The new version of Data Lake Insight (DLI) offers the following new features:
- The process of submitting Spark jobs through ClusterAgent has been optimized by decoupling it into a standalone container execution. This enhancement makes ClusterAgent more independent and flexible, reducing its reliance on Spark JAR files.
- The cleanup mechanism for job tables has been strengthened to improve stability and reliability. Additionally, optimizations have been made to address the dependency coupling between orders and job tables. These improvements ensure that data in job tables is cleaned up more efficiently and accurately, preventing issues such as data redundancy and performance degradation caused by delayed or inaccurate cleanups.
- Added support for ODBC connections to DLI on 64-bitsystems.
- Introduced a memory dump method when the driver undergoes full garbage collection. This allows for quick diagnosis and resolution of potential memory-related issues during runtime, enhancing system reliability and stability.
- The auto-renewal mechanism for CCE client tokens has been reinforced to improve system security and user experience.
- Enabled seamless upgrades for the DLI CCE version without requiring business migration, ensuring uninterrupted operations.
- The management plane logic has been separated from the data processing logic, allowing both components to operate and scale independently. This reduces direct dependencies and enhances overall efficiency.
These issues were resolved:
- Link for redirecting to the user guide on the DLI management console is invalid.
- Error prompt issue when saving a Flink job.
- Queue tags were not recognized on TMS.
- Solved the job table aging problem and strengthened the aging mechanism.
- Fixed the failure of the Save As button after stopping a Flink Jar job with checkpointing enabled only through the console.
Further information can be found in Data Lake Insight (DLI) Help Center.