You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The concept of compaction traditionally refers to the process of making something denser or more tightly packed. In its current implementation, the Upsert-Compaction task in Apache Pinot operates at the segment level, where it rebuilds individual segments by removing unused or invalid rows. This approach has proven highly effective in controlling the disk usage of upsert tables.
However this task focuses on addressing the issue of the continuously growing number of segments in upsert tables. To mitigate this challenge, we propose a multi-segment compaction model for upsert tables. In this model, multiple segments will be combined and re-uploaded as a single, consolidated segment, with invalid or unused rows removed. This approach aims to reduce the overall segment count while maintaining the storage efficiency benefits of the current upsert-compaction mechanism.
Sharing the design doc here for review and feedback from the community.
The text was updated successfully, but these errors were encountered:
The concept of compaction traditionally refers to the process of making something denser or more tightly packed. In its current implementation, the Upsert-Compaction task in Apache Pinot operates at the segment level, where it rebuilds individual segments by removing unused or invalid rows. This approach has proven highly effective in controlling the disk usage of upsert tables.
However this task focuses on addressing the issue of the continuously growing number of segments in upsert tables. To mitigate this challenge, we propose a multi-segment compaction model for upsert tables. In this model, multiple segments will be combined and re-uploaded as a single, consolidated segment, with invalid or unused rows removed. This approach aims to reduce the overall segment count while maintaining the storage efficiency benefits of the current upsert-compaction mechanism.
Sharing the design doc here for review and feedback from the community.
The text was updated successfully, but these errors were encountered: