[feature-request] Prune Invalid Docs During Segment Commit for Upsert Tables

[ankitsultana]

While Upsert Compaction is great, for tables which have very high ingestion throughput, it'd be ideal to prune invalid docs during segment commit itself, since compaction is costly and in many cases not able to catch up. In one of our use-cases, I think this feature would lead to a further reduction in table size of 2-3x.

This should be relatively simple to do for Full Upsert tables but I think would be harder for Partial Upsert tables.

cc: @tibrewalpratik17