Search before asking
Motivation
Now that Paimon enables remote index files for data, we should also be able to use them where possible on the read path.
Some jobs have a capped local disk, and so being able to quickly fetch these back rather than rebuild them is very useful for making lookup joins lower latency. For now, we'll keep things simple - this should only run when lookup files are being persisted to object store, deletion vectors are off (i.e. MOR tables), and we can actually use the SSTable files persisted by compaction jobs (the join consumer wants the full row schema as opposed to a projection of the data).
Solution
No response
Anything else?
No response
Are you willing to submit a PR?
Search before asking
Motivation
Now that Paimon enables remote index files for data, we should also be able to use them where possible on the read path.
Some jobs have a capped local disk, and so being able to quickly fetch these back rather than rebuild them is very useful for making lookup joins lower latency. For now, we'll keep things simple - this should only run when lookup files are being persisted to object store, deletion vectors are off (i.e. MOR tables), and we can actually use the SSTable files persisted by compaction jobs (the join consumer wants the full row schema as opposed to a projection of the data).
Solution
No response
Anything else?
No response
Are you willing to submit a PR?