Add initial dhat heap profile tests

[cetra3]

Which issue does this PR close?

Related to #20714 but doesn't actually fix any issues at the moment, merely highlights that memory accounting is currently inaccurate

Here's a table of measured heap profiles based upon these tests:

Test	Operator	Pool	Peak	Ratio	Assert
heap_profile_repartition	RepartitionExec	10MB	20.7MB	1.97x	TODO
heap_profile_hash_join	HashJoinExec	40MB	44.4MB	1.06x	active
heap_profile_sort_merge_join	SortMergeJoinExec	40MB	33.3MB	0.79x	active
heap_profile_sort	SortExec	10MB	65.9MB	6.28x	TODO
heap_profile_hash_aggregate	GroupedHashAggregate	10MB	121.7MB	11.60x	TODO
heap_profile_window	WindowAggExec	10MB	33.4MB	3.18x	TODO
heap_profile_parquet_sort	Parquet + SortExec	20MB	66.7MB	3.18x	TODO

Rationale for this change

This adds some preliminary heap profile testing using dhat-rs to record heap allocations and report on memory usage for a handful of canned queries. Each of them allow some head room as there can be some overhead (1.1x) in other parts of the process.

While more test types can be added in the future, already most of these heap profile tests blow out memory usage.

For the failing tests, these are commented out with some information about why they might fail. Essentially we should raise PRs to fix memory accounting and then add the appropriate assertions.

What changes are included in this PR?

• Adds dhat as a dev dependency
• Creates a handful of tests for the heap profile.

Are these changes tested?

Yes, as they are only tests

Are there any user-facing changes?

None at this stage.

[comphead]

Thats awesome, thanks @cetra3, please help me to read the table correctly

heap_profile_repartition	RepartitionExec	10MB	20.7MB	1.97x

it means 10MB accounted by the internal memory but the real usage was 20.7?

[cetra3]

Yes, that's right. The memory limit on the datafusion test is 10mb, but the measured heap is actually twice that amount. You would expect there is a bit of overhead in data structures etc.. but double the amount of memory is far too much I think

[comphead]

Yes, that's right. The memory limit on the datafusion test is 10mb, but the measured heap is actually twice that amount. You would expect there is a bit of overhead in data structures etc.. but double the amount of memory is far too much I think

true, in Comet we also were struggling with unexpected OOMs

@andygrove @mbutrovich FYI

let me go through the approach

[martin-g]

I am not sure whether these new IT tests are going to stay but if they do then maybe the dhat dev-dependency should be optional and loaded only when some new feature is enabled, e.g. memory-profiling, and all IT tests be gated behind this feature.

[martin-g]

Here you start the profiling after the setup but in many of the other IT tests this is done before the setup

[martin-g]

The memory limit on the datafusion test is 10mb, but the measured heap is actually twice that amount

The limit is only for the FairSpillPool, not for the whole application.
DHat provides heap dump functionality and it could be used to find what consumes the bigger chunks of the used memory.