Skip to content

sketch out improved performance by refactoring codec pipeline logic #3719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
d-v-b wants to merge 38 commits into
base: main
Choose a base branch
from

Merge branch 'main' into perf/smarter-codecs

116e417
Select commit
Loading
Failed to load commit list.
Draft

sketch out improved performance by refactoring codec pipeline logic #3719

Merge branch 'main' into perf/smarter-codecs
116e417
Select commit
Loading
Failed to load commit list.
CodSpeed HQ / CodSpeed Performance Analysis succeeded Mar 11, 2026 in 0s

Performance Gate Passed

⚡ 59 improved benchmarks
✅ 7 untouched benchmarks
⏩ 6 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ WallTime test_sharded_morton_single_chunk[(32, 32, 32)-memory] 1,875.7 µs 687.2 µs ×2.7
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-gzip] 1,014.9 ms 285.3 ms ×3.6
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] 555.8 ms 193.7 ms ×2.9
⚡ WallTime test_slice_indexing[(50, 50, 50)-(0, 0, 0)-memory] 1,758 µs 537.8 µs ×3.3
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-None] 1,616.3 ms 596.8 ms ×2.7
⚡ WallTime test_slice_indexing[(50, 50, 50)-(0, 0, 0)-memory_get_latency] 4 ms 3.2 ms +26.19%
⚡ WallTime test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory_get_latency] 4.1 ms 2.2 ms +82.89%
⚡ WallTime test_sharded_morton_single_chunk[(30, 30, 30)-memory] 1,965 µs 651.9 µs ×3
⚡ WallTime test_slice_indexing[None-(slice(None, None, None), slice(0, 3, 2), slice(0, 10, None))-memory] 3.7 ms 1.1 ms ×3.4
⚡ WallTime test_sharded_morton_indexing_large[(33, 33, 33)-memory] 10.2 s 1.8 s ×5.8
⚡ WallTime test_sharded_morton_indexing_large[(30, 30, 30)-memory] 7.7 s 1.3 s ×5.8
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-gzip] 9.5 s 1.2 s ×7.7
⚡ WallTime test_slice_indexing[(50, 50, 50)-(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory] 418.6 ms 83.5 ms ×5
⚡ WallTime test_slice_indexing[(50, 50, 50)-(slice(0, None, 4), slice(0, None, 4), slice(0, None, 4))-memory] 413.4 ms 78.8 ms ×5.2
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(100,), shards=(1000000,))-None] 5,355.3 ms 253.9 ms ×21
⚡ WallTime test_sharded_morton_single_chunk[(33, 33, 33)-memory] 1,959.6 µs 703.4 µs ×2.8
⚡ WallTime test_slice_indexing[(50, 50, 50)-(slice(None, None, None), slice(None, None, None), slice(None, None, None))-memory_get_latency] 435.3 ms 117.6 ms ×3.7
⚡ WallTime test_sharded_morton_write_single_chunk[(30, 30, 30)-memory] 147,069.6 µs 727.5 µs ×200
⚡ WallTime test_write_array[local-Layout(shape=(1000000,), chunks=(1000,), shards=None)-None] 1,212.5 ms 626.5 ms +93.53%
⚡ WallTime test_write_array[memory-Layout(shape=(1000000,), chunks=(1000,), shards=(1000,))-gzip] 2,117.8 ms 712.4 ms ×3
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Comparing d-v-b:perf/smarter-codecs (116e417) with main (fa61ed8)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

View more details on CodSpeed HQ