Skip to content

PERF: preserve idx slice in EGI MFF _read_segment_file to avoid np.take on large buffers - #14137

Merged
scott-huberty merged 5 commits into
mne-tools:mainfrom
PragnyaKhandelwal:egi-mff-idx-slice-opt
Aug 7, 2026
Merged

PERF: preserve idx slice in EGI MFF _read_segment_file to avoid np.take on large buffers#14137
scott-huberty merged 5 commits into
mne-tools:mainfrom
PragnyaKhandelwal:egi-mff-idx-slice-opt

Conversation

@PragnyaKhandelwal

@PragnyaKhandelwal PragnyaKhandelwal commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Reference issue (if any)

Part of #13926 (Phase 4: Read Optimization).

What does this implement/fix?

_read_segment_file in egimff.py unconditionally converted idx from a slice to
an ndarray via np.arange before channel bounds-checking. This forced _mult_cal_one
into its slow path — np.take(one, idx, axis=0, out=data_view) (fancy indexing) —
instead of the fast path data_view[:] = one[idx] (contiguous slice copy).

Fix: introduce idx_arr (ndarray) for bounds-checking only; pass the original idx
(slice or ndarray) through to both _mult_cal_one call sites.

Profiling evidence (warm cProfile run, imports pre-loaded):

Bottleneck Before % of total
numpy.ndarray.take inside _mult_cal_one 7.97 s 60.6 %

The same bottleneck appears in preload=False repeated get_data() calls (5.69 s
across 10 calls). The fix eliminates np.take in both paths.

Additional information

AI disclosure: I profiled the reader using cProfile, line_profiler, and snakeviz to
identify the bottleneck. Claude (claude.ai) helped me interpret the profiling output and
understand the two code paths in _mult_cal_one. The fix and all testing were done by me.

@PragnyaKhandelwal PragnyaKhandelwal changed the title PERF: preserve idx slice in EGI MFF _read_segment_file for _mult_cal_… PERF: preserve idx slice in EGI MFF _read_segment_file to avoid np.take on large buffers Aug 5, 2026
@PragnyaKhandelwal
PragnyaKhandelwal marked this pull request as ready for review August 5, 2026 13:48
@scott-huberty

Copy link
Copy Markdown
Contributor

@PragnyaKhandelwal If before this took 7.97 s for your test file, what is the time after this optimization?

@PragnyaKhandelwal

PragnyaKhandelwal commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@PragnyaKhandelwal If before this took 7.97 s for your test file, what is the time after this optimization?

After the fix:
numpy.ndarray.take is completely gone from the profile. _mult_cal_one drops from
8.15s cumtime -> 0.95s

Remaining bottleneck is mffpy.get_physical_samples at 1.80s — that's actual binary
block reading + float32 calibration inside mffpy, which would require changes to mffpy itself.

image image

@scott-huberty

Copy link
Copy Markdown
Contributor

Oh Ok : ) what is the wall time of read_raw_egi (that's the function that calls this path right?) on main and on this PR? Using some test data or toy data.

@PragnyaKhandelwal

Copy link
Copy Markdown
Contributor Author

Oh Ok : ) what is the wall time of read_raw_egi (that's the function that calls this path right?) on main and on this PR? Using some test data or toy data.

Wall-clock timing on the same file (42-min, 128-ch MFF), warm imports, 3 runs each:

Branch Run 1 Run 2 Run 3 Mean
main 37.4s 21.8s 8.2s 22.5s
this PR 10.7s 16.2s 10.4s 12.4s

~45% faster mean. The variance across runs is OS disk page cache filling up — later runs benefit from the file being cached in RAM. The controlled warm cProfile comparison (pre-warmed, single run) showed 13.15s → 7.50s (43% faster), consistent with the wall-clock result.

@scott-huberty
scott-huberty enabled auto-merge (squash) August 7, 2026 19:32
@scott-huberty
scott-huberty merged commit 231288c into mne-tools:main Aug 7, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants