You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
_read_segment_file in egimff.py unconditionally converted idx from a slice to
an ndarray via np.arange before channel bounds-checking. This forced _mult_cal_one
into its slow path — np.take(one, idx, axis=0, out=data_view) (fancy indexing) —
instead of the fast path data_view[:] = one[idx] (contiguous slice copy).
Fix: introduce idx_arr (ndarray) for bounds-checking only; pass the original idx
(slice or ndarray) through to both _mult_cal_one call sites.
The same bottleneck appears in preload=False repeated get_data() calls (5.69 s
across 10 calls). The fix eliminates np.take in both paths.
Additional information
AI disclosure: I profiled the reader using cProfile, line_profiler, and snakeviz to
identify the bottleneck. Claude (claude.ai) helped me interpret the profiling output and
understand the two code paths in _mult_cal_one. The fix and all testing were done by me.
PragnyaKhandelwal
changed the title
PERF: preserve idx slice in EGI MFF _read_segment_file for _mult_cal_…
PERF: preserve idx slice in EGI MFF _read_segment_file to avoid np.take on large buffers
Aug 5, 2026
@PragnyaKhandelwal If before this took 7.97 s for your test file, what is the time after this optimization?
After the fix: numpy.ndarray.take is completely gone from the profile. _mult_cal_one drops from
8.15s cumtime -> 0.95s
Remaining bottleneck is mffpy.get_physical_samples at 1.80s — that's actual binary
block reading + float32 calibration inside mffpy, which would require changes to mffpy itself.
Oh Ok : ) what is the wall time of read_raw_egi (that's the function that calls this path right?) on main and on this PR? Using some test data or toy data.
Oh Ok : ) what is the wall time of read_raw_egi (that's the function that calls this path right?) on main and on this PR? Using some test data or toy data.
Wall-clock timing on the same file (42-min, 128-ch MFF), warm imports, 3 runs each:
Branch
Run 1
Run 2
Run 3
Mean
main
37.4s
21.8s
8.2s
22.5s
this PR
10.7s
16.2s
10.4s
12.4s
~45% faster mean. The variance across runs is OS disk page cache filling up — later runs benefit from the file being cached in RAM. The controlled warm cProfile comparison (pre-warmed, single run) showed 13.15s → 7.50s (43% faster), consistent with the wall-clock result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference issue (if any)
Part of #13926 (Phase 4: Read Optimization).
What does this implement/fix?
_read_segment_fileinegimff.pyunconditionally convertedidxfrom aslicetoan
ndarrayvianp.arangebefore channel bounds-checking. This forced_mult_cal_oneinto its slow path —
np.take(one, idx, axis=0, out=data_view)(fancy indexing) —instead of the fast path
data_view[:] = one[idx](contiguous slice copy).Fix: introduce
idx_arr(ndarray) for bounds-checking only; pass the originalidx(slice or ndarray) through to both
_mult_cal_onecall sites.Profiling evidence (warm cProfile run, imports pre-loaded):
numpy.ndarray.takeinside_mult_cal_oneThe same bottleneck appears in
preload=Falserepeatedget_data()calls (5.69 sacross 10 calls). The fix eliminates
np.takein both paths.Additional information
AI disclosure: I profiled the reader using cProfile, line_profiler, and snakeviz to
identify the bottleneck. Claude (claude.ai) helped me interpret the profiling output and
understand the two code paths in
_mult_cal_one. The fix and all testing were done by me.