-
Notifications
You must be signed in to change notification settings - Fork 252
Open
Description
I am processing some LFP on 60s chunks (100s for illustration here). I switched from fork to forkserver in multiprocessing and I get some warnings about the chunks being 1s when I save the array.
write_binary_recording
engine=process - n_jobs=8 - samples_per_chunk=72,000 - chunk_memory=105.47 MiB - total_memory=843.75 MiB - chunk_duration=60.00s (1.00 minutes)
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/datajoint/plugin.py:4: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources # requires setuptools<82
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/spikeinterface/preprocessing/filter.py:122: UserWarning: The margin size (25000 samples) is more than 20% of the global chunk size 2500 samples. This may lead to performance bottlenecks when chunking. Consider increasing the chunk_size or chunk_duration to minimize margin overhead.
warnings.warn(
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/spikeinterface/preprocessing/filter.py:122: UserWarning: The margin size (25000 samples) is more than 20% of the global chunk size 2500 samples. This may lead to performance bottlenecks when chunking. Consider increasing the chunk_size or chunk_duration to minimize margin overhead.
warnings.warn(
write_binary_recording (workers: 2 processes forkserver): 100%|██████████| 2/2 [00:24<00:00, 12.10s/it]
I have set the global kwargs at the start of the processing, the initial line shows the correct chunk size (60s) and it is taking the same time as before so I actually think it might be the warning that is wrong.
Note: The datajoint-related warning is a line I get everytime I run the script.py file I am using. When using forkserver a new process is created to be the server of processes so I imagine that's where that is printed.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels