Skip to content

Wrong (?) warning about margins when using forkserver (vs fork) #4469

@ecobost

Description

@ecobost

I am processing some LFP on 60s chunks (100s for illustration here). I switched from fork to forkserver in multiprocessing and I get some warnings about the chunks being 1s when I save the array.

write_binary_recording 
engine=process - n_jobs=8 - samples_per_chunk=72,000 - chunk_memory=105.47 MiB - total_memory=843.75 MiB - chunk_duration=60.00s (1.00 minutes)
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/datajoint/plugin.py:4: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources  # requires setuptools<82
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/spikeinterface/preprocessing/filter.py:122: UserWarning: The margin size (25000 samples) is more than 20% of the global chunk size 2500 samples. This may lead to performance bottlenecks when chunking. Consider increasing the chunk_size or chunk_duration to minimize margin overhead.
  warnings.warn(
/home/ecobos/enigma-ephys/.venv/lib/python3.14/site-packages/spikeinterface/preprocessing/filter.py:122: UserWarning: The margin size (25000 samples) is more than 20% of the global chunk size 2500 samples. This may lead to performance bottlenecks when chunking. Consider increasing the chunk_size or chunk_duration to minimize margin overhead.
  warnings.warn(
write_binary_recording (workers: 2 processes forkserver): 100%|██████████| 2/2 [00:24<00:00, 12.10s/it]

I have set the global kwargs at the start of the processing, the initial line shows the correct chunk size (60s) and it is taking the same time as before so I actually think it might be the warning that is wrong.

Note: The datajoint-related warning is a line I get everytime I run the script.py file I am using. When using forkserver a new process is created to be the server of processes so I imagine that's where that is printed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions