Skip to content

fix(python): use mimalloc as global allocator to fix RSS leak#7245

Open
mansiverma897993 wants to merge 2 commits into
lance-format:mainfrom
mansiverma897993:fix-mimalloc-arena-leak
Open

fix(python): use mimalloc as global allocator to fix RSS leak#7245
mansiverma897993 wants to merge 2 commits into
lance-format:mainfrom
mansiverma897993:fix-mimalloc-arena-leak

Conversation

@mansiverma897993

Copy link
Copy Markdown

Resolves #7242

This PR configures mimalloc as the global allocator in the Python library (pylance) to resolve an unbounded RSS memory leak caused by glibc per-thread arena fragmentation.

Root Cause

Under glibc, sequential merge_insert workloads on Tokio worker threads allocate temporary buffers. Glibc does not eagerly return pages from these per-thread arenas to the OS after they are freed, causing RSS memory to grow monotonically until the process is terminated by the OOM killer.

Solution

  • Added mimalloc dependency to python/Cargo.toml (with default-features = false to keep dependencies minimal).
  • Registered mimalloc::MiMalloc as the #[global_allocator] in python/src/lib.rs.

@github-actions github-actions Bot added the A-python Python bindings label Jun 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@mansiverma897993 mansiverma897993 changed the title Use mimalloc as global allocator in Python library to fix RSS leak fix(python): use mimalloc as global allocator to fix RSS leak Jun 12, 2026
@github-actions github-actions Bot added the bug Something isn't working label Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Apparent RSS memory leak in repeated merge_insert caused by glibc per-thread arena fragmentation

1 participant