Skip to content

Paginate delete blockers to prevent OOM#8018

Open
acwhite211 wants to merge 4 commits intomainfrom
issue-7515
Open

Paginate delete blockers to prevent OOM#8018
acwhite211 wants to merge 4 commits intomainfrom
issue-7515

Conversation

@acwhite211
Copy link
Copy Markdown
Member

Fixes #7515
Contributed by @foozleface
Based on #7939 with additional fixes.

The delete_blockers endpoint used Django's Collector.collect() which loads ALL related objects into memory before returning. For a Taxon like "Life" with 900K+ determinations, this causes out-of-memory kills. This PR replaces the Collector with targeted COUNT queries and limited ID lists per PROTECT relationship, keeping memory bounded regardless of how many blocking rows exist.

Implementation

  • Replace Collector.collect() in _collect_delete_blockers with iteration over obj._meta.related_objects, filtering for protect_with_blockers on_delete
  • For each PROTECT relationship, run a COUNT query and fetch at most id_limit (default 100) primary keys
  • Add total_count field to the response so the frontend knows the full count even when IDs are capped
  • Update frontend test fixtures to match the new isstrict: undefined serialization
  • Add comprehensive backend tests for the pagination behavior

Testing instructions

  • Navigate to a record with many dependent records (e.g., a high-level Taxon node) and attempt to delete it
  • Verify the delete blocker dialog shows the correct count and a reasonable number of linked record IDs
  • Confirm no OOM or timeout on records with very large numbers of blockers
  • Run the test suite: python manage.py test specifyweb.backend.delete_blockers.tests.test_pagination

Copy link
Copy Markdown
Collaborator

@emenslin emenslin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Navigate to a record with many dependent records (e.g., a high-level Taxon node) and attempt to delete it
  • Verify the delete blocker dialog shows the correct count and a reasonable number of linked record IDs
  • Confirm no OOM or timeout on records with very large numbers of blockers
  • Run the test suite: python manage.py test specifyweb.backend.delete_blockers.tests.test_pagination

Delete blockers and linked records don't seem to be working at all

Issue branch:

04-27_12.52.mp4

Main:

Image

On https://hujinnhc20260129-issue-7515.test.specifysystems.org/specify/ in the Vascular collection as it has a large tree.

@github-project-automation github-project-automation Bot moved this from 📋Back Log to Dev Attention Needed in General Tester Board Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Dev Attention Needed

Development

Successfully merging this pull request may close these issues.

[Large Databases]: Paginate the delete blockers

3 participants