Skip to content

Batch triggerer unused trigger cleanup deletes#68269

Open
BeauDevCode wants to merge 1 commit into
apache:mainfrom
BeauDevCode:batch-trigger-clean-unused-deletes
Open

Batch triggerer unused trigger cleanup deletes#68269
BeauDevCode wants to merge 1 commit into
apache:mainfrom
BeauDevCode:batch-trigger-clean-unused-deletes

Conversation

@BeauDevCode

Copy link
Copy Markdown

closes: #68243

Why

Trigger.clean_unused() currently removes all unreferenced trigger rows in one unbounded delete. On busy deployments this can hold locks on the trigger table for the duration of a large cleanup and stall the triggerer loop.

What changed

  • Select unused trigger IDs before deleting, preserving the existing unused-trigger criteria.
  • Delete matching trigger rows in configurable batches, committing between batches.
  • Add [triggerer] unreferenced_triggers_cleanup_batch_size, defaulting to 500; setting it to 0 keeps single-transaction cleanup behavior.
  • Add a unit test that uses a batch size of 2 and verifies five unused triggers are removed across three committed delete batches.

Related context: PR #68244 changes the anti-join shape in the same method, but does not add bounded delete batches. This PR intentionally focuses on the batching behavior requested in #68243.

Tests

  • git diff --check
  • python3 -m compileall -q airflow-core/src/airflow/models/trigger.py airflow-core/tests/unit/models/test_trigger.py
  • Parsed airflow-core/src/airflow/config_templates/config.yml with PyYAML and verified the new option shape

Not run locally: focused pytest/config validation could not complete because this environment lacks parts of Airflow's development dependency set (pathspec during test collection; requests for dev/validate_version_added_fields_in_config.py).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch the triggerer's Trigger.clean_unused() cleanup deletes

1 participant