feat: time-based alert retention policy

[aditya-786]

Summary

Closes #4802

Adds a time-based alert retention policy so the alert tables stop growing forever (#6524 reports a ~20GB postgres database after a month). When KEEP_ALERT_RETENTION_DAYS is set to a positive value, Keep periodically deletes alerts older than that many days, together with their incident links. The feature is disabled by default, so nothing changes for existing deployments.

This is a native, database-agnostic version of the MySQL cleanup CronJob referenced in the issue (helm-charts db-cleanup-cronjob.yaml), which does not work for postgres or sqlite deployments.

Env var	Purpose	Default
KEEP_ALERT_RETENTION_DAYS	Delete alerts older than this number of days (0 disables the feature)	0
KEEP_ALERT_RETENTION_INTERVAL	Seconds between retention runs	86400
KEEP_ALERT_RETENTION_BATCH_SIZE	Alerts deleted per batch	1000

How it works

• delete_alerts_by_retention in keep/api/core/db.py deletes in batches and commits per batch, so there are no long transactions and an interrupted run resumes cleanly on the next cycle. Deletion order per batch keeps foreign keys satisfied: lastalerttoincident, then lastalert, then alerttoincident, then alert.
• LastAlert rows are selected by alert_id rather than by timestamp: a fingerprint that is still active keeps its LastAlert (which points at a fresh alert) while its old alert rows are purged, and a fully stale fingerprint is removed entirely.
• keep/api/tasks/process_retention_task.py mirrors process_watcher_task.py: with Redis it runs through arq guarded by an NX lock, otherwise it runs as an asyncio loop guarded by a file lock. The arq cron is only registered when retention is enabled, since a destructive job should be opt-in.
• Per-status retention (the second part of the issue) needs alert status to be queryable at purge time and is left for a follow-up; this PR delivers the age-based policy the referenced cronjob already implements.

Two deliberate divergences from the cronjob, to keep behavior safe and the scope reviewable:

• alerttogroup is not touched because the table was dropped from the schema in 2024 and has no model anymore.
• Fingerprint-keyed tables (alertenrichment, alertaudit, alertraw, alertfield) are not purged. Enrichments are keyed by fingerprint and also serve incident and alert-instance enrichment, so deleting them based on alert age is not safe without more care.

Tests

tests/test_alert_retention.py, running on the standard db_session fixture:

• purges old alerts across multiple batches (batch_size=2 over 3 alerts) and removes their LastAlert, AlertToIncident and LastAlertToIncident rows while fresh alerts survive
• a fingerprint with old history but fresh activity keeps its LastAlert and latest alert while old rows are purged
• soft-deleted LastAlertToIncident rows are cleaned up for purged fingerprints
• no-op when nothing is older than the cutoff

[codecov]

Codecov Report

❌ Patch coverage is 22.72727% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.46%. Comparing base (98dbd7e) to head (56378f0).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
keep/api/core/db.py	10.52%	17 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6616      +/-   ##
==========================================
- Coverage   30.49%   30.46%   -0.04%     
==========================================
  Files         101      101              
  Lines       11741    11774      +33     
==========================================
+ Hits         3581     3587       +6     
- Misses       8160     8187      +27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aditya-786]

The patch coverage number is a measurement artifact rather than missing tests. The unit tests job runs coverage run -m pytest -n auto, and coverage.py does not follow pytest-xdist worker processes, so only lines executed in the controller process (imports and definitions) get recorded. That is why delete_alerts_by_retention shows 17 missing lines even though tests/test_alert_retention.py calls it directly in all 4 tests.

Reproduced both ways locally: with -n auto the report shows the same 17 missing lines, without it the new function reports 18 of 19 lines covered, the remaining one being the decorator line. If useful I can open a separate PR to capture worker coverage (pytest-cov, or COVERAGE_PROCESS_START with parallel mode), which would also make the project number reflect reality.