Skip to content

Add eval seed timing logs and devservices mode#115523

Draft
JoshFerge wants to merge 2 commits into
masterfrom
jferg-codex/eval-seed-timing-logs
Draft

Add eval seed timing logs and devservices mode#115523
JoshFerge wants to merge 2 commits into
masterfrom
jferg-codex/eval-seed-timing-logs

Conversation

@JoshFerge
Copy link
Copy Markdown
Member

@JoshFerge JoshFerge commented May 13, 2026

Summary

  • add an evals devservices mode for seeded local evals
  • keep the mode focused on error-event eval ingestion: Relay, Snuba/Postgres, taskbroker, ingest-events, post-process-forwarder-errors, and dedicated taskworkers for relay, ingest.errors, ingest.errors.postprocess, and issues
  • add seer_eval_seed.* timing logs for eval project ingest, save, post-process/code-mapping enqueue, polling endpoints, auto source-code config, and Relay project-config builds
  • scope logs to eval-* projects where the code path has project context

Why

This is a draft/debug PR for the local seeded Seer eval bottleneck. The logs showed ingest-events consuming the 50 events within a few seconds, while sentry.tasks.store.save_event could start roughly 63s later because ingest.errors work was competing with issues tasks on the catch-all taskworker.

The new evals mode moves the queue topology into Sentry instead of requiring Seer to run manual taskworker commands. It avoids the broad ingest mode’s extra transactions/attachments workers and avoids a generic catch-all taskworker for the eval path.

Verification

  • Sentry pre-commit hooks on commit and push
  • git diff --check
  • .venv/bin/prek run -q check-yaml --files devservices/config.yml
  • devservices config loader check for evals mode and supervisor dependency types
  • Used local logs to identify:
    • ingest consumer queue lag around 3.2-3.8s in the slow run
    • save_event since_received_seconds around 63s in the slow run
    • improved seed probe after dedicated workers: 11.93s, then 18.72s

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 13, 2026
@JoshFerge JoshFerge changed the title Add eval seed timing logs Add eval seed timing logs and devservices mode May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant