Skip to content

Monitor by tag_columns (defaults with name) - #117

Draft
BetterBoyer wants to merge 3 commits into
mainfrom
monitor_name_groupings_and_owners
Draft

Monitor by tag_columns (defaults with name)#117
BetterBoyer wants to merge 3 commits into
mainfrom
monitor_name_groupings_and_owners

Conversation

@BetterBoyer

Copy link
Copy Markdown

What

All delayed.job.* monitor metrics are now grouped by — and tagged with — a configurable set of job columns, in addition to priority and queue. The default is the job's name (when the jobs table has the name column), so out of the box every metric series answers "which job?":

# delayed.job.count emits one series per (priority, queue, name):
{ priority: 'interactive', queue: 'default', name: 'SimpleJob', value: 1 }
{ priority: 'interactive', queue: 'default', value: 0 }  # untagged zero baseline, always emitted

The tagged columns are configurable, including custom columns an application adds to its jobs table:

# example with jobs table having both `name` and `owner` columns
Delayed::Monitor.tag_columns = %i(name owner)  # or [] to disable tagging entirely

Why

When delayed.job.max_age alerts, a common first question is "which job is stuck?" (and often "whose is it?"). Previously that required querying the database by hand. Now it's a max by {queue, priority, name} away in the metrics provider.

How

  • grouped_query accepts extra_group_columns, threaded into both the inner and outer GROUP BYs, so tagging costs no additional queries — still one query each for live/pending/failed counts per run
  • Delayed::Monitor.tag_columns= validates columns against the schema and raises ArgumentError at assignment (i.e. at boot when set in an initializer) rather than failing later in the monitor run loop
    ->-> Note this means setting it in an initializer requires a DB connection at boot <-<-
  • The default resolves to %i(name) only when the name column exists, so older schemas need no configuration and no rollout — they just emit untagged metrics until migrated
  • NULL tag values are reported as 'unset' (e.g. jobs enqueued before the column existed)

Behavioral notes / compatibility

  • No new event names, and no removed ones
    • Existing subscribers keep working. But by default every metric's cardinality now scales with the number of distinct job names, and untagged zero baselines are emitted for every (priority, queue) regardless of whether tagged series exist
    • Dashboards aggregating these gauges should sum by / max by accordingly
  • The README.md documents the behavior, the cardinality tradeoff, and a three-deploy rollout guide for adding a custom tag column

Testing

  • Ensured the full test suite passes after including coverage for:
    • multiple names per (priority, queue)
    • empty tag_columns
    • schema without a name column
    • 'unset' reporting
    • custom columns
    • assignment-time validations
  • Updated SQL snapshots for the new GROUP BY shape across sqlite/mysql/postgres

Other notes

Initially, the idea was to just add the ability to tag ONLY by name. This is captured in the first commit. Then the thinking evolved to enable fully customizable (metric, tag column) configurations. This was rejected as it makes more sense to have each tag column emitted across all metrics which exist instead as it seemed unlikely we would want to configure max_age_by_name and not, for example, failed_count_by_name. The final commit is where this effort landed so that each tagged column is emitted against each metric instead. Noting this evolution in case it (notably the second commit) helps highlight future capabilities that might be desired.

@BetterBoyer BetterBoyer changed the title Monitor name groupings and owners Monitor by tag_columns (defaults with name) Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant