Skip to content

Track worker occupancy in stats - #377

Draft
neob91-close wants to merge 1 commit into
masterfrom
andrzej/p2/worker-duty-cycle-stats
Draft

Track worker occupancy in stats#377
neob91-close wants to merge 1 commit into
masterfrom
andrzej/p2/worker-duty-cycle-stats

Conversation

@neob91-close

Copy link
Copy Markdown
Contributor

The utilization stat counts only time inside task code, so a worker saturated with many short tasks reads well below 100% — the dequeue/scan/lock overhead between tasks is invisible. This broke utilization-based autoscaling for us: workers scaled down during a large backlog.

Adds time_idle (blocking waits for new work, measured via measure_idle() around the two wait sites), and emits time_overhead and occupancy = 100 * (total - idle) / total — the "can this worker take more work" signal. Purely additive: utilization and time_busy keep their meaning.

Caveat for consumers that monkey-patch StatsThread (custom stats adapters): the worker now calls measure_idle()/report_idle_*, so adapters need those methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant