feat: report retryable scheduler outcome categories#831
Conversation
Greptile SummaryThis PR adds observability to the async scheduler's retryable-error handling: retryable exceptions are now classified into four categories (rate_limit, timeout, internal_server, connection), rolling and cumulative counts are tracked, first-occurrence per-kind warnings are emitted without logging provider bodies, and all metrics are included in the scheduler's health/completion diagnostics.
|
| Filename | Overview |
|---|---|
| packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py | Adds retryable-outcome classification, rolling/cumulative counters, and first-occurrence warnings; non-retryable LLM failures are incorrectly bucketed as "success" in the new metrics. |
| packages/data-designer-engine/tests/engine/dataset_builders/test_async_scheduler.py | New parametrized test validates error-kind classification and sanitized output; does not cover the non-retryable-exception call path (retryable=False with exc set). |
| pyproject.toml | Adds explicit workspace sources for data-designer packages in [tool.uv.sources]; straightforward dependency configuration change. |
Sequence Diagram
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Task as LLM Task
participant Exec as _execute_task_inner
participant Record as _record_retryable_outcome
participant Counts as _retryable_outcome_counts
participant Rolling as _recent_retryable_kinds
alt Task succeeds
Task-->>Exec: return result
Exec->>Record: "retryable=False, exc=None"
Record->>Record: "kind = success"
Record->>Counts: "counts[success] += 1"
Record->>Rolling: append(success)
else Retryable failure
Task-->>Exec: raise ModelRateLimitError
Exec->>Record: "retryable=True, exc=retryable_exc"
Record->>Record: "kind = rate_limit etc"
Record->>Counts: "counts[kind] += 1"
Record->>Rolling: append(kind)
else Non-retryable failure
Task-->>Exec: raise GenerationValidationFailureError
Exec->>Record: "retryable=False, exc=non_retryable_exc"
Record->>Record: "kind = success mis-labeled"
Record->>Counts: "counts[success] += 1"
Record->>Rolling: append(success)
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Task as LLM Task
participant Exec as _execute_task_inner
participant Record as _record_retryable_outcome
participant Counts as _retryable_outcome_counts
participant Rolling as _recent_retryable_kinds
alt Task succeeds
Task-->>Exec: return result
Exec->>Record: "retryable=False, exc=None"
Record->>Record: "kind = success"
Record->>Counts: "counts[success] += 1"
Record->>Rolling: append(success)
else Retryable failure
Task-->>Exec: raise ModelRateLimitError
Exec->>Record: "retryable=True, exc=retryable_exc"
Record->>Record: "kind = rate_limit etc"
Record->>Counts: "counts[kind] += 1"
Record->>Rolling: append(kind)
else Non-retryable failure
Task-->>Exec: raise GenerationValidationFailureError
Exec->>Record: "retryable=False, exc=non_retryable_exc"
Record->>Record: "kind = success mis-labeled"
Record->>Counts: "counts[success] += 1"
Record->>Rolling: append(success)
end
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py:1571
**Non-retryable failures misclassified as "success"**
`kind` is always `"success"` when `retryable=False`, but the exception handler at line 1751 also calls `_record_retryable_outcome(retryable=False, exc=exc)` for non-retryable LLM failures (auth errors, schema errors, code bugs). Those failures increment `_retryable_outcome_counts["success"]` and are appended to `_recent_retryable_kinds` as `"success"`, so the diagnostics inflate the success count and understate actual failure rates. An operator seeing `cumulative_counts: {success: 80, rate_limit: 20}` cannot tell whether the 80 represent genuine successes or a mix of successes and non-retryable failures.
Reviews (1): Last reviewed commit: "Declare Data Designer workspace sources" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| rate. Only retryable errors (rate-limit, timeout, 5xx, connection) count | ||
| toward the rate; non-retryable failures register as 0. | ||
| """ | ||
| kind = self._retryable_error_kind(exc) if retryable else "success" |
There was a problem hiding this comment.
kind is always "success" when retryable=False, but the exception handler at line 1751 also calls _record_retryable_outcome(retryable=False, exc=exc) for non-retryable LLM failures (auth errors, schema errors, code bugs). Those failures increment _retryable_outcome_counts["success"] and are appended to _recent_retryable_kinds as "success", so the diagnostics inflate the success count and understate actual failure rates. An operator seeing cumulative_counts: {success: 80, rate_limit: 20} cannot tell whether the 80 represent genuine successes or a mix of successes and non-retryable failures.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py
Line: 1571
Comment:
**Non-retryable failures misclassified as "success"**
`kind` is always `"success"` when `retryable=False`, but the exception handler at line 1751 also calls `_record_retryable_outcome(retryable=False, exc=exc)` for non-retryable LLM failures (auth errors, schema errors, code bugs). Those failures increment `_retryable_outcome_counts["success"]` and are appended to `_recent_retryable_kinds` as `"success"`, so the diagnostics inflate the success count and understate actual failure rates. An operator seeing `cumulative_counts: {success: 80, rate_limit: 20}` cannot tell whether the 80 represent genuine successes or a mix of successes and non-retryable failures.
How can I resolve this? If you propose a fix, please make it concise.
Code Review — PR #831: Report retryable scheduler outcome categoriesSummaryThis PR extends
The sanitization approach (extract only transport metadata, never provider free Findings1. Non-retryable failures are bucketed as
|
danecor
changed the title
1 participant
Summary
Behavior
Retryability, task deferral, salvage, AIMD, and early-shutdown behavior are unchanged. The change is observability-only.
Validation
Big Iron pins this branch by immutable commit while the change is reviewed.