Skip to content

Structured node-failure capture: target/_aeb/_failures.jsonl (code/node/phase/stderr) #9

Description

@paul-hammant

Motivation

blazerules never lets a bad record silently vanish. A malformed/type-bad input is diverted to a dead-letter record carrying the error code, the offending column_name, and a message — enough context to debug the producer, without polluting the hot path.

aeb's parallel: when a node fails under make -jN -k, the failure context is scattered. The driver collects a .rc marker per node, but the why (which toolchain invocation, the captured stderr) is interleaved into make's output and lost to any structured consumer. A CI system, a dashboard, or a follow-up aeb invocation has no single place to read "what failed and why."

This is a cheap, high-value win precisely because the .rc-marker infrastructure already exists in tools/aeb-driver.ae — we're adding a richer sibling record, not new orchestration.

Proposal

On any node failure, the driver appends a structured record to target/_aeb/_failures.jsonl (one JSON object per line):

{"label":"foo/.build.ae:test","node":"foo/.build.ae","tag":"test","rc":1,"phase":"link","command":"aetherc ...","stderr_tail":"...last N lines...","ts_ms":...}
  • Written alongside the existing .rc marker the driver already collects — same collection point, richer payload.
  • stderr_tail is bounded (last N lines) so the file stays small on a wide -k run with many failures.
  • File is truncated at the start of each build (like other target/_aeb/ scratch), so it reflects the last run only.
  • Rendered by the telemetry block (a FAIL summary can point at it) and consumable by any downstream tool — the read-from-artifacts pattern, same as _edges.txt.

Acceptance criteria

  • A failing node under -jN -k produces one line in target/_aeb/_failures.jsonl with label, node, tag, rc, and a bounded stderr tail.
  • Multiple concurrent failures each append their own line without corruption (append-only, one write per node).
  • File is reset at build start.
  • AEB_JOBS=1 sequential path writes the same records.
  • The [telemetry] FAIL render references the file when it's non-empty.

Not being asked

Not a dead-letter queue or retry mechanism — just structured capture of failure context. No new exec; the driver already has the rc + the child's output.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions