[draft/wip] test(stovepipe): cover process pipeline integration

[mnoah1]

Summary

• strengthen the Stovepipe E2E happy path to verify the durable ingest -> process -> build outcome, not only a process-consumer acknowledgment
• add process integration coverage for cold-start admission, stale-head coalescing, idempotent redelivery, and non-retryable DLQ reconciliation
• centralize bounded polling, storage inspection, queue-offset observation, build-payload decoding, scenario seeding, and diagnostic progress logging in shared Stovepipe test helpers
• stop asserting transient process-message rows, which may be garbage-collected after acknowledgment

Behaviors covered

Cold-start admission and build handoff

A request ingested through the public gRPC API must reach processing, use the full-build strategy with no baseline, claim exactly one in-flight slot, retain the queue's latest-head pointer, and publish a decodable BuildRequest for the same request ID. This proves the process stage performed its business transition and downstream handoff; an ack by itself could also represent a deliberate no-op.

Stale-head coalescing

When an older accepted request is delivered after latest_request_id points to a newer request, the older request must become superseded, the newer request must remain accepted, and no build slot may be consumed. This protects queue capacity and prevents obsolete work from reaching build.

Idempotent redelivery

Delivering a distinct queue message that references an already-processing request must leave the request processing, keep in_flight_count at one, and retain exactly one build message. This verifies at-least-once delivery cannot double-claim capacity or duplicate downstream work.

Non-retryable failure and DLQ reconciliation

A valid process payload that fails request-ID comparison non-retryably must route through process_dlq; the DLQ consumer must reconcile the request to recorded_not_green, leave no slot claimed, and publish no build work. This verifies the real classifier, MySQL queue DLQ routing, and reconciliation consumer fail closed together.

Test plan

• ./tool/bazel test //test/integration/stovepipe:go_default_test //test/e2e/stovepipe:go_default_test --test_output=errors