rfc(submitqueue): add modular queue wiring

[JamyDev]

Summary

RFC at doc/rfc/submitqueue/modular-queue-wiring.md proposing a declare-don't-assemble engine for the orchestrator's per-queue extension wiring, topic registry, and controller lifecycle:

1. platform/lifecycle — Component interface (Start/Stop) and ordered Group with rollback-on-partial-failure
2. platform/pipeline — Construct[D] engine that unifies topic registry, controller creation, DLQ pairing, and consumer lifecycle into one typed call over a []Stage[D] table
3. Service self-declaration — each service exports a Deps struct (seams), a Stages slice (topology), and a Controllers struct; no assembly code
4. Host-private profiles — per-queue extension selection stays in the host's profiles.go, never crosses the library boundary
5. Fluent builder — submitqueue.New().WithQueue(...).Create() wraps the engine for the common single-orchestrator case

Recommends profiles stay in Go code (Option C) rather than adding hints to QueueConfig. Also adds the RFC to doc/rfc/index.md.

Doc-only change, no code modifications.

Test Plan

CI/Build validation

Revert Plan

git revert

API Changes

No API changes

Monitoring and Alerts

Standard monitoring via existing dashboards

[behinddwalls]

Design counter-proposal: declare, don't assemble

Full end-to-end walkthrough (every construct defined, every file shown step by step): gist — declare, don't assemble. Summary below.

The test that drove it: could another repo import submitqueue as a library and build an fx / plain-main / test-harness service from it?

The core move: the RFC's extractions (and our first drafts of this feedback) still leave each service implementing assembly — four copies kept aligned by convention and review. The counter-proposal: each service declares itself as data, one shared engine assembles.

piece	owner	form
engine — construction, lifecycle, DLQ pairing, start/stop ordering	platform/pipeline.Construct + platform/lifecycle — written once, tested once	code
service definition — Deps (seams) · Stages (queue controllers, one row = controller + topic + DLQ reconciler) · Controllers (RPC, unbound to any wire contract)	each domain pkg	data + plain funcs
deployment policy — impl selection (profiles.go), creds, topic names, classifiers, transport, signals	each main (our service/ mains become reference hosts)	host code

Highlights, each shown concretely in the gist:

• Adding a stage = one table row. Today it takes 3 hand-coordinated edits (topic config, registration, registerDLQControllers); "every consumed stage has a DLQ pair + reconciler" becomes true by construction instead of by review — just in time for the speculation stages.
• One lifecycle.Component handle; the library exposes no server, reads no env, traps no signals. An fx host maps 1:1 (each block an fx.Provide, Construct one fx.Invoke, lifecycle two fx.Hook lines).
• Per-queue impl variation is the existing Factory contract ("identity in, resolve internally") — profiles stay host-private in profiles.go, so core/queueprofile stays trigger-gated exactly per the in-code TODO.
• Controllers are the portable Go surface (proto messages, no server interface — binding to a proto service/transport is ~10 lines of host glue), and the committed api/ protos are the cross-language surface: other languages regenerate stubs from the same sources, one contract, two codegens.
• Not the DI framework this RFC rightly rejects: no runtime graph, no reflection, no topo-sort — a typed engine over declarative data; one-offs enter via Options.

The problem statement and the QueueConfig A/B/C analysis survive nearly verbatim — this mostly changes the extraction plan's shape. Happy to pair on the rewrite.

[albertywu]

@JamyDev @behinddwalls can we have something fluent like this?

app, err := submitqueue.New().
    WithStorage(store).
    WithMessageQueue(mq).
    WithQueue(
        submitqueue.Queue("go-code").
            WithChangeProvider(changeProvider).
            WithBuildRunner(buildkiteBuildRunner).
            WithScorer(defaultScorer).
            WithConflictAnalyzer(tango),
    ).
    WithQueue(
        submitqueue.Queue("java-code").
            WithChangeProvider(changeProvider).
            WithBuildRunner(buildkiteBuildRunner).
            WithScorer(defaultScorer).
            WithConflictAnalyzer(tango),
    ).
    Build()
if err != nil {
    return fmt.Errorf("build SubmitQueue app: %w", err)
}

if err := app.Start(ctx); err != nil {
    return fmt.Errorf("start SubmitQueue app: %w", err)
}
defer app.Stop(context.Background())

return app.Wait(ctx)

or if some config needs to come from YAML for practical reasons (Uber infra):

cfg := loadDeploymentConfig()

app, err := submitqueue.New().
    WithStorage(newStorage(cfg.Database)).
    WithMessageQueue(newMessageQueue(cfg.MessageQueue)).
    WithQueue(newGoCodeQueue(cfg.GoCode)).
    WithQueue(newJavaCodeQueue(cfg.JavaCode)).
    Build()

func newGoCodeQueue(cfg MonorepoConfig) submitqueue.QueueBuilder {
    return submitqueue.Queue("go-code").
        WithBuildRunner(
            buildkite.New(buildkite.Config{
                Endpoint: cfg.BuildEndpoint,
                Timeout:  cfg.BuildTimeout,
            }),
        ).
        WithScorer(
            heuristic.New(goCodeScoringBuckets),
        ).
        WithConflictAnalyzer(
            targets.NewGoAnalyzer(cfg.Repository),
        ).
        WithPrioritizer(
            sticky.New(cfg.MaxBuilds),
        )
}

[JamyDev]

I think we can add something on top to allow for a fluent builder yea

[behinddwalls]

we can drop With feels verbose