146: Stand up factory-test-infra: Cloudflare-based ephemeral verification environments (Workers for Platforms + Containers) + factory provider wiring - #182
Conversation
|
End-to-end merge blocker from the Factory-wide verification pass:
I am marking this draft so the current provider/unit-test green cannot be read as satisfying the issue's explicit real-environment acceptance criteria. Ready-to-merge evidence needs the infrastructure repository and scripts, a trusted real-Cloudflare workflow run with teardown artifacts, every guardrail assertion demonstrated (including failure when removed), current-main local/hosted suites green, and zero unresolved review threads. |
…force-factory-80c19421
|
Addressed the blocker in the coupled implementation now published as AgentWorkforce/factory-test-infra#2. Concrete fixes:
Validation on these heads: Factory build + 71 files/1,332 tests + CLI/featuremap + tier-2 dry-run/status; infra typecheck + 14 tests + Wrangler Container/reaper dry-runs + Terraform fmt/validate + actionlint; exact Miniflare up/run-suite/down with 403 cross-environment isolation, quota/concurrency refusals, cost/time aborts, orphan deletion, 200 requests/0 errors, and idempotent teardown. The remaining acceptance item is intentionally not overstated: a trusted owner must supply the protected Cloudflare Resource secret and run the real job so the dispatch namespace, Container, k6, and teardown artifacts exist. |
|
Final safe-work update before the owner-gated real run:
The real Cloudflare job is the sole remaining acceptance proof. The dedicated repository currently has no factory-verification environment or cloudflare_resource_json secret, and this runner has no Cloudflare Resource credentials, so I have not represented the skipped real job as success. Happy to walk an owner through the Resource shape, protection settings, expected artifacts, or any implementation trade-off before that run. |
…force-factory-80c19421
…force-factory-80c19421
|
Current-base verification update:
The protected real-Cloudflare job remains the only unmet issue acceptance item; neither repository secrets nor this runner currently provide the required Resource credentials. |
Stale-PR triage: CLOSE — subsumed by #194, not obsoleted by mainTriage sweep of the 6 open PRs, evaluated against To be precise about the grounds: nothing on main superseded this PR. There is no Cloudflare implementation on main at all, and These are competing implementations, not complementary onesBoth branches diverged from the same base and neither is an ancestor of the other: Both add #194 is the superset: 1405-line provider vs 758 here; it has a concrete This branch's Cloudflare path has never been executedThe workflow this PR adds, The green Do not read #146's closure as "this work is done"#146 shows Factory has no PR #2895 — its numbering is in the ~270s. PR #2895 is in Salvage before closingThe genuinely unique content here is not the provider — it is the scaffolding:
Worth lifting into #194 or a follow-up. Note the private For the record, the rebase cost here is trivial — one conflict block in a 48-line file ( Recommendation only — I do not close or merge. Posted by an automated triage lane. |
Cross-PR triage summary (all 6 stale PRs)Recording the full result here as well as by DM, because the DM channel proved unreliable during this run (details at the end) and this surface is read. Per-PR evidence is in the individual comments on #207, #204, #199, #194, #182 and #178. Baseline for every claim below:
Every rebase was tested in a dedicated detached worktree and removed afterwards. No
One finding that generalises beyond these PRsTwo of the six PRs track issues that are marked
This repository has no PRs at those numbers — its range is in the ~270s. Both belong to a different repository in the org, and both are RelayAuth changes with no relationship to Cloudflare ephemeral environments (#146) or teammate discovery (#139). That is a cross-repo issue/PR number collision in the reconciliation path: a merge in one repo closed unrelated open issues in another. It is worth checking whether this is a one-off or generic, because if it is generic then other issues closed since 2026-08-13 may be equally bogus — and a falsely-closed issue is invisible work, not finished work. I am a triage lane with a comment-only mandate, so I have not filed this; flagging it for whoever owns the reconciliation path. Practical consequence for this triage: I did not treat either closure as evidence that #182's or #178's work was delivered, and neither should anyone reading those issues. DM reliability noteMy agent token was invalidated repeatedly between being issued and being used — Two things worth knowing from that: agent listing works without a valid token, which is the only reason I could recover my own identity; and a lane can finish its entire assignment and then silently lose the ability to report it — which is indistinguishable, from the outside, from a lane that died. Recommendations only — I do not close or merge. Posted by an automated triage lane. |
Triage: CLOSE — this is a second, divergent implementation of the same file as #194Stale-PR triage pass. Recommendation only; no ruling has been made yet. Evidence against 1. Is the work already on main by another route? — No, but it is duplicated by an open sibling PR.This is the decisive finding. #182 and #194 both add
Neither branch is an ancestor of the other ( 2. Does the parent issue still describe something we want? — Its two structural decisions have both been reversed by what shipped.#146 argued two things explicitly:
Separately, #146's blocking note — " 3. How big is the conflict? — Smallest of the four. 1 file, 82 commits behind.
This is a ~10-minute rebase. It is still not a reason to land it, because cheap-to-rebase and correct-to-merge are different questions, and this one duplicates #194. Also inherited from #194The same credential wall applies. The RecommendationCLOSE — as superseded by #194, not as work rejected. Rationale:
Note this recommendation is conditional on #194's fate: if #194 is converted to an issue (my recommendation there), then #182 should be closed into that same issue rather than into #194. Either way #182 should not be the surviving branch. One correction to the framing this was triaged under: #268 is about running Factory dispatch on a cloud node (Daytona, fresh-sandbox-per-agent, for agent execution hosts). #182/#194 are about ephemeral verification environments (Cloudflare, for bringing up a repo's stack to test it). Related in spirit, but #268 does not supersede these and its "fresh sandbox per agent" correction does not directly retarget them. What actually retargets them is the Kubernetes provider shipping on main. I'd rather say that plainly than lean on #268. Holding for a ruling — not closing anything on my own. |
|
Closing under the accepted stale-PR ruling. Parent #146 remains open and now records why this approach was abandoned. This is a second, divergent implementation of the same Continue the current-platform verification-environment work in #146. |
Part of Epic: Factory Verification & Environments (#141). This is the tier-5/6 data-plane home. The provider (#142) and gate (#145) run on what this stands up. Best done first in Track C.
Decision: Cloudflare-first, in a dedicated
factory-test-infrarepofactoryis a library (@agent-relay/factory) — it must not own or operate the environment substrate. The org is already all-in on Cloudflare + SST + wrangler + workerd/miniflare, so tier-5/6 verification environments run on Cloudflare, not a bespoke Kubernetes cluster:@cloudflare/vitest-pool-workers/ miniflare (cloud already runs this —test:integration,miniflare-sage).Kept in its own repo (not merged into cloud's
infra/) so the spiky, ephemeral verification account/config is isolated from product infra for blast-radius + cost accounting. K8s/Daytona remain optional escape-hatch providers behind the sameEnvironmentProviderport (#142) for the minority of target stacks that genuinely can't run on Cloudflare — Cloudflare Containers is not a general Kubernetes replacement for arbitrary heavy container stacks.What (in
factory-test-infra)verification-stackdescriptors (Verification-stack descriptor contract + deployer: repos declare their full stack, Factory brings it up #143) render into.What (in
factory)CloudflareEnvironmentProvider([factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) reads thefactory-test-infraaccount/API-token config (via aResource.*-style secret, notprocess.env); a reusable CI workflow that runs the Track-C E2E suites ([factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142–Verification gate: provision → deploy → E2E → load → evaluate → teardown, wired into the merge gate #145) against miniflare + a real dispatch namespace, and tears down.Acceptance
factory-test-infrarepo exists with Cloudflare account/config as code (dispatch namespace(s), Containers, bindings) documented + scripted; miniflare path for CI.CloudflareEnvironmentProviderconnects; reusable CI workflow runs [factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142–Verification gate: provision → deploy → E2E → load → evaluate → teardown, wired into the merge gate #145 suites and tears down.✅ End-to-end verification — Factory success criteria (REQUIRED)
"Compiles" is not done. Provision a real dispatch namespace + Container, prove the guardrails bind, and prove CI runs the suite end to end.
Run it
The E2E must prove the guardrails bind (not just exist):
Deliverable
factory-test-infraci/verification-env/{up,run-suite,down}.sh+ reusable CI workflow, one-command.Anchor points
factory-test-infra:infra/(SST or wrangler + Terraform Cloudflare provider),containers/(Container service defs),ci/verification-env/(up/run/down + miniflare), Cron-Trigger reaper worker.factory:src/environments/cloudflare-provider.ts(from [factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) reads the account/token;src/config/schema.tsenv config;.github/workflows/reusable verification workflow.src/orchestrator/reaper.ts(identity-checked reaper pattern to mirror).relaycast-cloud(workerd +@cloudflare/vitest-pool-workers), cloud'sminiflare-sageservice +test:integration.Fixes #146
Summary by cubic
Adds Cloudflare-based ephemeral verification environments and a reusable CI workflow. Each run gets an untrusted Workers for Platforms dispatch namespace with a sentinel metadata Worker, strict guardrails, and an identity-checked reaper, meeting #146.
CloudflareEnvironmentProvider: provisions one dispatch namespace per run; attaches a metadata Worker; enforces TTL, max-concurrency, run-cost, and container instance caps; adds a serializing + cross-instance reconciliation fix to close concurrency races; identity-checked destroy and a TTL/owner reaper.resourcename; credentials come from a linked Resource object at runtime (noprocess.env), with a narrow token used by an HTTP client..github/workflows/cloudflare-verification.ymlrunsfactory-test-infra’sci/verification-env/{up,run-suite,down}.shin Miniflare and real modes, materializes a JSON resource file for the real run, always tears down, and uploads artifacts.docs/cloudflare-environment-provider.md(lifecycle, guardrails, workflow usage);factory-test-infra/README.mdhandoff updated for Cloudflare; config schema, exports, and tests added.Written for commit 52bf6b4. Summary will update on new commits.