Run fetch and update in separate containers behind isolated_fetch_update - #647
Draft
v-robaiken wants to merge 1 commit into
Draft
Run fetch and update in separate containers behind isolated_fetch_update#647v-robaiken wants to merge 1 commit into
v-robaiken wants to merge 1 commit into
Conversation
With the experiment on, the CLI clones in one updater container and updates in another. The clone travels between them on a Docker volume mounted at DEPENDABOT_REPO_CONTENTS_PATH in both, so it is made once rather than twice. Each side gets its own proxy on its own network pair, so the update container has no route to the fetch proxy and the two credential sets can diverge later. They are identical for now; the call site marks the seam. `--local` populates the fetch container only. Debug shells, the OpenTelemetry collector and use_case_insensitive_filesystem are rejected rather than silently misbehaving under the split. Combined mode drops the `bin/run fetch_files &&` prefix. That entrypoint used to be a no-op stub, so this is byte-identical to today, and it avoids fetching twice now that dependabot-core makes it do real work. Also fixes container output being truncated: RunCmd piped through io.Copy and prefixer, but Prefixer.WriteTo returns without writing pending data at EOF and io.Copy only calls WriteTo once, so the tail of every command was dropped. Running `bin/run` twice used to mask it, which is what the "unless I echo here the json doesn't output" comment in input.txt was working around.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CLI half of running fetch and update in separate containers (ADR 0051). Pairs with dependabot/dependabot-core#15862 — they need to land together.
This only moves the clone. No file stripping, no serialization, no credential changes.
What it does
Behind the
isolated_fetch_updateexperiment,runIsolatedclones in one container and updates in another. The clone travels on a Docker volume mounted atDEPENDABOT_REPO_CONTENTS_PATHin both, so it's made once instead of twice.Each side gets its own proxy on its own network, so the update container has no route to the fetch proxy. Credentials are identical for both today — the seam is marked at the call site.
--localpopulates the fetch container only. Debug shells, the OTel collector anduse_case_insensitive_filesystemare rejected under the split rather than silently misbehaving.Combined mode drops the
bin/run fetch_files &&prefix. That was a no-op stub, so this is byte-identical to today, and it avoids fetching twice now that core makes it real.Drive-by bugfix
RunCmdwas dropping the tail of every container's output —Prefixer.WriteTodoesn't flush pending data at EOF andio.Copyonly calls it once. Runningbin/runtwice used to mask it, which is what the"unless I echo here the json doesn't output"comment ininput.txtwas working around. Happy to split this out if preferred.Testing
New
testdata/scripts/isolated.txtasserts the clone crosses the volume. Full script suite passes exceptproxy, which also fails onmain.Verified against the real gomod image and
dependabot-fixtures/go-modules-lib: onegit clonein both topologies, zerocontentsAPI calls from the update container, identicalcreate_pull_requestpayloads.Not a security boundary yet
The update container still calls the target repo twice — branch validation from the duplicate
FileFetcherCommandrun, and PR name prefixing — so its credential can't be dropped. The samegit_sourcecredential also serves dependency metadata lookups, so it can't simply be deleted.