Skip to content

fix(ssh,agent,chart,security): the SSH tunnel verified no host key, and four more - #509

Merged
cevheri merged 2 commits into
mainfrom
chore/backlog-round14
Aug 26, 2026
Merged

fix(ssh,agent,chart,security): the SSH tunnel verified no host key, and four more#509
cevheri merged 2 commits into
mainfrom
chore/backlog-round14

Conversation

@cevheri

@cevheri cevheri commented Aug 26, 2026

Copy link
Copy Markdown
Member

Round 14 of the docs/BACKLOG.md sweep. Five entries closed, five new findings recorded.
Every fix was measured on both arms with a control arm, and three of the five were driven against
something real rather than a mock.

Ordered by cost/value, cheapest-and-highest first.

What is fixed

D11 - the SSH tunnel accepted any host key it was offered (security)

createSSHTunnel passed no hostVerifier, and ssh2 has no default. Read out of the library's
own source (node_modules/ssh2/lib/protocol/kex.js), with the callback absent it takes the branch
that logs "Host accepted by default (no verification)" and completes the handshake. So the tunnel
connected to whatever answered on the bastion's address, and everything the tunnel carries - the
database password among it - was readable to anything that could occupy that address. Nothing in the
product ever showed a fingerprint a user could compare.

Policy decided: trust-on-first-use, pinned per connection. Requiring a pasted fingerprint up
front makes tunnelling unusable for someone reaching their own bastion; TOFU is what every SSH
client does on first contact and is strictly better than verifying nothing. It is additive to
SSHTunnelConfig, so no storage migration.

Driven against a real openssh-server container, three arms:

arm result
no pin accepted, reported SHA256:JWLQciyX8RYEeuK+bwRLW/YSpStz0/L7BlbVcejL1/U - byte-identical to ssh-keyscan -p 2226 127.0.0.1 | ssh-keygen -lf -
correct pin connects
wrong pin refused, naming both fingerprints and telling the user how to confirm the key

Non-vacuity: before the fix the mismatch arm logged Tunnel created for pin-mismatch - it
connected. The fingerprint format was separately checked against ssh-keygen -lf on generated
ed25519 and RSA keys, 3 of 3 exact.

The fingerprint is classified public in connection-secrets.ts and stored in the clear on
purpose: it authenticates the bastion to Studio, grants no access, and has to be readable for the
comparison to be possible at all.

B68 - a conversation was single-connection by record, not by database

Every link checked connectionId, which is an identity check on the record. A connection edited
to address another server keeps its id, so a follow-up was handed the earlier steps' claims about
the old database while reading the new one. Nothing refused, nothing wrong to look at.

Each run now records connectionIdentity - the same fingerprint the held context snapshot is filed
under - and a follow-up whose database does not match its predecessor's declines instead of
carrying.

That fingerprint gained the SSH tunnel, which was a real omission: host/port are resolved at
the far end of the hop, so the same db.internal:5432 reached through two different bastions is two
different databases, and the old fingerprint could not see it. Excluded on purpose: the password
(rotating a credential must not end a conversation) and the host-key pin (it records what the
connection trusts, not where it goes).

Three arms, all green: the named repoint test carries no steps; an unchanged connection still
carries; and a rotated password / renamed connection / rotated bastion credential still carries -
the arm that proves the identity is about the database and not the record.

Then driven for real in Chrome via Playwright, against a live Gemini model - four actual agent
runs on the embedded SQLite sample, not a harness:

  1. First question (arun_55cfff9e…) succeeded with no thread notice, which is correct for a run
    that starts its own conversation. Its ledger's run-opened event carries
    "connectionIdentity":"55c9b0fe3a6ed062…" - live proof the identity is actually recorded, not just
    plumbed in tests.
  2. Follow-up with the predecessor's recorded identity changed - the same condition a repoint
    produces, staged by editing the recorded value rather than by faking any product behaviour, byte
    length preserved so the ledger's NUL framing stayed intact. The run declined, and the rail
    rendered the corrected sentence: "The earlier step could not be carried into this question, so it
    started on its own."
    - asserted positively, and asserted NOT to contain "could not be reached".
  3. Control: follow-up whose predecessor identity matches. It carried - and this was checked
    from the ledger rather than from the absence of a notice, which would have been ambiguous:
    run-opened shows thread: { threadId: "arun_caac3db2…", steps: [ … ] }.

One process note, since it changes how much the middle arm is worth: an earlier attempt at the
control was invalid - a .click() issued from inside page.evaluate did not drive React the way a
real click does, and I was re-reading the previous run's state and calling it a control. Redone with
a genuine Playwright click, which is where arm 3 above comes from.

B30 - a green ledger probe promised the wrong thing

The probe ran ensureDataDir's four steps. The world calls initDataDir, which then reads and
parses version.txt. A truncated, empty or incompatible one answered green: the rail rendered,
the operator clicked Start, and the run died when the world was built.

The probe now reads an existing version.txt itself, mirroring upstream's parse (read out of
@workflow/world-local/dist/init.js), and reports LEDGER_INCOMPATIBLE - deliberately distinct
from LEDGER_UNAVAILABLE, because the path is writable and the action is replacing one file, so an
operator sent to a permission would find nothing wrong with the directory.

Two properties are load bearing and asserted in every arm:

  • It never writes. The only upstream entry point that answers this question is initDataDir,
    which writes version.txt as a side effect - calling it would turn a page-load visibility probe
    into something that initialises the ledger.
  • An absent version.txt stays green. That is the fresh install; refusing it would make every
    first run report an unavailable agent.

A version that parses but differs is green too - upstream hands a mismatch to upgradeVersion,
which logs and returns. Refusing it would have been the mirror-image error: removing a working rail.

Verified live against the running app: a corrupt file flips GET /api/agent/config to
{"enabled":false,"reason":"LEDGER_INCOMPATIBLE","detail":"…records a version this release cannot read (\"truncated-no-at-sign\")…"}, the file is byte-identical afterwards, and no probe file is left
behind.

And verified in a real Chrome via Playwright, three arms, because the point of the entry is what
the operator sees:

arm route says the rail
healthy ledger (no version.txt) agent-rail-panel, Start and objective all render
version.txt = truncated-no-at-sign enabled:false, LEDGER_INCOMPATIBLE withheld entirely — not one agent-* element in the DOM
file removed again enabled:true the rail comes back

The third arm is the one that matters: it makes the withholding causally attributable to that file
rather than to load order or a cache. Before this change the corrupt file answered green, so the rail
rendered and the operator met the failure at the first Start.

AU2 - no security header reached the paths the proxy matcher skips

src/proxy.ts's matcher excludes .*\..*, _next/static and _next/image, so proxy() never ran
for public/, /monaco/vs/*.js or _next/static - header delivery and auth redirection were two
concerns sharing one matcher. The dot exclusion is correct for auth and stays.

next.config.ts's headers() now delivers the two headers meaningful on a subresource -
X-Content-Type-Options: nosniff and X-Frame-Options: DENY - reading their values from the same
module proxy() uses rather than spelling them twice.

The four that were left out are the point of the change. A next.config header set is baked at
build time, while src/lib/security/config.ts reads its env per process. So a copy of the
CSP would strand the CSP_REPORT_ONLY escape hatch that exists for an operator who cannot rebuild a
prebuilt image; and a copy of HSTS - which is host-scoped, so the last value the browser receives
wins
- would let a request for /logo.svg silently downgrade the policy the document just set.
Referrer-Policy and Permissions-Policy act on a document and are inert on an image or script.

Verified over real HTTP, which is the only thing that can prove delivery:

/logo.svg                              -> nosniff, DENY
/monaco/vs/loader.js                   -> nosniff, DENY
/_next/static/chunks/2opt17kb2-y1b.js  -> nosniff, DENY
/login (control)                       -> exactly 1x HSTS, 1x CSP, 1x XFO

_next/static was the one path the implementing agent could not claim from Next's own docs. It
works, so AU2's scope did not need narrowing.

N2 - the chart could not set the one flag a plain-HTTP install needs

AUTH_COOKIE_SECURE was reachable in every distribution channel except the chart, and it is the
answer for a browser reaching the app over plain HTTP on a non-loopback host - otherwise cookies
carry Secure, the browser rejects them, and login silently loops. Already diagnosed on three
channels. Chart users had to reach for extraEnv.

Now config.authCookieSecure, rendered through templates/configmap.yaml like its siblings.
Three-state, and that is the substance: it renders on kindIs "invalid" rather than truthiness,
because truthiness would silently drop false - the value that matters most here. The precedent is
the chart's own agent.enabled. All three arms rendered and checked:

unset (or explicit null)  -> no AUTH_COOKIE_SECURE line at all
--set …=false             -> AUTH_COOKIE_SECURE: "false"
--set …=true              -> AUTH_COOKIE_SECURE: "true"

src/lib/auth.ts was read to confirm absent really reaches the app as unset, and the operator's
verbatim chart copy is mirrored (operatorCopyViolations() empty).

Chart 0.1.51 is already released, so Chart.yaml version is bumped to 0.1.52 by hand.
appVersion and package.json are untouched; chart:bump would not have done it, since
appVersion is already in sync.

Also

The rail's sentence for a declined conversation said the earlier step "could not be reached". That
became false for a repoint - the step can be reached, the database moved - so it now reads "could
not be carried into this question"
, which is true of all six causes declined: "unavailable"
collapses. The stale claim in src/proxy.ts's matcher comment and the overstated header claim in
SECURITY.md were corrected too.

New backlog entries (with measurements)

  • D32 - nothing writes the durable SSH pin. There is no input for it, seed configs do not model
    sshTunnel, and an accepted fingerprint is not written back - so the shipped protection is the
    process-scoped memory: a key that changes within a server's lifetime is refused, a restart
    trusts afresh. Stated plainly in docs/providers/README.md rather than implied.
  • D33 - the live SSH arm was driven once by hand (this PR) and no fixture keeps it true; CI
    still runs the mocked handshake.
  • AU3 - Cross-Origin-Resource-Policy / Cross-Origin-Opener-Policy are the two headers that
    would genuinely protect a subresource and the two nobody has decided on. Both are constant, so the
    build-time objection does not apply; neither has been measured against Monaco's same-origin
    workers, where a wrong CORP value breaks the editor.
  • B75 - a connection repointed mid-flight is still carried by a resumed drive. The material
    to close it now exists; what a run should do when its connection moves under it is undecided.
  • B76 - declined: "unavailable" collapses six causes, so one sentence covers all of them and is
    specific about none. The repoint case is the one whose remedy differs.

Verification

All gates run locally, re-run after the rebase onto the moved main (a pre-rebase pass does not
vouch for the merged tree):

gate result
format clean
lint 0 errors, 128 warnings (all pre-existing repo-wide)
typecheck clean
knip clean
core tests 352/352 files
component groups 33/33
evals 197 pass
build clean
build:lib + attw clean
coverage:check 43553/43553 lines (100.00%)

Rebase note

main moved while this was in flight (#503, #505 merged). Rebased, and docs/BACKLOG.md conflicted
because #503 had also claimed the id D30. Resolved by keeping their entry and renumbering mine
to D32/D33, then fixing the cross-references in docs/providers/README.md. My entries also cited
D11, which this PR deletes - reworded, since a citation to a deleted entry is exactly the rot T4
records.

What was NOT verified

Narrowed after a Playwright pass in a real Chrome (see B30 and B68 above - both are now verified in
the browser, B68 against a live model). What genuinely remains:

  • The repoint itself was staged, not performed. B68's browser arm changed the predecessor's
    recorded identity, which is the condition a repoint produces; it did not edit a connection through
    the dialog and wait out the seed cache. That the two are equivalent is an inference from the code,
    not an observation.
  • CI runs the mocked ssh2 handshake, not the container one I drove by hand - recorded as D33.
  • AU2 was verified over real HTTP rather than through the browser's network panel. curl -I on
    /logo.svg, /monaco/vs/loader.js and a hashed _next/static chunk is the same evidence a browser
    would show, and stronger than a unit test, but no page was loaded to observe it.
  • D11 and N2 have no UI to drive. D11's live arm was a real sshd container; N2's was three
    helm template renders plus a read of how src/lib/auth.ts treats unset.

Not merged - for review.

…nd four more

Round 14 of the BACKLOG sweep. Five entries closed, each measured on both arms with a
control, and five new findings recorded with their measurements.

D11 - the SSH tunnel accepted any host key it was offered. `createSSHTunnel` passed no
`hostVerifier`, and ssh2 has no default: read out of its own source, the library takes the
branch that logs "Host accepted by default (no verification)" and completes the handshake.
So the tunnel connected to whatever answered on the bastion's address, and everything it
carries - the database password among it - was readable to anything that could occupy that
address. The policy chosen is trust-on-first-use pinned per connection: requiring a pasted
fingerprint up front makes tunnelling unusable for someone reaching their own bastion, and
TOFU is both what every SSH client does on first contact and strictly better than verifying
nothing. Driven against a REAL `openssh-server` container, three arms: no pin accepted and
reported `SHA256:JWLQciyX8RYEeuK+bwRLW/YSpStz0/L7BlbVcejL1/U`, byte-identical to
`ssh-keyscan | ssh-keygen -lf -`; the correct pin connected; a wrong pin was refused naming
both fingerprints. Before the fix the mismatch arm logged `Tunnel created` - it connected.

B68 - a conversation was single-connection by record, not by database. Every link checked
`connectionId`, so a connection edited to address another server kept its id and a follow-up
was handed the earlier steps' claims about the old database while reading the new one.
Nothing refused, nothing wrong to look at. Each run now records `connectionIdentity` - the
same fingerprint the held snapshot is filed under - and a follow-up whose database does not
match declines instead of carrying. That fingerprint gained the SSH tunnel: host and port are
resolved at the far end of the hop, so the same `db.internal:5432` reached through two
different bastions is two different databases, and the old fingerprint could not see it.
Rotating a credential or renaming a connection keeps the conversation, and a predecessor that
recorded no identity at all is carried rather than refused, so no conversation in flight
across a deploy is ended by a silence.

B30 - a green ledger probe promised the wrong thing. It ran `ensureDataDir`'s four steps; the
world calls `initDataDir`, which then reads and parses `version.txt`. A truncated or
incompatible one answered green, the rail rendered, and the run died at the first Start. The
probe now reads an existing `version.txt` itself, mirroring upstream's parse (read out of
`@workflow/world-local/dist/init.js`), and reports `LEDGER_INCOMPATIBLE` - a reason distinct
from `LEDGER_UNAVAILABLE`, because the path is writable and the action is replacing one file.
Two properties are load bearing and asserted in every arm: it never writes, because the only
upstream entry point that answers this question initialises the ledger as a side effect; and
an ABSENT `version.txt` stays green, because that is the fresh install. A version that parses
but differs is green too - upstream hands a mismatch to `upgradeVersion`, which logs and
returns. Verified live: a corrupt file flips the route to `LEDGER_INCOMPATIBLE` with the file
named, and the file is byte-identical afterwards with no probe file left behind.

AU2 - no security header reached the paths the proxy matcher deliberately skips: `public/`,
`/monaco/vs/*.js`, `_next/static`. Header delivery and auth redirection were two concerns
sharing one matcher. `next.config.ts`'s `headers()` now delivers the two that are meaningful
on a subresource, reading their values from the same module the proxy uses rather than
spelling them twice. The four that were left out are the point of the change: a `next.config`
header set is baked at BUILD time while `src/lib/security/config.ts` reads its env per
process, so a copy of the CSP would strand the `CSP_REPORT_ONLY` escape hatch, and a copy of
HSTS - host-scoped, so the last value the browser receives wins - would let a request for
`/logo.svg` silently downgrade the policy the document just set. Verified over real HTTP on
all three paths, including `_next/static`, which no unit test can reach; documents still
carry exactly one of each header.

N2 - `AUTH_COOKIE_SECURE` was reachable in every distribution channel except the chart, and
it is the one setting a plain-HTTP LAN or home-server install most needs. Now
`config.authCookieSecure`, rendered through the configmap like its siblings. Three-state, and
that is the substance: it renders on `kindIs "invalid"` rather than truthiness, because
truthiness would silently drop `false` - the value that matters most here. The precedent is
the chart's own `agent.enabled`. All three arms rendered and checked, `src/lib/auth.ts`
confirmed to treat absent as unset, and the operator's verbatim chart copy mirrored.

Chart 0.1.51 is released, so `Chart.yaml` version is bumped to 0.1.52 by hand; `appVersion`
and `package.json` are untouched.

New entries, each with its measurement: D30 (nothing writes the durable SSH pin, so the
protection is process-scoped and resets on restart), D31 (the live SSH arm was driven once by
hand and no fixture keeps it true), AU3 (CORP/COOP are the two headers that would protect a
subresource and the two nobody has decided on), B75 (a connection repointed mid-flight is
still carried by a resumed drive), B76 (`declined: "unavailable"` collapses six causes into
one sentence). The rail's sentence for that collapse was corrected: the earlier step "could
not be carried into this question" rather than "could not be reached", which was false for a
repoint.

Gates, all run locally: format, lint (0 errors), typecheck, knip, 352/352 core files, 33/33
component groups, 197 evals, build, build:lib, attw, coverage 43539/43539 lines (100.00%).
@cevheri
cevheri merged commit 3529414 into main Aug 26, 2026
10 checks passed
@cevheri
cevheri deleted the chore/backlog-round14 branch August 26, 2026 21:59
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant