Move the CLI-level integration tests to a TypeScript e2e suite

.github/workflows/ci.yml

@@ -272,6 +272,86 @@ jobs:
             exit 1
           fi
 
+  # Prototype TypeScript e2e suite (test/e2e). Deliberately NOT in the release
+  # job's needs: and not required by the branch ruleset while it is being
+  # evaluated against the Go integration suite.
+  test-e2e:
+    name: E2E Tests (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v7
+
+      - name: Set up Go
+        uses: actions/setup-go@v7
+        with:
+          go-version-file: go.mod
+          cache-dependency-path: go.sum
+
+      # The action reads its version from a package.json's "packageManager" field,
+      # and this repo has none at the root — the suite's lives in test/e2e.
+      - name: Set up pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          package_json_file: test/e2e/package.json
+
+      # Node 26: the suite is type-erasable TypeScript, matching what Node itself
+      # can run by stripping types.
+      - name: Set up Node.js
+        uses: actions/setup-node@v7
+        with:
+          node-version-file: test/e2e/.node-version
+          cache: pnpm
+          cache-dependency-path: test/e2e/pnpm-lock.yaml
+
+      - name: Install e2e dependencies
+        shell: bash
+        working-directory: test/e2e
+        run: pnpm install --frozen-lockfile
+
+      - name: Typecheck e2e suite
+        shell: bash
+        working-directory: test/e2e
+        run: pnpm exec tsc --noEmit
+
+      # bash so the shared script works on the Windows runner too; container
+      # tests skip themselves there (no Linux containers available).
+      - name: Run e2e tests
+        shell: bash
+        run: make test-e2e
+        env:
+          CREATE_JUNIT_REPORT: "true"
+          LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
+          # Linux has every prerequisite, so a skipped one there is a defect, not
+          # an environment fact. macOS runners have no container runtime and
+          # Windows cannot run Linux containers or shim the browser opener, so
+          # those legs stay lenient.
+          LSTK_E2E_REQUIRE_ALL: ${{ matrix.os == 'ubuntu-latest' && '1' || '' }}
+
+      - name: Upload test results
+        uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: e2e-test-results-${{ matrix.os }}
+          path: test-e2e-results.xml
+
+      # fail-on-error is off so a setup failure — which produces no report at all —
+      # surfaces as the one real failure instead of also failing here, where the
+      # message would be "no files found" rather than the actual cause.
+      - name: Test report
+        uses: dorny/test-reporter@v3
+        if: always()
+        with:
+          name: E2E Test Results (${{ matrix.os }})
+          path: test-e2e-results.xml
+          reporter: java-junit
+          fail-on-error: false
+
   test-launcher:
     name: Launcher Tests
     runs-on: ubuntu-latest

.gitignore

@@ -12,3 +12,5 @@ test/integration/test-samples/**/.terraform/
 test/integration/test-samples/**/.terraform.lock.hcl
 test/integration/test-samples/**/*.tfstate
 test/integration/test-samples/**/*.tfstate.*
+test/e2e/node_modules/
+test-e2e-results.xml

CLAUDE.md

@@ -18,6 +18,7 @@ This installs a [gitleaks](https://github.com/gitleaks/gitleaks) hook that scans
 make build              # Compiles to bin/lstk
 make test               # Run unit tests (cmd/ and internal/) via gotestsum
 make test-integration   # Run integration tests (rebuilds bin/lstk via `build`, requires Docker)
+make test-e2e           # Run the TypeScript e2e suite (rebuilds bin/lstk; requires Node >= 26, pnpm, Docker)
 make lint               # Run golangci-lint (version pinned via .tool-versions)
 make govulncheck        # Run govulncheck (reachability-based vuln scan)
 make mock-generate      # Regenerate mocks (mockgen via go:generate)
@@ -34,9 +35,15 @@ Run a single integration test:
 make test-integration RUN=TestStartCommandSucceedsWithValidToken
 ```
 
+Run a subset of the e2e suite (`RUN` is a substring match on the test name):
+```bash
+make test-e2e RUN="injects the endpoint"
+```
+
 Notes:
 - Integration tests require `LOCALSTACK_AUTH_TOKEN` environment variable for valid token tests.
 - `test/integration` is a **separate Go module** (own `go.mod`); `make lint` runs golangci-lint twice — repo root and `test/integration` — and fails if the installed golangci-lint version doesn't match `.tool-versions`. `golangci-lint run --fix` auto-fixes many findings.
+- `test/e2e` is a **third** self-contained tree after the repo root and `test/integration`, and the only one that isn't Go: a Node/pnpm workspace running TypeScript under vitest. It drives the built `bin/lstk` and imports no lstk source, so nothing there is covered by `make lint`/`make govulncheck`.
 - `make govulncheck` also runs twice for the same reason (root + `test/integration`). It complements the dependency-version scan in `trivy.yml` with call-graph reachability analysis — it only flags known vulnerabilities in code actually called from the repo. It has no severity filter (most Go vulnerability reports carry no CVSS data), so it gates on reachability alone: any reachable known vulnerability fails the job. CI (`ci.yml`'s `govulncheck` job) runs it on every push/PR and uploads a SARIF report to the Security tab, same pattern as Trivy, but it is **not yet** in `release`'s `needs:` — it's a new check on a staged rollout and should be promoted to a hard release gate once it's proven false-positive-free.
 - Mocks are generated with mockgen (go.uber.org/mock) via per-file `//go:generate mockgen ...` directives (e.g. `internal/snapshot/remote.go`); adding a mock means adding a directive, then `make mock-generate`.
 - Set `CREATE_JUNIT_REPORT=1` to get a JUnit XML report from `make test` / `make test-integration`.
@@ -220,8 +227,17 @@ When drafting Slack messages, PR descriptions, review replies, release notes, or
 
 # Testing
 
-- Prefer integration tests to cover most cases. Use unit tests when integration tests are not practical.
-- **When fixing a bug, always add an integration test** that fails before the fix and passes after. This prevents regressions and documents the exact scenario that was broken.
+Three suites, and the first question for any new test is which one owns the area:
+
+- **`test/e2e/` (TypeScript/vitest)** owns CLI-level behaviour for the areas ported to it: the proxy commands `aws` and `terraform`, `--json` envelopes, exit codes, `--non-interactive`, lifecycle (`stop`/`restart`/`status`/`reset`), `logs`, `volume`, `--endpoint-url`/`LSTK_ENDPOINT_URL`, config resolution, completion, `docs`, the login journey, and the TUI paths. It drives the built binary and asserts whole output (`toPrintExactly`), never a substring and never a recorded snapshot. Conventions: [test/e2e/README.md](test/e2e/README.md); the coverage map against the Go suite: [test/e2e/PORTING.md](test/e2e/PORTING.md).
+- **`test/integration/` (Go)** still owns snapshots, IaC end-to-end (`terraform_e2e`, `cdk`, `sam`), most of `start`, extensions and signal forwarding, update/install, license, telemetry, `az` / `setup azure` / `awsconfig`, and anything needing Docker-SDK-level setup or the real OS keyring.
+- **Go unit tests** under `cmd/` and `internal/` are unaffected by the split and remain the right tool for logic with no CLI surface.
+
+- Prefer an end-to-end/integration test to a unit test where one is practical, in whichever of the two suites owns the area.
+- **When fixing a bug, always add a test** that fails before the fix and passes after, in the suite that owns the area. This prevents regressions and documents the exact scenario that was broken.
+
+The rest of this section is specific to the Go integration suite.
+
 - Integration tests that run the CLI binary with Bubble Tea must use a PTY (`github.com/creack/pty`) since Bubble Tea requires a terminal. Use `pty.Start(cmd)` instead of `cmd.CombinedOutput()`, read output with `io.Copy()`, and send keystrokes by writing to the PTY (e.g., `ptmx.Write([]byte("\r"))` for Enter).
 - Mark every integration test with `t.Parallel()` unless it shares external state with other tests. Today the main blocker is the Docker daemon: tests that start LocalStack containers cannot run concurrently because lstk's container discovery matches by `(image, internal port)`, so two parallel runs would cross-contaminate. Tests that only touch the filesystem, mock servers, or the CLI binary itself should be parallel.
 - Never let an integration test inherit the developer's real `$HOME`. Pass an isolated env via `testEnvWithHome(t.TempDir(), "")` (or build on top of it with `env.With(...)`) instead of `nil` or `os.Environ()`. Inheriting HOME pollutes the user's `~/.config/lstk/`, `~/.aws/`, and `~/.cache/lstk/`, and makes parallel runs interfere through shared `lstk.log`, license cache, and file-keyring fallback.

Makefile

@@ -5,7 +5,7 @@ endif
 BUILD_DIR=bin
 export CGO_ENABLED=0
 
-.PHONY: build clean test test-integration lint govulncheck mock-generate otel
+.PHONY: build clean test test-integration test-e2e lint govulncheck mock-generate otel
 
 # Always invoke `go build` and let Go's build cache handle incrementality; a
 # file target on bin/lstk would be skipped when the binary exists, even with
@@ -23,6 +23,9 @@ test:
 test-integration: build
 	@RUN="$(RUN)" ./scripts/test-integration.sh
 
+test-e2e: build
+	@RUN="$(RUN)" ./scripts/test-e2e.sh
+
 otel:
 	docker compose -f docker-compose.tracing.yaml up -d
 

scripts/test-e2e.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Run the TypeScript end-to-end suite against the built binary (bin/lstk).
+#
+# Requires Node >= 26 (the suite is type-erasable TypeScript, run through vitest)
+# and pnpm. See test/e2e/README.md.
+#
+# Honors:
+#   CREATE_JUNIT_REPORT  Emit JUnit XML to test-e2e-results.xml when set.
+#   SHARD_INDEX          1-based shard index (used with SHARD_TOTAL).
+#   SHARD_TOTAL          Total number of shards; passed to vitest --shard.
+#   RUN                  Substring filter passed to vitest -t.
+
+set -euo pipefail
+
+cd "$(dirname "$0")/../test/e2e"
+
+if ! command -v pnpm >/dev/null 2>&1; then
+  echo "pnpm is required to run the e2e suite: https://pnpm.io/installation" >&2
+  exit 1
+fi
+
+NODE_MAJOR=$(node -p 'process.versions.node.split(".")[0]' 2>/dev/null || echo 0)
+if [ "$NODE_MAJOR" -lt 26 ]; then
+  echo "Node >= 26 is required to run the e2e suite (found $(node -v 2>/dev/null || echo none))." >&2
+  echo "The suite is type-erasable TypeScript targeting Node's native type stripping." >&2
+  exit 1
+fi
+
+if [ -n "${CI:-}" ]; then
+  pnpm install --frozen-lockfile
+else
+  # A full install on every local run is slow; install only when deps are missing.
+  [ -d node_modules ] || pnpm install
+fi
+
+ARGS=()
+if [ -n "${SHARD_TOTAL:-}" ]; then
+  ARGS+=(--shard "${SHARD_INDEX:-1}/${SHARD_TOTAL}")
+fi
+if [ -n "${RUN:-}" ]; then
+  ARGS+=(-t "$RUN")
+fi
+
+exec pnpm exec vitest run ${ARGS[@]+"${ARGS[@]}"}

test/e2e/.node-version

@@ -0,0 +1 @@
+26

test/e2e/PORTING.md

@@ -0,0 +1,130 @@
+# Porting status: Go integration suite → TypeScript e2e suite
+
+**Intent:** this suite owns the CLI boundary for the areas listed under "Covered". Go
+**unit** tests (`cmd/`, `internal/`) stay as they are — they are the right tool for logic
+with no CLI surface. `test/integration` keeps everything under "Still owned by Go" and
+does not go away wholesale; it shrinks as areas move across.
+
+**Status: 225 tests across 26 files** (210 pass, 15 skip on this machine — an auth token,
+a native Linux daemon and SSL_CERT_FILE certificate trust are the prerequisites not met
+here). Full-suite wall clock ≈ 75s.
+
+The Go integration suite has been trimmed accordingly: against `main` it goes from **427
+→ 269** test functions across **55 → 40** files, 14,758 → 10,623 lines.
+
+## What "ported" means here
+
+Not a line-by-line translation. Many Go integration tests assert mechanism rather
+than behaviour, and those were deliberately **not** carried across:
+
+- **Telemetry assertions** (`assertCommandTelemetry`, `mockAnalyticsServer`) — dropped
+  throughout. What lstk sends to analytics is not something a CLI user observes. Per-command
+  telemetry stays in Go, consolidated into one table in
+  `test/integration/command_telemetry_test.go` rather than scattered across the tests whose
+  behavioural halves moved here.
+- **Container introspection** (`docker inspect` of `Config.Env`, `HostConfig.Binds`)
+  — replaced by the CLI-observable equivalent where one exists, e.g. `restart
+  --persist` is asserted through the `• Persistence: Enabled` line rather than the
+  container's environment.
+- **Token storage** — the keyring assertions across `login_test.go` and `logout_test.go`
+  became behavioural ones in `login-journey.pty.test.ts` (see the README). "A failed
+  login stores nothing" is asserted as *`start` still demands credentials and a retry
+  reopens the browser* rather than by reading the store. Nothing in `test/e2e` touches
+  credential storage; `keyring: "file" | "system"` only selects which backend the binary
+  uses. That is not just tidiness — the two Go tests that did reach in
+  (`TestLogoutCommandNotesWhenEmulatorStillRunning` and its multi-emulator sibling)
+  wrote through the *test process's* `$HOME` while running lstk under a temp one, and
+  passed on Linux CI while failing locally.
+
+Two Go tests turned out to be **vacuous** and were re-targeted rather than copied:
+`TestConfigWithUnknownFieldsIsAccepted` and `TestConfigWithMissingOptionalTagSucceeds`
+assert config acceptance through `lstk config path`, which never parses the file (see
+`cmd/config.go`) — they pass against a nonexistent path. The ports use `lstk logout`, which
+really calls `config.Get()`.
+
+## Covered
+
+| Area | Vitest files | Tests |
+| --- | --- | --- |
+| Proxy commands (`aws`, `terraform`) | `aws-proxy`, `terraform-proxy` | 41 |
+| `--json` envelope, exit codes, `--non-interactive` | `json-envelope`(+`.pty`), `json-flag`, `exit-codes`, `non-interactive.pty` | 42 |
+| Lifecycle (`stop`, `restart`, `status`, `reset`) | `stop-restart`, `status`, `reset.pty` | 28 |
+| `logs`, `volume` | `logs.pty`, `volume.pty` | 21 |
+| Config, completion, docs | `config`, `completion`, `docs` | 27 |
+| Start paths, emulator selection, login journey, TUI | `start`, `start-local-image`, `emulator-select.pty`, `emulator-type`, `login-journey.pty`, `tui-runtime-error.pty` | 18 |
+| `--endpoint-url` / `LSTK_ENDPOINT_URL` | `endpoint-url`(+`.pty`), `endpoint-url-https` | 36 |
+| Harness self-tests (not product behaviour) | `harness/strip-ansi`, `harness/print-exactly` | 12 |
+
+### What that removed from the Go suite
+
+Deleted outright: `json_envelope`, `exit_code`, `non_interactive`, `completion`,
+`aws_completion`, `docs`, `terraform_cmd`, `logs`, `reset`, `volume`, `stop`, `restart`,
+`logout`, `status`, `endpoint_url`, `endpoint_url_https`.
+
+`aws_completion_test.go` arrived on `main` (#424) after this port, and its one bash-driver
+case depends on a helper inside the `completion_test.go` this branch deletes. Both files
+move: `lstk aws <TAB>` delegating to `aws_completer` is CLI-observable, and
+`completion.test.ts` already drives the generated script under a bare bash.
+
+`config_test.go` likewise gained `TestConfigWithInvalidContainerNameFails` on `main`
+(custom `container_name`). That one is pure CLI output rejected at config load — no
+daemon, no token — so it was ported to `tests/config.test.ts` next to its `port is
+required` sibling rather than kept. Its companion `TestStartCommandUsesCustomContainerName`
+stays in Go: it needs a real start plus a container inspect.
+
+Trimmed, with the reason each remainder stayed:
+
+| Go file | Kept | Why it could not move |
+| --- | --- | --- |
+| `json_flag` | 2 of 7 | Both are table-driven proxy tests covering `az`, which TypeScript cannot reach without a completed `lstk setup azure` |
+| `aws_cmd` | 3 of 18 | Spinner timing under a PTY |
+| `config` | 1 of 12 | `TestConfigFlagEnvVarsPassedToContainer` inspects the container's environment |
+| `emulator_type` | 7 of 10 | |
+| `emulator_select` | 7 of 9 | |
+| `start` | 34 of 36 | |
+| `login` | 1 of 4 | `TestDeviceFlowSuccess` is the only remaining telemetry assertion for `login` |
+
+## Still owned by Go
+
+| Area | Go files | Cases | Would need |
+| --- | --- | --- | --- |
+| Snapshots | `snapshot_*_test.go`, `start_snapshot_test.go` | 78 | Mock cloud/S3 remotes; AWS SDK assertions against a live emulator |
+| IaC end-to-end | `terraform_e2e`, `terraform_s3backend_e2e`, `cdk_*`, `sam_*` | 46 | Real terraform/cdk/sam installs (the `_cmd` half of terraform is done) |
+| `start` remainder | `start_test.go`, `docker_unhealthy`, `docker_windows` | 41 | Never-healthy image via `docker commit`; bind/port introspection |
+| Trimmed leftovers | `emulator_type`, `emulator_select`, `login`, `aws_cmd`, `json_flag`, `config` | 21 | See the table above — each has its own blocker |
+| `az` proxy, `setup azure`, `awsconfig` | `az_*`, `setup_azure`, `awsconfig` | 22 | Isolated `~/.azure` assertions; `setup azure` completion marker |
+| Extensions, signal forwarding | `extension`, `signal_forwarding` | 22 | Reference extension build; process-group signalling |
+| Update & install | `update`, `multiple_installs`, `version_resolution` | 16 | Mock GitHub releases API; fake Homebrew/npm layouts |
+| Telemetry, license, logging | `telemetry`, `license`, `logging`, `command_telemetry` | 15 | Mechanism by design — a mock analytics server and a mock license API, neither of which is user-observable |
+
+## Fixtures available for the remaining work
+
+`support/`: `lstk()` and `lstkPty()` runners, `tempHome()`, `docker`,
+`useExclusiveEmulator()`, `emulator-stub.ts` (stand-in container + log writing),
+`fake-binary.ts` (fake `aws`/`terraform`/… recording argv, env, cwd),
+`extension-fixture.ts`, `platform.ts` (`mockPlatform()` login flow + `fakeBrowser()`),
+`emulator-api.ts` (the emulator's own HTTP API, http or https, for `--endpoint-url`),
+`license.ts`, `os-config-dir.ts`, `envelope.ts`, `requirements.ts`.
+
+## Consequences worth accepting deliberately
+
+- **The CLI boundary for the ported areas is now covered only by Node.** The e2e job has
+  to be a required check.
+- **Windows loses the login journey.** `pkg/browser` invokes `rundll32` there rather
+  than a shimmable script. No loss against today: `login_test.go` already skips Windows.
+- **Windows terminal coverage may improve** — node-pty drives ConPTY, where
+  `creack/pty` has no Windows support at all. Unproven until the Windows CI leg reports.
+- **Real-keyring coverage narrows to one journey run** (`keyring: "system"`, CI or
+  opt-in). The adapter logic below it stays covered by the mocked unit tests in
+  `internal/auth/token_storage_test.go`.
+- **Two prerequisites are gated rather than assumed**, both `requirement()`-checked so a
+  missing one skips instead of failing — and hard-fails on the CI leg that has
+  everything (`LSTK_E2E_REQUIRE_ALL=1`). The https tests need certificate trust an
+  exec'd lstk actually reads, which Go's x509 verifier only takes from SSL_CERT_FILE on
+  Linux; the "status reports the bound port" test needs a daemon that can publish on the
+  127.0.0.2 loopback alias, which Docker Desktop's VM networking refuses.
+- **Container behaviour is verified on exactly one CI leg.** Ubuntu has Docker and the
+  auth token, and `LSTK_E2E_REQUIRE_ALL=1` turns a missing prerequisite into a hard
+  failure there; macOS and Windows runners cannot run Linux containers, so 63 and 72
+  tests respectively skip. Roughly 51 tests could stop running on those legs without
+  anything going red. A per-platform skip budget asserted in CI would close that gap.