Add Docker-based integration test environment #67
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a tmux-driven test harness that launches codey in a real terminal session, sends keystrokes, and uses an LLM evaluator to verify that TUI rendering (approvals, tool output) is correct. - Dockerfile: multi-stage build (Rust compile + slim runtime with tmux/chromium/python) - harness.py: CodeySession (tmux driver) + LLMEvaluator (Anthropic API) - Test scenarios for read_file, edit_file, write_file, and shell tools - docker-compose.yml + CI workflow (triggered on src changes or manual dispatch) - run.sh for local execution with ANTHROPIC_API_KEY https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
Wait conditions now match the real UI indicators: - Approval: [y]es [n]o - Completion: ✓ icon - Denial: ⊘ icon / "Denied by user" - Idle: absence of [y]es and ⚙ LLM evaluation prompts updated to reference the actual rendered format (? / ✓ / ⊘ status icons, [y]es [n]o prompt text). https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
- Rust 1.83 → 1.85: genai dependency uses edition2024 (stabilized in 1.85) - SIMD=0: skip ratatui-core patching (needs patch + git clone, unnecessary for tests) - Remove musl-tools (not cross-compiling) - Remove tmpfs mount on /workspace (was wiping the git workspace created at build time) https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
- rust:1.85 still too old (darling, time, etc. need 1.88+). Use latest to avoid chasing version pins as deps update. - Split apt-get install and make release into separate RUN layers for clearer error attribution and better caching. https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
The tmux session was dying silently when codey crashed on startup, leaving no server for send_keys to connect to. Now: - Create a plain tmux session first, then set remain-on-exit on - Launch codey via send-keys so the pane survives a crash - After startup delay, verify the pane is alive and capture diagnostics if codey failed to start https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
rust:latest is based on trixie/sid (glibc 2.39) but the runtime was bookworm-slim (glibc 2.36), causing "GLIBC_2.38 not found" at startup. Also improved the startup health check to use tmux's pane_dead flag instead of heuristic text matching on pane content. https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a tmux-driven test harness that launches codey in a real
terminal session, sends keystrokes, and uses an LLM evaluator to
verify that TUI rendering (approvals, tool output) is correct.
https://claude.ai/code/session_01CNSLtDvXV6M6STFNNxfdVt