fix: clear exec_and_wait's spawn lock between logical invocations

[keepkeen]

What

exec_and_wait's retry protection makes retries impossible: a second invocation with the same tag executes nothing and silently returns the first invocation's exit code and log.

Root cause

The detached spawn is guarded so that a transport-level replay of the spawn RPC can't double-execute:

f"chmod +x {launcher}; "
f"mkdir {lock_dir} 2>/dev/null || exit 0; "   # dedupe guard
f"rm -f {out_file} {done_file}; "             # stale-marker cleanup — BEHIND the guard
f"setsid bash {launcher} < /dev/null > {out_file} 2>&1 &",

Two problems compose:

1. lock_dir is never removed, so the guard fires for every later logical invocation of the same tag, not just for transport replays of the same spawn.
2. The stale-marker cleanup sits behind the guard, so when the guard fires, the previous run's .done file is still there — _await_done_marker returns immediately with the previous run's exit code, and the .out tail is the previous run's log.

Concrete victim

harness.common.install_npm_cli retries a failed npm install three times by design (NPM_INSTALL_RETRIES = 3, "Detached install with a few in-place retries for transient disk flakes"), all with tag="harness-npm-install". Attempts 2 and 3 execute nothing and replay attempt 1's result — the backoff sleeps are the only thing that actually runs. RuntimeError("npm install failed after 3 attempts") really means "after 1 attempt", and a transient flake that would have succeeded on retry becomes a hard failure.

Fix

Clear the per-invocation state in its own idempotent RPC before the guarded spawn:

await sb.exec(f"rm -rf {lock_dir}; rm -f {out_file} {done_file}", ..., idempotent=True)
await sb.exec(f"chmod +x {launcher}; mkdir {lock_dir} 2>/dev/null || exit 0; setsid ...", ...)

The guard's actual purpose is preserved: a severed-response replay of the spawn RPC still hits mkdir and exits without double-executing. It just no longer outlives the invocation. (Callers must not overlap two same-tag calls — that was already true, and is now documented at the call site.)

Also fixes the adjacent Protocol mismatch: Sandbox.exec didn't declare idempotent, yet both exec_and_wait RPCs pass it — a third-party sandbox implementing the documented signature would raise TypeError. (tests/test_agent/_fakes.py already carried the parameter.)

Test

New tests/test_agent/test_sandbox_exec_and_wait.py, wired into the agent-test CI job. The fake sandbox interprets the actual shell strings exec_and_wait issues (mkdir-guard short-circuit, rm cleanup, setsid launch, marker polls), so the tests pin semantics rather than the exact command layout:

• a same-tag reinvocation must really re-run — fails on main with assert 1 == 2 (second invocation never spawned) and returns attempt 1's stale (1, "attempt-1 failed");
• a replayed spawn RPC must stay deduped (the guard's real job), and the cleanup must not ride inside the guarded command.

The full tests/test_agent/ suite passes (64 passed, 1 pre-existing skip).