fix(backtest): scope job sweep to this terminal, stop leaking tester processes, tail the real log

[Marinski]

Three defects found while investigating backtests reported as failed on terminals that had in fact run them. They compound each other, so they ship together. No configuration changes are required and single-terminal installs behave exactly as before.

1. The startup sweep failed other terminals' running jobs

Every backtest API on a host shares one logs/backtest-jobs directory — the sweep_orphans() docstring already says so. Sibling terminals also share broker and account; they differ only by instance, which the job record did not carry. So the sweep failed every recently-touched in-flight job it found, and restarting one terminal's API marked every other terminal's running backtest as "API restarted before completion" while those runs went on to finish and write valid reports.

The job now records its instance, and the sweep skips jobs belonging to another terminal. A job that names no terminal is still swept — that is what a single-terminal install writes, and what every job written before this field existed looks like — so installs with nothing to distinguish are unaffected.

2. Failure messages carried a months-old MetaEditor tail

_tail_terminal_log() took the alphabetically last *.log in the terminal's log directory. That is always metaeditor.log ("m" sorts after every "<date>.log"), and it is written once at install and never touched again. So every backtest failure message quoted stale compile output instead of the run that had just died.

Not cosmetic: it is why a genuine agent bind error on 127.0.0.1:3000 [10048] sat unread in the real log while the failure text showed unrelated MetaEditor lines. Now selected by modification time, with metaeditor.log excluded.

3. Leaked tester processes kept the agent ports bound

A test runs as terminal64.exe plus one metatester64.exe per agent, and it is the agents that bind the localhost ports MT5 allocates from 3000 upward.

subprocess.run kills the process it started, so the timeout path looked covered — but it does not touch the agents, nor a terminal MT5 relaunched in place of the one we spawned (the case _await_self_relaunch exists for). Those survivors keep their ports bound for as long as the host stays up, and every later run on that terminal then dies instantly with bind error [10048] and no report.

The timeout path now clears the whole terminal directory's tester processes, and startup does the same. Terminate first, then kill whatever ignores it. Matching is by terminal directory, so a sibling instance is never touched, and the startup pass is gated on mode: backtest because that cleanup thread also runs in live mode, where the terminal is meant to stay up.

The startup kill is deliberately not conditional on the sweep having found anything: a run whose state file predates the sweep lookback — an API killed while a long test was live — is never swept, and gating the kill on that would leave exactly the process that is holding the ports.

Testing

20 new tests across tests/test_backtest_jobs.py, tests/test_backtest_log_tail.py and tests/test_backtest_process_cleanup.py, covering sibling-terminal isolation, the legacy job shape, instance normalisation, metaeditor.log exclusion and mtime selection, kill escalation, the live-mode guard, and the not-swept-but-still-leaking case.

Full suite in the container test image (Dockerfile.test, the same one make test builds): 399 passed, 2 skipped, 1 xfailed.

Scope

These fixes do not attempt to stop concurrent terminals colliding on agent ports — only the persistent damage, where a leaked process keeps ports bound and every later run on that terminal fails until the host is rebooted.

Worth knowing if you are reading [10048] lines while chasing this: MT5 appears to retry upward when a port is taken. Terminals here have accumulated Tester/Agent-127.0.0.1-<port> directories spanning 3000–3332, the newest of them well above the base, so a single bind error is not on its own evidence that a run died of a collision.

---

4. The wickworks TA sidecar silently orphans when the VM container restarts

The TA sidecar shares the VM container's network namespace via network_mode: service:mt5. When the VM container is restarted or recreated, Docker leaves the sidecar in the old, now-orphaned namespace. Its own loopback /health keeps answering, so the image's built-in healthcheck stays green while the VM can no longer reach wickworks — every /rates/ta call then fails with connection refused and surfaces as a 502 ("wickworks unreachable").

This shipped as commit b3cd464. It adds scripts/wickworks-healthcheck.py, a self-heal healthcheck that probes the dockurr gateway services (20.20.20.1:445/139/5900/5700) which only exist while the sidecar still shares the LIVE mt5 netns, and kills the uvicorn child when they stop answering — making restart: unless-stopped recreate the container into the current netns. (kill 1 from an exec'd healthcheck is not delivered to the container init, so the uvicorn child is targeted instead.)

Wired into docker-compose.yml.j2 and docker-compose.yml.example via a read-only volume mount plus a healthcheck (30s start period so a slow boot is not mistaken for orphaning). No configuration changes required. 5 new tests: 4 unit tests for the healthcheck and 1 compose-generation test asserting the rendered wickworks services carry the mount and healthcheck.

[psyb0t]

Re-checked against this exact head rather than relying on the earlier pass. The intended fixes are sound, but these three cases still let one terminal affect another or prevent the new self-heal from firing. The existing test/lint checks pass; these need fixing before merge.

[psyb0t]

This is substring matching, not a directory-boundary check: when this terminal is ...\\a, a sibling at ...\\a2\\terminal64.exe (or ...\\aa\\metatester64.exe) is yielded and then terminated/killed. The new test only checks a versus b, so it misses the collision. Please compare normalized Windows path components and add an a/a2 regression case.

[psyb0t]

The upgrade case is still unowned. Pre-PR job JSON already contains broker and account; this diff only starts adding instance. Thus an old active job for the same broker/account but no instance is claimed by every clone (a, b, and default). The new legacy test omits broker/account too, so it does not cover that actual old record shape. Please make missing instance safe for multi-clone installs and regression-test that shape.

[psyb0t]

The four probes are serial and each can wait PROBE_TIMEOUT = 2, so the all-unreachable orphan path takes up to 8 seconds before _kill_main_process() runs. Docker aborts this healthcheck after 5 seconds, so the self-termination/restart never happens in precisely that case. Please keep the aggregate probe time below this timeout (or raise it) and add a regression test for the relationship.