feat: Squid Core Service — REST+SSE API and MCP bridge rewrite

[hongquanli]

Summary

Replaces the ad-hoc TCP control server with a transport-agnostic squid_service package exposed as a REST + SSE API (port 8060, served inside the GUI process) and a rewritten MCP bridge over that API. Driven by the internal Core Service spec and customer URS LA-WC-0001 (scheduler integration).

The legacy TCP server (port 5050) is untouched and still runs — deprecated for one release. Existing .mcp.json, the GUI launcher, and pre-approved microscope_* permissions keep working unchanged.

Why

The old microscope_control_server.py fused transport, validation, state, and hardware access in one 1,600-line class. This branch splits those concerns and, along the way, retires real bugs it surfaced:

• Systemic FieldInfo default bug — pydantic Field() used as plain function defaults meant optional params silently received FieldInfo objects (e.g. move_to(y_mm=5) passed FieldInfo into move_x_to).
• Well-coordinate bug — the TCP server's hand-rolled well math omitted WELLPLATE_OFFSET_X/Y_mm, so its coordinates disagreed with the GUI.
• Dead progress code — get_acquisition_status read worker attributes (current_fov_index, total_fovs) that don't exist.
• Outcome-blind acquisitions — status cleared to idle with no success/failure distinction; abort/error reported as SUCCESS whenever Slack wasn't configured.

What's in it

New software/squid_service/ package

• Canonical Fault model — 8 categories, numeric codes in 1000-blocks, scheduler_action hint, recoverable/terminal/plate_removable, monotonic sequence numbers, sanitized messages (no stack traces to remote callers)
• State machine — UNINITIALIZED → INITIALIZING → INITIALIZED → ACQUIRING → PROCESSING → ERROR → RECOVERING; state-changing commands rejected unless INITIALIZED; both GUI- and API-started acquisitions drive it, so API clients always see truthful state; acquisitions that end in error transition to ERROR with the fault preserved
• Job store — full lifecycle, progress (FOVs, elapsed, AF/save failure counts), outcome, skipped_fovs; last job persisted across restart
• Event bus — SSE with Last-Event-Id replay, resume_gap, replay/live dedupe
• Method registry — named server-side acquisition configs (URS API-METH-001..005), no client filesystem paths; full CRUD + validate, hot-reload

REST + SSE API (squid_service/rest/) — FastAPI on port 8060: system/motion/imaging/autofocus/acquisitions/jobs/methods/sample_formats/debug + /v1/events. GET /docs Swagger UI, GET /openapi.json. Bearer auth (off by default on loopback, mandatory on non-loopback binds), 202 + Location for acquisitions, canonical fault on every non-2xx.

Three ways to run a scan (exactly one per request): inline grid, named method, or GUI-saved yaml_path — each with wells/output_path/sample_format/autofocus overrides.

Rewritten MCP bridge (mcp_microscope_server.py) — thin curated-tool HTTP client (39 tools), all legacy names/args preserved; tool errors surface the canonical fault JSON so agents branch on category/terminal.

scripts/run_acquisition.py ported to REST (--method flag, SQUID_API_TOKEN), plus a latent exit-code bug fixed.

Docs — new quickstart-api.md (copy-paste, verified against the simulator) and quickstart-mcp.md (plain-English Claude prompts); core-service-api.md reference; updated automation.md / mcp_integration.md.

Added after PR creation (2026-07-03)

• Wells-by-name methods + run-time Z policy (d4764ae4..74cc0119): a wellplate method can now specify wellplate_scan.wells: "A1:B3" — X/Y derived from the plate definition; no more hand-written regions[].center_mm for plate scans (the coordinate form still parses). New z_reference field on acquisition requests: "current" (default, today's behavior), {"z_mm": <float>} (explicit baseline, limit-validated at preflight), or "autofocus" (requires AF enabled and a stored reference, enforced at preflight so it fails early — not mid-plate).
• Behavior change: the API path now honors z_stacking_config (previously ignored); the service sets z_range[0] = baseline and the worker performs the FROM CENTER shift — matching the GUI exactly.
• Headless mode (00702d14, 2026-07-05): python3 main_headless.py [--simulation] serves the identical API from a QApplication-free process — for scheduler-driven instruments and remote operation. squid_service/headless.py mirrors the GUI's controller wiring with the Qt-free base classes. main_headless.py owns SIGINT/SIGTERM itself (uvicorn's main-thread signal capture replays the signal after serve(), killing the process before any teardown — so uvicorn runs in a thread); shutdown aborts an in-flight acquisition (bounded 60 s), stops the server, closes the hardware, and reaps leftover JobRunner children so unattended instruments don't accumulate orphaned subprocesses. Live-verified in simulation: acquisition to SUCCESS; SIGTERM mid-384-FOV-run → ABORTED + clean exit, port released, zero orphans; a disk-full run surfaced the canonical ACQUISITION/6002 fault.
• Test count now 180 locally; CI green on the current head.

Testing

• 136 tests across service / REST / MCP bridge / acquisitions (simulated Microscope), including full acquisition lifecycle, graceful abort → ABORTED, busy-rejection, ERROR-state recovery, method CRUD, auth on/off, SSE replay/gap/dedupe, and forced hardware-fault paths.
• Full CI-style suite green (one pre-existing failure — test_multi_point_with_laser_af — confirmed identical on master; one Qt-abort in test_widgets.py confirmed environmental, passes in-process).
• black==25.12.0 (CI version) clean; ruff F401/F841 clean.
• Live smoke against the simulation GUI: both ports serving, stage moved via REST, real acquisition run to SUCCESS, MCP bridge round-trip verified.
• Every task individually reviewed; final whole-branch review passed after its 3 Criticals were fixed and re-verified.

Reviewer notes

• Port 8060, not the spec's 5060 — browsers block 5060 (SIP) as ERR_UNSAFE_PORT, which would make /docs unreachable. INI-configurable via [CORE_SERVICE] port.
• Not in this PR (need MultiPointWorker internals; matching the vendor "to be implemented" URS responses): emergency_stop, plate interlocks (confirm_plate_loaded / prepare_for_unload), per-acquisition AF/save-failure thresholds and populated skipped_fovs manifest, RESERVED state. Reserved endpoints return canonical 501s; the skipped_fovs field exists and is forward-compatible.
• First hardware test checklist (things simulation can't prove): initialize(home=true) with induced homing failure; two back-to-back API runs with a Z move between (z_range refresh); real abort → stage/illumination safe; REST polling during GUI live view (camera-SDK thread affinity); MCP home/long move timeout behavior; kill -TERM mid-acquisition → port 5060/8060 rebinds on restart; main_headless.py on real hardware (camera SDK callback threads have never run without a Qt event loop in this codebase).
• Follow-up PR (small, mechanical): event-bus publish-under-lock, _on_acq_finished ordering, reset() under lock, save-fallback DRY, Settings toggle label ("MCP Control Server" now gates both servers), explicit uvicorn/sse-starlette/httpx deps, a GET /v1/jobs list endpoint.

🤖 Generated with Claude Code

[hongquanli]

Review guide

This is a big diff (43 files, +7.2k), but it layers cleanly. Suggested reading order — each layer only depends on the ones before it:

1. squid_service/faults.py — the fault taxonomy every layer speaks (categories, codes, HTTP mapping). ~160 lines, sets the vocabulary.
2. squid_service/state.py, events.py, jobs.py, wells.py — small, self-contained, unit-tested.
3. squid_service/service.py — the core. Everything interesting is here: state guards (_exclusive), the acquisition path (start_acquisition → _configure_regions/_configure_controller), controller-callback chaining (_wrap_controller_callbacks — how GUI and API acquisitions share one state machine), Z handling (_resolve_z_reference, _reset_z_range_and_focus_map).
4. squid_service/rest/ — deliberately thin delegation; sse.py has the one subtle bit (replay/dedupe).
5. mcp_microscope_server.py — pure HTTP client, tool table.
6. main_hcs.py wiring, control/acquisition_yaml_loader.py (additive change only), docs.
7. main_headless.py + squid_service/headless.py (added 2026-07-05) — the same service/REST layers served from a GUI-free process. The factory mirrors gui_hcs.load_objects() wiring with the Qt-free controller base classes; the entry point owns SIGINT/SIGTERM itself (uvicorn's main-thread signal capture replays the signal after serve(), which would skip teardown) and on shutdown aborts an in-flight acquisition, closes the hardware, and reaps leftover JobRunner children.

Where human judgment matters most (machine review already did line-level correctness — every task had an independent implement/review cycle plus a final whole-branch review):

• GUI/API coexistence: both drive the shared MultiPointController; API runs are 409-guarded, GUI runs are tracked via chained callbacks. A simultaneous-start race window exists and is documented as a known limitation in core-service-api.md.
• Z semantics: API path now honors z_stacking_config (previously ignored) and resets z_range + focus-map state on every run — intended, matches GUI behavior, but worth a domain-expert eye.
• Security posture decisions: bearer auth off by default on loopback / mandatory on non-loopback binds; python_exec explicitly unsandboxed behind a GUI opt-in toggle.
• Port 8060 instead of the spec's 5060 (browsers hard-block 5060 as SIP → ERR_UNSAFE_PORT).
• Deprecation stance: legacy TCP 5050 untouched and still starts alongside; both gated by the same Settings toggle.

Fastest way to see it work: python3 main_hcs.py --simulation --start-server, then open http://127.0.0.1:8060/docs (live Swagger against the sim) or follow software/docs/quickstart-api.md — every example there was verified against the simulator. For the GUI-free variant: python3 main_headless.py --simulation.

🤖 Generated with Claude Code