feat: coordinate concurrent CBM sessions#1139
Draft
DeusData wants to merge 41 commits into
Draft
Conversation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ion-daemon Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> # Conflicts: # src/mcp/mcp.c # tests/test_cli.c
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> # Conflicts: # Makefile.cbm
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Unix-VFS MAX_PATHNAME bump from 512 to 4096 shipped with the daemon work but left no paper trail, so the next amalgamation refresh would silently revert it. Document it in vendored/sqlite3/PATCHES.md with the rationale and re-apply procedure, and point to it from THIRD_PARTY.md. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The six suites that spawn coordinated worker subprocesses or bind local endpoints (index_supervisor, daemon_application, daemon_runtime, daemon_frontend, daemon_bootstrap, daemon_ipc) ran in the parallel wave, where the saturated 3-4-core CI runners starve their fixed readiness deadlines: index_supervisor waits at most 3 s for a worker marker while the worker is a full re-exec of the ASan runner plus the admission handshake. Both macOS legs failed the same four readiness assertions deterministically and ubuntu-latest added four daemon_frontend and one daemon_runtime timing failures, while an idle machine passes the same suites 6/6 in the wave. Moving them to the quiet serial tail follows the runner's existing rule for this class (cli/subprocess/watcher/...) and also keeps the shared per-account coordination namespace free of cross-suite admission traffic. Same suites, same tests, same union-guarded totals - only the schedule changes. Locally re-verified: 6742 passed, 0 failed, 2 skipped (120 suites, 17 serial-tail). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The launcher answered every ownership/ACL problem with one generic "launcher ownership or access policy is unsafe", which made the CI guard failures (and any field report) undiagnosable: the same message covers an untrusted owner SID, a mutation-capable ACE, a reparse point, a hard-link clone, and a plain open failure across the whole ancestry walk. Record the failing check at its site - offending path, owner or ACE SID in SDDL string form, rights mask, Win32 error - and print it once alongside the refusal. No policy changes; refusals stay refusals. The interactive MCP smoke now also dumps daemon-conflicts.ndjson next to the daemon log tail on failure, so an admission conflict names itself instead of surfacing as a bare 30 s client timeout. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…guards Completes the handoff task for the Windows harness: the native test job ran with the MSYS-shared /tmp and the runner's inherited LocalAppData\Temp ACLs, which the daemon's strict private-filesystem checks reject by design. The test-windows job now creates a per-user root under the profile with an owner-stamped, protected current-SID DACL and routes TEMP/TMP (native form) and TMPDIR (POSIX form) through it before scripts/test.sh runs. The guards script already hardened its root DACL but ownership is never inherited on Windows: bundle copies created by the admin-group runner token can come out Administrators-owned, and the launcher's exe policy requires the exact current user as owner. Stamp the current SID on the staged bundle dir and both executables explicitly. Job topology, gates, and triggers are unchanged; the new step is a sub-second pwsh block in the existing test-windows job. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…bjects The Windows SECURITY_ATTRIBUTES used for every private-namespace creation (runtime dirs, rendezvous/startup/participant/lifetime/cohort lock files, log files, pipes) carried a DACL but no owner, so created objects took the token DEFAULT owner. Admin-group tokens can default that to BUILTIN\Administrators - standard policy on Windows Server, including CI runners - while every strict validation gate (win_file_owner_secure(require_current_user=true), private_win_owner_only_dacl) demands the exact token-user SID. Result: the daemon rejected objects it had just created itself; the client refused the daemon's rendezvous record, retried silently for its full 30 s window, and each clientless generation clean-stopped, looping until the frontend gave up. The legacy kernel mutex was the one object that already force-set its owner post-create - it is the template this change generalizes. Both descriptor builders now set the owner explicitly at creation, so objects are born satisfying the same strict gates on every default- owner policy. The gates themselves are unchanged. The Windows client connect path also stops failing mute: terminal rendezvous refusals, deadline expiry (with last status), and pipe server-identity rejections now log their reason once, matching the launcher-side refusal diagnostics. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stamping the owner at creation covers objects born through the daemon's own security descriptors, but the cache root is created by plain mkdir_p before the gate runs, and harness- or user-created directories arrive the same way. On an Administrators-default-owner token those directories are born owned by BUILTIN\Administrators and the exact-user gate refused them permanently - including a real elevated admin's own cache directory. win_runtime_directory_secure (the final component of both the runtime dir and the cache-root validation) now treats a TRUSTED owner - the same SYSTEM/Administrators/TrustedInstaller set the launcher already accepts for directories - as repairable: the owner is re-stamped to the exact token user inside the same set_security_info call that already re-protects the DACL. Any other owner is refused exactly as before, the post-repair validation still demands the exact user, and the handle falls back to the old access mask when WRITE_OWNER is unavailable so no previously-passing configuration can newly fail. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The new pipe_rejected diagnostic proved the Windows client reaches the daemon's pipe and then refuses the SERVER identity for its whole 30 s window, while earlier sessions in the same run connect fine. The check compares the server-process token user, so the interesting failures are environmental: a dead server PID already reused by Windows (typically for a SYSTEM service) is indistinguishable from a hostile pipe without classification. The rejection now names the failing step (pid query, process open, token open/query) or the server SID class (system/admins/other) plus the server PID. Also reflows the two owner-stamping call sites the lint gate flagged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…l legs The local containers ran with every host core, which masks the exact failure class the 4-core GitHub runners catch: deadline/starvation misses in worker-readiness waits and watcher/subprocess deadlines. The test services now default to a 4-CPU cap (CBM_LOCAL_CI_CPUS overrides, 0 = unconstrained), so a local run exercises the same scheduling pressure as CI before CI ever sees the push. Compilation is cached through ccache in persistent named volumes with CCACHE_COMPILERCHECK=content: every hit is keyed on the compiler-binary content plus the fully preprocessed input, so a stale or foreign entry can only miss, never return wrong output. Also documents the Wine leg's honest limit: it compile-checks Windows but cannot reproduce real ACL/token/owner semantics. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…lity mode Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The infra previously assumed Docker Desktop: an implicitly running daemon and bundled CLI plugins. On the Colima runtime (free OSS, CLI-first) neither is implicit, and a missing daemon surfaced as cryptic compose errors. run.sh now fails fast with the exact setup commands (colima start with --vz-rosetta for fast amd64 legs, plugin symlinks for the brew docker client) and documents the runtime in the header. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The four readiness/terminal failures on the CI macOS legs persist even with the suite serialized on a quiet machine, so the worker is exiting rather than starving - and its exit reason lives in the worker log and response files, which teardown deletes before the asserts print. Dump them at each failure site (async readiness, probe non-terminal, backlog logging, oversize containment) so the next failing run names the worker's actual error instead of a bare ASSERT. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two failure classes stay invisible to the container legs: kernel-real Windows ACL/token/owner semantics (Wine only compile-checks) and the GitHub-runner macOS environment (a developer Mac differs in temp ancestry, ACLs, and policies - runner-only failures never reproduce locally). Add real-VM legs for both, strictly opt-in: run.sh detects the tooling and the named VM and prints setup guidance with a distinct exit code when absent - explicitly requested legs are never silently skipped. windows-vm: UTM + CrystalFetch (both free), driven over ssh; the in-VM bootstrap (vm/windows-bootstrap.ps1) installs the CLANG64 toolchain, enables sshd, and mirrors the runner's NoDefaultAdminOwner=0 policy so freshly created objects are owned by BUILTIN\Administrators - the exact class the daemon's owner gates must survive. mac-vm: Cirrus Labs' runner-equivalent image via tart (free OSS), booted headless, worktree synced over key-based ssh, scripts/test.sh executed in the runner-class environment. vm/README.md documents both setups end to end. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… pinned Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…fusal The cache-private identity failure printed a bare status name while the actual rule that refused - POSIX owner/mode/extended-ACL steps, Windows owner class, mutation-capable DACL entries, ancestry components, the owner/DACL repair - stayed mute, costing a blind CI round-trip per guess. Both platform gates now record the object and rule at the exact failing check (cbm_daemon_ipc_validation_detail), main.c appends it to the identity message, and the Windows client's server-identity check logs its two previously-silent early exits (server PID query, process open - a dead server PID that Windows reaped or reused). Validation policy is unchanged everywhere; refusals only stop being anonymous. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The worker-log dumps settled the CI macOS failures: workers were alive with created-but-EMPTY logs when the 3 s readiness / 5 s terminal deadlines expired - busy in startup, not hung and not refused. Worker startup re-hashes the entire executable image for the exact-build fingerprint before its first write; for the ASan test-runner that is a multi-hundred-MB read+SHA-256, which the 3-core CI runner VMs with cold IO cannot finish inside the old budgets, while a warm 18-thread dev machine finishes in well under a second. Raise the fixture budgets to 15 s readiness / 20 s terminal. These are hang guards, not benchmarks: a wedged worker still fails loudly, and the failure path now dumps the (empty) worker log as proof either way. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…runners Same class as the index-supervisor calibration, proven by the serial tail: the daemon_frontend backpressure/EOF tests and the cli forced- install quiesce child failed on the 4-core ubuntu leg even running alone on a quiet machine. Their 2 s request/cleanup/acquire budgets are below real startup cost there (identity capture, frontend spawn, lock acquisition on cold IO). Raised to 10-30 s; still hang guards - a wedged run fails loudly. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ccache with CCACHE_COMPILERCHECK=content everywhere: every cache entry is keyed on the CONTENT of the compiler binary plus the fully preprocessed translation unit, so a hit is provably the identical compilation - a stale, foreign, or corrupted cache can only MISS, never return wrong output. No CCACHE_BASEDIR and no path rewriting: debug info and sanitizer report paths stay exact. Locally scripts/env.sh routes compilers through ccache's masquerade directories when present (opt-out CBM_NO_CCACHE=1) - $CC keeps its plain name, so verify_compiler, make, and link lines are untouched. CI caches are strictly per-ref by policy on top of GitHub's own branch scoping: keys embed github.ref, so no base-branch fallback - a new PR builds cold once and only its own pushes warm it. Cached: the four test jobs and the pr-smoke matrix. Release builds in _build.yml stay deliberately uncached. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Container legs with content-verified ccache, CI-fidelity CPU mode (CBM_LOCAL_CI_CPUS=4 reproduces the runner starvation class locally - proven against this branch's own index-supervisor failures), Colima runtime with preflight checks, and the opt-in windows-vm/mac-vm real-VM legs for the two classes containers cannot reproduce. Landed together with the daemon so its remaining Windows work iterates on this surface. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Combines the ccache steps with this branch's Windows hardening: the protected temp root and profile staging keep their roles, the cache steps slot alongside them, and the strictly per-ref keys apply unchanged. The daemon branch's own CI legs warm from the second push on. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…efusals The classified smoke run pinned the Windows client loop on GetNamedPipeServerProcessId failing against a successfully opened pipe (step=server_pid_query, 12 hits in one 30 s window). Two data points were still missing: the Win32 error of that exact call (and whether the function pointer resolved at all), and the server-side view - the daemon's accept path disconnects unverifiable clients silently, which a client experiences as exactly this failure. Both now log. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The instrumented smoke run pinned it: the daemon rejects the connecting client's identity 242 times in one window (daemon.accept.client_rejected stage=client_identity), and the client's error 233 (ERROR_PIPE_NOT_ CONNECTED) is only the downstream symptom of the server disconnecting it. The rejection lives in win_pipe_client_is_current_user but its single false covered three very different causes. Split them: impersonate failure (ERROR_CANNOT_IMPERSONATE = client wrote no frame yet / accept fired on an empty pipe), thread-token open failure, and a genuine SID mismatch (now naming the impersonated client's SID class - anonymous/system/administrators/other). The 242-in-a-loop count points at an accept-spin on client-less connections; the next run's step label confirms which. Windows-only diagnostic path; unix gate is unaffected and green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds one mandatory per-account coordination daemon for long-lived MCP and hook sessions.
Coordination and activation
All concurrent CBM processes must agree on the exact version, executable build, coordination ABI, and canonical
CBM_CACHE_DIR. Conflicting processes fail before doing work, report the conflict to the launching session, and record it indaemon-conflicts.ndjson.Native
install,update, anduninstallcommands:Windows uses a permanent launcher with an atomically replaceable payload, allowing the same lifecycle guarantees despite executable-file locking. npm, PyPI, and Go wrappers verify and publish their private cached binaries safely.
Security and observability
Verification
Recent local verification on the merged checkpoint line:
Current-head GitHub CI is intentionally left running on this draft PR.
Handoff / where work stopped (2026-07-18)
Pushed checkpoint:
7696b1486a10f19e231f5b8285abe6eea71b9a3c.The latest checkpoint additionally:
/usr/bin/true, which is policy-dependent on older macOS runners;mainand its isolated parallel-suite runner.Known next task:
TEMP,TMP,TMPDIR, runtime/cache paths, and the MSYS/tmpmapping through it for the full Windows harness. Re-run exact-head Windows smoke/guards/native tests afterward.No merge was performed; the PR remains draft for the follow-up session.