iron — VxCloud heavy-task engine (C++23 / io_uring)

iron is the durable, heavyweight half of the VxCloud worker pair (ion is the Rust micro-worker; iron is the C++23 heavy worker). It executes long-running tenant tasks — supervised subprocesses (headless Chromium/Playwright, DB migrators, streaming parsers) and io_uring file pipelines — behind the frozen 93-byte vx_task_header_t ABI, and embeds a supervisor for an outbound-only cloudflared QUIC tunnel so a tenant VM needs zero open inbound ports.

Everything asynchronous in this repo is a C++23 coroutine parked in an io_uring SQE and resumed from the CQE loop. There are no thread pools hiding blocking syscalls behind a coroutine facade.

Architecture

                 vx_task_header_t (93B, packed LE)            vx_result_header_t (29B)
   host vxnode ────────────────────────────────► iron ────────────────────────► host
                                                  │
              ┌───────────────────────────────────┴───────────────────────────────┐
              │                            engine.hpp                              │
              │      decode (abi.hpp, std::expected)  →  dispatch by payload op    │
              └──────┬──────────────────────────┬──────────────────────────┬──────┘
                     │ op=exec                  │ op=file-pipeline         │ sidecar
              ┌──────▼──────┐            ┌──────▼──────┐            ┌──────▼──────────┐
              │  proc.hpp   │            │ awaitable   │            │   tunnel.hpp    │
              │ fork/execvp │            │ read/write  │            │ cloudflared QUIC│
              │ pipes+pidfd │            │ openat/close│            │ supervisor + SM │
              │ TERM→KILL   │            │ (io_uring)  │            │ backoff+jitter  │
              └──────┬──────┘            └──────┬──────┘            └──────┬──────────┘
                     │      co_await awaiters park handles in SQE user_data │
              ┌──────▼─────────────────────────▼───────────────────────────▼──────┐
              │                      ring.hpp — io_uring                           │
              │   SQPOLL / COOP_TASKRUN feature ladder · batched submit            │
              │   wait_cqe_timeout · peek_batch_cqe + cq_advance drain             │
              └────────────────────────────────────────────────────────────────────┘
                                        task.hpp — task<T>
                     lazy start · symmetric transfer · exception propagation
                     RAII cancellation · spawn()/joinable<T> structured join

                              shutdown.hpp — shutdown_controller
                SIGTERM/SIGINT → signalfd polled on the ring · bounded drain
                     → TERM→KILL escalation · truthful frames · 128+signo

The io_uring feature ladder

ring asks the kernel for the best setup it can get and degrades gracefully, recording every attempt:

rung	flags	why it can fail
1	SQPOLL | SINGLE_ISSUER	SQPOLL needs privilege — classic EPERM for unprivileged users on older kernels
2	SQPOLL	same
3	COOP_TASKRUN | SINGLE_ISSUER	kernel < 5.19 / not combinable with SQPOLL (kernel rejects the combination, which is why these are separate rungs)
4	COOP_TASKRUN	kernel < 5.19
5	plain ring (flags=0)	works on every io_uring kernel

With SQPOLL the kernel polls the submission queue from its own thread, so io_uring_submit() is just a tail-pointer store while the SQ thread is awake — no syscall per submit. The negotiated rung is exposed via ring::mode() and the whole attempt trace via ring::attempts(); tests force lower rungs through ring_config::max_mode and assert each one still completes real io.

Real output on this box (WSL2, kernel 6.6.87.2, running as root):

$ iron ring-info
requested ladder start : sqpoll+single_issuer
negotiated mode        : sqpoll+single_issuer
entries                : 256
attempts:
  sqpoll+single_issuer         flags=0x00001002 -> ok

requested ladder start : plain
negotiated mode        : plain
entries                : 256
attempts:
  plain                        flags=0x00000000 -> ok

On unprivileged GitHub runners rung 1–2 typically fail with EPERM and the ladder lands on rung 3 — CI is written to accept whatever rung the kernel grants (the suite asserts functionality per rung, not a specific rung).

Zero open inbound ports — the tunnel model

iron tunnel --token=… supervises cloudflared tunnel run. cloudflared dials outbound to the Cloudflare edge over QUIC (HTTP/3) — typically 4 connections to distinct edge PoPs — authenticates with the tunnel token, and then all ingress for the tenant's hostnames is multiplexed back over those established connections as independent HTTP/3 streams. Because every packet of every session rides connections the VM itself initiated:

• the VM's firewall/security group can drop all inbound traffic (22 included),
• NAT/CGNAT and private RFC1918 addressing are irrelevant,
• per-stream multiplexing means one tunnel carries many concurrent requests with no head-of-line blocking between streams (QUIC, unlike TCP muxing).

iron drives this state machine from cloudflared's own log lines:

Stopped → Starting → Registering → Connected(n edges)
              │                        │      ▲
              ▼                        ▼      │ edge re-registers
            Failed  ◄─ fatal token   Reconnecting ─ backoff+jitter restart

Supervision rules: log line Registered tunnel connection increments the edge count, Unregistered/Lost connection decrement (0 ⇒ Reconnecting), a malformed/unauthorized token is fatal (no crash-looping on a bad config), and process death triggers restart with exponential backoff + jitter (500 ms → 30 s cap by default) up to max_restarts. A missing cloudflared binary is the typed error tunnel_error::binary_not_found.

The state machine and the supervisor are tested against tests/fake_cloudflared.sh, which replays realistic log sequences (happy 4-edge bring-up, edge flap → Reconnecting → recovery, crash → backoff restarts → restarts_exhausted, fatal token), so the Connected/ Reconnecting transitions are genuinely exercised without the real binary.

$ iron tunnel --dry-run --token=EXAMPLE_TOKEN_PLACEHOLDER
would exec: cloudflared tunnel run --token <token>
binary 'cloudflared': found

(The token is never logged; pass it via --token= or embed in config management — this repo contains no credentials.)

Substrate matrix

substrate	how iron runs	notes
ephemeral VM (VxCloud instant sandbox)	single iron process supervised by vxnode; tunnel sidecar per VM	zero inbound ports; cgroup limits from vx_task_header_t applied by the host
long-lived tenant VM	systemd unit iron tunnel + iron run per task	SQPOLL available when granted CAP_SYS_NICE / root
Kubernetes pod	iron as the task container, cloudflared as a sidecar container	ladder lands on COOP_TASKRUN or plain under default seccomp; io_uring must not be blocked by the runtime profile
CI (GitHub ubuntu-24.04)	build + ctest	SQPOLL usually EPERM ⇒ fallback rungs carry the suite

Task ABI

third_party/vx/worker_abi.h is the frozen v1 contract (vendored byte-identical; Apache-2.0). iron validates magic 0x58575601, rejects truncated/oversized records via std::expected (no exceptions on the hot path), reads all fields with memcpy at static-asserted offsets (no misaligned/aliasing UB on the packed layout), and always answers with a well-formed 29-byte vx_result_header_t:

condition	vx_task_state_t
handler succeeded, exit 0	VX_STATE_COMPLETED
nonzero exit / bad payload / io error	VX_STATE_FAILED
wall-clock limit hit (TERM→KILL escalation)	VX_STATE_KILLED_TIMEOUT
child died on a signal it wasn't sent by us	VX_STATE_KILLED_SIGNAL
in-flight child cut short by a shutdown drain	VX_STATE_KILLED_SIGNAL + interrupted=1 in payload
task refused because iron was already draining	VX_STATE_PENDING + skipped=1 in payload ("reschedule me")

Payload format ("iron task payload v1", line-oriented key=value; the ABI treats payloads as opaque bytes so JSON/Protobuf handlers can be added without a contract change):

op=exec                    |  op=file-pipeline
timeout_ms=5000            |  src=/data/in.csv
max_output=1048576         |  dst=/data/out.csv     (optional copy)
cwd=/tmp                   |
env=K=V                    |
arg=/usr/bin/python3       |
arg=migrate.py             |

Graceful shutdown — iron itself is drainable

A docker stop, systemd restart or Kubernetes eviction SIGTERMs iron, not its children. Dying instantly would abandon a DB migrator mid-migration, orphan a headless Chromium, and emit no result frame for in-flight work — so iron installs its own drain. SIGTERM and SIGINT are blocked and read from a signalfd whose IORING_OP_POLL_ADD is parked on the ring: a signal is just another CQE handled on the single engine thread. Nothing runs in async-signal context (no sig_atomic_t, no self-pipe), and SIGPIPE is ignored so a dead pipe surfaces as EPIPE instead of killing a drain.

The sequence, per signal:

1. first SIGTERM/SIGINT — stop accepting tasks: work not yet started is answered with a VX_STATE_PENDING frame (skipped=1, "reschedule me"). In-flight tasks keep running up to min(their own timeout, drain grace). Already-parked pidfd waits are woken through the ring with IORING_OP_ASYNC_CANCEL (cancel-by-fd, kernel ≥ 5.19; older kernels fall back to a 1 s re-arm slice, so a drain can never hang on a parked wait).
2. grace expires — every surviving child group gets the existing SIGTERM → term_grace → SIGKILL escalation, is reaped (waitpid, no zombies), and its task is answered with VX_STATE_KILLED_SIGNAL (interrupted=1). For iron tunnel the cloudflared sidecar is stopped (TERM, then KILL past the grace).
3. second SIGTERM/SIGINT — the grace window is abandoned immediately (what an impatient operator, or docker stop followed by docker kill, expects); escalation starts now.
4. exit status distinguishes the outcomes: 0 everything completed (even if a shutdown arrived while the last task drained), 1 a task failed on its own, 128+signo (143 TERM / 130 INT) a requested shutdown cut work short — never confusable with a crash.

The grace is --drain-ms=N (default 5000 ms): it must fit inside the outer supervisor's kill window, and the tightest common one is docker stop's 10 s — 5 s of drain plus a worst-case 2 s TERM→KILL escalation still leaves 3 s of headroom for result frames and exit (Kubernetes defaults to 30 s, systemd to 90 s, so those are comfortable too). --drain-ms=0 escalates immediately.

Captured on this box (iron run with a 1 s task in flight and a second task queued; TERM sent 0.4 s in):

ts=… lvl=warn comp=shutdown msg="signal received, draining" signo=15 grace_ms=8000
ts=… lvl=warn comp=engine msg="refusing task, shutdown in progress" task_id=9102
task_id=9101 state=2 exit_code=0 duration_us=1001847 … -> demo1.task.result   # COMPLETED — drained
task_id=9102 state=0 exit_code=0 duration_us=0       … -> demo2.task.result   # PENDING — refused
exit=143 elapsed_ms=1005

and the same TERM-ignoring 30 s child with one signal vs. two (--drain-ms=3000, term_grace_ms=400): one signal escalates exactly at the 3.0 s grace (elapsed ≈ 3.8 s), a second signal 0.5 s in collapses it to elapsed ≈ 1.3 s. Children never inherit iron's shutdown plumbing: spawn() clears the blocked mask and resets SIGPIPE before execvp, otherwise kill(-pgid, SIGTERM) could never reach them.

CLI

$ iron --version
iron 0.4.0 (abi v1, header 93B/29B)

$ iron exec --timeout-ms=2000 -- /bin/sh -c 'echo hello from a supervised child; uname -r'
hello from a supervised child
6.6.87.2-microsoft-standard-WSL2
iron-exec: exit_code=0 term_signal=0 timed_out=0 interrupted=0 duration_us=13442

$ iron exec --timeout-ms=300 -- /bin/sleep 10        # escalation demo
iron-exec: exit_code=-1 term_signal=15 timed_out=1 interrupted=0 duration_us=303025
$ echo $?
124

$ iron run demo.task        # 93B header + payload, produced by the host
task_id=9001 state=2 exit_code=0 duration_us=2726 payload_bytes=150 -> demo.task.result

$ iron selftest
  [ok] abi encode/decode roundtrip
  [ok] task record size
  [ok] coroutine sleep on io_uring
  ring mode: sqpoll+single_issuer
  [ok] subprocess capture
selftest: ALL OK

All output above is captured verbatim from this machine.

Build & test

Requirements: Linux ≥ 5.15 (6.x recommended), GCC 14 (-std=c++23, std::expected), liburing ≥ 2.5, CMake ≥ 3.28, Ninja.

cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER=g++-14 -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

The tree compiles with -Wall -Wextra -Wpedantic -Werror. Result of the full suite on this box:

100% tests passed, 0 tests failed out of 7      (task abi ring proc tunnel engine shutdown)

Sanitizers: -DIRON_SANITIZE=address,undefined builds the whole tree + tests under ASan+UBSan. The sanitized suite also passes 7/7 (it caught one dangling result_view over a temporary buffer in a test during development — the API docs now spell out that decoded views borrow the input buffer). The shutdown suite runs its end-to-end cases against the sanitized iron binary itself, so signal-driven drains are leak-checked too (the controller's watcher coroutine is cancelled and joined before the ring is torn down).

Benchmarks (real, measured)

Environment: WSL2 kernel 6.6.87.2-microsoft-standard, GCC 14.2 -O3, ring mode sqpoll+single_issuer, tmpfs file, single thread. bench/bench_ring.cpp, representative run:

ring mode        : sqpoll+single_issuer
nop batched      : n=500000 qd=64  7.58 Mops/s
nop single-op    : n=30000  p50=0.48us  p99=6.06us
read 4KiB QD1    : n=20000  p50=3.83us  p99=65.19us
read 4KiB QD32   : n=200000  411 kops/s  1605 MiB/s

Cold start (iron --version, 20 runs): min 2 ms / median 3 ms / max 4 ms; iron ring-info — which performs two full ring setups including the SQPOLL kernel thread — median 4 ms. Target was < 45 ms.

Numbers are from a shared dev VM; treat them as order-of-magnitude, and rerun bench_ring on your substrate.

Repository layout

include/iron/   task.hpp ring.hpp awaitable.hpp proc.hpp tunnel.hpp abi.hpp log.hpp engine.hpp
                shutdown.hpp (graceful-drain controller: signalfd on the ring)
src/            implementations + main.cpp (CLI)
tests/          self-contained harness + 7 suites + fake_cloudflared.sh
bench/          bench_ring.cpp
third_party/vx/ worker_abi.h (frozen ABI, vendored byte-identical)
cmake/          FindUring.cmake
scripts/        build.sh measure.sh
.github/        workflows/ci.yml (build + ctest + ASan job on ubuntu-24.04)

Security

• This repo is public: it contains no credentials, tokens, private IPs or tenant identifiers. Tunnel tokens come from --token=/environment at runtime.
• The subprocess supervisor runs children in their own process group, reaps them unconditionally, caps captured output, and never leaks pipe/pidfd fds (asserted by tests via /proc/self/fd).
• Decode paths are exception-free and bounds-checked before any field read.

License: Apache-2.0, matching prodxcloud/worker and prodxcloud/ion — the vendored worker_abi.h carries the same licence, so the whole stack is uniform.