Skip to content

Add headless Wayland compositor with GUI surface streaming#154

Open
pcarrier wants to merge 2 commits intomainfrom
omnirelease
Open

Add headless Wayland compositor with GUI surface streaming#154
pcarrier wants to merge 2 commits intomainfrom
omnirelease

Conversation

@pcarrier
Copy link
Copy Markdown
Contributor

@pcarrier pcarrier commented Apr 4, 2026

Embed a smithay-based headless Wayland compositor into every PTY session
(Linux and macOS) so GUI applications launched inside blit stream their
surfaces to remote clients as real-time H.264/AV1 video. This turns blit
from a terminal multiplexer into a full remote-desktop primitive: any
Wayland app — terminals, browsers, editors, media players — can be
launched inside a blit session and viewed in the browser alongside
terminal panes.

Server (Rust)

  • New crates/compositor — headless Wayland compositor (#[cfg(unix)]) implementing wl_shm, linux-dmabuf, xdg-shell, xdg-decoration, wp_viewporter, primary selection, fractional scaling, cursor shape, text input, and XDG activation protocols via smithay 0.7.
  • New crates/server/src/surface_encoder.rs — AV1 (rav1e), H.264 (openh264), and VA-API hardware-accelerated encoding behind a unified SurfaceEncoder interface. BLIT_SURFACE_QUALITY (low/medium/high/lossless) controls rav1e speed/quantizer presets. BLIT_SURFACE_ENCODER selects the codec (auto/av1/h264-software/h264-vaapi).
  • Per-surface subscription protocol (C2S_SURFACE_SUBSCRIBE / UNSUBSCRIBE) so clients only receive frames for surfaces they are viewing.
  • Wire-protocol extensions in crates/remote for surface lifecycle, frames, input, pointer, axis, resize, focus, clipboard, and capture.
  • CLI surface automation: surfaces, capture, click, key, type subcommands.
  • Compositor shutdown when all PTYs in a session exit or close.
  • LISTEN_PID validation before adopting systemd socket-activation fds.

Client (TypeScript)

  • SurfaceStore — tracks surface metadata and decodes H.264/AV1 frames via WebCodecs (avc1.42001f / av01, optimizeForLatency).
  • BlitSurfaceCanvas — OffscreenCanvas renderer with keyboard, mouse, pointer-axis, resize, focus, and clipboard forwarding.
  • BlitSurfaceView components for React and Solid.
  • Surface subscribe/unsubscribe messages in protocol.ts.
  • WebTransport close-reason surfacing into connection error state.

UI (js/web-appjs/ui)

  • Rename js/web-app to js/ui.
  • Surfaces promoted to first-class interactive BSP panes with resizable preview panel and off-screen session/surface thumbnails.
  • Passphrase encryption in URL hash (tweetnacl) — plaintext hashes are automatically replaced on first load.
  • Status bar: disconnect reason display, debounced connection state, full-bar background color instead of status circle.
  • Switcher overlay: surface entries, move-to-pane support, layout presets and history.

Infrastructure

  • Release workflow refactored: bin/release-prepare opens a PR, bin/release-tag creates a signed tag; CI verifies tag signature before publishing.
  • bin/dev supports DEV_INSTANCE for parallel dev stacks on non-colliding port blocks.
  • Nix devShell and packages updated with pixman, libxkbcommon, smithay, rav1e, and optional VA-API deps (Linux-only gating for Darwin CI).
  • install.sh defaults to user-local install.
  • E2E test stability: wait for DOM to settle after hash encryption, wait for async hash write in auth reload test.
  • Documentation updates across README, ARCHITECTURE, CONTRIBUTING, EMBEDDING, SKILL, UNSAFE, learn.md, and man pages.

@indent
Copy link
Copy Markdown
Contributor

indent bot commented Apr 4, 2026

PR Summary

This PR adds a headless Wayland compositor to every PTY session in blit, enabling GUI applications launched inside terminal sessions to stream their surfaces to remote clients via H.264/AV1 video encoding. The compositor integrates with smithay on the server side and WebCodecs on the client side, with full input forwarding and surface lifecycle management.

  • New smithay-based headless Wayland compositor (crates/compositor) supporting wl_shm, linux-dmabuf, and xdg-shell protocols
  • Surface video encoding via openh264 (H.264), rav1e (AV1), and VA-API hardware acceleration with auto codec selection
  • Wire protocol extensions: 8 new S2C message types and 10 new C2S message types for surface frames, metadata, input events, and codec negotiation
  • Client-side SurfaceStore with WebCodecs VideoDecoder and OffscreenCanvas rendering, plus Canvas2D fallback
  • React and Solid BlitSurfaceView components with pointer/keyboard input forwarding
  • Surface subscribe/unsubscribe with reference counting and keyframe-on-subscribe
  • SurfaceQuality presets (Low/Medium/High/Lossless) controlling rav1e quantizer and speed
  • CLI commands for surface automation: surfaces, capture, click, key, type
  • Renamed js/web-app to js/ui with surfaces as first-class BSP panes
  • Renamed BLIT_PASS env var to BLIT_PASSPHRASE across gateway, e2e, process-compose, and nix modules
  • Restored missing virtual-blit-wasm.d.ts type declaration
  • Formatting fixes (cargo fmt) across server and Windows PTY code

Issues

3 potential issues found:

  • Multiple ungated eprintln! calls in crates/server/src/lib.rs (lines 1964-1969, 1979, 1988, 2051-2054, 2061, 2064, 2077) fire on every force-keyframe and frame delivery cycle without a verbose guard. Other encoder logging in this file is gated behind state.config.verbose. → Autofix
  • S2C_SURFACE_FRAME parser will panic on truncated messages: the bounds check is data.len() < 12 but the parser accesses data[13], so a 12- or 13-byte malformed message causes an index-out-of-bounds panic. → Autofix
  • The BLIT_PASS to BLIT_PASSPHRASE rename is incomplete: man/blit-gateway.1.scd still references BLIT_PASS in its ENVIRONMENT section. → Autofix
8 issues already resolved
  • All six new surface send methods (sendSurfaceInput, sendSurfacePointer, sendSurfaceAxis, sendSurfaceResize, sendSurfaceFocus, sendClipboard) call this.transport.send() without checking this.transport.status === "connected", unlike every other send method in the class. (fixed by commit b190eed)
  • Both AV1 and VAAPI encoders clear force_keyframe = false before calling send_frame. If send_frame fails (its result is discarded with let _ = for AV1, and .ok()? for VAAPI), the keyframe request is permanently lost with no retry mechanism. (fixed by commit b190eed)
  • When BlitConnection disconnects or errors, handleStatusChange does not clear or destroy the surfaceStore. Stale surfaces with active decoders persist across reconnections, and if surface IDs are reused after reconnect, old decoders (potentially for a different codec) will be reused for new frames. (fixed by commit b190eed)
  • BlitConnection.dispose() calls this.store.destroy() but never calls this.surfaceStore.destroy(), leaking all VideoDecoder instances and canvas resources. Browsers limit hardware codec slots, so this will exhaust codec capacity in long-lived apps. (fixed by commit b190eed)
  • Tests at lines 821-827 reference SurfaceH264EncoderPreference::Software and ::Vaapi, but the actual enum variants are H264Software and H264Vaapi. These tests won't compile. (fixed by commit b190eed)
  • validate_surface_dimensions() rejects odd dimensions before codec selection, but AV1 handles odd dimensions correctly (the code even comments "AV1 has no even-dimension constraint"). Requesting AV1 or Auto with an odd-dimension surface (e.g., 1919x1079) fails with an H.264-specific error. (fixed by commit b190eed)
  • The C2S_CLOSE handler removes the PTY but never checks whether all remaining PTYs have exited, so it never triggers compositor shutdown. If PTYs are closed via C2S_CLOSE rather than exiting naturally, the compositor thread leaks until the server exits. (fixed by commit b190eed)
  • After a WebCodecs VideoDecoder error, the decoder enters "closed" state but is never reinitialized. Every subsequent frame throws at decode(), gets caught, sets pendingKeyframe = true, and repeats — permanently breaking the surface. (fixed by commit b190eed)

CI Checks

Same two persistent issues continue to fail CI across all runs.

Failing lint→ Autofix
  • cargo fmt check fails with formatting diffs in crates/server/src/lib.rs (6 locations), crates/server/src/pty/pty_windows.rs (3 locations), and crates/server/src/surface_encoder.rs (1 location). Run cargo fmt to fix.
Failing test (ubuntu-latest)→ Autofix
  • TypeScript typecheck fails: js/ui/src/wasm.ts(1,24): error TS2307: Cannot find module 'virtual:blit-wasm'. The virtual-blit-wasm.d.ts type declaration file was deleted during the js/web-app to js/ui rename and never recreated.
Failing e2e→ Autofix
  • Cascading failure — e2e depends on successful builds which fail due to the same cargo fmt and TypeScript issues.

⚡ Autofix All

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 4, 2026

🔗 Preview: https://blit-cbk92m527-indent.vercel.app

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 4, 2026

Coverage

Crate Lines Functions Regions
alacritty-driver 59.7% (529/886) 62.5% (40/64) 59.8% (754/1261)
browser 0.0% (0/807) 0.0% (0/65) 0.0% (0/1370)
cli 38.9% (1426/3666) 53.4% (190/356) 43.6% (2514/5765)
compositor 14.3% (123/859) 12.1% (8/66) 16.0% (200/1249)
fonts 76.9% (486/632) 85.5% (47/55) 78.2% (922/1179)
gateway 34.6% (241/696) 32.4% (23/71) 26.4% (291/1104)
remote 72.9% (1925/2639) 85.3% (185/217) 76.1% (3654/4804)
server 37.8% (1749/4633) 54.2% (199/367) 38.3% (2939/7673)
webrtc-forwarder 0.0% (0/1467) 0.0% (0/93) 0.0% (0/2505)
webserver 59.2% (248/419) 73.3% (44/60) 64.7% (473/731)
Total 40.3% (6727/16704) 52.1% (736/1414) 42.5% (11747/27641)

@pcarrier pcarrier force-pushed the omnirelease branch 3 times, most recently from e8bc05f to b18fff5 Compare April 4, 2026 11:31
@pcarrier pcarrier changed the title Add headless Wayland compositor with surface streaming Add headless Wayland compositor with GUI surface streaming Apr 4, 2026
@pcarrier pcarrier force-pushed the omnirelease branch 4 times, most recently from 5d7bf8c to db10561 Compare April 4, 2026 14:02
Embed a smithay-based headless Wayland compositor into every PTY session
(Linux and macOS) so GUI applications launched inside blit stream their
surfaces to remote clients as real-time H.264/AV1 video.  This turns blit
from a terminal multiplexer into a full remote-desktop primitive: any
Wayland app — terminals, browsers, editors, media players — can be
launched inside a blit session and viewed in the browser alongside
terminal panes.

Server (Rust)
─────────────
• New `crates/compositor` — headless Wayland compositor (#[cfg(unix)])
  implementing wl_shm, linux-dmabuf, xdg-shell, xdg-decoration,
  wp_viewporter, primary selection, fractional scaling, cursor shape,
  text input, and XDG activation protocols via smithay 0.7.
• New `crates/server/src/surface_encoder.rs` — AV1 (rav1e), H.264
  (openh264), and VA-API hardware-accelerated encoding behind a unified
  `SurfaceEncoder` interface.  BLIT_SURFACE_QUALITY (low/medium/high/
  lossless) controls rav1e speed/quantizer presets.  BLIT_SURFACE_ENCODER
  selects the codec (auto/av1/h264-software/h264-vaapi).
• Per-surface subscription protocol (C2S_SURFACE_SUBSCRIBE/UNSUBSCRIBE)
  so clients only receive frames for surfaces they are viewing.
• Wire-protocol extensions in `crates/remote` for surface lifecycle,
  frames, input, pointer, axis, resize, focus, clipboard, and capture.
• CLI surface automation: `surfaces`, `capture`, `click`, `key`, `type`
  subcommands in `crates/cli/src/agent.rs`.
• Compositor shutdown when all PTYs in a session exit or close.
• LISTEN_PID validation before adopting systemd socket-activation fds.

Client (TypeScript)
───────────────────
• `SurfaceStore` — tracks surface metadata and decodes H.264/AV1 frames
  via WebCodecs (`avc1.42001f` / `av01`, optimizeForLatency).
• `BlitSurfaceCanvas` — OffscreenCanvas renderer with keyboard, mouse,
  pointer-axis, resize, focus, and clipboard forwarding.
• `BlitSurfaceView` components for React and Solid.
• Surface subscribe/unsubscribe messages in protocol.ts.
• WebTransport close-reason surfacing into connection error state.

UI (js/web-app → js/ui)
────────────────────────
• Rename `js/web-app` to `js/ui`.
• Surfaces promoted to first-class interactive BSP panes with resizable
  preview panel and off-screen session/surface thumbnails.
• Passphrase encryption in URL hash (tweetnacl) — plaintext hashes are
  automatically replaced on first load.
• Status bar: disconnect reason display, debounced connection state,
  full-bar background color instead of status circle.
• Switcher overlay: surface entries, move-to-pane support, layout
  presets and history.

Infrastructure
──────────────
• Release workflow refactored: `bin/release-prepare` opens a PR,
  `bin/release-tag` creates a signed tag; CI verifies tag signature
  before publishing.
• `bin/dev` supports `DEV_INSTANCE` for parallel dev stacks on
  non-colliding port blocks.
• Nix devShell and packages updated with pixman, libxkbcommon, smithay,
  rav1e, and optional VA-API deps (Linux-only gating for Darwin CI).
• install.sh defaults to user-local install.
• E2E test stability: wait for DOM to settle after hash encryption,
  wait for async hash write in auth reload test.
• Documentation updates across README, ARCHITECTURE, CONTRIBUTING,
  EMBEDDING, SKILL, UNSAFE, learn.md, and man pages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant