Add headless Wayland compositor with GUI surface streaming#154
Open
Add headless Wayland compositor with GUI surface streaming#154
Conversation
Contributor
|
Contributor
|
🔗 Preview: https://blit-cbk92m527-indent.vercel.app |
Contributor
Coverage
|
e8bc05f to
b18fff5
Compare
5d7bf8c to
db10561
Compare
Embed a smithay-based headless Wayland compositor into every PTY session (Linux and macOS) so GUI applications launched inside blit stream their surfaces to remote clients as real-time H.264/AV1 video. This turns blit from a terminal multiplexer into a full remote-desktop primitive: any Wayland app — terminals, browsers, editors, media players — can be launched inside a blit session and viewed in the browser alongside terminal panes. Server (Rust) ───────────── • New `crates/compositor` — headless Wayland compositor (#[cfg(unix)]) implementing wl_shm, linux-dmabuf, xdg-shell, xdg-decoration, wp_viewporter, primary selection, fractional scaling, cursor shape, text input, and XDG activation protocols via smithay 0.7. • New `crates/server/src/surface_encoder.rs` — AV1 (rav1e), H.264 (openh264), and VA-API hardware-accelerated encoding behind a unified `SurfaceEncoder` interface. BLIT_SURFACE_QUALITY (low/medium/high/ lossless) controls rav1e speed/quantizer presets. BLIT_SURFACE_ENCODER selects the codec (auto/av1/h264-software/h264-vaapi). • Per-surface subscription protocol (C2S_SURFACE_SUBSCRIBE/UNSUBSCRIBE) so clients only receive frames for surfaces they are viewing. • Wire-protocol extensions in `crates/remote` for surface lifecycle, frames, input, pointer, axis, resize, focus, clipboard, and capture. • CLI surface automation: `surfaces`, `capture`, `click`, `key`, `type` subcommands in `crates/cli/src/agent.rs`. • Compositor shutdown when all PTYs in a session exit or close. • LISTEN_PID validation before adopting systemd socket-activation fds. Client (TypeScript) ─────────────────── • `SurfaceStore` — tracks surface metadata and decodes H.264/AV1 frames via WebCodecs (`avc1.42001f` / `av01`, optimizeForLatency). • `BlitSurfaceCanvas` — OffscreenCanvas renderer with keyboard, mouse, pointer-axis, resize, focus, and clipboard forwarding. • `BlitSurfaceView` components for React and Solid. • Surface subscribe/unsubscribe messages in protocol.ts. • WebTransport close-reason surfacing into connection error state. UI (js/web-app → js/ui) ──────────────────────── • Rename `js/web-app` to `js/ui`. • Surfaces promoted to first-class interactive BSP panes with resizable preview panel and off-screen session/surface thumbnails. • Passphrase encryption in URL hash (tweetnacl) — plaintext hashes are automatically replaced on first load. • Status bar: disconnect reason display, debounced connection state, full-bar background color instead of status circle. • Switcher overlay: surface entries, move-to-pane support, layout presets and history. Infrastructure ────────────── • Release workflow refactored: `bin/release-prepare` opens a PR, `bin/release-tag` creates a signed tag; CI verifies tag signature before publishing. • `bin/dev` supports `DEV_INSTANCE` for parallel dev stacks on non-colliding port blocks. • Nix devShell and packages updated with pixman, libxkbcommon, smithay, rav1e, and optional VA-API deps (Linux-only gating for Darwin CI). • install.sh defaults to user-local install. • E2E test stability: wait for DOM to settle after hash encryption, wait for async hash write in auth reload test. • Documentation updates across README, ARCHITECTURE, CONTRIBUTING, EMBEDDING, SKILL, UNSAFE, learn.md, and man pages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Embed a smithay-based headless Wayland compositor into every PTY session
(Linux and macOS) so GUI applications launched inside blit stream their
surfaces to remote clients as real-time H.264/AV1 video. This turns blit
from a terminal multiplexer into a full remote-desktop primitive: any
Wayland app — terminals, browsers, editors, media players — can be
launched inside a blit session and viewed in the browser alongside
terminal panes.
Server (Rust)
crates/compositor— headless Wayland compositor (#[cfg(unix)]) implementing wl_shm, linux-dmabuf, xdg-shell, xdg-decoration, wp_viewporter, primary selection, fractional scaling, cursor shape, text input, and XDG activation protocols via smithay 0.7.crates/server/src/surface_encoder.rs— AV1 (rav1e), H.264 (openh264), and VA-API hardware-accelerated encoding behind a unifiedSurfaceEncoderinterface.BLIT_SURFACE_QUALITY(low/medium/high/lossless) controls rav1e speed/quantizer presets.BLIT_SURFACE_ENCODERselects the codec (auto/av1/h264-software/h264-vaapi).C2S_SURFACE_SUBSCRIBE/UNSUBSCRIBE) so clients only receive frames for surfaces they are viewing.crates/remotefor surface lifecycle, frames, input, pointer, axis, resize, focus, clipboard, and capture.surfaces,capture,click,key,typesubcommands.LISTEN_PIDvalidation before adopting systemd socket-activation fds.Client (TypeScript)
SurfaceStore— tracks surface metadata and decodes H.264/AV1 frames via WebCodecs (avc1.42001f/av01,optimizeForLatency).BlitSurfaceCanvas— OffscreenCanvas renderer with keyboard, mouse, pointer-axis, resize, focus, and clipboard forwarding.BlitSurfaceViewcomponents for React and Solid.protocol.ts.UI (
js/web-app→js/ui)js/web-apptojs/ui.Infrastructure
bin/release-prepareopens a PR,bin/release-tagcreates a signed tag; CI verifies tag signature before publishing.bin/devsupportsDEV_INSTANCEfor parallel dev stacks on non-colliding port blocks.install.shdefaults to user-local install.