Skip to content

GrokImageCompression/glass2glass

BranchesTags

Repository files navigation

glass2glass (g2g)

A hardware-first, sans-IO, asynchronous multimedia graph framework in pure Rust — designed to replace GStreamer in AI-driven, real-time embedded (RTOS), cloud ingestion, and browser targets.

The name reflects the metric the project optimizes for: glass-to-glass latency, the time between physical photon capture and hardware presentation.

See DESIGN.md for the architecture specification.

The four pillars

  1. Async execution. Every element is a cooperative Future. The framework is runtime-agnostic (Tokio on servers, Embassy on RTOS, wasm-bindgen-futures in the browser).
  2. Hardware-first, zero-copy. Buffers live in DMABUF / Vulkan / CUDA / D3D11 / WebGPU memory domains; CPU memory copies are treated as system faults.
  3. no_std + alloc + sans-IO core. The same pipeline shape runs on a Cortex-M, a multi-threaded server, or wasm32.
  4. First-class ML. Tensor allocation, reshaping, and pipeline batching are part of graph orchestration.

Workspace

Crate Role Profile
g2g-core Traits, Frame/PipelinePacket, caps algebra, clock, runner. no_std + alloc
g2g-plugins Sources/sinks/transforms (RTSP, RTP in/out, HTTP/HLS/DASH/RTMP ingest, V4L2 / PipeWire / MF capture, ffmpeg, VAAPI, MF, Wayland, KMS, WASAPI, ALSA / PulseAudio / PipeWire audio, compositor, Embassy, web), container mux/demux (MP4, MPEG-TS, Matroska/WebM, FLV, Ogg), codec parsers + encoders (AV1, VP8/9, MJPEG), the tag system, and the gst-launch text DSL. mixed
g2g-ml ORT, Burn, WgpuPreprocess, TensorPostprocess. std
g2g-bridge GStreamer C-FFI bridge. std
g2g-enterprise Multi-stream tensor batcher. std

Build

Stable Rust, MSRV 1.75, resolver = "2".

cargo check --workspace          # no_std baseline
cargo test  --workspace          # default test suite (no platform features)
cargo clippy --workspace --all-targets

OS-coupled elements live behind cargo features:

Element Feature Platform / system dep
RtspSrc rtsp retina
H264Parse (default) —
FfmpegH264Dec (sw / NvdecCuvid / NvdecCuda) ffmpeg Linux + libavcodec
VaapiH264Dec vaapi Linux + libva + GBM
MfDecode / MfEncode / MfAacEncode / MfAacDecode mf-decode, mf-encode, mf-aac Windows + Media Foundation
WaylandSink wayland-sink Linux + Wayland
KmsSink kms-sink Linux + libdrm; needs DRM master / tty
D3D11Sink d3d11-sink Windows
CudaDownload, CudaGlSink cuda, cuda-gl Linux + NVIDIA driver (libcuda) + EGL + GL
UdpSink + RTP packetizer udp-egress —
UdpSrc (RTP ingest + jitter buffer + RTCP/NACK) udp-ingress —
RtmpSrc (RTMP publisher ingest) rtmp —
HttpSrc (HTTP(S) byte-stream source) http-src reqwest
HlsSrc (HLS: TS + fMP4/CMAF, live, AES-128 / SAMPLE-AES) hls reqwest + aes
DashSrc (DASH: SegmentTemplate / SegmentTimeline, live) dash reqwest + roxmltree
V4l2Src v4l2 Linux + V4L2 (/dev/videoN)
WasapiSink / WasapiSrc wasapi-sink, wasapi-src Windows
AlsaSink alsa-sink Linux + libasound
PulseSink pulse-sink Linux + libpulse
PipeWireSink / PipeWireSrc (audio) pipewire Linux + libpipewire
MfVideoSrc (camera) mf-video-src Windows + Media Foundation
Av1Enc (pure-Rust rav1e) av1-encode —
VpxEnc (VP8 / VP9 via libvpx) vpx libvpx
MjpegDec / MjpegEnc (pure Rust) mjpeg, mjpeg-encode —
AnalyticsOverlay (CPU) / VelloAnalyticsOverlay (GPU) / WgpuSink analytics, vello-overlay, wgpu-sink wgpu (GPU variants)
OrtInference (+ CUDA / DirectML EPs) ort, cuda, directml (in g2g-ml) onnxruntime
BurnInference burn (in g2g-ml) wgpu (Vulkan / Metal / DX12)
WgpuPreprocess wgpu (in g2g-ml) wgpu
Embassy / RTOS pool + clock embassy, embassy-link —
Browser elements web, web-codecs wasm32-unknown-unknown

The container parsers and muxers (mp4src / mp4sink, tsdemux / mpegtsmux, matroskademux / matroskamux, flvdemux / flvmux, oggdemux, fmp4demux), the bitstream parsers (h264parse, h265parse, aacparse, opusparse, vp8parse, vp9parse, av1parse), the software video/audio transforms (videoscale / videorate / videocrop / videoflip / videobalance / videobox / alpha, audioconvert / audioresample / audiomixer / volume / audiopanorama), the compositor, the tag system, and the gst-launch text DSL (parse_launch / gst-inspect) are all in the pure no_std + alloc default build (no feature flag).

Sample pipelines

The graph API is run_source_transform_sink / run_linear_chain / run_source_fanout / run_muxer_sink over typed elements (no string-keyed factory lookup). Examples are condensed; full versions live in the integration tests under g2g-plugins/tests/.

RTSP → ffmpeg decode → Wayland window

let src  = RtspSrc::new("rtsp://localhost:8554/pattern");
let dec  = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let sink = WaylandSink::new();

run_source_transform_sink(src, dec, sink, &clock, LatencyProfile::Live).await?;

Features: rtsp ffmpeg wayland-sink.

RTSP → NVDEC (CUDA device memory) → CUDA-GL display

Zero-copy after decode: NV12 stays in CUDA device memory until the GL fragment shader samples it.

let src  = RtspSrc::new(url);
let dec  = FfmpegH264Dec::with_backend(Backend::NvdecCuda);   // MemoryDomain::Cuda
let sink = CudaGlSink::new();                                  // EGL on Wayland, NV12 shader

run_source_transform_sink(src, dec, sink, &clock, LatencyProfile::Live).await?;

Features: rtsp ffmpeg cuda cuda-gl. Linux + NVIDIA only. See DESIGN.md §4.11.5.

RTSP → decode → KMS (tty / no compositor)

let src  = RtspSrc::new(url);
let dec  = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let sink = KmsSink::open("/dev/dri/card0")?;

run_source_transform_sink(src, dec, sink, &clock, LatencyProfile::Live).await?;

Features: rtsp ffmpeg kms-sink. Run from a tty after stopping the display manager (KMS sink needs DRM master).

RTSP → decode → ML preprocess → ORT inference → postprocess

let src       = RtspSrc::new(url);
let dec       = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let preproc   = WgpuPreprocess::new(w, h);                 // NV12 -> f32 NCHW on GPU
let inference = OrtInference::from_memory_with_cuda(model_bytes)?;
let post      = TensorPostprocess::topk_classification(5);

run_linear_chain(src, vec![&mut dec, &mut preproc, &mut inference, &mut post],
                 FakeSink::new(), &clock, LatencyProfile::Live).await?;

Features: rtsp ffmpeg (plugins) + wgpu cuda (g2g-ml). The CUDA execution provider falls back to CPU if no CUDA runtime is present.

File capture → H.264 parse → fMP4 record

let src   = FileSrc::open("in.h264")?;
let parse = H264Parse::new();
let sink  = Mp4Sink::open("out.mp4")?;

run_source_transform_sink(src, parse, sink, &clock, LatencyProfile::Live).await?;

MPEG-TS file → demux → H.264 parse → decode → Wayland

The container demuxers (tsdemux, matroskademux, flvdemux, oggdemux, fmp4demux) accept a Caps::ByteStream and split out elementary streams.

let src   = FileSrc::new("clip.ts", Caps::ByteStream { encoding: ByteStreamEncoding::MpegTs });
let demux = TsDemux::new().with_stream(TsStream::H264);   // PAT/PMT/PES -> Annex-B
let parse = H264Parse::new();
let dec   = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let sink  = WaylandSink::new();

run_linear_chain(src, vec![&mut demux, &mut parse, &mut dec], sink,
                 &clock, LatencyProfile::Live).await?;

Features: ffmpeg wayland-sink.

Adaptive streaming: HLS / DASH → decode → display

let src   = HlsSrc::new("https://example.com/master.m3u8");  // or DashSrc::new(mpd_url)
let demux = TsDemux::new().with_stream(TsStream::H264);
let parse = H264Parse::new();
let dec   = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let sink  = WaylandSink::new();

run_linear_chain(src, vec![&mut demux, &mut parse, &mut dec], sink,
                 &clock, LatencyProfile::Live).await?;

Features: hls ffmpeg wayland-sink (dash for the DASH front end). HlsSrc follows live playlist reloads and decrypts AES-128 / SAMPLE-AES segments; DashSrc handles SegmentTemplate / SegmentTimeline and dynamic (live) MPDs.

gst-launch text pipeline

parse_launch builds a runnable Graph from a GStreamer-style string against the default_registry, including caps filters, tee branching, and muxer fan-in. Registry::inspect(name) is the gst-inspect analog.

let graph = parse_launch(
    "videotestsrc num-buffers=90 pattern=ball ! video/x-raw,format=nv12 \
     ! videoflip method=rotate-180 ! matroskamux ! filesink location=out.mkv",
    &default_registry(),
)?;
run_graph(graph, &clock, LatencyProfile::Live).await?;

Registered launch elements include videotestsrc / audiotestsrc, the SW transforms, the demuxers (tsdemux, matroskademux, flvdemux, oggdemux) and muxers (mpegtsmux, matroskamux, flvmux, funnel, audiomixer), filesrc / filesink, and fakesink. Feature-gated capture / decode / display elements still need explicit Rust construction.

Camera → encode → RTP egress over UDP

let src  = VideoTestSrc::new(1920, 1080, 30, 0);         // RGBA test pattern, unbounded
let enc  = MfEncode::new_low_latency();                  // Windows; on Linux use the bridge
let sink = UdpSink::new("239.0.0.1:5004".parse()?)
    .with_rtp(96, 0x1234_5678);                          // payload type, SSRC

run_source_transform_sink(src, enc, sink, &clock, LatencyProfile::Live).await?;

Features: udp-egress (plus the platform encoder feature). UdpSink honors receive-side NACK by retransmitting from a bounded send history (with_retransmit).

RTP ingress over UDP → ffmpeg decode → Wayland

The receive-side inverse, with a jitter buffer (reorder / bounded-latency loss handling) and RTCP feedback (periodic receiver reports, NACK on gaps) built in.

let src  = UdpSrc::new("0.0.0.0:5004".parse()?)
    .with_jitter(50, 64)                                 // 50 ms hold, 64-packet depth
    .with_rtcp(1000, true);                              // 1 s reports, NACK enabled
let dec  = FfmpegH264Dec::new().with_output_format(OutputFormat::Nv12);
let sink = WaylandSink::new();

run_source_transform_sink(src, dec, sink, &clock, LatencyProfile::Live).await?;

Features: udp-ingress ffmpeg wayland-sink.

Picture-in-picture: webcam over a test pattern (compositor)

let bg   = VideoTestSrc::new(1280, 720, 30, 0).with_pattern(Pattern::MovingBar);
let cam  = V4l2Src::new("/dev/video0").with_size(640, 480);   // -> VideoConvert(RGBA) -> VideoScale
let comp = Compositor::new(1280, 720, vec![
    CompositorPad::at(0, 0),                              // background, timing driver
    CompositorPad::at(940, 460).with_zorder(1),          // webcam inset
]);
// bg -> comp.input(0); cam -> rgba -> scale -> comp.input(1); comp -> sink (see tests).

Features: v4l2 wayland-sink. Full graph in g2g-plugins/tests/pip_smoke.rs.

Running smoke tests

Most integration tests are marked #[ignore] because they need a live RTSP feed and/or a display. The pattern is the same across recipes:

cargo test -p g2g-plugins \
  --features "<comma-separated feature list>" \
  --test <test_name> -- --ignored --nocapture

A standing RTSP feed

A typical loopback setup uses mediamtx as the relay and ffmpeg as the publisher. In one terminal:

mediamtx                  # listens on 8554/tcp by default

In a second terminal, push a synthetic H.264 feed into it:

ffmpeg -re -f lavfi -i testsrc=size=1280x720:rate=30 \
       -c:v libx264 -preset ultrafast -tune zerolatency -g 30 \
       -f rtsp -rtsp_transport tcp rtsp://localhost:8554/pattern

A public RTSP feed (Wowza demo, IP camera on the LAN, etc.) also works — the smoke tests don't care where the stream comes from.

Software decode + Wayland

G2G_RTSP_TEST_URL=rtsp://localhost:8554/pattern \
  cargo test -p g2g-plugins \
  --features "rtsp ffmpeg wayland-sink" \
  --test wayland_smoke -- --ignored --nocapture

A window titled "glass2glass" appears showing the feed.

NVIDIA NVDEC (system memory) + Wayland

G2G_DECODER=nvdec \
G2G_RTSP_TEST_URL=rtsp://localhost:8554/pattern \
G2G_TARGET_FRAMES=300 \
  cargo test -p g2g-plugins \
  --features "rtsp ffmpeg wayland-sink" \
  --test wayland_smoke -- --ignored --nocapture

G2G_TARGET_FRAMES >= 300 is needed to amortize cuvid startup (libnvcuvid load, CUDA context, surface pool alloc) for meaningful p50 / p95 latency numbers. Compare against G2G_DECODER=software on the same feed.

NVIDIA NVDEC → CUDA → CUDA-GL zero-copy display

G2G_RTSP_TEST_URL=rtsp://localhost:8554/pattern \
  cargo test -p g2g-plugins \
  --features "rtsp ffmpeg cuda cuda-gl" \
  --test cuda_gl_smoke -- --ignored --nocapture

KMS scanout (no compositor)

Drop to a tty, stop the display manager, then:

G2G_RTSP_TEST_URL=rtsp://localhost:8554/pattern \
  cargo test -p g2g-plugins \
  --features "rtsp ffmpeg kms-sink" \
  --test kms_smoke -- --ignored --nocapture

ML inference

# ORT with the CUDA execution provider (silently falls back to CPU):
cargo test -p g2g-ml --features cuda --test ort_inference -- --nocapture

# Pure-Rust Burn over wgpu (any Vulkan/Metal/DX12 adapter):
cargo test -p g2g-ml --features burn --test burn_inference -- --nocapture

UDP egress (loopback, no network)

cargo test -p g2g-plugins --features udp-egress --test m47_udp_egress -- --nocapture

Binds a UDP receiver on localhost, drives the H.264 RTP packetizer, and asserts the datagrams parse back byte-exactly, with sequence numbers, marker bit, and FU-A reassembly all correct.

UDP ingress + resilience (loopback, no network)

cargo test -p g2g-plugins --features "udp-ingress udp-egress" --test udp_loopback -- --nocapture

End-to-end over localhost: depayload round-trip, the jitter buffer reordering out-of-order packets, and NACK-driven recovery (a lossy relay drops chosen sequences; the receiver NACKs, the sender retransmits, every access unit is recovered in order).

System dependencies

The cargo features pull pure Rust crates; OS-level dependencies must be present on the host.

Distro Decoder (ffmpeg) Wayland sink KMS sink VAAPI
Fedora ffmpeg-devel (RPM Fusion) or ffmpeg-free-devel wayland-devel libdrm-devel libva-devel
Debian / Ubuntu libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libwayland-dev libdrm-dev libva-dev
Arch ffmpeg wayland libdrm libva

For the CUDA path: install the NVIDIA driver and CUDA runtime your distribution ships, and ensure libnvcuvid.so and libcuda.so are on the linker path. The ffmpeg build must include cuvid support if you intend to use Backend::NvdecCuvid / Backend::NvdecCuda.

mediamtx for the loopback RTSP server is available as a single binary from https://github.com/bluenviron/mediamtx/releases; some distros also package it.

Layout

g2g-core/        # traits, runner, solver, frame, caps, clock
g2g-plugins/     # all source/sink/transform elements
g2g-ml/          # ORT, Burn, WgpuPreprocess, batcher
g2g-bridge/      # GStreamer C-FFI bridge (libgstglass2glass.so)
g2g-enterprise/  # multi-stream tensor batcher
DESIGN.md        # architecture specification
docs/            # GitHub Pages site

License

g2g-core, g2g-plugins, g2g-ml, g2g-bridge: LGPL v2.1+. g2g-enterprise: AGPL v3.

See LICENSE.

About

No description, website, or topics provided.

Resources

Readme

License

LGPL-2.1 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages