Add internal binary data-plane transport for low-latency service traffic

[JedimEmO]

Summary

Add an internal binary data-plane transport for low-latency, high-throughput RAS service traffic.

RAS already supports typed REST, JSON-RPC, file services, and bidirectional JSON-RPC. Those are good API, browser, control-plane, and general integration surfaces. Some RAS deployments may also need very low latency, sparse high-bandwidth transfer, long-lived channels, and reduced per-message auth overhead for internal service-to-service, worker, simulation, inference, or data-plane traffic.

This issue should define a first-class RAS binary transport model without committing v1 to a specific HPC substrate such as QUIC, TCP/TLS, RDMA/libfabric, Aeron, or shared memory. The initial goal is a Rust-contract-first, internal-only, adapter-oriented core that can later support concrete transports.

Related:

• Add outbound token framework with pluggable token sources #12 defines the outbound token framework used by internal service callers.
• Add RAS-native authorization control plane and internal service token issuer #13 defines RAS-native authorization, token/JWKS primitives, service identity, and internal service tokens.
• Add optional RAS auth gateway for multi-service browser frontends #14 defines the optional browser-facing auth gateway; binary data-plane traffic should not depend on it.
• Add RAS topology macro for compile-checked service graphs and generated artifacts #15 defines topology artifacts that can declare allowed service edges and generated authorization policy.

Problem

Today, RAS service contracts are optimized for interoperable application APIs:

• REST and file services are HTTP-friendly and browser/client-generator friendly.
• JSON-RPC is convenient for typed request/response APIs.
• Bidirectional JSON-RPC supports interactive WebSocket flows.

These transports are not ideal for every internal workload:

• repeated bearer-token parsing/signature validation can be too expensive per message
• JSON serialization can add latency and allocation overhead
• sparse high-bandwidth payloads may need zero-copy or near-zero-copy access
• stream setup, backpressure, and cancellation need to be explicit
• long-lived service-to-service channels should authenticate once and authorize operations cheaply
• teams may want to run over different substrates depending on environment: in-memory, QUIC, TCP/TLS, RDMA, Aeron, shared memory, or process-local adapters

RAS should be able to describe this traffic in the same typed contract ecosystem without turning into a general service mesh or coupling the framework to one HPC protocol.

Proposed Direction

Add a new binary transport crate family, likely under crates/binary/:

• ras-binary-core
  • channel/session model
  • operation metadata and operation IDs
  • codec-neutral zero-copy traits
  • unary call and bidirectional stream abstractions
  • auth/session binding types
  • flow-control, cancellation, and backpressure primitives
  • adapter traits for concrete transports
  • binary transport errors
• ras-binary-macro
  • binary_service! macro for declaring binary service contracts
  • generated service traits and server dispatch glue
  • generated typed clients
  • generated operation metadata
  • generated permission manifest entries
• ras-binary-test
  • in-memory adapter for tests and examples
  • fake codec helpers
  • auth/session test utilities
  • stream/backpressure test harnesses

The v1 implementation should be adapter-only with an in-memory adapter. Production adapters for QUIC, TCP/TLS, RDMA/libfabric, Aeron, shared memory, or process-local execution can be added later without changing service contracts.

Contract Model

The Rust macro contract should remain the source of truth, consistent with existing RAS design.

Example direction:

binary_service!({
    service_name: TensorWorker,
    audience: "tensor-worker",

    methods: [
        WITH_PERMISSIONS(["tensor:infer"])
            infer(InferRequest) -> InferResponse,
    ],

    streams: [
        WITH_PERMISSIONS(["tensor:stream"])
            stream_batches(BatchStreamConfig)
                client_stream: BatchFrame
                server_stream: BatchResult,
    ],
});

The exact syntax can evolve, but the design should support:

• unary request/response operations
• bidirectional streams
• generated server traits
• generated typed clients
• generated operation IDs and metadata
• generated permission manifests with a new binary transport kind
• API-crate feature gating similar to existing macro crates

This should not require schema files to become the source of truth. External schema systems may be used by codecs, but RAS should keep the Rust service definition as the contract boundary.

Payload And Codec Model

Binary transport should be codec-neutral and zero-copy capable.

ras-binary-core should define traits around borrowed buffers and encoded payload views rather than requiring serde for every message. Future codecs may include:

• rkyv for Rust-native archived values
• FlatBuffers or Cap'n Proto for cross-language zero-copy schemas
• custom frame codecs for domain-specific payloads
• simple bincode/postcard-style codecs for tests or less extreme workloads

The core should avoid forcing one codec in v1. It should provide enough trait shape to let generated code express:

• request and response encoding/decoding
• borrowed payload lifetimes where supported
• owned fallback values where needed
• large frame/chunk payloads
• sparse transfer metadata

Serde support may be useful as an adapter or test codec, but the binary data-plane should not collapse into "JSON-RPC with a binary serializer."

Auth And Crypto Model

Binary traffic should handle auth differently from HTTP request/response traffic.

Expected model:

1. A client opens a channel/session to a target RAS audience.
2. The adapter performs or relies on transport-level integrity/confidentiality where available.
3. The client authenticates once using RAS service identity or a RAS-issued internal token from Add RAS-native authorization control plane and internal service token issuer #13.
4. The channel is bound to subject/principal, audience, tenant/context where applicable, authz/session version, token type, expiry, and key ID/signing metadata.
5. Each unary call or stream open checks the generated operation permission requirement against the channel-bound identity.
6. Data frames do not carry bearer tokens by default.

The adapter trait should explicitly declare its security properties:

• authenticated transport or separate RAS token proof
• encrypted or plaintext
• integrity-protected or requiring frame-level integrity
• replay protection assumptions
• peer identity binding support

Adapters without sufficient built-in integrity/confidentiality may need frame MAC/encryption support, but this should be expressed as an adapter capability rather than hardcoded into the service macro.

Auth failures should occur before payload transfer whenever possible. Stream-open permission checks should happen before accepting client frames.

Topology And Authorization Relationship

Binary data-plane services should integrate with the RAS auth/topology layer:

• Add RAS-native authorization control plane and internal service token issuer #13 should issue internal tokens suitable for binary service audiences.
• Add RAS topology macro for compile-checked service graphs and generated artifacts #15 topology should be able to declare binary service nodes and allowed binary service-to-service edges.
• Topology-generated authz policy should constrain token issuance for binary calls the same way it constrains other internal service calls.
• Generated permission manifests should include binary operations and stream opens.
• Binary transport should be internal-only by default.

Binary services should not be exposed through public gateway profiles in v1. If a future deployment deliberately exposes binary traffic publicly, that should be a separate design with stronger abuse controls, handshake hardening, compatibility commitments, and operational guidance.

Security Requirements

• Binary transport is internal-only by default.
• Channel/session authentication must fail closed.
• Per-frame bearer tokens are not required or expected.
• Call and stream-open authorization must use generated RAS permission requirements.
• The authenticated channel binding must include target audience and token type.
• Token expiry/authz-version behavior must be documented for long-lived channels.
• Stream permissions must be checked before payload frames are processed.
• Adapters must declare integrity/confidentiality/replay assumptions.
• Payload/frame logs must not include raw token values or sensitive binary contents by default.
• Flow-control failures, cancellation, and partial stream errors must not bypass authorization checks.
• Public exposure must be out of scope for v1.

Acceptance Criteria

• A binary core crate defines channel/session, operation metadata, adapter, codec, call, stream, and error traits/types.
• A binary macro can declare unary operations and bidirectional streams from Rust service definitions.
• Generated binary service code emits service traits, typed clients, operation IDs, and permission manifest entries.
• ras-permission-manifest supports a binary transport kind and binary operation metadata.
• The v1 transport is adapter-oriented and includes an in-memory test adapter.
• Channel-level authentication binds subject/principal, audience, token type, expiry, tenant/context where applicable, and authz/session version where applicable.
• Unary calls and stream opens enforce generated permission requirements from the authenticated channel identity.
• Data frames do not require bearer tokens by default.
• Codec traits support zero-copy-capable borrowed payloads without forcing serde.
• The core does not require QUIC, TCP/TLS, RDMA/libfabric, Aeron, or shared memory in v1.
• Topology/authz integration can represent binary service edges and constrain token issuance.
• Documentation positions binary transport as an advanced internal data-plane option, not the default RAS API style.

Test Plan

• Macro tests for generated service traits, typed clients, operation IDs, and permission manifests.
• Compile tests for unary operation and bidirectional stream declarations.
• In-memory adapter tests for successful channel authentication and call dispatch.
• Auth failure tests for missing credentials, wrong audience, wrong token type, expired token, and authz-version mismatch.
• Permission tests proving unary calls and stream opens fail before payload processing when permissions are missing.
• Codec tests proving borrowed/zero-copy payload paths can be represented by the core traits.
• Stream tests for backpressure, cancellation, partial failure, and large sparse payload transfer through the in-memory adapter.
• Tests proving data frames do not carry or require bearer tokens after channel authentication.
• Tests showing generated binary permission manifests can be consumed by Add RAS-native authorization control plane and internal service token issuer #13/Add RAS topology macro for compile-checked service graphs and generated artifacts #15-style policy validation.

Non-Goals For v1

• Do not replace REST, JSON-RPC, file services, or browser-facing flows.
• Do not implement a production QUIC/TCP/RDMA/Aeron/shared-memory adapter in the first slice.
• Do not make an external schema file the RAS contract source of truth.
• Do not require every RAS app to understand binary transport.
• Do not expose binary transport publicly by default.
• Do not build a service mesh or scheduler.
• Do not require per-frame JWT validation.
• Do not force a single zero-copy codec across all users.

Notes

The value is not "RAS uses HPC protocol X." The value is that RAS can describe high-performance internal data-plane traffic with the same contract, auth, permission, topology, and testing discipline as the rest of the framework, while leaving concrete transport substrates pluggable.