Compositions documentation

.gitmodules

@@ -16,3 +16,6 @@
 [submodule "packages/python-server-sdk"]
 	path = packages/python-server-sdk
 	url = https://github.com/fishjam-cloud/python-server-sdk.git
+[submodule "api/composition"]
+	path = api/composition
+	url = git@github.com:fishjam-cloud/foundry.git

docs/api/reference.md

@@ -6,7 +6,7 @@ type: reference
 
 Describes APIs for direct interaction with Fishjam.
 
-Fishjam publishes documentation for the Sandbox API and Fishjam Server APIs.
+Fishjam publishes documentation for the Sandbox API, Fishjam Server APIs, and the Composition API.
 
 ## Sandbox API
 
@@ -51,3 +51,59 @@ the first message that must be sent is an `AuthRequest`,
 with a valid Management Token.
 
 Next, you can should subscribe to notifications by sending `SubscribeRequest` event with `SERVER_NOTIFICATION` event type.
+
+## Compositions
+
+[Compositions](../explanation/compositions) are managed through the Composition API: a REST API plus a WebSocket stream for engine events. All requests go to `https://rtc.fishjam.io`.
+
+### REST API
+
+[Composition REST API Reference](/api/compositions)
+
+The [OpenAPI document](https://github.com/fishjam-cloud/documentation/blob/main/static/api/composition-openapi.json) is generated from the service's source code and republished together with documentation updates.
+
+Alongside the input, output, and renderer endpoints, the API exposes three lifecycle calls: `POST …/start` starts a composition created with `autostart` off, `POST …/reset` tears down every registered input, output, and renderer and returns the composition to an empty, unstarted state, and `DELETE /api/composition/{composition_id}` destroys it.
+
+### WebSocket event stream
+
+Some engine events (for example, an output finishing) are delivered over a WebSocket rather than HTTP. Connect to:
+
+```
+GET wss://rtc.fishjam.io/api/composition/{composition_id}/ws
+```
+
+Because browsers cannot set an `Authorization` header on a WebSocket, authentication rides on the `Sec-WebSocket-Protocol` header, which must carry **two** subprotocols (order does not matter):
+
+- `json.fishjam.io`: selects the JSON wire format.
+- `bearer.auth.fishjam.io.<token>`: your token, appended to the literal prefix.
+
+```js
+const ws = new WebSocket(
+  `wss://rtc.fishjam.io/api/composition/${compositionId}/ws`,
+  ["json.fishjam.io", `bearer.auth.fishjam.io.${token}`],
+);
+```
+
+Messages are JSON text frames, each with a `type` field identifying the event.
+
+### Composition authentication
+
+| What you're calling                                                         | How it authenticates                                                                                                                                    |
+| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Control-plane calls (composition, inputs, outputs, renderers, events, room) | `Authorization: Bearer <token>`: your Fishjam **Management Token**, the same token used across Fishjam.                                                 |
+| Publishing to an input (`/whip/{input_id}`)                                 | The **input's** own bearer token, returned when you register a `whip_server` input (or the one you supplied). This is distinct from your account token. |
+| The WebSocket event stream                                                  | The subprotocol scheme described above.                                                                                                                 |
+
+Get your Management Token from the [**Fishjam developer panel**](https://fishjam.io/app).
+
+Compositions are not played back from the Composition API itself: outputs push to the destination you register, so viewers connect to that destination instead. A `whip_client` output pointed at a [livestream](../explanation/livestreams), for example, is watched through the livestream's own WHEP endpoint.
+
+### Errors
+
+Every non-2xx response of the Composition API is a JSON object:
+
+```json
+{ "message": "Composition not found", "http_status_code": 404 }
+```
+
+Common statuses are `400` (bad request), `401` (unauthorized), `404` (not found), `500` (server error), and `503` (no capacity, returned by composition creation, templated output registration, events, and the WebSocket stream).

docs/explanation/compositions.mdx

@@ -0,0 +1,110 @@
+---
+type: explanation
+title: Compositions
+sidebar_position: 4.5
+description: Understand compositions, the Fishjam feature that mixes multiple live media streams into a single composed output in real time.
+---
+
+# Compositions
+
+_Understanding real-time stream composition in Fishjam_
+
+A **composition** mixes multiple live media streams into a single output stream in real time. You send inputs (WebRTC, RTMP, HLS, or MP4), describe how they should be laid out, and it produces one composed output that you can publish anywhere, all without running any rendering infrastructure yourself.
+
+Compositions are built on [Smelter](https://smelter.dev), the open-source real-time video compositing engine by [Software Mansion](https://swmansion.com), and are a native part of Fishjam: they authenticate with the same Fishjam credentials and can compose the peers of a [Fishjam room](./rooms) directly.
+
+## What you can build
+
+- **Multi-party layouts**: arrange the cameras of a conference or livestream into grids, side-by-sides, or picture-in-picture.
+- **Branded streams**: overlay logos, captions, lower-thirds, and backgrounds on top of live video.
+- **Cross-protocol bridging**: take WebRTC inputs and republish the composed result to an RTMP destination, or vice versa.
+
+## Core concepts
+
+A **composition** is a single running compositing session. Into it you register:
+
+- **Inputs**: the live media sources being composed (WebRTC via WHIP/WHEP, RTMP, HLS, or MP4).
+- **Outputs**: where the composed result is sent (WebRTC via WHIP, or RTMP). Each output carries a **scene** that describes the layout.
+- **Renderers**: shared assets such as images and fonts you can place in a scene.
+
+An output's scene can either be described directly in the API or rendered by a **template**: a React component, written with the layout components and the [`@fishjam-cloud/composition`](./../how-to/compositions/write-and-deploy-a-template) hooks, that updates the layout live as the room changes.
+
+## Cost and lifecycle
+
+A running composition holds a GPU-backed rendering session for as long as it exists, and you are billed for that time whether or not anyone is watching. It is the most expensive thing in this part of Fishjam to leave running by accident.
+
+Two defaults keep that in check. A composition auto-starts, and it cleans itself up after five minutes in which none of its inputs carry any media. Between them, an experiment you walk away from stops costing you money on its own.
+
+`cleanup_without_inputs: false` turns that guard off. It widens the condition so cleanup needs both the inputs and the outputs to go quiet, which is what you want when inputs arrive late, such as a room whose peers have not joined yet, or when a composition legitimately has no inputs, such as an output that renders only text. The cost is that an idle composition then survives indefinitely, so anything created that way is yours to delete, and a forgotten one bills until you do.
+
+Delete a composition as soon as you are done with it:
+
+```bash
+curl -X DELETE "$COMPOSITION_URL/api/composition/$COMPOSITION" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+## Scenes
+
+Every video output carries a **scene**: a tree of components that describes how inputs, text, and images are arranged into the composed frame. Audio outputs carry an **audio scene** that describes which inputs are mixed together.
+
+### The video scene tree
+
+A video scene has a single `root` component. Each component has a `type` that determines how it lays out its children.
+
+```json
+{
+  "root": {
+    "type": "tiles",
+    "children": [
+      { "type": "input_stream", "input_id": "camera_1" },
+      { "type": "input_stream", "input_id": "camera_2" }
+    ]
+  }
+}
+```
+
+The available component types are:
+
+| `type`         | Purpose                                                                          |
+| -------------- | -------------------------------------------------------------------------------- |
+| `input_stream` | Renders one registered input. Identified by `input_id`.                          |
+| `view`         | A container you position and style; the basic building block for custom layouts. |
+| `tiles`        | Automatically arranges its children into a grid.                                 |
+| `rescaler`     | Fits a single child into a target area, preserving aspect ratio.                 |
+| `text`         | Renders a text string.                                                           |
+| `image`        | Renders a registered image. Identified by `image_id`.                            |
+
+Components nest freely: a `tiles` of `rescaler`s wrapping `input_stream`s, a `view` with a `text` caption over an `input_stream`, and so on. The styling and full property set of each component come from Smelter itself; the [Smelter HTTP API reference](https://smelter.dev/http-api/overview) documents every component and its properties.
+
+### The audio scene
+
+An audio scene lists the inputs to mix and, optionally, their relative volume:
+
+```json
+{
+  "inputs": [
+    { "input_id": "camera_1" },
+    { "input_id": "camera_2", "volume": 0.5 }
+  ]
+}
+```
+
+`volume` defaults to `1.0`. Only the inputs you list are audible in the output.
+
+### Setting a scene
+
+You provide the initial scene when you register an output, under `video.initial` (a video scene) and `audio.initial` (an audio scene). See [Choose inputs and outputs](./../how-to/compositions/inputs-and-outputs) for the full output shape.
+
+### Changing a scene over time
+
+A scene is not fixed for the life of an output. You can replace it while the composition is running, immediately or at a chosen point on the composition timeline.
+
+Either you push those updates yourself, or you hand the job to a **template**: a React component that receives the live room state and re-renders as the room changes. See [Choose inputs and outputs](./../how-to/compositions/inputs-and-outputs) to send an update, or [Write and deploy a template](./../how-to/compositions/write-and-deploy-a-template) to build one.
+
+## Where to go next
+
+- [Compositions tutorial](./../tutorials/compositions): create your first composition end to end.
+- [Write and deploy a template](./../how-to/compositions/write-and-deploy-a-template): build a React layout with the composition SDK.
+- [Compose a Fishjam room](./../how-to/compositions/compose-a-fishjam-room): turn a room's peers into one composed stream.
+- [Composition API](./../api/reference#compositions): the full REST surface.

docs/how-to/compositions/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Compositions",
+  "position": 5,
+  "link": {
+    "type": "generated-index",
+    "description": "Compose rooms and live streams into new outputs with templates, events, and multiple transports."
+  }
+}

docs/how-to/compositions/compose-a-fishjam-room.mdx

@@ -0,0 +1,181 @@
+---
+type: how-to
+sidebar_position: 2
+description: Forward a Fishjam room into a composition and render its peers with a live React template.
+---
+
+# Compose a Fishjam room
+
+A composition can include the peers of a [Fishjam room](./../../explanation/rooms). You forward the room's tracks to the composition with a single Fishjam API call: Fishjam then pushes each participant's media into the composition as inputs, and hooks from `@fishjam-cloud/composition` let a [template](./write-and-deploy-a-template) render one tile per participant and push the result to a [livestream](./../../explanation/livestreams).
+
+```
+Fishjam room (peers) ──forwarded──▶ composition (template) ──WHIP──▶ Fishjam livestream ──WHEP──▶ [viewers]
+```
+
+## Prerequisites
+
+- A [livestream](./../../explanation/livestreams) (or any other WHIP/RTMP destination) to publish the composed stream to.
+- A template project scaffolded with the composition CLI (see [Write and deploy a template](./write-and-deploy-a-template)).
+
+Compositions live on `https://rtc.fishjam.io`, while rooms and livestreams live on the Fishjam API. Both take the same Management Token:
+
+```bash
+export COMPOSITION_URL="https://rtc.fishjam.io"
+export FISHJAM_URL="https://fishjam.io/api/v1/connect/<YOUR_FISHJAM_ID>"
+export TOKEN="<YOUR_FISHJAM_TOKEN>"
+```
+
+## Step 1: Create a room and invite peers
+
+Compositions consume h264 video, so the room has to enforce that codec. It is the default, but set it explicitly so a change of default cannot break the composition later:
+
+```bash
+curl -X POST "$FISHJAM_URL/room" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{ "roomType": "conference", "videoCodec": "h264" }'
+```
+
+The room id comes back under `data.room.id`. Save it:
+
+```bash
+export ROOM_ID="<ROOM_ID>"
+```
+
+Every participant needs their own peer token. Create one per person:
+
+```bash
+curl -X POST "$FISHJAM_URL/room/$ROOM_ID/peer" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{ "type": "webrtc", "options": {} }'
+```
+
+Hand each `data.token` to a client and have it join, using [Connect to a room](./../client/connecting) or the [React quick start](./../../tutorials/react-quick-start). The composition renders whoever is publishing, so get at least one peer in with a camera on before you expect a picture.
+
+## Step 2: Write a room-aware template
+
+Hooks from `@fishjam-cloud/composition` give the template live room state; it re-renders automatically as participants join, leave, mute, or speak.
+
+```tsx
+// @jsx: react-jsx
+// ---cut-before---
+import { InputStream, Rescaler, Text, Tiles, View } from "@swmansion/smelter";
+import { usePeers, useSpeakingState } from "@fishjam-cloud/composition";
+import type { PeerWithStreams } from "@fishjam-cloud/composition";
+
+type PeerMetadata = { displayName?: string };
+
+function PeerTile({ peer }: { peer: PeerWithStreams<PeerMetadata> }) {
+  const camera = peer.cameraStream;
+  const cameraOn = camera?.video && !camera.video.paused;
+  const speaking = useSpeakingState(peer.id) === "speech";
+  const name = peer.metadata.peer?.displayName ?? peer.id;
+
+  return (
+    <View style={{ borderWidth: speaking ? 4 : 0, borderColor: "#00cc66ff" }}>
+      {cameraOn ? (
+        <Rescaler>
+          <InputStream inputId={camera.inputId} />
+        </Rescaler>
+      ) : (
+        <View>
+          <Text>{name}</Text>
+        </View>
+      )}
+    </View>
+  );
+}
+
+export default function App() {
+  const peers = usePeers<PeerMetadata>();
+  const connected = peers.filter((peer) => peer.streams.length > 0);
+
+  return (
+    <View style={{ backgroundColor: "#0b1020ff" }}>
+      <Tiles style={{ padding: 8 }}>
+        {connected.map((peer) => (
+          <PeerTile peer={peer} key={peer.id} />
+        ))}
+      </Tiles>
+    </View>
+  );
+}
+```
+
+| Hook                       | Returns                                                                                                        |
+| -------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| `usePeers()`               | All peers in the forwarded room, each with its streams (`cameraStream`, `screenShareStream`, `customStreams`). |
+| `usePeer(peerId)`          | A single peer, or `undefined`.                                                                                 |
+| `useRoom()`                | The forwarded room `{ id }`, or `undefined` before a room is forwarded.                                        |
+| `useSpeakingState(peerId)` | `"speech"` or `"silence"` for active-speaker highlighting.                                                     |
+
+The key link is `stream.inputId`: you pass it to `<InputStream inputId={…} />` to render that participant's forwarded track. A peer's streams fill in once its media actually starts flowing into the composition.
+
+Build the bundle as usual with `npm run build`.
+
+## Step 3: Create the composition and register the templated output
+
+Create the composition with auto-start off, because the room's inputs only appear once forwarding starts. Setting `cleanup_without_inputs` to `false` widens the cleanup condition to require both the inputs and the outputs to go quiet, so the composition survives the wait for the first peer to publish.
+
+:::warning
+That second flag switches off the guard that would otherwise delete an idle composition after five minutes, and a composition bills for GPU time the whole time it exists. Nothing will clean this one up for you, so delete it as soon as you are finished, and do not leave one running after a test. See [Cost and lifecycle](./../../explanation/compositions#cost-and-lifecycle).
+:::
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{ "autostart": false, "cleanup_without_inputs": false }'
+```
+
+Register a templated `whip_client` output that pushes to your livestream's WHIP endpoint. The output configuration and the template bundle go together in one multipart request:
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition/$COMPOSITION/output/main/template" \
+  -H "Authorization: Bearer $TOKEN" \
+  -F 'config={
+        "type": "whip_client",
+        "endpoint_url": "<LIVESTREAM_WHIP_URL>",
+        "bearer_token": "<LIVESTREAM_STREAMER_TOKEN>",
+        "video": { "resolution": { "width": 1280, "height": 720 }, "initial": { "root": { "type": "view" } } },
+        "audio": { "initial": { "inputs": [] } }
+      };type=application/json' \
+  -F "template=@dist/App.js"
+```
+
+## Step 4: Forward the room into the composition
+
+One call to the [Fishjam Server API](/api/rest) wires everything up:
+
+```bash
+curl -X POST "$FISHJAM_URL/room/$ROOM_ID/track_forwardings" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "{ \"compositionURL\": \"$COMPOSITION_URL/api/composition/$COMPOSITION\", \"selector\": \"all\" }"
+```
+
+Everything else happens automatically: Fishjam links the room to the composition, registers an input for every forwarded track, and streams the media in. Your template's `usePeers()` fills with the room's peers as their media starts flowing. You never register room inputs by hand.
+
+A room forwards to one composition at a time. Repeating the call with the same `compositionURL` is a no-op, while pointing the room at a different composition fails with `409 Conflict`.
+
+## Step 5: Start the composition
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition/$COMPOSITION/start" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+Viewers can now watch the composed grid through the livestream's WHEP endpoint.
+
+## Step 6: Clean up
+
+Delete the composition when you are done. Forwarding stops on the Fishjam side when the room itself stops, so delete the room too once you no longer need it. There is no separate call to remove a forwarding from a live room.
+
+```bash
+curl -X DELETE "$COMPOSITION_URL/api/composition/$COMPOSITION" \
+  -H "Authorization: Bearer $TOKEN"
+
+curl -X DELETE "$FISHJAM_URL/room/$ROOM_ID" \
+  -H "Authorization: Bearer $TOKEN"
+```