Skip to content

Commit a682f1d

Browse files
matt-aitkenmatt-aitken
authored
feat(hosting): default self-hosted realtime streams to v2 (s2-lite) (#4185)
## Summary Realtime streams (AI-agent token streaming and run streams) now default to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev) service. Self-hosting previously shipped no S2 configuration, so streams ran on the Redis-backed v1 path and there were no docs for wiring up v2. Both the Docker Compose stack and the Helm chart now provision s2-lite with persistent storage and set the stream env vars out of the box. ## What's included - **Docker Compose**: a persistent `s2` service (s2-lite), a basin init spec, and the `REALTIME_STREAMS_S2_*` plus `REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example` documents the v1 fallback and hosted-S2 options. - **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the non-root image user via `fsGroup`), an `s2` values block, and webapp env wiring with an existing-secret path for hosted S2. - **Docs**: the `REALTIME_STREAMS_S2_*` and `REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference, plus a "Realtime streams" section in the Docker and Kubernetes self-hosting guides. ## Notes - The OSS code default stays `v1`; v2 becomes the default purely through the self-hosting artifacts, so non-self-host deployments are unaffected. Disabling s2, or setting the version back to `v1`, cleanly reverts to Redis-backed v1. - With v2 enabled, the bundled s2 service is a required dependency for streaming: if it is down, streams error while the task itself still runs. That is the intended trade for the better v2 path. - You can point at a hosted S2 at s2.dev instead of the bundled server.
1 parent 6e827f1 commit a682f1d

10 files changed

Lines changed: 399 additions & 1 deletion

File tree

‎docs/self-hosting/docker.mdx‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,27 @@ EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2
354354
355355
This only affects new runs; existing runs continue to read from wherever their events were originally stored.
356356
357+
## Realtime streams
358+
359+
Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` service — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. It stores stream data in a persistent volume and is preconfigured in the webapp compose file, so no setup is required.
360+
361+
To fall back to the Redis-backed **v1** streams, set on the webapp in your `.env`:
362+
363+
```bash
364+
REALTIME_STREAMS_DEFAULT_VERSION=v1
365+
```
366+
367+
To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, point the endpoint at your basin and supply an access token:
368+
369+
```bash
370+
REALTIME_STREAMS_S2_BASIN=your-basin
371+
REALTIME_STREAMS_S2_ENDPOINT=https://your-basin.b.aws.s2.dev/v1
372+
REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false
373+
REALTIME_STREAMS_S2_ACCESS_TOKEN=your-access-token
374+
```
375+
376+
See the [webapp environment variables](/self-hosting/env/webapp) for the full list of realtime stream settings.
377+
357378
## Troubleshooting
358379
359380
- **Deployment fails at the push step.** The machine running `deploy` needs registry access. See the [registry setup](#registry-setup) section for more details.

‎docs/self-hosting/env/webapp.mdx‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,9 +139,14 @@ mode: "wide"
139139
| **Task events** | | | |
140140
| `EVENT_REPOSITORY_DEFAULT_STORE` | No | postgres | Where to store task events. Set to `clickhouse_v2` to store in ClickHouse (recommended for production). |
141141
| **Realtime** | | | |
142-
| `REALTIME_STREAM_VERSION` | No | v1 | Realtime stream protocol version. One of `v1`, `v2`. |
142+
| `REALTIME_STREAM_VERSION` | No | v1 | Stream version exposed to tasks via the `TRIGGER_REALTIME_STREAM_VERSION` variable. Distinct from `REALTIME_STREAMS_DEFAULT_VERSION`. One of `v1`, `v2`. |
143143
| `REALTIME_STREAM_MAX_LENGTH` | No | 1000 | Realtime stream max length. |
144144
| `REALTIME_STREAM_TTL` | No | 86400 (1d) | Realtime stream TTL (s). |
145+
| `REALTIME_STREAMS_DEFAULT_VERSION` | No | v1 | Server-side default the webapp uses when a stream request doesn't pin a version (modern SDKs request `v2`). The Docker and Helm self-hosting defaults set this to `v2`. One of `v1`, `v2`. |
146+
| `REALTIME_STREAMS_S2_BASIN` | No | — | S2 basin that holds v2 realtime streams. Required for `v2`. Must be at least 8 characters. |
147+
| `REALTIME_STREAMS_S2_ENDPOINT` | No | — | Custom S2 API endpoint, including the `/v1` suffix (e.g. `http://s2/v1` for the bundled s2-lite). Omit to use hosted S2 at s2.dev. |
148+
| `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` | No | false | Skip minting per-stream access tokens. Set to `true` for s2-lite, which needs no authentication. |
149+
| `REALTIME_STREAMS_S2_ACCESS_TOKEN` | No | — | S2 access token. Required for hosted S2 unless `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` is `true`. |
145150
| **Bootstrap** | | | |
146151
| `TRIGGER_BOOTSTRAP_ENABLED` | No | 0 | Trigger bootstrap enabled. |
147152
| `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME` | No | — | Trigger bootstrap worker group name. |

‎docs/self-hosting/kubernetes.mdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,32 @@ webapp:
375375

376376
This only affects new runs; existing runs continue to read from wherever their events were originally stored.
377377

378+
## Realtime streams
379+
380+
Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` deployment — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. The chart deploys it with a persistent volume, so no extra services are required.
381+
382+
To fall back to the Redis-backed **v1** streams, set the default version to `v1`:
383+
384+
```yaml
385+
s2:
386+
defaultStreamVersion: "v1"
387+
```
388+
389+
To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, disable the bundled deployment and point at your basin. Supply the access token via an existing secret:
390+
391+
```yaml
392+
s2:
393+
deploy: false
394+
skipAccessTokens: false
395+
external:
396+
endpoint: "https://your-basin.b.aws.s2.dev/v1"
397+
existingSecret: "s2-credentials"
398+
existingSecretAccessTokenKey: "access-token"
399+
basin: "your-basin"
400+
```
401+
402+
To disable realtime streams v2 entirely and use v1, set `s2.deploy: false` with no external endpoint. See `helm show values` for all `s2` options.
403+
378404
## Worker token
379405

380406
When using the default bootstrap configuration, worker creation and authentication is handled automatically. The webapp generates a worker token and makes it available to the supervisor via a shared volume.

‎hosting/docker/.env.example‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,22 @@ OBJECT_STORE_SECRET_ACCESS_KEY=very-safe-password
106106
# MINIO_ROOT_USER=admin
107107
# MINIO_ROOT_PASSWORD=very-safe-password
108108

109+
# Realtime streams
110+
# - Realtime streams power AI-agent token streaming and run streams
111+
# - They default to v2, backed by the bundled s2-lite service (open-source S2, https://s2.dev)
112+
# - To fall back to the Redis-backed v1 streams, set REALTIME_STREAMS_DEFAULT_VERSION=v1
113+
# REALTIME_STREAMS_DEFAULT_VERSION=v2
114+
# The bundled s2-lite creates whatever basin you set here (min 8 characters)
115+
# REALTIME_STREAMS_S2_BASIN=trigger-realtime
116+
# REALTIME_STREAMS_S2_ENDPOINT=http://s2/v1
117+
# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=true
118+
# To use a hosted S2 (https://s2.dev) instead of the bundled s2-lite:
119+
# - point the endpoint at your basin, disable token-skipping, and set an access token
120+
# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false
121+
# REALTIME_STREAMS_S2_ACCESS_TOKEN=
122+
# Pin the s2-lite image in production (full image reference, digest recommended)
123+
# S2_IMAGE=ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c
124+
109125
# Other image tags
110126
# - These are the versions of the other images to use
111127
# - You should lock these to a specific version in production

‎hosting/docker/webapp/docker-compose.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ services:
1818
- postgres
1919
- redis
2020
- clickhouse
21+
- s2
2122
networks:
2223
- webapp
2324
- supervisor
@@ -44,6 +45,15 @@ services:
4445
LOGIN_ORIGIN: ${LOGIN_ORIGIN:-http://localhost:8030}
4546
API_ORIGIN: ${API_ORIGIN:-http://localhost:8030}
4647
ELECTRIC_ORIGIN: http://electric:3000
48+
# Realtime streams v2, backed by the bundled s2-lite service. This powers
49+
# AI-agent token streaming and run streams. Point the endpoint at a hosted
50+
# S2 (https://s2.dev) instead by overriding these and setting an access token.
51+
# To fall back to the Redis-backed v1 streams, set the version to `v1`.
52+
REALTIME_STREAMS_DEFAULT_VERSION: ${REALTIME_STREAMS_DEFAULT_VERSION:-v2}
53+
REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime}
54+
REALTIME_STREAMS_S2_ENDPOINT: ${REALTIME_STREAMS_S2_ENDPOINT:-http://s2/v1}
55+
REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS: ${REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS:-true}
56+
REALTIME_STREAMS_S2_ACCESS_TOKEN: ${REALTIME_STREAMS_S2_ACCESS_TOKEN:-}
4757
DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable}
4858
DIRECT_URL: ${DIRECT_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable}
4959
SESSION_SECRET: ${SESSION_SECRET}
@@ -225,12 +235,62 @@ services:
225235
retries: 5
226236
start_period: 10s
227237

238+
# Generates the s2-lite basin spec from REALTIME_STREAMS_S2_BASIN so the bundled
239+
# basin always matches the basin the webapp targets. s2-lite is distroless (no
240+
# shell), so a tiny init container writes the spec into a shared volume first.
241+
s2-init:
242+
image: busybox:${BUSYBOX_IMAGE_TAG:-1.37}
243+
restart: "no"
244+
logging: *logging-config
245+
environment:
246+
REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime}
247+
command:
248+
- sh
249+
- -c
250+
- |
251+
cat > /config/s2-spec.json <<JSON
252+
{
253+
"basins": [
254+
{
255+
"name": "$$REALTIME_STREAMS_S2_BASIN",
256+
"config": { "create_stream_on_append": true, "create_stream_on_read": true }
257+
}
258+
]
259+
}
260+
JSON
261+
volumes:
262+
- s2-config:/config
263+
networks:
264+
- webapp
265+
266+
# s2-lite: the open-source, self-hostable S2 (https://s2.dev) server that backs
267+
# realtime streams v2. It's distroless (no shell), so a wget/curl healthcheck
268+
# always reports unhealthy even when the API is up — hence no healthcheck.
269+
# Runs as root so it can write to the named volume; storage lives in SlateDB
270+
# under /data, and the basin is created on startup from the generated spec.
271+
s2:
272+
image: ${S2_IMAGE:-ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c}
273+
restart: ${RESTART_POLICY:-unless-stopped}
274+
logging: *logging-config
275+
depends_on:
276+
s2-init:
277+
condition: service_completed_successfully
278+
command: ["lite", "--init-file", "/config/s2-spec.json", "--local-root", "/data"]
279+
user: "0:0"
280+
volumes:
281+
- s2-config:/config
282+
- s2:/data
283+
networks:
284+
- webapp
285+
228286
volumes:
229287
clickhouse:
230288
postgres:
231289
redis:
232290
shared:
233291
minio:
292+
s2:
293+
s2-config:
234294

235295
networks:
236296
docker-proxy:

‎hosting/k8s/helm/templates/_helpers.tpl‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -387,6 +387,27 @@ http://{{ include "trigger-v4.fullname" . }}-electric:{{ .Values.electric.servic
387387
{{- end -}}
388388
{{- end }}
389389

390+
{{/*
391+
Whether realtime streams v2 (S2) is wired up: either the bundled s2-lite is
392+
deployed, or an external S2 endpoint has been configured.
393+
*/}}
394+
{{- define "trigger-v4.s2.enabled" -}}
395+
{{- if or .Values.s2.deploy .Values.s2.external.endpoint -}}
396+
true
397+
{{- end -}}
398+
{{- end }}
399+
400+
{{/*
401+
S2 API endpoint URL, including the /v1 suffix expected by the client.
402+
*/}}
403+
{{- define "trigger-v4.s2.url" -}}
404+
{{- if .Values.s2.deploy -}}
405+
http://{{ include "trigger-v4.fullname" . }}-s2:{{ .Values.s2.service.port }}/v1
406+
{{- else -}}
407+
{{ .Values.s2.external.endpoint }}
408+
{{- end -}}
409+
{{- end }}
410+
390411
{{/*
391412
ClickHouse hostname
392413
*/}}

‎hosting/k8s/helm/templates/s2.yaml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
{{- if .Values.s2.deploy }}
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{ include "trigger-v4.fullname" . }}-s2-spec
6+
labels:
7+
{{- $component := "s2" }}
8+
{{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
9+
data:
10+
s2-spec.json: |
11+
{
12+
"$schema": "https://raw.githubusercontent.com/s2-streamstore/s2/main/cli/schema.json",
13+
"basins": [
14+
{
15+
"name": {{ .Values.s2.basin | quote }},
16+
"config": {
17+
"create_stream_on_append": true,
18+
"create_stream_on_read": true
19+
}
20+
}
21+
]
22+
}
23+
---
24+
apiVersion: apps/v1
25+
kind: StatefulSet
26+
metadata:
27+
name: {{ include "trigger-v4.fullname" . }}-s2
28+
labels:
29+
{{- $component := "s2" }}
30+
{{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
31+
spec:
32+
replicas: 1
33+
serviceName: {{ include "trigger-v4.fullname" . }}-s2
34+
selector:
35+
matchLabels:
36+
{{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 6 }}
37+
template:
38+
metadata:
39+
annotations:
40+
checksum/spec: {{ .Values.s2.basin | sha256sum }}
41+
{{- with .Values.s2.podAnnotations }}
42+
{{- toYaml . | nindent 8 }}
43+
{{- end }}
44+
labels:
45+
{{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 8 }}
46+
spec:
47+
{{- with .Values.s2.podSecurityContext }}
48+
securityContext:
49+
{{- toYaml . | nindent 8 }}
50+
{{- end }}
51+
containers:
52+
- name: s2
53+
{{- with .Values.s2.securityContext }}
54+
securityContext:
55+
{{- toYaml . | nindent 12 }}
56+
{{- end }}
57+
image: "{{ .Values.s2.image.registry }}/{{ .Values.s2.image.repository }}:{{ .Values.s2.image.tag }}{{ with .Values.s2.image.digest }}@{{ . }}{{ end }}"
58+
imagePullPolicy: {{ .Values.s2.image.pullPolicy }}
59+
command:
60+
- "lite"
61+
- "--init-file"
62+
- "/config/s2-spec.json"
63+
- "--local-root"
64+
- "/data"
65+
- "--port"
66+
- {{ .Values.s2.service.targetPort | quote }}
67+
ports:
68+
- name: http
69+
containerPort: {{ .Values.s2.service.targetPort }}
70+
protocol: TCP
71+
{{- if .Values.s2.livenessProbe.enabled }}
72+
livenessProbe:
73+
tcpSocket:
74+
port: http
75+
initialDelaySeconds: {{ .Values.s2.livenessProbe.initialDelaySeconds }}
76+
periodSeconds: {{ .Values.s2.livenessProbe.periodSeconds }}
77+
timeoutSeconds: {{ .Values.s2.livenessProbe.timeoutSeconds }}
78+
failureThreshold: {{ .Values.s2.livenessProbe.failureThreshold }}
79+
successThreshold: {{ .Values.s2.livenessProbe.successThreshold }}
80+
{{- end }}
81+
{{- if .Values.s2.readinessProbe.enabled }}
82+
readinessProbe:
83+
tcpSocket:
84+
port: http
85+
initialDelaySeconds: {{ .Values.s2.readinessProbe.initialDelaySeconds }}
86+
periodSeconds: {{ .Values.s2.readinessProbe.periodSeconds }}
87+
timeoutSeconds: {{ .Values.s2.readinessProbe.timeoutSeconds }}
88+
failureThreshold: {{ .Values.s2.readinessProbe.failureThreshold }}
89+
successThreshold: {{ .Values.s2.readinessProbe.successThreshold }}
90+
{{- end }}
91+
resources:
92+
{{- toYaml .Values.s2.resources | nindent 12 }}
93+
volumeMounts:
94+
- name: spec
95+
mountPath: /config
96+
readOnly: true
97+
- name: data
98+
mountPath: /data
99+
volumes:
100+
- name: spec
101+
configMap:
102+
name: {{ include "trigger-v4.fullname" . }}-s2-spec
103+
{{- if not .Values.s2.persistence.enabled }}
104+
- name: data
105+
emptyDir: {}
106+
{{- end }}
107+
{{- if .Values.s2.persistence.enabled }}
108+
volumeClaimTemplates:
109+
- metadata:
110+
name: data
111+
{{- if .Values.s2.persistence.retain }}
112+
annotations:
113+
helm.sh/resource-policy: keep
114+
{{- end }}
115+
spec:
116+
accessModes:
117+
- {{ .Values.s2.persistence.accessMode }}
118+
resources:
119+
requests:
120+
storage: {{ .Values.s2.persistence.size }}
121+
{{- if .Values.s2.persistence.storageClass }}
122+
storageClassName: {{ .Values.s2.persistence.storageClass }}
123+
{{- end }}
124+
{{- end }}
125+
---
126+
apiVersion: v1
127+
kind: Service
128+
metadata:
129+
name: {{ include "trigger-v4.fullname" . }}-s2
130+
labels:
131+
{{- $component := "s2" }}
132+
{{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
133+
spec:
134+
type: {{ .Values.s2.service.type }}
135+
ports:
136+
- port: {{ .Values.s2.service.port }}
137+
targetPort: http
138+
protocol: TCP
139+
name: http
140+
selector:
141+
{{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
142+
{{- end }}

‎hosting/k8s/helm/templates/validate-external-config.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ This template will fail the Helm deployment if external config is missing for re
3939
{{- end }}
4040
{{- end }}
4141

42+
{{- if and .Values.s2.deploy .Values.s2.external.endpoint }}
43+
{{- fail "S2 configuration conflict: s2.external.endpoint is set but s2.deploy=true, so the external endpoint would be ignored. Set s2.deploy=false to use an external S2 endpoint, or clear s2.external.endpoint to use the bundled s2-lite." }}
44+
{{- end }}
45+
4246
{{- if not .Values.registry.deploy }}
4347
{{- if or (not .Values.registry.external.host) }}
4448
{{- fail "Registry external configuration is required when registry.deploy=false. Please provide registry.external.host" }}

‎hosting/k8s/helm/templates/webapp.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,27 @@ spec:
188188
value: {{ .Values.webapp.apiOrigin | quote }}
189189
- name: ELECTRIC_ORIGIN
190190
value: {{ include "trigger-v4.electric.url" . | quote }}
191+
{{- if include "trigger-v4.s2.enabled" . }}
192+
# Realtime streams v2, backed by S2 (bundled s2-lite or an external endpoint)
193+
- name: REALTIME_STREAMS_DEFAULT_VERSION
194+
value: {{ .Values.s2.defaultStreamVersion | quote }}
195+
- name: REALTIME_STREAMS_S2_ENDPOINT
196+
value: {{ include "trigger-v4.s2.url" . | quote }}
197+
- name: REALTIME_STREAMS_S2_BASIN
198+
value: {{ .Values.s2.basin | quote }}
199+
- name: REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS
200+
value: {{ .Values.s2.skipAccessTokens | quote }}
201+
{{- if and .Values.s2.external.endpoint .Values.s2.external.existingSecret }}
202+
- name: REALTIME_STREAMS_S2_ACCESS_TOKEN
203+
valueFrom:
204+
secretKeyRef:
205+
name: {{ .Values.s2.external.existingSecret }}
206+
key: {{ .Values.s2.external.existingSecretAccessTokenKey }}
207+
{{- else if .Values.s2.external.accessToken }}
208+
- name: REALTIME_STREAMS_S2_ACCESS_TOKEN
209+
value: {{ .Values.s2.external.accessToken | quote }}
210+
{{- end }}
211+
{{- end }}
191212
{{- if include "trigger-v4.postgres.useSecretUrl" . }}
192213
- name: DATABASE_URL
193214
valueFrom:

0 commit comments

Comments
 (0)