Skip to content

Commit b87e5fc

Browse files
ConProgrammingConProgramming
authored
Merge branch 'triggerdotdev:main' into main
2 parents 79c7404 + 5dacab0 commit b87e5fc

100 files changed

Lines changed: 9216 additions & 575 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/cli-deploy-skip-rewrite-timestamp.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"trigger.dev": patch
3+
---
4+
5+
Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local self-hosted builds whose buildx driver doesn't support `rewrite-timestamp` alongside push (e.g. orbstack's default `docker` driver).

‎.changeset/mollifier-redis-worker-primitives.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"@trigger.dev/redis-worker": patch
3+
---
4+
5+
Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing.
6+
7+
MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts.
8+
9+
MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count.

‎.env.example‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,52 @@ REDIS_TLS_DISABLED="true"
2929
DEV_OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3030/otel"
3030
DEV_OTEL_BATCH_PROCESSING_ENABLED="0"
3131

32+
# Realtime streams v2 (Sessions, chat.agent, large stream backfills) backed
33+
# by S2 (https://s2.dev). The `s2` service in docker/docker-compose.yml runs
34+
# the open-source s2-lite binary and pre-creates a basin named `trigger-local`
35+
# (see docker/config/s2-spec.json). Comment these out to fall back to v1
36+
# (Redis-only) streams; Sessions and chat.agent then become unavailable.
37+
REALTIME_STREAMS_S2_BASIN=trigger-local
38+
REALTIME_STREAMS_S2_ACCESS_TOKEN=ignored
39+
REALTIME_STREAMS_S2_ENDPOINT=http://localhost:4566/v1
40+
REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=true
41+
REALTIME_STREAMS_DEFAULT_VERSION=v2
42+
43+
# Running multiple instances side by side (worktrees, branch experiments)
44+
#
45+
# Every host port in docker/docker-compose.yml is `${VAR:-default}` and the
46+
# project name comes from `COMPOSE_PROJECT_NAME`. To stand up a second stack
47+
# alongside the default one, uncomment the block below in this clone's `.env`
48+
# (pick any offset that doesn't clash with anything else running), then update
49+
# the URL/PORT vars further up to match. Default values are commented for
50+
# reference.
51+
#
52+
# --- core (pnpm run docker) ---
53+
# COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt
54+
# CONTAINER_PREFIX=alt-
55+
# POSTGRES_HOST_PORT=15432 # default 5432
56+
# REDIS_HOST_PORT=16379 # default 6379
57+
# ELECTRIC_HOST_PORT=13060 # default 3060
58+
# MINIO_API_HOST_PORT=19005 # default 9005
59+
# MINIO_CONSOLE_HOST_PORT=19006 # default 9006
60+
# CLICKHOUSE_HTTP_HOST_PORT=18123 # default 8123
61+
# CLICKHOUSE_TCP_HOST_PORT=19000 # default 9000
62+
# S2_HOST_PORT=14566 # default 4566
63+
# REMIX_APP_PORT=13030 # default 3030
64+
# --- extras (only needed if you also run `pnpm run docker:full`) ---
65+
# ELECTRIC_SHARD_1_HOST_PORT=13061 # default 3061
66+
# CH_UI_HOST_PORT=15521 # default 5521
67+
# TOXIPROXY_PROXY_HOST_PORT=40303 # default 30303
68+
# TOXIPROXY_API_HOST_PORT=18474 # default 8474
69+
# NGINX_H2_HOST_PORT=18443 # default 8443
70+
# OTEL_GRPC_HOST_PORT=14317 # default 4317
71+
# OTEL_HTTP_HOST_PORT=14318 # default 4318
72+
# OTEL_PROMETHEUS_HOST_PORT=18889 # default 8889
73+
# PROMETHEUS_HOST_PORT=19090 # default 9090
74+
# GRAFANA_HOST_PORT=13001 # default 3001
75+
# (and update DATABASE_URL / CLICKHOUSE_URL / REDIS_PORT / APP_ORIGIN /
76+
# LOGIN_ORIGIN / ELECTRIC_ORIGIN / REALTIME_STREAMS_S2_ENDPOINT to match)
77+
3278
# When the domain is set to `localhost` the CLI deploy command will only --load the image by default and not --push it
3379
DEPLOY_REGISTRY_HOST=localhost:5000
3480

@@ -106,7 +152,7 @@ POSTHOG_PROJECT_KEY=
106152
# INTERNAL_OTEL_TRACE_LOGGING_ENABLED=1
107153
# INTERNAL_OTEL_TRACE_INSTRUMENT_PRISMA_ENABLED=0
108154

109-
# Enable local observability stack (requires `pnpm run docker` to start otel-collector)
155+
# Enable local observability stack (requires `pnpm run docker:full` to bring up otel-collector + prometheus + grafana)
110156
# Uncomment these to send metrics to the local Prometheus via OTEL Collector:
111157
# INTERNAL_OTEL_METRIC_EXPORTER_ENABLED=1
112158
# INTERNAL_OTEL_METRIC_EXPORTER_URL=http://localhost:4318/v1/metrics

‎.github/workflows/pr_checks.yml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,21 @@ jobs:
1717
name: Detect changes
1818
runs-on: ubuntu-latest
1919
outputs:
20-
code: ${{ steps.filter.outputs.code }}
20+
code: ${{ steps.code_filter.outputs.code }}
21+
typecheck_self: ${{ steps.filter.outputs.typecheck_self }}
2122
webapp: ${{ steps.filter.outputs.webapp }}
2223
packages: ${{ steps.filter.outputs.packages }}
2324
internal: ${{ steps.filter.outputs.internal }}
2425
cli: ${{ steps.filter.outputs.cli }}
2526
sdk: ${{ steps.filter.outputs.sdk }}
2627
steps:
28+
# `code` uses `every` semantics so the negation patterns actually subtract.
29+
# With the default `some` quantifier, `**` matches every file and the
30+
# subsequent `!...` patterns are no-ops (each pattern is OR'd, not AND'd).
2731
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
28-
id: filter
32+
id: code_filter
2933
with:
34+
predicate-quantifier: every
3035
filters: |
3136
code:
3237
- '**'
@@ -37,6 +42,11 @@ jobs:
3742
- '!references/**'
3843
- '!**/*.md'
3944
- '!**/.env.example'
45+
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
46+
id: filter
47+
with:
48+
filters: |
49+
typecheck_self:
4050
- '.github/workflows/pr_checks.yml'
4151
- '.github/workflows/typecheck.yml'
4252
webapp:
@@ -95,7 +105,7 @@ jobs:
95105
96106
typecheck:
97107
needs: changes
98-
if: needs.changes.outputs.code == 'true'
108+
if: needs.changes.outputs.code == 'true' || needs.changes.outputs.typecheck_self == 'true'
99109
uses: ./.github/workflows/typecheck.yml
100110

101111
webapp:

‎.server-changes/magic-link-email-validation.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: fix
4+
---
5+
6+
Validate email format on the magic link login form.

‎.server-changes/mollifier-burst-protection.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: feature
4+
---
5+
6+
Lay the groundwork for an opt-in burst-protection layer on the trigger hot path. This release ships **monitoring only** — operators can observe per-env trigger storms via two opt-in modes, but no trigger calls are diverted or rate-limited yet (active burst smoothing follows in a later release). All new env vars are prefixed `TRIGGER_MOLLIFIER_*` and default off, so existing deployments see no behaviour change. With `TRIGGER_MOLLIFIER_SHADOW_MODE=1`, each trigger evaluates a per-env rate counter and logs `mollifier.would_mollify` when the threshold is crossed. With `TRIGGER_MOLLIFIER_ENABLED=1` plus a per-org `mollifierEnabled` flag, over-threshold triggers are also recorded in a Redis audit buffer alongside the normal `engine.trigger` call, drained by a background no-op consumer. The drainer has its own switch (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`) so multi-replica deployments can pin the polling loop to a single worker service while every replica still produces into the buffer; unset, it inherits `TRIGGER_MOLLIFIER_ENABLED` so single-container self-hosters need only one flag. Drainer misconfiguration (shutdown-timeout reconciliation against `GRACEFUL_SHUTDOWN_TIMEOUT`, or `TRIGGER_MOLLIFIER_ENABLED=1` with no buffer Redis) now throws `MollifierConfigurationError` at boot and crashes the process, so the misconfig surfaces to the orchestrator instead of disappearing into a log line; transient init failures (Redis blip) are still logged-and-swallowed. Emits the `mollifier.decisions` OTel counter for per-env rate visibility.

‎.server-changes/otel-attribute-utf16-sanitization.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
area: webapp
3+
type: fix
4+
---
5+
6+
Recover from ClickHouse `JSONEachRow` parse failures caused by lone
7+
UTF-16 surrogates in OTel attribute strings (`Cannot parse JSON object
8+
here ... ParallelParsingBlockInputFormat`).
9+
10+
`ClickhouseEventRepository.#flushBatch` and `#flushLlmMetricsBatch` now
11+
retry once after sanitizing every row in the batch: any string value
12+
containing a lone surrogate is replaced with `"[invalid-utf16]"`. If
13+
the sanitizer touched no fields (the parse error isn't a surrogate
14+
issue) or the retry still fails, the batch is dropped without further
15+
ClickHouse round-trips, `permanentlyDroppedBatches` increments, and an
16+
error log with a 1KB sample row is emitted. Non-parse errors propagate
17+
unchanged.
18+
19+
Detection reuses `detectBadJsonStrings` via `JSON.stringify(value)`,
20+
with a latent regex bug fixed: the low-surrogate hex nibble matched
21+
`[cd]` instead of `[c-f]`, missing the U+DE00–U+DFFF half of the range
22+
and false-flagging common emoji pairs. Healthy batches pay zero scan
23+
cost — the check only runs when ClickHouse has already rejected.

‎.server-changes/realtimestreams-dedupe.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: fix
4+
---
5+
6+
Dedupe the `realtimeStreams` array push on `PUT /realtime/v1/streams/:runId/:target/:streamId` so repeat stream-init calls for the same `(run, streamId)` skip the row UPDATE, mirroring the existing append handler.

‎.server-changes/webapp-sentry-fingerprint-p1001.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: improvement
4+
---
5+
6+
Group Prisma P1001 ("Can't reach database server") errors into a single Sentry issue via a `beforeSend` fingerprint rule, so DB outages no longer fan out into hundreds of distinct issues that bury other alerts. Adds a small extensible rule table for future collapsing rules.

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ See `ai/references/repo.md` for a more complete explanation of the workspaces.
1919
```bash
2020
pnpm run docker
2121
```
22+
Add `:full` (`pnpm run docker:full`) for the optional observability + chaos tooling. See `docker/docker-compose.extras.yml`.
2223
4. Run database migrations:
2324
```bash
2425
pnpm run db:migrate

0 commit comments

Comments
 (0)