This is the design record for the chat-moderation work — the goal and the path, here rather than in DMs so it can be argued with async. See Status below for where the code currently sits.
Goal
Let players use free-form chat without the failure modes that made preset-only the safe default: harassment, slurs, and threats reaching other players, with no moderation cost that scales per message and no third-party dependency. The bar: a blocked message never reaches another player, a clean message adds ~no latency, and an outage never lets anything through unmoderated.
Where we started
The original decision record settled on preset-only chat after rejecting blocklists (evadable), hosted ML (~$200/mo at our volume), and self-hosted ML (assumed to cost more than the server). Re-examining that: small guard models now run in-process on CPU at real-time latency, so the cost premise changed. Before building anything, we measured instead of assumed — the full 1.34M-message chat history: 99.3% completely clean; genuine harm ~0.03% (violent threats: 5 messages ever). The conclusion cuts both ways: chat is safe enough to open up, and the problem is small enough that deterministic rules do most of the work — the ML model is a backstop, not the foundation.
The path we took
A separate verdict-only moderation service; the relay stays the edge authority.
client → relay POST /chat → POST /moderate → verdict (allow / reject / rewrite)
↳ relay publishes on allow, maps rejects to HTTP
- The relay owns identity, membership, age gate, bans, publishing. It sends
{playerId (opaque UUID), displayName, lobbyCode, message} and gets a verdict. The service never sees Steam/Discord identity.
- The service owns content judgment: deterministic tiers first (curated preset allowlist ~5k entries built from our real chat, hard rules for directed threats, slur blocklist, link stripping, vocab rewrites), then a small fine-tuned guard model (~300ms, in-process, $0/message) only for what falls through. Clean messages short-circuit at 1–9ms and never touch the model.
- Fail-closed (deliberate): service unreachable ⇒ chat 503s, or degrades to allowlist-only (
MODERATION_OUTAGE_POLICY=presets) — presets are safe by construction. Free-form never proceeds unmoderated.
- No automatic retries: the sender gets instant feedback (see UX below), so the player is the retry loop; auto-retries would amplify load exactly when the service is struggling.
Player-facing UX (the part we care most about): your message echoes instantly when you hit enter. If it was blocked you immediately see "not delivered to other players"; if it was auto-cleaned you see "delivered as: …" — no silent shadow-banning, no mystery. Recipients only ever receive the delivered form.
Evidence: the reported-lobby buffer keeps the original text — a rewrite never launders the record a moderator later reviews. (The service used to keep its own message+verdict log too; it no longer keeps anything — see Status.)
Privacy split: word lists / rewrite rules / eval data live in a private config repo (so slurs never appear in public code); the service repo itself is public-ready.
v1 scope — deliberately small
v1 ships blocking + the delivery notices, nothing else. Player reports, appeals, and a moderator review console are already built but parked on branches — shipping the simple thing first, then iterating on what real usage shows we need.
Since first writing this, v1 got smaller still. The moderation service is now stateless: it takes a message, returns a verdict, and stores nothing — no chat log, no per-player record, no database at all. It has exactly two endpoints (/moderate and /health) and its only output besides the verdict is an operational log line with the message content stripped. Anything that needs to remember things about a player — reports, strikes, sanctions, review — is a separate service for later, not a thing this one grew into.
Muting is the client's job for now. The game keeps a local muted-players set and drops those senders from its own display; that needs no server support at all. So v1 has no server-issued mute, and nothing can silence someone for everyone at once — a moderator who needs that uses the relay's existing chat ban.
v2 and beyond (parked, mostly already built)
Roughly in the order real usage will probably demand them:
- Player reports — in-client report flow + relay intake endpoints with reported-message evidence capture. Built, then set aside: the client UI is archived and the service-side intake/review machinery was deleted in the stateless carve. The relay half survives on
feat/moderation-intake-full.
- Review surface — a Discord webhook for flagged/reported messages first (zero UI to build), an optional web console later. Both were built and removed; whatever replaces them will need its own retention answer, which is why question 2 below is only resolved for the verdict service.
- Sanctions ladder + appeals — server-issued mutes, the escalation policy (warn → timed mutes → permanent) driven by strikes, and an appeal path. Built and parked; this is the piece that would own player state, keeping the verdict service stateless.
- Wire contract as a package — see open question 4.
- Transport evolution — if the HTTP hop ever matters, the relay can subscribe to MQTT inbox topics and republish post-verdict; the verdict service itself doesn't change.
- Reuse — the same
POST /moderate can front any other chat surface unchanged.
None of this blocks v1 — it's listed so we agree on direction, not to commit to any of it now.
Status
Both PRs are closed and this is off the review queue for now — the code isn't going anywhere, it just isn't waiting on anyone. This issue is the design record — the useful place to argue with any of the above, whether it sits open or closed.
If this restarts, the rollout is unchanged: merge → set env vars → run service-side shadow mode on real traffic (logs would-blocks, blocks nothing) → flip enforcement when the numbers look right. Nothing about the design below depends on the PRs being open.
Open questions
- Rewrite vs block — mild profanity gets auto-rewritten ("delivered as…") rather than blocked. Is rewriting someone's words acceptable given the sender sees exactly what was delivered, or should everything just block?
Retention — resolved: the service stores no chat. This was going to be a proposal about retention windows and a purge job. It's moot now — a stateless service has nothing to retain, so there's no chat database to set a policy on, no purge job to run, and no second copy of player messages anywhere. If a review surface ships later it will need its own retention answer, and that's the right time to have the argument.- Hosting/ownership — the service runs on separate infra right now (HTTPS + bearer, CI-deployed). Happy to hand over, replicate, or keep running it.
- Wire contract — the band vocabulary is currently documented in the gateway, and the relay is built to tolerate drift (unknown bands degrade to a generic block), so v1 is fine as-is. Longer term the shared types should live in one place — either a tiny contract package published from the service repo, or
@bmp/types. This couples with question 3: whoever ends up owning the service should own the contract package.
This is the design record for the chat-moderation work — the goal and the path, here rather than in DMs so it can be argued with async. See Status below for where the code currently sits.
Goal
Let players use free-form chat without the failure modes that made preset-only the safe default: harassment, slurs, and threats reaching other players, with no moderation cost that scales per message and no third-party dependency. The bar: a blocked message never reaches another player, a clean message adds ~no latency, and an outage never lets anything through unmoderated.
Where we started
The original decision record settled on preset-only chat after rejecting blocklists (evadable), hosted ML (~$200/mo at our volume), and self-hosted ML (assumed to cost more than the server). Re-examining that: small guard models now run in-process on CPU at real-time latency, so the cost premise changed. Before building anything, we measured instead of assumed — the full 1.34M-message chat history: 99.3% completely clean; genuine harm ~0.03% (violent threats: 5 messages ever). The conclusion cuts both ways: chat is safe enough to open up, and the problem is small enough that deterministic rules do most of the work — the ML model is a backstop, not the foundation.
The path we took
A separate verdict-only moderation service; the relay stays the edge authority.
{playerId (opaque UUID), displayName, lobbyCode, message}and gets a verdict. The service never sees Steam/Discord identity.MODERATION_OUTAGE_POLICY=presets) — presets are safe by construction. Free-form never proceeds unmoderated.Player-facing UX (the part we care most about): your message echoes instantly when you hit enter. If it was blocked you immediately see "not delivered to other players"; if it was auto-cleaned you see "delivered as: …" — no silent shadow-banning, no mystery. Recipients only ever receive the delivered form.
Evidence: the reported-lobby buffer keeps the original text — a rewrite never launders the record a moderator later reviews. (The service used to keep its own message+verdict log too; it no longer keeps anything — see Status.)
Privacy split: word lists / rewrite rules / eval data live in a private config repo (so slurs never appear in public code); the service repo itself is public-ready.
v1 scope — deliberately small
v1 ships blocking + the delivery notices, nothing else. Player reports, appeals, and a moderator review console are already built but parked on branches — shipping the simple thing first, then iterating on what real usage shows we need.
Since first writing this, v1 got smaller still. The moderation service is now stateless: it takes a message, returns a verdict, and stores nothing — no chat log, no per-player record, no database at all. It has exactly two endpoints (
/moderateand/health) and its only output besides the verdict is an operational log line with the message content stripped. Anything that needs to remember things about a player — reports, strikes, sanctions, review — is a separate service for later, not a thing this one grew into.Muting is the client's job for now. The game keeps a local muted-players set and drops those senders from its own display; that needs no server support at all. So v1 has no server-issued mute, and nothing can silence someone for everyone at once — a moderator who needs that uses the relay's existing chat ban.
v2 and beyond (parked, mostly already built)
Roughly in the order real usage will probably demand them:
feat/moderation-intake-full.POST /moderatecan front any other chat surface unchanged.None of this blocks v1 — it's listed so we agree on direction, not to commit to any of it now.
Status
Both PRs are closed and this is off the review queue for now — the code isn't going anywhere, it just isn't waiting on anyone. This issue is the design record — the useful place to argue with any of the above, whether it sits open or closed.
MODERATION_SERVICE_URLis set), branch intact.If this restarts, the rollout is unchanged: merge → set env vars → run service-side shadow mode on real traffic (logs would-blocks, blocks nothing) → flip enforcement when the numbers look right. Nothing about the design below depends on the PRs being open.
Open questions
Retention— resolved: the service stores no chat. This was going to be a proposal about retention windows and a purge job. It's moot now — a stateless service has nothing to retain, so there's no chat database to set a policy on, no purge job to run, and no second copy of player messages anywhere. If a review surface ships later it will need its own retention answer, and that's the right time to have the argument.@bmp/types. This couples with question 3: whoever ends up owning the service should own the contract package.