Skip to content

feat(nixos): package buzz-relay and add a deliberately-disabled module - #2777

Open
cameronraysmith wants to merge 4 commits into
mainfrom
fm/vx-buzz-relay-magnetite
Open

feat(nixos): package buzz-relay and add a deliberately-disabled module#2777
cameronraysmith wants to merge 4 commits into
mainfrom
fm/vx-buzz-relay-magnetite

Conversation

@cameronraysmith

Copy link
Copy Markdown
Owner

Packages the Buzz Nostr relay and adds a NixOS module for it, shipped disabled.
magnetite imports the module but does not enable it, so nothing on the live host changes.
Not deployed.

Why this shape

This repository already contained an unsuperseded decision spike, docs/notes/development/buzz/self-hosting.md (2026-08-04), whose verdict was do not self-host the relay now.
Research confirmed its premises still hold at upstream 6e5c462: the relay hard-requires Postgres, Redis and an S3-compatible store, media-storage init is fatal at startup, and there is no SQLite or embedded path.
Rather than run past that or stop on it, the scope was settled deliberately: land the hard, reviewable work — packaging, the module, the specification — and leave the operational go/no-go as a separate gated change.

Three prerequisites are unmet and each is its own change:

  1. an object store that passes the relay's startup CAS conformance probe;
  2. Redis, a daemon class this fleet has never run;
  3. backups, which do not exist fleet-wide.

Postgres, Redis and object storage are modelled here as dependencies and options.
This module provisions none of them.

The R2 verdict — the most useful thing in this PR

R2 was required to be verified at source, not assumed. It was, and it fails.

The git object store runs a conformance probe at boot that races 32 writers against a single key for three rounds, twice (192 conditional writes per boot).
Cloudflare documents a hard limit of 1 write/second per key, returning HTTP 429, corroborated by error code 10058.
The failure is not confined to the probe: the pointer key is the sole writer-serialization primitive, and 429 is not mapped to Conflict, so concurrent pushes surface an opaque 500 instead of a 409 pull-and-retry — intermittent and load-dependent, the worst shape for a correctness-adjacent defect.

Three of the spike's own arguments were wrong and are corrected in the addendum:

  • it missed the BUZZ_GIT_CONFORMANCE_PROBE=false off-switch;
  • it asserted the 429 mechanism without checking the client — it is rust-s3 with RETRIES defaulting to 1, one blind retry that cannot absorb 32 writers (and it is status-blind, so it also retries 412 on every backend);
  • it overstated R2's consistency weakness — R2 documents strong read-after-write and list consistency, so the real gap is silence on CAS atomicity, not documented weakness.

BUZZ_GIT_PROBE_WRITERS=2 is worse than leaving it alone: it risks a false pass.
Recommendation is Garage single-node on fleet fit, with MinIO as the known-good fallback since upstream verifies against MinIO; the probe itself is the cheap adjudicator.
Garage's conditional-PUT CAS was not doc-verified in this pass, and the addendum says so.

What is deliberately not spent here

No nginx vhost, no DNS record, no tenant hostname.
The Host derived from RELAY_URL is the durable tenant key, persisted as communities.host and signed into every NIP-42/NIP-98 auth event, so it cannot be renamed later without invalidating history.
relayUrl therefore has no default and is required at enable time — as are database.url, redis.url and the object-store endpoint and bucket, because every upstream default is a working dev value that would otherwise be inherited silently.

Notable decisions

  • Separate relay-v* source pin. The relay ships at 0.2.1 on relay-v* while buzz-source tracks desktop-v* and feeds the CLI and credential helper the home configuration uses. Repinning the shared source would couple a server upgrade to a client upgrade.
  • BUZZ_REQUIRE_AUTH_TOKEN defaulted to true, departing from upstream's false. That default selects a published hardcoded dev key when no private key is supplied; with the identity generator always supplying one, true is safe and converts a silent fallback into a loud failure.
  • Secrets travel the sops-backed clan-vars lane and arrive via EnvironmentFile (the relay has no *_FILE variants). An earlier draft had both a generator and a manual credentialsFile option, so an operator would have supplied the same S3 credentials twice, by two mechanisms, one inert. Resolved by wiring the generators in and deleting the manual options.
  • Known gap: BUZZ_WEB_DIR/BUZZ_ADMIN_WEB_DIR are unset because web and admin-web are Vite/React sources needing a pnpm build. That means no admin UI and no invite landing page. Unset is upstream's own default; pointing them at a directory lacking index.html is instead a hard startup failure.

Verification

Every check below was run and observed to pass.

Check Result
nix build .#nixosConfigurations.magnetite.config.system.build.toplevel -L pass (x86_64-linux via magnetite as remote builder)
nix build .#checks.{aarch64-darwin,x86_64-linux}.package-buzz-relay{,-source} -L pass on both systems
openspec validate --all --strict 10 passed / 0 failed (was 9)
just lintprek (gitleaks + treefmt) pass; gitleaks scanned 142 KB of staged content, and ran again on each of the four commits
eval of all 6 nixosConfigurations + 4 darwinConfigurations toplevels pass

Why this selection covers the change. It adds two by-name packages and one module imported by exactly one host. The package checks cover the derivations on both systems that modules/checks/packages.nix will auto-generate — which is why building on aarch64-darwin mattered, and it does, so no blacklist entry was needed. The magnetite toplevel build covers the module import and the host's whole evaluation; the realised drvPath matched the independent fleet eval, so the thing evaluated is the thing built. The all-host eval covers the one genuine blast-radius risk, that a new file under modules/ reaches other machines through import-tree auto-import. gitleaks is load-bearing here because the change mints secret generators.

Deliberately not run: full nix flake check, which builds every package for every system and is far larger than this change; and builds (as opposed to evals) of the nine non-magnetite hosts, which this change cannot reach.

Inertness was proven, not asserted. No unit, no user, no firewall change, no clan-vars generator materialises, and buzz-relay is absent from all 21,231 derivations in magnetite's closure. A drvPath shift was traced to inputs.self source hashing and refuted by a control that added an empty unrelated module and reproduced the same shift.

Review notes

An adversarial review checked 24 upstream citations against 6e5c462; all 24 held. It found no blocking defect and four should-fix items, all in the enable path rather than the disabled-ship property. All four are fixed in this branch.

One nit is knowingly left: the no-public-bind assertion's isLoopbackOrMesh accepts the name localhost, making it a slightly weaker gate than cognee's address-based one. It only matters on the enable path, which gets its own change and its own review.

The relay releases independently of the desktop app: crates/buzz-relay
ships at 0.2.1 under relay-v* tags while the existing buzz-source tracks
desktop-v* and feeds the CLI, credential helper and signer that the home
configuration consumes. Repinning the shared source would couple a server
upgrade to a client upgrade, so the relay gets its own pin and its own
cargo vendor tree at relay-v0.2.1 (6e5c462, 2026-08-08).

Two pieces of builder knowledge are load-bearing and were verified against
upstream before being copied. The relay writes a fail-closed bash
pre-receive hook that shells out to curl and openssl, so its shebang is
rewritten to a store path (the hook is a compiled-in const, hence
substituteInPlace rather than patchShebangs). The wrapper puts
bashNonInteractive, coreutils, curl, gitMinimal, gnused and openssl on
PATH and sets SSL_CERT_FILE, because the relay also calls bare
Command::new("git") directly.

BUZZ_WEB_DIR and BUZZ_ADMIN_WEB_DIR are deliberately left unset: the web
and admin-web bundles are Vite/React sources needing a pnpm build, not
prebuilt assets. Unset is upstream's own default and yields no admin UI
and no invite landing page; pointing them at a directory without an
index.html is instead a hard startup failure.

Builds on aarch64-darwin and x86_64-linux, so no modules/checks blacklist
entry is needed. Reaches aws-lc-sys via cmake; openssl-sys is not
compiled.
magnetite imports the module; enable defaults to false and is not set, so
nothing on the live host changes. Inertness was proven rather than
asserted: no unit, no user, no firewall hole, no clan-vars generator, and
buzz-relay is absent from all 21231 derivations of magnetite's closure.

The module exists so the configuration surface is reviewable and the
credential slots are declared before anything is switched on. Three
operational prerequisites are unmet and each is its own change: an object
store that passes the relay's startup CAS conformance probe, Redis as a
daemon class this fleet has never run, and backups, which do not exist
fleet-wide.

Postgres, Redis and object storage are modelled as dependencies and
options. This module provisions none of them.

No nginx vhost, no DNS record and no tenant hostname are spent here. The
Host derived from RELAY_URL is the durable tenant key, persisted as
communities.host and signed into every NIP-42/NIP-98 auth event, so it
cannot be renamed later without invalidating history. relayUrl therefore
has no default and is required at enable time, as are database.url,
redis.url and the object-store endpoint and bucket, because every upstream
default is a working dev value that would otherwise be inherited silently.

Sixteen load-bearing settings are typed; the remaining ~90 environment
variables route through a free-form settings attrset rendered last.
Secrets travel the fleet's sops-backed clan-vars lane and arrive via
EnvironmentFile, since the relay has no *_FILE config variants. Four
generators are minted, named after the service rather than the upstream
variable so an upstream rename cannot strand sops material.

BUZZ_REQUIRE_AUTH_TOKEN is defaulted to true, departing from upstream's
false. That default selects a published hardcoded dev key when no private
key is supplied; with the identity generator always supplying one, true is
safe and converts a silent fallback into a loud failure.

The firewall is scoped to the mesh interface, and a build-time assertion
refuses a public bind, because net.ipv6.ip_nonlocal_bind makes a mistaken
public bind succeed silently.
Declares a new buzz-relay capability, following declarative-cognee-endpoint's
precedent that a service change declares its own rather than modifying an
existing one. Records the design decisions behind the packaging and the
module, and carries the enable path's prerequisites explicitly in the
Migration Plan so the later go/no-go change inherits them rather than
rediscovering them.

The spec deltas describe what actually ships, which is a module that
evaluates inert, asserts on missing required options and provisions no
daemons. They deliberately do not describe a running relay, because none
runs.

tasks.md carries only this change's work and is complete, so the change is
archivable on its own.

openspec validate --all --strict passes 10/0, up from 9.
The 2026-08-04 spike left Cloudflare R2 fitness as its central open gate.
It is now settled at source against upstream 6e5c462, and the addendum
records the answer alongside three corrections to the spike's own
reasoning.

R2 fails. The relay's git object store runs a conformance probe at boot
that races 32 writers against a single key for three rounds, twice, while
Cloudflare documents a hard limit of one write per second per key
returning HTTP 429. The failure is not confined to the probe: the pointer
key is the sole writer-serialization primitive, and a 429 is not mapped to
Conflict, so concurrent pushes surface an opaque 500 rather than a 409
pull-and-retry.

Three of the spike's arguments were wrong and are corrected. It missed the
BUZZ_GIT_CONFORMANCE_PROBE off-switch. It asserted the 429 mechanism
without checking the client, which is rust-s3 with RETRIES defaulting to 1,
a single blind retry that cannot absorb 32 writers. And it overstated R2's
consistency weakness: R2 documents strong read-after-write and list
consistency, so the real gap is silence on CAS atomicity rather than
documented weakness.

Setting BUZZ_GIT_PROBE_WRITERS=2 is worse than leaving it alone, because it
risks a false pass. Garage single-node is recommended for fleet fit, with
MinIO as the known-good fallback since upstream verifies against it; the
probe itself is the cheap adjudicator.

Appended only; the note's existing 428 lines are byte-identical.
@mergify

mergify Bot commented Aug 19, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant