Skip to content

Commit eff2582

Browse files
MikhusMikhus
committed
blog: horizontally auto-scaling Redis broker (promoter/unicaster)
New Serhiy Morenko article on scaling the @imqueue Redis broker layer: client-side ClusteredRedisQueue clustering, UDP broker discovery, the redis-broker-promoter (broadcast) and redis-broker-unicaster (K8s/GCP unicast) recipes, framed as horizontal auto-scaling. Also expose an agent-facing plain-markdown mirror of each article in llms.txt and add a _headers file marking the .md mirrors and llms*.txt noindex, so the AI surfaces don't create duplicate-content SEO issues.
1 parent 03d4464 commit eff2582

5 files changed

Lines changed: 307 additions & 1 deletion

File tree

‎images/blog/horizontally-scalable-redis-broker.png‎

91.6 KB
Loading

‎src/_shared/_includes/blog-art/broker-fleet.svg‎

Lines changed: 43 additions & 0 deletions
Loading

‎src/headers.liquid‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
permalink: /_headers
3+
eleventyExcludeFromCollections: true
4+
---
5+
# Cloudflare Pages response headers.
6+
# https://developers.cloudflare.com/pages/configuration/headers/
7+
#
8+
# The site publishes AI-agent-facing plain-markdown mirrors of its pages
9+
# (<page-url>index.md), a curated /llms.txt index and a concatenated
10+
# /llms-full.txt. They intentionally duplicate the canonical HTML content, so
11+
# keep them out of search-engine indexes (avoiding duplicate-content issues)
12+
# while leaving them fully fetchable by AI crawlers and agents — robots.txt
13+
# allows them, noindex only stops SERP indexing, not retrieval.
14+
15+
/*.md
16+
X-Robots-Tag: noindex
17+
18+
/llms.txt
19+
X-Robots-Tag: noindex
20+
21+
/llms-full.txt
22+
X-Robots-Tag: noindex

‎src/llms.liquid‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,9 @@ The open-source packages (@imqueue/core, @imqueue/rpc, @imqueue/cli) and their d
6565
- [Documentation home]({{ siteUrl }}/docs/): Entry point to all @imqueue documentation.
6666

6767
## Articles
68+
Every article is also published as agent-friendly plain markdown at `<article-url>index.md` — same content, no HTML. Canonical HTML pages are listed first, the markdown mirror after each summary.
6869
{%- for post in collections.posts %}
69-
- [{{ post.data.title }}]({{ siteUrl }}{{ post.url }}){% if post.data.summary %}: {{ post.data.summary | strip_newlines }}{% endif %}
70+
- [{{ post.data.title }}]({{ siteUrl }}{{ post.url }}){% if post.data.summary %}: {{ post.data.summary | strip_newlines }}{% endif %} — [markdown]({{ siteUrl }}{{ post.url }}index.md)
7071
{%- endfor %}
7172

7273
## Project

‎src/org/blog/posts/horizontally-scalable-redis-broker.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
---
2+
layout: post.html
3+
permalink: /blog/horizontally-scalable-redis-broker/
4+
templateEngineOverride: md
5+
title: "A horizontally auto-scaling Redis broker: recipes for networks with and without broadcast"
6+
summary: "One Redis behind your message bus is a ceiling and a single point of failure. The promoter and unicaster modules turn a fleet of plain Redis instances into a horizontally auto-scaling broker — here are the recipes for networks that deliver broadcast and for clouds like GCP that don't."
7+
description: "How to run a horizontally auto-scaling Redis message broker with @imqueue: client-side clustering with ClusteredRedisQueue, UDP-based broker discovery, redis-broker-promoter for broadcast-capable networks, and redis-broker-unicaster for GCP/Kubernetes where broadcast is blocked."
8+
keywords: "horizontally scalable redis broker, redis broker auto-scaling, horizontal auto-scaling redis, scale redis message queue, redis broker cluster, redis broker discovery, redis-broker-promoter, redis-broker-unicaster, imqueue cluster, udp broadcast discovery, GCP redis broker, kubernetes redis broker"
9+
date: 2026-07-24
10+
author: serhiy-morenko
11+
illustration: broker-fleet
12+
topics: [discovery, queue, architecture, patterns]
13+
ogType: article
14+
---
15+
16+
**A horizontally scalable Redis broker** is the missing half of scaling a
17+
message-driven system. Adding service instances is easy — with
18+
[`@imqueue`](/get-started/) two copies of a service just read the same queue —
19+
but all of that traffic still funnels through one Redis. At some point that
20+
single broker is both your throughput ceiling and your single point of failure.
21+
@imqueue's answer is not Redis Cluster and not a managed proxy: it's a **fleet
22+
of plain, independent Redis instances** that services discover at runtime, with
23+
producers spreading load across them and consumers draining all of them at
24+
once. And because discovery runs continuously, the fleet doesn't just scale —
25+
it **auto-scales**: add a broker and every service folds it into rotation
26+
within a second; remove one and traffic re-routes just as fast, no config
27+
pushes, no redeploys. The only part that changes between environments is *how
28+
brokers announce themselves* — and that's what the two recipes below are about.
29+
30+
> **TL;DR** — Load a tiny announcer module into every Redis broker and the
31+
> broker layer becomes horizontally **auto-scaling**: services discover the
32+
> fleet over UDP as brokers come and go. On networks that deliver limited
33+
> broadcast (bare metal, LANs, Docker bridge) use
34+
> [redis-broker-promoter](https://github.com/imqueue/redis-broker-promoter),
35+
> which shouts to `255.255.255.255`. On networks that drop broadcast — GCP
36+
> VPCs, most Kubernetes overlays — use
37+
> [redis-broker-unicaster](https://github.com/imqueue/redis-broker-unicaster),
38+
> which asks the Kubernetes API for pod IPs and unicasts the same datagram to
39+
> each of them. The service side is identical either way:
40+
> `clusterManagers: [new UDPClusterManager()]`.
41+
42+
## How @imqueue clusters the broker
43+
44+
Clustering lives on the client side, in
45+
[`ClusteredRedisQueue`](/api/core/latest/core.clusteredredisqueue/). You never
46+
instantiate it directly — the factory swaps it in whenever the options mention
47+
a cluster, so services and generated clients get it with zero code changes:
48+
49+
~~~typescript
50+
import { IMQServiceOptions, UDPClusterManager } from '@imqueue/rpc';
51+
52+
export const serviceOptions: Partial<IMQServiceOptions> = {
53+
// dynamic discovery (the subject of this post):
54+
clusterManagers: [new UDPClusterManager()],
55+
// …or, instead of a manager, a static fleet known up front:
56+
// cluster: [
57+
// { host: 'redis-1', port: 6379 },
58+
// { host: 'redis-2', port: 6379 },
59+
// ],
60+
};
61+
~~~
62+
63+
The model is deliberately simple. There is no sharding and no consistent
64+
hashing: every broker hosts an identically-named queue, producers pick a broker
65+
per message in **health-aware round-robin** (a broker whose connection is
66+
known to be down is skipped), and consumers run a blocking read against **all brokers
67+
concurrently**. Throughput scales with the number of brokers; losing one broker
68+
just narrows the rotation. The brokers themselves are stock standalone Redis —
69+
they never talk to each other, don't replicate, and don't even know they are
70+
part of a fleet. Neither announcer module registers a single Redis command.
71+
72+
## The discovery protocol both recipes share
73+
74+
Each broker loads a small C module that periodically emits a one-line,
75+
tab-separated UDP datagram:
76+
77+
~~~
78+
imq-broker 2cc7c345-3569-44bb-b57a-b72d729d7012 up 10.0.4.12:6379 1
79+
imq-broker 2cc7c345-3569-44bb-b57a-b72d729d7012 down 10.0.4.12:6379
80+
~~~
81+
82+
That's `name`, a per-process GUID, `up`/`down`, the advertised `host:port`, and
83+
— for `up` — the announce interval in seconds. On the service side,
84+
[`UDPClusterManager`](/api/core/latest/core.udpclustermanager/) listens on UDP
85+
port `63000` (its default) in a worker thread and translates datagrams into
86+
cluster changes:
87+
88+
- **`up`** — add the broker (deduplicated by GUID or address) and re-arm its
89+
liveness timer.
90+
- **`down`** — sent on graceful shutdown; the broker is removed immediately.
91+
- **silence** — a broker that misses heartbeats for
92+
`interval × 1000 + 5000 + 1` ms (about six seconds at the default 1-second
93+
interval) is evicted, which covers crashes and network partitions.
94+
95+
Both modules read the same environment variables — `REDIS_BROADCAST_NAME`
96+
(default `imq-broker`), `REDIS_BROADCAST_INTERVAL` (seconds, default `1`) —
97+
and emit byte-identical messages. They differ **only in how the datagram
98+
travels**, which is exactly why the client side doesn't care which one you run.
99+
100+
## Recipe 1: networks that deliver broadcast — redis-broker-promoter
101+
102+
If your brokers and services share an L2 segment — bare-metal boxes, on-prem
103+
VMs, a Docker bridge network, your laptop — the simplest transport is UDP
104+
*limited broadcast*: one `sendto()` to `255.255.255.255` reaches every host on
105+
the segment, no inventory required. That's all
106+
[redis-broker-promoter](https://github.com/imqueue/redis-broker-promoter) does:
107+
108+
~~~bash
109+
git clone https://github.com/imqueue/redis-broker-promoter.git
110+
cd redis-broker-promoter && make # needs libuuid
111+
REDIS_BROADCAST_NAME=imq-broker \
112+
REDIS_BROADCAST_INTERVAL=1 \
113+
redis-server --port 6379 --loadmodule $PWD/promoter.so
114+
~~~
115+
116+
On load the module spawns one announcer thread per network interface allowed by
117+
your Redis `bind` configuration (`0.0.0.0` means all of them) and broadcasts
118+
`up` every interval to `255.255.255.255:63000` (`REDIS_BROADCAST_PORT`
119+
configurable). On shutdown it broadcasts `down`. Scaling out is now an
120+
operational no-op: start another `redis-server` with the module loaded, and
121+
every service adds it to the rotation within roughly one interval. Stop it, and
122+
the fleet shrinks just as automatically. That is horizontal **auto-scaling** of
123+
the broker layer — hook broker instances to whatever triggers your scaling
124+
decisions and the services follow along; nothing else needs restarting or
125+
reconfiguring.
126+
127+
A useful side effect of limited broadcast: routers never forward
128+
`255.255.255.255`, so announcements are confined to the local segment. That's
129+
the constraint that breaks this recipe in the cloud — and a small security
130+
property everywhere else.
131+
132+
## Recipe 2: networks that block broadcast — redis-broker-unicaster (Kubernetes on GCP and other clouds)
133+
134+
Cloud VPCs are software-defined networks, and most of them — GCP explicitly —
135+
do not deliver broadcast or multicast at all. A datagram to `255.255.255.255`
136+
in a GCP VPC or across a typical Kubernetes overlay simply vanishes, and the
137+
promoter recipe goes silent.
138+
139+
[redis-broker-unicaster](https://github.com/imqueue/redis-broker-unicaster)
140+
*emulates* broadcast instead of relying on it. Every interval it asks the
141+
Kubernetes API for the pods in its namespace and sends the very same datagram
142+
as plain UDP **unicast to each pod IP** at port `63000`. Pods that aren't
143+
listening drop it; pods running `UDPClusterManager` get exactly what they would
144+
have gotten from a broadcast. Newly scheduled service pods start receiving
145+
announcements within one interval — no service registry, no headless-service
146+
DNS, no multicast anywhere. Scale the broker Deployment up or down — by hand or
147+
with an autoscaler — and the fleet follows: the same horizontal auto-scaling as
148+
the broadcast recipe, minus the broadcast.
149+
150+
One boundary to be clear about: this recipe lives *inside Kubernetes* — the
151+
module authenticates with the pod's mounted service-account token and talks to
152+
`kubernetes.default.svc`. On cloud VMs outside Kubernetes, reach for the static
153+
`cluster` list instead (last row of the table below). The broker's service
154+
account needs permission to list pods:
155+
156+
~~~bash
157+
git clone https://github.com/imqueue/redis-broker-unicaster.git
158+
cd redis-broker-unicaster && make # needs libuuid, libcurl, json-c
159+
160+
# inside the broker pod — the module needs the mounted service-account token
161+
DEPLOYMENT_ENV=production \
162+
SELECTED_INTERFACES=10. \
163+
redis-server --port 6379 --loadmodule $PWD/unicaster.so
164+
~~~
165+
166+
- **`DEPLOYMENT_ENV`** — the Kubernetes namespace to enumerate pods in (and
167+
therefore the blast radius of the announcements). Announcements reach only
168+
this namespace, so brokers and every service or client that should discover
169+
them must run in the same one.
170+
- **`SELECTED_INTERFACES`** — comma-separated IP prefixes (e.g.
171+
`10.,192.168.`) selecting which local interfaces announce themselves; unset
172+
means all of them, loopback included, so set it in real deployments.
173+
- The RBAC side is a `Role` with `list` on `pods` plus a `RoleBinding` to the
174+
broker pod's service account.
175+
- In the current implementation the announce destination port is fixed at
176+
`63000`, so leave `UDPClusterManagerOptions.port` at its default on the
177+
service side.
178+
179+
## The service side is the same in both recipes
180+
181+
Whatever transport the announcements take, services and clients configure one
182+
thing. A pattern that has served well in production keeps a static fallback one
183+
environment variable away:
184+
185+
~~~typescript
186+
const DISABLE_CLUSTER_MANAGER = !!+(process.env.DISABLE_CLUSTER_MANAGER || 0);
187+
const cluster = (process.env.REDIS_CLUSTER || 'localhost:6379')
188+
.split(/\s*,\s*/)
189+
.map(cfg => {
190+
const [host, port] = cfg.split(/\s*:\s*/);
191+
return { host, port: +port };
192+
});
193+
194+
Object.assign(serviceOptions, DISABLE_CLUSTER_MANAGER
195+
? { cluster } // static list
196+
: { clusterManagers: [new UDPClusterManager()] } // discovery
197+
);
198+
~~~
199+
200+
One rule matters: **apply the same cluster options to every service and every
201+
client**. Requests and replies flow through the whole fleet, so a client pinned
202+
to a single broker will miss responses that round-robin landed elsewhere.
203+
204+
## Life of the fleet
205+
206+
- **A broker joins.** Discovered within about one announce interval; @imqueue
207+
starts its queue, replays subscriptions, and folds it into the rotation. If a
208+
service sends before *any* broker is known (cold start), the send waits for
209+
the first discovery for up to 30 seconds (`IMQ_SEND_INIT_TIMEOUT`) instead of
210+
failing.
211+
- **A broker leaves gracefully.** The module's shutdown hook emits `down` and
212+
removal is immediate. That's reliable at the default 1-second announce
213+
interval; at longer intervals shutdown can outrun the announcer thread, in
214+
which case removal falls back to heartbeat eviction.
215+
- **A broker crashes.** No `down` arrives; the missed-heartbeat eviction
216+
removes it a few seconds later. Messages already queued on it stay in its
217+
Redis (subject to your persistence settings) and become consumable again when
218+
it returns — the fleet keeps flowing through the remaining brokers meanwhile.
219+
- **Auth.** Give every broker the same credentials (one shared ACL file works
220+
well), because any service may connect to any discovered broker.
221+
- **Security.** The datagrams are plain, unauthenticated UDP — anyone who can
222+
reach the port can inject or evict brokers. Broadcast confines itself to the
223+
L2 segment; for the unicaster, keep UDP `63000` cluster-internal with a
224+
NetworkPolicy and treat the namespace boundary as the trust boundary.
225+
226+
## Picking a recipe
227+
228+
| Environment | Recipe |
229+
| --- | --- |
230+
| Bare metal, on-prem VMs, one L2 segment | promoter (broadcast) |
231+
| Docker bridge network, local development | promoter (broadcast) |
232+
| Kubernetes — on GCP or any cloud VPC | unicaster (K8s-API unicast) |
233+
| Cloud VMs outside Kubernetes, fixed topology | static `cluster` list, no modules |
234+
235+
The broker fleet is the piece that turns "we can scale the services" into "the
236+
whole system auto-scales". If you're starting fresh, the
237+
[getting-started guide](/get-started/) gets a service and client running in
238+
minutes; for how the discovery mindset extends to services themselves, see
239+
[do Node.js backends need service discovery?](/blog/do-nodejs-backends-need-service-discovery/)
240+
and [load balancing without a load balancer](/blog/load-balancing-microservices-without-a-load-balancer/).

0 commit comments

Comments
 (0)