topology_hiding: keep the dialog-less state in a shared store (fixes long Contact / SUBSCRIBE)

[Lt-Flash]

Problem

When topology hiding runs without a dialog, the whole encoded state (route set, original Contact, flags, bind address) rides in the Contact URI parameter — making it long. Some UAs (e.g. Cisco CP-88xx) truncate it, after which topology_hiding_match() can't decode it and every sequential request of that call is lost. This is unavoidable for non-INVITE dialogs like SUBSCRIBE: dlg_create_dialog() is INVITE-only, so a SUBSCRIBE can never get a dialog to hold its state server-side, and the payload is dominated by the route set.

What this adds

An opt-in mode for the dialog-less path: when th_state_url points at a cachedb backend, the state is stored there under a short key and only the key travels in the Contact:

stateless:  ;thinfo=VG9JbzAdIFsyPz0MNS9yRmdPY31kQWFVd0djUGtSVF9CbzYQIFtiYWNBYEFzWmFbZmZiWQ--   (72 chars)
stateful:   ;thinfo=_9ea80d5443fe5c5a                                                           (17 chars)

The Contact stays short regardless of topology size, making SUBSCRIBE (and long route sets) workable on constrained endpoints. When th_state_url is unset, the encoding path is byte-for-byte unchanged. A call that does get a dialog never reaches this mode — the dialog holds its state and puts only the dialog id in the Contact, so force_dialog=1 is the better answer for calls; this mode serves what a dialog cannot.

The full detail — the marker-safety proof, the notifier-granted-expires handling, the measured TTLs, and the per-backend matrix — is in the module docs (admin.xml / README). This is the summary.

Design

• In-Contact path unchanged. A state that still rides in a Contact is XOR-obfuscated + word-encoded as before; a stored state skips both (pointless off the wire) and is kept as plain length-prefixed text. Decode tells them apart by a marker on the key; the restore logic is shared, not duplicated.
• The state is immutable (route set / contact / socket are fixed at call creation), so a plain KV store with a TTL is the right fit — no update or conflict path.
• One key per dialog leg, not per request. The key is derived (not random) from the leg it belongs to — the call-id + that party's own tag, keyed with the contact-encoding password — so every re-encode overwrites the same key. A live dialog holds exactly two states (one per leg) however often it refreshes; the password keeps the key unguessable from the clear-text call-id/tags. A message that cannot be keyed — no call-id, or no tag, as with the reply of an out-of-dialog transaction (a 200 OK to an OPTIONS keepalive carries a Contact but no To tag) — is simply not stored: its state stays inline in the Contact, and it has no sequential traffic to be matched against anyway.
• Coalesced writes. Since the state is immutable, a re-store only bumps the TTL. Each node keeps a small shm record of the expiry it last wrote per key and skips the backend write while the state still has enough life left — collapsing a busy dialog's identical writes into occasional refreshes. A write that would shorten a state (teardown) is never skipped.
• Availability. A sequential request may land on any node, so the backend must be shared (e.g. cachedb_redis); cachedb_local is fine for a single instance. The storage sits behind a small th_store.h interface (backend enum), so a clusterer-replication backend can be added later without touching the encode/decode sites.

State lifetime

A state is stored every time a Contact is encoded, but the deterministic key overwrites in place (no pile-up), and each write takes its TTL from the message, not a flat value:

Message	State lifetime
SUBSCRIBE / NOTIFY	subscription expires + 30 s (NOTIFY's granted Subscription-State wins, else Contact ;expires, else Expires)
expires 0 / Subscription-State: terminated	matched state dropped now; the outgoing Contact gets th_state_ttl_short (routes the final NOTIFY)
OPTIONS / MESSAGE / INFO / PUBLISH	th_state_ttl_short (default 60) — transaction-only, no target refresh
INVITE / UPDATE with session timers	Session-Expires + 30 s (the Expires ring timeout is deliberately not used)
INVITE / UPDATE without session timers	not stored — stays inline in the Contact (a call bounded by nothing must not be able to expire)
any message that cannot be keyed (no call-id/tag, e.g. an OPTIONS 200 OK with no To tag)	not stored — stays inline in the Contact
anything else	th_state_ttl (default 3600)

A message that carries its dialog's lifetime also refreshes the state it was sent to, in place — so a long-lived subscription that only ever re-SUBSCRIBEs doesn't lose the state it keeps using. An in-dialog OPTIONS does not refresh (its short TTL would otherwise cut a call to a minute). All of this is measured end-to-end against Redis; the full TTL / refresh / notifier-granted tables and the _-marker safety proof are in the docs.

Security

The stored state is in the clear — not encrypted, and (unlike the in-Contact copy) not obfuscated — so anyone who can read the backend sees the topology this hides (real Contact, route set, receiving socket). This is intentional: off the wire the obfuscation/encoding serve no purpose, and plain text is inspectable. The store is expected to be a trusted, internal backend reachable only by the OpenSIPS nodes that share it — deploy it on a private / DMZ network, bind it to the internal interface, enable the backend's own auth, and do not point th_state_url at anything reachable beyond that trust boundary.

Backend compatibility

Any cachedb backend that honours the expires argument of set() (and provides remove()) works — verified against cachedb_redis and cachedb_local, and applies to memcached / cassandra / couchbase / dynamodb / sql. cachedb_mongodb silently ignores the TTL (states would never expire), so it is rejected at startup by a URL-scheme check (the cachedb interface has no "supports expiration" capability — a general gap left out of this PR).

Configuration

loadmodule "cachedb_redis.so"
loadmodule "topology_hiding.so"
modparam("topology_hiding", "th_state_url", "redis://127.0.0.1:6379/0")  # shared, TTL-expiring backend
modparam("topology_hiding", "th_state_ttl", 3600)        # calls with no bound + fallback
modparam("topology_hiding", "th_state_ttl_short", 60)    # OPTIONS/MESSAGE/INFO/PUBLISH

route {
    if (has_totag()) {
        if (!topology_hiding_match())
            xlog("L_WARN", "no topology hiding state for $ru\n");
    } else {
        record_route();
        topology_hiding();   # works for SUBSCRIBE, no dialog needed
    }
}

cachedb_local:// (single instance) and a password-protected redis://:pass@host/db cluster are also supported.

Testing

Against cachedb_redis (and cachedb_local, confirming backend-agnostic):

• A dialog-less SUBSCRIBE gets its state stored; the Contact param shrinks 72 → 17 chars (a 16-hex key plus its _ marker); a sequential request carrying only the key is matched, decoded, and the R-URI restored to the original endpoint.
• Stateless mode unchanged — same 72-char Contact, nothing written.
• Per-message TTLs measured correct (subscription expires, session timers, short-TTL for OPTIONS, teardown removal, notifier-granted correction); the mongodb:// guard refuses startup.

Roadmap (not in this PR)

• Clusterer replication backend — keep states in shm and replicate to peers (mirroring dlg_replication.c), for deployments that don't want an external store in the signalling path; the immutable state makes it simpler (create / read / expire only).
• A CACHEDB_CAP_EXPIRES capability so TTL-ignoring backends are rejected through the interface rather than by a URL-scheme match (touches the core plus every backend).

Notes

• New files th_store.c / th_store.h; docs + regenerated README included. th_state_url declares a cachedb_url dependency so the backend module loads first.

---

Measured cost

The design section above says the state backend is a choice (cachedb_local for a single instance, a
shared store otherwise). That choice has a concrete price, so here it is measured, together with the
alternatives it competes with.

Test environment

Host	KVM guest, 8 vCPU
CPU	Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (Broadwell), hypervisor flag present
Memory	7.8 GB
OS / kernel	Debian 13 (trixie) / 6.12.96-amd64
OpenSIPS	4.0.0 — STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, F_PARALLEL_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
Runtime	-m 4096 shm, -M 32 pkg, udp_workers=8 (= vCPU count), tcp_workers=4
Generator	SIPp 3.6.0, 4 parallel UAC processes; SIPp UAS pool selected via dispatcher (weighted round-robin)

What was exercised

A dual-homed proxy: upstream leg and downstream pool on separate interfaces. Every call is a complete
INVITE -> 100 -> 200 -> ACK -> BYE -> 200 carrying Session-Expires, so a storable state is produced
on every call — deliberately the heaviest path for this feature. topology_hiding() runs on the
initial INVITE and topology_hiding_match() on every sequential request; a call is only counted as
successful if its BYE was matched and answered, so the decode path is verified, not just the encode.

Two measurement notes for anyone reproducing this: CPU must be sampled over a steady-state mid-run
window (including the call-teardown tail dilutes it badly — 46% vs the real 67% in one run), and a
single SIPp UAC process saturates one core at roughly 4.8k CPS, so the load has to be generated from
several processes or you measure the generator instead of the proxy.

UDP — what the state backend costs

Backend	CPU @3k CPS	CPU @6K CPS	Ceiling
no TH (plain record-route)	8 %	~47 %	~10 000 CPS
TH + dialog (force_dialog=1)	10 %	~73 %	~9 000 CPS
TH + cachedb_local	7 %	77 %	~7 100 CPS
TH + cachedb_redis, loopback	48 %	80 %	~6 900 CPS
TH + cachedb_redis, 3-master LAN cluster	54 %	—*	~5 300 CPS

* the LAN-cluster case never becomes CPU-bound — it caps at ~65%, see findings.

Findings (UDP)

1. The encode/decode is not what costs — the backend round-trip is. Holding the same state in a
   dialog runs at 10% CPU at 3k CPS; putting it through a loopback Redis costs 48% for identical calls,
   and the ceiling falls from ~9k to ~6.9k CPS. The per-call SET/GET dominates, not the topology
   hiding itself.
2. cachedb_local is the cheapest option at low/medium load but walls early. At 3k CPS it is the
   lowest of all five (7%), because it avoids both the socket round-trip and dialog bookkeeping. Then CPU
   goes 9% -> 77% -> 100% between 4k and 7.1k CPS: the shared cache hash serialises the workers, so it
   caps ~2k CPS below the dialog path despite being cheaper per call. A good single-instance choice up
   to a few thousand CPS; not the one to pick for peak throughput.
3. A remote store is latency-bound, not CPU-bound. The LAN cluster caps at ~5.3k CPS while the proxy
   still has ~35% CPU idle. The cachedb call is synchronous, so a worker blocks for the whole round-trip
   and stops draining its socket — every extra millisecond of backend RTT comes straight off the
   ceiling. Colocate the store where possible; this is also exactly what the coalesced-write path is
   worth the most for.
4. These are worst-case numbers for this feature. Session-Expires on every call means a state was
   stored for every call. Real traffic has far fewer storable messages, so production cost sits between
   the no-TH and TH lines — nearer the former the fewer states are actually stored. Note also that
   each call used a distinct key, so write coalescing is not what these figures measure.
5. The load is ~50/50 user/system CPU. At these rates a UDP proxy is as much kernel/packet-path bound
   as application bound; all workers share one UDP socket, which is why even the no-TH line plateaus
   around 83-97% rather than pinning all 8 cores.
6. Worker count should track core count. Raising udp_workers 8 -> 32 on the same 8 vCPU made things
   worse: ~1.6x the CPU for equal throughput and 2.5% call loss at only 1000 CPS (shared-socket
   contention plus scheduler churn), for a ~13% ceiling gain.

TCP — connection ceiling

Not specific to this PR — transport characterisation of the same box, included for completeness.
Connections were opened at a steady ~76/sec and held:

Findings (TCP)

1. Simultaneous TCP connections cap at tcp_max_connections (default 2048). At that point OpenSIPS
   logs maximum number of connections exceeded and refuses new connections. The failure is graceful:
   existing connections and UDP signalling keep working — only new TCP is rejected.
2. A much lower limit bites first for bursts: tcp_socket_backlog defaults to 10. The accept queue
   overflows on rapid arrivals (listen queue of a socket overflowed in netstat -s) long before 2048 is
   approached; reaching the real ceiling at all required raising it. For a public-facing edge proxy a
   backlog of 10 is easy to overrun with a reconnect storm or a burst of TLS handshakes — this may be
   worth revisiting as a default.
3. Kernel SYN handling was not implicated. No Possible SYN flooding messages, and
   tcp_max_syn_backlog (512) is far above the 10-deep accept queue; the constraint is purely the
   application's listen() backlog. File-descriptor limits were also not a factor (65536 available).

---

Under concurrent load: 50 000 calls held

The figures above ramp the call rate with calls torn down immediately, which measures the cost of
establishing state. This second test measures the cost of holding it: the hold time is set to
50000 / rate, so once the ramp completes there are ~50 000 calls in flight, and the load is then
sustained. Concurrency is verified rather than assumed - it is the summed in-flight call count reported
by the load generators, and it lands within 0.3% of 50 000 on every valid rung.

These numbers are not directly comparable to the section above. That test failed any call whose
response took over 8 s; this one allows 120 s, because calls are held deliberately. The proxy therefore
posts higher call rates here at considerably worse latency. They are two different measurements, not a
revision of the same one.

A rung counts only where the proxy actually kept pace: under 5% failures and in-flight calls still
near 50 000. That second condition matters. Judged on failure rate alone the loopback-Redis case appears
to reach 9 829 CPS at 1.7% failures - but its in-flight count had grown to 112 000, meaning calls were
arriving faster than they cleared. That is a growing backlog, not 50 000 calls being held, so it is not
counted.

Backend	Sustained CPS	at CPU	in-flight	CPU @4 000 CPS	Memory @4 000 CPS
no TH (plain record-route)	11 788	93 %	64 k	8 %	870 MB
TH + dialog (force_dialog=1)	9 875	90 %	54 k	17 %	1 203 MB
TH + cachedb_redis, loopback	6 000	80 %	56 k	57 %	924 MB
TH + cachedb_local, th=16	5 944	89 %	51 k	29 %	1 014 MB
TH + cachedb_local, default	5 907	97 %	52 k	45 %	913 MB
TH + cachedb_redis, 3-master LAN cluster	3 996	57 %	52 k	57 %	918 MB

Findings (50 000 held calls)

1. Holding state in a dialog is the strongest option once concurrency is high. 9 875 CPS against
   ~6 000 for every cachedb variant, and 17% CPU against 45-57% for identical traffic at 4 000 CPS. The
   ranking is the same as in the first test, but the gap widens sharply with in-flight state.
2. A remote store degrades furthest. The LAN cluster sustains 3 996 CPS and never exceeds 57-63% CPU
   • roughly a third of the machine stays idle while workers block on the synchronous round-trip. Backend
     RTT, not proxy CPU, sets that ceiling, which is the case the coalesced-write path helps most.
3. cachedb_local's default collection is undersized for tens of thousands of live states. The
   default is 1 << HASH_SIZE_DEFAULT = 512 buckets, so ~50 000 entries means ~100-entry chains walked
   under each bucket lock. Setting modparam("cachedb_local", "cache_collections", "th=16") (65 536
   buckets) cut CPU from 45% to 29% at 4 000 CPS. It barely moved the ceiling (5 907 -> 5 944), so it
   buys headroom below saturation rather than a higher wall - but it is a one-line change and worth
   documenting for anyone using cachedb_local to hold TH state at scale.
4. Dialog pays for its CPU advantage in memory: 1 203 MB at 4 000 CPS versus ~915 MB for the cachedb
   variants, growing to 2 304 MB by 10 000 CPS. Holding 50 000 calls costs roughly 350 MB of shared
   memory even with no topology hiding at all, from the in-flight transactions alone.
5. Failure counts alone are a misleading ceiling metric under sustained concurrency. Three of the six
   configurations post low failure rates at rungs where in-flight calls had doubled or tripled past the
   target. Any figure quoted from this kind of test should be paired with the in-flight count.

[Lt-Flash]

Hi, whenever you have a moment, could I kindly ask for a review of this PR? I'd really appreciate it.

I'll hold off on any further commits until the review is complete, so the branch stays stable for you to look at.

Thanks very much for your time!