Define message-size string accounting as UTF-8 byte length (with explicit string-length exceptions)

[sacOO7]

Problem

The client-side publish-size gate (RTO15d for ObjectMessage, TO3l8/maxMessageSize for Message) sums per-field sizes and rejects an over-limit publish before it hits the wire. The spec told SDKs to measure string fields by their "length", but never said whether "length" means UTF-8 bytes or UTF-16 code units. For ASCII these coincide; for non-ASCII they diverge (你 = 3 UTF-8 bytes / 1 UTF-16 code unit; 😊 = 4 UTF-8 bytes / 2 UTF-16 code units).

The spec was also internally contradictory:

• OM3d (extras): "the string length of its JSON representation" — reads as UTF-16 code units.
• OD3g (json): "the byte length of its JSON-encoded string representation" — explicitly bytes.
• OM3f / MCR3a1 / MST3c / MRM3a / OMP4a1 / OD3e / TM6f: bare "length" — undefined.

Because "length" was undefined, the SDKs diverged, and the same map key or clientId produced different gate results across SDKs. Concrete before-state (clientId = "émile", keys = "héllo👍", string data = "你好", extras = {"k":"你"}):

Field (spec point)	Value	cocoa (before)	java (before)	js (before)	After (all SDKs)
clientId (OM3f)	émile	6 (UTF-8)	6 (UTF-8)	5 (UTF-16)	6
MapSet key (MST3c)	héllo👍	10 (UTF-8)	10 (UTF-8)	7 (UTF-16)	10
map-state key (OMP4a1)	héllo👍	7 (UTF-16)	7 (UTF-16)	7 (UTF-16)	10
string data (OD3e)	你好	6 (UTF-8)	6 (UTF-8)	6 (UTF-8)	6
extras JSON (OM3d)	{"k":"你"}	9 (UTF-16)	9 (UTF-16)	9 (UTF-16)	9 (unchanged)

This surfaces the concrete gate-flip: a boundary ObjectMessage whose clientId/keys push it just over the limit is accepted by js (UTF-16 undercount) but rejected by cocoa/java (UTF-8) — the same publish, two verdicts. Two distinct defects:

1. Cross-SDK divergence. js under-counted clientId and operation keys (UTF-16) while cocoa/java counted them in UTF-8 bytes.
2. Intra-SDK inconsistency. cocoa/java measured the same map key two different ways depending on whether it arrived via an operation (MST3c, UTF-8) or via map state (OMP4a1, UTF-16).

The authoritative definition already exists in Ably's published billing/limits accounting — How is maximum message size measured? — verbatim:

The size is calculated as the sum of the name, clientId, data and extras properties before any compression or expansion occurs in the serialization process.

• name and clientId: "calculated as the size in bytes of their UTF-8 representation"
• data: "calculated as the size in bytes if it is in binary, or its UTF-8 byte length if it is a string"
• extras: "calculated as the string length of its JSON representation"
• binary on text transports: "the size is calculated using the actual size of the binary data, not its base64 encoded string"
• arrays: "the message size limit applies to the sum of all messages in the array"

Note the deliberate asymmetry in the published contract: name/clientId/string data are UTF-8 bytes, but extras is string length of its JSON representation.

Relationship to #331

This supersedes #331, the earlier attempt at the same disambiguation. This work substantially agrees with #331 — its conclusions and these changes converge on the same unit for every field both cover (string data, name, clientId, OD3e → UTF-8), and per-point coverage is documented below. Two deltas:

• extras deliberately follows the published docs (UTF-16 string length) where Message/PresenceMessage/Annotation size #331's discussion had converged on UTF-8. Measuring extras in UTF-8 would over-count (bytes ≥ units) and false-reject documented-valid messages — the worse failure direction. Message/PresenceMessage/Annotation size #331's UTF-8-for-extras was a hedged consistency preference ("technically a bug ... not one that matters much"), with no authoritative server-side statement that the server byte-counts extras. The docs are the currently-published enforced contract.
• This PR additionally disambiguates the LiveObjects map/operation keys (MCR3a1/MST3c/MRM3a/OMP4a1) and OD3g, which Message/PresenceMessage/Annotation size #331 predates, and uses a single canonical umbrella at TO3l8f instead of per-type duplication (addressing Message/PresenceMessage/Annotation size #331's "don't copy-paste size clauses per type" review feedback).

#331 is now CONFLICTING (the repo migrated textile/features.textile → specifications/features.md, so its diff no longer applies). It is left open for the author to close; this PR notes the supersession.

Solution

Match the service's published accounting per field. Every plain string and map key is measured in UTF-8 bytes; extras (and the regular-Message object/array data JSON) keep the documented "string length" (UTF-16 code units).

Component	Spec point(s)	Unit	Rationale
name (regular Message)	TM6a + TO3l8f default	UTF-8 bytes	docs: "size in bytes of UTF-8 representation"
clientId (regular + object)	(TM6a) / OM3f	UTF-8 bytes	docs verbatim
string data	TM6f	UTF-8 bytes	docs verbatim
binary data / bytes	TM6c / OD3c	raw bytes	actual binary, not base64
ObjectData.string	OD3e	UTF-8 bytes	same string family as values
ObjectData.json	OD3g	UTF-8 bytes	already byte-length; server counts bytes
ObjectData.number	OD3d	8	fixed
ObjectData.boolean	OD3b	1	fixed
operation keys	MCR3a1 / MST3c / MRM3a	UTF-8 bytes	keys are the same kind of string as values
map-state entry key	OMP4a1	UTF-8 bytes	changed from UTF-16 — consistency with operation keys
extras (regular + object)	TM6d / OM3d	UTF-16 string length of JSON	docs verbatim: "string length of its JSON representation"
object/array data (regular Message)	TM6b	UTF-16 string length after JSON-stringify	pre-existing convention; unchanged

Per-clause edits to specifications/features.md:

• → UTF-8 byte length: TM6f (string data), OM3f (object clientId), MCR3a1, MST3c, MRM3a, OMP4a1 (all map/operation keys), OD3e (string), OD3g (json — clarified "byte" → "UTF-8 byte").
• → explicit UTF-16 string length: TM6b (object/array data), TM6d and OM3d (extras) — reworded from bare "string length" to "string length (the number of UTF-16 code units)".
• Canonical umbrella at TO3l8f: one sentence establishing the default (all message-size string accounting is UTF-8 byte length unless a clause states otherwise, explicitly including the json OD3g payload) and enumerating the two UTF-16 exceptions (extras TM6d/OM3d and object/array data TM6b). Deliberately one umbrella rule rather than per-field/per-type duplication.

Wording-only change; no test additions in this repo.

SDK status

SDK	Sites	Fix	Status
ably-js	objectmessage.ts (clientId/OM3f, OMP4a1, MCR3a1, MST3c, MRM3a → dataSizeBytes); message.ts getMessageSize (name+clientId → Utils.dataSizeBytes)	UTF-8 for keys/clientId/name; extras stays JSON.stringify(...).length	companion PR: ably/ably-js#2289
ably-cocoa	ObjectMessage.swift ObjectsMap.size (OMP4a1: utf16.count → utf8.count — the one behavioural change); ARTBaseMessage.m core-Message alignment	UTF-8 keys; extras/object-array data UTF-16	changes ready on feature/liveobjects-implementation
ably-java	WireObjectMessage.kt WireObjectsMap.size (OMP4a1: .length → .byteSize)	UTF-8 (java was the UTF-8 reference impl); extras UTF-16	changes ready on refactor/uts-objects-unit-into-liveobjects

Notes

• The extras/UTF-16 rationale. The published accounting has an explicit rule for extras ("string length of its JSON representation") distinct from the byte rule for other strings. Matching it exactly (UTF-16 code units) makes the gate neither false-accept nor false-reject extras relative to the documented contract. UTF-8 would over-count and false-reject documented-valid messages. This is a deliberate documented exception, not an inconsistency; all three SDKs already agreed on UTF-16 here.
• Why keys/strings are UTF-8 (strict-dominance). For any string, UTF-8 byte length ≥ UTF-16 code-unit count (equality only for ASCII). Measuring a key in UTF-8 bytes equals the server's on-the-wire byte count, so the gate never false-accepts a key the server would reject. UTF-8 strictly dominates UTF-16 for keys.
• name/clientId for regular Message have no explicit per-field sub-clause in TM6; they are covered by the TO3l8f umbrella default (UTF-8).
• TP5 (PresenceMessage) needed no edit — it already delegates wholesale to TM6, and RSAN1a4 (annotations) delegates to TO3l8, so presence and annotation sizing inherit the rule with no separate edits.

[Copilot]

Pull request overview

Clarifies the specification’s message-size accounting rules by defining how to measure string sizes (defaulting to UTF-8 byte length) and explicitly documenting the UTF-16 code-unit exceptions, to eliminate cross-SDK divergence and internal inconsistencies.

Changes:

• Defines string data, clientId, and LiveObjects map key sizing as UTF-8 byte length (e.g., TM6f, OM3f, MCR3a1, MST3c, MRM3a, OMP4a1, OD3e, OD3g).
• Makes the UTF-16 code-unit basis explicit for JSON-stringified extras and object/array data sizing (TM6b, TM6d, OM3d).
• Adds an umbrella rule at TO3l8f defining the default and enumerating the exceptions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.