Skip to content

Commit 556db3f

Browse files
committed
*: add wire-protocol.md
Document the wire protocol for assembling packets from frames. For now this only covers the packet assembly algorithm. The wire format of frames and packets is not specified and can be added later. Validation is split into stream constraints and connection constraints. Stream constraints will be common to both mux and non-mux managers, while connection constraints would differ between them.
1 parent 4b35ab8 commit 556db3f

1 file changed

Lines changed: 184 additions & 0 deletions

File tree

docs/wire-protocol.md

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Wire Protocol: Frame Handling
2+
3+
Frames on the wire look like `[stream S, msg M, kind K, done D]`. A packet is
4+
one or more frames that share the same stream and message ID. Each frame carries
5+
a chunk of data; as frames arrive, their data is appended together to assemble
6+
the packet. The packet is complete when the final frame has `done=true`.
7+
8+
Validation happens at two layers: **stream constraints** are per-stream and
9+
shared between mux and non-mux, while **connection constraints** are at the
10+
manager level and differ between mux and non-mux.
11+
12+
---
13+
14+
## Stream Constraints
15+
16+
These are enforced by the stream itself and apply the same way for both mux and
17+
non-mux managers. The stream keeps track of:
18+
19+
- `nextMessageID`: the minimum accepted message ID, bumped to `messageID + 1`
20+
when a packet completes.
21+
- `assembling`: whether frames are currently being accumulated into a packet.
22+
Set when the first frame of a packet arrives, cleared when `done=true`
23+
completes it.
24+
- `pktKind`: the kind of the packet being assembled, set by the first frame.
25+
26+
1. **Stream ID must match.** A misrouted frame is a protocol error.
27+
2. **Message ID must not go backwards.** If the incoming frame's message ID
28+
(`messageID`) is less than `nextMessageID`, that's a monotonicity violation.
29+
3. **New packet vs continuation.** A frame starts a new packet if:
30+
- `messageID > nextMessageID` (a higher message arrived; any in-progress
31+
packet data is discarded), or
32+
- `messageID == nextMessageID` and `assembling` is false (previous packet
33+
completed, or this is the first frame on the stream).
34+
35+
A frame is a continuation if `messageID == nextMessageID` and `assembling`
36+
is true.
37+
38+
Discarding in-progress data when a higher message ID arrives supports
39+
asynchronous interrupts, where the sender abandons a packet mid-write (e.g.,
40+
due to context cancellation) and moves to a new message. Message IDs do not
41+
have to be contiguous; a jump from message 1 to message 5 is valid.
42+
43+
4. **Kind must stay consistent within a packet.** Continuation frames (same
44+
message ID while assembling) must carry the same kind as the first frame of
45+
that packet.
46+
5. **Append frame data** to the in-progress packet buffer.
47+
6. **Done completes the packet.** `nextMessageID` is bumped to `messageID + 1`
48+
and the assembling flag is cleared. Any future frame with the same or lower
49+
message ID gets rejected by rule 2.
50+
51+
## Non-mux Connection Constraints
52+
53+
These live in the non-mux manager's reader loop. A future mux manager will have
54+
its own set of rules since interleaved streams are expected there.
55+
56+
1. **Global frame monotonicity.** Frame IDs on the wire must be non-decreasing.
57+
The manager tracks the last frame ID it saw, and any frame with a lesser ID
58+
is a protocol error.
59+
2. **Old-stream frames are silently ignored.** When an incoming frame's stream
60+
ID is less than the locally created current stream's ID, it gets dropped
61+
without error. This comes up on the client side where the local stream ID can
62+
advance before the remote finishes responding. See the examples for how this
63+
differs from rule 1.
64+
3. **First frame of a stream must be Invoke.** The first frame for a new stream
65+
has to be `KindInvokeMetadata` or `KindInvoke`. Anything else is a protocol
66+
error.
67+
68+
## Layering
69+
70+
Both the manager and the stream enforce ordering, but at different scopes.
71+
72+
The stream works at the per-stream level and is the authority on message-level
73+
correctness.
74+
75+
The manager (non-mux) works at the connection level. It catches cross-stream
76+
ordering violations and malformed stream starts. For invoke frames, which the
77+
manager consumes directly and never forwards to a stream, the manager's check is
78+
the only line of defense. Similar to the stream, manager also bumps the ID when
79+
a packet completes. This protection could be left for the streams as it also
80+
does the same, but it's needed at manager level too to protect against the
81+
replay of an Invoke message if a stream is not created by the time next frame
82+
arrives.
83+
84+
---
85+
86+
## Examples
87+
88+
Frames are shown as `[stream S, msg M, kind K, done]` where done is `d=t` (true)
89+
or `d=f` (false). Only relevant fields are included.
90+
91+
### Stream rule 2: message ID must not go backwards
92+
93+
```
94+
[s1, m3, Message, d=t] <- OK, packet complete, nextMessageID becomes 4
95+
[s1, m2, Message, d=t] <- error: m2 < nextMessageID(4)
96+
```
97+
98+
### Stream rule 3: new packet vs continuation
99+
100+
```
101+
[s1, m1, Message, d=f] <- start accumulating
102+
[s1, m1, Message, d=f] <- continuation, append
103+
[s1, m2, Message, d=f] <- m1 data silently discarded, m2 starts fresh
104+
```
105+
106+
### Stream rule 4: kind consistency within a packet
107+
108+
```
109+
[s1, m1, Message, d=f] <- start packet, pktKind=Message
110+
[s1, m1, Error, d=t] <- error: kind changed mid-packet
111+
```
112+
113+
Kind is only checked for continuation frames. Different messages can have
114+
different kinds without issue:
115+
116+
```
117+
[s1, m1, Message, d=t] <- OK, packet complete
118+
[s1, m2, Close, d=t] <- OK, new message, no kind check against m1
119+
```
120+
121+
### Stream rule 6: done prevents replay
122+
123+
```
124+
[s1, m1, Message, d=t] <- packet complete, nextMessageID becomes 2
125+
[s1, m1, Message, d=t] <- error: m1 < nextMessageID(2)
126+
```
127+
128+
### Stream rule 6: multi-frame then next message
129+
130+
```
131+
[s1, m1, Message, d=f] <- assembling=true, accumulate
132+
[s1, m1, Message, d=f] <- continuation, append (kind matches)
133+
[s1, m1, Message, d=t] <- packet complete, assembling=false, nextMessageID=2
134+
[s1, m2, Close, d=t] <- not assembling, new packet, no kind check, OK
135+
```
136+
137+
### Connection rule 1: global monotonicity
138+
139+
```
140+
[s1, m5, d=t]
141+
[s1, m4, d=t] <- error: {1,4} < lastFrameID {1,5}
142+
```
143+
144+
Cross-stream:
145+
146+
```
147+
[s1, m3, d=t]
148+
[s2, m1, d=t] <- OK, new stream
149+
[s1, m4, d=t] <- error: {1,4} < lastFrameID {2,1}
150+
```
151+
152+
Stream 2's frame implicitly starts a new stream. A frame for stream 1 showing up
153+
after that means the sender is writing to a stream it should consider closed.
154+
155+
### Connection rule 2: old-stream frames silently ignored (client side)
156+
157+
This handles a case that rule 1 does not catch. On the client side, the local
158+
stream ID advances independently of the wire:
159+
160+
```
161+
Client creates stream 1, sends invoke.
162+
Server responds:
163+
[s1, m1, Message, d=t] <- delivered to stream 1
164+
165+
Client finishes stream 1, creates stream 2 (curr.ID=2).
166+
Server still responding:
167+
[s1, m2, Close, d=t] <- s1 < curr(2), silently ignored
168+
```
169+
170+
Global monotonicity passes here (m2 > m1, both on stream 1). But the client has
171+
already moved on. No error is raised because the server doesn't know yet.
172+
173+
### Connection rule 3: first frame must be Invoke
174+
175+
```
176+
[s1, m1, InvokeMetadata, d=t] <- OK, forwarded
177+
[s1, m2, Invoke, d=t] <- OK, stream created
178+
[s1, m3, Message, d=t] <- OK, delivered to stream 1
179+
```
180+
181+
```
182+
[s1, m4, Message, d=t] <- OK, delivered to stream 1
183+
[s2, m1, Cancel, d=t] <- error: first frame is not Invoke
184+
```

0 commit comments

Comments
 (0)