Skip to content

Commit ea52d0a

Browse files
jfb8856606jfb8856606
committed
docs: add MTU change support blog articles (Chinese + English)
1 parent ba34f04 commit ea52d0a

2 files changed

Lines changed: 477 additions & 0 deletions

File tree

‎docs/mtu_change_spec/mtu_change_support.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
F-Stack 2.0 Preview 7: MTU Change Support — Breaking the 1500 Hard Limit for 9000 Jumbo Frames, and the Two Pitfalls Along the Way
2+
3+
1. What this feature does and its key characteristics
4+
5+
Let's get straight to the point: F-Stack's DPDK-controlled NIC was previously locked at MTU 1500. This project turned the status quo — "decreasing MTU works, increasing MTU (jumbo frames) is completely unsupported" — into full support: set any MTU between 1500 and 9000 at startup via configuration, change it at runtime through the standard SIOCSIFMTU ioctl, with the protocol stack and DPDK hardware kept in sync.
6+
7+
First, what the pre-change baseline looked like. A conclusive research was done in mid-July 2026 (16 documents, `docs/mtu_change_spec/zh_cn/00~15`); the three-way cross-verified conclusion was "partial support":
8+
9+
- Decreasing MTU (≤1500): works out of the box. `ff_ioctl(SIOCSIFMTU)` goes through FreeBSD `ether_ioctl`, which writes values ≤ ETHERMTU directly into `if_mtu`.
10+
- Increasing MTU (>1500, jumbo): double-blocked. At the protocol-stack layer, `ether_ioctl` hardcodes `EINVAL` for `ifr_mtu > ETHERMTU(1500)`; at the DPDK hardware layer there is no MTU wiring at all — no `rte_eth_dev_set_mtu` call, the mbuf pool is fixed at `RTE_MBUF_DEFAULT_BUF_SIZE` (2048 usable dataroom), and `rxmode` has no jumbo/scatter configuration.
11+
12+
This conclusion matched F-Stack official issues #239/#490/#720 exactly: the jumbo support issue had been OPEN for a long time, and the maintainers explicitly replied "mtu cannot exceed 1500". So this project was not about applying patches — it was about connecting three broken layers: the protocol stack, the DPDK hardware layer, and the configuration system. The three keywords of this rework:
13+
14+
- **Software-hardware linkage**: one MTU with three views — the protocol-stack `if_mtu`, the DPDK port MTU, and the mbuf pool capacity. All three must agree; startup fails if any mismatches.
15+
- **Multi-process division of labor**: the DPDK physical port is shared state; the primary alone owns the hardware (sets both soft and hard), while secondaries set only their own process's software MTU — zero IPC between processes.
16+
- **Zero-regression commitment**: the new `mtu_enable` master switch; when disabled, legacy configs keep the exact MTU 1500, 2176B mbufs, and ioctl behavior.
17+
18+
Scale numbers: 16 research documents (landed 2026-07-17) → 7 implementation spec documents (07-21) → M1~M5 code implementation, 14 commits done in one day (07-21) → physical-machine validation + English translation (07-22). 59 unit tests passed (including 8 new MTU tests); IPv4/IPv6 8500-byte jumbo frames verified bidirectionally on a physical machine.
19+
20+
2. Main applicable scenarios
21+
22+
2.1 Large-packet throughput optimization
23+
24+
This is the most direct motivation. The historical conclusion of issue #1033 already noted that the performance drop in large-packet scenarios is partly caused by IP fragmentation forced by MTU 1500. With jumbo frames enabled, an 8500-byte packet is no longer split into 6 fragments, and fragmentation/reassembly overhead drops to zero. Suitable for internal high-speed networks, large data-block transfers, and storage networks where the link MTU can be unified.
25+
26+
2.2 Tunnel/overlay scenarios that need to flexibly reduce MTU
27+
28+
Reducing MTU always worked, but after this rework all runtime changes go through the standard SIOCSIFMTU semantics — a single `ff_ifconfig f-stack-0 mtu <N>` changes both the protocol stack and the hardware. VXLAN/GRE/tunnel overlay scenarios often need MTU pushed down to the 1400 range to leave room for tunnel headers.
29+
30+
2.3 Unified jumbo frames in multi-process production deployments
31+
32+
Under the classic 1 primary + N secondary deployment, each process sets MTU once to stay consistent (primary manages hardware, secondaries manage their own software views) — no cross-process coordination mechanism needed.
33+
34+
2.4 Scenarios that are not a good fit
35+
36+
- The link peer does not support jumbo: PMDs like virtio have limited jumbo capability, and the underlying NIC/vSwitch/peer must all support it; otherwise large frames get dropped or fragmented mid-path — code support is useless if the link doesn't follow.
37+
- Deep MTU linkage between KNI and the kernel stack: KNI veth interface MTU linkage is out of scope for this phase (only verified that both work independently after the mutual exclusion was lifted).
38+
- Transaction-level cross-process MTU consistency: this phase explicitly skips IPC coordination; a process that never sets MTU keeps its old software value — a known usage constraint.
39+
40+
3. Architectural characteristics
41+
42+
3.1 The three broken layers before the rework
43+
44+
The root-cause diagram drawn during the research phase — the starting point for understanding every change:
45+
46+
```
47+
┌────────────────────────────────────────────────┐
48+
│ Application: ff_ifconfig f-stack-0 mtu 9000 │
49+
│ ff_ioctl(SIOCSIFMTU) │
50+
└───────────────────┬────────────────────────────┘
51+
▼
52+
┌────────────────────────────────────────────────┐
53+
│ Protocol stack (ff_veth.c + trimmed FreeBSD) │
54+
│ ff_veth_ioctl → ether_ioctl │
55+
│ if (ifr_mtu > ETHERMTU) → EINVAL ← hardcoded 1500│
56+
│ if_mtu written only to ifnet, no HW linkage │
57+
└───────────────────┬────────────────────────────┘
58+
▼（gap: even success is not pushed to HW）
59+
┌────────────────────────────────────────────────┐
60+
│ DPDK hardware layer (ff_dpdk_if.c) │
61+
│ No rte_eth_dev_set_mtu call │
62+
│ rxmode has no mtu/max_rx_pkt_len/jumbo config │
63+
│ mbuf pool fixed 2048 dataroom ← can't fit big frames│
64+
└────────────────────────────────────────────────┘
65+
```
66+
67+
3.2 The SIOCSIFMTU path after the rework (split by process role)
68+
69+
```
70+
ff_ifconfig mtu <N>
71+
│
72+
ff_veth_ioctl(SIOCSIFMTU) ← intercepted, no longer delegates to ether_ioctl
73+
│
74+
rte_eal_process_type() role check
75+
┌──────────┴──────────┐
76+
▼ ▼
77+
primary secondary
78+
ff_dpdk_if_set_mtu() if_setmtu() software-only
79+
① -EBUSY → stop port │（never touches DPDK port control APIs）
80+
② rte_eth_dev_set_mtu │
81+
③ rte_eth_dev_get_mtu readback│
82+
④ rollback on failure: │
83+
restore old MTU │
84+
+ restart port │
85+
⑤ on success → if_setmtu(ifp)│
86+
⑥ if_notifymtu(ifp) │
87+
├ nd6_setmtu (IPv6 sync) │
88+
└ rt_updatemtu (route sync)│
89+
```
90+
91+
Key design constraint: `ff_veth.c` does not include `rte_ethdev.h`; all hardware operations go through the opaque interfaces `ff_dpdk_if_get_mtu/set_mtu/get_mtu_capability` in `ff_dpdk_if.h`, and DPDK negative errno values are converted to BSD positive errno via `ff_dpdk_errno_to_bsd()`.
92+
93+
3.3 The two mbuf carrying modes
94+
95+
```
96+
large mode scatter mode
97+
┌──────────────────────────┐ ┌──────────┐┌──────────┐
98+
│ single mbuf, data_room │ │ standard ││ standard │→ ...
99+
│ sized by HEADROOM+ │ │ mbuf ││ mbuf │
100+
│ max_mtu+L2_overhead │ │ 2048B ││ 2048B │
101+
└──────────────────────────┘ └────┬─────┘└────┬─────┘
102+
more memory, simpler path │ RX_OFFLOAD_SCATTER
103+
│ multi-seg chaining
104+
▼
105+
less memory, but requires PMD
106+
scatter + multi_segs support,
107+
and the multi-seg mbuf
108+
conversion path must be verified
109+
```
110+
111+
In large mode `data_room_size = align(HEADROOM + max_mtu + L2_overhead)`; exceeding UINT16_MAX must fail deterministically, so `max_mtu=65535` is unavailable in large mode. Scatter mode keeps standard mbufs, configurable up to 65535 but subject to PMD capability.
112+
113+
4. What was reworked and what problems were hit
114+
115+
The rest is a bit dry; skip this section if you don't need the implementation details and jump straight to Section 5.
116+
117+
4.1 Research conclusions turned directly into requirement decisions (D-MTU-01~06)
118+
119+
The research wrapped up with 6 decisions, all carried through the implementation:
120+
121+
| ID | Decision |
122+
|---|---|
123+
| D-MTU-01 | Implement both large and scatter mbuf modes, selected by configuration |
124+
| D-MTU-02 | `max_mtu` is configurable; defaults to 9000 when the feature is enabled |
125+
| D-MTU-05 | **Do not modify the freebsd/ tree**; intercept SIOCSIFMTU in lib/ff_veth.c to bypass ether_ioctl's 1500 hard check |
126+
| D-MTU-06 | With `mtu_enable=0`, behavior stays identical to the old version (MTU 1500, 2176B mbufs, ioctl unchanged) |
127+
128+
D-MTU-05 deserves one more sentence: the ETHERMTU hard check in `ether_ioctl` is a general FreeBSD semantic. F-Stack's choice was not to change it inside the trimmed stack, but to intercept SIOCSIFMTU in the ff_veth driver layer and handle it there. This keeps the freebsd/ subtree untouched — no new patch maintenance burden when the FreeBSD baseline is upgraded.
129+
130+
4.2 Code implementation (M1~M5, 14 commits in one day, 2026-07-21)
131+
132+
| Milestone | commit | Content |
133+
|---|---|---|
134+
| M1 | 97452db34 | Config parsing & validation: enum ff_mbuf_mode, ff_port_cfg.mtu, dpdk.{mtu_enable,max_mtu,mbuf_mode}; strict strtoul parsing; cross-field validation; 8 unit tests + 7 fixtures |
135+
| M2 | eec178902 | DPDK layer: ff_mtu_data_room_size() alignment, large-mode data_room sizing, rxmode.mtu + PMD capability check + scatter offload, set/get_mtu readback |
136+
| M3 | 0849f9f3a | ff_veth integration: opaque MTU API, DPDK errno→BSD errno conversion, SIOCSIFMTU interception (primary soft+hard / secondary soft only), initial if_mtu read from hardware at startup |
137+
| M4 | ef20b1abf | EBUSY state machine: -EBUSY → stop/set_mtu/get_mtu/start; rollback restore old MTU + restart port on failure |
138+
| M5 | 314df8a0a | Integration tests + architecture doc updates |
139+
| Wrap-up | 0f8f6991e / 332abf997 / 4c30d118f | magic numbers → named macros, SIOCSIFMTU handler fall-through dedup, bool→uint8_t clean-build fix |
140+
141+
One iron rule for config parsing: no `atoi()`. The new `ff_parse_u16/ff_parse_mbuf_mode` do strict parsing (strtoul + errno/endptr/range checks); any illegal configuration fails loudly at startup instead of silently falling back to 1500. `mtu_enable=0` combined with `mtu>1500` fails parsing outright with a hint to enable the feature first.
142+
143+
4.3 Problem 1: IPv6 jumbo replies fragmented at 1448 bytes (the biggest pitfall)
144+
145+
During physical-machine validation after the feature went live, a bizarre symptom appeared: **IPv4 jumbo worked bidirectionally, but IPv6 received an 8500-byte ping fine while the reply was split into 6 fragments of 1448 bytes**.
146+
147+
Back-calculation from the capture: 6 fragments = 1448×5 + 1268 = 8508, matching the fragmentation formula `len = (mtu - 40 - 8) & ~7`, which solves to mtu=1500 — the stack had if_mtu set to 9000, yet IPv6 fragmentation still used MTU 1500.
148+
149+
The root-cause chain spans 12 file:line references (full analysis in `15-ipv6-jumbo-frame-fragmentation-analysis.md`); the one-sentence summary:
150+
151+
```
152+
ether_ifattach(if_mtu=1500) → nd6_ifattach → nd6_setmtu0(ndi->maxmtu=1500)
153+
↓
154+
ff_veth if_setmtu(9000) → if_mtu=9000, but [nd6_setmtu not called] → ndi->maxmtu stuck at 1500
155+
↓
156+
IN6_LINKMTU = min semantics → maxmtu(1500) < if_mtu(9000) → returns 1500
157+
↓
158+
route nh_mtu=1500 → ip6_getpmtu → ip6_calcmtu → fragment len=(1500-48)&~7=1448
159+
```
160+
161+
The real pitfall: `if_setmtu()` only writes `ifp->if_mtu` — **it does not notify protocol families**. The standard FreeBSD kernel path `ifhwioctl → if_setmtu → if_notifymtu → nd6_setmtu + rt_updatemtu` synchronizes IPv6's `ndi->maxmtu` and the route MTU, but F-Stack took the "call if_setmtu directly" shortcut to bypass ether_ioctl's 1500 hard check and dropped the `if_notifymtu` step. The IPv4 output path judges fragmentation with `ifp->if_mtu` directly, so it was unaffected; IPv6 goes through IN6_LINKMTU and got hit.
162+
163+
The fix (commit 0f25ac495, +163/-1): both paths in `ff_veth.c` — startup initialization and runtime SIOCSIFMTU — now call `if_notifymtu(ifp)` after `if_setmtu`, aligning with standard kernel semantics. Physical-machine retest: `ping6 -M do -s 8500` replies are no longer fragmented; IPv6 jumbo works bidirectionally.
164+
165+
[Note 1] This pitfall has general value: in FreeBSD, "changing if_mtu" and "notifying protocol families" are two separate things. Any code that bypasses the standard ifhwioctl path and calls if_setmtu directly (driver init, ioctl interception) must add if_notifymtu itself; otherwise IPv6's nd_ifinfo and route nh_mtu stay at the old values forever. Incidentally, the secondary cause — a router advertisement lowering linkmtu — was also investigated; runtime verification confirmed linkmtu=0 (not affected by RA), so the primary-cause fix closed the loop.
166+
167+
4.4 Problem 2: the KNI/MTU mutual exclusion — banned first, lifted later
168+
169+
The initial requirement R-MTU-009 agreed: when `mtu_enable=1` and KNI/kernel coexistence are both enabled, reject at configuration time with EOPNOTSUPP — the rationale being the risk of dual-stack MTU inconsistency.
170+
171+
Physical-machine testing overturned this conservative decision (commit 989f1d2da, +1/-6): `mtu_enable=1` coexisting with `kni.enable=1` works perfectly — with veth0 MTU=1500 the kernel stack sends and receives normally with defragmentation, and with `ifconfig veth0 mtu 9000` the KNI path handles jumbo frames fine too. The exclusion was an unnecessary restriction; after lifting it, KNI users can also use jumbo frames.
172+
173+
[Note 2] This back-and-forth shows that conservative spec-stage constraints deserve re-examination after real measurements: an EOPNOTSUPP rejection is the safest wording, but if measurements show coexistence is fine, lifting the exclusion is friendlier to users than maintaining a "paper risk".
174+
175+
4.5 Other engineering details
176+
177+
- **EBUSY state machine**: `rte_eth_dev_set_mtu` may return -EBUSY for a running port; M4 implements stop/set_mtu/get_mtu/start within the primary process, with rollback restoring the old MTU and restarting the port on failure; secondaries simply return 0 without touching hardware.
178+
- **Multi-process consistency by usage convention**: zero IPC, zero transactions, zero message rings between processes; each process triggers SIOCSIFMTU once. A process that never sets MTU keeps its old software value (a known constraint, explicitly stated in the spec).
179+
- **Gates**: -Werror build, git-diff-verified zero changes to the freebsd/ tree, no rte_eth* references in ff_veth.c, no atoi, no IPC residue, zero regression across all paths with mtu_enable=0 — all six PASS.
180+
181+
5. How to use it, how to configure it, and the results
182+
183+
5.1 Configuration
184+
185+
```ini
186+
[dpdk]
187+
mtu_enable=1 # feature master switch, default 0 (behavior identical to the old version when off)
188+
max_mtu=9000 # runtime MTU upper bound + large-mode pool pre-allocation basis, default 9000
189+
mbuf_mode=large # large: one mbuf carries jumbo; scatter: standard mbuf multi-seg chain
190+
191+
[port0]
192+
mtu=9000 # protocol-stack + hardware MTU at port startup, default 1500
193+
```
194+
195+
Constraint quick reference:
196+
197+
- `mtu_enable=0` combined with `mtu>1500`: parsing fails, with a hint to enable the MTU feature first
198+
- In large mode `max_mtu` is bounded by `data_room_size <= UINT16_MAX` (65535 unavailable); scatter mode can go to 65535 but is subject to PMD capability
199+
- Illegal configurations fail at startup; there is no silent fallback
200+
201+
5.2 Runtime changes
202+
203+
```bash
204+
# primary process: changes soft + hardware together (including EBUSY → stop/set/start)
205+
ff_ifconfig -p 0 f-stack-0 mtu 9000
206+
207+
# query
208+
ff_ifconfig -p 0 f-stack-0
209+
f-stack-0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
210+
```
211+
212+
Multi-process convention: a change on the primary updates both soft and hardware; each secondary must trigger its own change (updating only its own software view). This is a usage convention, not a bug; cross-process IPC coordination is a future milestone outside this scope.
213+
214+
5.3 Results (physical-machine measurements)
215+
216+
| Test item | Result |
217+
|---|---|
218+
| IPv4 jumbo | `ping -M do -s 8972` bidirectional 8500-byte traffic fine, MTU=9000 without fragmentation ✅ |
219+
| IPv6 jumbo | `ping6 -M do -s 8500` fine (before the fix, replies were split into 6 fragments of 1448; after adding if_notifymtu, no fragmentation) ✅ |
220+
| KNI/MTU coexistence | `mtu_enable=1` + `kni.enable=1` coexist fine; veth0 MTU 1500/9000 both send and receive correctly ✅ |
221+
| Unit tests | 59 passed (including 8 new MTU tests), all 12 test binaries PASS ✅ |
222+
| Gates | -Werror / freebsd tree untouched / no rte_eth leakage / no atoi / no IPC residue / zero regression — all PASS ✅ |
223+
224+
Unit test coverage: 8 new config-parsing cases (UT-CFG-01..08) + 7 fixtures, focusing on config-validation boundaries (defaults, cross-field conflicts, illegal strings, overflow).
225+
226+
5.4 Advice for users
227+
228+
- Before enabling jumbo, verify the link: the physical NIC, vSwitch, and peer must all support jumbo; with an unsupporting link, mid-path drops of large frames are worse than 1500
229+
- Memory-sensitive scenarios: choose scatter — large mode pre-allocates the mbuf pool by max_mtu with significant memory cost; scatter saves memory but requires PMD RX_SCATTER + TX_MULTI_SEGS support
230+
- Just need a smaller MTU: no need to enable mtu_enable — `ff_ifconfig ... mtu 1400` works directly
231+
232+
Further reading:
233+
234+
- Research conclusions overview with three-way evidence: docs/mtu_change_spec/00-overview-index.md
235+
- IPv6 jumbo fragmentation root-cause chain (12 file:line): docs/mtu_change_spec/15-ipv6-jumbo-frame-fragmentation-analysis.md
236+
- Implementation report (M0~M5 + Post-M5): docs/mtu_change_spec/14-implementation-report.md
237+
- Interface and config design: docs/mtu_change_spec/08-interface-and-config-design.md
238+
- Related issues: #239 (set MTU in example), #490 (Why MTU MAX CONF is 1500), #720 (Enabling jumbo frames), #1033 (large-packet performance and MTU)
239+
- Three-layer architecture docs: docs/01-LAYER1-ARCHITECTURE.md
240+
- Knowledge graph: docs/KNOWLEDGE_GRAPH_WIKI.md

0 commit comments

Comments
 (0)