fix: benchmark pasta, not slirp4netns - #615
Conversation
Signed-off-by: Hayato Kiwata <dev@haytok.jp>
Signed-off-by: Hayato Kiwata <dev@haytok.jp>
| |`gvisor-tap-vsock` | 1.55 Gbps | 5.40 Gbps | ||
| |`lxc-user-nic` | 29.3 Gbps | 30.4 Gbps | ||
| |(rootful veth) | (35.0 Gbps)| (36.2 Gbps) | ||
| |`slirp4netns` | 1.69 Gbps | 8.11 Gbps |
There was a problem hiding this comment.
I guess the GHA machine spec was recently upgraded?
| |`slirp4netns` | 1.69 Gbps | 8.11 Gbps | ||
| |`slirp4netns` (with sandbox + seccomp) | 1.68 Gbps | 8.32 Gbps | ||
| |`vpnkit` | 0.14 Gbps |(Unsupported) | ||
| |`pasta` | 0.24 Gbps | 31.9 Gbps |
There was a problem hiding this comment.
Seems extremely sensitive to MTU 👀
Eventually we should also have some benchmark with the real Internet
There was a problem hiding this comment.
I got curious and looked into this.
Root cause:
pasta never sets txqueuelen on tap0, so it stays at the kernel default of 1000 packets. However, the receive window pasta advertises is ~5.5 MB — measured in bytes, independent of MTU. At MTU=1500 that is ~3,700 packets, so every window-sized burst overflows the tun ring buffer. Confirmed with perf record -e skb:kfree_skb, which reports as follows:
$ sudo perf record -e skb:kfree_skb -o /tmp/drops.data -- sleep 180
...
$ sudo perf script -i /tmp/drops.data | grep pasta | head -1
pasta 477878 [003] 442889.154999: skb:kfree_skb: skbaddr=0xffff0000ec228b00 rx_sk=(nil) protocol=2048 location=tun_net_xmit+0x25c reason: FULL_RING
Workaround:
Raising the queue alone (ip link set dev tap0 txqueuelen 2000) is enough even when MTU stays 1500:
[ ID] Interval Transfer Bitrate Retr
[ 6] 0.00-30.00 sec 88.3 GBytes 25.3 Gbits/sec 3074 sender
[ 6] 0.00-30.00 sec 88.3 GBytes 25.3 Gbits/sec receiver
The
benchmark::iperf3::pastafunction was running rootlesskit with--net=slirp4netnsAs a result, the Benchmark: Network (MTU=65520, network driver=pasta) jobs in the CI have been measuring
slirp4netns, notpasta.After applying this fix,
./benchmark-iperf3-net.sh pasta 65520benchmarksrootlesskit --state-dir=<state> --net=pasta --mtu=65520 -- iperf3 -t 30 -c 10.0.2.2Details
Note that since updating the benchmark script changes the benchmark results, I have also updated the documentation.