atlantic stops receiving packets after a while under high load

[hoshinolina]

This issue has been happening randomly for a while now, but I only just got around to setting up an alternate network interface to investigate.

After some time with high load/IO, the 10GbE interface on a j274 stops receiving network packets. TX still works, which eventually devolves to just ARP requests (which are received by other hosts). The failure is not instant, but rather seems to take time ranging up to several minutes, during which somehow network connectivity is degraded between certain hosts.

In this instance, the failure happened at 20:02 initially and most traffic flows were interrupted, but the (remote) monitoring still worked. It somehow recovered at 20:17, before failing completely at 20:31.

This is currently only happening on a j274. I don't think I've ever seen it on other Mac Mini variants (M1 Pro, M2). I'm not sure if that's just a coincidence, though (it could be related to CPU performance relative to network IO, for example).

I have a vague feeling this might be IRQ related. If there are multiple RX queues assigned to different network flows, some IRQs or queues failing first would explain the strange failure behavior where it fails partially before it fails fully.

/proc/interrupts dump during the failure:

124:        131          0     316956     321397          0    6301482    5757779    7076594 PCI-MSIX-0000:03:00.0   0 Edge      end0
125:         62          0   22534244          0     180882     109157      19396     163910 PCI-MSIX-0000:03:00.0   1 Edge      end0
126:        178          0          0     150623    9052354    3688564    1227219    4592258 PCI-MSIX-0000:03:00.0   2 Edge      end0
127:   16596356          0          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   3 Edge      end0
128:         81          0          0     236171      26504    3987434    4310273    5243217 PCI-MSIX-0000:03:00.0   4 Edge      end0
129:         85          0          0         75       8166    7977179   19628093    7158307 PCI-MSIX-0000:03:00.0   5 Edge      end0
130:        141          0          0     740925    8342324    4133795    2094545    1913432 PCI-MSIX-0000:03:00.0   6 Edge      end0
131:         80   27867953          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   7 Edge      end0
132:          0          0          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   8 Edge      end0

In this case, IRQ 128 (MSI 4) is ticking up with TX packets, but nothing else is.

ip link shows all RX packets as dropped (dropped and mcast ticking up, as well as TX normally):

2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a4:77:f3:05:50:1b brd ff:ff:ff:ff:ff:ff
    RX:     bytes    packets errors dropped  missed   mcast
     725823928713  552326168	  0   20032	  0  449035
    TX:     bytes    packets errors dropped carrier collsns
    1818579085663 1247873976	  0	  0	  0	  0
    altname enp3s0
    altname enxa477f305501b

So the PHY is receiving packets, but they aren't making it into the kernel.

Nothing in dmesg when the problem happens. ip link set end0 down && ip link set end0 up fixes the situation.

[hoshinolina]

Somewhat related bug:

• https://bugzilla.kernel.org/show_bug.cgi?id=202651
• https://bugzilla.redhat.com/show_bug.cgi?id=1499321

Possibly unrelated but I'm pretty sure I'm seeing this too sometimes (symptom is, when NetworkManager sees the link and brings up networking, the link drops and comes back again sometimes, and the drops take several seconds).

Note to self, ethtool -S end0 has more stats.

[jannau]

I had occasional issues with atlantic on my M1 Ultra Studio which I never managed to diagnose. I guess this was around 2023. Usage of that machine shifted and don't think I it reoccurred in the last 1-2 years. Since then high load/IO would have been a rare though. Since ip link set end0 down/up recovers network connectivity I assume a cable unplug/re-plug would do as well.
My M1 Mac Mini with atlantic serves as distfile/package proxy and file server. It sees spiky high load/IO when other systems update or when it compiles gentoo updates. There were a few events of lost network connectivity (1 or 2, maybe 3 over years).
The M1 systems doesn't do anything with IRQ affinity and all end0 interrupts are handled CPU core 0.

Is Mac Mini variants (M1 Pro ... a typo for "M2 Pro"? There was no M1 Pro Mac Mini only M1 Max/Ultra Mac Studios.

[hoshinolina]

Yes sorry, M2 pro. I do not believe cable replug fixes it, I'm pretty sure I tried that before. down/up would tear down and set up the packet interface, but I think cable replug only really affects the PHY, so they are different.

I haven't done anything with affinity manually, though irqbalance does move the affinities around (Fedora default). I've turned it off and manually assigned the IRQs to CPUs 0-7 (plus another 7 for the last one since there are 9), let's see if it happens again.

[hoshinolina]

The actual load on the system isn't particularly extreme. You can see the gaps where it crashed in the past couple of days.

I'm going to artificially make it churn a bit more to see if this keeps happening or I can correlate it with the irqbalance change.

Also the crash yesterday lined up eerily precisely with an earthquake... but it wasn't particularly extreme here in Tokyo, and it certainly doesn't crash when I shake it around by hand, so I don't know if that was just a crazy coincidence or what...

[jannau]

This doesn't trivially reproduce using parallel iperf3 on a M1 Ultra with 10Gbps link speed.

[hoshinolina]

It's not purely based on network load, I think it has something to do with system load/IO load etc. too or some other unknown factor. As you can see from the graphs, the last crash the network was barely doing 100-200mbps. The crashes do consistently correlate with the machine/network being busy for long periods of time (moving data around the network and disks), even if the actual total load isn't high, but I don't know what particular factor is needed to repro.

It's worth noting that my workload has many parallel TCP connections, around 300 right now (this is normal and expected for this use case). That might be a factor.

I think I'm just going to try to keep things "untouched" for a few weeks other than the irqbalance stuff, and see if I can correlate it with something. I have a script polling network connectivity and restarting the interface now, so at least it won't cause an outage, but it'll still leave logs behind.

[hoshinolina]

This did not repro in a few days after stopping irqbalance, so I tried this:

while true; do for i in $(seq 124 132); do echo $((RANDOM%8)) >| $i/smp_affinity_list;done;sleep 120;done

Where 124-132 are the IRQs for atlantic. This reproduced it fairly quickly, though it isn’t happening again easily, so maybe I just got lucky. I don’t yet have a reliable enough repro/enough data to be able to say for sure that this is migration related.

There seems to be a lot of subtlety around migration of MSIs and other edge-triggered IRQs. Search keywords: IRQCHIP_MOVE_DEFERRED and irqd_set_resend_when_in_progress. I suspect there’s something misconfigured or racy for AICv1 (and this explains why AICv2 is unaffected, because it doesn’t do affinity). Calling the latter on APCIE IRQs might be a good idea.

Relevant thread: https://lore.kernel.org/lkml/20230317095300.4076497-1-jgowans@amazon.com/

My guess is that there is no auto-masking (active state) for edge-triggered MSI IRQs in AIC, and the IRQ can be retriggered immediately after the event register is read. Normally that just queues up another of the same event on the same CPU, which won't run until aic_handle_irq() loops around. However, if the affinity is changed, another CPU could end up processing that IRQ in parallel, leading to the situation mentioned in that thread. This does beg the question of what happens when the IRQ is auto-balanced as in AICv2 (or AICv1 with >1 CPU in the mask?). Maybe there is an active state, but the problem is that this code masks/unmasks the IRQ, which would lead to essentially a premature EOI when the IRQ moves, allowing it to trigger on another core. That could be surgically fixed in the AIC code but it would need a spinlock to close the race beween event register read and another core unmasking, so it probably still makes more sense to just call irqd_set_resend_when_in_progress() for APCIE MSI IRQs (or all edge-triggered AIC IRQs).

Aside: On AICv2 machines, the MSI domain claims affinity is supported (because it unconditionally forwards to parent) which makes /proc/irq/XX/smp_affinity writable, but actually trying to write it returns -ENOSYS (see irq_chip_set_affinity_parent). This might be worth fixing so affinity is properly reported as not configurable.

[jannau]

There is MSI_FLAG_NO_AFFINITY for struct msi_parent_ops.required_flags which makes smp_affinity read-only. I unfortunately see no clean way to determine whether the parent irq domain supports irq affinity.
Best workarounds I can come up with is either checking if the parent irqdomain's fwnode is "apple,aic"-compatible or check if the device itself is "apple,t8103-pcie". Doubt any of the two options will be received enthusiastically upstream.

[jannau]

If irq affinity is the cause my issue on the M1 Ultra is unrelated.
A possible cause are thermal issues in the switch. I've mounted it (Zyxel XGS1250 (8x 1000Base-T, 3x 10GBase-T, 1x SFP+)) under my desk and apparently that wasn't compatible with its thermal requirements. I later discovered that one of the 10GBase-T ports died.

[hoshinolina]

I originally suspected thermal issues on the atlantic side too, but I've reasonably eliminated those by now.

I unfortunately see no clean way to determine whether the parent irq domain supports irq affinity.

Just check whether ops->irq_set_affinity is not NULL?

[jannau]

I unfortunately see no clean way to determine whether the parent irq domain supports irq affinity.

Just check whether ops->irq_set_affinity is not NULL?

The only way I see to access the irq_chip / ops is via irq_data which isn't yet allocated for the MSI AIC irq. That happens only in the call stack from msi_create_parent_irq_domain(). I could use one of port irqs knowning that both use AIC but that doesn't seem clean either. I asked maz for ideas.

[jannau]

Have you tried provoking the issue by inserting a generous udelay() in the interrupt handler? I'm currently testing AICv2 with

diff --git i/drivers/irqchip/irq-apple-aic.c w/drivers/irqchip/irq-apple-aic.c
index 4607d28f2388..223c12b98da8 100644
--- i/drivers/irqchip/irq-apple-aic.c
+++ w/drivers/irqchip/irq-apple-aic.c
@@ -48,6 +48,7 @@
 #include <linux/bits.h>
 #include <linux/bitfield.h>
 #include <linux/cpuhotplug.h>
+#include <linux/delay.h>
 #include <linux/io.h>
 #include <linux/irqchip.h>
 #include <linux/irqchip/arm-vgic-info.h>
@@ -412,9 +413,10 @@ static void __exception_irq_entry aic_handle_irq(struct pt_regs *regs)
 		type = FIELD_GET(AIC_EVENT_TYPE, event);
 		irq = FIELD_GET(AIC_EVENT_NUM, event);
 
-		if (type == AIC_EVENT_TYPE_IRQ)
+		if (type == AIC_EVENT_TYPE_IRQ) {
 			generic_handle_domain_irq(aic_irqc->hw_domain, event);
-		else if (type == AIC_EVENT_TYPE_IPI && irq == 1)
+			udelay(50);
+		} else if (type == AIC_EVENT_TYPE_IPI && irq == 1)
 			aic_handle_ipi(regs);
 		else if (event != 0)
 			pr_err_ratelimited("Unknown IRQ event %d, %d\n", type, irq);

with 10Gbps iperf3 and reading ~700MB/s from a SSD (USB).

The issue only occurring under high load and IO would be consistent with the race described in the linked lkml thread.

[hoshinolina]

I don't think that udelay is where it needs to be. It needs to be in the irq flow, before eoi is called, while the IRQ is considered active (and so it firing on another core would be ignored). In the AIC loop it won't do anything other than slow down IRQ processing in general.

[jannau]

yes, at least from the lkml thread it was the wrong place, yet iperf3 ran into

[  5][TX-C] 181.00-182.00 sec   694 MBytes  5.82 Gbits/sec  649    519 KBytes       
[  7][RX-C] 181.00-182.00 sec   948 MBytes  7.96 Gbits/sec                  
[  5][TX-C] 182.00-183.00 sec  0.00 Bytes  0.00 bits/sec    2   1.39 KBytes       
[  7][RX-C] 182.00-183.00 sec  1.07 GBytes  9.22 Gbits/sec                  
[  5][TX-C] 183.00-184.00 sec  0.00 Bytes  0.00 bits/sec    1   1.39 KBytes       
[  7][RX-C] 183.00-184.00 sec  1.08 GBytes  9.30 Gbits/sec                  
...
[  5][TX-C] 301.00-302.00 sec  0.00 Bytes  0.00 bits/sec    0   1.39 KBytes       
[  7][RX-C] 301.00-302.00 sec  1.08 GBytes  9.29 Gbits/sec                  
[  5][TX-C] 302.00-303.00 sec  0.00 Bytes  0.00 bits/sec    0   1.39 KBytes       
[  7][RX-C] 302.00-303.00 sec  1.09 GBytes  9.33 Gbits/sec                  
[  5][TX-C] 302.00-303.00 sec  0.00 Bytes  0.00 bits/sec    0   1.39 KBytes       
[  7][RX-C] 302.00-303.00 sec  1.09 GBytes  9.33 Gbits/sec                  
iperf3: error - control socket has closed unexpectedly

Probably unrelated as the network connection did not break down completely and the next iperf3 executin ran with the expected speed. Didn't reproduce quickly (in 30 minutes) either.

I've moved the udelay into atlantic's aq_vec_isr().

[hoshinolina]

I haven't seen a repro again with the loop, so I might have just gotten lucky. I think this will be a longer term watch and see thing unless either of us can work out a faster repro.