Unbalanced traffic over LACP bond

[rowlap]

I'm using openvswitch2.17.x86_64 from an RHEL8 derivative, and appreciate that this is an old version, however rebasing the package is not straightforward.

When setting up a bonded LACP interface, one interface gets all the egress traffic, and the other almost zero, perhaps the LACPDUs alone.

The interface config is not adventurous:

# ovs-appctl bond/show
---- bond0 ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 2
bond-hash-basis: 0
lb_output action: disabled, bond-id: -1
updelay: 0 ms
downdelay: 0 ms
lacp_status: negotiated
lacp_fallback_ab: false
active-backup primary: <none>
active member mac: 30:3e:a7:11:92:a9(eth0)

member eth0: enabled
  active member
  may_enable: true

member eth2: enabled
  may_enable: true

There is a workaround:

# ovs-vsctl set port bond0 other_config:bond-hash-basis=1

or

# ovs-vsctl set port bond0 other_config:bond-rebalance-interval=10000

causes egress traffic to flow over both interfaces. The weird part is that reverting either setting to 0 doesn't affect the balanced egress, as if there's some internal state to the interface which is being initialized only when we touch other_config.

I'm not expecting fixes for 2.17.0, but are there any other commands which could shine a light on what we're seeing here, before the heavier lift of upgrading?

[igsilya]

10000 is actually the default value for the bond-rebalance-interval, does it default to zero in your build? That would explain why rebalancing is not working.

The bond/show supposed to show the bucket hashes with the load they are carrying. Is there anything interesting in this information?

How much traffic are you sending through the bond? It may not rebalance when the traffic rate is too low.

[rowlap]

I think the zero bond-rebalance-interval is coming from

# grep OVS_OPT /etc/sysconfig/network-scripts/ifcfg-bond0
OVS_OPTIONS="bond_mode=balance-tcp lacp=active other_config:bond-rebalance-interval=0"

but I have no good rationale for this (the config was inherited by my team). When applying bond-rebalance-interval=10000 we indeed do see 256 hash buckets with kB load statistics. There is plenty of traffic, although the flows are highly variable in throughput.

My mental model of hash-based LACP is different to this. Should I think of each flow having affinity due to 5-tuple hashing, or what was least-busy when the flow was created or the bond interface last rebalanced, or ?

If there's still a problem report here, and I'm not convinced there is, should it be harder to bring up the bond interface without hashing working correctly / at all?

[igsilya]

Should I think of each flow having affinity due to 5-tuple hashing, or what was least-busy when the flow was created or the bond interface last rebalanced, or ?

For each packet we have a 5-tuple hash (it's not really a 5-tuple hash in older OVS/kernel, it's just an skb hash, which is a 5-tuple hash in most cases, but may also not be), last 8 bits of the hash value define a hash bucket this packet (traffic flow) goes to. These hash buckets (256 of them), a.k.a. "hashes" are getting balanced between members of the bond. If the bucket is assigned to port A, all the traffic with this 5-tuple hash hash goes to port A.

If I were guessing what happened in your setup is one port likely went DOWN, or it was down at the moment of the bond creation, so all the hash buckets were assigned to the one port that was UP. And since the rebalancing is effectively disabled (interval is zero), this situation was never repaired when the port came back UP. If you touch the configuration, this triggers rebalancing and some hash buckets are getting moved to the other port.

[rowlap]

OK, tried upgrading to Open vSwitch 3.4.1, and we no longer see single-member utilization. I stared at the bonding docs harder and think I now understand the egress design, although it's still surprising to me, having not encountered any other LACP implementation with stateful flow tracking.

Presumably this comes at a cost of tracking and balancing flows into buckets, and periodic rebalancing the buckets across bond members? What's the recommendation for "pure" hashing which tracks nothing once the packet has been egressed?

(Feel free to close this at any time with "not an issue in the latest version")

[igsilya]

There is no real tracking, packets are hashed and sent out through one of the ports based on the hash. We're only counting how much traffic went through a particular hash bucket, maintaining the moving average value, and if it turns out that one of the ports is carrying significantly more than the other, we move some hashes between them, so packets with the same hash start egressing form the different port.

[rowlap]

A minimum-viable-workaround is to apply

ovs-vsctl set port bond0 other_config:bond-hash-basis=1
ovs-vsctl set port bond0 other_config:bond-hash-basis=0

as part of the network startup, however this has a race with LACP initialization, so a sleep 10 helps to apply the settings at the right time.

As this appears to be resolved in a newer version, I'll close this issue. It would be interesting to bisect versions and find out when it changed, but that would take time I haven't got right now.

[Lalufu]

I stubled across this issue observing the same behaviour on OL9, using OVS3.3.
The theory that buckets get initialized on bond creation, and that not all members may be present at that point makes some sense to me. The way we're bringing up OVS takes some seconds because the cards are put into SRIOV mode, and this is not a fast process across multiple cards.
We're running with bond-rebalance-interval=0 because we do not want active flows to be reshuffled (this can have consequences further upstream, too), unless there is a change in the membership of the bond. Obviously, when a bond member goes away, flows that were assigned to that member need to go somewhere else.
I had assumed (but not checked), that changes in bond membership would trigger redistribution of hashes and their associated flows across the new members, but this does not seem to be the case?
Given that the original poster reports this no longer being an issue in 3.4.1, was there a change between 3.3 and 3.4.1 that would explain this?

[igsilya]

I had assumed (but not checked), that changes in bond membership would trigger redistribution of hashes and their associated flows across the new members, but this does not seem to be the case?

No, it is not the case. As documented, when rebalancing interval is zero, only the link failures cause flows to move. If rebalancing causes some sort of issues further upstream from the OVS, then rebalancing on link state changes will cause these issues too, I would expect.

At the same time, I'm not sure what kind of issues we're talking about here. If the switch on the other side of the cable is properly configured, it should not be surprised by the traffic shift. We're not rebalancing that often, and trying to minimize the number of flows we're moving.

Given that the original poster reports this no longer being an issue in 3.4.1, was there a change between 3.3 and 3.4.1 that would explain this?

I do not remember any intentional changes around this behavior. Most likely some change skewed timings in the bond creation so ports were already up. It will still not rebalance back after a link flap, will practically operate as active-backup at this point.

[Lalufu]

I admit my use case may not be the most typical one. The server is connected to a multi stage L3 CLOS network, where each switch (from the ToR switch to the spine and super spine layers) usually has multiple L3 paths to each destination, and uses ECMP to select one. The input interface is in some cases part of the input into the hash algorithm. Additionally, the servers are typically connected to an MLAG pair, with some members of the LACP bundle on one switch, and some on the other. When OVS rebalances flows, flows might not just end up on a different interface on the same switch, but even on a different switch, which influences the further path the packet takes. For unicast destinations this, ideally, doesn't matter, as all paths lead to Rome, as they say. There is always the possibility of certain paths being subtly or not so subtly broken, which means that a flow that just worked fine suddenly does not. Root causing these issues can be somewhat troublesome even in the best of cases. If your network uses anycast destinations things get a lot weirder, as a reshuffle of a flow can mean that you're suddenly hitting an entirely different endpoint, which will lead to a connection reset in TCP. Now you're correct in saying that a link failure (on the server or in other places in the network) will have the same effect. They are a lot rarer through, and easier to correlate to events, compared to flows potentially being moved every few seconds. This is why we disable this feature in OVS. As for the OVS version, we'll test 3.4 to see if this improves things, and if not we'll have to see if we can trigger a manual reshuffle on interface changes. There isn't a CLI command to request bond rehash, besides changing configuration like the hash base or the automatic timer, right?

[Lalufu]

For the record, we tested 3.4, and it does not solve this issue. For an easy test, disable link on a member interface (`ip link set down dev eth0`), them re-enable it (`ip link set up dev eth0`). Then observe that no flows will be scheduled to `eth0` when looking at `ovs-appctl dpctl/dump-flows --names`. For now, we moved the LACP/bond to a `bonding` kernel interface, and plugged that into OVS as a system interface.

[rowlap]

It would be useful to inspect the internal state of the hashing, beyond what ovs-appctl bond/show displays.

Is there any admin-visible output which shows in more detail the precise per-packet egress hashing decision? (Although ~related, I'm not asking about the periodic egress bucket rebalancing.)

[rowlap]

Re-opening, because this still affects LACP, is seen in newer versions, and I have a slightly better repro.

Testing with 3.5.2, we still see unbalanced egress. More specifically, between bond member change and rebalance, no traffic is observed to egress the added member interface.

As above, this implies that bond-rebalance-interval=0 never obtains balanced egress.

Steps to reproduce:

• Build a vanilla LACP bond with two members
• Set bond-rebalance-interval=60000
• On some nearby host run iperf -s
• On the repro host run iperf -b 1M -e -i 5 -t 3600 -P 10 -c OTHERHOSTNAME
• This should generate approx 10 MB/s of egress traffic across multiple flows
• Choose one of the bond members (e.g. eth0) and run ifdown eth0; sleep 5; ifup eth0
• Observe that eth0 is now UP but not egressing traffic
• Repeat for the other interface
• Observe that traffic has now switched to egress only from eth0
• Observe the ovs-appctl bond/show next rebalance countdown, when egress will self-rebalance

I don't currently understand why bond member changes don't trigger rebalancing.