Skip to content

Report when no usable GID is found - #4191

Merged
zcbenz merged 1 commit into
ml-explore:mainfrom
erwinzhang7:fix-3467-gid-fail-fast
Aug 12, 2026
Merged

Report when no usable GID is found#4191
zcbenz merged 1 commit into
ml-explore:mainfrom
erwinzhang7:fix-3467-gid-fail-fast

Conversation

@erwinzhang7

Copy link
Copy Markdown
Contributor

Related to #3467.

Connection::info() scans the GID table for an IPv4-mapped GID. If none matches, the loop
simply ends: ibv_gid gid; is never assigned and never initialized, and that value is
written into src.global_identifier and used for the queue pair. The error surfaces later
from the RTR transition as errno 22, which does not point at the GID table.

This is not a rare path. Thunderbolt RDMA ports publish only a link-local IPv6 GID until the
interface has an IPv4 address, so any Thunderbolt setup hits it before configuring one:

rdma_en3:  link_layer: Thunderbolt
           GID[0] = fe80::347c:94ff:fecd:45c4      <- no IPv4-mapped GID

After the change, the same configuration reports the cause and the remedy at the point of
failure:

[jaccl] No IPv4-mapped GID for this device. Thunderbolt RDMA ports only publish one
once the interface has an IPv4 address; assign a link-local address to it, for
example: ifconfig <interface> inet 169.254.0.1 netmask 255.255.0.0 alias

Verified

Two M4 Pro Mac minis on macOS 26.5.1, connected directly over Thunderbolt 5,
RDMA enabled from Recovery OS.

  • With no IPv4 on the port, stock fails at RTR with errno 22; with this change it throws
    the message above from info().
  • With an IPv4 assigned, behaviour is unchanged: info() returns ::ffff:169.254.10.1 and
    a two-rank mesh runs a correct all_sum.
  • Checked on an active port. An inactive one fails earlier, in
    allocate_protection_domain, and never reaches this code.

Connection::info() left gid uninitialized when the loop matched nothing and
handed it to the queue pair, where it surfaced later as an opaque errno from
the RTR transition. Thunderbolt ports only publish an IPv4-mapped GID once the
interface has an IPv4 address, so this is a configuration people hit rather
than an exotic one.
@zcbenz
zcbenz merged commit 21d897d into ml-explore:main Aug 12, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants