Static link-local IPv4 per Thunderbolt port for JACCL meshes
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
A static IPv4 per Thunderbolt port means each cabled Thunderbolt RDMA interface (`enN`, with verbs device `rdma_enN`) carries its own fixed address, so its IPv4-mapped GID exists at boot and after re-link without waiting for DHCP or APIPA.
These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- A static IPv4 per Thunderbolt port means each cabled Thunderbolt RDMA interface (`enN`, with verbs device `rdma_enN`) carries its own fixed address, so its IPv4-mapped GID exists at boot and after re-link without waiting for DHCP or APIPA. [source]
- Exo's maintainers describe the requirement as "each rdma port ... routable" and give two equivalent recipes. Recipe A: delete the Thunderbolt bridge, create a new network service per Thunderbolt port, and enable DHCP or IPv6 auto-assignment. Recipe B: delete the bridge, set manual IP addresses on each port and route traffic between them manually with the `route` command. [source]
- Exo bundles a script for recipe A with its app (`tmp/set_rdma_network_config.sh`), which a maintainer calls "a little destructive to existing networking settings". [source]
- An exo maintainer said in February 2026 that the interfaces need reachable addresses, "but after macOS 26.2 they do not self-assign by default". In the reporter's `ifconfig`, `en2` to `en7` showed `status: inactive` with no address, while the one self-assigned address (`169.254.138.233`) sat on a different interface, `en8`. [source]
- One reported exo cluster of four Mac Studio M3 Ultra (macOS 26.3) ran with network location `exo`, Thunderbolt Bridge disabled and all `EXO Thunderbolt 1-6` services on DHCP link-local. That is dynamic link-local per port, not static. [source]
- ds4's docs use `sudo ifconfig en1 inet 10.99.0.2/30 alias` on one Mac and `sudo ifconfig en6 inet 10.99.0.1/30 alias` on the other, so a private /30 per link, not 169.254.0.0/16. ds4 needs only some IPv4 address on the cabled member interface. [source] — privacy-ok (example address quoted from the cited source)
- JACCL's own error text suggests the opposite shape: `ifconfig <interface> inet 169.254.0.1 netmask 255.255.0.0 alias`. [source]
- The no-IPv4 failure appeared in exo in February 2026 as `Changing queue pair to RTR failed with errno 22` on two nodes with unaddressed ports (issue 1390), two months before the JACCL refactor that issue 3467 blames. [source]
- Three schemes exist and none is shown to be better: dynamic link-local per port (a reported exo cluster), static link-local per port from one block (the issue 3467 reporter's, on a 4x M3 Ultra mesh), and a private /30 per link (ds4's docs). Each only has to give every cabled member interface some IPv4 address. [source]
- A single block shared by all ports (for example one /24 for `169.254.250.N`) gives one host several interfaces in one subnet. Whether that affects non-RDMA IP traffic is not tested in the sources; RDMA setup uses the peer's GID, not an IP route. [source]
- Exo's recipe B includes manual routing with the `route` command; the static-block and /30 recipes in the other sources do not mention routes. [source]
- Static versus DHCP link-local. The issue 3467 reporter found DHCP/APIPA fragile at boot and after link changes and moved to static; a reported exo cluster runs DHCP link-local per port. [source]
- Bridge deleted or kept. Exo deletes the bridge in both recipes; ds4's docs only say addresses must be on member interfaces and not only on the bridge. [source]
- Whether a static address survives a macOS 27 upgrade and the reworked RDMA library. [source]
- Whether the `route` step in exo's recipe B is needed for JACCL itself or only for exo's IP traffic. [source]
- Exo's maintainers give two recipes for routable Thunderbolt RDMA ports: new network services with DHCP or IPv6 auto, or manual per-port addresses with manual routes, both after deleting the bridge. [source]
- Exo bundles `tmp/set_rdma_network_config.sh` to apply recipe A and warns it is destructive to existing network settings. [source]
- An exo maintainer said in February 2026 that since macOS 26.2 the Thunderbolt RDMA interfaces no longer self-assign IPv4 addresses by default. [source]
- RTR errno 22 on unaddressed ports was reported in exo in February 2026, before MLX PR 3412. [source]
- One reported four-node exo cluster ran with the bridge disabled and every `EXO Thunderbolt N` service on DHCP link-local. [source]
- ds4's docs configure a private /30 per Thunderbolt link with `ifconfig ... alias`, not a link-local block. [source]
- JACCL's error text suggests `ifconfig <interface> inet 169.254.0.1 netmask 255.255.0.0 alias`. [source]
- Any IPv4 address on the cabled member interface satisfies both ds4 and JACCL; the sources use link-local, /30 and DHCP-assigned forms. [source]
Children
- No children recorded.