JACCL RDMA completion status checking and silent corruption
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
zcbenz's closing reason: it is easier for maintainers to write and verify such changes themselves than to review contributor PRs over several rounds
These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- zcbenz's closing reason: it is easier for maintainers to write and verify such changes themselves than to review contributor PRs over several rounds [source]
- angeloskath measured about 10% slowdown (about 1 us) on 16 KB 4-way all-reduce and held the PR for a JACCL refactor and optimisation pass, noting the team treats Thunderbolt as a reliable channel [source]
- The rebased PR changed 3 files (+61 lines) and added 4 check sites in mesh_impl.h and 5 in ring_impl.h [source]
- The first commit message cites IBV_WC_RETRY_EXC_ERR as an example of an RDMA transport error that was silently ignored [source]
- The original PR also made Connection::poll() and the free poll() throw on a negative ibv_poll_cq return [source]
- The PR author tested 256, 512, 5478+433 and 8490+512 token generations on the 612 GB model with zero RDMA errors after the fix, and saw no regression on models under 10 GB per rank [source]
- The PR author states small models are unaffected because their RDMA operations always succeed, so the bug is latent [source]
- After a peer is SIGKILLed, ibv_poll_cq returned 0 about 2.64 billion times, never negative, never a bad wc.status, so status checking cannot detect peer loss [source]
- Related exo report: macOS 26.3.1 4x M3 Ultra clusters hit "[jaccl] Recv failed with errno=2", errno=60 and "Changing queue pair to RTR failed with errno 22"; the reporter says the #3451 race fix removed the post-kill RTR failures [source]
- Issue #3467 root cause: after the #3412 refactor Connection::info() only accepts IPv4-mapped GIDs (::ffff:x.x.x.x) but Apple Thunderbolt RDMA can expose only link-local fe80:: GIDs, leaving the GID uninitialised and giving RTR errno 22 [source]
Corrections and disagreements
- CONTRADICTS distributed-inference-across-macs.md: PR #3152 is closed, not open; zcbenz closed it without merging on 2026-10-02 [source]
Children
- No children recorded.