Per-collective all_sum latency in decode over JACCL RDMA
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
Existing coverage: tensor-vs-pipeline-vs-expert-parallelism-for-moe.md has TB5 RDMA about 50 us vs about 300 us TCP and 25 GbE about 200 us; mlx-metal-fast-synch dossier says the 0.26 ms FAST_SYNCH saving is unconfirmed; exo-cluster-software.md has the exo tweet of 3 us. This file reconciles them
These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- Existing coverage: tensor-vs-pipeline-vs-expert-parallelism-for-moe.md has TB5 RDMA about 50 us vs about 300 us TCP and 25 GbE about 200 us; mlx-metal-fast-synch dossier says the 0.26 ms FAST_SYNCH saving is unconfirmed; exo-cluster-software.md has the exo tweet of 3 us. This file reconciles them [source]
- PR 4530 is likewise closed unmerged, so no JACCL liveness check or progress timeout exists in mainline MLX or in the exo pin [source]
- The same guruswami INTERCONNECTS page gives two different TB5 RDMA latencies: 'about 50 us per op' in the speed hierarchy table and 'about 100 us per all-reduce' in the Llama 405B TP2 table, so the author's own figure is 2x apart depending on the table [source]
- Per-collective figures now stand at about 3 us (exo tweet, Dec 2025), about 50 us and about 100 us (guruswami, Mar 2026), and 1.9 ms for a 16 MB all_sum on a 2-node mesh (PR 4530 clean-path test); these differ by payload size and by what is timed, so none is a decode-sized collective measurement [source]
- angeloskath measured about 1 us (about 10 percent) added by the wc_status check on a 16 KB 4-way all-reduce, which implies a baseline of about 10 us for that small collective on JACCL mesh [source]
- The guruswami Llama 405B TP2 model counts 126 all-reduces per token (one per layer) and gets 12.6 ms of sync at 100 us each against a stated 35 ms token time; standard TP needs about two all-reduces per layer (after attention output and after MLP down projection), so the true count is likely near 252 [source]
- The page's own measured TP2 for Llama 405B Q4 is 4.3 tok/s, about 233 ms per token, so the '35 ms token gen time' row cannot be the measured value and the 36 percent overhead figure is a model, not a measurement [source]
- At the author's 100 us and 126 collectives, sync is 12.6 ms of about 233 ms measured, about 5 percent, which would not explain the observed 4.3 vs 3.0 tok/s TP2 scaling; compute and memory effects dominate [source]
- The guruswami page says Q8 Qwen 32B at 54 ms per token has 23 percent sync overhead and Q2 at 21 ms has 60 percent, both computed from the same 12.6 ms; this is the arithmetic behind its 'TP scaling is quant-dependent' finding, not a per-quant measurement [source]
- No fetched source reports a per-layer all_sum latency measured during decode on JACCL (for example with an 8 KB tensor); the model-free loop proposed in exo-vs-mlx-launch-tensor-parallel-throughput-comparison.md remains the way to get it [source]
Corrections and disagreements
- CONTRADICTS distributed-inference-across-macs.md and apple-libthunderboltrdma-tbt-post-recv-sigsegv.md on PR status: MLX PR 3152 and PR 4530 are both closed unmerged by zcbenz (3152 on 2026-10-02, 4530 on 2026-09-29); both pages in the cache show the 'closed this' event and no merge [source]
Children
- No children recorded.