Silent-corruption testing for distributed collectives with rank-distinct data
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness.
These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness. [source]
- Ring all_reduce switches to one direction and one wire at 32768 bytes or less, mesh multi-wire starts at 512 KiB, and mesh switches algorithm at 32 KiB per wire when size is above 2, so a corruption test must cover sizes on both sides of each [source]
- Ring sum_scatter changes wire count at 65536 bytes [source]
- Ring all_gather splits every rank's region into `2 * n_wires` slices of `ceil(n_bytes / (2 * n_wires))` bytes, so tail clamping is exercised only when the size is not a multiple of `2 * n_wires` [source]
- With constant input, an all_gather or all_sum that moves the wrong slice returns the same values, so constant data cannot detect it [source]
- UC delivery never reports a lost message, so integrity tests need a timeout as well as an equality check [source]
- Use `MLX_JACCL_RING=1` (or `--backend jaccl-ring`) to test the ring path separately from the mesh path [source]
- Integer dtypes give exact expected sums, unlike bfloat16 [source]
Children
- No children recorded.