# Silent-corruption testing for distributed collectives with rank-distinct data > Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness. Parent: [Mac local LLMs: Clusters, RDMA, exo and ds4](https://llms-explorer.com/tree/mac-local-llms-clusters-rdma-exo-ds4/) · 1 facets · 8 facts · page: https://llms-explorer.com/tree/silent-corruption-testing-for-distributed-collec/ ## Facts - Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness. — source: `asserted` - Ring all_reduce switches to one direction and one wire at 32768 bytes or less, mesh multi-wire starts at 512 KiB, and mesh switches algorithm at 32 KiB per wire when size is above 2, so a corruption test must cover sizes on both sides of each — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/ring.cpp) - Ring sum_scatter changes wire count at 65536 bytes — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/ring.cpp) - Ring all_gather splits every rank's region into `2 * n_wires` slices of `ceil(n_bytes / (2 * n_wires))` bytes, so tail clamping is exercised only when the size is not a multiple of `2 * n_wires` — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/ring_impl.h) - With constant input, an all_gather or all_sum that moves the wrong slice returns the same values, so constant data cannot detect it — source: `asserted` - UC delivery never reports a lost message, so integrity tests need a timeout as well as an equality check — [source](https://raw.githubusercontent.com/antirez/ds4/main/ds4_tp.c) - Use `MLX_JACCL_RING=1` (or `--backend jaccl-ring`) to test the ring path separately from the mesh path — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/python/mlx/_distributed_utils/launch.py) - Integer dtypes give exact expected sums, unlike bfloat16 — source: `asserted`