JACCL protection domain exhaustion on repeated mx.distributed.init
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
The library does carry release paths: `Connection` has a destructor and the verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq`. The reported leak is therefore not a missing call in JACCL's own code path as read here; it matches the reports that the kernel side does not return PDs af...
These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- The library does carry release paths: `Connection` has a destructor and the verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq`. The reported leak is therefore not a missing call in JACCL's own code path as read here; it matches the reports that the kernel side does not return PDs after teardown. [source]
- A larger mesh or more wires per pair burns the pool faster per init, so a 5-node mesh should hit the limit in fewer restarts than a 2-node pair. The reported counts do not fit this simple model (about 60 sessions at 2 nodes against 2 to 3 broadcasts at 5 nodes), so other consumers of the pool probably exist (orphans, probes). [source]
- Ring init allocates a PD, a completion queue and a queue pair per connection, `2 * n_conns` PDs per rank [source]
- Mesh init allocates a PD per connection to every peer, so a rank uses `(size - 1) * n_wires` PDs [source]
- JACCL registers each send and receive buffer to the PD of the connection it serves [source]
- JACCL's verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq` and `Connection` has a destructor [source]
- mlx-lm issue 955 recommends one N-node broadcast over N-1 sequential 2-node transfers to limit init cycles, and periodic reboots for heavy use [source]
- mlx-lm issue 955 says RDMA device names change after every reboot, so `mlx.distributed_config --auto-setup` must be rerun and the hostfile redistributed after a PD-exhaustion reboot [source]
Children
- No children recorded.