# JACCL protection domain exhaustion on repeated mx.distributed.init > The library does carry release paths: `Connection` has a destructor and the verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq`. The reported leak is therefore not a missing call in JACCL's own code path as read here; it matches the reports that the kernel side does not return PDs af... Parent: [Mac local LLMs: Clusters, RDMA, exo and ds4](https://llms-explorer.com/tree/mac-local-llms-clusters-rdma-exo-ds4/) · 1 facets · 8 facts · page: https://llms-explorer.com/tree/jaccl-protection-domain-exhaustion-on-repeated-m/ ## Facts - The library does carry release paths: `Connection` has a destructor and the verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq`. The reported leak is therefore not a missing call in JACCL's own code path as read here; it matches the reports that the kernel side does not return PDs after teardown. — source: `asserted` - A larger mesh or more wires per pair burns the pool faster per init, so a 5-node mesh should hit the limit in fewer restarts than a 2-node pair. The reported counts do not fit this simple model (about 60 sessions at 2 nodes against 2 to 3 broadcasts at 5 nodes), so other consumers of the pool probably exist (orphans, probes). — source: `asserted` - Ring init allocates a PD, a completion queue and a queue pair per connection, `2 * n_conns` PDs per rank — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/ring.cpp) - Mesh init allocates a PD per connection to every peer, so a rank uses `(size - 1) * n_wires` PDs — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/mesh.cpp) - JACCL registers each send and receive buffer to the PD of the connection it serves — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/ring.cpp) - JACCL's verbs table includes `dealloc_pd`, `destroy_qp` and `destroy_cq` and `Connection` has a destructor — [source](https://raw.githubusercontent.com/ml-explore/mlx/main/mlx/distributed/jaccl/lib/jaccl/rdma.h) - mlx-lm issue 955 recommends one N-node broadcast over N-1 sequential 2-node transfers to limit init cycles, and periodic reboots for heavy use — [source](https://github.com/ml-explore/mlx-lm/issues/955) - mlx-lm issue 955 says RDMA device names change after every reboot, so `mlx.distributed_config --auto-setup` must be rerun and the hostfile redistributed after a PD-exhaustion reboot — [source](https://github.com/ml-explore/mlx-lm/issues/955)