<!-- llms-explorer concept facts · https://llms-explorer.com/tree/uninterruptible-us-plus-rank-processes-in-the-th/ · pack 2026-10-05 · ~491 tokens -->

# Uninterruptible Us+ rank processes in the Thunderbolt RDMA driver

> `Us+` ranks drift into the state gradually after their launcher dies, so a sweep at the next restart can still kill the newest orphans

Parent: [Mac local LLMs: Clusters, RDMA, exo and ds4](https://llms-explorer.com/tree/mac-local-llms-clusters-rdma-exo-ds4/) · 1 facets · 7 facts · page: https://llms-explorer.com/tree/uninterruptible-us-plus-rank-processes-in-the-th/

## Facts

- `Us+` ranks drift into the state gradually after their launcher dies, so a sweep at the next restart can still kill the newest orphans — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- Orphaned ranks and their parent `sshd` both hang, so the orphan's ppid is the stuck `sshd`, not 1 — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- `launchd` KeepAlive restarts re-ran `mlx.launch` within seconds and accumulated peer orphans each cycle — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- The fix kills all processes matching the model's `--tmpdir` pattern with `kill -9` before launching, with no ppid filter — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- A PID-file lock released just before `exec` prevents a manual run and a launchd restart from killing each other's ranks, since an EXIT trap does not fire after `exec` — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- Killed `mlx_lm.share` ranks left temp directories of 817 GB and 868 GB on two nodes; kill processes before deleting them — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
- Ranks already in `Us+` survive `kill -9` and only a node reboot clears them — [source](https://valensas.com/blog/how-we-fixed-a-silent-killer-in-our-mac-studio-llm-cluster)
