mlx.launch remote teardown bug (tmp.None pid file)

Facts

• The bug is in `RemoteProcess.terminate` in `mlx/_distributed_utils/launch.py`: it builds a remote kill script from `self._pidfile`, which is only set after the first chunk of rank output has been read. If it was never set properly, the script reads a missing file and the rank is not killed. [source]
• The launch script created by `make_launch_script` runs `pidfile=$(mktemp); echo $$ > $pidfile; printf "%s\n" $pidfile;` and ends with `exec "${cmd[@]}"`, so `$$` is the pid of the rank itself and the pid file path is the first line printed. [source]
• `RemoteProcess.__init__` sets `self._pidfile = None`. `preprocess_output` takes the first newline-terminated line of the first data chunk it receives as the pid file path, and it does this for stdout and stderr alike because the caller passes both through it. [source]
• `terminate` first calls `self._process.terminate()` and waits, then runs `make_kill_script(self._pidfile)` through `ssh <host>`; the script is `pid=$(cat <pidfile>); if ps -p $pid ...; then kill $pid; echo 1; else echo 0; fi; rm <pidfile>`. [source]
• The kill call uses `check=True`. When the pid file is missing, `cat` fails, `ps -p` with an empty pid fails, `echo 0` prints, and the last command `rm <pidfile>` exits 1, so the ssh command returns 1 and Python raises `CalledProcessError`, matching the traceback in the write-up. [source]
• Because the error is raised inside the launcher's stop path, `exit_codes[rank]` is never recorded for that rank and later ranks in the loop may not be torn down. [source]
• The write-up quotes the code as `from mlx/_distributed_utils/launch.py` with a pid file path ending `tmp.None`; the current main branch fetched on 2026-10-04 still has the same structure (no guard for `_pidfile is None`, no timeout on the ssh kill). [source]
• The literal text `tmp.None` is not what Python's f-string would give for a `None` pid file (that would print `None`); the write-up shows a `/var/folders/.../tmp.None` path, so the exact value is unexplained by the source. One plausible route is that the first output line is not the `mktemp` path, for example a stderr warning or an ssh banner arriving first. [source]
• The kill sends SIGTERM (`kill $pid`), never SIGKILL, and only to the one rank pid. A rank that is already in uninterruptible sleep in the RDMA driver ignores it. [source]
• The pid file lives in the remote user's `$TMPDIR` (`mktemp` default, a `/var/folders/...` path on macOS) and is removed only by this kill script, so a launcher crash that skips it leaves a pid file behind. [source]
• The same path is used for the local rank when the host is `127.0.0.1`: `terminate` then runs the kill script without ssh. [source]
• Remote ranks are launched with `ssh -tt -o LogLevel=QUIET`; the forced tty means stderr and stdout share one stream on the remote side and `stty -echo` is run first. [source]
• None between sources; the write-up reports the symptom and the source shows the mechanism. The explanation for the exact string `tmp.None` is the only unresolved point. [source]
• Whether an upstream issue or PR exists for the `_pidfile is None` case; none was found in the cached pages. [source]
• Whether `ssh -tt` hangup alone (SIGHUP to the remote session) would stop a healthy rank, making the explicit kill redundant in the clean case. [source]
• `launch.py` stores `self._pidfile = None` at construction and fills it from the first output chunk's first line. [source]
• `preprocess_output` is applied to both stdout and stderr, so a first stderr line would be taken as the pid file path. [source]
• The remote script `exec`s the rank command, so the stored pid is the rank's own pid. [source]
• The kill script ends with `rm <pidfile>`, so a missing pid file makes the whole ssh command exit 1 and `check=True` raises `CalledProcessError`. [source]
• The teardown sends SIGTERM only. [source]
• The ssh kill call has no timeout and no fallback to SIGKILL on main as fetched on 2026-10-04. [source]
• The write-up's `tmp.None` text is not reproduced by `f"cat {None}"`; the origin of that string is unexplained. [source]