MLX-LM integration contracts for external teacher student distillation

Parent: MLX-LM API and adapter contracts in external post-training integrations · Published reference · snapshot 2026-10-23 · viability 3.80 / 5 (research)

↓ Facts as markdown all context files

Source-anchored research for MLX-LM integration contracts for external teacher student distillation. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

Tokenizer equality is a requirement of direct token-level teacher/student comparison, not of text-only teacher-response SFT. AutoContext rejects a vocabulary mismatch for its local OPD path; a Hugging Face TRL issue records that passing student token IDs directly into a differently-tokenized teacher gives incorrect teacher probabilities. Cross-tokenizer logit training needs explicit alignment or another objective; `mlx-go-lm` exposes a token-logit remapper and counts unmapped tokens. (high confidence) (verified as of 2026-10-23) [source]
MLX-LM's DWQ is a scoped teacher-guided path: the documentation says it tunes quantized-model parameters such as scales and biases against a non-quantized teacher, and the implementation computes KL targets while updating the quantized model. This is weight-quantization distillation, not a general task-level teacher/student training interface; the September 2026 feature request explicitly distinguishes DWQ from general knowledge distillation. (high confidence) (verified as of 2026-10-23) [source]

Structure and components

AutoContext OPD writes adapters/adapters.safetensors from student.trainable_parameters() and adapter_config.json containing fine_tune_type=lora, num_layers, and lora_parameters. Pinned native load(base,adapter_path=that_directory) reads those two files, recreates adapter layers, and validates allowed names and shapes before loading. This is a concrete file-contract match for the matching base/config, not a live interoperability test; a standalone weights file or MLXSmith ADAPTER.txt placeholder does not satisfy it. (medium confidence) (verified as of 2026-10-23) [source]

How it works

Current external integrations separate offline teacher-text SFT from online token-level distillation. AutoContext describes its `mlxlm` route as SFT over a fixed teacher-written dataset and runs OPD in a separate in-process loop with a frozen teacher and LoRA student; MLXSmith likewise separates generated-response SFT from its OPD scoring mode. (high confidence) (verified as of 2026-10-23) [source]

Problems, failure modes and limitations

Token-level KL cannot compare raw teacher and student logits as if token IDs shared meaning when their tokenizers or vocabulary sizes differ. In TRL issue 4562, the maintainer says this constraint follows from GKD/MiniLLM's token-level objective; TRL's experimental GOLD trainer instead aligns text spans and uses ULD for cross-tokenizer cases. An MLX-LM integration must preserve a shared token coordinate system or implement an explicit alignment method; passing student token IDs directly to a different-tokenizer teacher is unsafe. (high confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

To adapt an external distillation objective to the pinned native trainer, supply train(student,optimizer,...,loss=my_loss,iterate_batches=my_batches): standard my_loss receives (model,token_batch,lengths) and returns (scalar_loss,token_count). Build an explicit completion/padding mask and token denominator rather than assuming the inclusive default_loss boundary excludes every padded target. The native loop differentiates only the supplied student module, sums gradients, averages across workers, divides by args.grad_accumulation_steps on updates, and calls the supplied optimizer; values below one are rejected. AutoContext instead calls nn.value_and_grad(student,loss_fn) in its own loop with teacher targets stopped. (medium confidence) (verified as of 2026-10-23) [source]
Native --resume-adapter-file calls model.load_weights(...,strict=False), then builds a new optimizer and starts the normal loop; it does not restore teacher identity/weights, optimizer moments, scheduler progress, RNG state, or rollout state. AutoContext OPD saves adapter tensors/config rather than those states, while MLXSmith may apply a source adapter but initializes a fresh optimizer and random.Random(seed). Exact external-training continuation therefore needs a separate checkpoint for that omitted state and resolved teacher/student revisions. (medium confidence) (verified as of 2026-10-23) [source]

Facts and statements

The documented MLX-LM LoRA CLI consumes JSONL chat, tools, completions, or text records. Its `--mask-prompt` option computes loss on completion tokens for chat and completion datasets; for chat data the final message is treated as the completion. (high confidence) (verified as of 2026-10-23) [source]
The Python `mlx_lm.tuner.trainer.train` function accepts a loss callable, differentiates that callable with `nn.value_and_grad`, and updates the supplied optimizer. External code can provide a student-logit objective at this lower-level seam; the stock LoRA CLI contract remains supervised fine-tuning. (high confidence) (verified as of 2026-10-23) [source]
Hosted teacher APIs can feed the offline text path: AutoContext documents API or hosted teacher trace collection before training, while the MedTech and Distill-SQL projects describe teacher-generated reasoning or labels rendered as training records and then learned with MLX-LM LoRA. (medium confidence) (verified as of 2026-10-23) [source]
`mlx-go-lm` documents a teacher-logit cache contract that writes precomputed targets in the same order used by training and streams cached logits batch by batch. An external MLX-LM integration that adopts cached logits therefore needs to preserve sample order and vocabulary coordinates across precompute and student training. (high confidence) (verified as of 2026-10-23) [source]
The documented `mlx_lm.lora` data contract accepts local JSONL with `chat`, `tools`, `completions`, or `text` records, one example per line. An external teacher's generated response can enter the student's ordinary SFT path after the caller stores it as a prompt/completion or message record. `--mask-prompt` supports completion-only loss for chat and completion datasets; the documented default computes loss for every token. (high confidence) (verified as of 2026-10-23) [source]
The Python `mlx_lm.tuner.trainer.train` contract accepts caller-supplied `loss` and `iterate_batches` functions plus a `training_callback` object implementing report methods. Its default loss shifts the token batch, computes masked next-token cross-entropy, and returns loss plus a token count. The current signature has no teacher-model argument, so direct logit distillation needs a custom loss and batch path or a separate training loop. (high confidence) (verified as of 2026-10-23) [source]
A hosted teacher that supplies response text gives supervised examples, not token-probability targets. AutoContext documents provider-generated rationales as SFT data and says token-level distillation needs local or open-weight teacher logits; MLX-LM's JSONL loader and custom-loss hooks provide the student-side data and training extension points for those two distinct paths. (high confidence) (verified as of 2026-10-23) [source]
AutoContext run_on_policy_distillation at f72c1541 imports mlx_lm.load and linear_to_lora_layers. It loads/freeze()s the teacher, obtains student and tokenizer from load(student_model), freezes the student base, checks _model_logit_vocab(student/teacher) via a one-token forward, then installs LoRA on num_layers with rank=8, scale=20.0, dropout=0.0. Prompts use the student tokenizer chat template. This validates output-logit dimension equality; it does not validate identical token-to-ID mappings between tokenizers. (low confidence) (verified as of 2026-10-23) [source]
AutoContext full_kl OPD feeds [B,T,V] student and teacher raw logits into its own reverse_kl_per_token, applies temperature-scaled log-softmax internally, stop_gradient to teacher logits, and computes KL(student||teacher). Its fixed-length student rollout has no early EOS and masks prediction positions P-1 through P+max_tokens-2; normalization uses the mask sum with an epsilon floor. Native kl_div_loss/js_div_loss also accept raw logits and return per-position losses, but substituting them requires matching the intended divergence orientation and completion mask, not merely matching tensor shapes. (medium confidence) (verified as of 2026-10-23) [source]
MLXSmith run_distill at 3737578f uses a different OPD route: the student generates token IDs and behavior logprobs, the teacher scores that completion (or supplies a sequence-average fallback), and importance_sampling_loss uses a teacher-minus-behavior advantage. It aligns/truncates completion scores and accumulates gradients over cfg.train.grad_accum before applying averaged student gradients. Its offline branch writes teacher text then calls run_sft. If loading raises BackendNotAvailable, it writes ADAPTER.txt and returns; that placeholder is not a trained native adapter. (low confidence) (verified as of 2026-10-23) [source]

Children

MLX-LM custom loss and batch iterator contract (frontier)
Cross-tokenizer alignment for teacher/student logits (frontier)
Teacher-generated dataset filtering and leakage control (frontier)

Frontier under this node: Cross-tokenizer alignment for teacher/student logits, MLX-LM custom loss and batch iterator contract, Teacher-generated dataset filtering and leakage control

← the whole tree · 3D view· how to read this page