MLX-LM training and adapter lifecycle

Parent: MLX and mlx-lm on Apple silicon · Published reference · snapshot 2026-10-23

Source-anchored research for MLX-LM training and adapter lifecycle. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Structure and components

Before training, the entry point rejects num_layers greater than len(model.layers) with ValueError: Requested to train N layers but the model only has M layers. It freezes the model before selecting full-training layers or converting adapter modules. (low confidence) (verified as of 2026-10-23) [source]
The current MLX Adam and AdamW implementations default bias_correction to false. Treat optimizer settings as part of the run configuration when comparing learning-rate experiments or porting settings from another framework. (high confidence) (verified as of 2026-10-23) [source]
MLX-LM's guide also recommends reducing `--batch-size`, `--num-layers`, or example length to lower memory; it cautions that training fewer layers may reduce quality on data-heavy runs, while research treats microbatching and checkpointing as separate techniques with workload-dependent memory and compute tradeoffs. (medium confidence) [source]
--grad-checkpoint enables training gradient checkpointing. --batch-size reduces examples in each minibatch, while --num-layers reduces the last-layer subset selected for training. For one worker, batch_size B and --grad-accumulation-steps A combine A minibatches before an optimizer update, giving an effective example batch size B*A. (medium confidence) (verified as of 2026-10-23) [source]

How it works

The current `mlx_lm.fuse` command loads a base model and adapter directory, fuses model modules that expose a `fuse` method, and saves the combined model at `--save-path`; the defaults are `mlx_model`, `adapters`, and `fused_model`. It can also upload the saved model when given `--upload-repo`. Apple describes the fused result as a self-contained model for simpler deployment and sharing. (high confidence) (verified as of 2026-10-23) [source]
The current schedule builder uses the named MLX optimizer schedule with the configured arguments; when warmup is enabled, it linearly ramps from warmup_init to arguments[0] and joins the main schedule at warmup_steps + 1. Inspect the logged learning rate to verify that the configured schedule follows the intended trajectory. (high confidence) (verified as of 2026-10-23) [source]
Load YAML training settings with --config <file.yaml> or -c <file.yaml>. main() copies each YAML value only when the parsed CLI value for that key is None, then fills remaining None values from CONFIG_DEFAULTS. Explicit non-None CLI values take precedence; parser defaults that are already non-None can also block a YAML override. (low confidence) (verified as of 2026-10-23) [source]
For a local --data directory, MLX-LM reads JSONL records from train.jsonl for training, accepts valid.jsonl as an optional validation split, and reads test.jsonl for testing. The current loader returns an empty split when a file is missing, then raises for an empty training or test split and warns but continues when validation is empty. If the --data path does not exist, the loader treats it as a Hugging Face dataset ID. (high confidence) (verified as of 2026-10-23) [source]
Prompt masking is disabled by default. Without --mask-prompt, MLX-LM computes loss for every sample token; with it, the documented supported formats are chat and completion, and the final chat message is treated as the completion. The current loader raises ValueError when prompt masking is requested for a text dataset. (high confidence) (verified as of 2026-10-23) [source]
Chat masking computes one prefix offset from all messages except the final message. Earlier assistant actions in a multi-turn tool trace are therefore masked when a later tool result or answer follows them; a practitioner report documented this behavior and split rows so each assistant action became a final-message target. Keep the intended assistant target last in each row. (medium confidence) (verified as of 2026-10-23) [source]
A CompletionsDataset prompt-masking crash was reported in issue #786 and addressed by PR #967, which merged into main on 2026-03-11 and changed the mask calculation to template a prompt-only message list. The current loader uses that prompt-only list when computing the completion offset. (high confidence) (verified as of 2026-10-23) [source]
train() rejects grad_accumulation_steps below one with ValueError("grad_accumulation_steps must be at least 1"). It sums successive microbatch gradients; at an update it calls average_gradients, divides each accumulated gradient by grad_accumulation_steps when greater than one, then calls optimizer.update. (low confidence) (verified as of 2026-10-23) [source]

How-to and procedures

Use an unfused adapter for generation with mlx_lm.generate --model <base_model> --adapter-path <adapter_directory> --prompt "<prompt>". LORA.md documents --upload-repo as the fused-model upload destination and --hf-path as the original model for its card, but the pinned fuse.py parser has no --hf-path option. Treat that documented upload invocation as a guide/parser disagreement and inspect the installed mlx_lm.fuse --help before publishing. (high confidence) (verified as of 2026-10-23) [source]
Configure adapter rank and module selection in YAML under lora_parameters: rank sets the low-rank dimension and keys supplies module paths such as self_attn.q_proj. Without keys, the converter discovers modules with to_lora or supported linear, switch-linear, and embedding types. --num-layers selects the last N model layers; -1 selects all. (medium confidence) (verified as of 2026-10-23) [source]
Use consistent JSONL row shapes: text records contain {"text":"..."}; completion records contain {"prompt":"...","completion":"..."}; chat records contain {"messages":[{"role":"user","content":"..."},{"role":"assistant","content":"..."}]}. prompt_feature and completion_feature rename completion columns; chat_feature and text_feature rename the other formats. (low confidence) (verified as of 2026-10-23) [source]

Problems, failure modes and limitations

On `mlx-lm` 0.31.3 with `mlx` 0.32.0, issue #1723 reported `[save_gguf] can only serialize row-major arrays` during `--export-gguf` because `save()` ignored `donate_model=False` and emptied weights before conversion. Upstream merged PR #1743 on 2026-08-17 to pass that flag through. Treat the error as version-specific and check the installed build before reapplying workarounds. (high confidence) (verified as of 2026-08-17) [source]
A 2026 llama.cpp issue reports that its separate `convert_hf_to_gguf.py` path rejected `lm_head.scales` and sometimes `lm_head.biases` in an MLX-fused artifact. That report is specific to the downstream converter and does not establish a failure in MLX-LM's built-in `--export-gguf` implementation. (low confidence) [source]
Memory and quality do not follow a universal ranking. In one 2026 FLAN-T5 Query-DSL experiment, full-precision fine-tuning had the best average evaluation loss among the configurations that could be run, while average memory ranking changed with model size: LoRA and QLoRA used more memory than full precision at FLAN-T5 Large, and QLoRA enabled FLAN-T5 XL under that study’s resource limit. The QLoRA paper reports preserving 16-bit task performance in its own evaluations, so results depend on experimental conditions. (medium confidence) [source]
A 2026 theoretical preprint analyzes a simple linear-regression setting and identifies conditions where LoRA can have lower excess risk than full fine-tuning when the downstream task shift is effectively low-rank. This is a conditional result, not a general LLM quality guarantee; the EAMT translation study provides a contrasting task-specific result. (medium confidence) [source]
For local data, train.jsonl is the training split, valid.jsonl supplies validation, and test.jsonl supplies --test evaluation. A missing file becomes an empty split: the final loader rejects empty training/test splits when requested and warns for missing validation. A zero-line existing JSONL file instead reaches create_dataset(data[0]) without an empty-data guard and can fail before those final split checks; ensure every supplied split file has records. (low confidence) (verified as of 2026-10-23) [source]
A June 2025 user report recorded Metal `Insufficient Memory` at a reported 34.108 GB peak with batch size 1, `--grad-checkpoint`, and a 5,000-token maximum sequence; after reinstalling MLX-LM from source, the follow-up log reached iteration 100, so the report does not isolate a memory-control limit or give a portable capacity estimate. (low confidence) [source]
Activation memory grows with batch size and sequence length, so the MLX-LM guide recommends lowering `--batch-size` and shortening long examples when memory is constrained; it notes that smaller batches may slow training. (high confidence) (verified as of 2026-10-23) [source]
iterate_batches truncates examples longer than max_seq_length, emits a warning, and updates the recorded lengths to the truncated lengths. If the dataset contains fewer examples than batch_size, it raises ValueError stating that the dataset must have at least batch_size examples. Pre-split long examples and choose a batch size no larger than the dataset. (low confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

--resume-adapter-file loads model weights with model.load_weights(..., strict=False). It does not restore optimizer state or a previous iteration counter in this entry point, so it is a weight-based continuation rather than a complete training-state restart. (low confidence) (verified as of 2026-10-23) [source]
Pass the base model through --model <local_directory_or_Hugging_Face_repo>. --fine-tune-type accepts lora, dora, and full; fine_tune_type defaults to lora. These choices select adaptation mechanics; a quantized base is supplied as the model rather than selected by a separate qlora fine_tune_type. (medium confidence) (verified as of 2026-10-23) [source]
Learning-rate selection is consequential and should be tested for the model and run: an LLM learning-rate study argues that policies designed for traditional DNN training may not transfer directly, and a reported mlx-lm 0.31.3 case saw one 1e-4 run produce 162 errors on a held-out task versus 18 errors in a later 1e-5 run. The author notes that the example order also changed and there was only one run per setting, so it does not establish a universal rate or isolate learning rate as the sole cause. (medium confidence) [source]
An mlx-lm issue reports that the same stated data and configuration produced different training and validation loss behavior after upgrading from 0.21.5 to 0.22.0. The thread does not establish the cause, so this is a version-specific user report rather than evidence of a confirmed regression. (low confidence) [source]
--steps-per-report controls training report intervals; --steps-per-eval controls validation intervals; --val-batches -1 selects the entire validation iterator instead of a fixed number of batches. on_train_loss_report receives iteration, train_loss, learning_rate, iterations_per_second, tokens_per_second, trained_tokens, and peak_memory. (medium confidence) (verified as of 2026-10-23) [source]
Prompt masking is not a universal best choice. A 2026 TACL survey reports that including prompt-token loss can improve generalization in some long-instruction, low-data settings, while a prompt-loss-weighting study says the useful weight depends on the task and dataset. Evaluate masking choices on held-out task outputs instead of assuming completion-only loss always wins. (medium confidence) [source]
A 2024 practitioner experiment using Llama-2-7b on a RACE subset reported that completion-only training reached 80% validation accuracy versus 75% with full-sequence loss and converged faster. The author cautioned that the result may not generalize beyond that dataset and setup. (low confidence) [source]
MLX-LM's `--grad-checkpoint` trades extra computation for lower activation memory by recomputing intermediate values needed during backpropagation; its guide says the option is more helpful with larger batches or sequences. Framework documentation describes the same recomputation tradeoff and notes that checkpoint placement affects peak-memory savings. (high confidence) (verified as of 2026-10-23) [source]
As of 2026-10-23, open MLX-LM issue #1206 contains reports of Qwen3.5 training OOM despite `--grad-checkpoint` on M5 Max and M3 Pro systems, alongside a base-M5 report that the OOM was not reproducible and training completed normally through iteration 5; treat this as configuration-specific evidence rather than a general memory limit. (low confidence) (verified as of 2026-10-23) [source]

Facts and statements

Current MLX-LM training writes the current adapter weights to `adapters.safetensors` and, at each configured save interval, also writes a numbered `<step>_adapters.safetensors` snapshot beside it. The loader called by fusion reads `adapter_config.json` and `adapters.safetensors` from the `--adapter-path` directory, so using a numbered snapshot requires exposing the chosen weights under the expected `adapters.safetensors` name beside the corresponding config. (high confidence) (verified as of 2026-10-23) [source]
The current `--export-gguf` path accepts only `model_type` values `llama`, `mixtral`, and `mistral`; the docs also scope support to those model styles at fp16 and give `fused_model/ggml-model-f16.gguf` as the default output, overridable with `--gguf-path`. The optional `--dequantize` path dequantizes before saving and removes quantization metadata from the saved config. A `gpt_oss` export rejection was reported through the same model-type gate. (high confidence) (verified as of 2026-10-23) [source]
Two case reports show quantized fusion can change observed behavior: an independent test found a rank-8 adapter's behavior disappeared after fusion into a 4-bit base without `--dequantize`, while issue #654 reports that direct fusion changed hidden reasoning for `gpt_oss-20b` and that dequantizing before a separate 4-bit conversion restored it. These reports are model-specific and do not establish a general degradation rate; evaluate the target model after fusion. (medium confidence) (verified as of 2026-07-30) [source]
The --adapter-path option chooses the adapter directory. Training creates that directory and writes adapters.safetensors plus adapter_config.json there; --save-every controls the periodic checkpoint interval by populating TrainingArgs.steps_per_save. (high confidence) (verified as of 2026-10-23) [source]
In the MLX-LM guide pinned to commit 0d4d3d04, adapter fine-tuning defaults to LoRA; a quantized input model uses QLoRA; and full-weight fine-tuning is exposed with `--fine-tune-type full`. The README also advertises low-rank and full fine-tuning with quantized-model support. (high confidence) (verified as of 2026-10-23) [source]
LoRA freezes pretrained base weights and trains low-rank matrices, while full fine-tuning updates all original model parameters. LoRA therefore uses fewer trainable parameters, although its adapter matrices and optimizer states still consume memory. (high confidence) [source]
QLoRA propagates gradients through a frozen quantized pretrained model into trainable LoRA adapters. The original QLoRA method uses a 4-bit base and introduces NF4, double quantization, and paged optimizers; MLX-LM documents QLoRA as its LoRA training path for quantized models. (high confidence) [source]
LoRA’s quality relative to full fine-tuning is conditional. An LREC-COLING study reports comparable performance for appropriately ranked LoRA in high-resource multitask instruction tuning, while Thinking Machines reports a performance gap at larger batch sizes in its experiments; spectral-analysis research finds that LoRA and full fine-tuning can reach similar task performance with different weight-space structures. (high confidence) [source]
Quality parity is task-dependent. The original LoRA paper reports on-par or better quality than full tuning on its evaluated RoBERTa, DeBERTa, GPT-2, and GPT-3 tasks, and the QLoRA paper reports preserving full 16-bit fine-tuning task performance on its evaluations. In contrast, an EAMT study of Llama-2 13B for business translation found full tuning superior to LoRA and QLoRA; its 4-bit QLoRA preliminary results were unsatisfactory, so the reported experiments used 8-bit QLoRA. (medium confidence) [source]
lora_parameters.scale directly multiplies the low-rank update; LoRALinear returns base_linear(x) + scale*((dropout(x) @ lora_a) @ lora_b). It does not divide scale by rank. lora_parameters.dropout is passed as nn.Dropout's probability on the adapter input. The entry-point defaults are rank=8, scale=20.0, dropout=0.0. (medium confidence) (verified as of 2026-10-23) [source]
The current MLX-LM guide accepts YAML through -c/--config, and a repeated command-line flag overrides the corresponding YAML key. Apple's WWDC25 walkthrough says the config file controls learning-rate schedules, optimizer settings, and evaluation intervals. (high confidence) (verified as of 2026-10-23) [source]
The guide says an optional valid.jsonl enables periodic validation-loss reporting; --test evaluates test.jsonl and reports test-set perplexity. It also supports sending training metrics to Weights & Biases and/or SwanLab. (low confidence) (verified as of 2026-10-23) [source]
Training and validation loss should be paired with a task-level evaluation: an MLX-LM practitioner case study reported validation loss improving while hard-set macro-F1 remained very poor, and a separate local-LLM evaluation guide recommends task-specific measures or human review alongside training metrics. (medium confidence) [source]
For memory-constrained training, the current guide recommends lowering --batch-size from its default of 4 to 2 or 1 at possible speed cost; --grad-accumulation-steps can increase effective batch size without increasing memory, and --grad-checkpoint trades memory for computation. (low confidence) (verified as of 2026-10-23) [source]
MLX-LM exposes configuration for iteration count, batch size, validation batches and intervals, loss-report intervals, gradient accumulation, maximum sequence length, learning-rate schedules, and optimizer settings; these controls let a run be adjusted to its training and evaluation needs. (high confidence) (verified as of 2026-10-23) [source]
The current trainer reports training loss, the realized learning rate, iterations per second, tokens per second, cumulative trained tokens, and peak memory; validation reports validation loss and elapsed time at the first iteration, configured evaluation intervals, and the final iteration. (high confidence) (verified as of 2026-10-23) [source]
Training or validation loss alone can misstate task quality: in the reported mlx-lm 0.31.3 case, validation loss reached 0.39 while held-out macro-F1 fell to 0.05–0.09 from 0.64–0.78 for the untrained model; a separate BERT fine-tuning study also found different test performance among runs with the same training loss. Pair loss curves with metrics that represent the intended task. (medium confidence) [source]
--learning-rate sets the scalar learning rate. --optimizer accepts adam, adamw, muon, sgd, and adafactor. Put optimizer-specific keyword arguments such as weight_decay under optimizer_config.<optimizer_name> in YAML; the entry point passes that selected dictionary to the optimizer constructor. (low confidence) (verified as of 2026-10-23) [source]
For a schedule, set lr_schedule.name to an mlx.optimizers.schedulers function and lr_schedule.arguments to its positional arguments. An optional warmup count builds a linear schedule from warmup_init (default 0.0) to arguments[0], then joins it to the requested schedule at warmup+1. A truthy lr_schedule takes precedence over the scalar learning_rate. (medium confidence) (verified as of 2026-10-23) [source]
For Hugging Face datasets, YAML fields such as prompt_feature, completion_feature, chat_feature, and text_feature map dataset columns to MLX-LM inputs. MLX-LM applies the model's chat template to chat, tools, and completion records by default; Hugging Face documents that chat templates and control tokens vary by model, so the training serialization should match the tokenizer. (high confidence) (verified as of 2026-10-23) [source]
The current loader chooses a row format from the first record: prompt and completion keys take precedence, then the chat key (messages by default), then the text key (text by default). Configurable feature names can rename those keys. Completion rows become a user/assistant message pair, and chat rows can pass a tools field to the tokenizer's chat template, so records within a split should use a consistent shape. (high confidence) (verified as of 2026-10-23) [source]
MLX-LM calculates the prompt boundary in token space with tokenizer.apply_chat_template: chat data uses the template for all messages except the last, and completion data uses a prompt-only message with add_generation_prompt=True. Hugging Face documents that chat templates serialize messages into model-specific token sequences and that generation prompts affect this serialization, so the tokenizer and template used for training must match the intended format. (high confidence) (verified as of 2026-10-23) [source]
Research does not establish completion-only loss as a universal optimum. The 2024 EMNLP study reports that useful prompt-token weights varied with generation length and benchmark, while the 2025 TACL study reports that the standard zero-prompt/full-response objective was often suboptimal across its tested models, datasets, and benchmarks; target-specific evaluation is needed before choosing masking. (medium confidence) [source]
--mask-prompt removes loss on the prompt prefix. Completion masking templates the user prompt alone with add_generation_prompt=True to locate that prefix; chat masking templates all messages except the last. Chat and completion records support masking; text records raise ValueError("Prompt masking not supported for text dataset."). (low confidence) (verified as of 2026-10-23) [source]
MLX-LM's memory guide says `--grad-accumulation-steps N` accumulates gradients across N batches before updating, so users can keep each microbatch small while increasing effective batch size without increasing batch memory; Hugging Face's training documentation describes the same effective-batch relationship. (high confidence) (verified as of 2026-10-23) [source]

Children

Dataset format validation (frontier)
Chat-template compatibility (frontier)
Per-turn loss masking (frontier)
MLX-LM adapter checkpoints fusion and export
MLX-LM LoRA and QLoRA versus full fine-tuning
MLX-LM training configuration optimization and metrics
MLX-LM training datasets and prompt masking
MLX-LM training memory and gradient control

← the whole tree · 3D view· how to read this page