MLX-LM LoRA and QLoRA versus full fine-tuning
Parent: MLX-LM training and adapter lifecycle · Published reference · snapshot 2026-10-23 · viability 4.80 / 5 (research)
↓ Facts as markdownall context files
Source-anchored research for MLX-LM LoRA and QLoRA versus full fine-tuning. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Before training, the entry point rejects num_layers greater than len(model.layers) with ValueError: Requested to train N layers but the model only has M layers. It freezes the model before selecting full-training layers or converting adapter modules. (low confidence) (verified as of 2026-10-23) [source]
How-to and procedures
- Configure adapter rank and module selection in YAML under lora_parameters: rank sets the low-rank dimension and keys supplies module paths such as self_attn.q_proj. Without keys, the converter discovers modules with to_lora or supported linear, switch-linear, and embedding types. --num-layers selects the last N model layers; -1 selects all. (medium confidence) (verified as of 2026-10-23) [source]
Problems, failure modes and limitations
- Memory and quality do not follow a universal ranking. In one 2026 FLAN-T5 Query-DSL experiment, full-precision fine-tuning had the best average evaluation loss among the configurations that could be run, while average memory ranking changed with model size: LoRA and QLoRA used more memory than full precision at FLAN-T5 Large, and QLoRA enabled FLAN-T5 XL under that study’s resource limit. The QLoRA paper reports preserving 16-bit task performance in its own evaluations, so results depend on experimental conditions. (medium confidence) [source]
- A 2026 theoretical preprint analyzes a simple linear-regression setting and identifies conditions where LoRA can have lower excess risk than full fine-tuning when the downstream task shift is effectively low-rank. This is a conditional result, not a general LLM quality guarantee; the EAMT translation study provides a contrasting task-specific result. (medium confidence) [source]
Comparisons and alternatives
- Pass the base model through --model <local_directory_or_Hugging_Face_repo>. --fine-tune-type accepts lora, dora, and full; fine_tune_type defaults to lora. These choices select adaptation mechanics; a quantized base is supplied as the model rather than selected by a separate qlora fine_tune_type. (medium confidence) (verified as of 2026-10-23) [source]
Facts and statements
- In the MLX-LM guide pinned to commit 0d4d3d04, adapter fine-tuning defaults to LoRA; a quantized input model uses QLoRA; and full-weight fine-tuning is exposed with `--fine-tune-type full`. The README also advertises low-rank and full fine-tuning with quantized-model support. (high confidence) (verified as of 2026-10-23) [source]
- LoRA freezes pretrained base weights and trains low-rank matrices, while full fine-tuning updates all original model parameters. LoRA therefore uses fewer trainable parameters, although its adapter matrices and optimizer states still consume memory. (high confidence) [source]
- QLoRA propagates gradients through a frozen quantized pretrained model into trainable LoRA adapters. The original QLoRA method uses a 4-bit base and introduces NF4, double quantization, and paged optimizers; MLX-LM documents QLoRA as its LoRA training path for quantized models. (high confidence) [source]
- LoRA’s quality relative to full fine-tuning is conditional. An LREC-COLING study reports comparable performance for appropriately ranked LoRA in high-resource multitask instruction tuning, while Thinking Machines reports a performance gap at larger batch sizes in its experiments; spectral-analysis research finds that LoRA and full fine-tuning can reach similar task performance with different weight-space structures. (high confidence) [source]
- Quality parity is task-dependent. The original LoRA paper reports on-par or better quality than full tuning on its evaluated RoBERTa, DeBERTa, GPT-2, and GPT-3 tasks, and the QLoRA paper reports preserving full 16-bit fine-tuning task performance on its evaluations. In contrast, an EAMT study of Llama-2 13B for business translation found full tuning superior to LoRA and QLoRA; its 4-bit QLoRA preliminary results were unsatisfactory, so the reported experiments used 8-bit QLoRA. (medium confidence) [source]
- lora_parameters.scale directly multiplies the low-rank update; LoRALinear returns base_linear(x) + scale*((dropout(x) @ lora_a) @ lora_b). It does not divide scale by rank. lora_parameters.dropout is passed as nn.Dropout's probability on the adapter input. The entry-point defaults are rank=8, scale=20.0, dropout=0.0. (medium confidence) (verified as of 2026-10-23) [source]
Children
- No children recorded.