# MLX-LM LoRA and QLoRA versus full fine-tuning > Source-anchored research for MLX-LM LoRA and QLoRA versus full fine-tuning. Run status: COMPLETED. Parent: [MLX-LM training and adapter lifecycle](https://llms-explorer.com/tree/mlx-lm-training-and-adapter-lifecycle/) · 5 facets · 11 facts · page: https://llms-explorer.com/tree/mlx-lm-lora-and-qlora-versus-full-fine-tuning/ ## Structure and components - Before training, the entry point rejects num_layers greater than len(model.layers) with ValueError: Requested to train N layers but the model only has M layers. It freezes the model before selecting full-training layers or converting adapter modules. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/lora.py) ## How-to and procedures - Configure adapter rank and module selection in YAML under lora_parameters: rank sets the low-rank dimension and keys supplies module paths such as self_attn.q_proj. Without keys, the converter discovers modules with to_lora or supported linear, switch-linear, and embedding types. --num-layers selects the last N model layers; -1 selects all. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/lora.py) ## Problems, failure modes and limitations - Memory and quality do not follow a universal ranking. In one 2026 FLAN-T5 Query-DSL experiment, full-precision fine-tuning had the best average evaluation loss among the configurations that could be run, while average memory ranking changed with model size: LoRA and QLoRA used more memory than full precision at FLAN-T5 Large, and QLoRA enabled FLAN-T5 XL under that study’s resource limit. The QLoRA paper reports preserving 16-bit task performance in its own evaluations, so results depend on experimental conditions. (medium confidence) — [source](https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2026.1665992/full) - A 2026 theoretical preprint analyzes a simple linear-regression setting and identifies conditions where LoRA can have lower excess risk than full fine-tuning when the downstream task shift is effectively low-rank. This is a conditional result, not a general LLM quality guarantee; the EAMT translation study provides a contrasting task-specific result. (medium confidence) — [source](https://arxiv.org/abs/2605.19018) ## Comparisons and alternatives - Pass the base model through --model . --fine-tune-type accepts lora, dora, and full; fine_tune_type defaults to lora. These choices select adaptation mechanics; a quantized base is supplied as the model rather than selected by a separate qlora fine_tune_type. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/lora.py) ## Facts and statements - In the MLX-LM guide pinned to commit 0d4d3d04, adapter fine-tuning defaults to LoRA; a quantized input model uses QLoRA; and full-weight fine-tuning is exposed with `--fine-tune-type full`. The README also advertises low-rank and full fine-tuning with quantized-model support. (high confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/LORA.md) - LoRA freezes pretrained base weights and trains low-rank matrices, while full fine-tuning updates all original model parameters. LoRA therefore uses fewer trainable parameters, although its adapter matrices and optimizer states still consume memory. (high confidence) — [source](https://arxiv.org/abs/2106.09685) - QLoRA propagates gradients through a frozen quantized pretrained model into trainable LoRA adapters. The original QLoRA method uses a 4-bit base and introduces NF4, double quantization, and paged optimizers; MLX-LM documents QLoRA as its LoRA training path for quantized models. (high confidence) — [source](https://arxiv.org/abs/2305.14314) - LoRA’s quality relative to full fine-tuning is conditional. An LREC-COLING study reports comparable performance for appropriately ranked LoRA in high-resource multitask instruction tuning, while Thinking Machines reports a performance gap at larger batch sizes in its experiments; spectral-analysis research finds that LoRA and full fine-tuning can reach similar task performance with different weight-space structures. (high confidence) — [source](https://aclanthology.org/2024.lrec-main.206/) - Quality parity is task-dependent. The original LoRA paper reports on-par or better quality than full tuning on its evaluated RoBERTa, DeBERTa, GPT-2, and GPT-3 tasks, and the QLoRA paper reports preserving full 16-bit fine-tuning task performance on its evaluations. In contrast, an EAMT study of Llama-2 13B for business translation found full tuning superior to LoRA and QLoRA; its 4-bit QLoRA preliminary results were unsatisfactory, so the reported experiments used 8-bit QLoRA. (medium confidence) — [source](https://arxiv.org/abs/2106.09685) - lora_parameters.scale directly multiplies the low-rank update; LoRALinear returns base_linear(x) + scale*((dropout(x) @ lora_a) @ lora_b). It does not divide scale by rank. lora_parameters.dropout is passed as nn.Dropout's probability on the adapter input. The entry-point defaults are rank=8, scale=20.0, dropout=0.0. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/lora.py)