MLX-LM native DoRA and quantized adapter mechanics
Parent: MLX-LM DoRA and distributed training extensions · Published reference · snapshot 2026-10-23 · viability 4.40 / 5 (research)
↓ Facts as markdownall context files
Source-anchored research for MLX-LM native DoRA and quantized adapter mechanics. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- As of 2026-10-23, MLX-LM exposes fine_tune_type=dora and maps nn.Linear and nn.QuantizedLinear layers to DoRALinear; its dispatcher raises for SwitchLinear and QuantizedSwitchLinear, so native DoRA conversion does not cover those MoE layer types. (high confidence) (verified as of 2026-10-23) [source]
- For an nn.QuantizedLinear base, DoRALinear.from_base recovers the logical input dimension from the packed weight width using 32 // bits; DoRALinear then dequantizes weight, scales, and biases using the layer's group_size, bits, and mode for its forward calculation. (high confidence) (verified as of 2026-10-23) [source]
- MLX-LM exposes `--fine-tune-type dora` and maps nn.Linear and nn.QuantizedLinear targets to DoRALinear; its DoRALinear.fuse merges the low-rank delta and magnitude scaling, then requantizes a quantized layer with its original group_size, bits, and mode unless dequantize=True. (high confidence) (verified as of 2026-10-23) [source]
- The fine-tuning entry point calls model.freeze() before converting selected layers to DoRA. The new DoRA wrapper introduces trainable lora_a, lora_b and magnitude m while retaining the frozen pretrained base layer; full fine-tuning instead explicitly unfreezes selected base layers. (high confidence) (verified as of 2026-10-23) [source]
Comparisons and alternatives
- For ordinary LoRA on a quantized base, MLX-LM retains the wrapped base layer and calls self.linear(x) before adding the low-rank branch; its guide calls LoRA training on a quantized model QLoRA. This preserves the base module's own forward implementation, unlike DoRALinear's explicit full-weight dequantization path. (high confidence) (verified as of 2026-10-23) [source]
- The adapter converter routes nn.QuantizedEmbedding to DoRAEmbedding, but DoRAEmbedding.from_base explicitly checks for nn.QuantizedLinear rather than nn.QuantizedEmbedding; the independent guide flags that the packed embedding path can therefore miss the intended guard and fail later on shape mismatch, so quantized embedding targets need version-specific verification. (medium confidence) (verified as of 2026-10-23) [source]
- DoRAEmbedding.from_base contains the exact error message "DoRAEmbedding does not yet support quantization." Its guard tests nn.QuantizedLinear rather than nn.QuantizedEmbedding, although the converter selects DoRAEmbedding for nn.QuantizedEmbedding. Do not interpret the message as proof that every quantized embedding is rejected cleanly. (high confidence) (verified as of 2026-10-23) [source]
Facts and statements
- DoRA reparameterizes pretrained weights into direction and magnitude, applying a low-rank LoRA update to direction while learning magnitude separately; MLX-LM's DoRALinear implements this with lora_a, lora_b, and a trainable per-output magnitude vector m. (high confidence) [source]
- For a QuantizedLinear base, MLX-LM's DoRALinear forward calls _dequantized_weight(), computes the base output with a dense x @ w.T, constructs the full adapted weight matrix, and computes its row norms on each call. A quantized base is therefore accepted, but the unmerged DoRA forward path does not preserve the base QuantizedLinear forward path; an independent practitioner report describes the same dequantization and dense-normalization behavior. (high confidence) (verified as of 2026-10-23) [source]
- A practitioner report dated 2026-08-08 measured 4.1 tokens/s for one DoRA adapter on a 4-bit Qwen base on an M5 Max and about 33 tokens/s after fusing the adapter or switching to LoRA. These are setup-specific measurements, not a portable speed ratio. (low confidence) (verified as of 2026-08-08) [source]
- In MLX-LM's DoRALinear, the learned base magnitude is stored per output row; each forward pass adds the scaled low-rank update, computes the adapted row norms with stop_gradient, and rescales the output by the ratio of base magnitude to adapted norm. (high confidence) (verified as of 2026-10-23) [source]
- DoRALinear uses scale directly: z=(dropout(x) @ lora_a) @ lora_b and out=x @ w.T + scale*z before magnitude normalization. The adapted weight is w + (scale*lora_b.T) @ lora_a.T. This code does not divide scale by rank; its quantized base w comes from mx.dequantize. (low confidence) (verified as of 2026-10-23) [source]
Children
- MLX-LM DoRA support for MoE SwitchLinear layers (frontier)
- Quantized DoRA inference performance measurement (frontier)
Frontier under this node: MLX-LM DoRA support for MoE SwitchLinear layers, Quantized DoRA inference performance measurement