LLM Fine-Tuning & PEFT

Parent: LLM Models and APIs · researched 2026-05-31T21:21:01.962Z· 20 sources · 11 concepts · skill llm-fine-tuning-peft

Adapting a pretrained LLM to a specific task, domain, format, or behavior by

LLM Fine-Tuning & PEFT

Adapting a pretrained LLM to a specific task, domain, format, or behavior by [source]
continuing training on labeled examples - and doing it cheaply with [source]
parameter-efficient fine-tuning (PEFT), which freezes the base model and [source]
trains a tiny set of new weights instead of all of them. [source]
Five things this reference answers: [source]
Should I even fine-tune? Full FT vs PEFT vs RAG vs prompting. [source]
How does LoRA work, and how do I set its knobs? rank, alpha, target modules, init. [source]
Which PEFT method? LoRA family (QLoRA / DoRA / rsLoRA / LoRA+), adapters, (IA)^3, prefix/P-tuning/prompt-tuning. [source]
How do I run it? The HuggingFace PEFT workflow, SFT data prep + chat templating, the tooling stack. [source]
How do I ship it? Merge vs swap for serving, catastrophic-forgetting mitigation, and evaluating the result. [source]

Scope boundary (read first)

This reference = supervised fine-tuning + the PEFT method zoo. SFT data [source]
preparation and chat templating are here because they are the input to any [source]
fine-tune (PEFT or full). [source]
Preference optimization / RLHF / DPO - turning a preference signal [source]
(pairwise comparisons, reward models) into model behavior - is the sibling [source]
references/llm-alignment-post-training.md. SFT is the post-training *base [source]
step* that precedes RLHF/DPO; once you have preference data, go there. The [source]
DPO-variant family, PPO loop, reward modeling, and alignment eval all live there. [source]
Quantization algorithm internals. QLoRA fine-tunes LoRA adapters on top of a [source]
frozen 4-bit NF4 base. The NF4 data type, double quantization, and the [source]
PTQ/QAT landscape are the sibling references/llm-compression.md reference — [source]
this reference treats NF4 as a black-box dependency of QLoRA. [source]
**Multi-LoRA serving runtime. Deciding merge vs swap** and what an adapter [source]
costs at inference is here. Tuning the engine that serves many adapters [source]
(S-LoRA/Punica kernels inside vLLM, PagedAttention, continuous batching, [source]
autoscaling) is the sibling references/llm-inference-serving.md. [source]
Offline benchmark-harness mechanics (running MMLU/HELM, LLM-as-judge [source]
scaffolding) → da-7-machine-learning. The fine-tune-specific eval design [source]
(held-out task set + base-model regression check) is here. [source]
Reasoning RL (GRPO/RLVR/DeepSeek-R1-style) is neither SFT nor preference [source]
optimization → the reasoning-models material (pointer only). [source]

Part 1 — Should you fine-tune at all? (the decision framework)

Fine-tuning is the most expensive and slowest of the three adaptation levers. [source]
Climb the ladder; stop at the first rung that clears your quality bar. [source]
The canonical order (2025 consensus): start with prompt engineering → [source]
add RAG when you need current/proprietary knowledge → fine-tune only when [source]
behavior stays inconsistent after prompts and RAG, or when a **small fine-tuned [source]
model is cheaper than a large general one** on your narrow task. [source]
Knowledge vs behavior is the load-bearing distinction. RAG is for *what the [source]
model knows; fine-tuning is for how the model acts*. Fine-tuning is a poor way [source]
to inject facts (they go stale and the model still hallucinates around them) and [source]
RAG is a poor way to fix formatting/tone. [source]
They compose. The highest-performing production systems often do both: [source]
fine-tune to shape behavior/format/domain reasoning, RAG to supply current facts [source]
at inference. Fine-tuning and RAG are not mutually exclusive. [source]
Good fine-tune use cases: consistent structured output prompts can't enforce; [source]
domain-specific reasoning absent from pretraining; style/tone calibration beyond [source]
what prompts achieve; cost optimization (a fine-tuned 8B beating a prompted [source]
70B on your task at a fraction of the inference cost); behavior cloning / distillation. [source]

Part 2 — Full fine-tuning vs PEFT

Full fine-tuning (FFT) updates every weight. PEFT freezes the base and [source]
trains a small add-on (often <1% of params). The trade-off is memory/cost vs [source]
Memory. FFT of a 7B model needs ~100-120 GB VRAM (weights + gradients + [source]
Adam's two moments + activations, all in fp16/bf16 → roughly 16-20 bytes/param). [source]
The same model fine-tunes with QLoRA on a single 24 GB RTX 4090. PEFT broadly [source]
cuts training memory 10-20x while retaining 90-95%+ of FFT quality on [source]
typical adaptation tasks. You also store a few-MB adapter instead of a full [source]
model checkpoint per task. [source]
When PEFT (LoRA) is ~equal to FFT: instruction-following, style transfer, [source]
classification, most NLU (GLUE/SuperGLUE). Well-configured LoRA reaches [source]
When FFT still wins: large new-knowledge infusion (continued pretraining on [source]
20B tokens) and hard generative skills (code, math). The "LoRA Learns Less and [source]
Forgets Less" paper (Biderman et al., 2024) found LoRA **substantially [source]
underperforms FFT** on programming and math in both instruction-tuning (~100K [source]
pairs) and continued-pretraining (20B tokens) regimes, because full fine-tuning [source]
learns weight perturbations with a rank 10-100x higher than typical LoRA [source]
configs, so low rank is genuinely capacity-limited there. [source]
The upside of "learning less": the same paper shows LoRA forgets less. [source]
It better preserves the base model's out-of-domain capabilities and maintains more [source]
diverse generation, acting as a **stronger regularizer than weight decay or [source]
dropout. So the FFT-vs-LoRA choice is a plasticity-vs-stability** trade: [source]
FFT for max new capability, LoRA when retaining general ability and avoiding [source]
> Closing the gap: a 2024-2026 line of work ("LoRA vs Full Fine-tuning: An [source]
> Illusion of Equivalence") argues even when LoRA matches FFT on the target [source]
> metric it does so via "intruder dimensions" (new singular directions [source]
> unlike the pretrained weights) which drive forgetting. The practical levers: [source]
> raise the rank and apply LoRA to all linear layers (Part 4) to behave [source]
> more like FFT, or use the intruder-dimension mitigation (Part 8). [source]

Part 3 — LoRA mechanics (the one method to understand deeply)

LoRA (Low-Rank Adaptation; Hu et al., 2021) freezes the pretrained weight [source]
matrix W ∈ R^(d×k) and learns a low-rank update: W' = W + ΔW = W + (α/r)·BA, [source]
where B ∈ R^(d×r), A ∈ R^(r×k), and r ≪ min(d,k). Only A and B train. [source]
The hypothesis: the update a model needs for a downstream task has low [source]
"intrinsic rank," so a thin BA product captures it with a fraction of the params. [source]

The four knobs

r (rank) - capacity. Small r = fewer params, cheaper, more [source]
regularization; large r = more capacity but more memory and overfitting risk. [source]
Rules of thumb: r=4-8 for easy/well-covered tasks (classification, [source]
sentiment); r=16-32 typical for instruction tuning; r=64-256 when [source]
approaching FFT quality on hard tasks (code/math) per "LoRA Learns Less." When in [source]
doubt start at r=16 and sweep. [source]
lora_alpha (α) - scaling. The update is scaled by α/r. α controls how [source]
strongly the adapter speaks relative to the frozen base. The widespread [source]
heuristic is α = 2·r (e.g. r=16 → α=32). Because the effective scale is [source]
α/r, raising r without raising α shrinks each update: this is exactly the [source]
pathology rsLoRA fixes (Part 5). [source]
target_modules - where. Which nn.Linear layers get an adapter. Original [source]
LoRA targeted only attention q_proj, v_proj (PEFT's default). Modern best [source]
practice (QLoRA, "LoRA Learns Less") is target_modules="all-linear": every [source]
linear layer including the MLP (gate_proj/up_proj/down_proj) and [source]
k_proj/o_proj, which closes most of the gap to FFT at modest extra cost. For [source]
MoE models whose experts are fused nn.Parameter tensors, use target_parameters. [source]
lora_dropout - regularization. Dropout on the LoRA path (e.g. 0.05-0.1 for [source]
small datasets, 0 for large clean ones). [source]

Initialization

Default PEFT init: A ~ Kaiming-uniform, B = zeros → BA = 0 at start, so [source]
the adapter begins as an identity transform (training starts exactly at the [source]
base model - critical for stability). init_lora_weights="gaussian" uses a [source]
Gaussian A (Diffusers convention). Data-driven inits that converge faster / [source]
preserve knowledge better: PiSSA (principal singular values/vectors of W), [source]
OLoRA (QR decomposition), EVA (SVD of input activations + adaptive [source]
per-layer rank via rho), CorDA (task- or knowledge-oriented decomposition, [source]
KPM mode mitigates forgetting), LoRA-GA (aligns to FFT gradient), and [source]
LoftQ (init to minimize quantization error for QLoRA). [source]

Why LoRA is "free" at inference

Because ΔW = (α/r)BA is just a matrix, you can fold it into W after [source]
training (W' = W + ΔW) → a standalone model with **zero added latency or [source]
params**. This is the merge path (Part 7). Keep it unmerged only when you need [source]

Part 4 — The LoRA family: QLoRA, DoRA, rsLoRA, LoRA+

These keep LoRA's low-rank update but fix a specific weakness. [source]

QLoRA (Dettmers et al., 2023) — memory

Fine-tune LoRA adapters on top of a base model quantized to 4-bit, so the [source]
frozen weights occupy ~1/4 the VRAM while gradients flow through them in bf16. [source]
Three ingredients: (1) NF4 (4-bit NormalFloat, information-theoretically [source]
optimal for the ~normally-distributed weights - *internals live in [source]
llm-compression.md*), (2) double quantization (quantize the quantization [source]
constants too), (3) paged optimizers (page optimizer state to CPU to survive [source]
memory spikes). Result: fine-tune a 65-70B model on a single 48 GB GPU with [source]
quality matching 16-bit LoRA and 16-bit FFT. Enable in PEFT by loading the base [source]
with a bitsandbytes 4-bit quantization_config, then attaching LoRA as usual. [source]
Pair with LoftQ init for best quantized-training quality. "QDoRA" = QLoRA + DoRA. [source]

DoRA (Liu et al., ICML 2024) — low-rank quality

Weight-Decomposed LoRA. Decompose each weight into magnitude (a scalar [source]
vector) and direction; let LoRA update only the direction while a separate [source]
learnable parameter handles magnitude. This decoupling makes DoRA's learning [source]
pattern closer to FFT and beats LoRA especially at low rank (r=4-8) on [source]
commonsense reasoning and multimodal tasks, with no extra inference cost once [source]
merged. Enable: LoraConfig(use_dora=True). Caveats: bigger training overhead [source]
than plain LoRA (mitigated by DoraCaching / ephemeral_gpu_offload); supports [source]
linear/embedding/Conv2d only; merge for inference to erase the overhead. [source]

rsLoRA (Kalajdzievski, 2023) — stable high rank

Rank-Stabilized LoRA changes the scaling from α/r to α/√r. With the [source]
original α/r, gradients collapse as r grows, so large ranks learn no better [source]
than small ones (the reason "just raise the rank" historically failed). With [source]
α/√r gradients stay healthy and higher ranks finally pay off: better [source]
perplexity/quality at large r, zero inference cost. Enable: [source]
LoraConfig(use_rslora=True). Use it whenever you want r ≥ 32. [source]

LoRA+ (Hayou et al., 2024) — efficient feature learning

Vanilla LoRA updates A and B with the same learning rate, which is [source]
provably suboptimal for feature learning in wide models. LoRA+ uses a **higher LR [source]
for B** than A by a fixed ratio (loraplus_lr_ratio, e.g. 16). Result: [source]
~1-2% accuracy and up to ~2x faster convergence at the same compute. [source]
Enable via create_loraplus_optimizer(...). (Related: a 2026 line of work argues [source]
careful LR tuning alone often suffices, so always tune LR before reaching for [source]
> Picking within the family: start LoRA; tight on VRAM → QLoRA; low [source]
> rank but want more quality → DoRA; want high rank to work → rsLoRA; want [source]
> faster/slightly-better at no cost → LoRA+. They compose (e.g. QLoRA + rsLoRA + LoRA+). [source]

Part 5 — The other PEFT families (non-LoRA)

PEFT methods differ in where they put the new parameters. (Survey framing: [source]
Han et al. 2024; HuggingFace PEFT.) [source]
Adapters (Houlsby 2019 / Pfeiffer 2021). Insert small **bottleneck MLP [source]
modules** (down-project → nonlinearity → up-project, with residual) inside each [source]
transformer block. Houlsby = two adapters per layer (after attention and [source]
after FFN); Pfeiffer = one (after FFN only) - cheaper, near-equal quality. [source]
Match FFT within ~95%+ at <5% params. Downside vs LoRA: adapters add **layers in [source]
series → real inference latency** that you cannot merge away (LoRA can). [source]
(IA)^3 (Liu et al., 2022). "Infused Adapter by Inhibiting and Amplifying [source]
Inner Activations." Learns three element-wise scaling vectors that rescale [source]
keys, values, and FFN activations. Extremely parameter-light - **~0.5 M params [source]
for a 7B model** (one scalar per activation dim, no matrices). Designed to beat [source]
few-shot in-context learning more cheaply. Often slightly below LoRA on [source]
accuracy; shines when parameter budget is the hard constraint. [source]
Prefix-tuning (Li & Liang, 2021). Prepend trainable **continuous vectors [source]
("virtual tokens") to the keys/values at every** layer; the real model stays [source]
frozen. Steers behavior without touching weights. [source]
P-tuning v2 (Liu et al., 2021). Deep prompt tuning - trainable prompts at [source]
every layer (not just the input). Effectively prefix-tuning generalized to [source]
NLU; the first prompt-based method to match FFT across scales/tasks. [source]
Prompt tuning (Lester et al., 2021). The lightest: trainable **soft-prompt [source]
embeddings at the input layer only**. Competitive only at large model scale; [source]
weaker on smaller models and harder tasks. [source]
> The mental model: LoRA/adapters/(IA)^3 = reparameterize the weights; [source]
> prefix/P-tuning/prompt-tuning = learn a soft prompt, weights untouched. [source]
> In 2024-2026 practice LoRA (and its family) is the default; (IA)^3 for [source]
> extreme parameter thrift; prompt-based methods are mostly of historical / [source]
> multi-task-serving interest. Prefix/prompt methods also **consume context [source]
> length** at inference. [source]

Part 6 — The HuggingFace PEFT workflow

peft is the standard library; it wraps any transformers model. [source]
Knob summary in LoraConfig: r, lora_alpha, target_modules [source]
(or "all-linear"), lora_dropout, use_rslora, use_dora, init_lora_weights [source]
(True/"gaussian"/"pissa"/"olora"/"eva"/"loftq"/"corda"), [source]
rank_pattern/alpha_pattern (per-layer overrides), target_parameters (MoE [source]
experts), modules_to_save (fully-train extra modules like a new classifier head), [source]
trainable_token_indices (train just new special-token embeddings). [source]
Multiple adapters on one base (Part 7): `PeftModel.from_pretrained(base, id, [source]
adapter_name="a"), then model.load_adapter(id2, adapter_name="b")`, [source]
model.set_adapter("b") to switch, model.disable_adapter() context for the raw [source]
base, model.delete_adapter("b") to drop. LoRA+ optimizer: [source]
create_loraplus_optimizer(model, optimizer_cls, lr, loraplus_lr_ratio). [source]
PEFT supports LoRA + variants (DoRA/rsLoRA/PiSSA/…), adapters, (IA)^3, [source]
prefix-tuning, P-tuning, prompt-tuning, LoHa/LoKr, and more - same wrap-and-train [source]

Part 7 — Multi-LoRA serving: merge vs swap (+ adapter merging)

You trained an adapter. Two ways to serve it, and a third way to combine several. [source]
Merge (merge_and_unload). Fold ΔW into W to get a **standalone model with [source]
zero added latency. Use when one adapter serves all traffic. It is not [source]
in-place, so assign the return value. Lossy for quantized bases** (merging fp16 [source]
deltas into a 4-bit base reintroduces error) and irreversible; for QLoRA, either [source]
serve unmerged or dequantize-then-merge. DoRA/MoE-LoRA should be merged to erase [source]
their inference overhead. [source]
Swap / multi-tenant (keep unmerged). Keep the frozen base resident once and [source]
hot-swap small adapters per request - N tasks served from 1 base + N few-MB [source]
adapters instead of N full models. The economic win behind LoRA serving. PEFT can [source]
even mix adapters within one batch via the adapter_names argument [source]
(base/adapter_fr/adapter_de rows in the same forward pass). At scale, the [source]
serving engine does this efficiently: [source]
S-LoRA - custom heterogeneous CUDA kernels + unified paging (adapters in [source]
CPU memory, active slices paged to GPU alongside KV-cache). Serves thousands [source]
of concurrent adapters; up to 4x throughput over naive PEFT/vLLM LoRA. [source]
Punica - SGMV kernel fuses heterogeneous LoRA deltas (different adapters [source]
and ranks) into one batched matmul. [source]
vLLM / TGI / SGLang ship multi-LoRA serving built on these ideas; [source]
mid-sequence adapter switching is still the open overhead. [source]
> The kernel/runtime side of multi-LoRA (PagedAttention, continuous batching, [source]
> KV-aware routing, autoscaling) is the sibling references/llm-inference-serving.md. [source]
> This reference owns the decision (merge vs swap) and the adapter-combination [source]
Combining several adapters into one - `add_weighted_adapter(adapters=[...], [source]
weights=[...], combination_type=...). combination_type` options: [source]
linear - weighted sum of the deltas (e.g. blend an SFT and a DPO adapter [source]
cat - concatenate (ranks add; no information loss, larger adapter). [source]
ties / dare_ties / dare_linear - sign-resolution / random-drop [source]
merge methods that reduce interference between task adapters (these merge [source]
algorithms are detailed in llm-compression.md's model-merging section). [source]
svd - SVD-based combine (not supported in fp16/bf16). [source]
> aLoRA (Activated LoRA) is a serving-time variant that activates the adapter [source]
> only after an invocation token, so it reuses the base model's KV cache — [source]
> an order-of-magnitude speedup when the base does most of the work and the adapter [source]
> handles a checking/correcting sub-task. aLoRA cannot be merged by definition. [source]

Part 8 — Catastrophic forgetting & mitigations

Catastrophic forgetting: fine-tuning on a narrow task degrades the base model's [source]
general abilities (it overwrites pretrained knowledge). The classic symptom is a [source]
fine-tune that nails your task but loses MMLU points and general chat quality. [source]
Mitigations, roughly strongest-first: [source]
Use PEFT, especially LoRA. Because the base is frozen and only a small [source]
add-on trains, parameter isolation structurally protects pretrained [source]
weights (the "LoRA forgets less" result, Part 2), and LoRA out-forgets weight [source]
decay and dropout. The single biggest lever. [source]
Experience replay / rehearsal. Mix a slice of general / prior-task data [source]
(or pretraining-style data) into the fine-tune set. The most effective standalone [source]
technique; recent work prioritizes rehearsing "collateral-damage" examples (ones [source]
the base got right but the fine-tune started getting wrong). [source]
Regularization toward the base. Weight decay, dropout, **lower learning [source]
rate, fewer epochs, early stopping** on a held-out set. [source]
Forgetting-aware init / structure. CorDA-KPM (knowledge-preserved init) [source]
and OPLoRA (orthogonal-projection LoRA) explicitly protect base knowledge; [source]
KappaTune targets only the most isotropic (high-entropy) layers, leaving [source]
specialized layers intact. [source]
Intruder-dimension reduction (reduce_intruder_dimension) - post-hoc remove [source]
the "intruder" singular directions a LoRA introduced; a tunable trade-off [source]
between task accuracy kept and base knowledge restored. [source]
Always quantify it: run a general-capability benchmark (e.g. MMLU) on the base [source]
and the fine-tune. A >2-3 point drop signals forgetting (Part 10). [source]

Part 9 — SFT data preparation & chat templating

The fine-tune's quality is bounded by its data. Supervised fine-tuning (SFT) = [source]
training on (instruction/prompt → desired response) pairs so the model shifts [source]
from generic next-token prediction to following instructions in your format. [source]
(SFT is also the first stage of post-training that precedes RLHF/DPO → [source]
llm-alignment-post-training.md.) [source]
Quality over quantity. A few thousand **clean, diverse, correctly-formatted, [source]
deduplicated** examples beat a noisy large set (the LIMA "less is more" finding). [source]
Curate for correctness, format consistency, and coverage of the behaviors you want; [source]
decontaminate against your eval set. [source]
Dataset formats (TRL SFTTrainer conventions): [source]
Conversational - `{"messages": [{"role": "system"/"user"/"assistant", [source]
"content": ...}]}`. Preferred for chat models; the trainer applies the model's [source]
chat template for you. [source]
Prompt-completion - {"prompt": ..., "completion": ...}. [source]
Instruction (Alpaca-style) - {"instruction", "input", "output"}, usually [source]
rendered into one of the above. [source]
Chat templating is non-negotiable. Chat models were trained with an exact [source]
token format (role markers + special tokens, e.g. <|im_start|>user … <|im_end|>). [source]
The template is a Jinja string shipped on the tokenizer [source]
(tokenizer.apply_chat_template(...)). **Mismatched formatting between fine-tuning [source]
and inference is the #1 silent fine-tune killer**: train and serve with the same [source]
template and special tokens. When introducing genuinely new special tokens, [source]
resize_token_embeddings and train them (PEFT trainable_token_indices does this [source]
Completion-only / loss masking. You almost always want loss computed **only on [source]
the assistant/response tokens**, not the prompt - set the prompt-token labels to [source]
the ignore index -100 so cross-entropy skips them. This focuses learning on [source]
generating the response rather than memorizing the instruction. TRL's [source]
SFTTrainer does completion-only masking for prompt-completion data by default; [source]
for conversational data use its assistant-only-loss option. [source]
Packing. Concatenate short examples into full-length sequences to avoid wasted [source]
padding compute (packing=True) - watch that cross-example attention is masked. [source]

Part 10 — Tooling stack

All support LoRA + QLoRA; the differences are ergonomics, kernel speed, and [source]
multi-GPU. For preference optimization (DPO/PPO) the same tools route into TRL / [source]
OpenRLHF → llm-alignment-post-training.md. [source]

Part 11 — Evaluating a fine-tune

A fine-tune eval needs two prongs - and you must beat a real baseline. [source]
Task improvement. A held-out test set of your task (never seen in [source]
training), scored with a task-appropriate metric: exact-match/F1 for extraction, [source]
pass@k for code, an LLM-as-judge rubric for open-ended generation, [source]
classification metrics for labels. (Harness mechanics → da-7-machine-learning.) [source]
Capability-regression check. Run a general benchmark (e.g. MMLU) on the [source]
base and the fine-tune. A >2-3 point drop = catastrophic forgetting — [source]
address with Part 8 before shipping. [source]
Always compare against the base model on the same held-out set to prove the [source]
fine-tune actually helped (and ideally against a strong prompted base - sometimes [source]
prompting alone matches it). [source]
Detect overfitting: track validation loss during training and stop when it [source]
turns up (early stopping); a train-loss that keeps dropping while val-loss rises is [source]
the tell. Watch for benchmark contamination: the NeurIPS-2023 fine-tuning [source]
competition found top models heavily overfit popular benchmarks, so a clean, [source]
private held-out set is worth more than a public leaderboard number. [source]

Anti-patterns

Fine-tuning to add knowledge that changes often. It bakes in stale facts and [source]
the model still hallucinates. Use RAG. Fine-tune behavior, retrieve facts. [source]
Reaching for fine-tuning before exhausting prompting + RAG. It's the slowest, [source]
costliest lever; most "fine-tune" problems are prompt/RAG problems. [source]
Train/inference chat-template mismatch. Different template or special tokens [source]
at serving than at training → silent quality collapse. The #1 fine-tune bug. [source]
Computing loss on prompt tokens. Teaches the model to parrot instructions; [source]
mask the prompt with -100 / use completion-only. [source]
Targeting only q_proj,v_proj and expecting FFT quality. For hard tasks use [source]
all-linear and a higher rank (and rsLoRA so the higher rank helps). [source]
Raising r without rsLoRA. With α/r scaling, gradients collapse and the [source]
larger rank buys nothing - use use_rslora=True. [source]
Merging an fp16 adapter into a 4-bit QLoRA base and expecting no loss. Merge [source]
reintroduces quantization error; serve unmerged or dequantize first. [source]
No base-model regression check. Shipping a fine-tune that quietly lost 5 MMLU [source]
points. Always eval both prongs (Part 11). [source]
Huge noisy dataset over a small clean one. Quality, diversity, and dedup beat [source]
raw volume; decontaminate against eval. [source]
Same learning rate as full fine-tuning. LoRA usually wants a higher LR [source]
(e.g. 1e-4 to 3e-4) than FFT (~1e-5); tune it before reaching for exotic variants. [source]

Troubleshooting

Fine-tune nails the task but general chat degraded → catastrophic forgetting: [source]
switch to LoRA, add replay data, lower LR / fewer epochs, check MMLU delta (Part 8). [source]
LoRA underperforms FFT on code/math → raise rank to 64-256, use all-linear, [source]
use_rslora=True; or accept FFT for that workload (Part 2). [source]
Garbage/looping generations after fine-tuning → almost always a chat-template [source]
or special-token mismatch, or EOS not learned; verify apply_chat_template [source]
parity train↔serve (Part 9). [source]
OOM during training → QLoRA (4-bit base), Unsloth, gradient checkpointing, [source]
smaller batch + gradient accumulation, paged optimizer, lower rank. [source]
Adapter "does nothing" at inference → forgot to set_adapter/load it, or [source]
merged then tried to swap; confirm the active adapter. [source]
High inference latency with the adapter → merge_and_unload for single-task [source]
serving; DoRA/MoE-LoRA especially must be merged. [source]
Loss not decreasing → LR too low (LoRA likes higher LR), or loss masked [source]
wrong, or B not actually training (check print_trainable_parameters). [source]
QLoRA quality below expectation → use LoftQ init to minimize quantization [source]
error; consider QDoRA. [source]

References

LoRA: Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" - https://arxiv.org/abs/2106.09685 [source]
QLoRA: Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" - https://arxiv.org/abs/2305.14314 [source]
DoRA: Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation" (ICML 2024) - https://arxiv.org/abs/2402.09353 [source]
rsLoRA: Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA" - https://arxiv.org/pdf/2312.03732 [source]
LoRA+: Hayou et al., "LoRA+: Efficient Low Rank Adaptation of Large Models" - https://arxiv.org/pdf/2402.12354 [source]
"LoRA Learns Less and Forgets Less": Biderman et al. - https://arxiv.org/pdf/2405.09673 [source]
"LoRA vs Full Fine-tuning: An Illusion of Equivalence" - https://arxiv.org/html/2410.21228v3 [source]
(IA)^3 / T-Few: Liu et al., "Few-Shot PEFT is Better and Cheaper than In-Context Learning" - https://proceedings.neurips.cc/paper_files/paper/2022/file/0cde695b83bd186c1fd456302888454c-Paper-Conference.pdf [source]
PEFT survey: Han et al., "Parameter-Efficient Fine-Tuning for Large Models: A Survey" - https://link.springer.com/article/10.1007/s10462-025-11236-4 [source]
S-LoRA: "Serving Thousands of Concurrent LoRA Adapters" (MLSys 2024) - https://arxiv.org/pdf/2311.03285 [source]
Punica: "Multi-Tenant LoRA Serving" - https://arxiv.org/pdf/2310.18547 [source]
HuggingFace PEFT - LoRA developer guide - https://huggingface.co/docs/peft/main/en/developer_guides/lora [source]
HuggingFace TRL - SFTTrainer - https://huggingface.co/docs/trl/en/sft_trainer [source]
bitsandbytes 4-bit + QLoRA (HF blog) - https://huggingface.co/blog/4bit-transformers-bitsandbytes [source]
LoRA hyperparameters (rank/alpha/target modules) - https://mbrenndoerfer.com/writing/lora-hyperparameters-rank-alpha-target-modules [source]
Fine-tuning framework comparison (Unsloth/Axolotl/torchtune/Llama-Factory) - https://modal.com/blog/fine-tuning-llms [source]
RAG vs Fine-tuning vs Prompt Engineering (IBM) - https://www.ibm.com/think/topics/rag-vs-fine-tuning-vs-prompt-engineering [source]
Catastrophic-forgetting rehearsal scheme - https://arxiv.org/html/2402.08096 [source]
OPLoRA (orthogonal-projection LoRA, forgetting) - https://arxiv.org/pdf/2510.13003 [source]

Children

Fine-tune vs RAG vs prompt decision framework (frontier)
Full fine-tuning vs PEFT (memory, plasticity-stability) (frontier)
LoRA mechanics (rank/alpha/target_modules/init) (frontier)
LoRA family (QLoRA, DoRA, rsLoRA, LoRA+) (frontier)
Non-LoRA PEFT (adapters, (IA)^3, prefix/P-tuning/prompt-tuning) (frontier)
HuggingFace PEFT + TRL workflow (frontier)
Multi-LoRA serving: merge vs swap + adapter merging (frontier)
Catastrophic forgetting and mitigations (frontier)
SFT data preparation and chat templating (frontier)
Fine-tuning tooling stack (Unsloth/Axolotl/Llama-Factory/torchtune) (frontier)
Evaluating a fine-tune (task + capability-regression) (frontier)

Frontier under this node: Catastrophic forgetting and mitigations, Evaluating a fine-tune (task + capability-regression), Fine-tune vs RAG vs prompt decision framework, Fine-tuning tooling stack (Unsloth/Axolotl/Llama-Factory/torchtune), Full fine-tuning vs PEFT (memory, plasticity-stability), HuggingFace PEFT + TRL workflow, LoRA family (QLoRA, DoRA, rsLoRA, LoRA+), LoRA mechanics (rank/alpha/target_modules/init), Multi-LoRA serving: merge vs swap + adapter merging, Non-LoRA PEFT (adapters, (IA)^3, prefix/P-tuning/prompt-tuning), SFT data preparation and chat templating

← the whole tree · 3D view· how to read this page

LLM Fine-Tuning &amp; PEFT