--- title: "LLM Pretraining & Scaling Laws" description: "PROVENANCE: This reference is part of the ai-agent-engineering hub." --- # LLM Pretraining & Scaling Laws Pretraining is the expensive part: take a randomly-initialized transformer and run **next-token prediction over trillions of tokens** until it becomes a *base model* — a raw next-token predictor with broad world knowledge but no instruction-following. Everything in `llm-alignment-post-training.md`, `llm-fine-tuning-peft.md`, and `reasoning-models.md` *starts from this checkpoint*. This reference answers the four questions that define a pretraining run: **what objective**, **what data**, **how big / how long** (scaling laws + the compute budget), and **how do I know it worked** (base-model eval). **The one identity that anchors everything here — `C ≈ 6ND`.** A dense transformer with `N` parameters trained on `D` tokens costs about `6ND` floating-point operations. That single equation is the budget line: every scaling-law result (Kaplan, Chinchilla, data-constrained, inference-aware) is an answer to *"given a fixed `C`, how do I split it between `N` and `D` to minimize loss?"* Hold `C = 6ND` and the whole field becomes a constrained-optimization story. The 2022 → 2026 arc is the field realizing that the loss-optimal split (`N : D`) and the *deployment*-optimal split are different things — and that data, not parameters, is now the binding constraint. ## Scope boundary (read first) - **This reference** = pretraining a base model: objectives (causal LM, MLM, prefix-LM, FIM, UL2), the data pipeline (curation, dedup, filtering, mixtures, tokenizer *training*, decontamination), scaling laws (Kaplan, Chinchilla, data-constrained, inference-aware), the `6ND` compute budget, emergent abilities + the mirage debate, LR schedules at scale (cosine, WSD), data curriculum/annealing, continual/domain-adaptive pretraining, and base-model evaluation. - **The transformer architecture** — attention, MoE, RoPE, RMSNorm, and the tokenizer *algorithms* (BPE/byte-level-BPE/SentencePiece/tiktoken) → `transformer-architecture.md`. The line: **what to train and on how much data is here; how the block is wired is there.** Tokenizer *training* (picking vocab size, the training corpus, measuring fertility) is here because it is a *data* decision; the tokenization *algorithm* is architecture. - **Distributed-training infrastructure** — FSDP, DeepSpeed ZeRO, tensor/pipeline/expert parallelism, gradient checkpointing, MFU (Model FLOPs Utilization) — is the *engineering* of spending `6ND` FLOPs across a cluster → `distributed-training.md`. We own the *budget arithmetic* (`6ND`, token/param ratios) and *what to train*; that sibling owns *how the GPUs cooperate*. - **Post-training** (SFT, RLHF/PPO, DPO family, Constitutional AI, RLAIF) → `llm-alignment-post-training.md`. **Pretraining produces the base model; alignment turns it into an assistant.** The handoff is the base checkpoint. - **Fine-tuning / PEFT** (LoRA/QLoRA, domain SFT, instruction tuning) → `llm-fine-tuning-peft.md`. *Continual pretraining* (more self-supervised next-token prediction on a new corpus, §10) lives here; *supervised* adaptation lives there. - **Reasoning RL & test-time compute** (GRPO, RLVR, long-CoT, the DeepSeek-R1 recipe) → `reasoning-models.md`. Note: "inference-time scaling laws" *there* (spend more compute at answer time) are a different axis from the *pretraining* scaling laws *here* (spend more compute at train time) — §6 distinguishes them. - **Benchmark-harness mechanics** (how MMLU/HELM/LLM-as-Judge are scored) → `da-7-machine-learning`. Base-eval *strategy* and *decontamination* are here. --- ## 1. Pretraining objectives — what loss is the model minimizing? The objective decides what the base model is good at before any post-training. Five matter (2024–2026). - **Causal / autoregressive LM (CLM, next-token prediction).** Predict token `t` from tokens ``, ``, ``), then train next-token as usual. The model learns to generate the middle given both ends. Key results: it is **"free"** — the **FIM rate** (fraction of docs transformed, 50–90% is fine) buys infilling with *no loss* on left-to-right ability (the "FIM-for-free property"); **context-level** FIM (transform after chunking to context length) beats **document-level**; **joint PSM+SPM** training transfers positively. This is why code models (StarCoder, Codestral, DeepSeek-Coder) and modern general models support `` infilling and IDE tab-completion. - **UL2 — Mixture-of-Denoisers (MoD).** *(Tay et al. 2022, arXiv:2205.05131.)* Unify objectives by training on a **mixture of span-corruption "denoisers"**: **R-denoiser** (regular T5-style short spans, ~15% / span≈3), **S-denoiser** (sequential = prefix-LM, corrupt a contiguous tail → forces generation), **X-denoiser** ("extreme": long spans and/or high corruption rate, the hardest). A **paradigm token** (`[R]`/`[S]`/`[X]`) prepended at train time lets you **mode-switch** the model toward the most suitable behavior downstream. UL2 showed one objective family can be competitive across both understanding and generation; the *conceptual* legacy (mix easy + hard self-supervised tasks; the prefix-LM-as-S-denoiser framing) outlived the specific recipe. **Decision in practice (2026):** generative LLMs use **CLM**, often **+ FIM** (especially for code), trained on **packed** sequences (multiple documents concatenated to a fixed context length with separators — packing keeps GPUs full; whether to mask cross-document attention is a live choice). MLM is for encoders. UL2/prefix-LM matter mostly historically and for encoder-decoder niches. --- ## 2. The pretraining data pipeline — curation, dedup, quality filtering Data quality, not architecture, is the dominant lever on a fixed compute budget. The canonical open reference is **FineWeb** *(Penedo et al. 2024, arXiv:2406.17557)*: 15T GPT-2 tokens from **96 Common Crawl snapshots (2013–early 2024)**, with every design choice **ablated** (each stage shown to monotonically improve downstream benchmarks). The pipeline stages: 1. **Text extraction.** Pull main content from raw HTML/WARC (FineWeb uses `trafilatura`); good extraction beats Common Crawl's own WET text. 2. **Language ID + filtering.** Classifier-based language detection; keep target languages above a confidence threshold. 3. **Quality / heuristic filtering.** Rule-based filters (line-length, symbol-to-word ratio, fraction of duplicate lines, bad-words, repetition) à la MassiveText/Gopher and C4. Removes boilerplate, SEO spam, gibberish. 4. **Deduplication (the single highest-impact stage).** **Near-dedup** with **MinHash + LSH** on n-gram shingles (FineWeb: per-snapshot MinHash dedup). Removes the long tail of near-identical pages. Counter-intuitive finding from FineWeb: **global cross-snapshot dedup can hurt** — it disproportionately removes *recently* re-crawled, often higher-quality content and over-upweights ancient low-quality pages; **per-snapshot** dedup worked better. Dedup also matters because *repeated* data interacts with epoch-counting (§5). 5. **Model-based quality classification.** Train a lightweight classifier to score "is this high-quality / educational?" **FineWeb-Edu** is a **1.3T-token** subset filtered by an **educational-quality classifier** (a linear/small head trained on **Llama-3-70B annotations** of educational value). Models pretrained on FineWeb-Edu show **large gains on knowledge/reasoning benchmarks (MMLU, ARC)** versus unfiltered FineWeb — strong evidence that *aggressive* quality filtering pays off, even at the cost of raw token count. **Synthetic data** is now a standard ingredient (rephrased web, textbook-style generation à la Phi, distilled chains). Risk: **model collapse** (degenerate distributions when training on too much un-curated model output across generations) — mitigate by anchoring to real data and limiting synthetic share. > Cross-ref: the *tokenization algorithm* (how text becomes IDs) is in `transformer-architecture.md` §10. *Tokenizer training* (choosing the vocab, the corpus to train it on) is §4 here because it is a data decision. --- ## 3. Data mixtures & domain weighting — how much of each source? Once you have cleaned sources (web, code, books, arXiv, Wikipedia, math, multilingual), you must choose **mixture proportions** (domain weights). This is a first-class hyperparameter — getting it wrong wastes compute. - **The Pile / fixed heuristic weights** — early models hand-set weights (upweight Wikipedia/books, cap web). Simple, manual, suboptimal. - **DoReMi (Domain Reweighting with Minimax Optimization)** *(Xie et al. 2023, arXiv:2305.10429, NeurIPS 2023).* Train a **small proxy model** (e.g. 280M) with **Group DRO** to find domain weights that minimize *worst-case* excess loss versus a reference model — *without* knowing downstream tasks. Reuse those weights to train a model **30× larger** (8B). Result: **+6.5% average few-shot accuracy** over The-Pile default weights and the baseline accuracy reached in **2.6× fewer steps**; perplexity improved across *all* domains even ones it downweighted. The principle — **use a cheap proxy run to set mixture weights for the expensive run** — generalizes (online/learned mixtures, RegMix, data-mixing laws). - **Mixture scaling laws.** A live 2024–2026 thread: the *optimal* mixture **shifts with scale and with token budget** (and with repetition — §5). Web fraction that is optimal at 1T tokens is not optimal at 15T; code/math get upweighted as you train longer. Treat the mixture as scale-dependent, not fixed. **Practical pattern:** set a base mixture (heuristic or DoReMi-derived), then **upsample high-value domains (code, math, curated/synthetic) during the annealing phase** (§9) rather than uniformly — the decay phase is where domain emphasis is cheapest and most effective (the MiniCPM/Yi-Lightning recipe). --- ## 4. Tokenizer training & eval-set decontamination **Tokenizer training (a data decision; algorithm → `transformer-architecture.md`).** Before pretraining you *train* the tokenizer on a sample of the corpus and freeze it; vocab choice then constrains everything. - **Vocabulary size.** A bigger vocab → fewer tokens per document (lower *fertility*, cheaper sequences, more text per context) but a larger embedding/unembedding matrix and rarer-token undertraining. **~128K is the modern sweet spot** for multilingual models (Llama 3 moved to 128K byte-level BPE; many 2024–2026 models sit at 128K–256K). There are even *scaling laws for vocabulary* — larger models warrant larger vocabularies. - **Training corpus & multilinguality.** Train the tokenizer on a mixture *representative of the pretraining mix* — an English-heavy tokenizer gives terrible fertility on other languages (more tokens per word → more expensive, worse). Returns on tokenizer *training data* diminish (1GB → 900GB studied; gains saturate early). - **Fertility & parity** (tokens-per-word; cross-language token-count ratio) are the standard intrinsic metrics — but **caveat: they are not always predictive of downstream quality**, so validate the tokenizer on a small pretraining proxy, not on fertility alone. **Eval-set decontamination (do not skip — it is how you avoid lying to yourself).** Web-scale corpora contain copies of benchmark test sets; if MMLU/GSM8K leak into pretraining, your eval is inflated. - **n-gram overlap** is the standard filter: scan the corpus and remove (or flag) documents overlapping a test item. **GPT-3 used 13-gram overlap; GPT-4 used a 50-character span.** Maximum-matching-subsequence (MMS) is a variant. - **n-gram matching is fragile.** *(Yang et al. 2023, "Rethinking Benchmark and Contamination… with Rephrased Samples", arXiv:2311.04850.)* **Paraphrased or translated test items slip past string matching** and still contaminate — a model can memorize a rephrased benchmark and ace it while passing decontamination. Mitigations: embedding/semantic-similarity decontamination, **contamination-resistant / freshly-collected benchmarks** (post-cutoff data, e.g. LiveCodeBench, private held-out sets), and **canary strings**. Always **decontaminate, then prefer time-gated evals** for the headline number. --- ## 5. Scaling laws I — Kaplan vs Chinchilla (compute-optimal allocation) Scaling laws predict loss as a smooth power law in model size `N`, data `D`, and compute `C`, and tell you how to *split* a fixed `C` between `N` and `D`. - **Kaplan et al. 2020 (arXiv:2001.08361).** First clean power laws: test loss falls predictably as a power of `N`, `D`, and `C`. Their compute-optimal prescription **favored very large models** — given more compute, grow `N` fast and `D` slowly (`N_opt ∝ C^0.73`). This drove the GPT-3 / Gopher / MT-NLG "scale parameters" era. *(It turned out to be skewed — see the reconciliation below.)* - **Chinchilla — Hoffmann et al. 2022 (arXiv:2203.15556).** Refit the laws carefully and found Kaplan-era models were **massively under-trained on data**. Compute-optimal scaling is **balanced**: `N_opt ∝ C^0.50` and `D_opt ∝ C^0.50` — i.e. **scale `N` and `D` equally** (every doubling of params should double tokens). The famous heuristic: **≈ 20 tokens per parameter** is compute-optimal. They trained **Chinchilla (70B on 1.4T tokens)** and it **beat Gopher (280B)** using the *same* compute — a smaller, longer-trained model won. Loss is fit as `L(N, D) = E + A/N^α + B/D^β`. **This reset the field**: GPT-3 (175B/300B tokens) was ~10× too few tokens. - **Reconciling the two** *(Porian et al. 2024, "Resolving Discrepancies…", arXiv:2406.12907, NeurIPS 2024; also Besiroglu et al. replication arXiv:2404.10102).* Kaplan's `0.73` vs Chinchilla's `0.50` is **mostly an artifact**, explained by three things: (1) **Kaplan counted only non-embedding parameters** (Chinchilla counts *all* params); (2) Kaplan **under-counted FLOPs** by ignoring the last-layer/embedding cost at the small scales they used; (3) **warmup duration and optimizer tuning** were not adapted per model size. Fix all three and the curves **collapse onto Chinchilla's 0.50**. **Lesson: count *total* params and *all* FLOPs, tune warmup/LR per scale — then scaling fits agree.** Epoch AI's replication broadly **confirmed** Chinchilla's estimates while noting its confidence intervals were too tight. --- ## 6. Scaling laws II — data-constrained & inference-aware (why Chinchilla is not the answer in 2026) Chinchilla is **training-compute-optimal**, not **deployment-optimal**, and assumes **unlimited unique data**. Both assumptions break in practice. - **Inference-aware / "over-training"** *(Sardana & Frankle 2023, "Beyond Chinchilla-Optimal", arXiv:2401.00448).* If you will *serve* the model to many users, total cost = training **+** inference, and inference cost scales with `N`. So you should deliberately train a **smaller model on far more tokens than Chinchilla** (`>20` tok/param) — paying more at train time to get a cheaper, faster model forever. **"Over-training" is a misnomer**: it is only "over" relative to the *training*-optimal point. **Llama 3 8B is the canonical example: ~15T tokens ≈ 1,875 tokens/param** (vs Chinchilla's ~200B / 20× for an 8B), and loss kept improving log-linearly far past the Chinchilla point. **Mid-2026 reality: almost every shipped model is deliberately over-trained.** (Related 2026 thread: *test-time scaling can make over-training even more attractive* — a small over-trained model + inference-time compute beats a compute-optimal bigger one at equal serving cost; reasoning-time scaling itself → `reasoning-models.md`.) - **Data-constrained scaling** *(Muennighoff et al. 2023, arXiv:2305.16264, JMLR 2025).* When unique tokens run out (we are approaching the limit of high-quality web text), you **repeat data (multiple epochs)**. Findings, now load-bearing: **repeating up to ~4 epochs is almost as good as fresh data** (negligible loss penalty); gains continue but **decay out to ~16 epochs** (a repeated token retains ~63% of a fresh token's value around there) and approach **zero by ~40 epochs**. Their scaling law adds a **decay term for repeated tokens and excess parameters**, and prescribes **smaller models trained for more epochs** when data-bound (the opposite of naively applying Chinchilla to repeated data). **Allocating excess compute to more *params* also decays** once data is fixed. Practical rule: **≤4 epochs is safe; 4–16 is diminishing; >16 wastes compute** — and budget extra FLOPs into *quality filtering / synthetic data* rather than blind repetition. **Synthesis of the three regimes:** Chinchilla (balanced) is the textbook answer when **data and inference are free**. **Inference-aware** (over-train a small model) is the answer when you will **serve at scale**. **Data-constrained** (repeat ≤4 epochs, prefer smaller models) is the answer when you are **out of unique tokens**. Mid-2026 frontier runs live at the **intersection**: small-ish models, heavily over-trained, on **heavily-filtered + synthetic** data, a few epochs at most. --- ## 7. The compute budget — `C ≈ 6ND` and how to use it The arithmetic that turns all of the above into a project plan. - **`C ≈ 6ND` FLOPs** to train a dense model of `N` non-embedding params on `D` tokens. The **6** decomposes as **2 (forward) + 4 (backward)** FLOPs per parameter per token: a matmul is one multiply + one add = **2 FLOPs** per weight (so `2N` forward per token), and the backward pass does **~2× the forward matmuls** (gradient w.r.t. inputs *and* weights) = `4N`. Multiply by `D` tokens → `6ND`. **Inference is `≈ 2ND`** (forward only). *(Standard derivation; see the Chinchilla appendix and "Transformer FLOPs", Casson 2023.)* - **What it is good for.** Back-of-envelope budgeting: pick any two of `{C, N, D}` and solve the third. *E.g.* an 8B model on 15T tokens ≈ `6 × 8e9 × 15e12 ≈ 7.2e23` FLOPs. Combine with hardware throughput and **MFU** (Model FLOPs Utilization, the fraction of peak FLOP/s actually used — typically 30–55%) to get wall-clock and GPU-hours: `time ≈ C / (peak_FLOP/s × MFU × num_GPUs)`. - **Caveats.** `6ND` **ignores attention FLOPs** (the `O(seq²·d)` term), which is fine while `seq` ≪ `d_model · layers / seq` but bites at long context. For **MoE**, use **active** params, not total, in `N` (only the routed experts fire per token). The *engineering* of hitting high MFU across a cluster — FSDP/ZeRO sharding, parallelism, gradient checkpointing — is the **distributed-training sibling's** job; `6ND` is the *budget*, MFU-engineering is *spending it efficiently*. --- ## 8. Emergent abilities & the "mirage" debate - **The claim (Wei et al. 2022, arXiv:2206.07682).** Some abilities are **emergent**: absent in small models, present in large ones, appearing **sharply and unpredictably** at a scale threshold (e.g. multi-step arithmetic, word unscrambling, certain BIG-Bench tasks) — not extrapolable from smaller models' performance. - **The rebuttal — "Are Emergent Abilities a Mirage?" (Schaeffer et al. 2023, arXiv:2304.15004, NeurIPS 2023 Outstanding Paper).** The sharp jumps are often an **artifact of the metric**, not the model. **Discontinuous/nonlinear metrics** (exact-match, multiple-choice accuracy — all-or-nothing) manufacture apparent step-changes; switch to **continuous/smooth metrics** (token edit distance, Brier score, log-likelihood per token) on the *same* models and the curve becomes **smooth and predictable**. They reproduce "emergence" by choosing metrics and erase it by changing them, and predict where it will/won't appear on the GPT-3 family + BIG-Bench. - **The 2024–2026 synthesis (what to actually believe).** Both are partly right and the distinction is practical: **per-token loss scales smoothly and predictably** (this is what scaling laws fit and what you should plan against); **downstream task *scores* under harsh metrics can still jump**, because crossing a usefulness threshold (the model finally gets the *whole* multi-step answer right) is real for the user even if the underlying capability grew smoothly. **Operationally: forecast with smooth metrics (loss, log-prob), but don't be surprised when a hard pass/fail benchmark lurches.** This is also why a base model can look unimpressive on accuracy yet be a fine pretraining checkpoint (§11). --- ## 9. Learning-rate schedules at scale — cosine vs WSD, and the annealing phase The LR schedule is a pretraining-specific lever with a surprising amount of pull on the final loss. - **Warmup.** Linearly ramp LR from 0 over a few hundred–few thousand steps (and tune warmup *per scale* — it was one of the Kaplan/Chinchilla reconciliation factors, §5). Skipping warmup destabilizes early training. - **Cosine decay (the long-time default).** After warmup, decay LR following a cosine curve to a small floor **over the whole planned token budget**. Works well but has a **coupling problem**: the schedule is **tied to a pre-committed total step count**, so you cannot cleanly extend a run or take a good intermediate checkpoint (any checkpoint before the end is at a high, un-decayed LR and underperforms). - **Warmup-Stable-Decay (WSD)** *(Hu et al. 2024, MiniCPM, arXiv:2404.06395).* Three phases: **(1) warmup → (2) a long *stable* phase at a constant high LR → (3) a short, sharp *decay/cooldown*** (often the last ~10–20%). Properties that made it a 2024–2026 favorite: **(a)** during the stable phase loss sits *higher* than cosine, but the **decay phase drops loss sharply**, often *below* cosine's final loss; **(b)** it is **compute-agnostic** — the stable phase can run indefinitely and you decay *whenever* you decide to stop, so you can **reuse the stable checkpoint** for runs of different lengths; **(c)** the decay phase is the natural place to **anneal in high-quality / domain / synthetic data** (see below). MiniCPM used WSD to run efficient **data-model scaling studies** and reported a much higher **compute-optimal data/model ratio of ~192×** (vs Chinchilla's 20×) — reinforcing the over-training story (§6). A "river-valley loss landscape" account (arXiv:2410.05192) explains *why* the sharp decay helps. - **The annealing/decay phase = where the data curriculum lives (§3, §10).** The dominant 2024–2026 pattern is **two-phase pretraining**: phase 1 trains on broad web data at high/stable LR; the **annealing phase** (the WSD decay, or the cosine tail) **upsamples high-quality, instruct-like, synthetic, math/code, and rare-language data** while LR drops. Yi-Lightning's three-stage recipe (diversity → upsample-high-quality-during-anneal → fast-decay-on-best-data) is representative. Caveat (arXiv:2511.18903): **aggressive LR decay can *waste* your best data** (the model is barely learning by the time you feed it) — a *moderate* decay, or **decoupling the curriculum from LR via model averaging / a constant LR + checkpoint-averaging**, can do better. Continual-pretraining "infinite LR" schedules (arXiv:2503.02844) push the WSD idea further for never-ending training. --- ## 10. Continual & domain-adaptive pretraining You rarely retrain from scratch to add a domain, a language, or fresher data. **Continual pretraining (CPT)** = keep doing **next-token prediction** on a new corpus starting from an existing base checkpoint. (Distinct from fine-tuning/PEFT, which uses *supervised* data → `llm-fine-tuning-peft.md`.) - **The naive failure.** Resuming on new data at the *decayed* end-of-run LR barely adapts; resuming at the *original high* LR causes a **loss spike** and **catastrophic forgetting** of the old distribution. - **The recipe that works** *(Gupta et al. 2023, arXiv:2308.04014; Ibrahim et al. 2024, "Simple and Scalable Strategies to Continually Pre-train LLMs", arXiv:2403.08763).* Three ingredients: **(1) LR re-warming** (ramp the LR back up at the start of the new phase) **+ (2) LR re-decaying** (cosine/WSD decay over the new phase) **+ (3) replay** (mix in a **modest fraction of the *old* data**, e.g. 1–5%, to prevent forgetting). This combination **matches the performance of fully retraining from scratch on the union of old+new data, at a fraction of the compute** — the headline result that made CPT standard practice for domain/language/freshness updates. - **Domain-adaptive pretraining (DAPT).** The same machinery aimed at a *domain* (code, biomed, legal, finance): continue pretraining on in-domain text before any task fine-tuning. Strong when the domain is far from the base distribution; combine with replay to avoid losing general ability. Related: GQA "uptraining" (converting MHA→GQA with ~5% of pretraining compute) is a continual-pretraining-style cheap conversion — see `transformer-architecture.md`. > Where CPT meets the schedule: re-warm/re-decay is literally a fresh WSD/cosine cycle (§9); the new domain data is often introduced in the **decay/annealing phase** for the same reason §9 gives. --- ## 11. Evaluating a base (pre-instruct) model A base model is a **raw next-token predictor** — it does **not follow instructions** or chat. Evaluating it requires different methods than an aligned model, and confusing the two is a common error. - **Intrinsic: perplexity / loss on held-out text.** The most reliable signal *during* pretraining — smooth, comparable across checkpoints, no prompt-format confound. Compare on a **fixed, decontaminated** held-out set (and remember §8: per-token loss is the smooth metric to forecast against). Caveat: perplexity is tokenizer-dependent, so only compare same-tokenizer models. - **Few-shot / in-context, log-likelihood scored.** Base models are evaluated **few-shot** (provide k exemplars in the prompt) because they have no zero-shot instruction-following. The standard tool is **EleutherAI's `lm-evaluation-harness`** (`lm-eval`), which scores any causal LM on the *same* inputs via **log-likelihood of answer choices** (multiple-choice: pick the highest-likelihood option — MMLU, ARC, HellaSwag) and **constrained generation** (GSM8K, etc.). Using the harness is how results stay comparable across papers. - **Base vs instruct is not "instruct is always better."** A 2025–2026 finding (arXiv:2601.13244, arXiv:2501.08716): **base models often *beat* their instruction-tuned versions in pure few-shot/zero-shot settings** (drops of ~30%+ reported for some instruct models zero-shot) — instruct tuning trades raw few-shot capability for prompt-following and safety. So **evaluate the base model on its own terms (few-shot, log-likelihood)**; do not judge it by chat behavior or zero-shot instruction tasks. This also informs the pretraining→alignment handoff: a strong base on few-shot benchmarks is the right thing to hand to `llm-alignment-post-training.md`. - **Always decontaminate first (§4).** A headline benchmark number is meaningless without decontamination, and even then prefer **time-gated / contamination-resistant** benchmarks for the number you trust. --- ## Anti-patterns (the costly mistakes) - **Applying Chinchilla 20× literally when you will serve the model.** You will ship a too-big, too-expensive model. Over-train a smaller one (§6). - **Applying Chinchilla to *repeated* data.** Chinchilla assumes unique tokens; with repetition use the data-constrained law and **train a smaller model for more epochs** (§6). And don't blow past ~4–16 epochs expecting fresh-token value. - **Skipping decontamination, or trusting n-gram decontamination alone.** Inflated evals; paraphrased leakage passes string matching (§4). Decontaminate *and* time-gate. - **Counting non-embedding params / under-counting FLOPs in scaling fits.** This is *exactly* the Kaplan artifact (§5) — count total params and all FLOPs, tune warmup per scale. - **Committing a cosine schedule to a fixed length, then wanting to extend or branch.** Use WSD so you can decay whenever and reuse the stable checkpoint (§9). - **Decaying LR too aggressively over your best (annealing-phase) data.** The model is barely learning by then — use moderate decay or decouple curriculum from LR (§9). - **Continual pretraining without re-warming or without replay.** Either fails to adapt (no re-warm) or catastrophically forgets (no replay) (§10). - **Judging a base model by chat / zero-shot instruction behavior.** Base models predict tokens; evaluate few-shot with log-likelihood (§11). - **Uniform data mixture / hand-set weights at scale.** Use a proxy-model-derived mixture (DoReMi) and upsample high-value domains in the anneal (§3, §9). - **Treating `6ND` as exact at long context or for MoE.** Add attention FLOPs at long seq; use *active* params for MoE (§7). ## Troubleshooting - **Loss plateaus / model under-performs at a given compute.** Check the `N:D` split against the regime (§5–6): likely under-trained on tokens (raise `D`, the Chinchilla lesson) or wrong mixture (§3). Verify warmup and LR floor (§9). - **Eval scores look too good.** Suspect contamination (§4) — re-run decontamination (semantic, not just n-gram) and re-test on a post-cutoff benchmark. - **A capability "suddenly appears" (or fails to).** Likely a metric artifact (§8) — re-measure with a smooth metric (log-prob, edit distance) to see the true trend before concluding. - **Adding a domain wrecked general ability.** Forgetting from CPT without replay (§10) — add 1–5% old-data replay and re-warm/re-decay. - **Multilingual / code tokenization is wasteful (high fertility).** Tokenizer trained on too-English a corpus or too-small a vocab (§4) — retrain on a representative mix, consider 128K+ vocab. - **Intermediate checkpoint is much worse than final under cosine.** Expected — the LR has not decayed yet; this is the WSD motivation (§9). ## References (primary; 2024–2026 unless seminal) **Scaling laws** 1. Kaplan et al. (2020), *Scaling Laws for Neural Language Models* — arXiv:2001.08361. 2. Hoffmann et al. (2022), *Training Compute-Optimal Large Language Models* (Chinchilla, ≈20 tok/param) — arXiv:2203.15556. 3. Muennighoff et al. (2023), *Scaling Data-Constrained Language Models* (≤4 epochs ≈ free, decay to ~40) — arXiv:2305.16264; JMLR 26 (2025) 24-1000. 4. Sardana & Frankle (2023/2024), *Beyond Chinchilla-Optimal: Accounting for Inference in LM Scaling Laws* — arXiv:2401.00448. 5. Porian et al. (2024), *Resolving Discrepancies in Compute-Optimal Scaling of Language Models* — arXiv:2406.12907 (NeurIPS 2024). Besiroglu et al. (2024), *Chinchilla Scaling: A replication attempt* — arXiv:2404.10102. 6. Casson (2023), *Transformer FLOPs* (the `6ND` / `2N`-per-token derivation) — adamcasson.com. **Objectives** 7. Bavarian et al. (2022, OpenAI), *Efficient Training of Language Models to Fill in the Middle* (FIM, PSM/SPM, FIM-for-free) — arXiv:2207.14255. 8. Tay et al. (2022, Google), *UL2: Unifying Language Learning Paradigms* (Mixture-of-Denoisers, R/S/X, mode-switch) — arXiv:2205.05131. **Data pipeline & mixtures** 9. Penedo et al. (2024, HuggingFace), *The FineWeb Datasets: Decanting the Web…* (15T tokens, ablated pipeline, FineWeb-Edu 1.3T) — arXiv:2406.17557. 10. Xie et al. (2023), *DoReMi: Optimizing Data Mixtures Speeds Up LM Pretraining* (Group DRO proxy reweighting) — arXiv:2305.10429 (NeurIPS 2023). 11. Yang et al. (2023), *Rethinking Benchmark and Contamination for LMs with Rephrased Samples* (n-gram decontamination is fragile) — arXiv:2311.04850. Survey: *Benchmark Data Contamination of LLMs* — arXiv:2406.04244. 12. Tokenizer training: *Tokenizer Choice for LLM Training: Negligible or Crucial?* — arXiv:2310.08754; *Diminishing Returns of Tokenization Training Data* — arXiv:2502.20273. **LR schedules, curriculum, continual pretraining** 13. Hu et al. (2024), *MiniCPM* (Warmup-Stable-Decay; ~192× data/model ratio) — arXiv:2404.06395. *River-valley landscape view of WSD* — arXiv:2410.05192. 14. Gupta et al. (2023), *Continual Pre-Training of LLMs: How to (re)warm your model?* — arXiv:2308.04014. 15. Ibrahim et al. (2024), *Simple and Scalable Strategies to Continually Pre-train LLMs* (re-warm + re-decay + replay = match full retrain) — arXiv:2403.08763. 16. *How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining* — arXiv:2511.18903. *Mid-Training of LLMs: A Survey* — arXiv:2510.06826. **Emergence & base-model evaluation** 17. Wei et al. (2022), *Emergent Abilities of Large Language Models* — arXiv:2206.07682. 18. Schaeffer et al. (2023), *Are Emergent Abilities of Large Language Models a Mirage?* (metric artifact; NeurIPS 2023 Outstanding Paper) — arXiv:2304.15004. 19. EleutherAI, *Language Model Evaluation Harness* (`lm-eval`) — github.com/EleutherAI/lm-evaluation-harness. Base-vs-instruct few-shot: arXiv:2601.13244, arXiv:2501.08716. *Compiled via /dr deep-research, 2026-05-31. Scaling-law constants, token/param ratios, and dataset sizes are research findings, not laws of nature — re-derive against your own data, tokenizer, and hardware. For the transformer block, tokenizer algorithms, and attention internals see `transformer-architecture.md`; for spending the `6ND` budget across a cluster (FSDP/ZeRO/parallelism) see `distributed-training.md`.*