LLM Pretraining & Scaling Laws

Parent: LLM Models and APIs · researched 2026-05-31T21:38:54.141Z· 19 sources · 11 concepts · skill llm-pretraining-scaling-laws

PROVENANCE: This reference is part of the ai-agent-engineering hub.

Overview

PROVENANCE: This reference is part of the ai-agent-engineering hub. [source]
Source: /dr deep-research run, 2026-05-31. Topic - LLM pretraining objectives, data pipeline, and scaling laws (2024–2026). [source]
Routed as a hub reference (not a standalone top-level skill) per hub-and-spoke strategy. [source]
Owns the LLM pretraining layer - how a base model is TRAINED FROM SCRATCH on a token budget: the objective, the data, the compute math, the LR schedule, and how to read a base checkpoint. This is the "pretraining sibling" that transformer-architecture.md defers objectives/scaling-laws questions to. [source]
Primary sources: Kaplan et al. 2020 (arXiv:2001.08361); Hoffmann et al. 2022 Chinchilla (arXiv:2203.15556); Muennighoff et al. 2023 data-constrained (arXiv:2305.16264, JMLR 2025); Sardana & Frankle 2023 inference-aware "Beyond Chinchilla-Optimal" (arXiv:2401.00448); Bavarian et al. 2022 FIM (arXiv:2207.14255); Tay et al. 2022 UL2 (arXiv:2205.05131); Hu et al. 2024 MiniCPM/WSD (arXiv:2404.06395); Schaeffer et al. 2023 emergent-abilities-mirage (arXiv:2304.15004); Wei et al. 2022 emergent abilities (arXiv:2206.07682); Penedo et al. 2024 FineWeb (arXiv:2406.17557); Xie et al. 2023 DoReMi (arXiv:2305.10429); Gupta et al. 2023 re-warming (arXiv:2308.04014); Ibrahim et al. 2024 continual pretraining (arXiv:2403.08763); Porian et al. 2024 reconciling Kaplan/Chinchilla (arXiv:2406.12907, NeurIPS 2024); EleutherAI lm-evaluation-harness. [source]
The transformer BLOCK itself - attention/MoE/RoPE/norm, and tokenizer ALGORITHMS (BPE/SentencePiece/tiktoken) → transformer-architecture.md. Tokenizer training (vocab-size choice, training corpus, fertility) is HERE; the algorithm is THERE. [source]
Distributed-training infra - FSDP/ZeRO/3D parallelism, MFU engineering, the actual GPU mechanics of spending 6ND FLOPs → distributed-training.md (pointer only). Here we own the compute budget math and what to train, not the cluster. [source]
Post-training - SFT/RLHF/DPO/Constitutional AI → llm-alignment-post-training.md. Pretraining ENDS at the base checkpoint; everything after is alignment. [source]
Fine-tuning / PEFT (LoRA/QLoRA, domain SFT) → llm-fine-tuning-peft.md. Continual pretraining (more next-token prediction on new corpora) is HERE; adapting with supervised data is THERE. [source]
Reasoning RL / test-time compute (GRPO, RLVR, long-CoT) → reasoning-models.md. [source]
Offline benchmark-harness mechanics (HELM/MMLU/LLM-as-Judge scoring internals) → da-7-machine-learning (da-analytical-methods). Here we cover base-model eval strategy and decontamination, not the harness internals. [source]

LLM Pretraining & Scaling Laws

Pretraining is the expensive part: take a randomly-initialized transformer and run next-token prediction over trillions of tokens until it becomes a base model - a raw next-token predictor with broad world knowledge but no instruction-following. Everything in llm-alignment-post-training.md, llm-fine-tuning-peft.md, and reasoning-models.md starts from this checkpoint. This reference answers the four questions that define a pretraining run: what objective, what data, how big / how long (scaling laws + the compute budget), and how do I know it worked (base-model eval). [source]
The one identity that anchors everything here - C ≈ 6ND. A dense transformer with N parameters trained on D tokens costs about 6ND floating-point operations. That single equation is the budget line: every scaling-law result (Kaplan, Chinchilla, data-constrained, inference-aware) is an answer to "given a fixed C, how do I split it between N and D to minimize loss?" Hold C = 6ND and the whole field becomes a constrained-optimization story. The 2022 → 2026 arc is the field realizing that the loss-optimal split (N : D) and the deployment-optimal split are different things - and that data, not parameters, is now the binding constraint. [source]

Scope boundary (read first)

This reference = pretraining a base model: objectives (causal LM, MLM, prefix-LM, FIM, UL2), the data pipeline (curation, dedup, filtering, mixtures, tokenizer training, decontamination), scaling laws (Kaplan, Chinchilla, data-constrained, inference-aware), the 6ND compute budget, emergent abilities + the mirage debate, LR schedules at scale (cosine, WSD), data curriculum/annealing, continual/domain-adaptive pretraining, and base-model evaluation. [source]
The transformer architecture - attention, MoE, RoPE, RMSNorm, and the tokenizer algorithms (BPE/byte-level-BPE/SentencePiece/tiktoken) → transformer-architecture.md. The line: what to train and on how much data is here; how the block is wired is there. Tokenizer training (picking vocab size, the training corpus, measuring fertility) is here because it is a data decision; the tokenization algorithm is architecture. [source]
Distributed-training infrastructure - FSDP, DeepSpeed ZeRO, tensor/pipeline/expert parallelism, gradient checkpointing, MFU (Model FLOPs Utilization) - is the engineering of spending 6ND FLOPs across a cluster → distributed-training.md. We own the budget arithmetic (6ND, token/param ratios) and what to train; that sibling owns how the GPUs cooperate. [source]
Post-training (SFT, RLHF/PPO, DPO family, Constitutional AI, RLAIF) → llm-alignment-post-training.md. Pretraining produces the base model; alignment turns it into an assistant. The handoff is the base checkpoint. [source]
Fine-tuning / PEFT (LoRA/QLoRA, domain SFT, instruction tuning) → llm-fine-tuning-peft.md. Continual pretraining (more self-supervised next-token prediction on a new corpus, §10) lives here; supervised adaptation lives there. [source]
Reasoning RL & test-time compute (GRPO, RLVR, long-CoT, the DeepSeek-R1 recipe) → reasoning-models.md. Note: "inference-time scaling laws" there (spend more compute at answer time) are a different axis from the pretraining scaling laws here (spend more compute at train time) - §6 distinguishes them. [source]
Benchmark-harness mechanics (how MMLU/HELM/LLM-as-Judge are scored) → da-7-machine-learning. Base-eval strategy and decontamination are here. [source]

1. Pretraining objectives — what loss is the model minimizing?

The objective decides what the base model is good at before any post-training. Five matter (2024–2026). [source]
Causal / autoregressive LM (CLM, next-token prediction). Predict token t from tokens <t with a causal mask; cross-entropy loss on every position. This is the dominant objective for every modern generative LLM (GPT, Llama, Mistral, Qwen, DeepSeek, Gemma). It is self-supervised (the label is the next token), trivially parallel over positions during training (teacher forcing), and produces a model that can generate. Loss is reported as cross-entropy in nats; perplexity = exp(loss). [source]
Masked LM (MLM). Corrupt ~15% of tokens with [MASK] and predict them from bidirectional context (BERT). Produces strong encoders for understanding/embedding tasks but cannot generate left-to-right. Largely displaced for generative pretraining; still relevant for embedding models and retrievers. Boundary: embedding-model use lives in rag-architecture.md / ai-datastores.md. [source]
Prefix-LM. A hybrid: bidirectional attention over a prefix, causal attention over the continuation (one sequence, a "non-causal prefix"). Used by the UL2 S-denoiser and by some encoder-decoder setups; lets the model fully attend to the conditioning context while still generating. [source]
Fill-in-the-Middle (FIM). (Bavarian et al. 2022, arXiv:2207.14255.) Teach a causal model to infill by a pure data transformation - no architecture change. Split a document into (prefix, middle, suffix) and reorder into Prefix–Suffix–Middle (PSM) or Suffix–Prefix–Middle (SPM) with sentinel tokens (<PRE>, <SUF>, <MID>), then train next-token as usual. The model learns to generate the middle given both ends. Key results: it is "free" - the FIM rate (fraction of docs transformed, 50–90% is fine) buys infilling with no loss on left-to-right ability (the "FIM-for-free property"); context-level FIM (transform after chunking to context length) beats document-level; joint PSM+SPM training transfers positively. This is why code models (StarCoder, Codestral, DeepSeek-Coder) and modern general models support <FIM> infilling and IDE tab-completion. [source]
UL2 - Mixture-of-Denoisers (MoD). (Tay et al. 2022, arXiv:2205.05131.) Unify objectives by training on a mixture of span-corruption "denoisers": R-denoiser (regular T5-style short spans, ~15% / span≈3), S-denoiser (sequential = prefix-LM, corrupt a contiguous tail → forces generation), X-denoiser ("extreme": long spans and/or high corruption rate, the hardest). A paradigm token ([R]/[S]/[X]) prepended at train time lets you mode-switch the model toward the most suitable behavior downstream. UL2 showed one objective family can be competitive across both understanding and generation; the conceptual legacy (mix easy + hard self-supervised tasks; the prefix-LM-as-S-denoiser framing) outlived the specific recipe. [source]
Decision in practice (2026): generative LLMs use CLM, often + FIM (especially for code), trained on packed sequences (multiple documents concatenated to a fixed context length with separators - packing keeps GPUs full; whether to mask cross-document attention is a live choice). MLM is for encoders. UL2/prefix-LM matter mostly historically and for encoder-decoder niches. [source]

2. The pretraining data pipeline — curation, dedup, quality filtering

Data quality, not architecture, is the dominant lever on a fixed compute budget. The canonical open reference is FineWeb (Penedo et al. 2024, arXiv:2406.17557): 15T GPT-2 tokens from 96 Common Crawl snapshots (2013–early 2024), with every design choice ablated (each stage shown to monotonically improve downstream benchmarks). The pipeline stages: [source]
Text extraction. Pull main content from raw HTML/WARC (FineWeb uses trafilatura); good extraction beats Common Crawl's own WET text. [source]
Language ID + filtering. Classifier-based language detection; keep target languages above a confidence threshold. [source]
Quality / heuristic filtering. Rule-based filters (line-length, symbol-to-word ratio, fraction of duplicate lines, bad-words, repetition) à la MassiveText/Gopher and C4. Removes boilerplate, SEO spam, gibberish. [source]
Deduplication (the single highest-impact stage). Near-dedup with MinHash + LSH on n-gram shingles (FineWeb: per-snapshot MinHash dedup). Removes the long tail of near-identical pages. Counter-intuitive finding from FineWeb: global cross-snapshot dedup can hurt - it disproportionately removes recently re-crawled, often higher-quality content and over-upweights ancient low-quality pages; per-snapshot dedup worked better. Dedup also matters because repeated data interacts with epoch-counting (§5). [source]
Model-based quality classification. Train a lightweight classifier to score "is this high-quality / educational?" FineWeb-Edu is a 1.3T-token subset filtered by an educational-quality classifier (a linear/small head trained on Llama-3-70B annotations of educational value). Models pretrained on FineWeb-Edu show large gains on knowledge/reasoning benchmarks (MMLU, ARC) versus unfiltered FineWeb - strong evidence that aggressive quality filtering pays off, even at the cost of raw token count. [source]
Synthetic data is now a standard ingredient (rephrased web, textbook-style generation à la Phi, distilled chains). Risk: model collapse (degenerate distributions when training on too much un-curated model output across generations) - mitigate by anchoring to real data and limiting synthetic share. [source]
> Cross-ref: the tokenization algorithm (how text becomes IDs) is in transformer-architecture.md §10. Tokenizer training (choosing the vocab, the corpus to train it on) is §4 here because it is a data decision. [source]

3. Data mixtures & domain weighting — how much of each source?

Once you have cleaned sources (web, code, books, arXiv, Wikipedia, math, multilingual), you must choose mixture proportions (domain weights). This is a first-class hyperparameter - getting it wrong wastes compute. [source]
The Pile / fixed heuristic weights - early models hand-set weights (upweight Wikipedia/books, cap web). Simple, manual, suboptimal. [source]
DoReMi (Domain Reweighting with Minimax Optimization) (Xie et al. 2023, arXiv:2305.10429, NeurIPS 2023). Train a small proxy model (e.g. 280M) with Group DRO to find domain weights that minimize worst-case excess loss versus a reference model - without knowing downstream tasks. Reuse those weights to train a model 30× larger (8B). Result: +6.5% average few-shot accuracy over The-Pile default weights and the baseline accuracy reached in 2.6× fewer steps; perplexity improved across all domains even ones it downweighted. The principle - use a cheap proxy run to set mixture weights for the expensive run - generalizes (online/learned mixtures, RegMix, data-mixing laws). [source]
Mixture scaling laws. A live 2024–2026 thread: the optimal mixture shifts with scale and with token budget (and with repetition - §5). Web fraction that is optimal at 1T tokens is not optimal at 15T; code/math get upweighted as you train longer. Treat the mixture as scale-dependent, not fixed. [source]
Practical pattern: set a base mixture (heuristic or DoReMi-derived), then upsample high-value domains (code, math, curated/synthetic) during the annealing phase (§9) rather than uniformly - the decay phase is where domain emphasis is cheapest and most effective (the MiniCPM/Yi-Lightning recipe). [source]

4. Tokenizer training & eval-set decontamination

Tokenizer training (a data decision; algorithm → transformer-architecture.md). Before pretraining you train the tokenizer on a sample of the corpus and freeze it; vocab choice then constrains everything. [source]
Vocabulary size. A bigger vocab → fewer tokens per document (lower fertility, cheaper sequences, more text per context) but a larger embedding/unembedding matrix and rarer-token undertraining. ~128K is the modern sweet spot for multilingual models (Llama 3 moved to 128K byte-level BPE; many 2024–2026 models sit at 128K–256K). There are even scaling laws for vocabulary - larger models warrant larger vocabularies. [source]
Training corpus & multilinguality. Train the tokenizer on a mixture representative of the pretraining mix - an English-heavy tokenizer gives terrible fertility on other languages (more tokens per word → more expensive, worse). Returns on tokenizer training data diminish (1GB → 900GB studied; gains saturate early). [source]
Fertility & parity (tokens-per-word; cross-language token-count ratio) are the standard intrinsic metrics - but caveat: they are not always predictive of downstream quality, so validate the tokenizer on a small pretraining proxy, not on fertility alone. [source]
Eval-set decontamination (do not skip - it is how you avoid lying to yourself). Web-scale corpora contain copies of benchmark test sets; if MMLU/GSM8K leak into pretraining, your eval is inflated. [source]
n-gram overlap is the standard filter: scan the corpus and remove (or flag) documents overlapping a test item. GPT-3 used 13-gram overlap; GPT-4 used a 50-character span. Maximum-matching-subsequence (MMS) is a variant. [source]
n-gram matching is fragile. (Yang et al. 2023, "Rethinking Benchmark and Contamination… with Rephrased Samples", arXiv:2311.04850.) Paraphrased or translated test items slip past string matching and still contaminate - a model can memorize a rephrased benchmark and ace it while passing decontamination. Mitigations: embedding/semantic-similarity decontamination, contamination-resistant / freshly-collected benchmarks (post-cutoff data, e.g. LiveCodeBench, private held-out sets), and canary strings. Always decontaminate, then prefer time-gated evals for the headline number. [source]

5. Scaling laws I — Kaplan vs Chinchilla (compute-optimal allocation)

Scaling laws predict loss as a smooth power law in model size N, data D, and compute C, and tell you how to split a fixed C between N and D. [source]
Kaplan et al. 2020 (arXiv:2001.08361). First clean power laws: test loss falls predictably as a power of N, D, and C. Their compute-optimal prescription favored very large models - given more compute, grow N fast and D slowly (N_opt ∝ C^0.73). This drove the GPT-3 / Gopher / MT-NLG "scale parameters" era. (It turned out to be skewed - see the reconciliation below.) [source]
Chinchilla - Hoffmann et al. 2022 (arXiv:2203.15556). Refit the laws carefully and found Kaplan-era models were massively under-trained on data. Compute-optimal scaling is balanced: N_opt ∝ C^0.50 and D_opt ∝ C^0.50 - i.e. scale N and D equally (every doubling of params should double tokens). The famous heuristic: ≈ 20 tokens per parameter is compute-optimal. They trained Chinchilla (70B on 1.4T tokens) and it beat Gopher (280B) using the same compute - a smaller, longer-trained model won. Loss is fit as L(N, D) = E + A/N^α + B/D^β. This reset the field: GPT-3 (175B/300B tokens) was ~10× too few tokens. [source]
Reconciling the two (Porian et al. 2024, "Resolving Discrepancies…", arXiv:2406.12907, NeurIPS 2024; also Besiroglu et al. replication arXiv:2404.10102). Kaplan's 0.73 vs Chinchilla's 0.50 is mostly an artifact, explained by three things: (1) Kaplan counted only non-embedding parameters (Chinchilla counts all params); (2) Kaplan under-counted FLOPs by ignoring the last-layer/embedding cost at the small scales they used; (3) warmup duration and optimizer tuning were not adapted per model size. Fix all three and the curves collapse onto Chinchilla's 0.50. **Lesson: count total params and all FLOPs, tune warmup/LR per scale - then scaling fits agree. Epoch AI's replication broadly confirmed** Chinchilla's estimates while noting its confidence intervals were too tight. [source]

6. Scaling laws II — data-constrained & inference-aware (why Chinchilla is not the answer in 2026)

Chinchilla is training-compute-optimal, not deployment-optimal, and assumes unlimited unique data. Both assumptions break in practice. [source]
Inference-aware / "over-training" (Sardana & Frankle 2023, "Beyond Chinchilla-Optimal", arXiv:2401.00448). If you will serve the model to many users, total cost = training + inference, and inference cost scales with N. So you should deliberately train a smaller model on far more tokens than Chinchilla (>20 tok/param) - paying more at train time to get a cheaper, faster model forever. "Over-training" is a misnomer: it is only "over" relative to the training-optimal point. Llama 3 8B is the canonical example: ~15T tokens ≈ 1,875 tokens/param (vs Chinchilla's ~200B / 20× for an 8B), and loss kept improving log-linearly far past the Chinchilla point. Mid-2026 reality: almost every shipped model is deliberately over-trained. (Related 2026 thread: test-time scaling can make over-training even more attractive - a small over-trained model + inference-time compute beats a compute-optimal bigger one at equal serving cost; reasoning-time scaling itself → reasoning-models.md.) [source]
Data-constrained scaling (Muennighoff et al. 2023, arXiv:2305.16264, JMLR 2025). When unique tokens run out (we are approaching the limit of high-quality web text), you repeat data (multiple epochs). Findings, now load-bearing: repeating up to ~4 epochs is almost as good as fresh data (negligible loss penalty); gains continue but decay out to ~16 epochs (a repeated token retains ~63% of a fresh token's value around there) and approach zero by ~40 epochs. Their scaling law adds a decay term for repeated tokens and excess parameters, and prescribes smaller models trained for more epochs when data-bound (the opposite of naively applying Chinchilla to repeated data). **Allocating excess compute to more params also decays once data is fixed. Practical rule: ≤4 epochs is safe; 4–16 is diminishing; >16 wastes compute** - and budget extra FLOPs into quality filtering / synthetic data rather than blind repetition. [source]
Synthesis of the three regimes: Chinchilla (balanced) is the textbook answer when data and inference are free. Inference-aware (over-train a small model) is the answer when you will serve at scale. Data-constrained (repeat ≤4 epochs, prefer smaller models) is the answer when you are out of unique tokens. Mid-2026 frontier runs live at the intersection: small-ish models, heavily over-trained, on heavily-filtered + synthetic data, a few epochs at most. [source]

7. The compute budget — `C ≈ 6ND` and how to use it

The arithmetic that turns all of the above into a project plan. [source]
C ≈ 6ND FLOPs to train a dense model of N non-embedding params on D tokens. The 6 decomposes as 2 (forward) + 4 (backward) FLOPs per parameter per token: a matmul is one multiply + one add = 2 FLOPs per weight (so 2N forward per token), and the backward pass does ~2× the forward matmuls (gradient w.r.t. inputs and weights) = 4N. Multiply by D tokens → 6ND. Inference is ≈ 2ND (forward only). (Standard derivation; see the Chinchilla appendix and "Transformer FLOPs", Casson 2023.) [source]
What it is good for. Back-of-envelope budgeting: pick any two of {C, N, D} and solve the third. E.g. an 8B model on 15T tokens ≈ 6 × 8e9 × 15e12 ≈ 7.2e23 FLOPs. Combine with hardware throughput and MFU (Model FLOPs Utilization, the fraction of peak FLOP/s actually used - typically 30–55%) to get wall-clock and GPU-hours: time ≈ C / (peak_FLOP/s × MFU × num_GPUs). [source]
Caveats. 6ND ignores attention FLOPs (the O(seq²·d) term), which is fine while seq ≪ d_model · layers / seq but bites at long context. For MoE, use active params, not total, in N (only the routed experts fire per token). The engineering of hitting high MFU across a cluster - FSDP/ZeRO sharding, parallelism, gradient checkpointing - is the distributed-training sibling's job; 6ND is the budget, MFU-engineering is spending it efficiently. [source]

8. Emergent abilities & the "mirage" debate

The claim (Wei et al. 2022, arXiv:2206.07682). Some abilities are emergent: absent in small models, present in large ones, appearing sharply and unpredictably at a scale threshold (e.g. multi-step arithmetic, word unscrambling, certain BIG-Bench tasks) - not extrapolable from smaller models' performance. [source]
The rebuttal - "Are Emergent Abilities a Mirage?" (Schaeffer et al. 2023, arXiv:2304.15004, NeurIPS 2023 Outstanding Paper). The sharp jumps are often an artifact of the metric, not the model. Discontinuous/nonlinear metrics (exact-match, multiple-choice accuracy - all-or-nothing) manufacture apparent step-changes; switch to continuous/smooth metrics (token edit distance, Brier score, log-likelihood per token) on the same models and the curve becomes smooth and predictable. They reproduce "emergence" by choosing metrics and erase it by changing them, and predict where it will/won't appear on the GPT-3 family + BIG-Bench. [source]
The 2024–2026 synthesis (what to actually believe). Both are partly right and the distinction is practical: per-token loss scales smoothly and predictably (this is what scaling laws fit and what you should plan against); **downstream task scores under harsh metrics can still jump**, because crossing a usefulness threshold (the model finally gets the whole multi-step answer right) is real for the user even if the underlying capability grew smoothly. Operationally: forecast with smooth metrics (loss, log-prob), but don't be surprised when a hard pass/fail benchmark lurches. This is also why a base model can look unimpressive on accuracy yet be a fine pretraining checkpoint (§11). [source]

9. Learning-rate schedules at scale — cosine vs WSD, and the annealing phase

The LR schedule is a pretraining-specific lever with a surprising amount of pull on the final loss. [source]
Warmup. Linearly ramp LR from 0 over a few hundred–few thousand steps (and tune warmup per scale - it was one of the Kaplan/Chinchilla reconciliation factors, §5). Skipping warmup destabilizes early training. [source]
Cosine decay (the long-time default). After warmup, decay LR following a cosine curve to a small floor over the whole planned token budget. Works well but has a coupling problem: the schedule is tied to a pre-committed total step count, so you cannot cleanly extend a run or take a good intermediate checkpoint (any checkpoint before the end is at a high, un-decayed LR and underperforms). [source]
Warmup-Stable-Decay (WSD) (Hu et al. 2024, MiniCPM, arXiv:2404.06395). Three phases: **(1) warmup → (2) a long stable phase at a constant high LR → (3) a short, sharp decay/cooldown (often the last ~10–20%). Properties that made it a 2024–2026 favorite: (a)** during the stable phase loss sits higher than cosine, but the decay phase drops loss sharply, often below cosine's final loss; (b) it is compute-agnostic - the stable phase can run indefinitely and you decay whenever you decide to stop, so you can reuse the stable checkpoint for runs of different lengths; (c) the decay phase is the natural place to anneal in high-quality / domain / synthetic data (see below). MiniCPM used WSD to run efficient data-model scaling studies and reported a much higher compute-optimal data/model ratio of ~192× (vs Chinchilla's 20×) - reinforcing the over-training story (§6). A "river-valley loss landscape" account (arXiv:2410.05192) explains why the sharp decay helps. [source]
The annealing/decay phase = where the data curriculum lives (§3, §10). The dominant 2024–2026 pattern is two-phase pretraining: phase 1 trains on broad web data at high/stable LR; the annealing phase (the WSD decay, or the cosine tail) upsamples high-quality, instruct-like, synthetic, math/code, and rare-language data while LR drops. Yi-Lightning's three-stage recipe (diversity → upsample-high-quality-during-anneal → fast-decay-on-best-data) is representative. Caveat (arXiv:2511.18903): **aggressive LR decay can waste your best data** (the model is barely learning by the time you feed it) - a moderate decay, or decoupling the curriculum from LR via model averaging / a constant LR + checkpoint-averaging, can do better. Continual-pretraining "infinite LR" schedules (arXiv:2503.02844) push the WSD idea further for never-ending training. [source]

10. Continual & domain-adaptive pretraining

You rarely retrain from scratch to add a domain, a language, or fresher data. Continual pretraining (CPT) = keep doing next-token prediction on a new corpus starting from an existing base checkpoint. (Distinct from fine-tuning/PEFT, which uses supervised data → llm-fine-tuning-peft.md.) [source]
The naive failure. Resuming on new data at the decayed end-of-run LR barely adapts; resuming at the original high LR causes a loss spike and catastrophic forgetting of the old distribution. [source]
The recipe that works (Gupta et al. 2023, arXiv:2308.04014; Ibrahim et al. 2024, "Simple and Scalable Strategies to Continually Pre-train LLMs", arXiv:2403.08763). Three ingredients: (1) LR re-warming (ramp the LR back up at the start of the new phase) + (2) LR re-decaying (cosine/WSD decay over the new phase) + (3) replay (mix in a **modest fraction of the old data, e.g. 1–5%, to prevent forgetting). This combination matches the performance of fully retraining from scratch on the union of old+new data, at a fraction of the compute** - the headline result that made CPT standard practice for domain/language/freshness updates. [source]
Domain-adaptive pretraining (DAPT). The same machinery aimed at a domain (code, biomed, legal, finance): continue pretraining on in-domain text before any task fine-tuning. Strong when the domain is far from the base distribution; combine with replay to avoid losing general ability. Related: GQA "uptraining" (converting MHA→GQA with ~5% of pretraining compute) is a continual-pretraining-style cheap conversion - see transformer-architecture.md. [source]
> Where CPT meets the schedule: re-warm/re-decay is literally a fresh WSD/cosine cycle (§9); the new domain data is often introduced in the decay/annealing phase for the same reason §9 gives. [source]

11. Evaluating a base (pre-instruct) model

A base model is a raw next-token predictor - it does not follow instructions or chat. Evaluating it requires different methods than an aligned model, and confusing the two is a common error. [source]
Intrinsic: perplexity / loss on held-out text. The most reliable signal during pretraining - smooth, comparable across checkpoints, no prompt-format confound. Compare on a fixed, decontaminated held-out set (and remember §8: per-token loss is the smooth metric to forecast against). Caveat: perplexity is tokenizer-dependent, so only compare same-tokenizer models. [source]
Few-shot / in-context, log-likelihood scored. Base models are evaluated few-shot (provide k exemplars in the prompt) because they have no zero-shot instruction-following. The standard tool is EleutherAI's lm-evaluation-harness (lm-eval), which scores any causal LM on the same inputs via log-likelihood of answer choices (multiple-choice: pick the highest-likelihood option - MMLU, ARC, HellaSwag) and constrained generation (GSM8K, etc.). Using the harness is how results stay comparable across papers. [source]
Base vs instruct is not "instruct is always better." A 2025–2026 finding (arXiv:2601.13244, arXiv:2501.08716): **base models often beat their instruction-tuned versions in pure few-shot/zero-shot settings (drops of ~30%+ reported for some instruct models zero-shot) - instruct tuning trades raw few-shot capability for prompt-following and safety. So evaluate the base model on its own terms (few-shot, log-likelihood)**; do not judge it by chat behavior or zero-shot instruction tasks. This also informs the pretraining→alignment handoff: a strong base on few-shot benchmarks is the right thing to hand to llm-alignment-post-training.md. [source]
Always decontaminate first (§4). A headline benchmark number is meaningless without decontamination, and even then prefer time-gated / contamination-resistant benchmarks for the number you trust. [source]

Anti-patterns (the costly mistakes)

Applying Chinchilla 20× literally when you will serve the model. You will ship a too-big, too-expensive model. Over-train a smaller one (§6). [source]
**Applying Chinchilla to repeated data. Chinchilla assumes unique tokens; with repetition use the data-constrained law and train a smaller model for more epochs** (§6). And don't blow past ~4–16 epochs expecting fresh-token value. [source]
Skipping decontamination, or trusting n-gram decontamination alone. Inflated evals; paraphrased leakage passes string matching (§4). Decontaminate and time-gate. [source]
Counting non-embedding params / under-counting FLOPs in scaling fits. This is exactly the Kaplan artifact (§5) - count total params and all FLOPs, tune warmup per scale. [source]
Committing a cosine schedule to a fixed length, then wanting to extend or branch. Use WSD so you can decay whenever and reuse the stable checkpoint (§9). [source]
Decaying LR too aggressively over your best (annealing-phase) data. The model is barely learning by then - use moderate decay or decouple curriculum from LR (§9). [source]
Continual pretraining without re-warming or without replay. Either fails to adapt (no re-warm) or catastrophically forgets (no replay) (§10). [source]
Judging a base model by chat / zero-shot instruction behavior. Base models predict tokens; evaluate few-shot with log-likelihood (§11). [source]
Uniform data mixture / hand-set weights at scale. Use a proxy-model-derived mixture (DoReMi) and upsample high-value domains in the anneal (§3, §9). [source]
Treating 6ND as exact at long context or for MoE. Add attention FLOPs at long seq; use active params for MoE (§7). [source]

Troubleshooting

Loss plateaus / model under-performs at a given compute. Check the N:D split against the regime (§5–6): likely under-trained on tokens (raise D, the Chinchilla lesson) or wrong mixture (§3). Verify warmup and LR floor (§9). [source]
Eval scores look too good. Suspect contamination (§4) - re-run decontamination (semantic, not just n-gram) and re-test on a post-cutoff benchmark. [source]
A capability "suddenly appears" (or fails to). Likely a metric artifact (§8) - re-measure with a smooth metric (log-prob, edit distance) to see the true trend before concluding. [source]
Adding a domain wrecked general ability. Forgetting from CPT without replay (§10) - add 1–5% old-data replay and re-warm/re-decay. [source]
Multilingual / code tokenization is wasteful (high fertility). Tokenizer trained on too-English a corpus or too-small a vocab (§4) - retrain on a representative mix, consider 128K+ vocab. [source]
Intermediate checkpoint is much worse than final under cosine. Expected - the LR has not decayed yet; this is the WSD motivation (§9). [source]

References (primary; 2024–2026 unless seminal)

Kaplan et al. (2020), Scaling Laws for Neural Language Models - arXiv:2001.08361. [source]
Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla, ≈20 tok/param) - arXiv:2203.15556. [source]
Muennighoff et al. (2023), Scaling Data-Constrained Language Models (≤4 epochs ≈ free, decay to ~40) - arXiv:2305.16264; JMLR 26 (2025) 24-1000. [source]
Sardana & Frankle (2023/2024), Beyond Chinchilla-Optimal: Accounting for Inference in LM Scaling Laws - arXiv:2401.00448. [source]
Porian et al. (2024), Resolving Discrepancies in Compute-Optimal Scaling of Language Models - arXiv:2406.12907 (NeurIPS 2024). Besiroglu et al. (2024), Chinchilla Scaling: A replication attempt - arXiv:2404.10102. [source]
Casson (2023), Transformer FLOPs (the 6ND / 2N-per-token derivation) - adamcasson.com. [source]
Bavarian et al. (2022, OpenAI), Efficient Training of Language Models to Fill in the Middle (FIM, PSM/SPM, FIM-for-free) - arXiv:2207.14255. [source]
Tay et al. (2022, Google), UL2: Unifying Language Learning Paradigms (Mixture-of-Denoisers, R/S/X, mode-switch) - arXiv:2205.05131. [source]
Data pipeline & mixtures [source]
Penedo et al. (2024, HuggingFace), The FineWeb Datasets: Decanting the Web… (15T tokens, ablated pipeline, FineWeb-Edu 1.3T) - arXiv:2406.17557. [source]
Xie et al. (2023), DoReMi: Optimizing Data Mixtures Speeds Up LM Pretraining (Group DRO proxy reweighting) - arXiv:2305.10429 (NeurIPS 2023). [source]
Yang et al. (2023), Rethinking Benchmark and Contamination for LMs with Rephrased Samples (n-gram decontamination is fragile) - arXiv:2311.04850. Survey: Benchmark Data Contamination of LLMs - arXiv:2406.04244. [source]
Tokenizer training: Tokenizer Choice for LLM Training: Negligible or Crucial? - arXiv:2310.08754; Diminishing Returns of Tokenization Training Data - arXiv:2502.20273. [source]
LR schedules, curriculum, continual pretraining [source]
Hu et al. (2024), MiniCPM (Warmup-Stable-Decay; ~192× data/model ratio) - arXiv:2404.06395. River-valley landscape view of WSD - arXiv:2410.05192. [source]
Gupta et al. (2023), Continual Pre-Training of LLMs: How to (re)warm your model? - arXiv:2308.04014. [source]
Ibrahim et al. (2024), Simple and Scalable Strategies to Continually Pre-train LLMs (re-warm + re-decay + replay = match full retrain) - arXiv:2403.08763. [source]
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining - arXiv:2511.18903. Mid-Training of LLMs: A Survey - arXiv:2510.06826. [source]
Emergence & base-model evaluation [source]
Wei et al. (2022), Emergent Abilities of Large Language Models - arXiv:2206.07682. [source]
Schaeffer et al. (2023), Are Emergent Abilities of Large Language Models a Mirage? (metric artifact; NeurIPS 2023 Outstanding Paper) - arXiv:2304.15004. [source]
EleutherAI, Language Model Evaluation Harness (lm-eval) - github.com/EleutherAI/lm-evaluation-harness. Base-vs-instruct few-shot: arXiv:2601.13244, arXiv:2501.08716. [source]

Children

Pretraining objectives (causal/autoregressive LM, masked LM, prefix-LM, FIM, UL2 mixture-of-denoisers) (frontier)
Pretraining data pipeline (web curation, MinHash/LSH dedup, quality filtering, FineWeb/FineWeb-Edu) (frontier)
Data mixtures & domain weighting (DoReMi Group-DRO proxy reweighting) (frontier)
Tokenizer training (vocab size ~128K, fertility/parity) & eval-set decontamination (frontier)
Kaplan vs Chinchilla compute-optimal scaling (N∝C^0.5, ~20 tokens/param) & reconciliation (frontier)
Data-constrained scaling (Muennighoff, ≤4 epochs) & inference-aware / over-training (Sardana & Frankle) (frontier)
The C≈6ND compute budget (2N forward + 4N backward, MoE active-params, MFU) (frontier)
Emergent abilities & the mirage debate (Wei vs Schaeffer) (frontier)
LR schedules at scale (cosine vs Warmup-Stable-Decay/WSD, MiniCPM) & data annealing (frontier)
Continual & domain-adaptive pretraining (LR re-warm + re-decay + replay) (frontier)
Base-model (pre-instruct) evaluation (perplexity, few-shot log-likelihood, lm-evaluation-harness) (frontier)

Frontier under this node: Base-model (pre-instruct) evaluation (perplexity, few-shot log-likelihood, lm-evaluation-harness), Continual & domain-adaptive pretraining (LR re-warm + re-decay + replay), Data mixtures & domain weighting (DoReMi Group-DRO proxy reweighting), Data-constrained scaling (Muennighoff, ≤4 epochs) & inference-aware / over-training (Sardana & Frankle), Emergent abilities & the mirage debate (Wei vs Schaeffer), Kaplan vs Chinchilla compute-optimal scaling (N∝C^0.5, ~20 tokens/param) & reconciliation, LR schedules at scale (cosine vs Warmup-Stable-Decay/WSD, MiniCPM) & data annealing, Pretraining data pipeline (web curation, MinHash/LSH dedup, quality filtering, FineWeb/FineWeb-Edu), Pretraining objectives (causal/autoregressive LM, masked LM, prefix-LM, FIM, UL2 mixture-of-denoisers), The C≈6ND compute budget (2N forward + 4N backward, MoE active-params, MFU), Tokenizer training (vocab size ~128K, fertility/parity) & eval-set decontamination

← the whole tree · 3D view· how to read this page