LLM Pretraining & Scaling Laws

Parent: LLM Models and APIs · Published reference · snapshot 2026-09-08 · skill ai-llm-model-layer/references/llm-pretraining-scaling-laws.md

↓ Facts as markdown all context files

PROVENANCE: This reference is part of the ai-agent-engineering hub.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Overview

[archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#overview]

LLM Pretraining & Scaling Laws

Pretraining is the expensive part: take a randomly-initialized transformer and run next-token prediction over trillions of tokens until it becomes a base model - a raw next-token predictor with broad world knowledge but no instruction-following. Everything in llm-alignment-post-training.md, llm-fine-tuning-peft.md, and reasoning-models.md starts from this checkpoint. This reference answers the four questions that define a pretraining run: what objective, what data, how big / how long (scaling laws + the compute budget), and how do I know it worked (base-model eval). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#llm-pretraining-scaling-laws]
The one identity that anchors everything here - C ≈ 6ND. A dense transformer with N parameters trained on D tokens costs about 6ND floating-point operations. That single equation is the budget line: every scaling-law result (Kaplan, Chinchilla, data-constrained, inference-aware) is an answer to "given a fixed C, how do I split it between N and D to minimize loss?" Hold C = 6ND and the whole field becomes a constrained-optimization story. The 2022 → 2026 arc is the field realizing that the loss-optimal split (N : D) and the deployment-optimal split are different things - and that data, not parameters, is now the binding constraint. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#llm-pretraining-scaling-laws]

Scope boundary (read first)

This reference = pretraining a base model: objectives (causal LM, MLM, prefix-LM, FIM, UL2), the data pipeline (curation, dedup, filtering, mixtures, tokenizer training, decontamination), scaling laws (Kaplan, Chinchilla, data-constrained, inference-aware), the 6ND compute budget, emergent abilities + the mirage debate, LR schedules at scale (cosine, WSD), data curriculum/annealing, continual/domain-adaptive pretraining, and base-model evaluation. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
The transformer architecture - attention, MoE, RoPE, RMSNorm, and the tokenizer algorithms (BPE/byte-level-BPE/SentencePiece/tiktoken) → transformer-architecture.md. The line: what to train and on how much data is here; how the block is wired is there. Tokenizer training (picking vocab size, the training corpus, measuring fertility) is here because it is a data decision; the tokenization algorithm is architecture. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
Distributed-training infrastructure - FSDP, DeepSpeed ZeRO, tensor/pipeline/expert parallelism, gradient checkpointing, MFU (Model FLOPs Utilization) - is the engineering of spending 6ND FLOPs across a cluster → distributed-training.md. We own the budget arithmetic (6ND, token/param ratios) and what to train; that sibling owns how the GPUs cooperate. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
Post-training (SFT, RLHF/PPO, DPO family, Constitutional AI, RLAIF) → llm-alignment-post-training.md. Pretraining produces the base model; alignment turns it into an assistant. The handoff is the base checkpoint. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
Fine-tuning / PEFT (LoRA/QLoRA, domain SFT, instruction tuning) → llm-fine-tuning-peft.md. Continual pretraining (more self-supervised next-token prediction on a new corpus, §10) lives here; supervised adaptation lives there. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
Reasoning RL & test-time compute (GRPO, RLVR, long-CoT, the DeepSeek-R1 recipe) → reasoning-models.md. Note: "inference-time scaling laws" there (spend more compute at answer time) are a different axis from the pretraining scaling laws here (spend more compute at train time) - §6 distinguishes them. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]
Benchmark-harness mechanics (how MMLU/HELM/LLM-as-Judge are scored) → da-7-machine-learning. Base-eval strategy and decontamination are here. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#scope-boundary-read-first]

1. Pretraining objectives — what loss is the model minimizing?

The objective decides what the base model is good at before any post-training. Five matter (2024–2026). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
- Causal / autoregressive LM (CLM, next-token prediction). Predict token t from tokens <t with a causal mask; cross-entropy loss on every position. This is the dominant objective for every modern generative LLM (GPT, Llama, Mistral, Qwen, DeepSeek, Gemma). It is self-supervised (the label is the next token), trivially parallel over positions during training (teacher forcing), and produces a model that can generate. Loss is reported as cross-entropy in nats; perplexity = exp(loss). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
- Masked LM (MLM). Corrupt ~15% of tokens with [MASK] and predict them from bidirectional context (BERT). Produces strong encoders for understanding/embedding tasks but cannot generate left-to-right. Largely displaced for generative pretraining; still relevant for embedding models and retrievers. Boundary: embedding-model use lives in rag-architecture.md / ai-datastores.md. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
- Prefix-LM. A hybrid: bidirectional attention over a prefix, causal attention over the continuation (one sequence, a "non-causal prefix"). Used by the UL2 S-denoiser and by some encoder-decoder setups; lets the model fully attend to the conditioning context while still generating. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
- Fill-in-the-Middle (FIM). (Bavarian et al. 2022, arXiv:2207.14255.) Teach a causal model to infill by a pure data transformation - no architecture change. Split a document into (prefix, middle, suffix) and reorder into Prefix–Suffix–Middle (PSM) or Suffix–Prefix–Middle (SPM) with sentinel tokens (<PRE>, <SUF>, <MID>), then train next-token as usual. The model learns to generate the middle given both ends. Key results: it is "free" - the FIM rate (fraction of docs transformed, 50–90% is fine) buys infilling with no loss on left-to-right ability (the "FIM-for-free property"); context-level FIM (transform after chunking to context length) beats document-level; joint PSM+SPM training transfers positively. This is why code models (StarCoder, Codestral, DeepSeek-Coder) and modern general models support <FIM> infilling and IDE tab-completion. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
- UL2 - Mixture-of-Denoisers (MoD). (Tay et al. 2022, arXiv:2205.05131.) Unify objectives by training on a mixture of span-corruption "denoisers": R-denoiser (regular T5-style short spans, ~15% / span≈3), S-denoiser (sequential = prefix-LM, corrupt a contiguous tail → forces generation), X-denoiser ("extreme": long spans and/or high corruption rate, the hardest). A paradigm token ([R]/[S]/[X]) prepended at train time lets you mode-switch the model toward the most suitable behavior downstream. UL2 showed one objective family can be competitive across both understanding and generation; the conceptual legacy (mix easy + hard self-supervised tasks; the prefix-LM-as-S-denoiser framing) outlived the specific recipe. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]
Decision in practice (2026): generative LLMs use CLM, often + FIM (especially for code), trained on packed sequences (multiple documents concatenated to a fixed context length with separators - packing keeps GPUs full; whether to mask cross-document attention is a live choice). MLM is for encoders. UL2/prefix-LM matter mostly historically and for encoder-decoder niches. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#1-pretraining-objectives-what-loss-is-the-model-minimizing]

2. The pretraining data pipeline — curation, dedup, quality filtering

Data quality, not architecture, is the dominant lever on a fixed compute budget. The canonical open reference is FineWeb (Penedo et al. 2024, arXiv:2406.17557): 15T GPT-2 tokens from 96 Common Crawl snapshots (2013–early 2024), with every design choice ablated (each stage shown to monotonically improve downstream benchmarks). The pipeline stages: [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
- Text extraction. Pull main content from raw HTML/WARC (FineWeb uses trafilatura); good extraction beats Common Crawl's own WET text. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
- Language ID + filtering. Classifier-based language detection; keep target languages above a confidence threshold. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
- Quality / heuristic filtering. Rule-based filters (line-length, symbol-to-word ratio, fraction of duplicate lines, bad-words, repetition) à la MassiveText/Gopher and C4. Removes boilerplate, SEO spam, gibberish. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
- Deduplication (the single highest-impact stage). Near-dedup with MinHash + LSH on n-gram shingles (FineWeb: per-snapshot MinHash dedup). Removes the long tail of near-identical pages. Counter-intuitive finding from FineWeb: global cross-snapshot dedup can hurt - it disproportionately removes recently re-crawled, often higher-quality content and over-upweights ancient low-quality pages; per-snapshot dedup worked better. Dedup also matters because repeated data interacts with epoch-counting (§5). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
- Model-based quality classification. Train a lightweight classifier to score "is this high-quality / educational?" FineWeb-Edu is a 1.3T-token subset filtered by an educational-quality classifier (a linear/small head trained on Llama-3-70B annotations of educational value). Models pretrained on FineWeb-Edu show large gains on knowledge/reasoning benchmarks (MMLU, ARC) versus unfiltered FineWeb - strong evidence that aggressive quality filtering pays off, even at the cost of raw token count. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
Synthetic data is now a standard ingredient (rephrased web, textbook-style generation à la Phi, distilled chains). Risk: model collapse (degenerate distributions when training on too much un-curated model output across generations) - mitigate by anchoring to real data and limiting synthetic share. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]
> Cross-ref: the tokenization algorithm (how text becomes IDs) is in transformer-architecture.md §10. Tokenizer training (choosing the vocab, the corpus to train it on) is §4 here because it is a data decision. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#2-the-pretraining-data-pipeline-curation-dedup-quality-filtering]

3. Data mixtures & domain weighting — how much of each source?

Once you have cleaned sources (web, code, books, arXiv, Wikipedia, math, multilingual), you must choose mixture proportions (domain weights). This is a first-class hyperparameter - getting it wrong wastes compute. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#3-data-mixtures-domain-weighting-how-much-of-each-source]
- The Pile / fixed heuristic weights - early models hand-set weights (upweight Wikipedia/books, cap web). Simple, manual, suboptimal. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#3-data-mixtures-domain-weighting-how-much-of-each-source]
- DoReMi (Domain Reweighting with Minimax Optimization) (Xie et al. 2023, arXiv:2305.10429, NeurIPS 2023). Train a small proxy model (e.g. 280M) with Group DRO to find domain weights that minimize worst-case excess loss versus a reference model - without knowing downstream tasks. Reuse those weights to train a model 30× larger (8B). Result: +6.5% average few-shot accuracy over The-Pile default weights and the baseline accuracy reached in 2.6× fewer steps; perplexity improved across all domains even ones it downweighted. The principle - use a cheap proxy run to set mixture weights for the expensive run - generalizes (online/learned mixtures, RegMix, data-mixing laws). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#3-data-mixtures-domain-weighting-how-much-of-each-source]
- Mixture scaling laws. A live 2024–2026 thread: the optimal mixture shifts with scale and with token budget (and with repetition - §5). Web fraction that is optimal at 1T tokens is not optimal at 15T; code/math get upweighted as you train longer. Treat the mixture as scale-dependent, not fixed. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#3-data-mixtures-domain-weighting-how-much-of-each-source]
Practical pattern: set a base mixture (heuristic or DoReMi-derived), then upsample high-value domains (code, math, curated/synthetic) during the annealing phase (§9) rather than uniformly - the decay phase is where domain emphasis is cheapest and most effective (the MiniCPM/Yi-Lightning recipe). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#3-data-mixtures-domain-weighting-how-much-of-each-source]

4. Tokenizer training & eval-set decontamination

Tokenizer training (a data decision; algorithm → transformer-architecture.md). Before pretraining you train the tokenizer on a sample of the corpus and freeze it; vocab choice then constrains everything. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
- Vocabulary size. A bigger vocab → fewer tokens per document (lower fertility, cheaper sequences, more text per context) but a larger embedding/unembedding matrix and rarer-token undertraining. ~128K is the modern sweet spot for multilingual models (Llama 3 moved to 128K byte-level BPE; many 2024–2026 models sit at 128K–256K). There are even scaling laws for vocabulary - larger models warrant larger vocabularies. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
- Training corpus & multilinguality. Train the tokenizer on a mixture representative of the pretraining mix - an English-heavy tokenizer gives terrible fertility on other languages (more tokens per word → more expensive, worse). Returns on tokenizer training data diminish (1GB → 900GB studied; gains saturate early). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
- Fertility & parity (tokens-per-word; cross-language token-count ratio) are the standard intrinsic metrics - but caveat: they are not always predictive of downstream quality, so validate the tokenizer on a small pretraining proxy, not on fertility alone. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
Eval-set decontamination (do not skip - it is how you avoid lying to yourself). Web-scale corpora contain copies of benchmark test sets; if MMLU/GSM8K leak into pretraining, your eval is inflated. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
- n-gram overlap is the standard filter: scan the corpus and remove (or flag) documents overlapping a test item. GPT-3 used 13-gram overlap; GPT-4 used a 50-character span. Maximum-matching-subsequence (MMS) is a variant. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]
- n-gram matching is fragile. (Yang et al. 2023, "Rethinking Benchmark and Contamination… with Rephrased Samples", arXiv:2311.04850.) Paraphrased or translated test items slip past string matching and still contaminate - a model can memorize a rephrased benchmark and ace it while passing decontamination. Mitigations: embedding/semantic-similarity decontamination, contamination-resistant / freshly-collected benchmarks (post-cutoff data, e.g. LiveCodeBench, private held-out sets), and canary strings. Always decontaminate, then prefer time-gated evals for the headline number. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#4-tokenizer-training-eval-set-decontamination]

5. Scaling laws I — Kaplan vs Chinchilla (compute-optimal allocation)

Scaling laws predict loss as a smooth power law in model size N, data D, and compute C, and tell you how to split a fixed C between N and D. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#5-scaling-laws-i-kaplan-vs-chinchilla-compute-optimal-allocation]
- Kaplan et al. 2020 (arXiv:2001.08361). First clean power laws: test loss falls predictably as a power of N, D, and C. Their compute-optimal prescription favored very large models - given more compute, grow N fast and D slowly (N_opt ∝ C^0.73). This drove the GPT-3 / Gopher / MT-NLG "scale parameters" era. (It turned out to be skewed - see the reconciliation below.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#5-scaling-laws-i-kaplan-vs-chinchilla-compute-optimal-allocation]
- Chinchilla - Hoffmann et al. 2022 (arXiv:2203.15556). Refit the laws carefully and found Kaplan-era models were massively under-trained on data. Compute-optimal scaling is balanced: N_opt ∝ C^0.50 and D_opt ∝ C^0.50 - i.e. scale N and D equally (every doubling of params should double tokens). The famous heuristic: ≈ 20 tokens per parameter is compute-optimal. They trained Chinchilla (70B on 1.4T tokens) and it beat Gopher (280B) using the same compute - a smaller, longer-trained model won. Loss is fit as L(N, D) = E + A/N^α + B/D^β. This reset the field: GPT-3 (175B/300B tokens) was ~10× too few tokens. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#5-scaling-laws-i-kaplan-vs-chinchilla-compute-optimal-allocation]
- Reconciling the two (Porian et al. 2024, "Resolving Discrepancies…", arXiv:2406.12907, NeurIPS 2024; also Besiroglu et al. replication arXiv:2404.10102). Kaplan's 0.73 vs Chinchilla's 0.50 is mostly an artifact, explained by three things: (1) Kaplan counted only non-embedding parameters (Chinchilla counts all params); (2) Kaplan under-counted FLOPs by ignoring the last-layer/embedding cost at the small scales they used; (3) warmup duration and optimizer tuning were not adapted per model size. Fix all three and the curves collapse onto Chinchilla's 0.50. **Lesson: count total params and all FLOPs, tune warmup/LR per scale - then scaling fits agree. Epoch AI's replication broadly confirmed** Chinchilla's estimates while noting its confidence intervals were too tight. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#5-scaling-laws-i-kaplan-vs-chinchilla-compute-optimal-allocation]

6. Scaling laws II — data-constrained & inference-aware (why Chinchilla is not the answer in 2026)

Chinchilla is training-compute-optimal, not deployment-optimal, and assumes unlimited unique data. Both assumptions break in practice. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#6-scaling-laws-ii-data-constrained-inference-aware-why-chinchilla-is-not-the-answer-in-2026]
- Inference-aware / "over-training" (Sardana & Frankle 2023, "Beyond Chinchilla-Optimal", arXiv:2401.00448). If you will serve the model to many users, total cost = training + inference, and inference cost scales with N. So you should deliberately train a smaller model on far more tokens than Chinchilla (>20 tok/param) - paying more at train time to get a cheaper, faster model forever. "Over-training" is a misnomer: it is only "over" relative to the training-optimal point. Llama 3 8B is the canonical example: ~15T tokens ≈ 1,875 tokens/param (vs Chinchilla's ~200B / 20× for an 8B), and loss kept improving log-linearly far past the Chinchilla point. Mid-2026 reality: almost every shipped model is deliberately over-trained. (Related 2026 thread: test-time scaling can make over-training even more attractive - a small over-trained model + inference-time compute beats a compute-optimal bigger one at equal serving cost; reasoning-time scaling itself → reasoning-models.md.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#6-scaling-laws-ii-data-constrained-inference-aware-why-chinchilla-is-not-the-answer-in-2026]
- Data-constrained scaling (Muennighoff et al. 2023, arXiv:2305.16264, JMLR 2025). When unique tokens run out (we are approaching the limit of high-quality web text), you repeat data (multiple epochs). Findings, now load-bearing: repeating up to ~4 epochs is almost as good as fresh data (negligible loss penalty); gains continue but decay out to ~16 epochs (a repeated token retains ~63% of a fresh token's value around there) and approach zero by ~40 epochs. Their scaling law adds a decay term for repeated tokens and excess parameters, and prescribes smaller models trained for more epochs when data-bound (the opposite of naively applying Chinchilla to repeated data). **Allocating excess compute to more params also decays once data is fixed. Practical rule: ≤4 epochs is safe; 4–16 is diminishing; >16 wastes compute** - and budget extra FLOPs into quality filtering / synthetic data rather than blind repetition. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#6-scaling-laws-ii-data-constrained-inference-aware-why-chinchilla-is-not-the-answer-in-2026]
Synthesis of the three regimes: Chinchilla (balanced) is the textbook answer when data and inference are free. Inference-aware (over-train a small model) is the answer when you will serve at scale. Data-constrained (repeat ≤4 epochs, prefer smaller models) is the answer when you are out of unique tokens. Mid-2026 frontier runs live at the intersection: small-ish models, heavily over-trained, on heavily-filtered + synthetic data, a few epochs at most. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#6-scaling-laws-ii-data-constrained-inference-aware-why-chinchilla-is-not-the-answer-in-2026]

7. The compute budget — `C ≈ 6ND` and how to use it

The arithmetic that turns all of the above into a project plan. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#7-the-compute-budget-c-6nd-and-how-to-use-it]
- C ≈ 6ND FLOPs to train a dense model of N non-embedding params on D tokens. The 6 decomposes as 2 (forward) + 4 (backward) FLOPs per parameter per token: a matmul is one multiply + one add = 2 FLOPs per weight (so 2N forward per token), and the backward pass does ~2× the forward matmuls (gradient w.r.t. inputs and weights) = 4N. Multiply by D tokens → 6ND. Inference is ≈ 2ND (forward only). (Standard derivation; see the Chinchilla appendix and "Transformer FLOPs", Casson 2023.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#7-the-compute-budget-c-6nd-and-how-to-use-it]
- What it is good for. Back-of-envelope budgeting: pick any two of {C, N, D} and solve the third. E.g. an 8B model on 15T tokens ≈ 6 × 8e9 × 15e12 ≈ 7.2e23 FLOPs. Combine with hardware throughput and MFU (Model FLOPs Utilization, the fraction of peak FLOP/s actually used - typically 30–55%) to get wall-clock and GPU-hours: time ≈ C / (peak_FLOP/s × MFU × num_GPUs). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#7-the-compute-budget-c-6nd-and-how-to-use-it]
- Caveats. 6ND ignores attention FLOPs (the O(seq²·d) term), which is fine while seq ≪ d_model · layers / seq but bites at long context. For MoE, use active params, not total, in N (only the routed experts fire per token). The engineering of hitting high MFU across a cluster - FSDP/ZeRO sharding, parallelism, gradient checkpointing - is the distributed-training sibling's job; 6ND is the budget, MFU-engineering is spending it efficiently. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#7-the-compute-budget-c-6nd-and-how-to-use-it]

8. Emergent abilities & the "mirage" debate

The claim (Wei et al. 2022, arXiv:2206.07682). Some abilities are emergent: absent in small models, present in large ones, appearing sharply and unpredictably at a scale threshold (e.g. multi-step arithmetic, word unscrambling, certain BIG-Bench tasks) - not extrapolable from smaller models' performance. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#8-emergent-abilities-the-mirage-debate]
The rebuttal - "Are Emergent Abilities a Mirage?" (Schaeffer et al. 2023, arXiv:2304.15004, NeurIPS 2023 Outstanding Paper). The sharp jumps are often an artifact of the metric, not the model. Discontinuous/nonlinear metrics (exact-match, multiple-choice accuracy - all-or-nothing) manufacture apparent step-changes; switch to continuous/smooth metrics (token edit distance, Brier score, log-likelihood per token) on the same models and the curve becomes smooth and predictable. They reproduce "emergence" by choosing metrics and erase it by changing them, and predict where it will/won't appear on the GPT-3 family + BIG-Bench. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#8-emergent-abilities-the-mirage-debate]
The 2024–2026 synthesis (what to actually believe). Both are partly right and the distinction is practical: per-token loss scales smoothly and predictably (this is what scaling laws fit and what you should plan against); **downstream task scores under harsh metrics can still jump**, because crossing a usefulness threshold (the model finally gets the whole multi-step answer right) is real for the user even if the underlying capability grew smoothly. Operationally: forecast with smooth metrics (loss, log-prob), but don't be surprised when a hard pass/fail benchmark lurches. This is also why a base model can look unimpressive on accuracy yet be a fine pretraining checkpoint (§11). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#8-emergent-abilities-the-mirage-debate]

9. Learning-rate schedules at scale — cosine vs WSD, and the annealing phase

The LR schedule is a pretraining-specific lever with a surprising amount of pull on the final loss. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#9-learning-rate-schedules-at-scale-cosine-vs-wsd-and-the-annealing-phase]
- Warmup. Linearly ramp LR from 0 over a few hundred–few thousand steps (and tune warmup per scale - it was one of the Kaplan/Chinchilla reconciliation factors, §5). Skipping warmup destabilizes early training. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#9-learning-rate-schedules-at-scale-cosine-vs-wsd-and-the-annealing-phase]
- Cosine decay (the long-time default). After warmup, decay LR following a cosine curve to a small floor over the whole planned token budget. Works well but has a coupling problem: the schedule is tied to a pre-committed total step count, so you cannot cleanly extend a run or take a good intermediate checkpoint (any checkpoint before the end is at a high, un-decayed LR and underperforms). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#9-learning-rate-schedules-at-scale-cosine-vs-wsd-and-the-annealing-phase]
- Warmup-Stable-Decay (WSD) (Hu et al. 2024, MiniCPM, arXiv:2404.06395). Three phases: **(1) warmup → (2) a long stable phase at a constant high LR → (3) a short, sharp decay/cooldown (often the last ~10–20%). Properties that made it a 2024–2026 favorite: (a)** during the stable phase loss sits higher than cosine, but the decay phase drops loss sharply, often below cosine's final loss; (b) it is compute-agnostic - the stable phase can run indefinitely and you decay whenever you decide to stop, so you can reuse the stable checkpoint for runs of different lengths; (c) the decay phase is the natural place to anneal in high-quality / domain / synthetic data (see below). MiniCPM used WSD to run efficient data-model scaling studies and reported a much higher compute-optimal data/model ratio of ~192× (vs Chinchilla's 20×) - reinforcing the over-training story (§6). A "river-valley loss landscape" account (arXiv:2410.05192) explains why the sharp decay helps. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#9-learning-rate-schedules-at-scale-cosine-vs-wsd-and-the-annealing-phase]
- The annealing/decay phase = where the data curriculum lives (§3, §10). The dominant 2024–2026 pattern is two-phase pretraining: phase 1 trains on broad web data at high/stable LR; the annealing phase (the WSD decay, or the cosine tail) upsamples high-quality, instruct-like, synthetic, math/code, and rare-language data while LR drops. Yi-Lightning's three-stage recipe (diversity → upsample-high-quality-during-anneal → fast-decay-on-best-data) is representative. Caveat (arXiv:2511.18903): **aggressive LR decay can waste your best data** (the model is barely learning by the time you feed it) - a moderate decay, or decoupling the curriculum from LR via model averaging / a constant LR + checkpoint-averaging, can do better. Continual-pretraining "infinite LR" schedules (arXiv:2503.02844) push the WSD idea further for never-ending training. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#9-learning-rate-schedules-at-scale-cosine-vs-wsd-and-the-annealing-phase]

10. Continual & domain-adaptive pretraining

You rarely retrain from scratch to add a domain, a language, or fresher data. Continual pretraining (CPT) = keep doing next-token prediction on a new corpus starting from an existing base checkpoint. (Distinct from fine-tuning/PEFT, which uses supervised data → llm-fine-tuning-peft.md.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#10-continual-domain-adaptive-pretraining]
- The naive failure. Resuming on new data at the decayed end-of-run LR barely adapts; resuming at the original high LR causes a loss spike and catastrophic forgetting of the old distribution. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#10-continual-domain-adaptive-pretraining]
- The recipe that works (Gupta et al. 2023, arXiv:2308.04014; Ibrahim et al. 2024, "Simple and Scalable Strategies to Continually Pre-train LLMs", arXiv:2403.08763). Three ingredients: (1) LR re-warming (ramp the LR back up at the start of the new phase) + (2) LR re-decaying (cosine/WSD decay over the new phase) + (3) replay (mix in a **modest fraction of the old data, e.g. 1–5%, to prevent forgetting). This combination matches the performance of fully retraining from scratch on the union of old+new data, at a fraction of the compute** - the headline result that made CPT standard practice for domain/language/freshness updates. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#10-continual-domain-adaptive-pretraining]
- Domain-adaptive pretraining (DAPT). The same machinery aimed at a domain (code, biomed, legal, finance): continue pretraining on in-domain text before any task fine-tuning. Strong when the domain is far from the base distribution; combine with replay to avoid losing general ability. Related: GQA "uptraining" (converting MHA→GQA with ~5% of pretraining compute) is a continual-pretraining-style cheap conversion - see transformer-architecture.md. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#10-continual-domain-adaptive-pretraining]
> Where CPT meets the schedule: re-warm/re-decay is literally a fresh WSD/cosine cycle (§9); the new domain data is often introduced in the decay/annealing phase for the same reason §9 gives. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#10-continual-domain-adaptive-pretraining]

11. Evaluating a base (pre-instruct) model

A base model is a raw next-token predictor - it does not follow instructions or chat. Evaluating it requires different methods than an aligned model, and confusing the two is a common error. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#11-evaluating-a-base-pre-instruct-model]
- Intrinsic: perplexity / loss on held-out text. The most reliable signal during pretraining - smooth, comparable across checkpoints, no prompt-format confound. Compare on a fixed, decontaminated held-out set (and remember §8: per-token loss is the smooth metric to forecast against). Caveat: perplexity is tokenizer-dependent, so only compare same-tokenizer models. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#11-evaluating-a-base-pre-instruct-model]
- Few-shot / in-context, log-likelihood scored. Base models are evaluated few-shot (provide k exemplars in the prompt) because they have no zero-shot instruction-following. The standard tool is EleutherAI's lm-evaluation-harness (lm-eval), which scores any causal LM on the same inputs via log-likelihood of answer choices (multiple-choice: pick the highest-likelihood option - MMLU, ARC, HellaSwag) and constrained generation (GSM8K, etc.). Using the harness is how results stay comparable across papers. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#11-evaluating-a-base-pre-instruct-model]
- Base vs instruct is not "instruct is always better." A 2025–2026 finding (arXiv:2601.13244, arXiv:2501.08716): **base models often beat their instruction-tuned versions in pure few-shot/zero-shot settings (drops of ~30%+ reported for some instruct models zero-shot) - instruct tuning trades raw few-shot capability for prompt-following and safety. So evaluate the base model on its own terms (few-shot, log-likelihood)**; do not judge it by chat behavior or zero-shot instruction tasks. This also informs the pretraining→alignment handoff: a strong base on few-shot benchmarks is the right thing to hand to llm-alignment-post-training.md. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#11-evaluating-a-base-pre-instruct-model]
- Always decontaminate first (§4). A headline benchmark number is meaningless without decontamination, and even then prefer time-gated / contamination-resistant benchmarks for the number you trust. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#11-evaluating-a-base-pre-instruct-model]

Anti-patterns (the costly mistakes)

Applying Chinchilla 20× literally when you will serve the model. You will ship a too-big, too-expensive model. Over-train a smaller one (§6). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
**Applying Chinchilla to repeated data. Chinchilla assumes unique tokens; with repetition use the data-constrained law and train a smaller model for more epochs** (§6). And don't blow past ~4–16 epochs expecting fresh-token value. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Skipping decontamination, or trusting n-gram decontamination alone. Inflated evals; paraphrased leakage passes string matching (§4). Decontaminate and time-gate. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Counting non-embedding params / under-counting FLOPs in scaling fits. This is exactly the Kaplan artifact (§5) - count total params and all FLOPs, tune warmup per scale. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Committing a cosine schedule to a fixed length, then wanting to extend or branch. Use WSD so you can decay whenever and reuse the stable checkpoint (§9). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Decaying LR too aggressively over your best (annealing-phase) data. The model is barely learning by then - use moderate decay or decouple curriculum from LR (§9). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Continual pretraining without re-warming or without replay. Either fails to adapt (no re-warm) or catastrophically forgets (no replay) (§10). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Judging a base model by chat / zero-shot instruction behavior. Base models predict tokens; evaluate few-shot with log-likelihood (§11). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Uniform data mixture / hand-set weights at scale. Use a proxy-model-derived mixture (DoReMi) and upsample high-value domains in the anneal (§3, §9). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]
Treating 6ND as exact at long context or for MoE. Add attention FLOPs at long seq; use active params for MoE (§7). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#anti-patterns-the-costly-mistakes]

Troubleshooting

Loss plateaus / model under-performs at a given compute. Check the N:D split against the regime (§5–6): likely under-trained on tokens (raise D, the Chinchilla lesson) or wrong mixture (§3). Verify warmup and LR floor (§9). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]
Eval scores look too good. Suspect contamination (§4) - re-run decontamination (semantic, not just n-gram) and re-test on a post-cutoff benchmark. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]
A capability "suddenly appears" (or fails to). Likely a metric artifact (§8) - re-measure with a smooth metric (log-prob, edit distance) to see the true trend before concluding. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]
Adding a domain wrecked general ability. Forgetting from CPT without replay (§10) - add 1–5% old-data replay and re-warm/re-decay. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]
Multilingual / code tokenization is wasteful (high fertility). Tokenizer trained on too-English a corpus or too-small a vocab (§4) - retrain on a representative mix, consider 128K+ vocab. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]
Intermediate checkpoint is much worse than final under cosine. Expected - the LR has not decayed yet; this is the WSD motivation (§9). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#troubleshooting]

References (primary; 2024–2026 unless seminal)

Kaplan et al. (2020), Scaling Laws for Neural Language Models - arXiv:2001.08361. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla, ≈20 tok/param) - arXiv:2203.15556. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Muennighoff et al. (2023), Scaling Data-Constrained Language Models (≤4 epochs ≈ free, decay to ~40) - arXiv:2305.16264; JMLR 26 (2025) 24-1000. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Sardana & Frankle (2023/2024), Beyond Chinchilla-Optimal: Accounting for Inference in LM Scaling Laws - arXiv:2401.00448. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Porian et al. (2024), Resolving Discrepancies in Compute-Optimal Scaling of Language Models - arXiv:2406.12907 (NeurIPS 2024). Besiroglu et al. (2024), Chinchilla Scaling: A replication attempt - arXiv:2404.10102. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Casson (2023), Transformer FLOPs (the 6ND / 2N-per-token derivation) - adamcasson.com. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Bavarian et al. (2022, OpenAI), Efficient Training of Language Models to Fill in the Middle (FIM, PSM/SPM, FIM-for-free) - arXiv:2207.14255. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Tay et al. (2022, Google), UL2: Unifying Language Learning Paradigms (Mixture-of-Denoisers, R/S/X, mode-switch) - arXiv:2205.05131. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Data pipeline & mixtures [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Penedo et al. (2024, HuggingFace), The FineWeb Datasets: Decanting the Web… (15T tokens, ablated pipeline, FineWeb-Edu 1.3T) - arXiv:2406.17557. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Xie et al. (2023), DoReMi: Optimizing Data Mixtures Speeds Up LM Pretraining (Group DRO proxy reweighting) - arXiv:2305.10429 (NeurIPS 2023). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Yang et al. (2023), Rethinking Benchmark and Contamination for LMs with Rephrased Samples (n-gram decontamination is fragile) - arXiv:2311.04850. Survey: Benchmark Data Contamination of LLMs - arXiv:2406.04244. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Tokenizer training: Tokenizer Choice for LLM Training: Negligible or Crucial? - arXiv:2310.08754; Diminishing Returns of Tokenization Training Data - arXiv:2502.20273. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

LR schedules, curriculum, continual pretraining [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Hu et al. (2024), MiniCPM (Warmup-Stable-Decay; ~192× data/model ratio) - arXiv:2404.06395. River-valley landscape view of WSD - arXiv:2410.05192. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Gupta et al. (2023), Continual Pre-Training of LLMs: How to (re)warm your model? - arXiv:2308.04014. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Ibrahim et al. (2024), Simple and Scalable Strategies to Continually Pre-train LLMs (re-warm + re-decay + replay = match full retrain) - arXiv:2403.08763. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining - arXiv:2511.18903. Mid-Training of LLMs: A Survey - arXiv:2510.06826. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Emergence & base-model evaluation [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Wei et al. (2022), Emergent Abilities of Large Language Models - arXiv:2206.07682. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
Schaeffer et al. (2023), Are Emergent Abilities of Large Language Models a Mirage? (metric artifact; NeurIPS 2023 Outstanding Paper) - arXiv:2304.15004. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]
EleutherAI, Language Model Evaluation Harness (lm-eval) - github.com/EleutherAI/lm-evaluation-harness. Base-vs-instruct few-shot: arXiv:2601.13244, arXiv:2501.08716. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-pretraining-scaling-laws/#references-primary-20242026-unless-seminal]

Where this helps

Deciding how to split a fixed training compute budget between model size and token count before committing to a pretraining run, since Chinchilla, inference-aware, and data-constrained regimes each answer a different version of that question. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Diagnosing why a model's benchmark scores look implausibly good, by checking eval-set decontamination and n-gram overlap before trusting a headline number. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Planning a continual or domain-adaptive pretraining run on top of an existing checkpoint, where naive resumption at the wrong learning rate causes catastrophic forgetting. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Choosing a learning-rate schedule when the total training-token budget isn't fixed in advance, or when you want to branch or extend a run later without committing to a cosine curve's endpoint. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Project ideas

Build a small-scale (100M–1B parameter) pretraining run that fits a Chinchilla-style loss curve across a few (N, D) combinations to build intuition for the C ≈ 6ND identity before scaling up. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Implement a data pipeline stage that applies MinHash+LSH near-deduplication to a raw web corpus and measure how much of the corpus that single stage removes. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build an eval-set decontamination checker that scans a training corpus for n-gram overlap against a benchmark like MMLU or GSM8K before a training run starts. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Prototype a Warmup-Stable-Decay (WSD) learning-rate scheduler and compare a mid-run checkpoint's usability against an equivalent point in a cosine-decay run. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Applying the classic ~20-tokens-per-parameter Chinchilla ratio literally when the model will be served to many users, instead of over-training a smaller model once inference cost dominates total cost. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Reusing the Chinchilla compute-optimal ratio on a corpus with heavy repetition, instead of switching to the data-constrained scaling law once unique tokens run out. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Trusting a benchmark score without first checking eval-set decontamination — n-gram overlap alone misses paraphrased leakage. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Committing to a fixed-length cosine LR schedule and then wanting to extend the run, when the schedule is coupled to the planned token budget in a way WSD is not. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

The C ≈ 6ND identity ignores attention FLOPs, a fine approximation until sequence length grows large relative to model width, at which point the O(seq²·d) term starts to matter. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
For MoE models, N in the scaling-law arithmetic should be active parameters, not total parameters, or the compute estimate is wrong. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Whether "emergent abilities" are real discontinuities or a metric artifact is still actively contested — per-token loss scales smoothly even when a downstream metric appears to jump. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Domain-adaptive continual pretraining without replaying a small fraction of the original data risks catastrophic forgetting of general capability, even when the new-domain data is high quality. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

Pretraining objectives (causal/autoregressive LM, masked LM, prefix-LM, FIM, UL2 mixture-of-denoisers) (frontier)
Pretraining data pipeline (web curation, MinHash/LSH dedup, quality filtering, FineWeb/FineWeb-Edu) (frontier)
Data mixtures & domain weighting (DoReMi Group-DRO proxy reweighting) (frontier)
Tokenizer training (vocab size ~128K, fertility/parity) & eval-set decontamination (frontier)
Kaplan vs Chinchilla compute-optimal scaling (N∝C^0.5, ~20 tokens/param) & reconciliation (frontier)
Data-constrained scaling (Muennighoff, ≤4 epochs) & inference-aware / over-training (Sardana & Frankle) (frontier)
The C≈6ND compute budget (2N forward + 4N backward, MoE active-params, MFU) (frontier)
Emergent abilities & the mirage debate (Wei vs Schaeffer) (frontier)
LR schedules at scale (cosine vs Warmup-Stable-Decay/WSD, MiniCPM) & data annealing (frontier)
Continual & domain-adaptive pretraining (LR re-warm + re-decay + replay) (frontier)
Base-model (pre-instruct) evaluation (perplexity, few-shot log-likelihood, lm-evaluation-harness) (frontier)

Frontier under this node: Base-model (pre-instruct) evaluation (perplexity, few-shot log-likelihood, lm-evaluation-harness), Continual & domain-adaptive pretraining (LR re-warm + re-decay + replay), Data mixtures & domain weighting (DoReMi Group-DRO proxy reweighting), Data-constrained scaling (Muennighoff, ≤4 epochs) & inference-aware / over-training (Sardana & Frankle), Emergent abilities & the mirage debate (Wei vs Schaeffer), Kaplan vs Chinchilla compute-optimal scaling (N∝C^0.5, ~20 tokens/param) & reconciliation, LR schedules at scale (cosine vs Warmup-Stable-Decay/WSD, MiniCPM) & data annealing, Pretraining data pipeline (web curation, MinHash/LSH dedup, quality filtering, FineWeb/FineWeb-Edu), Pretraining objectives (causal/autoregressive LM, masked LM, prefix-LM, FIM, UL2 mixture-of-denoisers), The C≈6ND compute budget (2N forward + 4N backward, MoE active-params, MFU), Tokenizer training (vocab size ~128K, fertility/parity) & eval-set decontamination

← the whole tree · 3D view· how to read this page