Reasoning Models and Test-Time Compute

Reasoning Models & Test-Time Compute

The frontier (2024–2026) shift from “scale the model and prompt it well” to “train the model to reason, then spend extra compute at inference to reason harder.” Two coupled ideas drive it:

  1. Test-time / inference-time compute (TTC): a model can be made more accurate on a hard problem by spending more compute when answering — sampling many solutions, searching over reasoning steps, or generating a long internal chain of thought — instead of (or in addition to) growing the model. Snell et al. (2024) showed this can be more parameter-efficient than scaling pretraining on a fixed compute budget for many problems.
  2. Reasoning models (o1/o3, DeepSeek-R1, Gemini “thinking”, Claude extended thinking): models post-trained with reinforcement learning to produce long chains of thought before answering. The RL teaches the model to use its inference budget well, so the two ideas reinforce each other: the model both generates a long CoT and benefits from being allowed to.

Scope boundary (read first)

1. Chain-of-thought (CoT) and long-CoT

Chain-of-thought prompting (Wei et al., NeurIPS 2022, arXiv:2201.11903) is the origin: prompting a model to emit intermediate reasoning steps before its final answer (“Let’s think step by step” / few-shot exemplars with worked reasoning) sharply improves arithmetic, commonsense, and symbolic reasoning. The paper’s key empirical claim: CoT is an emergent ability of scale — it does not help (and can hurt) small models, and only yields gains at roughly ≥100B-parameter scale. CoT works because autoregressive decoding lets the model externalize computation into tokens: each generated step conditions the next, so the model can do serial work it cannot do in a single forward pass. A useful mental model is that CoT trades extra output tokens for extra effective “depth.”

Long-CoT is the 2024–2026 evolution. Where prompted CoT produces a few tidy steps, reasoning models produce thousands of tokens of unpolished internal reasoning — exploring, backtracking, self-correcting, re-checking (“Wait, let me reconsider…”), trying multiple approaches. This is not just “more steps”; it qualitatively includes reflection, verification, and search-like behavior in a single linear stream. DeepSeek-R1-Zero exhibited a now-famous “aha moment”: mid-training, the model spontaneously learned to stop and re-evaluate its approach, allocating more thinking to harder sub-problems — an emergent behavior from RL, not something explicitly taught. Long-CoT is what reasoning-model RL produces, and what makes test-time-compute scaling work: a longer, self-correcting chain is more likely to reach the right answer.

CoT faithfulness caveat: the visible chain is not guaranteed to be the model’s true computation. Models can reach a correct answer via a reasoning trace that is post-hoc or partially confabulated, so a CoT should be treated as an aid to accuracy, not a transparent log of mechanism — relevant for both interpretability and safety.

2. Reinforcement learning for reasoning — GRPO and the DeepSeek-R1 recipe

The central insight of reasoning models: you can RL a base model toward better reasoning using only an automatic correctness signal (does the final answer match? does the code pass tests?), with no human preference labels and no learned reward model. This is what separates reasoning-RL from RLHF.

GRPO (Group Relative Policy Optimization)

GRPO (introduced in DeepSeekMath, arXiv:2402.03300; used at scale in R1) is the workhorse algorithm. It is a PPO-style policy-gradient method that removes the value/critic network. Instead of a learned baseline, GRPO samples a group of G outputs for each prompt, scores them all with the reward function, and uses the group’s mean reward as the baseline — each sample’s advantage is its reward standardized against its group ((r_i − mean) / std). Benefits: no critic model (roughly halves memory/compute vs PPO’s four-model loop), and a baseline that is naturally calibrated per-prompt. It keeps PPO’s clipped surrogate objective and a KL penalty to a reference policy to prevent drift. Known issue: the original formulation can induce a response-length bias (especially inflating length on incorrect answers); several 2025 variants (e.g., length-normalized / token-level objectives, Dr. GRPO) correct this.

The R1-Zero and R1 recipe (arXiv:2501.12948)

3. RLVR — Reinforcement Learning with Verifiable Rewards

RLVR (named and formalized in Tülu 3, Lambert et al., arXiv:2411.15124) is the general principle behind reasoning-RL: replace RLHF’s learned reward model with a deterministic verification function, and give reward only when the output is verifiably correct. The verifier can be an exact-match check on a math answer, a unit-test suite for code, a format/constraint checker for instruction-following, or a symbolic checker. Because the signal is grounded truth rather than a learned proxy, RLVR is far less prone to reward hacking than RLHF — there is no reward model to exploit (though verifier gaming and reward-spec gaps still exist).

The RLVR effectiveness debate (important, unresolved as of mid-2026)

A prominent 2025 result (Yue et al., Tsinghua, arXiv:2504.13837) argued that RLVR may not teach genuinely new reasoning — under pass@k with large k, RLVR-trained models do not exceed the base model’s coverage of solvable problems; RL appears to sharpen / up-weight reasoning paths already in the base distribution (raising pass@1) rather than discovering paths the base could never find. Follow-ups (e.g., arXiv:2506.14245) counter that RLVR implicitly improves the correctness of sampled reasoning. The practical takeaway: RLVR reliably makes a model better at finding its good reasoning faster (sample-efficiency / pass@1), but whether it expands the reasoning ceiling beyond the base model is contested — relevant when deciding between RL and just distilling from a stronger teacher (§7).

4. Process reward models (PRM) vs outcome reward models (ORM)

When you score reasoning, you can reward the outcome (final answer only) or the process (each intermediate step). This choice shapes both RL training and test-time verification.

Two uses of reward models — keep them distinct:

  1. As a training signal (in RL): PRMs can densify the RL reward. However, the R1 team found PRMs hard to use as the RL reward at scale — defining “a good step” is fuzzy, step-level labels are expensive, and PRMs are themselves reward-hackable. R1 therefore used simple rule-based outcome+format rewards for RL, not a PRM. This is a key practical lesson: PRMs shine at test-time verification more than as the RL objective.
  2. As a test-time verifier (§5–6): a PRM scores candidate solutions/steps so search or best-of-N can pick the best — this is where PRMs deliver the most value.

The PRM survey (arXiv:2510.08049) traces the field from outcome signals to process supervision and covers automatic PRM-label generation (e.g., Monte-Carlo rollouts that label a step by how often continuing from it reaches a correct answer — “Math-Shepherd” style), which removes the human-labeling bottleneck.

5. Parallel test-time compute — best-of-N, self-consistency, verifiers

The simplest way to spend more inference compute: sample multiple independent solutions and aggregate. This is “parallel” scaling (independent samples) as opposed to “sequential” scaling (one long, self-correcting chain — §9).

Verifier quality is the ceiling on BoN. A weak verifier makes BoN plateau or even degrade as N grows (you increasingly select confidently-wrong answers — “reward over-optimization” at inference). With a perfect verifier (e.g., unit tests for code), BoN is extremely strong: pass@N rises steeply.

6. Search-based test-time compute — beam, lookahead, MCTS, reward-guided decoding

Beyond independent samples, you can search over the space of reasoning steps, using a process verifier to guide which partial paths to expand. The tree-search-for-reasoning survey (arXiv:2510.09988) unifies these.

When search beats best-of-N (Snell et al., 2024): on easier problems, simple best-of-N / sequential revision is compute-optimal; on harder problems, search (beam/lookahead) over a PRM uses the budget better. The optimal strategy is difficulty-dependent — there is no single best TTC method (§7). Caveat: much of the strongest search work uses small models + strong PRMs on math; with a top-tier reasoning model that already does long-CoT, a long single chain plus self-consistency often matches elaborate external search at lower engineering cost.

7. Inference-time scaling laws and compute-optimal test-time scaling

Just as pretraining has scaling laws, inference has scaling laws: accuracy improves predictably as you spend more test-time compute, up to a point.

8. The reasoning-model landscape — the o-series, R1-class, and mid-2026 SOTA

9. Budget forcing and thinking-token control

Once a model produces long-CoT, you need a dial on how much it thinks. Two layers:

10. Distilling reasoning into smaller models

You don’t have to RL a small model to make it reason — you can distill a big reasoning model’s long-CoT into a smaller one via plain SFT on its traces.

11. Reasoning benchmarks — and why they keep breaking

Reasoning models are evaluated on verifiable, hard tasks (so RLVR/verifier signals apply):

Evaluation hygiene (treat as the real lesson):

12. Cost, latency, and accuracy trade-offs (the operating decision)

Reasoning is expensive: thinking tokens are billed (usually at output rates) and dominate latency — a hard query can burn tens of thousands of hidden tokens before the first visible output. The engineering job is allocating that budget.

Practical patterns

Anti-patterns

Troubleshooting

References (primary sources)