Agentic RL — Reinforcement Learning for LLM Agents

Agentic RL — Reinforcement Learning for LLM Agents

The 2024–2026 frontier discipline of training an LLM to act over many turns — calling tools, searching, browsing, editing code, driving a computer — by optimizing the whole multi-step trajectory against a reward grounded in environment outcomes. This is a distinct discipline from single-turn reasoning RL (the GRPO/RLVR/DeepSeek-R1 recipe that produces a long chain of thought and checks one final answer). The defining survey (arXiv:2509.02547) calls the shift one “from passive sequence generators into autonomous, decision-making agents embedded in complex, dynamic worlds.”

The one-sentence framing: reasoning RL optimizes a single response; agentic RL optimizes a trajectory of interleaved (think → act → observe) steps where the environment talks back between actions. Everything hard about agentic RL follows from that — credit assignment across many steps, masking the tokens the environment produced, instability over long horizons, and the rollout infrastructure to run environments in the training loop.

Scope boundary (read first)

• This reference = RL for LLM agents: the multi-turn POMDP framing and long-horizon credit assignment; agentic RLVR (reward from environment outcomes — tests pass, task success, search success); RL environments/gyms and the step/reset interface; reward design and long-horizon reward hacking; GRPO/PPO adapted to multi-turn (observation masking, trajectory- vs step-level advantage); agentic-RL rollout infrastructure (async generation-in-the-loop); tool-use RL / agent-as-policy; the 2025–26 agentic-RL model wave; and agent RL benchmarks.
• Single-turn reasoning RL — GRPO (the critic-free group-baseline algorithm), RLVR (verifiable rewards), the DeepSeek-R1 / R1-Zero recipe, PRM vs ORM — is the algorithmic base this reference assumes. It lives in reasoning-models.md. The line: reasoning RL = one prompt → one (long-CoT) response checked against one answer; agentic RL = a trajectory of tool/environment steps with feedback interleaved. GRPO is borrowed from reasoning RL and adapted here (multi-turn masking, nested credit assignment). When the question is “what is GRPO / how is R1 trained,” go there; when it is “how do I RL an agent that calls tools over many turns,” stay here.
• Classic preference alignment — RLHF with PPO, the DPO-variant family (DPO/IPO/KTO/ORPO/SimPO/CPO), RLAIF, Constitutional AI — optimizes human preference via a learned reward model. It lives in llm-alignment-post-training.md. Agentic RL optimizes verifiable task outcomes, not preference. (DPO is used agentically — on successful-vs-failed trajectories, ETO-style — but the preference-alignment machinery itself is there.)
• Inference-time agent orchestration — building the harness, action space, tool definitions, and the prompted agent loop that runs without any training — is agent-harness-construction.md, agent-ecosystem.md, and autonomous-loops.md. The line: those references make a fixed model act as an agent at inference; this reference changes the model’s weights so it acts better. A ReAct loop you prompt is harness/ecosystem; a ReAct loop whose policy you train with RL is here.
• General distributed-training infrastructure — FSDP/ZeRO/tensor-pipeline-expert parallelism, the optimizer/gradient-sync mechanics of training across GPUs — is distributed-training.md. The agentic-RL-specific rollout infrastructure (async environment rollouts, generator-in-the-loop vLLM/SGLang, actor–learner separation for RL) is here.
• Serving an agent for throughput (vLLM batching, paged KV) → llm-inference-serving.md. Agentic RL uses vLLM/SGLang as the in-the-loop generator, but serving-engine tuning is there. Offline benchmark-harness mechanics (HELM/MMLU scaffolding) → da-analytical-methods (references/da-7-machine-learning.md); agent-specific benchmarks (SWE-bench, τ-bench, WebArena) are summarized here (§9).

1. The multi-turn POMDP — why agentic RL is a different problem

The formal heart of the field (survey arXiv:2509.02547, Table 1). Single-turn LLM RL — the survey calls it PBRFT (preference-based RL fine-tuning, i.e. RLHF/DPO and reasoning-RL) — is a degenerate single-step MDP: one static prompt state {s₀}, a pure-text action, horizon T=1, γ=1, objective E_{a∼π}[r(a)]. Agentic RL is a genuine POMDP ⟨𝒮, 𝒜, 𝒫, ℛ, γ, 𝒪⟩:

Two consequences define the discipline:

1. The action space is hybrid. The agent interleaves thinking tokens (reasoning) with action tokens (a tool call, a search query, a shell command, a click). The policy must learn both what to think and when/what to act.
2. Reward is sparse and delayed. Success is usually known only at the end (did the tests pass? did the DB reach the goal state?), so the gradient must be propagated back across a long trajectory of mostly-unrewarded steps. The survey names temporal credit assignment as the central bottleneck for long-horizon tool use. This is the problem §5 (algorithms) exists to solve.

The survey organizes agent capabilities the RL is meant to improve into six dimensions: planning, tool use, memory, reasoning, self-improvement, perception.

2. Agentic RLVR — verifiable rewards from environment outcomes

Agentic RL inherits RLVR from reasoning models (reward = an automatic verifier, not a learned reward model — see reasoning-models.md §3) but changes where the verifier’s signal comes from: a real environment outcome rather than a math-answer string match. This is the single most important practical idea — it is what makes agentic RL trainable without human labels.

Canonical instances (and their exact reward functions):

• Code / software engineering — SWE-RL (Meta, arXiv:2502.18449). The reward is a piecewise rule-based score: −1 on format failure; otherwise a continuous similarity score in [0,1] between the predicted patch and the oracle patch, computed by Python’s difflib.SequenceMatcher. Optimized with GRPO over a seed dataset distilled from 11M GitHub pull requests (code snapshots + issues + PRs — “open software evolution”). Result: Llama3-SWE-RL-70B → 41.0% on SWE-bench Verified, best among <100B models and near GPT-4o’s 38.8%. Notably, RL on only issue-solving produced generalized out-of-domain gains (code reasoning, math, language). (SWE-RL’s core generation is single-turn — issue+files → patch — wrapped in the “Agentless Mini” scaffold at inference; the trajectory-level SWE agents are the OpenHands/SWE-Gym line.)
• Search / QA — Search-R1 (arXiv:2503.09516). A simple outcome-based reward — exact-match / F1 on the final answer only, no process reward — drives multi-turn search. +26% (Qwen2.5-7B) over RAG/SOTA. R1-Searcher incentivizes search with a two-stage outcome-based RL.
• The general taxonomy (survey): verifier types are rule-based (code execution, unit tests), symbolic (proof/format checkers), and neural reward models; reward can be outcome (final completion) or process (intermediate step feedback).

Why this differs from reasoning-model RLVR: in reasoning-models, the verifier checks a closed-form answer the model produced in one shot. Here the verifier runs the artifact in an environment — executes the patched repo’s test suite, checks the final database state against a goal, or scores whether the retrieved answer matches — and the agent took many actions to get there. Same RLVR principle (grounded truth, hard to hack vs a learned RM), different reward source and a long trajectory to assign it across.

3. RL environments & gyms — the step/reset interface

In agentic RL, an “environment” is a sandboxed, resettable process the RL loop drives via a Gymnasium-style interface — concretely, the thing that takes the agent’s action, mutates world state, and returns an observation plus (eventually) a reward. Getting this interface standard, sandboxed, parallelizable, and reproducible is half the engineering.

• OpenEnv (Meta-PyTorch + HuggingFace, meta-pytorch/OpenEnv) is the emerging standard: a Gymnasium-style interface with three APIs — step(), reset(), state(). Environments are isolated (each agent instance in its own sandbox) and scalable (deployed as FastAPI servers in Docker containers, driven over type-safe HTTP, enabling distributed rollouts across a cluster). Reference envs: echo (testing), coding, Atari, OpenSpiel. The client–server/HTTP design decouples the env from the trainer.
• SkyRL-Gym (NovaSky, NovaSky-AI/SkyRL) is “a gymnasium of tool-use tasks — math, coding, search, SQL — implemented in the Gymnasium API.” SkyRL splits the stack into skyrl-train (the RL trainer), skyrl-agent (long-horizon agents), and skyrl-gym (the environments).
• RAGEN (arXiv:2504.20073) ships controlled symbolic envs — Bandit, Sokoban, FrozenLake, WebShop — chosen to isolate single-turn vs multi-turn, deterministic vs stochastic, and open-domain grounding.
• verl’s AgentLoop (see §6) is the env/agent-interaction abstraction inside the verl trainer; BrowserGym unifies observation/action spaces for web agents (over WebArena/WorkArena tasks).

Reproducibility & sandboxing are first-class because code-execution and computer-use environments run untrusted, side-effecting actions — Docker isolation (OpenEnv) and deterministic seeds/state are what make a rollout repeatable and safe to parallelize.

4. Reward design & long-horizon failure modes (the Echo Trap)

Reward design is harder over long horizons because failure modes appear that do not exist in single-turn RL. The key primary source is RAGEN (arXiv:2504.20073).

• Outcome vs process, sparse vs dense. Outcome reward (signal only at task completion) is clean and hard to game but gives almost no gradient on long trajectories; process/dense sub-rewards (step-level progress) improve sample efficiency but invite gaming. The survey frames this explicitly as a tension and most systems combine a sparse task reward with dense shaping. Format rewards (did the agent emit a well-formed tool call / <answer> tag?) are a common cheap dense component (SWE-RL’s −1-on-bad-format is the minimal case).
• The “Echo Trap” (RAGEN’s signature finding) — the characteristic collapse of naive multi-turn RL: agents initially improve, then overfit to locally rewarded reasoning patterns and collapse to deterministic, repetitive templates. It has three measurable symptoms: (1) entropy collapse (policy converges to fixed phrasing), (2) a reward-variance cliff — reward standard deviation drops before task performance degrades, making it an early-warning signal, and (3) gradient-norm spikes marking the irreversible collapse point (e.g. ~step 170 in their Bandit-PPO run).
• Reasoning does not emerge for free. RAGEN Finding 2: “reasoning hardly emerges through multi-turn RL” without explicit fine-grained, reasoning-aware reward — even when prompts force <think> tokens, the model suppresses reasoning if it confers no reward advantage (response length collapsed 307→89.5 tokens on Sokoban). If you want the agent to actually plan, the reward has to pay for planning.
• Reward hacking over long horizons = the agent finds a high-reward degenerate policy: looping, padding, exploiting a dense-reward proxy, or gaming a weak verifier. The longer the horizon, the more room to hack. Mitigations: keep the outcome verifier authoritative, shape sparingly, and watch entropy/variance as collapse predictors. Gated/partial-credit reward schemes (arXiv:2508.10548) are an active fix for long-horizon stability.

5. GRPO/PPO adapted to multi-turn — observation masking & credit assignment

Naive single-turn GRPO/PPO fails in the multi-turn setting (RAGEN Finding 1: vanilla adaptations get early gains then collapse). Two adaptations are essentially mandatory.

5a. Observation / tool-token masking (the non-negotiable fix)

Compute the policy-gradient loss only over the tokens the agent generated, and mask out every token the environment returned (tool outputs, retrieved documents, observations). Reasons: those tokens were not produced by the policy (training on them is “fundamentally incorrect”), and long observations would otherwise dominate the loss weight. Search-R1 calls this “retrieved token masking” and shows it is required for stable RL; WebAgent-R1/VAGEN (arXiv:2505.16421) document the same for web/VLM agents. This is the multi-turn analog of SFT’s prompt-token masking, and the most common silent bug when people first extend GRPO to agents.

5b. Credit assignment — trajectory-level vs step-level advantage

The core problem: with reward only at the end, how do you decide which of 20 actions deserved credit? Two patterns:

• Trajectory-level — StarPO (RAGEN’s “State-Thinking-Actions-Reward Policy Optimization”). Objective J(θ) = E_τ[R(τ)] over the whole trajectory τ = {s₀,a₀,r₀,…,s_K}, decomposed to token-level likelihoods for autoregressive LLMs. The whole trajectory is the unit of optimization. StarPO-S is the stabilized variant with three knobs: (1) variance-based trajectory filtering — keep only the ~top-25% highest-reward-variance prompts (drop low-information rollouts), (2) a token-level critic with GAE (γ=λ=1.0) for smoother advantage, (3) gradient shaping — KL-term removal + asymmetric “Clip-Higher” clipping. Collapse is “largely mitigated when more than half of the trajectories are filtered.”
• Nested / step-level — GiGPO (Group-in-Group Policy Optimization, NeurIPS’25, arXiv:2505.10978). Keeps GRPO’s critic-free, low-memory, group-baseline property but nests two levels of group-relative advantage: an episode level (sample a group of full trajectories under identical task/initial state, compute macro advantage from total returns) and a step level via an “anchor-state grouping” mechanism — retroactively group all actions taken from the same recurring environment state across trajectories, and compute a localized advantage among them. This gives fine-grained per-step credit for free (no extra rollouts, same GPU memory as GRPO): +12% on ALFWorld, +9% on WebShop over the GRPO baseline.
• The variant family (survey) used agentically: DAPO, GSPO, Dr.GRPO, Step-GRPO, ProRL, StarPO, GiGPO, TreePO, Pass@k Training, plus tool-integrated optimizers (ToolRL, OTC-PO, ASPO). Most are GRPO descendants tuned for sparse/long-horizon/multi-turn settings.

Practical mental model: start from GRPO (the reasoning-models base), then (1) mask observation tokens, (2) pick a credit-assignment scheme (trajectory-level StarPO if you mainly have an end reward; nested GiGPO if states recur and you want step-level signal), and (3) stabilize (trajectory filtering, clip-higher, watch entropy/variance for the Echo Trap).

6. Rollout infrastructure for agentic RL

The rollout — running the agent through the environment to collect trajectories — is the agentic-RL bottleneck: every training step needs fresh on-policy trajectories, and each trajectory is many slow generate→act→observe round-trips. The architectural fix is asynchronous, server-based generation with actor–learner separation.

• verl / HybridFlow (EuroSys’25, verl-project/verl) — a Hybrid-Controller that models the RL algorithm (PPO/GRPO/DAPO) as a multi-stage, multi-model, parallelizable dataflow graph. v0.5 added the AgentLoop abstraction (define a custom multi-turn agent/tool loop; ReactAgentLoop adapts LangGraph agents) plus server-based async rollout: generation is pulled out into per-conversation async vLLM/SGLang servers so each dialogue advances at its own pace, returns out of order, and is reassembled for training — without intrusive edits to the inference engine. SGLang calls async_generate via a Ray actor; vLLM calls generate over ZMQ. v0.5 also prototypes disaggregated async training and a GenerativeRM.
• SkyRL-Agent (arXiv:2511.16108) — a fine-grained asynchronous dispatcher for scheduling rollouts (1.55× faster async dispatch), a tool-centric task interface with dynamic tool registration + verifiers, and a backend bridge that is backend-agnostic (SkyRL-train / verl / Tinker). Trained SA-SWE-32B, lifting Qwen3-32B 24.4% → 39.4% Pass@1 on SWE-bench.
• Generator-in-the-loop. The same engine (vLLM/SGLang) that serves models in production is run inside the RL loop as the trajectory generator; the actor (generation) and learner (gradient update) are separated so generation can run async/ahead and GPUs stay busy. Other frameworks in this space (named in the ecosystem; not individually deep-verified here): NeMo-RL, OpenRLHF, AReaL, ROLL, AgentGym-RL.

The distinction from distributed-training: that reference owns how gradients are sharded/synced across GPUs (FSDP/ZeRO/parallelism). This owns how trajectories are generated and fed to the trainer — the rollout half that only exists in RL.

7. Tool-use RL & the agent-as-policy view

The agent is the policy: instead of prompting a fixed model to call tools (harness work → agent-harness-construction.md), you train the model with RL to decide when and how to call tools, interleaving reasoning with tool calls and learning the strategy from outcome feedback. The survey groups this as tool-integrated reasoning (TIR).

• ReTool (arXiv:2504.11536): cold-start SFT on code-augmented reasoning traces → outcome-reward RL with multi-turn real-time code execution interleaved into the reasoning. The model learns strategic tool use (when to drop into code) from task outcomes alone. 32B reaches 67% on AIME with 400 RL steps vs a text-only-RL baseline’s 40% at 1080 steps, and exhibits emergent code self-correction (a tool-use “aha moment”).
• ToRL (Tool-Integrated RL, arXiv:2503.23383): RL (not SFT) lets the model explore and discover optimal tool-use strategies — learned tool use generalizes past the demonstrations that SFT would cap it at.
• Learned vs prompted tool use is the key contrast: a prompted ReAct agent uses whatever tool-use policy the base model already has; tool-use RL moves the weights so the model gets better at the decision of invoking tools. The survey lists ToolRL, OTC-PO, AutoTIR, ReTool, ToRL, ASPO in this family, and an “RL as internal driver” pattern that applies DPO to successful-vs-failed trajectories (ETO).

8. The 2025–2026 agentic-RL model wave

The field moved sharply from single-turn RLVR to multi-turn, environment-grounded agentic RL as the dominant post-training frontier (survey arXiv:2509.02547).

• Deep-research / search agents. Kimi-Researcher (Moonshot) — a deep-research agent trained end-to-end with agentic RL on the Kimi k-series, deliberately “zero-structure” (no preset workflow/prompt scaffold; the agent learns the whole search-and-reason loop via trial-and-error). Reported 26.9% pass@1 on Humanity’s Last Exam (SOTA at release) and 69% pass@1 on xbench-DeepSearch (beating o3-with-tools); typical task ~23 reasoning steps, ~200 URLs. Plus Search-R1, R1-Searcher, WebAgent-R1, ParallelSearch (arXiv:2508.09303).
• SWE / coding agents. SWE-RL (Meta, 41% SWE-bench Verified), SWE-Gym training environments, SkyRL SA-SWE-32B (OpenHands-style scaffold).
• Computer-use / GUI agents. Trained against OSWorld / AppWorld / Android-in-the-Wild with task-completion rewards (survey).
• General trend: the recipe has consolidated to base reasoning model → agentic RL in an environment with a verifiable outcome reward, masking observations, with async rollout infra. Contested / unresolved (mid-2026): how far agentic RL expands capability vs sharpens existing behavior (the same sharpening-vs-new-capability debate as reasoning RLVR — see reasoning-models.md §3); how to keep long-horizon training stable at scale; and how much dense reward shaping helps before it invites hacking.

9. Agent RL evaluation / benchmarks

What you optimize against (and report). Most are task-success benchmarks — the reward signal and the eval are often the same environment.

Eval hygiene specific to agents: report pass^k / reliability (τ-bench) not just pass@1 — an agent that succeeds once in eight tries is not production-ready; and beware that the training environment and eval benchmark overlapping invites the same contamination/over-fit concerns as any RL setup.

Practical patterns

• Start from a reasoning-capable base, then RL in the environment. The reasoning-models base (long-CoT + GRPO) gives the model the thinking substrate; agentic RL teaches it to act. Don’t RL a base model with no reasoning ability and expect agentic behavior to emerge for free (RAGEN Finding 2).
• Mask observation tokens — always. The single most common bug. Loss over agent-generated tokens only; mask tool outputs / retrieved docs / observations (Search-R1, verl).
• Pick credit assignment by reward shape. End-only reward → trajectory-level (StarPO). Recurring states + want per-step signal → nested GiGPO. Both keep GRPO’s critic-free economy.
• Watch entropy and reward-variance as collapse predictors. They drop before task reward degrades (RAGEN Echo Trap). Filter low-variance trajectories; use Clip-Higher; consider re-introducing a light critic.
• Use async server-based rollouts. verl AgentLoop or SkyRL-Agent with vLLM/SGLang generators; separate actor (generation) from learner (update) so GPUs aren’t idle waiting on slow environment round-trips.
• Keep the outcome verifier authoritative; shape sparingly. Dense/process rewards help sample efficiency but are the attack surface for long-horizon reward hacking.
• Sandbox and seed environments. Docker isolation (OpenEnv) for untrusted, side-effecting actions; deterministic resets for reproducible rollouts.

Anti-patterns

• Training on tool/observation tokens (no masking) — destabilizes RL because the policy is graded on text it never generated.
• Applying vanilla single-turn GRPO/PPO to a multi-turn task and expecting it to hold — it gets early gains then hits the Echo Trap (RAGEN Finding 1).
• End-only reward with no credit-assignment scheme on long horizons — the gradient is too sparse; the agent learns nothing or collapses.
• Over-dense reward shaping — invites looping/padding/proxy-gaming; the longer the horizon, the worse.
• Confusing this with prompting an agent loop — if you are not changing weights, that is harness/orchestration (agent-harness-construction, autonomous-loops), not agentic RL.
• Reporting only pass@1 — agents need pass^k / reliability (τ-bench).
• Treating GRPO/RLVR here as the same thing as in reasoning-models — same algorithm family, but the multi-turn masking + nested credit assignment + rollout infra are what make it agentic.

Troubleshooting

• Reward climbs then collapses; outputs become repetitive → Echo Trap. Check entropy (collapsing?) and reward std-dev (cliff?). Apply StarPO-S: variance-based trajectory filtering, Clip-Higher, optional critic.
• Training unstable / loss dominated by long sequences → you are probably training on observation tokens. Verify the loss mask excludes tool/retrieved/env tokens.
• Agent “succeeds” but does something degenerate → reward hacking. Tighten/verify the outcome verifier; reduce dense-reward weight; inspect trajectories.
• GPUs idle, throughput dominated by rollouts → switch to async server-based generation (verl AgentLoop / SkyRL-Agent async dispatcher); decouple actor and learner.
• No reasoning/planning emerging → the reward doesn’t pay for it (RAGEN Finding 2). Add a fine-grained reasoning-aware or process reward, or accept the model will shortcut.
• Good pass@1, unreliable in practice → measure pass^k (τ-bench); reliability over trials is the production metric.

References

• Survey — “The Landscape of Agentic Reinforcement Learning for LLMs” (arXiv:2509.02547) — the canonical taxonomy: POMDP-vs-PBRFT formalism, six capability dimensions, full RL-algorithm and benchmark/framework catalog.
• RAGEN / StarPO (arXiv:2504.20073) — multi-turn agent RL; the Echo Trap, StarPO trajectory objective, StarPO-S stabilization, the three findings.
• GiGPO (arXiv:2505.10978, NeurIPS’25) — nested episode + anchor-state step-level group-relative credit assignment, critic-free.
• SWE-RL (Meta, arXiv:2502.18449; facebookresearch/swe-rl) — rule-based difflib reward, GRPO, 11M PRs, 41% SWE-bench Verified.
• Search-R1 (arXiv:2503.09516; PeterGriffinJin/Search-R1) — interleaved reason+search, retrieved-token masking, outcome EM reward. R1-Searcher — two-stage outcome RL for search.
• ReTool (arXiv:2504.11536) and ToRL (arXiv:2503.23383) — tool-use RL / tool-integrated reasoning; learned strategic tool invocation.
• verl / HybridFlow (EuroSys’25; verl-project/verl; agentic-RL docs + v0.5 release) — AgentLoop, async server-based vLLM/SGLang rollouts.
• SkyRL (NovaSky-AI/SkyRL) + SkyRL-Agent (arXiv:2511.16108) — full-stack RL (train/agent/gym), async dispatcher, backend-agnostic.
• OpenEnv (meta-pytorch/OpenEnv) — Gymnasium-style step/reset/state, sandboxed Dockerized environment hub.
• τ-bench (arXiv:2406.12045) / τ²-bench (arXiv:2506.07982; sierra-research/tau2-bench) — tool-agent-user benchmark, pass^k reliability, dual-control.
• Kimi-Researcher (Moonshot tech report) — end-to-end agentic RL deep-research agent; HLE 26.9%, xbench-DeepSearch 69%.
• WebAgent-R1 / VAGEN (arXiv:2505.16421) — multi-turn web/VLM agent RL; observation-masking rationale, M-GRPO.
• Base RL algorithms — see reasoning-models.md (GRPO, RLVR, DeepSeek-R1) and llm-alignment-post-training.md (PPO, DPO family).