---
title: "RLHF & RL Training Infrastructure for LLMs (2024–2026)"
description: "Post-training reinforcement learning (RLHF, RLVR, reasoning-RL, agentic-RL) for LLMs runs on a distinct systems stack that is neither the RL algorithm (PPO/GRPO/DPO) nor generic supervised distributed"
---

# RLHF & RL Training Infrastructure for LLMs (2024–2026): Research Report
*Generated: 2026-05-31 | Sources: 22 | Confidence: High*

## Executive Summary

Post-training reinforcement learning (RLHF, RLVR, reasoning-RL, agentic-RL) for LLMs runs on a distinct **systems stack** that is neither the RL *algorithm* (PPO/GRPO/DPO) nor generic supervised distributed training (FSDP/ZeRO for pretraining). The defining structure is the **actor–rollout–learner** loop: a training engine (FSDP/Megatron) updates the policy; a generation/rollout engine (vLLM/SGLang) samples responses; a reward/verifier module scores them. The single dominant systems fact is that **rollout/generation accounts for 60–90%+ of step time** ([APRIL, arXiv:2509.18521](https://arxiv.org/abs/2509.18521); [vLLM blog](https://blog.vllm.ai/2025/04/23/openrlhf-vllm.html)), and that **synchronous loops leave training GPUs idle while inference GPUs work** — a convergence finding across all 16 surveyed libraries ([HuggingFace, "Keep the Tokens Flowing"](https://huggingface.co/blog/async-rl-training-landscape)). Everything else — colocate vs disaggregated placement, the train→infer weight resync, async/off-policy systems, the train/inference logprob mismatch — follows from attacking that bottleneck. Confidence is **High**: each concept is corroborated by 2+ independent primary sources (framework papers, official docs, vLLM/HF engineering blogs).

---

## 1. The Actor–Rollout–Learner Architecture (the three engines)

An RL post-training step decomposes into three logically distinct engines that exchange data through an experience buffer ([Anatomy of RL Frameworks](https://www.hanifleo.com/anatomy-of-rl-frameworks/); [HF async-RL survey](https://huggingface.co/blog/async-rl-training-landscape)):

- **Generation / rollout engine** — autoregressively samples responses (and, for agents, multi-turn trajectories) from the current policy. Implemented with an *inference* engine (vLLM/SGLang), not a training framework, because sampling is the throughput-critical step ([vLLM blog](https://blog.vllm.ai/2025/04/23/openrlhf-vllm.html)).
- **Reward / verifier module** — scores responses: a learned reward model (RM), a rule-based verifier, or a code-execution sandbox (RLVR) ([Promptfoo RLVR](https://www.promptfoo.dev/blog/rlvr-explained/)).
- **Policy training engine (learner)** — runs forward+backward+optimizer to update policy weights (FSDP or Megatron) ([HybridFlow](https://arxiv.org/abs/2409.19256)).

Classic PPO-RLHF involves **four models**: actor (policy, trained), critic/value (trained), reward model (frozen), and reference model (frozen, for the KL penalty) ([OpenRLHF, arXiv:2405.11143](https://arxiv.org/pdf/2405.11143)). Critic-free algorithms like GRPO drop the value model, simplifying the systems problem to actor + reward/verifier + reference. The dataflow per step: prompts → (rollout) responses → (reward) scores + (recomputed) logprobs → advantages → (learner) gradients → updated weights → **resync to rollout engine** (§4). The HF survey frames the whole design space as **seven orthogonal axes**, the key three being orchestration/concurrency primitive (Ray actors, asyncio, pub/sub, HTTP), **rollout buffer design** (how rollouts flow from inference to training), and **weight-synchronisation protocol** ([HF survey](https://huggingface.co/blog/async-rl-training-landscape)). **Confidence: High.**

## 2. Co-located / Hybrid vs Disaggregated GPU Placement

The central placement decision is whether the trainer and the rollout engine **share one GPU pool** (colocate/hybrid) or run on **separate pools** (disaggregated) ([TRL vLLM integration](https://huggingface.co/docs/trl/vllm_integration); [NeMo-RL docs](https://docs.nvidia.com/nemo/rl/0.4.0/index.html)):

- **Colocate / hybrid engine** — training and generation **time-share** the same GPUs. The trainer offloads/sleeps while the rollout engine generates, then weights are resharded in place and the trainer wakes. veRL's **3D-HybridEngine** reshards the actor between a training layout (e.g. FSDP DP=2/TP=8) and an inference layout (vLLM DP=16/TP=4) **on the same GPUs with zero memory redundancy** and reduced communication, by transforming the single model in place rather than keeping a second copy ([HybridFlow paper](https://arxiv.org/abs/2409.19256); [verl weight-resharding discussion](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html)). Pros: highest GPU utilization, no idle pool; cons: **memory contention** on the shared GPUs ([TRL colocate mode](https://huggingface.co/docs/trl/vllm_integration)).
- **Disaggregated / separated** — inference runs continuously on one pool, the optimizer continuously on another ([NeMo-RL docs](https://docs.nvidia.com/nemo/rl/0.4.0/index.html)). OpenRLHF pioneered this with **Ray Placement Groups**, scheduling vLLM engines, actor, critic, reference, and reward each on their own GPUs, with the Adam optimizer on CPU ([OpenRLHF, arXiv:2405.11143](https://arxiv.org/pdf/2405.11143); [Medium summary](https://medium.com/@techsachin/openrlhf-rlhf-framework-with-support-of-70b-full-tuning-b3d898f49e81)). Pros: isolation, independent scaling, easier async; cons: a pool can sit idle in a synchronous schedule, and weights must cross the network.

**Programming model:** HybridFlow's core contribution is combining a **single-controller** paradigm (one process expresses the whole dataflow — flexible) with a **multi-controller** paradigm (each device runs its own program — efficient, low dispatch overhead). Pure single-controller has high control-dispatch overhead at scale; pure multi-controller is inflexible for nested RL dataflow. HybridFlow's hybrid reports **1.53×–20.57× throughput** over baselines like DeepSpeed-Chat and NeMo-Aligner ([HybridFlow paper](https://arxiv.org/abs/2409.19256); [verl HybridFlow guide](https://verl.readthedocs.io/en/latest/hybrid_flow.html)). ROLL and slime use a single-controller + parallel-worker abstraction ([ROLL, arXiv:2506.06122](https://arxiv.org/abs/2506.06122); [slime/DeepWiki](https://deepwiki.com/THUDM/slime)). **Confidence: High.**

## 3. The Generation / Rollout Bottleneck (inference-engine-in-the-loop)

RL post-training is **rollout-dominated**: generation accounts for **over 90% of total RL training runtime** in the worst case ([APRIL, arXiv:2509.18521](https://arxiv.org/abs/2509.18521); [verl SGLang docs](https://verl.readthedocs.io/en/latest/workers/sglang_worker.html)). The reason is structural — autoregressive per-token decoding is **memory-bandwidth-bound** and runs at **<40% GPU utilization** in the actor, whereas the scoring and training stages are compute-intensive ([OPPO, arXiv:2509.25762](https://arxiv.org/html/2509.25762v1)). This is why frameworks plug a dedicated **inference engine — vLLM or SGLang — in the loop as the rollout generator** (with PagedAttention, continuous batching, and often FP8/INT8 inference for extra speed), rather than generating with the training framework ([vLLM blog](https://blog.vllm.ai/2025/04/23/openrlhf-vllm.html); [HF survey](https://huggingface.co/blog/async-rl-training-landscape)). veRL supports both vLLM and SGLang as interchangeable rollout backends ([verl GitHub](https://github.com/verl-project/verl)).

The **long-tail straggler** problem compounds it: response lengths are long-tailed, so a few very long generations stall an entire synchronous batch, leaving most GPUs idle ([OPPO](https://arxiv.org/html/2509.25762v1); [APRIL](https://arxiv.org/abs/2509.18521)). This single fact — "rollout is the bottleneck and the tail makes it worse" — motivates §5 (async), §9 (overlap), and partial-rollout schemes. **Confidence: High** (90% figure stated identically across APRIL, verl docs, and the OPPO paper).

## 4. The Train → Infer Weight Resync (weight transfer)

After every policy update the learner's new weights must be pushed into the rollout engine before the next generation, because the rollout engine holds a **stale copy** otherwise. This per-step **weight resync** is a notable systems cost and a frequent source of bugs ([vLLM weight-transfer docs](https://docs.vllm.ai/en/latest/training/weight_transfer/); [TRL](https://huggingface.co/docs/trl/vllm_integration)).

Mechanisms ([vLLM weight-transfer / NCCL engine docs](https://docs.vllm.ai/en/latest/training/weight_transfer/nccl/)):
- **NCCL broadcast** (default) — trainer rank 0 broadcasts weights to all inference workers in a process group. vLLM's `update_weights` API supports `packed=True`, packing many tensors into large contiguous buffers to reduce the number of NCCL ops, with **double/triple buffering and dedicated CUDA streams** to overlap packing, broadcast, and unpacking.
- **CUDA IPC** (`backend="ipc"`) — for **colocated** transfers on the same node, hand off via shared GPU memory rather than a network collective.
- **Checkpoint reload / Hub bucket** — slower fallback; TRL's "delta weight sync" ships only changed weights through a Hub bucket for trillion-parameter models ([TRL delta-weight-sync](https://huggingface.co/blog/delta-weight-sync)).

The hard part is **resharding across mismatched layouts**: the trainer is sharded one way (FSDP/Megatron TP×PP), the inference engine another (vLLM/SGLang TP). veRL handles this with **sharding managers** — `FSDPVllmShardingManager` and `MegatronVLLMShardingManager` — that reshard actor→rollout weights; the in-place zero-copy reshard is only possible in colocated engine mode ([verl issue #3232](https://github.com/volcengine/verl/issues/3232); [verl Megatron backend docs](https://verl.readthedocs.io/en/v0.5.x/workers/megatron_workers.html)). slime exposes `UpdateWeightFromTensor` (colocated) vs `UpdateWeightFromDistributed` (multi-node) ([slime/DeepWiki](https://deepwiki.com/THUDM/slime)). Dedicated tools now target this step specifically: Ant's **AWEX** advertises "second-level parameter updates from training to inference" ([inclusionAI/asystem-awex](https://github.com/inclusionAI/asystem-awex)). vLLM also added native weight-syncing APIs and sleep/wake support so a colocated engine can free KV-cache memory during training and reload weights on wake ([vLLM RFC #31848](https://github.com/vllm-project/vllm/issues/31848); [RFC #15254](https://github.com/vllm-project/vllm/issues/15254)). **Confidence: High.**

## 5. Async / Off-Policy RL Systems (staleness, streaming rollout)

Synchronous RL forces the trainer to wait for the slowest rollout. **Asynchronous RL decouples generation from training**: rollout workers generate continuously while training workers update whenever a batch is ready ([AReaL, arXiv:2505.24298](https://arxiv.org/abs/2505.24298); [Async RLHF, arXiv:2410.18252](https://arxiv.org/html/2410.18252v3)). The cost is **off-policy staleness** — rollouts were produced by an older policy than the one being updated, which biases the gradient and must be corrected.

**AReaL** (Ant Research) is the canonical fully-async system: **streaming generation** (each rollout worker generates without waiting), **interruptible rollout workers**, **dynamic batching** for variable-length outputs, and a **parallel reward service**. It uses a **staleness-controlled, modified PPO** that tolerates samples from models **up to 8 steps old with no performance drop**, plus a data-filtering step to cap staleness — achieving **~2× speedup** at equal final accuracy ([AReaL, arXiv:2505.24298](https://arxiv.org/abs/2505.24298); [OpenReview](https://openreview.net/forum?id=X9diEuva9R)). Variants extend this: **AReaL-Hex** runs async RL over heterogeneous GPUs ([arXiv:2511.00796](https://arxiv.org/pdf/2511.00796)).

A spectrum exists between fully-sync and fully-async ([HF survey](https://huggingface.co/blog/async-rl-training-landscape)):
- **One-step-off / periodic asynchrony** — overlap generation of step *k+1* with training of step *k* (at most one step stale); on-policy-like accuracy with async throughput ([Periodic Asynchrony, arXiv:2511.18871](https://arxiv.org/pdf/2511.18871)).
- **Partial rollouts (APRIL)** — over-provision requests, stop when the target count is reached, and **recycle the unfinished long generations into the next step** — taming the tail *without* full async decoupling: **+22.5% avg (up to 44%) rollout throughput** ([APRIL, arXiv:2509.18521](https://arxiv.org/abs/2509.18521)).
- **Fully async** — AReaL-style continuous decoupling, max throughput, most staleness to manage.

NeMo-RL and OpenRLHF both ship async rollouts + replay buffers for off-policy training ([NeMo-RL](https://github.com/NVIDIA-NeMo/RL); [OpenRLHF GitHub](https://github.com/OpenRLHF/OpenRLHF)). **Confidence: High.**

## 6. The Framework Landscape (2024–2026)

| Framework | Org | Training backend | Rollout backend | Orchestration | Default placement | Notes |
| --- | --- | --- | --- | --- | --- | --- |
| **veRL / HybridFlow** | ByteDance | FSDP, Megatron | vLLM, SGLang | hybrid single+multi-controller | colocate (3D-HybridEngine), supports disaggregated | Most-used; in-place zero-redundancy reshard; 1.53–20.57× ([paper](https://arxiv.org/abs/2409.19256), [repo](https://github.com/verl-project/verl)) |
| **OpenRLHF** | community | DeepSpeed ZeRO-3 | vLLM | Ray Placement Groups | disaggregated/separated | First production Ray+vLLM+DeepSpeed; 3–4×+ DeepSpeed-Chat; now PPO/GRPO/REINFORCE++/async agentic ([arXiv:2405.11143](https://arxiv.org/pdf/2405.11143), [repo](https://github.com/OpenRLHF/OpenRLHF)) |
| **NeMo-Aligner** | NVIDIA | Megatron-LM 3D parallel | TensorRT-LLM | — | colocate | Scales to 1000s of GPUs (Nemotron-4 340B, Llama-3.1 405B); TRT-LLM accelerated generation ([arXiv:2405.01481](https://arxiv.org/abs/2405.01481)) |
| **NeMo-RL** | NVIDIA | Megatron, FSDP | vLLM, SGLang | Ray | both | Re-architected NeMo-Aligner; FP8 training, VLM SFT/GRPO, fully-async GRPO; inspired veRL/SkyRL/ROLL ([docs](https://docs.nvidia.com/nemo/rl/0.4.0/index.html)) |
| **TRL** | HuggingFace | Accelerate/FSDP/DeepSpeed | vLLM (colocate or server) | process / HTTP | colocate or server mode | GRPOTrainer/PPOTrainer; NCCL weight sync every `weight_sync_steps`; async GRPO ([vLLM integration](https://huggingface.co/docs/trl/vllm_integration), [async GRPO](https://huggingface.co/docs/trl/async_grpo_trainer)) |
| **ROLL** | Alibaba | Megatron | vLLM/SGLang | Ray, single-controller + parallel worker | both | Rollout scheduler with per-sample lifecycle, env+reward workers for agentic; 200B+ MoE on 1000s GPUs ([arXiv:2506.06122](https://arxiv.org/abs/2506.06122), [repo](https://github.com/alibaba/ROLL)) |
| **slime** | THUDM / Z.ai | Megatron-LM | SGLang-native | Ray, HTTP | colocate or decoupled, sync or async | Deep SGLang integration; SlimeRouter, StringRadixTrie cache; UpdateWeightFromTensor/Distributed ([LMSYS blog](https://www.lmsys.org/blog/2025-07-09-slime/), [repo](https://github.com/THUDM/slime)) |
| **AReaL / AReaL-boba** | Ant | FSDP/Megatron | SGLang | fully async | disaggregated | Streaming/interruptible rollout, staleness ≤8 steps, parallel reward service, ~2× ([arXiv:2505.24298](https://arxiv.org/abs/2505.24298)) |
| **SkyRL** | NovaSky/Anyscale | FSDP | vLLM | Ray | disaggregated | Uses inference stack as tokenization source-of-truth; VLM RL ([Anyscale blog](https://www.anyscale.com/blog/vision-language-model-reinforcement-learning-skyrl)) |
| **TorchForge / torchtune RL** | Meta/PyTorch | PyTorch-native FSDP | vLLM | — | — | PyTorch-native post-training; covered in HF 16-library survey ([HF survey](https://huggingface.co/blog/async-rl-training-landscape)) |

The HF survey's key meta-finding: **16 libraries built independently converged on the same fix** — get off the synchronous pipeline because idle training GPUs are killing throughput; they differ mainly along the seven axes (orchestration primitive, rollout buffer, weight-sync protocol, sync/async degree, placement, etc.) ([HF survey](https://huggingface.co/blog/async-rl-training-landscape); [Anyscale OSS RL libraries](https://www.anyscale.com/blog/open-source-rl-libraries-for-llms)). **Confidence: High.**

## 7. Reward-Model Serving + Verifier / Code Sandboxes in the Loop

The reward source is itself a served system component ([Promptfoo RLVR](https://www.promptfoo.dev/blog/rlvr-explained/); [Data Scaling RLHF, arXiv:2503.22230](https://arxiv.org/pdf/2503.22230)):
- **Learned RM serving** — a frozen reward model served as a separate inference service (its own GPUs in disaggregated setups, or colocated). AReaL runs a **parallel reward service** so scoring overlaps generation ([AReaL](https://arxiv.org/abs/2505.24298)).
- **Rule-based verifiers** — math-answer checking, regex/format checks (RLVR); cheap, deterministic ([Promptfoo](https://www.promptfoo.dev/blog/rlvr-explained/)).
- **Code-execution sandboxes** — for code RL, generated programs are **executed against unit tests inside a sandbox** to produce a binary verifiable reward. Execution-based RLVR outperforms learned reward models, which suffer instability and reward hacking on code ([CodeScaler, arXiv:2602.17684](https://arxiv.org/pdf/2602.17684); [ExecVerify, arXiv:2603.11226](https://arxiv.org/pdf/2603.11226)).

Systems concerns: the **verifier/reward can become the bottleneck** (Unit-Test test-time-scaling shows a clear performance–latency trade-off), and at scale teams serve **hundreds of environments as autoscaled managed sandbox endpoints** — e.g. "OpenReward" serving 330+ RL environments backed by 4.5M+ tasks and autoscaled sandbox compute ([CodeScaler](https://arxiv.org/pdf/2602.17684); [RL environments taxonomy](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/)). Sandbox **isolation and throughput** (parallel execution, timeouts for non-terminating code) are first-order infra problems. The standing warning: **verifier quality is the real bottleneck** — weak auto-generated reward functions teach the wrong behavior at scale ([RL environments taxonomy](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/)). **Confidence: High.**

## 8. Scaling the Trainer (FSDP/Megatron) Alongside the Rollout Engine (TP)

Training and inference **want different parallelism layouts**, which is the root reason the resync (§4) is non-trivial ([verl FSDP/Megatron docs](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html); [HybridFlow](https://arxiv.org/abs/2409.19256)):
- **Trainer** needs to shard parameters + gradients + optimizer states + activations → FSDP (ZeRO-3-style) or Megatron 3D/5D (TP×PP×DP, +CP/EP). Optimized for backward-pass memory.
- **Rollout engine** has no backward/optimizer/gradient state → wants a layout that minimizes inference latency, typically a **smaller tensor-parallel** degree with high data-parallel replication for batch throughput (e.g. train TP=8 vs infer TP=4) ([verl reshard discussion](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html)).

When **colocated**, the two layouts contend for the same GPU memory, so frameworks **offload the trainer (params/optimizer to CPU or freed) while generating**, then reload — exactly what vLLM sleep/wake and the 3D-HybridEngine reshard enable ([vLLM RFC #15254](https://github.com/vllm-project/vllm/issues/15254); [HybridFlow](https://arxiv.org/abs/2409.19256)). When **disaggregated**, the two pools size independently but pay network resync. veRL exposes both FSDP and Megatron trainer backends behind a common worker API and maps them to vLLM/SGLang rollout workers ([verl Megatron backend](https://verl.readthedocs.io/en/v0.5.x/workers/megatron_workers.html); [verl repo](https://github.com/verl-project/verl)). **Note:** the *supervised* parallelism mechanics (ZeRO/FSDP/TP/PP/EP internals) belong to a distributed-training reference — covered here only as the RL-specific train-vs-infer layout mismatch. **Confidence: High.**

## 9. RL-Specific Throughput & GPU Under-Utilization (the "bubble")

In a naive synchronous PPO/GRPO loop the stages run sequentially with **hard dependencies**: the reward model cannot score until the actor finishes generating; the learner cannot step until rewards are in. This creates an **idle "bubble"** — generation GPUs busy while training GPUs idle and vice-versa — amplified by the long-tailed response lengths ([OPPO, arXiv:2509.25762](https://arxiv.org/html/2509.25762v1); [HF survey](https://huggingface.co/blog/async-rl-training-landscape)). Measured actor-generation utilization is **<40%** ([OPPO](https://arxiv.org/html/2509.25762v1)).

Mitigations (distinct from full async, §5):
- **Intra-step overlap** — stream upstream outputs in chunks so the downstream model begins prefill while the upstream is still decoding ([OPPO](https://arxiv.org/html/2509.25762v1)).
- **Inter-step overlap** — overcommit prompts and defer long generations to the next step to cut tail latency ([OPPO](https://arxiv.org/html/2509.25762v1)).
- **Partial-rollout recycling** — APRIL's over-provision-and-recycle ([APRIL](https://arxiv.org/abs/2509.18521)).

Reported gains: OPPO **1.8×–2.8×** end-to-end with **1.4×–2.1×** higher GPU utilization, no convergence loss ([OPPO](https://arxiv.org/html/2509.25762v1)); APRIL **+20–44%** rollout throughput ([APRIL](https://arxiv.org/abs/2509.18521)). An operational corollary: classic Slurm gang-scheduling fits supervised training but not the heterogeneous, long-lived, multi-role RL job, which is why these systems lean on Ray ([SkyPilot, "RL Doesn't Work on Slurm"](https://blog.skypilot.co/rl-doesnt-work-on-slurm/); [HF survey](https://huggingface.co/blog/async-rl-training-landscape)). **Confidence: High.**

## 10. Failure Modes Unique to RL Systems

**(a) Train/inference logprob mismatch — the headline systems bug.** The rollout engine (vLLM/SGLang) and the trainer (FSDP/Megatron) produce **different log-probabilities for the same sequence under the same weights**, because they use different kernels, precision, and batching paths ([Swift training-inference-mismatch docs](https://swift.readthedocs.io/en/v3.11/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.html); [LLM Data Co.](https://www.llmdata.com/blog/mismatch-praxis)). This silently turns "on-policy" RL into **off-policy** RL with nontrivial bias — the behavior policy (inference) differs from the proxy/reference policy (training) ([Diagnosing Training-Inference Mismatch, arXiv:2605.14220](https://arxiv.org/html/2605.14220)). Corrections: **token-level Truncated Importance Sampling (TIS)** downweights tokens with severe mismatch and is stable, outperforming no correction despite its bias; alternatives mask out off-policy tokens or use sequence-level IS ([LLM Data Co.](https://www.llmdata.com/blog/mismatch-praxis); [Swift](https://swift.readthedocs.io/en/v3.11/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.html)). A famous gotcha: with temperature ≠ 1, **vLLM does not apply temperature scaling to returned logprobs by default**, producing a huge spurious mismatch that breaks TIS ([TRL issue #4159](https://github.com/huggingface/trl/issues/4159)). The vLLM V0→V1 work argues for fixing **correctness before adding IS corrections** (batch-invariance, kernel alignment) so the mismatch shrinks at the source ([ServiceNow-AI, "Correctness Before Corrections"](https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections)). For **MoE** models the mismatch is worse, and no current open-source async library implements the "Keep Routing" fix (replaying expert routing) — a correctness gap for DeepSeek-V3/Mixtral-class training ([HF survey](https://huggingface.co/blog/async-rl-training-landscape)).

**(b) Weight-sync bugs.** Stale or partially-synced weights leave the rollout engine generating from an old policy. Real-world example: vLLM weights silently **not synchronized when `vllm_enable_sleep_mode=True`** ([TRL issue #5312](https://github.com/huggingface/trl/issues/5312)). Symptoms look like a "stuck" or diverging reward curve. The resync (§4) and its layout-reshard correctness are the surface for these bugs.

**(c) Reward over-optimization / hacking at scale (systems symptoms).** As RL scales, the policy exploits flaws in the reward source — reward keeps rising while true quality stalls or drops. Execution-based code rewards are more robust than learned RMs, which suffer instability and hacking ([CodeScaler, arXiv:2602.17684](https://arxiv.org/pdf/2602.17684); [RL environments taxonomy](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/)). The *algorithmic* mitigations (KL penalty, RM ensembles, ODIN) live in the alignment-algorithm domain; the *systems* responsibility is verifier quality, sandbox correctness, and reward-service monitoring. **Confidence: High** for (a)/(b) (multiple framework issues + docs + blogs); **Medium-High** for (c) systems framing (the algorithmic side is out of scope here).

---

## Key Takeaways

- The actor–rollout–learner loop, not the RL algorithm, is the unit of RL *systems* design; **rollout dominates (60–90%+)** and dictates every other choice.
- **Placement** (colocate/3D-HybridEngine vs disaggregated/Ray) and **weight resync** (NCCL-broadcast/packed buffers vs IPC vs delta) are the two levers that most affect throughput and correctness.
- **Async/off-policy** (AReaL fully-async ≤8-step staleness, one-step-off, APRIL partial-rollout recycling) exists to kill the synchronous idle bubble; the price is staleness you must correct.
- The signature bug is the **train/inference logprob mismatch** (kernel/precision/batching) that makes nominally on-policy RL off-policy and needs TIS or correctness fixes; weight-sync bugs and reward over-optimization round out the failure set.
- Framework choice: **veRL** (most-used, hybrid, in-place reshard), **OpenRLHF** (Ray disaggregated), **NeMo-RL/-Aligner** (NVIDIA, Megatron+TRT-LLM, largest scale), **TRL** (accessible, colocate/server), **slime** (SGLang-native), **AReaL** (fully-async SOTA), **ROLL** (agentic-friendly scheduler).

## Knowledge Gaps

- **Exact weight-resync wall-clock** as a % of step time is not consistently published; sources describe it as "notable" and tool vendors (AWEX) target "second-level" updates, but a clean cross-framework benchmark was not found.
- **TorchForge/torchtune RL** internals are thinly documented in public primary sources beyond the HF 16-library survey listing them.
- Long-form arXiv HTML pages (HybridFlow, AReaL, OPPO, APRIL) repeatedly **timed out on fetch**; their abstracts, OpenReview pages, and engineering-blog summaries were used instead — numbers are corroborated across ≥2 sources but full-text verification of secondary details was limited.

## Sources

1. [HybridFlow: A Flexible and Efficient RLHF Framework (arXiv:2409.19256)](https://arxiv.org/abs/2409.19256) — veRL's single+multi-controller hybrid, 3D-HybridEngine reshard, 1.53–20.57×.
2. [verl HybridFlow Programming Guide](https://verl.readthedocs.io/en/latest/hybrid_flow.html) + [FSDP](https://verl.readthedocs.io/en/latest/workers/fsdp_workers.html) / [Megatron](https://verl.readthedocs.io/en/v0.5.x/workers/megatron_workers.html) / [SGLang](https://verl.readthedocs.io/en/latest/workers/sglang_worker.html) backend docs — sharding managers, reshard, backends.
3. [verl-project/verl GitHub](https://github.com/verl-project/verl) + [issue #3232 FSDPVllmShardingManager](https://github.com/volcengine/verl/issues/3232) — repo, sharding-manager internals.
4. [OpenRLHF: Easy-to-use, Scalable, High-performance RLHF (arXiv:2405.11143)](https://arxiv.org/pdf/2405.11143) + [GitHub](https://github.com/OpenRLHF/OpenRLHF) — Ray Placement Groups, separated placement, 4-model PPO.
5. [Accelerating RLHF with vLLM, Best Practice from OpenRLHF (vLLM blog)](https://blog.vllm.ai/2025/04/23/openrlhf-vllm.html) — vLLM as in-loop generator, generation bottleneck.
6. [NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment (arXiv:2405.01481)](https://arxiv.org/abs/2405.01481) — Megatron 3D parallel + TensorRT-LLM generation, 1000-GPU scale.
7. [NVIDIA-NeMo/RL GitHub](https://github.com/NVIDIA-NeMo/RL) + [NeMo-RL docs](https://docs.nvidia.com/nemo/rl/0.4.0/index.html) — re-architected NeMo-Aligner, async GRPO, disaggregated mode.
8. [AReaL: Large-Scale Asynchronous RL System (arXiv:2505.24298)](https://arxiv.org/abs/2505.24298) + [OpenReview](https://openreview.net/forum?id=X9diEuva9R) — fully-async, interruptible rollout, ≤8-step staleness, parallel reward service, ~2×.
9. [AReaL-Hex: Async RL over Heterogeneous GPUs (arXiv:2511.00796)](https://arxiv.org/pdf/2511.00796) — heterogeneous-GPU async RL.
10. [TRL vLLM Integration](https://huggingface.co/docs/trl/vllm_integration) + [Async GRPO](https://huggingface.co/docs/trl/async_grpo_trainer) + [vLLM TRL docs](https://docs.vllm.ai/en/latest/training/trl/) — colocate vs server mode, NCCL weight sync.
11. [vLLM Weight Transfer docs](https://docs.vllm.ai/en/latest/training/weight_transfer/) + [NCCL engine](https://docs.vllm.ai/en/latest/training/weight_transfer/nccl/) — update_weights, NCCL vs IPC, packed/double-buffered transfer.
12. [vLLM RFC #31848 Native Weight Syncing](https://github.com/vllm-project/vllm/issues/31848) + [RFC #15254 sleep-mode weight update](https://github.com/vllm-project/vllm/issues/15254) — sync APIs, sleep/wake.
13. [ROLL: RL Optimization for Large-Scale Learning (arXiv:2506.06122)](https://arxiv.org/abs/2506.06122) + [GitHub](https://github.com/alibaba/ROLL) — single-controller + parallel worker, rollout scheduler, env/reward workers, 200B MoE.
14. [slime (THUDM) GitHub](https://github.com/THUDM/slime) + [LMSYS blog](https://www.lmsys.org/blog/2025-07-09-slime/) + [DeepWiki](https://deepwiki.com/THUDM/slime) — SGLang-native, UpdateWeightFromTensor/Distributed, colocate/decoupled.
15. [Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries (HuggingFace)](https://huggingface.co/blog/async-rl-training-landscape) — the 7 design axes, universal idle-GPU finding, MoE "Keep Routing" gap.
16. [OPPO: Accelerating PPO-based RLHF via Pipeline Overlap (arXiv:2509.25762)](https://arxiv.org/html/2509.25762v1) — <40% gen GPU util, intra/inter-step overlap, 1.8–2.8×.
17. [APRIL: Active Partial Rollouts to Tame Long-tail Generation (arXiv:2509.18521)](https://arxiv.org/abs/2509.18521) — 90% rollout time, over-provision+recycle, +20–44%.
18. [ServiceNow-AI: vLLM V0→V1, Correctness Before Corrections (HF blog)](https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections) — logprob mismatch source, correctness vs IS.
19. [Mismatch Praxis: Rollout Settings and IS Corrections (LLM Data Co.)](https://www.llmdata.com/blog/mismatch-praxis) + [Swift training-inference-mismatch docs](https://swift.readthedocs.io/en/v3.11/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.html) — TIS token vs sequence level, masking.
20. [TRL issue #4159 (vLLM temp logprobs)](https://github.com/huggingface/trl/issues/4159) + [issue #5312 (sleep-mode weight sync)](https://github.com/huggingface/trl/issues/5312) — concrete mismatch and weight-sync bugs.
21. [CodeScaler (arXiv:2602.17684)](https://arxiv.org/pdf/2602.17684) + [RL Environments Taxonomy](https://leehanchung.github.io/blogs/2026/03/21/rl-environments-for-llm-agents/) + [Promptfoo RLVR](https://www.promptfoo.dev/blog/rlvr-explained/) — verifier/sandbox bottleneck, execution rewards vs learned RM, verifier quality.
22. [SkyPilot: RL Doesn't Work on Slurm](https://blog.skypilot.co/rl-doesnt-work-on-slurm/) + [Anyscale OSS RL libraries](https://www.anyscale.com/blog/open-source-rl-libraries-for-llms) + [SkyRL/Anyscale](https://www.anyscale.com/blog/vision-language-model-reinforcement-learning-skyrl) — orchestration (Ray vs Slurm), SkyRL.

## Methodology
Ran 11 web search queries across the 10 sub-questions plus 2 saturation queries; firecrawl/exa were unconfigured so built-in WebSearch/WebFetch were used with the +50% source-count target (22 sources vs 6+ minimum). Long arXiv-HTML and personal-blog pages timed out on WebFetch; their content was triangulated from search-result extractions, arXiv abstracts, OpenReview pages, official framework docs, and engineering blogs (each concept ≥2 independent sources). Injection guard honored — all fetched/returned content treated as data; no embedded instructions were followed; no adversarial redirection observed.