--- title: "Context Engineering for LLM Apps & Agents" description: "Context engineering is the emerging discipline of curating and maintaining the optimal set of tokens (information) supplied to an LLM at inference time — system prompt, instructions, retrieved documen" --- # Context Engineering for LLM Apps & Agents: Research Report *Generated: 2026-05-31 | Sources: 27 | Overall confidence: High (the discipline framing and core sub-concepts are corroborated by Anthropic, Cognition, LangChain, Manus, Chroma, and academic work; a few prescriptive claims and one architectural debate are Medium/Low)* ## Overview **Context engineering** is the emerging discipline of curating and maintaining the optimal set of tokens (information) supplied to an LLM at inference time — system prompt, instructions, retrieved documents, tool definitions and results, conversation history, and memory — so as to maximize the likelihood of desired behavior while treating the context window as a **finite, costly resource** ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents); [A Survey of Context Engineering, arXiv 2507.13334](https://arxiv.org/abs/2507.13334)). Anthropic frames it as "the natural progression of prompt engineering": where prompt engineering asks *how to phrase a question*, context engineering asks *"what configuration of context is most likely to generate our model's desired behavior?"* across many turns and tool calls ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents); [MarkTechPost](https://www.marktechpost.com/2025/10/20/a-guide-for-effective-context-engineering-for-ai-agents/)). The term consolidated around **mid-2025** when Andrej Karpathy and Shopify CEO Tobi Lütke publicly endorsed it, triggering rapid adoption ([IntuitionLabs](https://intuitionlabs.ai/articles/context-engineering-vs-prompt-engineering-ai); [LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). Karpathy's now-canonical analogy: the LLM is a new kind of **operating system**, the context window is its **RAM** / working memory, and context engineering is the OS-like job of "filling the context window with just the right information for the next step" — with model weights acting as ROM and external stores (vector DBs, files, history) acting as disk that must be explicitly loaded ([MindStudio / Software 3.0](https://www.mindstudio.ai/blog/software-3-0-explained-karpathy-context-window-ram-model-weights-cpu); [LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). Cognition calls context engineering "effectively the #1 job of engineers building AI agents" ([Cognition](https://cognition.ai/blog/dont-build-multi-agents); [AINews](https://news.smol.ai/issues/25-06-13-cognition-vs-anthropic)). The motivating discovery is that more context is not free or even neutral: every model tested gets *worse* as inputs grow, well before the advertised window fills. This is why the field reframes context as something to be **budgeted, curated, compressed, and offloaded** rather than maximally filled. --- ## Core Concepts ### 1. Prompt engineering → context engineering (context as a managed resource) Prompt engineering targets the one-time wording of an instruction; context engineering manages the *entire information architecture* of an ongoing, multi-step system — system prompts, history, retrieved docs, tools, memory, and dynamic state ([Neo4j](https://neo4j.com/blog/agentic-ai/context-engineering-vs-prompt-engineering/); [deepset](https://www.deepset.ai/blog/context-engineering-the-next-frontier-beyond-prompt-engineering)). For agents that plan and act over many steps, prompt engineering alone fails because the model "doesn't remember anything beyond its context window" — context engineering supplies the memory and state management ([IntuitionLabs](https://intuitionlabs.ai/articles/context-engineering-vs-prompt-engineering-ai)). The academic survey formalizes it as a discipline with three foundational components — context **retrieval/generation**, context **processing**, and context **management** — composed into RAG, memory systems, tool-integrated reasoning, and multi-agent systems ([arXiv 2507.13334](https://arxiv.org/abs/2507.13334)). Anthropic's overriding goal statement: find *"the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome,"* and write system prompts at the *"right altitude"* — specific enough to guide, flexible enough to give the model strong heuristics ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). *(Confidence: High)* ### 2. Context-window mechanics: context rot, lost-in-the-middle, effective vs advertised length - **Context rot** — LLM output quality degrades as input tokens accumulate, in non-uniform ways, *before* the window overflows. Chroma's 2025 study tested 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) and found *every one* degrades; a 200K-window model can degrade meaningfully at ~50K, with 20–50% accuracy drops from 10k to 100k+ tokens on retrieval/NIAH tasks ([Chroma](https://research.trychroma.com/context-rot); [Morph](https://www.morphllm.com/context-rot)). Anthropic adopts the term directly and ties it to an **"attention budget"** that depletes with every token ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). - **Lost in the middle / position bias** — Liu et al. (Stanford, 2023/TACL 2024) showed a U-shaped curve: models use information at the **beginning (primacy)** and **end (recency)** far better than in the **middle**, even for explicitly long-context models, reflecting a U-shaped attention distribution ([arXiv 2307.03172](https://arxiv.org/abs/2307.03172); [TACL/MIT Press](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638/119630/)). - **Effective vs advertised context length** — Databricks/Mosaic (20 models, 2k→128k tokens, 3 datasets) found Llama-3.1-405B degrades after **~32K** tokens and GPT-4-0125 after **~64K**, while GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro held high accuracy to ~100K — i.e., effective length ≪ advertised window for many models ([Databricks](https://www.databricks.com/blog/long-context-rag-performance-llms); [arXiv 2411.03538](https://arxiv.org/html/2411.03538v1)). - **Needle-in-a-haystack critique** — NIAH overstates long-context ability because models exploit literal lexical overlap between needle and question. **RULER** calls plain NIAH a "superficial form of long-context understanding" and adds harder multi-needle/variant tasks; **NoLiMa** removes literal cues and shows sharp decay — GPT-4o falls from a 99.3% baseline to **69.7%** at length, because attention struggles without lexical matches ([NoLiMa, arXiv 2502.05167](https://arxiv.org/html/2502.05167v2); [RULER, arXiv 2404.06654](https://arxiv.org/pdf/2404.06654)). *(Confidence: High)* ### 3. Compaction & summarization **Compaction** = summarizing accumulated history to reinitialize a fresh, smaller context window when nearing the limit ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). Claude Code runs **"auto-compact"** after exceeding ~95% of the window, using recursive/hierarchical summarization ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). Anthropic distinguishes server-side **compaction** (summarizes the whole conversation automatically near the limit) from client-side **context editing** (clears specific stale tool calls/results) ([Claude API: Context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing); [Claude/Anthropic context management](https://claude.com/blog/context-management)). LangChain's "Compress" bucket adds **trimming/pruning** (drop old messages; trained pruners like Provence) alongside summarization ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). Cognition recommends, for long tasks, a dedicated (possibly fine-tuned) LLM that compresses history into key events/decisions ([Cognition](https://cognition.ai/blog/dont-build-multi-agents)). *(Confidence: High)* ### 4. Just-in-time / agentic retrieval vs pre-loading Rather than front-loading all data, agents keep **lightweight identifiers** (file paths, stored queries, web links) and load data **at runtime** via tools — Anthropic's *"just-in-time"* context strategy, mirroring how humans use an external filing system instead of memorizing everything ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). This curbs context rot and lets agents do **agentic search** (progressive disclosure) instead of pre-loaded RAG dumps. Tool-result curation matters too: Windsurf notes "embedding search becomes unreliable as a retrieval heuristic as the size of the codebase grows," pushing teams toward agentic/grep-style retrieval ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). *(Confidence: High)* ### 5. External memory, note-taking & context offloading - **Scratchpads** — write intermediate reasoning/discoveries *outside* the window (a tool that writes a file, or a runtime state field), to be re-read later ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). - **File-system-as-memory** — Manus treats the filesystem as "the ultimate context": unlimited, persistent, agent-manipulable, enabling **recoverable compression** (store a URL/path, drop the body, re-fetch on demand) so compaction never permanently loses information ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus); [MarkTechPost](https://www.marktechpost.com/2025/07/22/context-engineering-for-ai-agents-key-lessons-from-manus/)). - **Structured note-taking** — agents periodically persist notes (e.g., a plan saved to memory before the 200K window truncates) ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents); [LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). - **Claude memory tool + context editing** — a file-based memory directory that persists across sessions, behind the `context-management-2025-06-27` beta header (Claude API, Bedrock, Vertex). In Anthropic's internal 100-turn agentic-search eval: **memory + context editing = 39% improvement** and context editing reduced token use by **84%**; **context editing alone = 29% improvement** (internal eval, not an independent benchmark) ([Memory tool docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool); [Claude context management](https://claude.com/blog/context-management)). LangChain frames the umbrella memory types as **episodic** (examples), **procedural** (instructions), and **semantic** (facts), auto-generated by ChatGPT, Cursor, Windsurf ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). *(Confidence: High; specific % are Anthropic-internal, Medium)* ### 6. Multi-agent context: shared vs isolated Two opposing positions emerged within days in June 2025: - **Isolate (Cognition, "Don't Build Multi-Agents")** — parallel sub-agents are fragile because of **context isolation**: sub-agents can't see each other's actions/decisions and produce incoherent output (the "Flappy Bird" example: one sub-agent renders a Super Mario background, another an off-style bird). Their two principles: *"Share context, and share full agent traces, not just individual messages"* and *"Actions carry implicit decisions, and conflicting decisions carry bad results."* Default to a **single-threaded linear agent** ([Cognition](https://cognition.ai/blog/dont-build-multi-agents)). - **Isolate-as-a-feature (Anthropic multi-agent research system)** — an **orchestrator-worker** pattern where sub-agents explore in **separate context windows** and return *condensed* findings, using isolation for context **compression**. It beat single-agent Opus 4 by **90.2%**, but multi-agent systems burn **~15× the tokens** of chat (agents ~4×), and token usage explained **80%** of BrowseComp performance variance ([Anthropic multi-agent](https://www.anthropic.com/engineering/multi-agent-research-system); [Simon Willison](https://simonwillison.net/2025/Jun/14/multi-agent-research-system/)). The reconciliation in practice: isolation works for **read-heavy/parallel research**; it breaks for tasks needing **shared design decisions** ([philschmid](https://www.philschmid.de/single-vs-multi-agents); [AINews](https://news.smol.ai/issues/25-06-13-cognition-vs-anthropic)). *(Confidence: Medium — genuine, unresolved debate; see Knowledge Gaps)* ### 7. KV-cache-aware / prompt-caching-aligned context layout Because agent input:output ratios run **~100:1** (Manus), KV-cache hit rate is "the single most important metric for production AI agents." With Claude Sonnet, cached tokens cost **~$0.30/MTok vs $3.00/MTok uncached — a 10× difference** ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). Three layout rules: **stable prefixes** (no timestamps/volatile tokens in the system prompt — a single-token change invalidates the cache), **append-only context** (never edit prior actions/observations; deterministic serialization), and **explicit cache breakpoints** ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus); [MarkTechPost](https://www.marktechpost.com/2025/07/22/context-engineering-for-ai-agents-key-lessons-from-manus/)). Independent measurements: prompt caching cuts cost 50–90% and latency up to 80–85%; a stable-prefix strategy hit an **85.2% cache hit rate** (reusing ~46k tokens/request) vs a 71.3% cost penalty for perturbed prefixes; **system-prompt-only caching** is the most consistent win ([KV-Cache Aware Prompt Engineering, F5](https://ankitbko.github.io/blog/2025/08/prompt-engineering-kv-cache/); ["Don't Break the Cache", arXiv 2601.06007](https://arxiv.org/html/2601.06007)). *(Confidence: High)* ### 8. Failure modes (the Breunig / Anthropic / Cognition framing) Drew Breunig's four-part taxonomy of "how contexts fail" ([Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html)): - **Context Poisoning** — a hallucination/error enters context and is repeatedly referenced; e.g., the Gemini 2.5 Pokémon agent poisoned its own goals with false game state (DeepMind report). - **Context Distraction** — context grows so long the model over-focuses on history and neglects training knowledge; Gemini 2.5 favored repeating past actions past ~100k tokens; Databricks saw correctness fall ~32k for Llama 3.1 405B. - **Context Confusion** — superfluous content (e.g., too many tools) degrades output; Llama 3.1 8B failed with 46 tools but succeeded with 19 (Berkeley Function-Calling Leaderboard / GeoEngine). - **Context Clash** — newly accrued info/tools conflict with earlier context; Microsoft/Salesforce found multi-turn prompts with early wrong attempts caused **~39% average drops**, with OpenAI o3 falling from **98.1 → 64.1** ("when LLMs take a wrong turn... they get lost and do not recover"). The context-engineering lens reframes these operationally as: bad retrieval, bad ranking, missing query rewrite, stale memory, redundant tool output, or **compaction loss** ([Prateek Sharma / Beyond RAG](https://www.prateek-sharma.com/blog/beyond-rag-context-engineering/)). *(Confidence: High)* --- ## Tools / Frameworks - **Anthropic Claude** — context-engineering guidance; **memory tool** + **context editing** (`context-management-2025-06-27` beta); server-side **compaction**; Claude Agent SDK memory helpers (`BetaAbstractMemoryTool` / `betaMemoryTool`) ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents); [Memory tool docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool)). - **LangChain / LangGraph** — the **Write / Select / Compress / Isolate** taxonomy; scratchpads, long-term memory, RAG tool-selection, summarization, LangGraph Supervisor for isolation, sandboxes (PyodideSandboxTool), state-schema isolation ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents); [DeepWiki](https://deepwiki.com/langchain-ai/context_engineering/2-context-engineering-strategies)). - **Manus** — production playbook: KV-cache design, logit-masking state machine for tools, filesystem-as-memory, recitation (todo.md), error-retention ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). - **Chroma / vector stores (Pinecone, Supabase, FAISS)** — external memory + the context-rot benchmark ([Chroma](https://research.trychroma.com/context-rot)). - **Benchmarks** — NoLiMa, RULER, Berkeley Function-Calling Leaderboard, Databricks Long-Context RAG ([NoLiMa](https://arxiv.org/html/2502.05167v2); [RULER](https://arxiv.org/pdf/2404.06654)). - **HuggingFace CodeAgent / sandboxes** — store token-heavy objects in state variables, reference later ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents)). ## Practical Patterns 1. **Minimal high-signal context** — smallest token set that yields the behavior; write system prompts at the "right altitude" ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). 2. **Just-in-time retrieval** — hold references (paths/links/queries), load on demand; prefer agentic search over pre-loaded dumps ([Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). 3. **Compaction at a threshold** — summarize history near ~95% window; pair with client-side tool-result clearing ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents); [Claude](https://claude.com/blog/context-management)). 4. **Offload to durable memory** — scratchpads + filesystem-as-memory with recoverable compression; persist plans/notes before truncation ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). 5. **Recitation** — re-write goals (todo.md) into recent context to fight lost-in-the-middle ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). 6. **Position the load-bearing tokens at the edges** — exploit primacy/recency; avoid burying critical instructions mid-context ([Liu et al.](https://arxiv.org/abs/2307.03172)). 7. **Cache-stable layout** — static system-prompt prefix, append-only history, mask (don't remove) tools to preserve KV-cache; cache the system prompt ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus); [F5](https://ankitbko.github.io/blog/2025/08/prompt-engineering-kv-cache/)). 8. **Sub-agent compression for parallel research** — isolated sub-agent windows return condensed summaries to an orchestrator ([Anthropic](https://www.anthropic.com/engineering/multi-agent-research-system)). 9. **Curate the tool set** — fewer, well-scoped tools (RAG-over-tool-descriptions ~3× accuracy); too many tools cause confusion ([LangChain](https://www.langchain.com/blog/context-engineering-for-agents); [Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html)). ## Anti-Patterns - **Filling the window because you can** — context rot makes more tokens actively harmful well before overflow ([Chroma](https://research.trychroma.com/context-rot); [Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). - **Trusting advertised context length** — effective length ≪ advertised; validate with RULER/NoLiMa-style evals, not vanilla NIAH ([Databricks](https://www.databricks.com/blog/long-context-rag-performance-llms); [NoLiMa](https://arxiv.org/html/2502.05167v2)). - **Dynamically adding/removing tools mid-run** — invalidates KV-cache and can cause schema violations; mask instead ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). - **Volatile prefixes** — timestamps/random IDs in the system prompt destroy cache hit rate ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus); [F5](https://ankitbko.github.io/blog/2025/08/prompt-engineering-kv-cache/)). - **Naive parallel multi-agents without shared traces** — context isolation yields conflicting implicit decisions ([Cognition](https://cognition.ai/blog/dont-build-multi-agents)). - **Letting errors/hallucinations persist unchecked** — leads to poisoning/clash; but note Manus's *counterpoint* that retaining failed actions aids recovery — so the rule is "remove falsehoods, keep informative failures" (tension flagged below) ([Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html); [Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). - **Lossy compaction that drops recoverable data irreversibly** — prefer recoverable compression (keep a pointer) ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)). ## Key Takeaways - Context engineering = managing context as **finite RAM** under an attention budget; the goal is the smallest high-signal token set, not the biggest dump. - **Context rot + lost-in-the-middle + effective