LLM Observability
Parent: AI Agent Ecosystems · researched 2026-05-31T17:21:52.875Z· 31 sources · 12 concepts · skill ai-agent-engineering
AI & agent-engineering family ROUTER. Split into: ai-agents-orchestration (agent frameworks, multi-agent, memory, planning, guardrails, coding/GUI agents, autonomous loops, eval); ai-rag-retrieval (RA
ai-agent-engineering
- AI & agent-engineering family ROUTER. Split into: ai-agents-orchestration (agent frameworks, multi-agent, memory, planning, guardrails, coding/GUI agents, autonomous loops, eval); ai-rag-retrieval (RAG, iterative retrieval, vector/graph datastores); ai-llm-model-layer (training, fine-tuning, alignment/RLHF, compression, inference serving, transformer/multimodal architecture, model selection, observability); ai-mcp-sdk-prompting (MCP servers/builder, Anthropic SDK, prompt engineering, context engineering, LLM frameworks, tool-search, prompt lookup). Route to the matching sub-hub. [source]
- This hub routes to on-demand reference files under references/. See each spoke for depth. [source]
Children
- LLM/agent tracing and spans (frontier)
- OpenTelemetry GenAI semantic conventions (frontier)
- OpenInference span-kind standard (frontier)
- OpenLLMetry / Traceloop instrumentation (frontier)
- Token, cost, and latency monitoring (frontier)
- Online (reference-free) evaluation vs offline eval (frontier)
- LLM-as-judge production scoring (frontier)
- Hallucination / groundedness / faithfulness monitoring (frontier)
- Answer-relevance monitoring (frontier)
- Prompt / output / embedding drift detection (frontier)
- Runtime guardrails and safety monitoring (frontier)
- PII detection and redaction at runtime (frontier)
- RAG observability (RAGAS: context precision/recall, faithfulness) (frontier)
- Agent trajectory and tool-call observability (frontier)
- Loop / recursion detection in agents (frontier)
- Prompt management and versioning (frontier)
- Datasets and experiment tracking (frontier)
- Observability tooling landscape (Langfuse, LangSmith, Arize Phoenix, WhyLabs/LangKit, Helicone, Datadog, OpenLLMetry) (frontier)
Frontier under this node: Agent trajectory and tool-call observability, Answer-relevance monitoring, Datasets and experiment tracking, Hallucination / groundedness / faithfulness monitoring, LLM-as-judge production scoring, LLM/agent tracing and spans, Loop / recursion detection in agents, Observability tooling landscape (Langfuse, LangSmith, Arize Phoenix, WhyLabs/LangKit, Helicone, Datadog, OpenLLMetry), Online (reference-free) evaluation vs offline eval, OpenInference span-kind standard, OpenLLMetry / Traceloop instrumentation, OpenTelemetry GenAI semantic conventions, PII detection and redaction at runtime, Prompt / output / embedding drift detection, Prompt management and versioning, RAG observability (RAGAS: context precision/recall, faithfulness), Runtime guardrails and safety monitoring, Token, cost, and latency monitoring