Augmented Analytics and LLM-Assisted Analysis

Parent: data analysis · researched 2026-05-31T04:52:22.177Z· 28 sources · 7 concepts · skill da-39-augmented-analytics-llm-assisted

How AI augments or automates the analytical loop — preparing data, finding insights, answering natural-language questions, and explaining results — instead of a human writing every query and reading e

Augmented Analytics & LLM-Assisted Analysis

How AI augments or automates the analytical loop - preparing data, finding insights, answering natural-language questions, and explaining results - instead of a human writing every query and reading every chart. The audience is an analyst, data/BI engineer, or product owner deciding whether and how to put an LLM between users and data, and how to keep the answers correct. [source]
Scope boundary. This skill is about the AI/LLM layer over analytics. It deliberately does not re-teach: [source]
The semantic/metrics layer itself → da-18-semantic-layer-headless-bi. Here we only cover how an LLM is grounded in that layer. [source]
Unstructured-text NLP (topic modeling, sentiment, NER, text embeddings) → da-36-text-analytics-nlp. [source]
Generic ML / LLM training, fine-tuning, eval theory → da-7-machine-learning. [source]
Generic RAG pipeline architecture with no numeric/analytical angle → rag-architecture. [source]

1. Augmented analytics → agentic analytics (the Gartner arc)

Gartner (Rita Sallam, 2017) defined augmented analytics as using ML/AI to assist data preparation, insight generation, and insight explanation to augment how people explore and analyze data. Gartner's evaluation criteria span six capabilities: ML-assisted insight discovery, NLP/NLQ querying, automated explanations (NLG), data-prep assistance, GenAI integration, and augmented data science. [source]
The 2025 evolution is agentic analytics (Gartner Market Guide for Agentic Analytics, Feb 2025): AI agents that don't just assist but autonomously plan, investigate, and act. Gartner predicts ~75% of analytics content will use GenAI for contextual intelligence by 2027, evolving toward "autonomous analytics" managing a slice of business processes. The four classic analytics tiers map onto this: descriptive (what happened) → diagnostic (why) → predictive (what will) → prescriptive (what to do) - augmented/agentic analytics automates the first two and increasingly drives toward the latter two. [source]

2. Conversational BI / NLQ

Natural-Language Query (NLQ) turns a plain-English question into a query against governed data, returns a result, a chart, and (via NLG) a written explanation. The non-negotiable lesson of 2024-2026: accuracy depends on grounding the LLM in a governed semantic model, not the raw schema. The semantic layer (see da-18) supplies business term → table/column/metric mappings, relationships, synonyms, and metric definitions so "revenue" always means the same SQL. [source]
Snowflake Cortex Analyst is built around a YAML semantic model (now Semantic Views as the recommended form) plus a Verified Query Repository of approved question→SQL pairs that the model references at generation time. [source]
Guardrails: restrict to a curated model/views, prefer verified queries, validate generated SQL, constrain output (e.g., function-calling / JSON-schema-constrained SQL), enforce row/column security so the agent inherits the user's permissions. [source]

3. Text-to-SQL

The research workhorse of LLM-assisted analytics. [source]
Schema linking - selecting the relevant tables/columns for a question - is the dominant accuracy lever, especially on large/enterprise schemas. 2025 SOTA approaches use context-aware bidirectional retrieval and autonomous schema exploration (e.g., AutoLink reports ~97% strict linking recall on BIRD-dev, ~91% on Spider 2.0-Lite). Hybrid dense-vector + symbolic schema retrieval (Semantic-RAG, CSR-RAG) scales linking to enterprise schemas. [source]
Prompt patterns: provide schema (DDL), few-shot question→SQL exemplars, value/sample hints, and dialect notes; decompose complex questions; use RAG to retrieve schema fragments + similar verified queries. [source]
Correctness & self-correction: never trust SQL on syntax. Use execution-guided self-correction - run the SQL (or a dry-run/EXPLAIN), feed errors/empty-results back, and let the model repair (e.g., LitE-SQL: 72.1% EX on BIRD, 88.45% on Spider 1.0 via execution-guided correction without multi-candidate sampling). Majority-vote / consensus over candidates (ReFoRCE) filters unreliable outputs. [source]
Benchmarks & metrics: Spider (cross-domain) and BIRD (large, dirty, real-world DBs with efficiency scoring) are the classics; Spider 2.0 targets enterprise workflows (huge schemas, dialects, nested query plans) and is hard - frontier execution accuracy sits in the ~25-35% range vs. ~70%+ on BIRD-dev. Primary metrics: Execution Accuracy (EX) - does the result match the gold result - and the stricter, brittler Exact-Match (EM) on SQL text. Prefer EX; note that benchmark annotation errors are a known caveat (CIDR 2026 "Text-to-SQL Benchmarks are Broken"). [source]

4. Analytics agents (plan → query → analyze → narrate)

An analytics agent decomposes a goal into steps, calls tools (SQL, search, a sandboxed Python code interpreter), executes, reflects, and synthesizes a narrative. Pattern: select → aggregate → rank → explain, presented in plain language. [source]
Code interpreter / sandboxed Python: the agent writes and runs Python (pandas/plots) in an isolated sandbox to do analysis beyond SQL (stats, joins across sources, charts). Managed sandboxes (e.g., Amazon Bedrock AgentCore Code Interpreter) handle isolation/scaling; ReAct-style loops (LangGraph) drive write→execute→observe. [source]
Multi-agent specialization: planner / builder / critic / reflector agents make the final narrative more reliable (e.g., CoDA for collaborative visualization). [source]
Engineering guardrails: bound the number of tool calls and self-correction iterations, sandbox all code, scope DB credentials to the requesting user, and log every step for replay. [source]

5. Automated insight generation & NLG narratives

Diagnostic automation answers "what changed and why." Key-driver analysis decomposes a metric movement into contributing dimensions/segments (often shown as a waterfall), automatically ranking drivers. Anomaly detection surfaces unexpected movements proactively (Tableau Pulse's model). [source]
Significance-aware mining: only narrate insights that are statistically meaningful - guard against spurious "drivers" from multiple comparisons / small segments (ties to da-12 multiple-comparison discipline). [source]
NLG converts the result into a human-readable descriptive/diagnostic/prescriptive narrative attached to a chart or dashboard, making insight portable and actionable (heavy adoption in finance reporting). [source]

6. RAG over structured + unstructured analytical context (hybrid retrieval)

Analytical questions often need both numbers (in tables/warehouse) and context (in docs/metric definitions). Hybrid retrieval combines vector + keyword + metadata filtering: [source]
Route by question type: structured/aggregate questions → text-to-SQL against governed data (do not answer "what was Q3 revenue" from a vector store); definitional/context questions → vector retrieval over docs. [source]
RAG-to-SQL: retrieve schema fragments, FK relationships, column descriptions, and similar verified queries to improve schema linking and grounding (Semantic-RAG, CSR-RAG - ~80%+ recall at ~30ms on commodity hardware). [source]
Agentic RAG over long text in SQL tables handles documents stored alongside structured columns. [source]

7. Evaluation, trust & governance

LLM-produced numbers are a correctness problem, not just a fluency one. [source]
Hallucination types: faithfulness (output not grounded in the retrieved context/query result) vs factuality (wrong vs the real world). For analytics, faithfulness to the executed query result is the key bar - every number should trace to a query. [source]
Controls: execution-guided validation, verified-query repositories, self-consistency / consensus decoding, RAG grounding, span-level attribution (claim → source/query), and SelfCheckGPT-style inter-sample contradiction checks. [source]
Metric consistency: route metrics through the governed semantic layer so the same business term yields the same SQL every time (the anti-"metric sprawl" argument from da-18). [source]
Human-in-the-loop & auditability: keep humans verifying high-stakes answers; expose the generated SQL/query and lineage so analysts can audit how a number was produced; integrate agent observability (distributed tracing, span-level evaluators) to run quality checks on live traffic. [source]

Tooling landscape (2025-2026)

Choosing: if your data already lives in Snowflake → Cortex Analyst; Databricks lakehouse → Genie; Power BI/Microsoft → Copilot; want warehouse-independent NLQ → ThoughtSpot; building it yourself / embedding → Vanna or a LangChain/LlamaIndex SQL agent. [source]

Practical patterns

Ground before you generate. Put a governed semantic model / verified queries between the LLM and the warehouse. Raw-schema text-to-SQL is a demo, not a product. [source]
Execute to validate. Run (or EXPLAIN/dry-run) generated SQL, feed errors back, and self-correct. Prefer execution accuracy over trusting the text. [source]
Seed a verified-query repository. Curated question→SQL pairs are the single highest-ROI accuracy lever and double as regression tests. [source]
Route by question type. Aggregate/metric questions → SQL; definitional/context → vector RAG; complex/multi-source → agent with code interpreter. Don't answer numeric questions from a vector store. [source]
Bound the agent. Cap tool calls + self-correction loops, sandbox all code, scope credentials to the user, and log every step for replay/audit. [source]
Narrate only significant insights. Significance-test before NLG; rank drivers; show the waterfall and the supporting query. [source]
Make every number auditable. Surface the generated query + lineage; keep humans in the loop for high-stakes answers. [source]
Build an eval set. Golden NL→SQL→result triples; track EX (not just EM), faithfulness, and metric-consistency over time. [source]

Anti-patterns

Pointing text-to-SQL at a raw, ungoverned schema and expecting reliable answers - accuracy collapses without semantic grounding. [source]
Trusting SQL because it parses - syntactically valid SQL can return wrong or empty results; always execute-validate. [source]
Answering numeric questions via vector RAG - vector similarity does not aggregate; route to SQL. [source]
Narrating "drivers" without significance testing - manufactures spurious explanations from noise / multiple comparisons. [source]
Unbounded agent loops or unsandboxed code execution - cost blowups and security holes. [source]
Numbers with no audit trail - if you can't show the query and lineage behind a figure, it isn't trustworthy for decisions. [source]
Optimizing for Exact-Match - EM is brittle (many correct SQLs differ textually); optimize Execution Accuracy. [source]
One metric, many definitions - bypassing the semantic layer reintroduces metric sprawl. [source]

Troubleshooting

Wrong/empty results despite "good" SQL → schema-linking failure; add column descriptions, synonyms, sample values, and verified-query exemplars; check FK/join paths. [source]
Inconsistent numbers for the same question → route through the governed semantic layer; pin metric definitions; add the pair to the verified repository. [source]
Agent loops or runs up cost → cap iterations/tool calls; add a termination check; cache schema retrieval. [source]
Plausible-but-wrong narrative → faithfulness failure; require claims to cite the executed query result; add self-consistency / SelfCheckGPT-style checks; insert HITL for high stakes. [source]
Great on BIRD, fails in prod → enterprise schemas (Spider 2.0 regime) are far harder; invest in schema linking, dialect handling, and retrieval over the real catalog. [source]

References

Gartner, Augmented Analytics glossary & Market Guide for Agentic Analytics (2025) - https://www.gartner.com/en/information-technology/glossary/augmented-analytics ; https://www.gartner.com/en/newsroom/press-releases/2025-06-18-gartner-predicts-75-percent-of-analytics-content-to-use-genai-for-enhanced-contextual-intelligence-by-2027 [source]
Spider 2.0: Evaluating Language Models on Enterprise Text-to-SQL - https://openreview.net/pdf/a580c1b9fa846501c4bbf06e874bca1e2f3bc1d0.pdf [source]
AutoLink: Autonomous Schema Exploration for Scalable Schema Linking (2025) - https://arxiv.org/pdf/2511.17190 [source]
RSL-SQL: Robust Schema Linking in Text-to-SQL - https://arxiv.org/pdf/2411.00073 [source]
LitE-SQL: Lightweight Text-to-SQL with Execution-Guided Self-Correction - https://arxiv.org/pdf/2510.09014 [source]
ReFoRCE: A Text-to-SQL Agent - https://arxiv.org/pdf/2502.00675 [source]
Text-to-SQL Benchmarks are Broken (CIDR 2026) - https://www.vldb.org/cidrdb/papers/2026/p5-jin.pdf [source]
Semantic-RAG for Text-to-SQL - https://medium.com/@lbirjega/semantic-rag-for-text-to-sql-ed57fcdb0a45 ; CSR-RAG - https://arxiv.org/pdf/2601.06564 [source]
RAGFlow, From RAG to Context - 2025 year-end review - https://ragflow.io/blog/rag-review-2025-from-rag-to-context [source]
A review of faithfulness metrics for hallucination assessment in LLMs - https://arxiv.org/pdf/2501.00269 ; Faithfulness metric fusion - https://arxiv.org/pdf/2512.05700 [source]
Snowflake, Cortex Analyst docs + Agentic Semantic Model Improvement - https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst ; https://www.snowflake.com/en/blog/engineering/agentic-semantic-model-text-to-sql/ [source]
Databricks AI/BI Genie - https://zenlytic.com/blog/databricks-ai-bi-genie [source]
Tellius, Best AI Data Analysis Agents 2026 - https://www.tellius.com/resources/blog/best-ai-data-analysis-agents-in-2026-12-platforms-compared-for-nl-to-sql-autonomous-investigation-and-governance [source]
Simon Willison, Coding agents for data analysis (NICAR 2026) - https://simonw.github.io/nicar-2026-coding-agents/coding-agents.html [source]
AWS, Amazon Bedrock AgentCore Code Interpreter - https://aws.amazon.com/blogs/machine-learning/introducing-the-amazon-bedrock-agentcore-code-interpreter/ [source]
Vanna AI (RAG-powered text-to-SQL) - https://medium.com/mitb-for-all/text-to-sql-just-got-easier-meet-vanna-ai-your-rag-powered-sql-sidekick-e781c3ffb2c5 [source]
Diagnostic Analytics / key-driver - https://www.lumi-ai.com/analytics-101/diagnostic-analytics ; NLG for BI - https://automatedinsights.com/business-intelligence/ [source]

Related skills

da-18-semantic-layer-headless-bi - the governed semantic/metrics layer LLMs are grounded in. [source]
da-36-text-analytics-nlp - unstructured-text NLP (topics, sentiment, NER, embeddings). [source]
da-7-machine-learning - LLM landscape, training, and eval theory. [source]
rag-architecture - generic RAG pipeline design. [source]
da-8-data-visualization / da-9-reporting-communication - viz recommendation and reporting. [source]
da-12-ab-testing-causal-inference - significance discipline behind driver/diagnostic claims. [source]

Children

Augmented analytics and agentic analytics (Gartner) (frontier)
Conversational BI and NLQ (frontier)
Text-to-SQL (schema linking, self-correction, Spider/BIRD benchmarks) (frontier)
Analytics agents (plan-query-analyze-narrate, code interpreter) (frontier)
Automated insight generation and NLG narratives (frontier)
RAG over structured plus unstructured analytical context (hybrid SQL+vector) (frontier)
Evaluation, trust and governance for LLM-produced numbers (frontier)

Frontier under this node: Analytics agents (plan-query-analyze-narrate, code interpreter), Augmented analytics and agentic analytics (Gartner), Automated insight generation and NLG narratives, Conversational BI and NLQ, Evaluation, trust and governance for LLM-produced numbers, RAG over structured plus unstructured analytical context (hybrid SQL+vector), Text-to-SQL (schema linking, self-correction, Spider/BIRD benchmarks)

← the whole tree · 3D view· how to read this page