Real-Time OLAP and Analytical Databases

Parent: Data Analysis · Published reference · snapshot 2026-09-08 · skill da-data-engineering-platform/references/da-28-realtime-olap-databases.md

↓ Facts as markdown all context files

Real-time OLAP databases are the query and storage engines that answer

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Overview

Real-time OLAP databases are the query and storage engines that answer analytical questions (aggregations, group-bys, filters, top-N, time-series rollups) over large, continuously updating datasets with sub-second latency and high concurrency. They sit between stream processing (which transforms events in flight - da-14) and the BI/semantic layer (da-18), serving as the low-latency serving layer for dashboards, monitoring, and user/customer-facing analytics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]
What makes an engine "real-time OLAP" rather than a cloud data warehouse: [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]
- Fresh data: rows are queryable seconds (often sub-second) after they land, via streaming ingestion - not after a nightly batch load. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]
- High concurrency / QPS: built to serve thousands of concurrent queries (a per-user dashboard feature), not a handful of internal analysts. ClickHouse reports 1,000+ concurrent queries per node; Snowflake defaults to ~8 queries per warehouse, Redshift caps ~50 concurrent across queues (ClickHouse, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]
- Tight latency SLAs: tens of milliseconds, achieved with columnar storage, vectorized execution, and pre-aggregation/indexing. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]
This skill covers the engines and the query layer. It is the OLAP-database node of the data-analytics curriculum (da-1 onward); it does not cover stream processing (da-14), pipeline orchestration (da-13), or the metrics layer (da-18). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#overview]

1. Columnar storage

Data is stored by column, not by row. Analytical queries touch few columns but many rows, so columnar layout reads only the needed columns, drastically cutting I/O, and stores like-typed values together so they compress far better (delta, dictionary, RLE, LZ4/ZSTD). This is the foundational OLAP advantage over row stores (SQLFlash, 2025; Airbyte, 2024). Compression both saves storage and increases effective scan throughput. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#1-columnar-storage]

2. Vectorized execution

Instead of processing one row at a time through a tuple-at-a-time interpreter, the engine processes batches (vectors) of column values in tight loops, exploiting CPU SIMD instructions, cache locality, and amortized virtual-function/branch overhead. This yields multi-x to order-of-magnitude speedups and is paired with columnar storage in every modern engine (SQLFlash, 2025; Cockroach Labs). Even hybrid OLTP/OLAP systems (Google Spanner's columnar engine, OceanBase 4.3) adopt columnar + vectorized execution for analytics, up to ~200x faster on live data (InfoQ, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#2-vectorized-execution]

3. Real-time vs batch analytics (Lambda / Kappa)

Batch: data loaded and computed periodically; high latency, high accuracy, cheap recompute (classic warehouse / BI pattern). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#3-real-time-vs-batch-analytics-lambda-kappa]
Real-time: data queryable seconds after arrival; low latency, continuous ingestion. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#3-real-time-vs-batch-analytics-lambda-kappa]
Lambda keeps two paths: a batch layer for correctness and a speed layer for low latency, at the cost of dual code and reconciliation. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#3-real-time-vs-batch-analytics-lambda-kappa]
Kappa collapses everything into a single streaming path over an immutable log (e.g., Kafka), replaying when recompute is needed; simpler but harder for large historical batch jobs (bix-tech, 2025; Materialize). Most 2025 teams run a hybrid: streaming for operational decisions, batch for trusted reporting and model training (makitsol, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#3-real-time-vs-batch-analytics-lambda-kappa]

4. Streaming ingestion & upserts

Real-time OLAP engines ingest directly from Kafka, Pulsar, and Kinesis. Apache Pinot transforms bytes from Kafka into queryable segments with sub-second visibility from write to query as the default; end-to-end latency under 5s is normal in production (StarTree, 2025; Confluent). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#4-streaming-ingestion-upserts]
- Append-only is the simplest model (events never change). Pinot was append-only until 2022. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#4-streaming-ingestion-upserts]
- Upserts let the same key be ingested many times but return only the latest value at query time. Pinot supports full upsert (new row replaces the old entirely) and partial upsert (only specified columns update) (Pinot deep dive, 2026). StarRocks uses a primary-key table model (cloud-native PK index in shared-data) for upserts and CDC-style mutable data (StarRocks, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#4-streaming-ingestion-upserts]

5. Materialized views & pre-aggregation

Precompute aggregates so dashboard queries hit ready answers instead of scanning raw rows. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#5-materialized-views-pre-aggregation]
- Pinot star-tree index: precomputes selected aggregation paths during segment generation; returns aggregation/group-by over billions of rows in milliseconds when the query shape matches configured dimensions/metrics, with no separate MV maintenance. Benchmarks show ~126x throughput over an inverted index (27 to 3,494 QPS on 4 vCPU) (Pinot docs; StarTree, 2023). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#5-materialized-views-pre-aggregation]
- StarRocks / Doris asynchronous materialized views: precompute joins and aggregations; the optimizer automatically rewrites base-table queries to use the MV (transparent query rewrite). Can be built over external lake catalogs (Iceberg/Hive/Hudi/Paimon) (StarRocks docs; Doris docs). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#5-materialized-views-pre-aggregation]
- ClickHouse materialized views / projections: MVs are insert-time triggers that populate a target table (often with AggregatingMergeTree); projections store an alternate sorted/aggregated copy inside the part. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#5-materialized-views-pre-aggregation]

6. Indexing & the ClickHouse MergeTree model

ClickHouse's MergeTree family writes each INSERT as an immutable data part (one file per column + index), merged in the background. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#6-indexing-the-clickhouse-mergetree-model]
- Sparse primary index: one mark per granule (default 8,192 rows), stored in primary.idx; it does not enforce uniqueness, it lets the engine skip granules that cannot match a filter (ClickHouse docs). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#6-indexing-the-clickhouse-mergetree-model]
- Data-skipping indexes (minmax, set(N), bloom_filter, ngrambf_v1, tokenbf_v1) summarize non-key columns so granules can be skipped on secondary predicates; tune GRANULARITY and materialize after adding to existing data (oneuptime, 2026). Druid instead indexes within each segment (300-700 MB target) including inverted (bitmap) indexes for fast filtering (Druid docs). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#6-indexing-the-clickhouse-mergetree-model]

7. Star-schema-on-OLAP & denormalization

Two camps: (a) flatten/denormalize into one wide table for fastest scans (historically favored by Druid/ClickHouse, which are weaker at large joins); (b) keep the star/snowflake schema and join at query time. StarRocks explicitly preserves star/snowflake schemas and does real-time pre-processing at load, with a strong distributed join engine, so you avoid the maintenance burden of giant denormalized tables (StarRocks features; jusdb, 2026). Rule of thumb: denormalize when joins dominate latency and the engine joins poorly; keep the star schema when the engine joins well and dimensions change. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#7-star-schema-on-olap-denormalization]

8. Storage-compute separation / shared-data / tiered storage

Modern engines decouple cheap durable object storage (S3/GCS/Azure Blob) from elastic compute, mirroring cloud-DW architecture but for real-time workloads. StarRocks 3.0+ shared-data mode replaces storage-bearing backends with compute nodes (CN) that cache hot data and read cold data from S3, giving elastic scaling; 4.0 cut object-store API costs and reached 15-30s data freshness in this mode (StarRocks architecture; Medium/Ding, 2025). Tiered storage (hot local SSD to warm/cold object store) is now standard across ClickHouse, Druid, Pinot, and StarRocks. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#8-storage-compute-separation-shared-data-tiered-storage]

9. High-QPS user-facing / customer-facing analytics

"User-facing analytics" embeds analytics in the product, exposed to end users, so every user gets personalized metrics, producing hundreds of thousands of QPS rather than a few analyst sessions (Pinot). This is the workload real-time OLAP engines exist for and where cloud DWs fail on concurrency/cost. Pinot has served 20,000+ QPS at sub-second p99 with 99.99% availability via star-tree pre-aggregation; ClickHouse powers a customer-facing feature and a BI dashboard from one service (StarTree; ClickHouse, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#9-high-qps-user-facing-customer-facing-analytics]

Tools / Frameworks

Benchmarks (treat as directional, vendor-published): StarRocks reports ClickHouse ~2.2x and Druid ~8.9x slower on 13 SSB flat-table queries; Pinot reported 2-4x faster than Druid on some queries (StarRocks; StarTree). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#tools-frameworks]

Methodology — choosing & designing

Classify the workload. Internal batch BI / ad-hoc to cloud DW (Snowflake/ BigQuery/Redshift) or DuckDB for single-node. User-facing / sub-second / high QPS / streaming-fresh to a real-time OLAP engine. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Match engine to query shape. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
- Massive single-table aggregations, want simplicity: ClickHouse. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
- Per-user dashboards, very high QPS, fixed query shapes: Pinot (star-tree). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
- Interactive time-series exploration with high concurrency: Druid. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
- Real-time analytics needing joins on a star schema, lakehouse: StarRocks/Doris. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
- Embedded/local, no server, Parquet on disk: DuckDB. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Decide the freshness path (Kappa-style streaming vs hybrid Lambda) and the mutability model (append-only vs full/partial upsert). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Model the schema: denormalize for join-weak engines; keep star schema for StarRocks/Doris with strong join engines. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Pre-aggregate intentionally: star-tree (Pinot) or async MVs (StarRocks/ Doris) or MV+projections (ClickHouse) for the known dashboard query shapes. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Tune indexing: sort key / primary index ordered by your most selective filter; add data-skipping/bitmap indexes for secondary predicates. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]
Right-size storage: separate storage/compute (shared-data) and tier hot to cold to control cost at scale. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#methodology-choosing-designing]

Practical Patterns

Kafka to real-time OLAP serving layer: stream events to Kafka, ingest into Pinot/Druid/ClickHouse for sub-5s freshness, serve the product dashboard directly from the engine. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#practical-patterns]
CDC upserts: stream Postgres/MySQL changes (Debezium/ClickPipes) into a PK/upsert table so the OLAP store mirrors mutable source state. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#practical-patterns]
Query-shape-driven pre-aggregation: enumerate the dimensions/metrics your dashboards actually use, then build a matching star-tree or async MV; do not pre-aggregate everything. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#practical-patterns]
Two-tier serving: cloud DW for deep batch/historical + real-time OLAP engine for the hot, user-facing layer; sync via Iceberg or scheduled exports. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#practical-patterns]
Sort by the dominant filter: order the table by the column(s) most queries filter/range on (e.g., (tenant_id, timestamp)) so the sparse index skips the most data. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#practical-patterns]

Anti-Patterns

Using a cloud DW for user-facing analytics: concurrency caps (Snowflake ~8, Redshift ~50) and per-query cost make per-user dashboards slow and expensive (ClickHouse, 2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Real-time OLAP for OLTP: these engines are not for point updates/deletes, transactions, or single-row lookups by a service of record. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Pre-aggregating for query shapes you do not run: star-trees and MVs cost storage and ingestion CPU; build them for real query shapes only. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Over-indexing ClickHouse: too many data-skipping indexes slow inserts and rarely help; they also do not help with negations (oneuptime, 2026). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Expecting DuckDB to be a streaming/high-concurrency server: it is in-process, single-node, batch/interactive, not a serving engine (Kestra, 2026). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Many tiny inserts into MergeTree: floods the engine with small parts; batch inserts (or use async inserts) so merges keep up. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]
Treating vendor benchmarks as neutral: SSB/flat-table benchmarks favor the publisher; validate on your own query shapes and data. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#anti-patterns]

Troubleshooting

Slow aggregation / group-by: confirm pre-aggregation matches the query shape (star-tree dims/metrics, MV grouping keys); check vectorized path is used. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
High latency under concurrency: check QPS vs node count; add star-tree/MV; scale compute (shared-data CNs) horizontally. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
Stale data: inspect ingestion lag (Kafka consumer lag, segment commit/ handoff in Pinot/Druid, publish batching/freshness in StarRocks shared-data). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
Query scans too much data: primary/sort key not aligned to the dominant filter, or missing data-skipping/bitmap index; reorder sort key. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
Insert pressure / "too many parts" (ClickHouse): inserts too small/frequent; batch them; let background merges catch up. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
Upsert results look wrong: verify full vs partial upsert semantics and that the upsert primary key and partitioning are configured correctly. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]
Costs spiking on object store (shared-data): API call volume; enable batch publish / caching (StarRocks 4.0 addressed this) and size the local hot cache. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#troubleshooting]

References

ClickHouse - How the 5 major cloud data warehouses compare on cost-performance (2025): https://clickhouse.com/blog/cloud-data-warehouses-cost-performance-comparison [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
ClickHouse - ClickHouse vs Snowflake for Real-Time Analytics (2025): https://clickhouse.com/blog/clickhouse-vs-snowflake-for-real-time-analytics-comparison-migration-guide [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
ClickHouse docs - A practical introduction to primary indexes (sparse index/granules): https://clickhouse.com/docs/guides/best-practices/sparse-primary-indexes [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
oneuptime - Data Skipping with Sparse Indexes in ClickHouse (2026): https://oneuptime.com/blog/post/2026-03-31-clickhouse-data-skipping-sparse-indexes/view [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
oneuptime - Avoid Over-Indexing in ClickHouse (2026): https://oneuptime.com/blog/post/2026-03-31-clickhouse-avoid-over-indexing/view [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarTree - Inside the flight path of real-time ingestion in Apache Pinot (2025): https://startree.ai/resources/inside-the-flight-path-of-real-time-ingestion-in-apache-pinot/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarTree - A Tale of Three Real-Time OLAP Databases (Pinot/Druid/ClickHouse): https://startree.ai/resources/a-tale-of-three-real-time-olap-databases/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarTree - Star-Tree Index Part 2: High Concurrency (2023): https://startree.ai/resources/star-tree-indexes-in-apache-pinot-part-2-understanding-the-impact-during-high-concurrency/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Apache Pinot docs - Star-Tree Index: https://docs.pinot.apache.org/build-with-pinot/indexing/star-tree-index [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
pdpspectra - Apache Pinot Deep Dive 2026: User-Facing Analytics, Upserts (2026): https://pdpspectra.com/blog/apache-pinot-realtime-olap-2026/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Confluent - Real-Time Analytics with Apache Kafka and Pinot: https://www.confluent.io/blog/real-time-analytics-with-kafka-and-pinot/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarRocks - Benchmark: StarRocks vs ClickHouse, Druid, Trino: https://www.starrocks.io/blog/benchmark-test [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarRocks docs - Query rewrite with materialized views: https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
StarRocks docs - Architecture (shared-data / compute nodes): https://docs.starrocks.io/docs/introduction/Architecture/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
jusdb - StarRocks Database (2026): Architecture & Real-Time Analytics Guide (2026): https://www.jusdb.com/blog/starrocks-explained-the-complete-guide-to-real-time-analytics [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Apache Doris docs - Overview of Asynchronous Materialized Views: https://doris.apache.org/docs/query-acceleration/materialized-view/async-materialized-view/overview/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Apache Druid docs - Segments: https://druid.apache.org/docs/latest/design/segments/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Apache Druid docs - Ingestion: https://druid.apache.org/docs/latest/ingestion/index.html [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
SQLFlash - OLAP Database Architecture: Columnar Storage & Vectorized Execution (2025): https://sqlflash.ai/article/20250722_olap-database-architecture/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
InfoQ - Google Spanner Unifies OLTP and OLAP with Columnar Engine (2025): https://www.infoq.com/news/2025/09/google-spanner-oltp-olap-unify/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
bix-tech - Kappa vs. Lambda vs. Batch: https://bix-tech.com/kappa-vs-lambda-vs-batch-choosing-the-right-data-architecture-for-your-business/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Materialize - When Is Kappa Architecture Most Effective?: https://materialize.com/blog/when-is-kappa-architecture-most-effective/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
makitsol - Real-Time Analytics vs Batch Processing (2025): https://makitsol.com/real-time-analytics-vs-batch-processing-in-us-eu/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Kestra - Embedded Databases in 2026: DuckDB, SQLite, Polars, chDB (2026): https://kestra.io/blogs/embedded-databases [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Tinybird - OLAP databases: what's new and what's best in 2026 (2026): https://www.tinybird.co/blog/best-database-for-olap [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
Estuary - Top 10 Real-Time OLAP Databases in 2026 (2026): https://estuary.dev/blog/real-time-olap-databases/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]
pracdata - State of Open Source Real-Time OLAP Systems 2025 (2025): https://www.pracdata.io/p/state-of-open-source-read-time-olap-2025 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-28-realtime-olap-databases/#references]

Where this helps

Powering a customer-facing analytics dashboard that needs sub-second aggregate queries over billions of rows, where a traditional OLTP database or a batch warehouse can't meet the latency requirement. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Ingesting high-velocity event streams — clickstream, IoT, logs — continuously while keeping the data queryable in near real time, instead of waiting on a nightly batch load. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Choosing between a Lambda architecture (separate batch and speed layers reconciled later) and a Kappa architecture (a single streaming pipeline) for a given freshness requirement. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Designing pre-aggregated materialized views to serve a known set of dashboard queries fast, when computing them from raw fact tables on every request is too slow. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Deciding on storage-compute separation and tiered storage when years of historical data need to stay queryable without paying full hot-storage cost for all of it. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Project ideas

Stand up a ClickHouse (or similar MergeTree-based) instance, load a large event dataset, and compare columnar scan performance against a row-store baseline for a typical aggregate query. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build a streaming ingestion pipeline that upserts into a real-time OLAP table and measure end-to-end freshness from event time to queryable time. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Design a set of materialized views for a specific dashboard's known query patterns and measure the latency improvement over querying the raw fact table directly. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Prototype a customer-facing usage-analytics widget backed by a real-time OLAP store instead of a pre-computed nightly batch job. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Denormalizing everything into one wide table without considering the engine's sort-key and partition design, which determines whether common queries can actually skip data efficiently. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Treating a real-time OLAP database like an OLTP system, running frequent single-row updates or deletes against engines optimized for bulk columnar scans and append-heavy ingestion. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Over-relying on materialized views for every query pattern, multiplying storage and maintenance cost and creating staleness risk if the view refresh lags the source data. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Choosing a Lambda architecture's added complexity when the freshness requirement doesn't actually justify it, when a simpler Kappa-style streaming pipeline would do. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

Columnar, MergeTree-style engines are optimized for append-and-scan workloads; frequent point updates or deletes are comparatively expensive, often implemented as background merges rather than in-place writes. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Real-time ingestion with upserts adds write-path complexity — deduplication, ordering, late-arriving data — a purely batch pipeline doesn't have to solve. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Storage-compute separation reduces cost for cold data but adds query-time latency on cache-miss reads, so tiering decisions trade cost against worst-case query speed. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Vectorized execution gains the most speedup on numeric, columnar scans — highly relational, join-heavy workloads with many small lookups benefit less and can still bottleneck on join strategy. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

Columnar storage (frontier)
Vectorized (SIMD) execution (frontier)
Real-time vs batch analytics (Lambda/Kappa) (frontier)
Streaming ingestion and upserts (Kafka/Pulsar/Kinesis) (frontier)
Materialized views and pre-aggregation (frontier)
ClickHouse MergeTree indexing (sparse primary index, data-skipping) (frontier)
Star-schema-on-OLAP and denormalization (frontier)
Storage-compute separation / shared-data / tiered storage (frontier)
High-QPS user-facing analytics (frontier)
OLAP engines vs cloud data warehouses (frontier)
Embedded OLAP (DuckDB) (frontier)

Frontier under this node: ClickHouse MergeTree indexing (sparse primary index, data-skipping), Columnar storage, Embedded OLAP (DuckDB), High-QPS user-facing analytics, Materialized views and pre-aggregation, OLAP engines vs cloud data warehouses, Real-time vs batch analytics (Lambda/Kappa), Star-schema-on-OLAP and denormalization, Storage-compute separation / shared-data / tiered storage, Streaming ingestion and upserts (Kafka/Pulsar/Kinesis), Vectorized (SIMD) execution

← the whole tree · 3D view· how to read this page