MongoDB Aggregation Pipeline

Parent: MongoDB Expert Knowledge · researched 2026-05-28T15:12:34.482Z· 11 sources · 10 concepts · skill mongodb-aggregation-pipeline

Stages execute left-to-right; each stage receives the document stream from the previous

1. Pipeline Stages Reference

Stages execute left-to-right; each stage receives the document stream from the previous [source]
stage. Order matters enormously for performance (see Section 4). [source]

2a. Equality join (localField / foreignField)

MongoDB performs a hash-lookup on the foreign collection for each input document. [source]
Always index the foreignField - without an index MongoDB scans the entire [source]
foreign collection per input document (an O(N×M) table scan). [source]
One important footgun: if the as field name already exists on the input document, [source]
it is silently overwritten. Choose an as name that does not collide. [source]
Required index on the foreign side: [source]

2b. Correlated sub-query (let + pipeline)

Use when you need filtering beyond a simple equality, or when joining on multiple fields. [source]
let binds local document variables; reference them inside the sub-pipeline with $$varName. [source]

2c. Performance checklist for $lookup

Index every foreignField (or the first field in the sub-pipeline $match). [source]
Place $match inside the sub-pipeline to push filtering before the join materialises. [source]
Avoid interleaving $unwind → $lookup → $unwind; chain all lookups, then unwind. [source]
On sharded clusters, joining a sharded foreign collection is supported (MongoDB 5.1+), [source]
but each input document fans out to every shard holding the foreign collection [source]
(scatter-gather). Co-locate the collections on the same shard key, or use Atlas Data [source]
Federation, to avoid the fan-out. [source]

3a. $out — full collection replacement

$out atomically replaces the target collection after the full pipeline completes. [source]
Use for nightly full rebuilds where brief stale reads are acceptable. [source]
Target must be in the same database unless using { db, coll } object form (4.4+). [source]
Reads see the old collection until the atomic swap completes. [source]

3b. $merge — incremental / upsert update

$merge (introduced in 4.2) writes into an existing collection with [source]
per-document conflict control. Ideal for incrementally updating materialized views. [source]
whenMatched options: "replace" | "merge" | "keepExisting" | "fail" | [pipeline] [source]
whenNotMatched options: "insert" | "discard" | "fail" [source]
The [pipeline] form for whenMatched allows complex update logic: [source]

4a. Place $match (and $sort) early

The query planner can push a leading $match into the collection scan using an index. [source]
A $match + $sort sharing an index prefix eliminates the in-memory sort stage entirely. [source]

4b. Project down early to shrink per-document size

Large documents amplify memory usage across every downstream stage. [source]

5a. $expr in $match — field-to-field comparisons

$expr unlocks aggregation operators inside $match. An index can still be used when [source]
the expression references an indexed field at the top level. [source]

5d. $accumulator — custom group-level accumulation

$accumulator (4.4+) lets you write a fully custom accumulator using JavaScript. [source]
Use only when no native accumulator fits - it is significantly slower than built-ins [source]
and requires javascriptEnabled: true. [source]

5e. $function — inline custom JavaScript per document

Runs arbitrary JS inside mongod per document (not per group). Slower than $accumulator [source]
for grouped work; avoid in hot paths. [source]

6. Window Functions ($setWindowFields)

Introduced in MongoDB 5.0. Analogous to SQL OVER (PARTITION BY … ORDER BY …). [source]
Does not reduce the document count (unlike $group). [source]

7a. $densify — fill temporal gaps

$densify (5.1+) inserts synthetic documents for missing date/numeric values. [source]
Combine with $fill (5.3+) to interpolate values into those synthetic docs. [source]

7c. Native time series collections

With timeseries collection type (5.0+), $match on timeField or metaField [source]
pushes down into bucket metadata - MongoDB skips entire buckets without unpacking them. [source]

9a. Default per-stage limit

Each pipeline stage is limited to 100 MB of RAM (raised from 32 MB in 4.4). [source]
Stages most likely to hit this: $sort, $group, $bucket, $setWindowFields. [source]
When the limit is exceeded, MongoDB aborts with: [source]

9b. allowDiskUse

Spilled data is written to <dbPath>/_tmp. Increases latency; fix root cause first. [source]
Atlas tiers: allowDiskUse is disabled on M0/M2/M5 (shared) tiers; requires M10+. [source]

10c. Java — MongoDB Driver 5.x (sync)

The raw Document API works but is verbose. The idiomatic 5.x approach uses the [source]
Aggregates and Filters builder classes for type-safety and readability: [source]

References

Aggregation Pipeline Stages - MongoDB Manual [source]
$lookup (aggregation) - MongoDB Manual [source]
$merge (aggregation stage) - MongoDB Manual [source]
$setWindowFields - MongoDB Manual [source]
$densify (aggregation stage) - MongoDB Manual [source]
$fill (aggregation stage) - MongoDB Manual [source]
Aggregation Pipeline Optimization - MongoDB Manual [source]
Explain Results - MongoDB Manual [source]
Time Series Collections - MongoDB Manual [source]
$accumulator - MongoDB Manual [source]
Aggregation with the Java Driver - MongoDB Docs [source]

Time Series Collection Aggregation Notes

When running aggregation pipelines against time series collections (MongoDB 5.0+), the following behaviors differ from regular collections: [source]
Bucket-level pruning: The query planner uses control.min/control.max metadata on internal buckets to skip entire buckets that don't match time-range or metaField predicates. Always place $match on the metaField and timeField as the first stage to maximise pruning. [source]
$densify / $fill on time series: $densify partitionByFields supports dotted paths into metaField sub-fields but not measurement fields. For measurement field partitioning, use $addFields to promote the field before $densify. [source]
$setWindowFields performance: Window functions do not push down through bucket storage. A tight $match before $setWindowFields is critical - without it, MongoDB unpacks and scans all buckets. [source]
$dateTrunc for downsampling: Use $dateTrunc with binSize to downsample raw measurements into fixed time buckets (hourly/daily OHLCV, hourly averages). It is more efficient than $dateToString + $group for time-bucket aggregations. [source]
$out to time series (MongoDB 7.0.3+): $out can write directly into a time series collection. $merge into a time series collection is not supported - use $out instead. [source]
Cannot use distinct() on time series - use $group with a supporting metaField compound index instead. [source]
For full time series aggregation patterns including IoT multi-sensor, financial OHLCV, gap-fill dashboards, and working set sizing, see mongodb-time-series. [source]

Children

Pipeline Stages Reference (frontier)
$lookup Patterns (frontier)
$merge and $out Materialized Views (frontier)
Pipeline Optimization (frontier)
Aggregation Expressions (frontier)
Window Functions ($setWindowFields) (frontier)
Time Series Aggregation (frontier)
Anti-Patterns (frontier)
Memory Limits and allowDiskUse (frontier)
Driver Examples (Node.js, Python, Java) (frontier)
mongodb-time-series

Frontier under this node: $lookup Patterns, $merge and $out Materialized Views, Aggregation Expressions, Driver Examples (Node.js, Python, Java), Memory Limits and allowDiskUse, Pipeline Optimization, Pipeline Stages Reference, Time Series Aggregation, Window Functions ($setWindowFields)

← the whole tree · 3D view· how to read this page