MongoDB Aggregation Pipeline

Parent: MongoDB Expert Knowledge · Published reference · snapshot 2026-09-08 · skill mongodb-expert/references/mongodb-aggregation-pipeline.md

↓ Facts as markdown all context files

Stages execute left-to-right; each stage receives the document stream from the previous

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

1. Pipeline Stages Reference

Stages execute left-to-right; each stage receives the document stream from the previous stage. Order matters enormously for performance (see Section 4). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#1-pipeline-stages-reference]

2a. Equality join (localField / foreignField)

MongoDB performs a hash-lookup on the foreign collection for each input document. Always index the foreignField - without an index MongoDB scans the entire foreign collection per input document (an O(N×M) table scan). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2a-equality-join-localfield-foreignfield]
One important footgun: if the as field name already exists on the input document, it is silently overwritten. Choose an as name that does not collide. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2a-equality-join-localfield-foreignfield]
Required index on the foreign side: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2a-equality-join-localfield-foreignfield]

2b. Correlated sub-query (let + pipeline)

Use when you need filtering beyond a simple equality, or when joining on multiple fields. let binds local document variables; reference them inside the sub-pipeline with $$varName. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2b-correlated-sub-query-let-pipeline]

2c. Performance checklist for $lookup

Index every foreignField (or the first field in the sub-pipeline $match). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2c-performance-checklist-for-lookup]
Place $match inside the sub-pipeline to push filtering before the join materialises. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2c-performance-checklist-for-lookup]
Avoid interleaving $unwind → $lookup → $unwind; chain all lookups, then unwind. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2c-performance-checklist-for-lookup]
On sharded clusters, joining a sharded foreign collection is supported (MongoDB 5.1+), but each input document fans out to every shard holding the foreign collection (scatter-gather). Co-locate the collections on the same shard key, or use Atlas Data Federation, to avoid the fan-out. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#2c-performance-checklist-for-lookup]

3a. $out — full collection replacement

$out atomically replaces the target collection after the full pipeline completes. Use for nightly full rebuilds where brief stale reads are acceptable. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3a-out-full-collection-replacement]
- Target must be in the same database unless using { db, coll } object form (4.4+). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3a-out-full-collection-replacement]
- Reads see the old collection until the atomic swap completes. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3a-out-full-collection-replacement]

3b. $merge — incremental / upsert update

$merge (introduced in 4.2) writes into an existing collection with per-document conflict control. Ideal for incrementally updating materialized views. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3b-merge-incremental-upsert-update]
whenMatched options: "replace" | "merge" | "keepExisting" | "fail" | [pipeline] whenNotMatched options: "insert" | "discard" | "fail" [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3b-merge-incremental-upsert-update]
The [pipeline] form for whenMatched allows complex update logic: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#3b-merge-incremental-upsert-update]

4a. Place $match (and $sort) early

The query planner can push a leading $match into the collection scan using an index. A $match + $sort sharing an index prefix eliminates the in-memory sort stage entirely. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#4a-place-match-and-sort-early]

4b. Project down early to shrink per-document size

Large documents amplify memory usage across every downstream stage. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#4b-project-down-early-to-shrink-per-document-size]

5a. $expr in $match — field-to-field comparisons

$expr unlocks aggregation operators inside $match. An index can still be used when the expression references an indexed field at the top level. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#5a-expr-in-match-field-to-field-comparisons]

5d. $accumulator — custom group-level accumulation

$accumulator (4.4+) lets you write a fully custom accumulator using JavaScript. Use only when no native accumulator fits - it is significantly slower than built-ins and requires javascriptEnabled: true. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#5d-accumulator-custom-group-level-accumulation]

5e. $function — inline custom JavaScript per document

Runs arbitrary JS inside mongod per document (not per group). Slower than $accumulator for grouped work; avoid in hot paths. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#5e-function-inline-custom-javascript-per-document]

6. Window Functions ($setWindowFields)

Introduced in MongoDB 5.0. Analogous to SQL OVER (PARTITION BY … ORDER BY …). Does not reduce the document count (unlike $group). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#6-window-functions-setwindowfields]

7a. $densify — fill temporal gaps

$densify (5.1+) inserts synthetic documents for missing date/numeric values. Combine with $fill (5.3+) to interpolate values into those synthetic docs. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#7a-densify-fill-temporal-gaps]

7c. Native time series collections

With timeseries collection type (5.0+), $match on timeField or metaField pushes down into bucket metadata - MongoDB skips entire buckets without unpacking them. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#7c-native-time-series-collections]

9a. Default per-stage limit

Each pipeline stage is limited to 100 MB of RAM (raised from 32 MB in 4.4). Stages most likely to hit this: $sort, $group, $bucket, $setWindowFields. When the limit is exceeded, MongoDB aborts with: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#9a-default-per-stage-limit]

9b. allowDiskUse

Spilled data is written to <dbPath>/_tmp. Increases latency; fix root cause first. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#9b-allowdiskuse]
Atlas tiers: allowDiskUse is disabled on M0/M2/M5 (shared) tiers; requires M10+. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#9b-allowdiskuse]

10c. Java — MongoDB Driver 5.x (sync)

The raw Document API works but is verbose. The idiomatic 5.x approach uses the Aggregates and Filters builder classes for type-safety and readability: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#10c-java-mongodb-driver-5x-sync]

References

Aggregation Pipeline Stages - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$lookup (aggregation) - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$merge (aggregation stage) - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$setWindowFields - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$densify (aggregation stage) - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$fill (aggregation stage) - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
Aggregation Pipeline Optimization - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
Explain Results - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
Time Series Collections - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
$accumulator - MongoDB Manual [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]
Aggregation with the Java Driver - MongoDB Docs [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#references]

Time Series Collection Aggregation Notes

When running aggregation pipelines against time series collections (MongoDB 5.0+), the following behaviors differ from regular collections: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
Bucket-level pruning: The query planner uses control.min/control.max metadata on internal buckets to skip entire buckets that don't match time-range or metaField predicates. Always place $match on the metaField and timeField as the first stage to maximise pruning. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
$densify / $fill on time series: $densify partitionByFields supports dotted paths into metaField sub-fields but not measurement fields. For measurement field partitioning, use $addFields to promote the field before $densify. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
$setWindowFields performance: Window functions do not push down through bucket storage. A tight $match before $setWindowFields is critical - without it, MongoDB unpacks and scans all buckets. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
$dateTrunc for downsampling: Use $dateTrunc with binSize to downsample raw measurements into fixed time buckets (hourly/daily OHLCV, hourly averages). It is more efficient than $dateToString + $group for time-bucket aggregations. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
$out to time series (MongoDB 7.0.3+): $out can write directly into a time series collection. $merge into a time series collection is not supported - use $out instead. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
Cannot use distinct() on time series - use $group with a supporting metaField compound index instead. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]
For full time series aggregation patterns including IoT multi-sensor, financial OHLCV, gap-fill dashboards, and working set sizing, see mongodb-time-series. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-aggregation-pipeline/#time-series-collection-aggregation-notes]

Project ideas

Build a nightly reporting pipeline using $out to materialize a denormalized "orders with customer + product" view for a BI dashboard, refreshed via a scheduled job. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build an incremental analytics rollup ($merge) that updates hourly aggregate counters (daily active users, revenue by region) without re-scanning history each run. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Prototype a change-stream-to-search-index sync: reshape documents with an aggregation before pushing them into Atlas Search. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build a time-series downsampling job using $densify + $fill + $dateTrunc to backfill gaps in sensor data into hourly OHLCV-style buckets. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Where this helps

Denormalizing across collections at query time via $lookup, when write-side normalization matters more than read-side embedding. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Materializing a reporting or analytics view too expensive to compute per request ($out for full rebuilds, $merge for incremental updates). [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Computing running totals, rankings, or moving averages over ordered data ($setWindowFields) without pulling everything into application code. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Filling gaps in irregular time-series data for charting or trend detection ($densify/$fill). [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Running $lookup on an unindexed foreignField — turns an O(1) hash-lookup per document into a collection scan per document. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Interleaving $unwind -> $lookup -> $unwind instead of chaining lookups first and unwinding after — needlessly multiplies intermediate document count. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Reaching for $function or a custom $accumulator for logic a native operator already covers — $function runs arbitrary JS per document inside mongod and is drastically slower. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Placing $match or $project late in the pipeline instead of as early as possible — misses index pushdown and carries full document size through every downstream stage. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Choosing $merge for a full nightly rebuild (should be $out) or $out for incremental updates (should be $merge) — the wrong tool for the job's actual write pattern. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

Each stage defaults to a 100MB RAM limit; $sort/$group/$bucket/$setWindowFields hit it first on large datasets. allowDiskUse trades latency for headroom but is disabled on Atlas M0/M2/M5 shared tiers (needs M10+). [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
$lookup silently overwrites the `as` field name if it already exists on the input document — no error, just data loss. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
$merge into a time series collection is not supported (only $out, as of MongoDB 7.0.3+) — a $merge-shaped pipeline needs restructuring for a time series target. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Sharded $lookup (5.1+) fans each input document out to every shard holding the foreign collection — a hidden cross-shard-fanout cost that may not show up until data grows. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

$merge and $out Materialized Views
mongodb-aggregation-stages-deep
Pipeline Stages Reference (frontier)
Pipeline Optimization (frontier)
Aggregation Expressions
Window Functions ($setWindowFields) (frontier)
Time Series Aggregation (frontier)
Anti-Patterns
Memory Limits and allowDiskUse (frontier)
Driver Examples (Node.js, Python, Java) (frontier)
mongodb-time-series

Frontier under this node: Driver Examples (Node.js, Python, Java), Memory Limits and allowDiskUse, Pipeline Optimization, Pipeline Stages Reference, Time Series Aggregation, Window Functions ($setWindowFields)

← the whole tree · 3D view· how to read this page