mongodb-aggregation-stages-deep
Parent: MongoDB Aggregation Pipeline · researched 2026-05-28T18:45:10.179Z· 18 sources · 10 concepts · skill mongodb-aggregation-stages-deep
This skill is the deep-dive companion to mongodb-aggregation-pipeline. It
MongoDB Aggregation Stages — Deep Reference
- This skill is the deep-dive companion to mongodb-aggregation-pipeline. It [source]
- covers the highest-impact stages where most aggregation bugs and performance [source]
- problems hide: cross-collection joins, recursive graph traversal, parallel [source]
- facets, materialized-view writes, window functions, and time-series gap [source]
- filling. Each section gives syntax, index/memory requirements, working [source]
- examples, and the trade-offs that decide whether you should reach for the [source]
- When to use this skill: [source]
- Designing a pipeline that joins collections, traverses a hierarchy, runs [source]
- parallel facets, materializes a view, computes window functions, or [source]
- fills time-series gaps. [source]
- Diagnosing stage-level memory limits, disk spills, BSONObjectTooLarge [source]
- errors, or $lookup NestedLoopJoin warnings in explain output. [source]
- Choosing between $lookup vs denormalization, $out vs $merge, or [source]
- $bucket vs $bucketAuto. [source]
- When NOT to use this skill: [source]
- Basic pipeline mechanics ($match, $project, $group, $sort, [source]
- $addFields, $unwind) - see mongodb-aggregation-pipeline. [source]
- Index design and query planner reads - see mongodb-indexes-deep and [source]
- mongodb-query-performance. [source]
- Atlas-specific features (Search, Data Federation, Online Archive, [source]
- Charts) outside aggregation - see the corresponding mongodb-atlas-* [source]
- Document-level schema modeling decisions unrelated to a specific [source]
- aggregation stage - see mongodb-schema-design. [source]
- Cross-link reading list: [source]
- mongodb-aggregation-pipeline - pipeline mental model, optimizer rewrites, [source]
- $match/$project/$group/$sort basics. [source]
- mongodb-query-performance - $lookup join tuning, equality vs hash join [source]
- hints, INDEXED vs NLJ in explain output. [source]
- mongodb-schema-design - denormalize-vs-$lookup decision tree. [source]
- mongodb-views-materialized-views - on-demand materialized views via [source]
- $merge, db.createView semantics. [source]
- mongodb-time-series - when to use $densify and $fill with bucketed [source]
- time-series collections. [source]
- mongodb-indexes-deep - covering indexes for the foreignField / [source]
- connectToField of $lookup and $graphLookup. [source]
1. `$lookup` — left outer join
- $lookup performs an unsharded-style left outer join from the input [source]
- collection (the "local" side) to a foreign collection. The optimizer picks [source]
- between four physical operators based on input cardinality, index presence, [source]
- explain("executionStats") exposes the choice under [source]
- $lookup.strategy / eqLookupStrategy. **Treat NestedLoopJoin over [source]
- hundreds of thousands of documents as a red flag** - it scans the foreign [source]
- collection once per input document. [source]
1.1 Equality form (`localField`/`foreignField`)
- The as field is always an array, even when at most one document matches. [source]
- Add { $unwind: { path: "$stock", preserveNullAndEmptyArrays: true } } to [source]
- flatten while keeping unmatched rows. [source]
- If localField is an array, MongoDB matches any element of the array [source]
- against the scalar foreignField (no $unwind required since 3.4). [source]
- The foreign collection must live in the same database. [source]
- Index the foreign side on the foreignField (or compound index where [source]
- foreignField is the leading key for the optimizer to choose indexed [source]
1.2 Pipeline form with `let`
- The pipeline form unlocks correlated subqueries, multi-key joins, [source]
- projection, sorting, and limiting on the joined side: [source]
- Variables bound in let are referenced with the $$ prefix inside the [source]
- sub-pipeline; fields of the foreign documents use the normal $ prefix. [source]
- localField/foreignField and let/pipeline can coexist - when both [source]
- are present, the equality join runs first and the pipeline filters the [source]
- matched documents, which can make the optimizer choose [source]
- IndexedLoopJoin + NLJ for the second predicate. [source]
- For the indexed nested loop to engage on $expr pipeline joins, the [source]
- foreign index must cover the equality side; range predicates inside the [source]
- pipeline are evaluated post-fetch. [source]
1.3 Atlas Search inside `$lookup` (6.0+)
1.4 Sharded collections
- Before MongoDB 5.1, $lookup could not target a sharded foreign collection. [source]
- From 5.1 onward, sharded $lookup is supported but the optimizer routes [source]
- through the primary shard of the join collection for non-equality joins, [source]
- which can serialize throughput. Strategies: [source]
- Keep the foreign collection unsharded if it is small (a few hundred MB). [source]
- Use a covering index on the shard key so $lookup can target a single [source]
- For analytics, materialize via $merge to a denormalized collection. [source]
1.5 `$lookup` vs denormalization decision tree
- See mongodb-schema-design for the embed-vs-reference framework. [source]
1.6 Common `$lookup` pitfalls
- The as array grows large: a $lookup that joins to a 50 000-document [source]
- parent will return a 50 000-element array per input document, which can [source]
- push the result over 16 MB. Use the pipeline form with $match/$limit, [source]
- or add a $lookup-then-$unwind-then-aggregate sequence. [source]
- Type mismatch silently returns empty arrays: localField ObjectId [source]
- vs foreignField string will never match. Normalize types first or [source]
- use $expr with $toObjectId / $toString. [source]
- $lookup placed before $match loses pushdown: if the post-lookup [source]
- $match is on the input side, move it before the $lookup so it [source]
- shrinks the input set. The optimizer does this automatically only when [source]
- the predicate is on a field that exists pre-lookup. [source]
2. `$graphLookup` — recursive single-collection traversal
2.1 Syntax
- startWith - expression evaluated against the input document; can be a [source]
- scalar or array. Each value becomes a starting point. [source]
- connectFromField - the field on the matched documents whose value [source]
- feeds the next iteration's lookup. [source]
- connectToField - the field on the foreign documents to match [source]
- against. Index this field. [source]
- maxDepth - inclusive upper bound on recursion depth. 0 means a [source]
- single non-recursive lookup. Always set this on production queries to [source]
- guard against cycles. [source]
- depthField - when present, every output document gets a numeric [source]
- depthField indicating how many hops away from the start it is. [source]
- restrictSearchWithMatch - pre-filter applied at every recursion step; [source]
- cannot reference $$ROOT of the originating document but can use [source]
- standard query operators against the foreign collection. [source]
2.2 Tree / hierarchy patterns
- The classic org-chart query: [source]
2.3 Category trees and bill-of-materials
2.4 Cycle protection
- $graphLookup is naturally cycle-safe: it tracks already-visited documents [source]
- by _id and will not revisit them. However, without maxDepth a deep [source]
- or wide graph can still allocate huge intermediate state. Cap maxDepth [source]
- at the deepest meaningful value (org chart: 10–12 levels; permission [source]
2.5 Memory and performance
- Memory limit: like every aggregation stage, 100 MB before spill. [source]
- $graphLookup automatically writes temporary files to disk when it [source]
- exceeds 100 MB only if allowDiskUse: true is set on the [source]
- aggregation call (or allowDiskUseByDefault is true at server level). [source]
- Otherwise the stage errors out. [source]
- Index connectToField. Without an index every recursion step is a [source]
- restrictSearchWithMatch runs against the index when possible - use it [source]
- to prune dead branches early (e.g. { active: true }, [source]
- { tenantId: "..." }). [source]
- The result is one array per input document. Use $match upstream to [source]
- bound starting points. [source]
2.6 Common use cases
- Org charts: who reports (directly or transitively) to a manager? [source]
- Category trees: ancestry from leaf to root, or all descendants of a [source]
- Permission inheritance: which groups grant a user a permission via [source]
- nested group membership? [source]
- Friend-of-friend / social graph: limited-depth neighbors. [source]
- Bill of materials: all sub-parts beneath an assembly. [source]
- Dependency graphs: transitive package dependencies, file imports. [source]
3. `$facet` — parallel sub-pipelines
3.1 Syntax and structure
3.2 Constraints
- One document, ≤16 MB: the entire output document must fit in BSON's [source]
- 16 MB limit. If sub-pipelines return large arrays, the aggregation fails [source]
- with BSONObjectTooLarge. Bound each branch with $limit, $project [source]
- to drop fields, or move heavy branches outside $facet and union the [source]
- No cross-branch references: sub-pipelines cannot read each other's [source]
- output. They share only the upstream input. [source]
- No $facet nesting: you cannot put a $facet stage inside another [source]
- Disallowed sub-stages: $out, $merge, $collStats, $indexStats, [source]
- $planCacheStats, $facet (no nesting), $geoNear, $search, [source]
- $searchMeta, and $vectorSearch cannot appear inside a $facet branch. [source]
- Memory limit per branch: each branch is independent and subject to [source]
- its own 100 MB / allowDiskUse ceiling. [source]
3.3 When to reach for `$facet`
- The pagination idiom: [source]
3.5 `$facet` vs running pipelines in parallel from the client
- Because every branch shares the same upstream input, $facet avoids the [source]
- duplicated work of running N independent pipelines that each repeat the [source]
- filter and projection stages. Trade-offs: [source]
- $facet wins when the upstream pipeline is expensive and shared. [source]
- Multiple client-side pipelines win when each branch needs different [source]
- pre-filters or when the 16 MB output ceiling is at risk. [source]
4. `$bucket` and `$bucketAuto` — histograms
4.1 `$bucket` — explicit boundaries
- boundaries must be strictly increasing values of the same type as [source]
- the groupBy expression results. [source]
- Each bucket covers [boundaries[i], boundaries[i+1]) - lower-inclusive, [source]
- A document falls into the default bucket when its groupBy value is [source]
- outside the explicit range or its type does not match boundary type. [source]
- Without default, out-of-range documents cause an error. [source]
- The _id of each output document is the lower bound of the bucket [source]
- (or the literal default value for the catch-all). [source]
4.2 `$bucketAuto` — automatic boundaries
- $bucketAuto distributes documents as evenly as possible across the [source]
- requested number of buckets. granularity (optional) snaps boundaries to [source]
- a "preferred number" series: [source]
- Boundary values are multiplied by powers of 10 so they cover the actual [source]
- data range. Use POWERSOF2 for log-scale histograms (latency, file [source]
- size). Use Renard for engineering-style "nice number" bin edges. [source]
4.4 Histogram + summary together
- Combine with $facet to render a dashboard widget in one round-trip: [source]
5. `$out` vs `$merge` — writing pipeline results
- Both are terminal stages: they must appear last in the pipeline. [source]
5.1 `$out` — wholesale collection replacement
- Drops or atomically replaces the target collection. [source]
- Preserves indexes on the target collection if it already exists (MongoDB [source]
- re-creates the same indexes after the replacement). [source]
- Fails if the target is the source collection. [source]
- Cannot output to a sharded collection (on any version). The input/source [source]
- collection may be sharded, but the $out target must be unsharded - use [source]
- $merge to write into a sharded collection. [source]
- Does not allow you to write conditionally - every run is an unconditional [source]
5.2 `$merge` — incremental upsert
- into - target collection (string or { db, coll }). [source]
- on - single field name or array used as the unique key. Default is [source]
- _id. If you supply a custom on, a unique index must back it. [source]
- let - variables usable by the whenMatched pipeline. [source]
- whenMatched - behavior when a document with the same on value [source]
- exists in the target: [source]
- "merge" (default) - $set-style field merge (incoming fields [source]
- overwrite; existing-only fields are kept). [source]
- "replace" - replace the matched document entirely (preserves _id). [source]
- "keepExisting" - keep target untouched. [source]
- "fail" - error on any collision. [source]
- [ ...pipeline ] - custom update pipeline. Reference incoming fields [source]
- via $$new and existing fields with normal $ paths. [source]
- whenNotMatched - behavior when no match: [source]
- "insert" (default) - insert the incoming document. [source]
- "discard" - drop unmatched results silently. [source]
- "fail" - error if any input has no match in the target. [source]
5.3 Materialized view refresh patterns
5.4 Idempotency rules
- Set _id (or the on key) deterministically from the source data [source]
- so re-running the same window produces identical keys. [source]
- For additive metrics with incremental refresh, the math must compensate [source]
- if the same source documents are seen twice. Either (a) use a strict [source]
- watermark that never overlaps, or (b) make whenMatched "replace" and [source]
- aggregate over the full window each run. [source]
- For whenMatched: "merge", missing fields in the new document do not [source]
- remove existing fields. Use "replace" if you want hard overwrite. [source]
5.5 Pitfalls
- Same-collection $merge can loop: writing back to the source [source]
- collection in a way that changes document size or shard-key value can [source]
- cause documents to be re-read by the same pipeline. MongoDB docs [source]
- explicitly warn that this can result in documents being processed [source]
- multiple times or an infinite loop. Prefer $out or write to a [source]
- Unique index required for non-default on: without it, $merge [source]
- fails at execution time. [source]
- $merge/$out are not transactional with the source reads: a [source]
- client reading the source mid-merge sees a mix. Wrap downstream [source]
- consumers around a "last refresh at" marker. [source]
- Sharding: $merge is supported into sharded targets; the on key [source]
- must include the shard key (or be the shard key). $out cannot target a [source]
- sharded collection on any version - use $merge when the destination is [source]
6. `$setWindowFields` — window functions
6.1 Anatomy
- partitionBy - segments the input into independent groups. Each [source]
- partition gets its own sliding window. [source]
- sortBy - orders documents within a partition. Window boundaries are [source]
- defined relative to this order. [source]
- window - defines the inclusive range of documents/values the [source]
- documents: [a, b] - position-based, where "current", integers, or [source]
- "unbounded" are valid bounds. [source]
- range: [a, b] - value-based on the sortBy field; for date sort [source]
- fields, supply unit (e.g. "day"). [source]
- You cannot mix documents and range in the same window. [source]
6.2 Operator catalog
- Accumulators usable inside $setWindowFields.output: [source]
6.3 Ranking semantics
6.6 Derivative and integral for time-series
6.7 Memory limit
7. `$densify` — close gaps in numeric / time ranges
7.1 Syntax
- "full" - span the global min/max of the field across all input [source]
- documents (one synthetic sequence covers every partition). [source]
- "partition" - span the min/max within each partition independently. [source]
- [lower, upper] - explicit literal bounds. [source]
- unit is required when field is a date and optional for numeric fields. [source]
7.2 Hourly observations example
7.3 Numeric densification example
7.4 Rules
- $densify does not modify existing documents; it only adds new ones. [source]
- Inserted documents inherit only partitionByFields and the densified [source]
- field. Use $fill next to populate value fields. [source]
- If two existing documents collide with the same generated key, the [source]
- existing documents survive - $densify is a no-op for those positions. [source]
- $densify must precede operations that assume dense input [source]
- ($setWindowFields for moving averages, charts, time-aligned joins). [source]
8. `$fill` — populate null and missing values
8.1 Methods
- For linear and locf, you must supply sortBy so ordering is defined. [source]
8.3 Time-series gap fill (densify + fill)
- The canonical pattern from the MongoDB blog: [source]
8.4 `$linearFill` expression (vs `$fill`'s `linear` method)
8.5 Rules
- For locf, the first run of nulls before any non-null value remains [source]
- null - there is no value to carry forward. [source]
- For linear, the same boundary rule applies and additionally: leading [source]
- and trailing nulls (before the first or after the last non-null) [source]
- remain null because interpolation requires two endpoints. [source]
- $fill only writes the output fields you specify. Other fields are [source]
- $fill's partitionBy / partitionByFields must match the upstream [source]
- $densify partition keys, or you will see synthetic rows that never [source]
9. `$unionWith` — UNION ALL across collections
9.1 Syntax
9.3 Removing duplicates (`UNION` not `UNION ALL`)
- $unionWith includes duplicates. To deduplicate, follow it with $group: [source]
9.4 Rules
- All collections involved must be in the same database. [source]
- The combined stream of documents may have heterogeneous shapes. Project [source]
- to a common shape before grouping or merging. [source]
- $unionWith cannot be used inside a multi-document transaction. [source]
- Disallowed stages inside the inner pipeline: $out, $merge. [source]
- Each collection scan inside $unionWith is independent - index it [source]
- appropriately if you push down a $match. [source]
9.5 Use cases
- Sharded archive + hot collection: union an archive collection with [source]
- the live collection for unified reporting. [source]
- Cross-tenant report: union per-tenant collections (when the schema [source]
- forbids a single collection). [source]
- Migration: dual-read from old and new collections during cutover. [source]
- Heterogeneous corpora: union "cases", "slack", "meetings" into a [source]
- single search/scoring pipeline. [source]
10.1 The 100 MB-per-stage rule
- Each blocking aggregation stage may use up to 100 MB of RAM for its [source]
- in-memory state. Blocking stages include $sort, $group, $bucket, [source]
- $bucketAuto, $setWindowFields, $facet (per branch), $graphLookup, [source]
- and $lookup (when buffering). [source]
- When a stage exceeds the limit: [source]
- MongoDB 6.0+ with allowDiskUseByDefault: true (the default for [source]
- Atlas / current community builds) - the stage spills to temporary [source]
- allowDiskUseByDefault: false - the stage errors out with [source]
- QueryExceededMemoryLimitNoDiskUseAllowed unless the call explicitly [source]
- passes allowDiskUse: true. [source]
- Pre-6.0 - the stage always errors unless allowDiskUse: true is [source]
- set on the aggregation command. [source]
10.2 Enabling disk spill
- Server: db.adminCommand({ setParameter: 1, allowDiskUseByDefault: true }) [source]
- Atlas: allowDiskUseByDefault is enabled by default; it can be [source]
- Drivers: AggregateOptions.allowDiskUse(true) (Java), [source]
- aggregate({ allowDiskUse: true }) (Node), aggregate(..., allowDiskUse=True) (Python). [source]
10.3 `explain("executionStats")` for spill detection
- Key signals in the output: [source]
- usedDisk: true - at least one stage spilled to temporary files. [source]
- spillFileSize / spilledBytes / spilledRecords - magnitude of the [source]
- spill (newer versions). [source]
- executionTimeMillisEstimate per stage - locate the slow stage. [source]
- nReturned, totalKeysExamined, totalDocsExamined - index health. [source]
- $lookup.strategy / eqLookupStrategy — [source]
- "IndexedLoopJoin" (good), "NestedLoopJoin" (bad on large input), [source]
- "HashJoin", or "DynamicIndexedLoopJoin". [source]
10.4 The 16 MB document limit
- Every document at every stage must fit in 16 MB. [source]
- $facet output is one document containing every branch's array — [source]
- this is where the 16 MB ceiling hits hardest. Bound each branch. [source]
- $group with $push can blow up an array beyond 16 MB. Use [source]
- $accumulator chunks, $bucket + $group, or write to a collection [source]
- with $merge instead of returning to the client. [source]
- Pipelines that join with $lookup and then $unwind are fine; pipelines [source]
- that $lookup and then leave the joined array intact may overflow. [source]
10.5 Stage ordering for memory and performance
- The optimizer reorders stages where it can, but you should write [source]
- pipelines that already minimize the working set: [source]
- $match first - filter early, ideally on an indexed field. [source]
- $project / $unset next - drop fields you do not need. [source]
- $sort before $group only if the sort is needed for output, and [source]
- only after the dataset is reduced. [source]
- $lookup / $graphLookup after pruning - every joined document [source]
- Terminal $out / $merge last. [source]
10.6 Sharded aggregation specifics
- Stages run on each shard up to the first split point (a stage that [source]
- requires a global view, like $group without the shard key in _id, [source]
- $sort over the full result, or $facet). [source]
- After the split point, intermediate results stream to the mongos (or [source]
- the merging shard) for the rest of the pipeline. [source]
- $out / $merge execute on the merging side; the target collection [source]
- may need to be unsharded or $merge.on must include the shard key. [source]
13. Anti-patterns
- $lookup without an index on foreignField - turns every join [source]
- into an O(N*M) collection scan. Always index the foreign side. [source]
- $graphLookup without maxDepth - unbounded recursion can hit the [source]
- 100 MB ceiling or, with disk spill, run for hours on dense graphs. [source]
- $facet branches without $limit - easiest way to hit [source]
- BSONObjectTooLarge. Cap arrays in every branch. [source]
- $merge into the source collection - risk of infinite loops if [source]
- the merge changes document size or shard key. Write to a separate [source]
- $out to a sharded target - unsupported on any version; $out cannot [source]
- write to a sharded collection. Use $merge when the destination is sharded. [source]
- $setWindowFields over one giant partition - single-partition [source]
- window over millions of documents will spill. Add a partitionBy key [source]
- or split by tenant / day. [source]
- $densify without a following $fill - produces ghost rows with [source]
- missing fields that downstream code may not expect. [source]
- $unionWith followed by a $lookup - the join may need to scan [source]
- twice, once per input branch, without index pushdown for one side. [source]
- Pipelines that ignore explain - without executionStats, you [source]
- cannot detect usedDisk, NestedLoopJoin, or unindexed sorts. [source]
15. Sources
- MongoDB Docs - $lookup (aggregation stage) [source]
- MongoDB Docs - $graphLookup (aggregation stage) [source]
- MongoDB Docs - $facet (aggregation stage) [source]
- MongoDB Docs - $bucket (aggregation stage) [source]
- MongoDB Docs - $bucketAuto (aggregation stage) [source]
- MongoDB Docs - $merge (aggregation stage) [source]
- MongoDB Docs - $out (aggregation stage) [source]
- MongoDB Docs - $setWindowFields (aggregation stage) [source]
- MongoDB Docs - $densify (aggregation stage) [source]
- MongoDB Docs - $fill (aggregation stage) [source]
- MongoDB Docs - $linearFill (expression) [source]
- MongoDB Docs - $unionWith (aggregation stage) [source]
- MongoDB Docs - Aggregation Pipeline Limits [source]
- MongoDB Blog - Introducing Gap Filling For Time Series Data in MongoDB 5.3 [source]
- MongoDB Developer Hub - Preparing time-series data with $densify and $fill [source]
- Practical MongoDB Aggregations Book - Faceted Classification [source]
- MongoDB Performance Tuning (Guy Harrison) - Getting started with MongoDB 5.0 window functions [source]
- Percona - Window Functions in MongoDB 5.0 [source]
Children
- $lookup (frontier)
- $graphLookup (frontier)
- $facet (frontier)
- $bucket-bucketAuto (frontier)
- $merge-$out (frontier)
- $setWindowFields (frontier)
- $densify-$fill (frontier)
- $unionWith (frontier)
- allowDiskUse-100MB-stage (frontier)
- explain-executionStats-usedDisk (frontier)
Frontier under this node: $bucket-bucketAuto, $densify-$fill, $facet, $graphLookup, $lookup, $merge-$out, $setWindowFields, $unionWith, allowDiskUse-100MB-stage, explain-executionStats-usedDisk