# Read Offload via Caching and Precomputed Views ($merge/$out, Computed Pattern) > Source-anchored research for Read Offload via Caching and Precomputed Views ($merge/$out, Computed Pattern). Run status: COMPLETED. 8 facets · 78 facts · page: https://llms-explorer.com/tree/read-offload-via-caching-and-precomputed-views-merge-out-computed-pattern/ ## Definitions - The manual's Computed Pattern refresh job is one aggregation: `$group` the `screenings` collection by `movie_id` summing `num_viewers` and `revenue`, then `$merge` into `movies` with `on: '_id'` and `whenMatched: 'merge'`. The `merge` action adds the new total fields to the existing movie document and keeps its other fields. The job only reads `screenings` and only writes `movies`, so it can run at any time. (high confidence) — [source](https://www.mongodb.com/docs/manual/data-modeling/design-patterns/computed-values/computed-schema-pattern/) - An on-demand materialized view is a precomputed aggregation result stored on and read from disk, typically written by a `$merge` or `$out` stage. A standard view is computed at read time and not stored. MongoDB says the read-performance benefit grows with pipeline complexity and data size, and gives no benchmark numbers. (high confidence) — [source](https://www.mongodb.com/docs/current/core/materialized-views/) - A MongoDB on-demand materialized view is a pre-computed aggregation pipeline result that is stored on and read from disk, usually written by a `$merge` or `$out` stage. A standard view is computed each time it is read and is not stored, so an on-demand view reads faster than a standard view. (high confidence) — [source](https://www.mongodb.com/docs/manual/core/materialized-views/) ## Structure and components - Treat the derived store as disposable and non-authoritative. Azure says a materialized view can be entirely rebuilt from its source stores and is maintained by a dedicated component, 'so it's a specialized cache'. Facebook's memcache 'is not the authoritative source of the data and is therefore allowed to evict cached data'. (medium confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/materialized-view) - MongoDB's on-demand view example refreshes only part of the data. The `updateMovieStats(startYear)` function runs `$match` on `year >= startYear`, `$group` by year, then `$merge` with `whenMatched: 'replace'` and `whenNotMatched` defaulting to insert, so only groups from that year forward are rewritten and earlier groups are untouched. (high confidence) — [source](https://www.mongodb.com/docs/manual/core/materialized-views/) ## How it works - Precompute only when the same expensive result is read repeatedly, reads far outnumber writes and some staleness is acceptable. The Azure Architecture Center says the Materialized View pattern may not suit data that is simple to query, data that changes very quickly, or workloads where consistency is a high priority. MongoDB's manual frames the Computed Pattern the same way: repeated calculations and a high read-to-write ratio. (high confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/materialized-view) - `$out` is the full-rebuild option with an atomic swap. If the target exists, `$out` creates a temp collection, copies the existing index definitions to it, inserts the results, then calls `renameCollection` with `dropTarget: true`. If the aggregation fails, `$out` leaves the existing collection unchanged. The Azure Architecture Center calls atomic publication or versioned replacement typically the safest refresh, because readers keep using the last complete view while the new one builds. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out/) - The read side of a refresh can run off the primary. Since MongoDB 5.0, a pipeline ending in `$merge` or `$out` can run on replica-set secondaries when every node's featureCompatibilityVersion is 5.0 or higher and the read preference allows secondary reads; the reads happen on the secondary and the writes go to the primary. Not every driver version supports this, and before 5.0 these pipelines always ran on the primary. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$merge` errors if it would change an existing document's `_id` (or its shard key in a sharded target). When `on` does not include `_id`, remove `_id` from the results with a preceding `$unset` stage. (low confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Keep refresh source and target collections separate. Per the `$merge` reference, `$merge` into the collection being aggregated can update documents more than once or loop forever when an update moves a document on disk (the Halloween problem). The Computed Pattern example avoids this by reading `screenings` and writing `movies`. (medium confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Cache-aside (lazy loading): the application checks the cache, on a miss reads the database, stores the result in the cache and returns it. On a write, the Azure pattern updates the data store and then invalidates the cached entry; AWS describes write-through as the proactive alternative. AWS calls cache-aside the most common caching strategy, and its cache holds only data the application actually requested, which keeps cache size cost-effective. (high confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/cache-aside) - Order matters when invalidating a cache-aside entry: update the data store first, then remove the cached item. Removing the cache entry first lets a reader re-cache the old value in the gap before the data store changes, which leaves stale data in the cache. (low confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/cache-aside) - A cache can cost more than it saves. A miss adds round trips to both the cache and the database. Azure says cache-aside may not fit when most requests miss (the check-and-load overhead can outweigh the benefit), when the data is sensitive, or when the data set is static and fits in cache (prime it at startup and disable expiry instead). (medium confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/cache-aside) - A local in-process cache is private to each application instance, so instances can hold different copies of the same data and diverge quickly. The Azure Architecture Center suggests shorter expiry for private caches or a shared distributed cache. (low confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/cache-aside) - The VLDB stampede paper lists three mitigations: a background process that regenerates items before expiry, locking on the cache key (extra write, a lock TTL to tune, requests that lose the lock need a fallback, and no fault tolerance if the lock holder fails), and probabilistic early expiration, where each request may regenerate the item with a probability that rises as expiry nears. The authors present their XFetch algorithm as theoretically optimal; that is the authors' claim. (low confidence) — [source](https://www.vldb.org/pvldb/vol8/p886-vattani.pdf) - Atlas Scheduled Triggers run a Function on a schedule set as a basic interval or a five-field cron expression evaluated in UTC, which is one way to run a periodic refresh on Atlas. Atlas Triggers guarantee each event is processed at least once and bill on daily usage with a shared per-project free tier. The docs show no `$merge` refresh example, and whether an Atlas Function can run a long refresh was not verified. (medium confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-ui/triggers/) - The read-performance gain of an on-demand materialized view grows with the complexity of the pipeline and the size of the data aggregated (MongoDB manual). Azure's Materialized View pattern says the same mechanism, storing the results of complex computations, avoids recomputing them for every request and reduces overall resource consumption. (high confidence) — [source](https://www.mongodb.com/docs/manual/core/materialized-views/) - Precomputing pays when reads of the result far outnumber writes to its inputs. MongoDB's worked example is 1,000,000 reads and 1,000 writes per hour, where computing at write time divides the number of calculations by a factor of 1,000. Azure says a materialized view is a poor fit when the source data changes very quickly. (medium confidence) — [source](https://mongodb.com/company/blog/building-with-patterns-the-computed-pattern) - `$out` must be the last pipeline stage. If the target collection exists, `$out` creates a temp collection, copies the existing collection's index definitions, inserts the pipeline output, then calls `renameCollection` with `dropTarget: true`. The replacement is atomic on completion, and a failed aggregation leaves the existing collection unchanged. A community-forum answer recommends `$out` for a full refresh because it preserves collection metadata such as indexes. (medium confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out/) - Starting in MongoDB 5.0, a pipeline ending in `$merge` or `$out` can run its reads on a replica-set secondary when every node has featureCompatibilityVersion 5.0 or higher and the read preference allows secondary reads. The writes still go to the primary, and not every driver version supports sending these pipelines to secondaries. Earlier versions always ran them on the primary. (high confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Choose the refresh trigger from the write rate and the staleness readers accept. MongoDB says the computation can run with each source update in a low-write environment, and at defined intervals (for example hourly) where writes are regular; it also suggests a last-updated timestamp or a queue of pending computations. Azure says to define the acceptable staleness window first, and warns that event-driven refresh adds excessive overhead when the source changes rapidly. (high confidence) — [source](https://mongodb.com/company/blog/building-with-patterns-the-computed-pattern) - Monitor freshness. Azure says a lost or delayed refresh signal, such as a missed change-feed event or a failed scheduled task, makes a view silently serve stale results, so alert when the view's age exceeds the acceptable staleness window. MongoDB's Computed Pattern article suggests a last-updated timestamp on the document so the application can decide when to recompute. (medium confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/materialized-view) - Skip a materialized view when the source data is simple to query, changes very quickly, or consistency is a high priority (Azure). Skip a look-aside cache when most requests would miss, because 'the overhead of checking the cache and loading data into it might outweigh the benefits', or when the data set is small and static enough to preload (Azure). AWS adds that a write-through cache wastes space because most data is never read. (high confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/materialized-view) - Running `$merge` into the collection being aggregated, for example to write computed values back onto source documents, can update documents more than once or loop forever when the update moves a document's physical location (the Halloween problem). (low confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Inference, not stated in the manual: the windowed refresh in MongoDB's example (`$match` on `year >= startYear`, `$group` by year, `$merge` with `replace`) is correct only because the window boundary aligns with the group key. A boundary that splits a group would replace a complete group total with a partial one. (low confidence) — [source](https://www.mongodb.com/docs/manual/core/materialized-views/) - A cold or replaced cache sends traffic to the database. AWS says a new empty ElastiCache node makes each cache miss query the database. Azure says the workload falls back to the origin data store when the cache malfunctions. Inference: the database must still survive miss traffic, so size it for fallback, not hit traffic alone. (medium confidence) — [source](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Strategies.html) - Facebook's look-aside memcache had two races: a 'stale set' (reordered concurrent updates leave a value that is not the latest) and a 'thundering herd' (heavy write activity on a hot key keeps invalidating it, so many reads take the costly path). Leases fixed both: memcached gives a missing client a 64-bit token that it voids when it receives a delete for the key, and by default hands out one token per key every 10 seconds. (low confidence) — [source](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf) - Cache-aside can serve a miss or briefly stale data between a write and the next read, and does not guarantee consistency if another process changes the store. Azure points to write-through caching when read-heavy paths need read-after-write freshness. AWS says write-through keeps cached data 'never stale' at the cost of a write penalty. (high confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside) ## How-to and procedures - Set the acceptable staleness window before picking a refresh style. The Azure Architecture Center says the maximum staleness equals the refresh interval or event-processing lag, and that a lost or delayed refresh signal (a missed change-feed event, a failed scheduled task) makes the view silently serve stale results, so monitor view age and alert when it exceeds the window. MongoDB's manual notes that periodically refreshed computed values are not guaranteed exact. (medium confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/materialized-view) ## Measurements and reference values - MongoDB's own guidance is that a separate cache is often unnecessary: if indexes cover the queries and the working set fits in RAM, MongoDB serves queries from memory, and its comparison page says 'If you're already using MongoDB, you can achieve similar results without adding Redis'. The default WiredTiger internal cache is the larger of 50% of (RAM - 1 GB) or 0.256 GB, and the filesystem cache uses free memory beyond it. Check index coverage and working-set fit before adding cache infrastructure. A Redis client vendor concedes the working-set-fits case but argues result caching still matters (see disagreements). (medium confidence) (verified as of 2026-10-14) — [source](https://mongodb.com/resources/compare/mongodb-vs-redis) ## Problems, failure modes and limitations - An on-demand materialized view is a real collection, so you can create indexes directly on it; a standard view uses the indexes of its underlying collection and cannot have its own general indexes. The Azure Architecture Center adds that index maintenance on a materialized view adds write-path overhead on every refresh, so weigh read gains against longer refreshes and more refresh compute. (medium confidence) — [source](https://www.mongodb.com/docs/current/core/materialized-views/) - Change streams can watch an on-demand materialized view written with `$merge`; they cannot watch one written with `$out`. (low confidence) — [source](https://www.mongodb.com/docs/current/core/materialized-views/) - A `$merge` `whenMatched` pipeline can update the existing document from the incoming one. It may only use `$addFields`/`$set`, `$project`/`$unset` and `$replaceRoot`/`$replaceWith`, cannot change the `on` field, reads the existing document's fields directly, and reads the incoming document as `$$new` (or through `let` variables). The manual's example adds each day's comment count to a `monthlyCommentTotals` running total; the 2020 practitioner article uses the same technique for incremental counts. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Additive merges are not idempotent: a `whenMatched` pipeline that adds the incoming value adds it again if the same window runs twice. The Atlas Stream Processing streaming-view recipe guards its additive `$merge` with a `lastWindowStart` high-water mark so a replayed window does not double-count. Atlas Triggers guarantee at-least-once processing and can deliver duplicate events after a failure, so a trigger-driven additive refresh needs an equivalent guard (inference from these docs and the `$merge` reference). (medium confidence) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - `$merge` is not atomic across a run. If the aggregation fails after some writes, MongoDB does not roll those writes back, so readers can see a partly refreshed view. This holds for `whenMatched: 'fail'`, unique-index violations and `_id` changes, and with schema validation `validationAction: 'error'` `$merge` writes the valid documents and throws on the invalid ones. `$out` instead makes no change to the existing collection on failure. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$out` cannot write to a sharded collection or a capped collection (the input collection may be sharded). `$merge` can write to a sharded collection: its default `on` is `_id` plus all shard-key fields, any override must include every shard-key field, and a unique index on the `on` fields must exist. On a sharded cluster the output database must already exist for `$merge`. From MongoDB 7.0.3 and 7.1, `$out` can write to a time series collection; `$merge` cannot. (high confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$merge` needs a unique index whose keys are exactly the `on` fields (not partial; sparse is allowed; same collation as the aggregation). If the output collection does not exist, `on` must be `_id`; to match on another key, create the collection first with a unique index on that key. `on` fields cannot hold array values. A 2020 practitioner article notes the same unique-index requirement for a custom `on` field. (medium confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Version-specific `$merge` `on` rules: in MongoDB 8.0 and earlier the `on` fields cannot be missing or null; from 8.1, missing or null is allowed when the supporting unique index is not sparse. From 8.3 the server checks that the automatically created `_id` index matches the query collation and does not run the query if it does not. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$merge` must be the last pipeline stage and writes into a new or existing collection in the same or another database. For a matched document, `whenMatched` can be `replace`, `keepExisting`, `merge` (default), `fail` or an update pipeline. For an unmatched document, `whenNotMatched` can be `insert` (default), `discard` or `fail`. The `on` fields must be backed by a unique index; for a new or unsharded output collection `on` defaults to `_id`. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - The two stages accept different output targets. `$merge` can write to a sharded collection (an overridden `on` must include every shard-key field) but not to a time series collection. `$out` cannot write to a sharded collection (the input collection may be sharded) or a capped collection, and can write to a time series collection starting in MongoDB 7.0.3 and 7.1. (high confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Atlas Scheduled Triggers run a linked Atlas Function on a Basic interval or a five-field CRON expression, evaluated in UTC. The manual's example fires daily at 7:00 AM UTC, aggregates the previous day's orders and stores the result in a `store.reports` collection. It advises a `$match` to limit the documents the Function reads so it stays within Function memory limits. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-ui/triggers/scheduled-triggers/) - Publish a rebuilt view atomically. Azure recommends atomic publication or versioned replacement so the workload keeps serving the last complete view while the new one is built. `$out` fits: it swaps in the new collection by rename on completion and leaves the old one untouched if the aggregation fails. `$merge` does not: documents it wrote before an error are not rolled back. (high confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/materialized-view) - `$merge` only inserts or updates; it cannot remove target documents whose source documents were deleted. An incrementally refreshed view therefore keeps stale rows until you reconcile it, for example by deleting target `_id` values absent from the source or by rebuilding the view with `$out`. The manual lists no delete action among the `whenMatched` and `whenNotMatched` options, and MongoDB's streaming view handles deletes only because it reads `fullDocumentBeforeChange` and emits a negative delta. (medium confidence) — [source](https://oneuptime.com/blog/post/2026-03-31-mongodb-merge-sync-data-between-collections/markdown) - A `$merge`-maintained view can be watched with change streams but an `$out`-maintained view cannot (MongoDB manual). A collection drop or rename raises an `invalidate` event that closes the change-stream cursor and blocks `resumeAfter` (use `startAfter`), which is consistent with `$out` replacing the target by rename (inference). (medium confidence) — [source](https://www.mongodb.com/docs/manual/core/materialized-views/) - With schema validation set to `validationAction: error`, `$merge` throws `MongoServerError` for an invalid document, rejects only the invalid documents and writes the valid ones. `$out` instead makes no change to the existing collection when it hits an invalid document. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$merge` errors if a replace or merge would change an existing document's `_id` (or its shard-key value on a sharded target). When `on` does not include `_id`, remove `_id` from the pipeline output, for example with a preceding `$unset`. Writes completed before the error are not rolled back. (low confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) ## Comparisons and alternatives - The Computed Pattern stores a value derived from source data ahead of time, so a read needs one lookup instead of a computation over many documents. MongoDB's manual says to consider it when the application repeats the same calculations and has a high read-to-write ratio, and says it reduces CPU workload and increases application performance. Neither MongoDB page gives measured numbers. (high confidence) — [source](https://www.mongodb.com/docs/manual/data-modeling/design-patterns/computed-values/computed-schema-pattern/) - `$out` replaces the target by renaming a temp collection over it, which detaches an Atlas Search index on the target: the `$out` reference says to delete and re-create the search index and suggests `$merge` instead. A 2020 community thread reports a search index still holding old documents after `$out` (192,912 documents in the collection versus 325,113 in the index), a reply that the rename changes the collection UUID, and the reporter confirming `$merge` fixed it. (medium confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out/) - Atlas Stream Processing can keep a streaming materialized view current: a stream processor reads a source collection's change stream, computes each event's effect and `$merge`s it into the view, so the view needs no manual or scheduled refresh. MongoDB's comparison table lists on-demand views as manual or scheduled, minutes-to-days latency and a point-in-time snapshot, versus streaming views as change-driven, sub-second to low-second latency and perpetually synchronized. These figures are MongoDB's own, with no benchmark. (medium confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - Choose between per-event and batched refresh deliberately. The Azure Architecture Center says event-driven regeneration is ideal but can create excessive overhead when the source changes rapidly, and lists scheduled tasks, external triggers or manual runs as alternatives. The Atlas Stream Processing recipe batches events in a window so it writes one summarized count per interval instead of one write per event. (medium confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/materialized-view) - Cache-aside does not guarantee consistency. A change made outside the application does not appear until the item is reloaded, and between a write's invalidation and the next read a reader sees a miss or briefly stale data. Expiry is a trade-off: too short and applications keep re-fetching from the data store, too long and data goes stale. Write-through updates the cache with every write, so reads usually hit and the database does fewer reads, but it also caches rarely requested data, making the cache larger and costlier. AWS recommends combining write-through and lazy loading with an appropriate expiration. (high confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/cache-aside) - Cache stampede (thundering herd): when a popular cached item expires or is invalidated, many concurrent requests miss together and all recompute it against the database. The VLDB paper's example: an item read 10 times per second with a 3-second recompute makes 30 requests recompute it. Facebook's memcache uses leases, a per-key token handed out at most once every 10 seconds by default, and measured a peak database query rate of 1.3K/s with leases versus 17K/s without, over one week on keys prone to thundering herds. (high confidence) — [source](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf) - The Computed Pattern precomputes a value (a sum, count or ranking) and stores it in a document, so reads return the stored result instead of recomputing it. MongoDB's manual says to run the computation on insert or with a periodic task, and its example is a rank-ordered '100 Best-Reviewed Gadgets' list that is refreshed periodically while the underlying data changes independently. (high confidence) — [source](https://mongodb.com/company/blog/building-with-patterns-the-computed-pattern) - For a cache-aside write, update the data store first and then delete the cached item. Azure says deleting the cache entry first lets a concurrent reader re-cache the outdated item. Facebook deletes cached values instead of updating them because deletes are idempotent. (high confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside) ## Facts and statements - MongoDB keeps recently used data in RAM and, with indexes and a working set that fits in RAM, serves queries from memory. It does not cache query results for identical queries. A repeated expensive aggregation therefore does its full work on every run, which is the gap that the Computed Pattern, materialized views and external caches fill. (high confidence) — [source](https://www.mongodb.com/docs/manual/faq/fundamentals/) - Update timing for computed values depends on write rate. The manual says a low-write system can recompute on every write that changes the source data, and a system with regular writes can run the aggregation at intervals such as hourly. The 2019 MongoDB blog adds a last-updated timestamp or a queue of pending computations as options and leaves the choice to the developer. With periodic updates the stored value is not guaranteed to be exact. (high confidence) — [source](https://www.mongodb.com/docs/manual/data-modeling/design-patterns/computed-values/computed-schema-pattern/) - The 2019 MongoDB Computed Pattern blog gives one worked ratio: with 1,000,000 reads and 1,000 writes per hour, computing at write time divides the number of calculations by 1,000. The blog reports no measured latency, CPU or cost result. Its use cases are qualitative: sums such as revenue or viewers, time series, product catalogs, single view, event sourcing, and hours-long reuse of large aggregations. (low confidence) — [source](https://mongodb.com/company/blog/building-with-patterns-the-computed-pattern) - Refresh work has a cost of its own. The Azure Architecture Center says regenerating a view consumes compute proportional to source volume and transformation complexity, and that event-driven refresh on fast-changing sources or full rebuilds of large analytical views can be a significant cost driver. It advises right-sizing refresh frequency and scope. (low confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/materialized-view) - Precomputing trades read-time compute for storage and refresh work. MongoDB's stream-processing blog (Sep 2025) says 'You are trading more costly CPU and RAM for cheaper storage' and says offloading joins to a precomputed read model cuts primary load enough to allow smaller instances, with no numbers. The Azure Architecture Center warns that many query-specific views can create unacceptable storage capacity and cost. (medium confidence) — [source](https://mongodb.com/company/blog/technical/real-time-materialized-views-with-atlas-stream-processing) - Incremental refresh pattern: `$match` only the changed key range or time window, `$group` it, then `$merge` into the view collection. MongoDB's materialized-view page shows `updateMovieStats(startYear)`, which re-aggregates years from `startYear` and `$merge`s with `whenMatched: 'replace'`, so re-running the same window gives the same result. A 2020 practitioner article filters on `createdAt` newer than the last aggregation timestamp and runs from a cron job or on page view. (high confidence) — [source](https://www.mongodb.com/docs/current/core/materialized-views/) - `$merge` has no delete action among its documented `whenMatched` and `whenNotMatched` options, so it never removes target documents for keys the pipeline stops emitting. In the `$merge` reference example, re-running over `year >= 1963` leaves the 1939 and 1962 rows from the earlier run in `movieRatingSummary`. To drop orphans, rebuild with `$out` (which replaces the whole collection) or handle deletes explicitly: the Atlas Stream Processing recipe emits a -1 delta on delete events, and the Azure Architecture Center says to propagate source deletions to every view. (medium confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - `$merge` and `$out` must be the last pipeline stage. Neither can run inside a transaction, appear in a view definition (including nested pipelines), appear in the nested pipeline of `$lookup`, `$facet` or `$unionWith`, or be combined with `linearizable` read concern. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - Streaming-view recipe prerequisites (Atlas Stream Processing): enable change stream pre- and post-images on the source collection (`collMod` with `changeStreamPreAndPostImages`), set `$source` `fullDocument` and `fullDocumentBeforeChange` to `required`, compute a signed delta per insert, update and delete, and sum the deltas in a tumbling window. Every `$group` must sit inside a window, the window interval is the view's refresh interval (minimum 1 millisecond), and the view starts from zero, so seed it with a batch aggregation or `initialSync`. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - Limits of streaming materialized views: only incremental aggregations qualify (aggregations that need a full collection scan do not), and with `$lookup` later changes to the reference collection do not update documents the view already wrote. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - A materialized view is disposable: it can be rebuilt entirely from the source data, so the Azure Architecture Center calls it a specialized cache and says a transient view can live in a cache or less reliable store. If a refresh job fails midway, decide whether to serve the previous complete view, a partial view or none. (low confidence) — [source](https://learn.microsoft.com/azure/architecture/patterns/materialized-view) - Vendor and cloud sources say read offload lowers database cost, but none of the sources read quantifies it for Atlas. MongoDB's Atlas Stream Processing blog (2025-09-09) says the primary database 'can often run on smaller, less expensive instances' and gives no numbers. The AWS caching whitepaper says in-memory caching can reduce database costs. Azure warns that the compute cost of refreshing a view 'can be a significant cost driver'. (medium confidence) — [source](https://mongodb.com/company/blog/technical/real-time-materialized-views-with-atlas-stream-processing) - MongoDB contrasts two materialized-view styles. An on-demand view is refreshed manually or on a schedule: latency 'minutes to days', a point-in-time snapshot, best for batch reporting. A streaming view kept current by an Atlas Stream Processing processor is change-driven: sub-second to low-second latency, 'perpetually synchronized', best for live dashboards and operational analytics. (medium confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - Neither `$merge` nor `$out` can run inside a transaction, appear in a view definition, nest inside a `$lookup`, `$facet` or `$unionWith` sub-pipeline, or be combined with `linearizable` read concern. (high confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - An Atlas Stream Processing streaming materialized view reads the source collection's change stream with `fullDocument` and `fullDocumentBeforeChange` set to `required`, turns each insert, update and delete into a signed delta, sums the deltas per key in a windowed `$group`, and writes with `$merge` using a `whenMatched` pipeline that adds each window to the running total. The window interval is the view's refresh interval; the minimum is one millisecond. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/) - Atlas Stream Processing bills per hour per stream processor, by tier, only while the processor runs, in one-second increments. AWS us-east-1 hourly prices are SP2 $0.0550, SP5 $0.1100, SP10 $0.1900, SP30 $0.3900 and SP50 $1.5600; an always-on SP2 is about $40 per month at 730 hours (derived). Egress data transfer is billed separately, for example $0.01 per GB within an AWS region, measured on logical size before compression. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/billing/stream-processing-costs/) - Application-tier caches fill by three documented strategies. Lazy loading (Azure: Cache-Aside) loads an item on a miss, and Azure's version invalidates the item on write. Write-through updates the cache on every database write. Adding a TTL expires entries so later misses refresh them from the database. AWS lists the trade-offs: lazy loading has a miss penalty and can go stale; write-through adds write latency, leaves new nodes missing data and churns the cache with data never read. (high confidence) — [source](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Strategies.html) - Match the TTL to the access pattern. Azure warns that a too-short expiry makes the application 'continually retrieve data from the data store and add it to the cache', and that a too-long expiry leaves stale data. AWS says a TTL does not guarantee fresh data but 'keeps data from getting too stale' and limits cache clutter. (high confidence) — [source](https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside) - Facebook sizes its databases on peak load. Cutting the peak database query rate for a herd-prone key set from 17K/s to 1.3K/s over one week with leases was, in its words, 'a significant efficiency gain'. (low confidence) — [source](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf) - A `$merge` `whenMatched` update pipeline can add each run's result to a stored total, as in MongoDB's monthly comment-totals example (`$add: ['$count', '$$new.count']`). Re-running the same window would count it twice, so the Atlas Stream Processing docs keep a `lastWindowStart` high-water mark that 'prevents a replayed window from double-counting'. (medium confidence) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/merge/) - For computed counters that change very often, MongoDB pairs the Computed Pattern with the Approximation Pattern, which updates a stored value only in coarse steps (a city's population in increments of 100). MongoDB says the results are 'not precise but they are statistically valid'. (low confidence) — [source](https://www.mongodb.com/docs/manual/data-modeling/design-patterns/handle-computed-values/) - Do not add the Computed Pattern or a materialized view by reflex. MongoDB's pattern summary lists as cons that 'It may be difficult to identify the need for this pattern' and that 'Applying or overusing the pattern should be avoided unless needed'. Azure notes views are tailored to one or a few queries, so many queries can mean 'unacceptable storage capacity requirements and storage cost'. (high confidence) — [source](https://www.mongodb.com/company/blog/building-with-patterns-a-summary) - If you use `$out` on a collection that has an Atlas/MongoDB Search index, you must delete and re-create the search index; both the `$out` page and the Search index management page say to consider `$merge` instead. A 2020 forum answer says `$out` renames the collection, which changes its UUID and detaches the index; the asker reported that `$merge` kept the search index updated. (high confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out/) - A streaming materialized view has caveats. It starts empty, so seed it with a batch aggregation or enable `initialSync`. It needs change-stream pre- and post-images on the source collection. Only incremental aggregations qualify; one that needs a full collection scan does not. Later changes to a `$lookup` reference collection do not update documents the view already wrote. (low confidence) (verified as of 2026-10-14) — [source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/streaming-materialized-views/)