Storage and Data-Lifecycle Cost Levers

Parent: MongoDB Atlas Cost Optimization · Published reference · snapshot 2026-10-19 · skill mongodb-operations-expert/references/storage-and-data-lifecycle-cost-levers.md

Also known as: Archive Cost Model, Block Compression (snappy/zlib/zstd), Prefix Compression (indexes), Bucket Architecture and Columnar Compression

↓ Facts as markdown all context files

Source-anchored research for Storage and Data-Lifecycle Cost Levers. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

zlib and zstd are the alternative block compressors, set globally with storage.wiredTiger.collectionConfig.blockCompressor, or per collection/index at creation (e.g. configString block_compressor=zstd). Index prefix compression is disabled with storage.wiredTiger.indexConfig.prefixCompression. (high confidence) [source]

Structure and components

A covered query is answered entirely from an index with no document reads. It requires all queried and returned fields in the index, _id excluded in the projection unless indexed, no null comparisons, and no array-field coverage on multikey indexes. On sharded collections the index must contain the shard key. MongoDB attributes the speed to index keys being smaller than documents and typically in RAM. (high confidence) [source]
Shorter field names do not shrink indexes: index entries hold key values with a predefined structure, not field names. Allegro also found index size unchanged when switching _id from string to int32. (medium confidence) [source]

How it works

TTL deletion runs on a single-threaded background task on the primary, roughly every 60 seconds; each index pass stops at 50,000 deletes or 1 second. A new TTL index or a lowered expireAfterSeconds can make a huge backlog eligible at once and cause heavy delete load; MongoDB advises creating it off-hours or pre-deleting in batches. Mass deletion can fragment storage and may need compact or initial sync to reclaim space. (high confidence) [source]
$indexStats counters reset on mongod restart and internal operations can inflate them. Wait a full workload cycle (24h+ of normal load) before treating zero usage as removable, run it per shard primary on sharded clusters (not on mongos), and exclude _id and unique indexes. (low confidence) [source]
Overlapping indexes are redundant: Performance Advisor de-duplicates suggestions so only { a: 1, b: 1 } is suggested when { a: 1 } is its prefix. (low confidence) [source]
MongoDB's own guidance: field names compress well under WiredTiger block compression, so the additional benefit of short field names is modest; do not abbreviate to single letters because it obscures data. (high confidence) [source]

Measurements and reference values

Online Archive is billed through Atlas Data Federation: storage by GB and days stored (rate varies by cloud provider and region), plus $5.00 per TB of archived data processed with a 10 MB minimum per query, plus data returned/transferred. (high confidence) (verified as of 2026-10-18) [source]
Performance Advisor watches queries slower than 100 ms, groups them by shape, ranks suggested indexes by Impact (total wasted bytes read), and suggests an index only when scanned minus returned documents exceeds 500. The Impact formula is not published. (medium confidence) (verified as of 2026-10-18) [source]
No official MongoDB source found that quantifies or states that query/index tuning lets a cluster drop a tier. The query-targeting and Performance Advisor pages describe only performance impact, not cost. The tuning-to-smaller-tier link is an inference; third-party guidance suggests collecting about two weeks of metrics and checking cache hit rate above 90% before downsizing. (low confidence) (verified as of 2026-03-31) [source]
BSON stores field names inside every document, so field-name length multiplies by document count. Allegro (MongoDB 4.4, snappy) measured a flat test document dropping from about 3.4MB to 2MB (about 40%) with shorter field names alone. (medium confidence) [source]
Percona disputes treating compression as the whole story: document size also affects app memory, network transfer, the replication log, cache memory, compressor input and journal; only on-disk and journal involve compression. In its example, shorter field names cut a document from 54,879 to 41,392 bytes (about 25%). (medium confidence) [source]
Another test (Vicuesoft, 9.79KB avg docs) measured 711.54MB snappy, 449.48MB zlib, 380.43MB zstd. Ratios are data-dependent; MongoDB itself says the ratio depends on the data and library, and gives no universal figure. Blog ranges (snappy 2-3x, zstd 3-5x) are generic estimates. (medium confidence) [source]

Problems, failure modes and limitations

Federated query layer limits that scope the archive/federation option: no indexes on federated sources, object storage is read-only, no change streams or multi-document transactions, 30 concurrent queries per federated instance, 6-hour query limit, 16 MB document limit. (medium confidence) (verified as of 2026-10-18) [source]
MongoDB lists unnecessary indexes (rarely used, redundant because a compound index covers them, or unused) as a schema anti-pattern that bloats storage and slows writes. Procedure: find them with $indexStats (run on every node), hide with hideIndex(), then dropIndexes(). Atlas Performance Advisor (M10+) and Compass also surface them. (high confidence) (verified as of 2026-10-18) [source]

Comparisons and alternatives

A TTL index deletes documents; MongoDB's own TTL page tells users whose goal is reducing storage cost to consider Atlas Online Archive, which tiers infrequently accessed data to managed S3 instead of deleting it. (high confidence) [source]
Data Federation charges 'Data Processed' (bytes read from the source, rounded up to the MB) and 'Data Returned/Transferred'; AWS-side return/transfer within a region is $0.01 per GB. Partitioning S3 data lets a query read one partition (1 GB of 10 GB) instead of the whole file, but a blank $match still reads everything. (high confidence) (verified as of 2026-10-18) [source]
Third-party blog figures of about $0.25/GB/month for cluster storage versus about $0.023/GB/month for Online Archive are unverified; the blog page fetched gave no per-GB rates, so these numbers do not hold as a cost basis. (low confidence) (verified as of 2026-10-18) [source]
Atlas Scanned Objects / Returned alert fires when the server-wide average of documents scanned versus returned exceeds a threshold; the default is 1000:1. The Scanned / Returned alert (index keys examined) is not enabled by default. The ideal ratio is close to 1. (high confidence) (verified as of 2026-10-18) [source]
Write-penalty magnitude is workload dependent. One author benchmark (N=1, laptop, 1000-index vs no-index updates) showed about 50% slower writes at 1000 indexes and a minimal penalty at 100 indexes or fewer, assuming indexes fit in RAM. A MongoDB forum thread on ~14 indexes per collection attributes slowness to mixed causes (many collections, dhandle count, disk) rather than index count alone. (medium confidence) [source]
Compression sharply reduces the field-name saving: on uncompressed MongoDB 2.2.5, short names saved about 10% (22.47 GB to 20.42 GB, 100M synthetic docs); on TokuMX with zlib, long vs short names were 4.38 GB vs 4.27 GB (about 2.5%). (medium confidence) [source]
Value types matter: Allegro found an int32 ID roughly halved on-disk collection size versus a string ID; Percona warns that numbers and dates stored as strings waste space. Enum labels replaced by ordinals saved little with snappy on but considerably more with compression off. (medium confidence) [source]
Allegro (synthetic data): storing explicit nulls took more than 1MB per 1M documents versus omitting the field; removing the spring-data _class field cut a minimal test document by over 50%, though the percentage is much smaller for documents with many fields. (low confidence) [source]
Flattening nested sub-documents saved about 0.4MB (3.4MB to 3MB, roughly 12% of the personal-data fields) in Allegro's synthetic test; a modest lever compared with type and field-name choices. (low confidence) [source]
Bloated documents (related fields not read together stored in one document) inflate the working set so it may stop fitting in RAM, and raise bandwidth. MongoDB's fix is to split hot fields and detail fields into separate linked collections; the trade-off is $lookup cost. This is mainly a RAM/IOPS cost lever (instance tier), not a disk-bytes lever. (medium confidence) (verified as of 2026-10-18) [source]
Percona test (PSMDB 6.0.4, 4vCPU/14GB): 14.95GB of data became 10.75GB with snappy and 7.69GB with zstd level 6, with average CPU 34% vs 31.72%. Percona concludes snappy suits write-heavy low-CPU workloads and zstd suits read-heavy or storage-saving goals. (medium confidence) [source]

Facts and statements

Cost controls for archive queries: configure query limits to cap processed data, run explain and multiply typical processed bytes by monthly query count, set billing alerts. The Online Archive federated-instance page shows bytes processed and query counts but not cost; cost appears only on the Atlas billing page. (high confidence) [source]
Archive queries that omit the archive partition fields (for example createdAt) scan more data and cost more, so include partition-field filters. (medium confidence) [source]
Online Archive is fully managed; pointing Data Federation at your own S3 bucket allows more tuning but adds complexity and ongoing management. Keep Data Federation in the same AWS region as the S3 source to avoid excess charges. (low confidence) [source]
Restoring archived data to the cluster requires pausing the archive, connecting to the archive-only endpoint and running a $merge aggregation, which incurs query charges; archive queries are also reported (anecdotally) to add seconds of latency. (low confidence) [source]
A Query Targeting alert can fire with no Performance Advisor suggestion, so advisor output is not a complete list of inefficient queries; use explain(), Query Profiler or Namespace Insights too. (high confidence) [source]
Every index must be updated on each write. Updates that change an indexed field remove and reinsert the entry in each affected index, so unused indexes add sustained disk I/O and CPU per write. (high confidence) [source]
In WiredTiger, index pages compete with documents for cache space, so a large unused index reduces memory for active data and can raise latency via application-thread eviction. (low confidence) [source]
WiredTiger block compression covers collections (snappy default in 3.0, zlib optional) and prefix compression covers indexes; compression is page-level, so repeated field names across documents in a page compress well, while random and already-compressed binary data does not. (medium confidence) (verified as of 2026-10-18) [source]
WiredTiger uses snappy block compression for most collections and prefix compression for all indexes by default; time-series collections default to zstd. Compression costs extra CPU in exchange for lower storage use. (high confidence) (verified as of 2026-10-18) [source]
Collection data in the WiredTiger internal cache is uncompressed, so block compression saves disk, not cache RAM. The filesystem cache holds the compressed on-disk format. Index prefix compression deduplicates common prefixes and also reduces index RAM in the cache. (high confidence) [source]
storageSize() reports compressed on-disk size and can be smaller than dataSize(); totalIndexSize() reflects prefix-compressed size. Compare these to measure savings. (high confidence) [source]
zstd level defaults to 6 in WiredTiger and is tunable (MongoDB blockCompressorQuality 1-22 per blogs). WiredTiger does not store which compressor wrote a block. (medium confidence) (verified as of 2026-10-18) [source]
Changing the global blockCompressor affects only newly created collections; existing collections keep their compressor. Blogs say compact does not recompress, so switching needs copy to a new collection (or dump/restore). (low confidence) [source]
Deleted-document space is reused inside data files but not returned to the OS except via compact or resyncing a replica set member; dropping a collection or index frees its space. Compression savings therefore do not shrink billed disk until space is reclaimed. (medium confidence) [source]

Children

Data Lifecycle Cost Model (TTL vs Online Archive vs Data Federation)
Query Efficiency as a Cost Lever
Schema Design for Storage Cost
WiredTiger Block and Prefix Compression as Storage Cost Lever
Atlas storage auto-scaling and disk reclaim via compact
Covered queries and projection design (frontier)
Data Federation over customer S3 with partitioning
Data type selection for storage
Field-name abbreviation trade-offs (frontier)
Journal compression (frontier, viability 2.35 / 5, below research gate)
Network (wire) compression (frontier, viability 2.90 / 5, below research gate)
Online Archive partition-field selection
Query limits and billing alerts for archive spend (frontier)
Right-sizing Atlas tier from cache and CPU metrics
Time series collection expireAfterSeconds (frontier, viability 2.80 / 5, below research gate)

← the whole tree · 3D view· how to read this page