# MongoDB time-series working-set sizing and sharding > Everything the scanned sources say about MongoDB time-series working-set sizing and sharding: 52 source-anchored units across 1 sources, grouped by facet. Parent: [Global AI Hub Research Corpus](https://llms-explorer.com/tree/global-ai-hub-research-corpus/) · 8 facets · 52 facts · page: https://llms-explorer.com/tree/mongodb-time-series-sizing-and-sharding/ ## Definitions - If collection data is compressed (which is the [`default for WiredTiger`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--wiredTigerCollectionBlockCompressor)), the storage size reflects the compressed size and may be smaller than the value for [`collStats.size`.](https://www.mongodb.com/docs/manual/reference/command/collstats.md#mongodb-data-collStats.size) — [source](https://www.mongodb.com/docs/manual/reference/command/collStats.md#output) ## Structure and components - A document that contains metrics on the WiredTiger **spill instance**. When MongoDB writes to disk to fulfill certain operations, it utilizes a separate WiredTiger instance, which contains its own in-memory cache. This separate cache isolates operations from the main WiredTiger cache. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#spillwiredtiger) - - [`Auto-scaling: Compute auto-scaling initiated for base tier`](https://www.mongodb.com/docs/atlas/reference/alert-conditions.md#mongodb-alert-Auto-scaling--Compute-auto-scaling-initiated-for-base-tier) occurs if Atlas starts compute auto-scaling for any of the [operational nodes](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-operational-node) in your dedicated cluster. Atlas can scale disk capacity as part of this event. — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-autoscaling-alerts.md#alert-conditions) - - [`Auto-scaling: Compute auto-scaling initiated for analytics tier`](https://www.mongodb.com/docs/atlas/reference/alert-conditions.md#mongodb-alert-Auto-scaling--Compute-auto-scaling-initiated-for-analytics-tier) occurs if Atlas starts compute auto-scaling for any of the [analytics nodes](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-analytics-node) in your dedicated cluster. Atlas can scale disk capacity as part of this event. — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-autoscaling-alerts.md#alert-conditions) - | Aspect | Atlas Core | Atlas Infinite | | --- | --- | --- | | Architecture | Compute and storage run on the same node. Each data-bearing node runs on a virtual machine with attached storage. | Compute and storage run as independent layers. MongoDB manages the storage layer separately from the compute nodes. | | Compute scaling | You change compute capacity by selecting a different cluster tier. Atlas restarts the cluster's nodes when you change the tier, with no downtime on dedicated clusters. | You scale the cluster tier independently of storage. Scaling doesn't move your data, so it is fast — [source](https://www.mongodb.com/docs/atlas/index.md#compare-editions) - Messages related to compaction operations performed by the [WiredTiger](https://www.mongodb.com/docs/manual/core/wiredtiger.md#std-label-storage-wiredtiger) storage engine. To specify the log level for [`WTCMPCT`](https://www.mongodb.com/docs/manual/reference/log-messages.md#mongodb-data-WTCMPCT) components, use the [`systemLog.component.storage.wt.wtCompact.verbosity`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-systemLog.component.storage.wt.wtCompact.verbosity) setting. — [source](https://www.mongodb.com/docs/manual/reference/log-messages.md#components) - Sharding distributes data across the shards in the cluster, allowing each shard to contain a subset of the total cluster data. As the data set grows, additional shards increase the storage capacity of the cluster. — [source](https://www.mongodb.com/docs/manual/sharding/) ## How it works - For example, in a collection with six encrypted fields, you can reduce the size of the `ESC` by at least 1000 documents when total documents inserted and unique field/value pairs inserted since the last compaction meet the following conditions: — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#example) - The default values for the diagnostic data capture interval and the maximum sizes are configured to provide data to MongoDB engineers with minimal impact on performance and storage size. Typically, these values need modification only when requested by MongoDB engineers for specific diagnostic purposes. — [source](https://www.mongodb.com/docs/manual/reference/parameters.md#diagnostic-parameters) - The number of incoming connections the server rejected because the server doesn't have the capacity to accept additional connections or the [`net.maxIncomingConnections`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-net.maxIncomingConnections) setting is reached. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#connections) - The ticket capacity across all pools. When [`usesPrioritization`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.queues.execution.usesPrioritization) is `false`, this value only includes tickets in the normal-priority queue. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#queue-information) - Specifies the default compression for collection data. You can override this on a per-collection basis when creating collections. — [source](https://www.mongodb.com/docs/manual/reference/program/mongod.md#wiredtiger-options) - - [`Auto-scaling: Disk auto-scaling initiated`](https://www.mongodb.com/docs/atlas/reference/alert-conditions.md#mongodb-alert-Auto-scaling--Disk-auto-scaling-initiated) occurs when Atlas starts auto-scaling disk capacity. — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-autoscaling-alerts.md#alert-conditions) - These enhancements further reduce size of data on disk when compressed with `zstd`, and also significantly reduce space used in the WiredTiger cache. — [source](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-compression.md#column-compression) ## How-to and procedures - To reduce the size of the `ESC` metadata collection by at least `t` documents, run metadata compaction when the following formula is satisfied: — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - Use [`mongostat`](https://www.mongodb.com/docs/database-tools/mongostat.md#mongodb-binary-bin.mongostat) to understand the distribution of operation types and to inform capacity planning. See the [`mongostat`](https://www.mongodb.com/docs/database-tools/mongostat.md#mongodb-binary-bin.mongostat) reference page for details. — [source](https://www.mongodb.com/docs/manual/administration/monitoring.md#mongostat) - Use this data to monitor the state and storage capacity of a specific database. This output also allows you to compare use between databases and to determine the average [document](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-document) size in a database. — [source](https://www.mongodb.com/docs/manual/administration/monitoring.md#dbstats) ## Measurements and reference values - - If your WiredTiger cache fill ratio regularly exceeds 90% — [source](https://www.mongodb.com/docs/manual/tutorial/build-indexes-on-replica-sets.md#about-this-task) - Manually run [metadata collection compaction](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#std-label-qe-metadata-compaction) when your metadata collections exceed 1 GB. Compaction decreases the size of the metadata collections associated with encrypted fields, and improves performance. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/reference/limitations.md#manual-metadata-collection-compaction) ## Problems, failure modes and limitations - To hold the additional representative queries, MongoDB 8.3 also increases the storage capacity for representative queries in query settings beyond the original 16 MB BSON document limit. — [source](https://www.mongodb.com/docs/manual/reference/command/setQuerySettings.md#set-query-settings-by-passing-in-a-query-shape-hash) ## Comparisons and alternatives - Horizontal Scaling involves dividing the system dataset and load over multiple servers, adding more servers to increase capacity as required. Each machine handles a subset of the overall workload, which can cost less than high-end hardware for a single machine. The trade-off is increased complexity in infrastructure and maintenance. — [source](https://www.mongodb.com/docs/manual/sharding/) ## Facts and statements - You can check the size of a metadata collection using [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) and running the [`db.collection.totalSize()`](https://www.mongodb.com/docs/manual/reference/method/db.collection.totalsize.md#mongodb-method-db.collection.totalSize) command. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#running-metadata-compaction) - As you insert or update documents, the metadata collections change and grow. Metadata collection compaction empties the `ECOC` and reduces the size of the `ESC`. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#metadata-collection-compaction) - In the worst case, running metadata compaction reveals the number of unique field/value pairs inserted across all documents since the previous compaction. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - For details on the exact information revealed by metadata compaction, see "Section 6: Theoretical Analysis" and "Section 9: Guidelines" in MongoDB's [Queryable Encryption Technical Paper](https://www.mongodb.com/collateral/queryable-encryption-technical-paper). — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - - `docinserts`: the number of documents inserted since the last compaction. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - - `valinserts`: the number of unique field/value pairs inserted since the last compaction. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - The number of encrypted fields per document multiplied by the number of insertions since last compaction, minus the number of unique field/value pairs inserted across all documents since the last compaction, should be greater than or equal to the number of documents to remove from the `ESC`. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#scheduling-metadata-compaction) - The formula would be satisfied, for example, if 200 documents were inserted since the last compaction with a total of 200 unique field/value pairs across all documents, or if 400 documents were inserted since the last compaction with 700 unique field/value pairs. — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#example) - You must run metadata compaction manually. Use [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) and run the `db.collection.compactStructuredEncryptionData()` command: — [source](https://www.mongodb.com/docs/manual/core/queryable-encryption/fundamentals/manage-collections.md#running-metadata-compaction) - Specifies whether the instance runs automatic background [compaction.](https://www.mongodb.com/docs/manual/reference/privilege-actions.md#mongodb-authaction-compact) — [source](https://www.mongodb.com/docs/manual/reference/parameters.md#storage-parameters) - - Upscale your Search Nodes to give `mongot` more CPU and memory for replication. To learn more about adding or scaling Search Nodes, see [Tips for Sizing and Scaling Search Nodes.](https://www.mongodb.com/docs/search/deployment/deployment-options.md#std-label-fts-search-node-sizing-tips) — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-search-alerts.md#alert-conditions) - The `scale` argument affects this value. Data compression does not affect this value. — [source](https://www.mongodb.com/docs/manual/reference/command/collStats.md#output) - - Clustered collections have a lower storage size, which improves performance for queries and bulk inserts. — [source](https://www.mongodb.com/docs/manual/core/clustered-collections.md#benefits) - The number of bytes to be written to disk by the external sorter before compression. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#indexbulkbuilder) - The total capacity of the ticket pool. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#queue-information) - The total capacity of the low-priority ticket pool. — [source](https://www.mongodb.com/docs/manual/reference/command/serverStatus.md#queue-information) - Specifies the type of compression to use to compress WiredTiger journal data. — [source](https://www.mongodb.com/docs/manual/reference/program/mongod.md#wiredtiger-options) - - [`Auto-scaling: Compute auto-scaling down didn't initiate for base tier due to storage requirements`](https://www.mongodb.com/docs/atlas/reference/alert-conditions.md#mongodb-alert-Auto-scaling--Compute-auto-scaling-down-didn-t-initiate-for-base-tier-due-to-storage-requirements) occurs if Atlas couldn't start compute auto-scaling for any of the [operational nodes](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-operational-node) in your dedicated cluster as the configured storage size isn't supported by the target cluster tier. — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-autoscaling-alerts.md#alert-conditions) - - [`Auto-scaling: Compute auto-scaling down didn't initiate for analytics tier due to storage requirements`](https://www.mongodb.com/docs/atlas/reference/alert-conditions.md#mongodb-alert-Auto-scaling--Compute-auto-scaling-down-didn-t-initiate-for-analytics-tier-due-to-storage-requirements) occurs if Atlas couldn't start compute auto-scaling for any of the [analytics nodes](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-analytics-node) in your dedicated cluster as the configured storage size isn't supported by the target cluster tier. — [source](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/atlas-autoscaling-alerts.md#alert-conditions) - - Drive your app's retrieval capacity with [Voyage AI](https://www.mongodb.com/docs/voyageai.md#std-label-voyage-landing) embedding models and APIs that turn data into high-accuracy vectors natively in the platform. — [source](https://www.mongodb.com/docs/atlas/index.md#features) - - You want to grow storage capacity without scaling the cluster tier — [source](https://www.mongodb.com/docs/atlas/index.md#choose-an-edition) - - Array compression (starting in MongoDB 6.0) — [source](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-compression.md#column-compression) - Column compression ensures that if you are using objects and arrays in your documents, you receive the same compression benefits had those embedded fields existed at the root level of your document. — [source](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-compression.md#object-and-array-compression) - Provides a lower compression rate than `zlib` or `zstd` but has a lower CPU cost than either. — [source](https://www.mongodb.com/docs/manual/administration/production-notes.md#compression) - Provides better compression rate than `snappy` but has a higher CPU cost than both `snappy` and `zstd`. — [source](https://www.mongodb.com/docs/manual/administration/production-notes.md#compression) - Provides better compression rate than both `snappy` and `zlib` and has a lower CPU cost than `zlib`. — [source](https://www.mongodb.com/docs/manual/administration/production-notes.md#compression) - By default, WiredTiger uses [snappy](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-snappy) compression library. To change the compression setting, see [`storage.wiredTiger.collectionConfig.blockCompressor`.](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.collectionConfig.blockCompressor) — [source](https://www.mongodb.com/docs/manual/administration/production-notes.md#compression) - Database systems with large data sets or high throughput applications can challenge the capacity of a single server. For example, high query rates can exhaust the CPU capacity of the server. Working set sizes larger than the system's RAM stress the I/O capacity of disk drives. — [source](https://www.mongodb.com/docs/manual/sharding/) - Vertical Scaling increases the capacity of a single server by using a more powerful CPU, adding more RAM, or expanding storage. Available technology and cloud provider hardware configurations impose a practical maximum for vertical scaling. — [source](https://www.mongodb.com/docs/manual/sharding/) - Performance issues may indicate that the database is operating at capacity and that it is time to add additional capacity to the database. In particular, the application's working set should fit in the available physical memory. — [source](https://www.mongodb.com/docs/manual/administration/analyzing-mongodb-performance.md) - Many concurrent application requests may overwhelm the server's ability to keep up with demand. If this is the case, increase the capacity of your deployment. — [source](https://www.mongodb.com/docs/manual/administration/analyzing-mongodb-performance.md#number-of-connections) ## Related concepts - working set for time series — is a measure of MongoDB time-series working-set sizing and sharding