mongodb-time-series

Parent: mongodb-schema-design · researched 2026-05-28T16:45:08.510Z· 12 sources · 15 concepts · skill mongodb-time-series

MongoDB Time Series Collections, introduced in MongoDB 5.0 (GA), are a specialized collection type optimized for time-stamped measurement data. They use an internal columnar storage format with automa

Overview

MongoDB Time Series Collections, introduced in MongoDB 5.0 (GA), are a specialized collection type optimized for time-stamped measurement data. They use an internal columnar storage format with automatic bucketing, delta encoding, and Zstd compression to achieve 50-90% storage reduction over regular collections while dramatically improving query performance for time-range access patterns. [source]
Time series collections are the preferred choice over the manual bucket pattern for IoT sensor data, server metrics, financial tick data, application events, observability signals, and any domain where data is appended in timestamp order and queried by time range. [source]
> Skill boundaries: [source]
> - Use this skill (mongodb-time-series) for: collection creation options, bucket internals, TTL, time-series-specific index constraints, $densify/$fill/$dateTrunc/$setWindowFields in a time-series context, Atlas triggers/change-stream limitations, migration, sharding for time series, and performance sizing. [source]
> - Use mongodb-aggregation-pipeline for: general pipeline stage design, $lookup, $merge/$out to regular collections, explain profiling, memory/allowDiskUse tuning. [source]
> - Use mongodb-schema-design for: the manual bucket pattern, embedding vs referencing decisions, general IoT schema modeling without the native time series collection type. [source]
> - Use mongodb-indexes-deep for: ESR compound index design, partial/sparse/wildcard/text index types on regular collections. [source]
Version timeline: [source]
MongoDB 5.0: Initial release (create, insert, query, TTL, basic indexing) [source]
MongoDB 5.1: $densify aggregation stage [source]
MongoDB 5.2: Columnar compression format (major storage improvement) [source]
MongoDB 5.3: $fill aggregation stage [source]
MongoDB 6.0: partial index support with $or/$in/$geoWithin [source]
MongoDB 6.3: Custom bucketing parameters (bucketMaxSpanSeconds, bucketRoundingSeconds) [source]
MongoDB 7.0: $out can write to time series collections; TTL partial filter on metaField [source]
MongoDB 8.0: Block processing - direct write into column-compressed format (2-3x throughput, 10-20x cache reduction); timeField shard key deprecated [source]
MongoDB 8.3: timeField cannot start with $; creating "_id_" index returns error [source]
Atlas (2023+): Atlas Stream Processing introduced - time series collections can be a sink but not a $source (no change stream support) [source]
MongoDB Time Series Documentation [source]
MongoDB 8.0 Block Processing Blog [source]
Columnar Storage Cost Savings Blog [source]

1. Collection Creation and Configuration

Time series collections are created with db.createCollection() using a timeseries subdocument. The timeField is the only required parameter; all others are optional but significantly affect performance. [source]
Parameter reference: [source]
Granularity and bucket time spans: [source]
Changing parameters after creation: Use collMod to update granularity, bucketMaxSpanSeconds, bucketRoundingSeconds, and expireAfterSeconds. You can only increase bucket span, never decrease it. timeField and metaField are permanently immutable. [source]
Time Series Considerations [source]
Create and Query Procedures v7.0 [source]
Community: Granularity and metaField [source]

2. Internal Bucket Architecture

MongoDB stores time series documents in internal system.buckets.<collectionName> bucket documents, not as individual BSON records. The view layer (<collectionName>) presents unpacked measurements to applications. [source]
Bucket structure: [source]
Bucket lifecycle: [source]
A bucket is opened when the first measurement for a given metaField value arrives. [source]
A bucket is closed when it reaches either ~1,000 measurements OR ~125 KB, whichever comes first, or when the bucket's time span limit (determined by granularity) is exceeded. [source]
Closed buckets are compressed and eligible for WiredTiger cache eviction. [source]
Compression mechanisms (MongoDB 5.2+): [source]
Column-oriented storage: Values for each field (temperature, humidity, pressure) are stored together rather than per-document. This enables delta encoding and RLE to be applied across entire columns. [source]
Delta encoding: Stores the first value absolutely, then subsequent values as differences (+0.1, -0.2). Highly effective for monotonic timestamps and slowly changing sensor readings. [source]
Run-Length Encoding (RLE): Repeated values (e.g., status: "active" for 1,000 measurements) stored as (value, count). [source]
Metadata deduplication: Field names and BSON types stored once per bucket rather than per document. [source]
Zstd block compression (WiredTiger level): Applied on top of the already column-compressed data. [source]
MongoDB 8.0 block processing: Documents are written directly into column-compressed format, eliminating the decompression-recompression cycle on write. This results in 2-3x write throughput improvement and 10-20x cache usage reduction compared to MongoDB 7.0 for time series workloads. [source]
Columnar Storage Blog [source]
Time Series Compression [source]
High vs Low Ingestion Study [source]
MongoDB 8.0 Block Processing [source]

3. Secondary Indexes

Time series collections index at the bucket level, not the document level. The control.min and control.max values on each bucket function as a clustered range index that enables bucket-level pruning for time-range queries. [source]
Default clustered index: A clustered index on the metaField and timeField is automatically created. No explicit _id index is created (unlike regular collections). In MongoDB 5.0, only a single compound index on metaField + timeField was supported; secondary indexes on measurement fields were added in later versions. [source]
> Skill boundary: For general compound index design (ESR rule, multikey, partial, sparse, wildcard), use mongodb-indexes-deep. This section covers only time-series-specific index constraints and patterns. [source]
Supported secondary index types: [source]
Adding a compound secondary index: [source]
Key indexing constraints: [source]
partialFilterExpression can only reference the metaField (not measurement fields). [source]
Unique indexes are not supported - duplicate prevention must be handled at the application layer or using $match + $group in aggregation. [source]
Text indexes are not supported - consider Atlas Search for full-text needs. [source]
The distinct() command is not efficiently supported; use $group with a supporting compound index instead. [source]
Query on object metaField - use sub-field dot notation: [source]
Time Series Indexes [source]
Add Secondary Indexes [source]
Time Series Limitations [source]

4. TTL and Automatic Data Expiration

Time series collections support bucket-granularity TTL via expireAfterSeconds. Unlike regular collection TTL indexes (which delete individual documents), TTL on time series collections deletes entire buckets once all measurements within the bucket are older than the threshold. [source]
Modify after creation (cannot use createIndex): [source]
The background TTL task runs every 60 seconds. [source]
A bucket is deleted only when all measurements in that bucket have expired (i.e., control.max.timestamp + expireAfterSeconds < now). [source]
Because of bucket aggregation, actual deletion may be delayed by up to bucket-span + 60s after expiration. [source]
A bucket created with granularity: "hours" covering a 30-day span won't be deleted until all 30 days of measurements within it have expired. [source]
Tiered TTL pattern (MongoDB 7.0+ with partial filter): [source]
TTL for Time Series [source]
TTL Indexes [source]

5. Aggregation Pipeline — Time Series Optimizations

MongoDB provides three specialized aggregation stages that are particularly valuable for time series analysis: [source]
> Skill boundary - aggregation stages: For general aggregation pipeline design ($lookup, $group, $merge, $out, explain profiling, memory limits), use mongodb-aggregation-pipeline. This section covers only time-series-optimized stages ($densify, $fill) and time-series-specific $setWindowFields usage. $dateTrunc (for downsampling into time buckets) is also covered here as it is the primary time-bucketing operator. [source]
#### $dateTrunc - Time-Bucket Downsampling [source]
$dateTrunc truncates a date to a specified granularity boundary. It is the canonical operator for downsampling raw measurements into fixed time buckets (minute candles, hourly rollups, daily aggregates). [source]
The binSize parameter (MongoDB 5.0+) groups dates into multiples of the unit - e.g., binSize: 5, unit: "minute" snaps all timestamps to 5-minute boundaries. [source]
#### $densify (MongoDB 5.1+) [source]
Fills gaps in a time series by inserting synthetic documents at regular intervals where data is missing. Critical for dashboards and window function inputs that assume uniform spacing. [source]
"full" - spans min to max across all documents in the collection. [source]
"partition" - spans min to max within each partition group. [source]
[lower, upper] - explicit range; lower inclusive, upper exclusive. [source]
#### $fill (MongoDB 5.3+) [source]
Populates null or missing fields in densified documents using interpolation or last-observed-carry-forward (LOCF). [source]
"linear" - calculates value proportionally between surrounding non-null values. [source]
"locf" (Last Observation Carried Forward) - repeats the last known non-null value. [source]
#### $setWindowFields (MongoDB 5.0+) [source]
Applies window functions over ordered partitions without collapsing documents (unlike $group). Enables rolling averages, cumulative sums, lag/lead comparisons, and rankings - all SQL-standard window function patterns. [source]
documents: ["unbounded", "current"], [-N, M] - count-based. [source]
range: [-N, M] with unit for time-based (ms, second, minute, hour, day, week, month, quarter, year). [source]
Important performance note: Window functions on time series collections do not automatically push down through the bucket storage format. Use $match on metaField and timeField before $setWindowFields to minimize the scanned document set. [source]
$densify Reference [source]
$setWindowFields Reference [source]
Percona Window Functions in MongoDB 5.0 [source]
MongoDB Developer: time-series-window-functions [source]

6. Atlas-Specific Features

#### Atlas Charts Integration [source]
Atlas Charts works natively with time series collections. The time-series-optimized aggregation engine (bucket-level pruning, columnar projection) applies to Charts queries automatically - no special configuration needed. [source]
Use cases with Atlas Charts: [source]
Real-time IoT sensor dashboards using time-range filters [source]
Environmental monitoring with rolling average overlays [source]
Infrastructure metrics with aggregated panels (mean, p95, max) [source]
Financial dashboards showing OHLCV candlestick data [source]
Limitation: Embedded charts querying time series collections with a high-cardinality metaField can generate expensive scatter-gather queries. Use time-range and metaField equality filters in the embedded chart filter to scope queries. [source]
Visualizing Atlas Data with Charts [source]
IoT + Atlas Charts Example [source]
#### Atlas Triggers - Not Supported [source]
Time series collections do not support change streams and therefore cannot use Database Triggers. The optimized bucket storage format does not emit per-document change events. [source]
Workaround patterns: [source]
Dual-write to a regular collection: Write events to both a regular collection (for triggers) and a time series collection (for historical queries). The regular collection can be capped or have a short TTL. [source]
Scheduled triggers: Use scheduled Atlas triggers to run aggregations over the time series collection at regular intervals and emit derived events or aggregated results to another collection. [source]
Atlas Stream Processing: Use Kafka or Atlas Stream Processing $source stage to consume events before they enter the time series collection and react in real-time. Note: time series collections cannot serve as a $source in ASP. [source]
Community: Change Stream Workaround [source]
Triggers Limitations [source]
#### Atlas Flex Clusters [source]
Atlas Flex clusters (the replacement for M2/M5 and Serverless instances, as of January 2026) support time series collections as they run MongoDB 5.0+ wire protocol. However, Flex clusters have limitations compared to Dedicated clusters: [source]
No Continuous backup / Point-in-Time Restore (snapshots only) [source]
No cross-region replication [source]
Private Endpoints support is limited - verify current availability in the Atlas docs, as Flex private endpoint support has been expanding since 2025 [source]
For production time series workloads requiring PITR, guaranteed HA, or private networking, use Dedicated clusters (M10+). [source]
Manage Flex Clusters [source]
Flex Migration Guide [source]

7. Sharding Time Series Collections

Sharding enables horizontal scaling for very high ingestion rates. Time series sharding has several important constraints that differ from regular collection sharding. [source]
Shard key must contain only the metaField, sub-fields of metaField, or (deprecated) the timeField. [source]
timeField as a shard key component is deprecated in MongoDB 8.0 because monotonically increasing values cause write hotspots on a single shard. [source]
metaField can be used as a ranged or hashed shard key. [source]
Zone sharding is not supported for time series collections. [source]
Recommended shard key patterns: [source]
Anti-pattern - timeField-only shard key: [source]
Pre-splitting: If device groups or regions are known in advance, pre-split chunks before ingestion to avoid initial primary-shard hotspot. [source]
Shard a Time Series Collection [source]
Time Series Limitations - Sharding [source]

8. Performance Benchmarks and Working Set Sizing

#### Storage Compression [source]
#### Write Performance (MongoDB 8.0 vs 7.0) [source]
#### Working Set Sizing for Time Series [source]
Unlike regular collections where the working set is the "hot" subset of documents, for time series the working set is primarily: [source]
Open buckets (currently being written) - proportional to metaField cardinality. [source]
Recently queried time ranges - based on your typical query lookback window. [source]
Granularity bucket span seconds reference: [source]
seconds granularity → 3,600 s (1 hour) [source]
minutes granularity → 86,400 s (24 hours) [source]
hours granularity → 2,592,000 s (30 days) [source]
Example (10,000 IoT sensors, minutes granularity = 86,400 s span, 1-hour lookback): [source]
(At seconds granularity the same 1-hour lookback covers exactly 1 bucket span, so recent-query RAM ≈ open-bucket RAM = ~1.2 GB - an important difference when choosing granularity.) [source]
Recommendation: Size WiredTiger cache (storage.wiredTiger.engineConfig.cacheSizeGB) at 50-60% of available RAM, targeting > 95% cache hit rate. Monitor wiredTiger.cache.bytes currently in the cache and page faults in Atlas metrics. [source]
Columnar Storage Cost Savings Blog [source]
Time Series Compression Docs [source]
Bucket Behavior Study [source]
Medium: Storage Comparison [source]

Pattern 1: IoT Multi-Sensor Ingestion

Batched insertion (critical for performance): [source]

Pattern 5: Versioning for Correctable Measurements

Time series collections cannot update measurement fields. Use the versioning pattern to handle corrections: [source]
Source: Versioning Pattern with Time Series Data [source]

Migration: Regular Collection to Time Series

You cannot convert an existing collection in-place. Migration always requires creating a new time series collection and copying data. [source]

Method 3: Kafka Connector (streaming cutover)

For live production systems with continuous ingestion, use the MongoDB Kafka Connector to dual-write during cutover: [source]
Configure source connector reading from existing collection. [source]
Configure sink connector writing to new time series collection. [source]
Once data is synced and validated, cut application writes over to the time series collection. [source]
Drain and stop connectors. [source]
Migrate with Aggregation Pipeline [source]
Migrate with Database Tools [source]
Kafka Connector Migration Tutorial [source]

Anti-Pattern 2: Wrong Granularity for Ingestion Rate

Mismatch 1 - granularity too coarse (high-frequency data): Setting granularity: "hours" for a sensor that reports every second means each bucket can remain open for up to 30 days before the time limit triggers a close. In practice, the measurement-count limit (~1,000 documents) is hit first (after ~17 minutes at 1/s), but this still produces far more bucket churn than needed and misrepresents the intended data cadence to the storage engine, degrading compression locality. [source]
Mismatch 2 - granularity too fine (low-frequency data): Setting granularity: "seconds" for a sensor that only reports once per hour means each bucket closes after 1 hour (time limit), typically containing only ~1 measurement. This destroys compression - you lose all the benefit of columnar storage across many measurements. [source]

Anti-Pattern 3: High metaField Cardinality with Unbounded Values

Each unique metaField value maintains a separate open bucket in the working set. If metaField includes a UUID or a user-specific ID that changes per request, the working set explodes. [source]

Issue: Buckets are Too Large / Too Small

Fix: Adjust granularity or bucketMaxSpanSeconds with collMod. Remember: you can only increase span, not decrease it. [source]

Issue: Queries Are Slow Despite Indexes

Look for COLLSCAN on system.buckets.* - this indicates missing indexes or the query optimizer not using bucket-level pruning. [source]
Querying measurement fields in $match without preceding metaField filter. [source]
Not using dot notation on metaField sub-fields. [source]
Missing compound index for the combination of metaField sub-field + timeField. [source]

Issue: High Memory / WiredTiger Cache Pressure

Symptoms: High cache utilization, frequent evictions, rising page faults. [source]
Diagnosis: High metaField cardinality generating too many open buckets. [source]
Reduce metaField cardinality by grouping sensors into logical partitions. [source]
Increase granularity to close buckets faster (shorter time span per bucket). [source]
Upgrade to MongoDB 8.0 for 10-20x cache reduction from block processing. [source]
Scale up cluster tier (more RAM) or scale out (sharding). [source]

Issue: TTL Not Deleting Data

Verify expiration config: [source]
expireAfterSeconds was never set at creation (default: no expiration). [source]
Bucket span is too large - the bucket won't delete until ALL measurements in it expire. [source]
Background TTL task has lag (up to 60s + bucket span after last measurement expires). [source]

References

MongoDB Time Series Collections - Official Documentation - Core reference for all time series features. [source]
Time Series Limitations - Comprehensive list of unsupported operations. [source]
Best Practices for Time Series Collections - Official best practices: compression, batching, metaField design. [source]
Columnar Storage Cost Savings - MongoDB Engineering Blog - Delta encoding, RLE, and Zstd compression mechanics with benchmarks. [source]
MongoDB 8.0 Block Processing - 2-3x throughput and 10-20x cache reduction from direct columnar writes. [source]
High vs Low Ingestion Bucket Behavior Study - Empirical study of granularity impact on bucket lifecycle under different ingestion rates. [source]
$densify Reference - Full parameter reference and examples. [source]
$setWindowFields Reference - Window function accumulator and range options. [source]
Migrate Data into a Time Series Collection - Official migration procedures. [source]
Versioning Pattern with Time Series Data - Pattern for handling measurement corrections. [source]
Window Functions and Time Series Performance - Medium - Performance analysis of $setWindowFields with time series collections. [source]
Shard a Time Series Collection - Sharding rules and shard key selection. [source]

Children

Time Series Collection Creation (timeField, metaField, granularity) (frontier)
Bucket Architecture and Columnar Compression (frontier)
Secondary Indexes on Time Series (frontier)
TTL and Automatic Bucket Deletion (frontier)
$densify and $fill Gap Filling (frontier)
$dateTrunc Downsampling (frontier)
$setWindowFields Window Functions (frontier)
Atlas Charts Integration (frontier)
Atlas Triggers Workarounds (no change streams) (frontier)
Time Series Sharding Patterns (frontier)
Working Set Sizing for Time Series (frontier)
Migration from Regular Collections (frontier)
Granularity Anti-Patterns (frontier)
metaField Cardinality Anti-Patterns (frontier)
MongoDB 8.0 Block Processing (frontier)

Frontier under this node: $dateTrunc Downsampling, $densify and $fill Gap Filling, $setWindowFields Window Functions, Atlas Charts Integration, Atlas Triggers Workarounds (no change streams), Bucket Architecture and Columnar Compression, Granularity Anti-Patterns, Migration from Regular Collections, MongoDB 8.0 Block Processing, Secondary Indexes on Time Series, TTL and Automatic Bucket Deletion, Time Series Collection Creation (timeField, metaField, granularity), Time Series Sharding Patterns, Working Set Sizing for Time Series, metaField Cardinality Anti-Patterns

← the whole tree · 3D view· how to read this page