Time series backfill ordering and batching
Parent: Bucket fragmentation remediation (re-insert/merge) · Published reference · snapshot 2026-10-31
↓ Facts as markdownall context files
Source-anchored research for Time series backfill ordering and batching. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Keep BSON field order consistent across reinserted measurements, including nested fields, because the time-series best-practices guide associates consistency with better insert and compression performance. This field-layout recommendation is distinct from chronological document ordering. (low confidence) [source]
- A May 2025 first-person MongoDB 8 TSV-replay benchmark reported periodic insert latency variation across batch sizes 500–5000, with no accepted root cause in the retrieved thread. Treat this as an unresolved observation, not evidence that changing batch size or disabling write concern fixes the underlying issue. (low confidence) [source]
- For historical re-insertion, process measurements oldest to newest within each metadata series and group equal metaField values into batches. This combines the migration guide's chronological recommendation with the insert guide's metadata-local batching advice; it is a preparation strategy, not a guarantee of minimal bucket count. (medium confidence) [source]
Problems, failure modes and limitations
- The server README describes append-only compressed column builders: out-of-time-order arrivals can turn sorted V2 buckets into unsorted V3 buckets, particularly during low-cardinality concurrent bulk loads. V3 buckets can have worse query performance because queries cannot assume sorted measurements. Therefore, scheduling multiple writers against overlapping times for the same series deserves a workload-specific test. (low confidence) (verified as of 2026-10-14) [source]
- Do not assume larger batches alone resolve a slow historical load. A March 2023 MongoDB 6 user report found no noticeable improvement across batch sizes 100 through 100000 with locally sorted but globally unordered chunks. Separately, benchANT warns its time-series benchmark configuration cannot substitute for application-specific measurements. These support testing the actual replay workload, not adopting a universal batch-size target. (medium confidence) [source]
- SERVER-95067 documents potential silent loss of the first measurement on MongoDB 7.0.0–7.0.15 and 8.0.0–8.0.3 when retryable writes, multi-document ordered:false inserts, a qualifying metadata/time index (or time-only index without metadata), and concurrent writers targeting the same bucket coincide. Its January 29, 2025 advisory identifies fixes in 7.0.16 and 8.0.4 and ordered:true as a workaround if the index cannot be removed. Check this historical version boundary before applying unordered batching guidance. (low confidence) (verified as of 2025-01-29) [source]
- In a 2021 first-person late-arrival investigation, the user ultimately reported that late insertion worked and traced failures to invalid dates, timezone conversion and unvalidated client time. This is evidence against treating timestamp disorder as inherently invalid data; it does not quantify modern backfill throughput. (low confidence) [source]
- ordered:false does not promise timestamp order: insertMany() may reorder operations and continues after individual insert errors, whereas ordered writes stop after an insert error. A backfill loader using the recommended unordered mode must account for successful and failed writes separately. (medium confidence) [source]
- The insertMany() reference gives a 100,000-operation maxWriteBatchSize and says the high-level driver API splits larger input arrays. This protocol batching limit is not an optimal time-series backfill batch-size recommendation. (low confidence) (verified as of 2026-10-14) [source]
Comparisons and alternatives
- MongoDB recommends insertMany instead of repeated insertOne calls, batching identical metaField values together where possible, and ordered:false for time-series insert throughput. The recommendation does not specify one universally optimal application batch size. (low confidence) [source]
- The Database Tools migration example uses mongorestore --maintainInsertionOrder to preserve dump order. Its text says mongodump exports natural order; consequently this option preserves the source sequence rather than sorting an arbitrarily ordered dump by the measurement timestamp. (low confidence) [source]
- MongoDB recommends insertMany() rather than repeated insertOne() calls, batching identical metaField values when possible, and ordered:false for time-series insert throughput. QuestDB's March 2023 account independently describes MongoDB's metadata-grouped batching recommendation, but its experiment used MongoDB 6.0.4. (medium confidence) [source]
Facts and statements
- For a backfill intended to improve bucketing, use oldest-to-newest measurement time within each metadata series as the baseline schedule. The migration guide explicitly requires chronological insertion; the server README explains that descending-time batches previously caused inefficient bucketing. This is a performance baseline, not proof that every late insert is invalid. (medium confidence) [source]
- The server README's Batched Inserts section describes grouping each incoming batch by metadata and sorting it by ascending time before bucket targeting. It contrasts this with older paths that followed user order and could bucket descending input poorly. Treat this as implementation-specific evidence, not proof that every deployed release sorts batches. (low confidence) (verified as of 2026-10-14) [source]
- If Estuary Batch Incremental reads the time-series collection using timeField as its cursor, its documented correctness condition is strictly increasing cursor values for new documents. Inference: inserting older historical measurements behind an already advanced cursor violates that condition and needs a capture strategy review. Chronologically sorting only the backfill batch does not restore the global cursor condition. (low confidence) (verified as of 2026-10-27) [source]
- The Logtide observability benchmark explicitly describes a MongoDB metrics implementation without time-series-specific compression. Its reported batch-throughput result therefore does not establish the best batch size or ordering for a native time-series bucket-fragmentation backfill. (low confidence) [source]
- The server README says that, beginning in MongoDB 8.0, out-of-order arrivals into an append-only compressed V2 bucket can promote it to unsorted V3, particularly during low-cardinality concurrent bulk loads; queries may then lose sorted-bucket optimizations. Per-batch sorting therefore does not establish global ordering across concurrent batches. (low confidence) (verified as of 2026-10-14) [source]
- benchANT's cached Time Series: DevOps specification uses a 1,000-document batch, while QuestDB's 2023 native-time-series experiment uses ten loader workers. These are workload-specific experimental settings. Both publish configuration context, and benchANT explicitly warns against extrapolating its measurements to another application; tune batching and concurrency for the intended backfill. (medium confidence) [source]
Children
- No children recorded.