FTDC Chunk Decompression & Stream Parsing

Parent: FTDC and Log Tooling · Published reference · snapshot 2026-10-15

Depth-first rabbithole dossier for FTDC Chunk Decompression & Stream Parsing; source-anchored research pack.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

[low] The Compressed Chunk is a delta, run length encoded, and varint compressed array of numbers, where all numbers are encoded little endian. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] The metrics_chunk is not a BSON document but raw bytes containing an uncompressed_size uint32_t and a zlib compressed_chunk uint8_t array. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] Each FTDC file is a bare concatenation of BSON documents with no header, index, or framing beyond BSON's leading int32 length, requiring readers to advance by each document's length. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] In FTDC statistics, Value is the metric value at the beginning of the sample, Avg is the mean of the metric's deltas (first derivative), and Var is the variance related to the absolute second derivative. Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]

Structure and components

[low] In the FTDC chunk structure, counts come after the reference document, not before it, and the total samples in the chunk equals deltaCount + 1. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[high] FTDC captures only metadata and does not inspect network packets, include storage capacity data, or contain user data, serving as the first artifact MongoDB support requests for performance tickets. Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md, https://www.percona.com/blog/the-diagnostic-data-mongodb-atlas-doesnt-hand-you/ [source]
[low] The archive file is an append-only file readable with bsondump, while the interim file is constantly overwritten and always contains just one document of the metric type representing the most recent uncompressed metrics sample. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] FTDC metric documents have a structure where metric type 1 contains a DateTime _id and BinData(0) data field, while metadata_delta type 2 contains a DateTime _id, int32 count, and collectors_doc or collectors_delta_doc. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] The compressed_chunk contains a reference_document BSON Document, metric_count uint32_t, sample_count uint32_t, and a compressed_metrics_array uint8_t array. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] The Clip function trims the chunk to contain as little data as possible while keeping data within the given interval, returning false and leaving the chunk unmodified if it is entirely outside the range. Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]

How it works

[low] FTDC thread failures are fatal and stop the parent mongod or mongos process. Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md [source]

Measurements and reference values

[low] From the busy-replset chunk 0 on mongod 6.0.26, there are 1,685 columns, and doubles constitute 1.4% of columns, requiring no path of their own since mongod stores them as int64. (verified as of 2026-10-15) Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]

Problems, failure modes and limitations

[low] FTDC data is stored only on the local host and is lost when the host is terminated or rebuilt; it cannot be used for diagnostics if forwarded by OTel or other log shipping pipelines. Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md [source]
[low] In rare circumstances, a write can fail due to cache pressure, causing MongoDB to issue a TemporarilyUnavailable error and increment the temporarilyUnavailableErrors counter in the slow query log and FTDC. Sources: https://www.mongodb.com/docs/manual/reference/parameters.md [source]

Comparisons and alternatives

[low] FTDC reports utilization statistics from the container's perspective, calculating memory utilization against the container's RAM limit rather than the host operating system's limit. Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md [source]
[low] Trailing whitespace in FTDC metric paths is significant, and a reflexive .trim() makes those metrics permanently unresolvable with a symptom of 'no such metric' rather than a parse error. (verified as of 2026-10-15) Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]

Facts and statements

[low] The entire FTDC chunk block is compressed with the zlib compression algorithm. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] The last chunk of an FTDC file is routinely truncated mid-write, so a decoder must return every fully decoded chunk and treat a short read at EOF as normal termination, not corruption. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] A path absent from an FTDC chunk produces NaN for that chunk's sample range, and decoders should not shift or forward-fill during decode. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] The Chunks function takes an FTDC diagnostic file as an io.Reader and yields metric chunks (type 1) on a channel, silently skipping type 0 and type 2 documents for backward compatibility. Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]
[low] In the FTDC Go library, Chunk represents a metric chunk (type 1), DocMetadata is a metadata document (type 0) collected on startup/rotation, DocMetricChunk is a compressed metric chunk (type 1), and DocPeriodicMetadata is a periodic metadata document (type 2) new in MongoDB ~7.x. (verified as of 2026-10-15) Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]
[high] FTDC data files are compressed and not human-readable, containing no user data, database names, or collection names, only counters and metrics. (verified as of 2026-10-15) Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md, https://www.percona.com/blog/the-diagnostic-data-mongodb-atlas-doesnt-hand-you/, https://github.com/saqibameen86/mdbkit [source]
[high] FTDC is enabled by default and writes approximately 5,700 metrics every second into a folder called diagnostic.data. (verified as of 2026-10-15) Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md, https://www.percona.com/blog/the-diagnostic-data-mongodb-atlas-doesnt-hand-you/ [source]
[low] FTDC writes two types of files in diagnostic.data: archive files named metrics.%Y-%m-%dT%H-%M-%SZ-CCCCC and an interim file named metrics.interim. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] The FTDC compressor extracts uint64_t numbers from BSON fields of types double, int64, int32, boolean, datetime, decimal128, and timestamp, converting all numeric fields to uint64_t. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] Timestamps are extracted as two uint64_t values (seconds followed by increment), and doubles are handled with NaN encoded as zero, values less than MIN_INT64 stored as MIN_INT64, and values greater than MAX_INT64 stored as MAX_INT64. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] To reduce space, the FTDC compressor subtracts the previous sample's value from the current sample, using the reference document's values as the baseline for the first sample after the reference. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] A sequence of zeros is compressed to a pair of numbers [0, x] where x is a non-zero positive integer indicating the number of additional zeros in the sequence. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] Metadata delta files are plain BSON using a simple diff algorithm to de-duplicate data, resetting delta tracking and emitting the full sample if the sequence of field names changes. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/src/mongo/db/ftdc/README.md [source]
[low] Decoders should validate that metricsCount equals flattenedMetrics.length and reject the chunk if it differs, as this is the cheapest corruption detector. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] Edge cases for FTDC numeric encoding include NaN becoming 0, values >= 2^63 and +Inf becoming INT64_MAX, and values < -2^63 and -Inf becoming INT64_MIN. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] FTDC samples are nominally 1 second apart but irregular spacing and inter-chunk gaps are real signal indicating mongod was down, stalled, or the host froze, so every sample's time must be derived from the column. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] Three duplicated paths out of 2,203 in FTDC data can corrupt consumers treating the key list as a set, as metricsCount counts columns and duplicates occupy their own columns with their own data. (verified as of 2026-10-15) Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]
[low] FTDC files are compatible with those produced by MongoDB 4.x through 8.x. (verified as of 2026-10-15) Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]
[low] To disable FTDC, start mongod or mongos with the diagnostic data capture disabled. Sources: https://www.mongodb.com/docs/manual/administration/full-time-diagnostic-data-capture.md [source]
[low] The Expand function accumulates all deltas to give values of diagnostic data for each sample represented by the Chunk. Sources: https://pkg.go.dev/github.com/jenunes/ftdc-utils [source]
[low] Fast-sampling with diagnosticDataCollectionPeriodMillis=100 cuts chunk time from 5 minutes to 30 seconds. Sources: https://raw.githubusercontent.com/zelmario/Big-hole/main/docs/ftdc-format.md [source]

Related concepts

FTDC — is a part of FTDC Chunk Decompression & Stream Parsing
Chunk — is a part of FTDC Chunk Decompression & Stream Parsing

Children

No children recorded.

← the whole tree · 3D view· how to read this page