Streaming BSON Parser Architecture

Parent: Diagnostic Tool Design · Published reference · snapshot 2026-10-15

Depth-first rabbithole dossier for Streaming BSON Parser Architecture; source-anchored research pack.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

[low] BSON is a binary format storing zero or more ordered key/value pairs as a single entity, with each type serialized in little-endian format. Sources: https://bsonspec.org/spec.html [source]
[low] A BSON document is defined as an `int32` total byte count followed by an element list and a trailing `unsigned_byte(0)`, where the `int32` is the total number of bytes constituting the document. Sources: https://bsonspec.org/spec.html [source]
[low] BSON `ObjectId` is a `signed_byte(7)` followed by an element name and 12 bytes of value, and `datetime` is a `signed_byte(9)` followed by an element name and an `int64` representing UTC milliseconds since the Unix epoch. Sources: https://bsonspec.org/spec.html [source]
[low] BSON `Timestamp` is a `signed_byte(17)` followed by a `uint64`, where the first 4 bytes are an increment and the second 4 bytes are a timestamp, used internally by MongoDB replication and sharding. Sources: https://bsonspec.org/spec.html [source]
[low] BSON `string` is an `int32` length followed by bytes and a trailing null byte, where the `int32` is the number of bytes in the string plus one for the trailing null byte. Sources: https://bsonspec.org/spec.html [source]
[low] BSON `binary` is an `int32` length followed by a subtype and bytes, where the `int32` is the number of bytes in the binary data. Sources: https://bsonspec.org/spec.html [source]
[low] A BSON array is a document with integer values for keys starting at 0 and continuing sequentially. Sources: https://bsonspec.org/spec.html [source]
[low] A BSON vector is a dense array of numeric values stored in a binary format efficient for storage and retrieval, where all values within the vector must be of the same type. Sources: https://bsonspec.org/spec.html [source]

Structure and components

[low] The `bson_reader_t` structure is used for reading a sequence of BSON documents from a file descriptor, memory region, or custom callbacks. Sources: https://mongoc.org/libbson/current/bson_reader_t.html [source]
[low] BSON regex consists of a first cstring for the pattern and a second cstring for the options string. Sources: https://bsonspec.org/spec.html [source]

How it works

[low] Parsing large blobs of BSON is difficult with certain libraries because they require pulling the entirety of the BSON blob into memory. Sources: https://jira.mongodb.org/browse/NODE-2793 [source]
[low] In `wirebson`, accessing an index out of bounds or encountering invalid BSON types causes a panic. Sources: https://pkg.go.dev/github.com/FerretDB/wire/wirebson [source]

Problems, failure modes and limitations

[low] The `wirebson` `Equal` function panics if an invalid BSON type or decoding error is encountered, so it should not be used with user input. Sources: https://pkg.go.dev/github.com/FerretDB/wire/wirebson [source]

Facts and statements

[low] The `BSONEachRow` format parses data as a sequence of Binary JSON (BSON) documents without separators, where each row is a single document and each column is a BSON field with the column name as a key. Sources: https://clickhouse.com/docs/interfaces/formats/BSONEachRow [source]
[low] The `BSONEachRow` format supports parsing big integers and decimals such as `Int128`, `UInt128`, `Int256`, `UInt256`, `Decimal128`, and `Decimal256` from BSON Binary values with the `\x00` subtype, validating that the binary data size equals the expected value size. Sources: https://clickhouse.com/docs/interfaces/formats/BSONEachRow [source]
[low] The `wirebson` package implements BSON encoding and decoding, passing composite types (Document and Array) by pointers while raw composite types and scalars are passed by values. Sources: https://pkg.go.dev/github.com/FerretDB/wire/wirebson [source]
[low] BSON data types include `int32` (4 bytes), `int64` (8 bytes), `double` (8 bytes), and `decimal128` (16 bytes). Sources: https://bsonspec.org/spec.html [source]
[low] BSON binary subtypes include `unsigned_byte(0)` for generic binary, `unsigned_byte(4)` for UUID, `unsigned_byte(6)` for encrypted BSON, `unsigned_byte(8)` for sensitive data, and `unsigned_byte(9)` for vectors. Sources: https://bsonspec.org/spec.html [source]
[low] Compressed BSON column data types use delta and delta-of-delta compression and run-length-encoding for efficient element storage. Sources: https://bsonspec.org/spec.html [source]
[low] The `deserializeStream` method in some BSON libraries does not accept piped streams and still requires a buffer to be passed in. Sources: https://jira.mongodb.org/browse/NODE-2793 [source]
[low] The `bsondump` tool converts BSON files into human-readable formats, including JSON, using Extended JSON v2.0 (Canonical mode) to format its data. Sources: https://www.mongodb.com/docs/database-tools/bsondump.md [source]
[low] By default, `bsondump` enables `--objcheck` to validate each BSON object before outputting it in JSON format; starting in MongoDB 9.0, this option has no effect on server behavior as `bsondump` always validates BSON objects. (verified as of 2026-10-15) Sources: https://www.mongodb.com/docs/database-tools/bsondump.md [source]

Related concepts

BSON — is a part of Streaming BSON Parser Architecture

Children

No children recorded.

← the whole tree · 3D view· how to read this page