Pinned layered WiredTiger page-delta read reconstruction and corruption boundaries
Parent: WiredTiger checkpoints and dirty-page write cost · Published reference · snapshot 2026-11-02
↓ Facts as markdownall context files
Source-anchored research for Pinned layered WiredTiger page-delta read reconstruction and corruption boundaries. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- The disaggregated corruption path sets `WT_CONN_DATA_CORRUPTION`. It returns `WT_ERROR` when the block manager is verifying or `WT_SESSION_READ_CORRUPT_OK` permits the read; otherwise it enters the fatal-read panic path. These branches handle detected corruption; they contain no data-repair operation. (high confidence) [source 1] [source 2] [source 3] [source 4]
- The leaf merge drops a real base value whose stop is globally visible. It drops a selected delta carrying `WT_DELTA_LEAF_IS_DELETE` or a globally visible stop. An omitted base value cell denotes a live zero-length value with no time window; the merge bypasses the stop check for that case because its lookahead unpack structure may refer to the next key. A selected zero-length delta with an empty time window is emitted using the empty-value representation, and the packing helper omits its value cell. (medium confidence) [source 1] [source 2] [source 3]
How it works
- For an ordinary chain, the expected checksum starts with the address-cookie checksum. Each buffer must have the expected header checksum and pass a checksum recomputation with the stored checksum field zeroed; the reader then assigns `previous_checksum` as the expected checksum for the next older buffer. The broken-middle-link fixture relinks individually self-consistent newer blocks and expects `WT_ERROR` when the chain no longer links to the older block. (high confidence) [source 1] [source 2] [source 3]
- The leaf merge compares the next base key with the minimum delta key. A delta wins equality, and the matching base entry is consumed as well. The chosen delta is consumed even when its delete causes it to emit no output, preventing an older base value from reappearing. (low confidence) [source]
- The leaf merger avoids prefix compression for the first output key, then uses the btree prefix-compression setting. It finalizes entry count, image size, row-leaf type and timestamp page version, takes write generation from the newest delta, and initializes the block header. It sets all-empty/no-empty value flags only when the output has entries. (low confidence) [source 1] [source 2]
Problems, failure modes and limitations
- A result marked `WT_BLOCK_DISAGG_MODIFIED` must be the only returned image. For that offline-rewritten base, `previous_checksum` must match the cookie while the image checksum validates the rewritten bytes. The reader retains the cookie checksum and uses the page-log delta count for block metadata. The fixture expects a matching original checksum to succeed, a mismatching original checksum to fail, and body corruption under the full-image checksum to fail. (high confidence) [source 1] [source 2] [source 3]
- The conversion helpers reject missing decryptor or decompressor configuration and invalid compressed sizes, and treat a decompression result-length mismatch as an error. Their corruption handler sets the connection corruption flag and panics during ordinary reads, while btree verify or corruption-tolerant sessions receive the error. The multi-read transform also rejects an unencrypted original base when an encryptor is configured. (low confidence) [source 1] [source 2]
- The internal merger emits the first base key and its address before merging the remaining streams; its comment explains that customized collators cannot handle a truncated first key. For remaining keys, a delta wins equality and consumes the matching base entry. A delta whose raw value-cell type is `WT_CELL_ADDR_DEL_VISIBLE_ALL` is omitted. (high confidence) [source 1] [source 2]
- In diagnostic builds, the read path verifies the reconstructed image using the emitted time aggregate and `WT_VRFY_DISK_EMPTY_PAGE_OK | WT_VRFY_DISK_FROM_DELTA`; a verification failure triggers `WT_ASSERT_ALWAYS`. It copies the used image into a right-sized allocation and frees the original oversized buffer and input base/delta buffers. (low confidence) [source 1] [source 2]
- The read path can transfer the reconstructed image to the shared disk cache; on a concurrent cache insertion it frees its redundant image and uses the cached one. `__wti_page_inmem` attaches the selected image to the new page and marks shared ownership when a shared item is supplied. The read error path checks whether a page took ownership before freeing the reconstructed copy, and restores the previous reference state. A shared disk-image cache hit bypasses disk multi-read and reconstruction. Cache insertion transfers ownership by clearing the local memory pointer. The error path discards a built page or releases a shared disk item, frees remaining buffers, clears the reading flag and restores the previous reference state; success clears the reading flag and sets the reference to its in-memory state. (high confidence) [source 1] [source 2]
- The checksum-prefix counterexample is limited and explicit: the reader covers the whole returned image only when the data-checksum flag is present; otherwise it covers the smaller of the image size and `WT_BLOCK_COMPRESS_SKIP`. The fixture with no data-checksum flag flips body byte `300` and asserts a successful read with no corruption flag, while a mutation at `WT_BLOCK_COMPRESS_SKIP - 8` is asserted to fail. This rules out an unconditional claim that this checksum step detects all body corruption. (high confidence) [source 1] [source 2] [source 3]
- Delta leaf value-format decoding uses an always assertion when custom metadata unpacking fails, and diagnostic reconstructed-image validation also uses an always assertion. These explicit assertions are separate boundaries from the checksum/header corruption branch that can return an error in verify or corruption-tolerant mode. (medium confidence) [source 1] [source 2] [source 3] [source 4]
Comparisons and alternatives
- After checksum validation, the reader checks base-versus-delta magic, reader compatibility and combined header size. It applies the version-one header-size fix before the size check and byte-swaps the page header only after those checks pass. The combined header size must hold the minimum headers, remain within the format maximum, and fit inside the block. The version-one correction is conditional on the debug setting and changes both the image and swapped header. (high confidence) [source 1] [source 2] [source 3] [source 4]
- When more than one buffer was read, `__page_read` passes the base buffer plus the remaining delta buffers to a full-image builder. That builder selects the leaf merge for a row-leaf base and the internal merge otherwise. Both merge implementations append cells directly into one reconstructed disk image rather than constructing an intermediate page for each delta. (medium confidence) [source 1] [source 2]
Facts and statements
- The address-cookie multi-read wrapper unpacks the page ID, LSN, base LSN, size and expected checksum and passes them to the disaggregated reader. The page-log callback supplies the result buffers; the reader walks from the highest result index back to the base at index zero. (medium confidence) [source 1] [source 2]
- The multi-buffer cache reader may use a cached base-and-delta set or call the block manager. It decrypts before decompressing. Base-image conversion flags come from the page header; delta conversion flags come from the block header. It returns an array with the base first and subsequent delta buffers for reconstruction. (high confidence) [source 1] [source 2] [source 3] [source 4]
- Both minimum-delta selectors scan streams from the highest index toward the lowest. They compare keys with the btree collator, prefer the newer stream on equal keys, and consume older duplicates by clearing the unpacked flag and reducing the remaining cell count by two. The internal merge state advances its input pointer during unpacking, before consumption. Leaf selection expands prefix-compressed keys before comparison; the internal selector uses decoded internal keys. (low confidence) [source 1] [source 2] [source 3] [source 4]
- For retained leaf values, `__time_window_clear_obsolete` clears a globally visible start timestamp, durable start timestamp and transaction ID, but does not clear stops. The caller drops globally visible stops instead; surviving time windows are passed to cell packing and contribute to the diagnostic time aggregate. (medium confidence) [source 1] [source 2]
- Internal key/address packing selects the winning base or delta key, address bytes, address cell type, page-delete information and time aggregate. It grows the output buffer and recomputes its write pointer after growth. The pinned helper explicitly passes `false` for prepared fast-truncate handling and carries `FIXME-WT-17663` to pass the correct value; address preservation must retain that qualification. (high confidence) [source 1] [source 2]
- The internal merger aggregates emitted child time aggregates in diagnostic builds and finalizes a row-internal image with updated entry count, image size, newest-delta write generation, timestamp page version and `WT_PAGE_FT_UPDATE`, then initializes its block header. (low confidence) [source 1] [source 2] [source 3]
- The checksum fixture forges block images and returns them through a stub page log. Its assertions cover a valid base and valid chain, cookie mismatches, a broken predecessor link below the newest block, and whole-image body corruption. These are attributed fixture assertions about supplied bytes. The fixture uses a real connection and session with a fabricated block manager and tree. It normally sets the quiet-corruption session flag. (low confidence) [source 1] [source 2]
- A separate fixture section clears the quiet-corrupt session flag, enables block-manager verify, corrupts the body and asserts `WT_ERROR`, a set corruption flag and an unset `WT_CONN_PANIC`. Header-validation sections assert rejection of base/delta magic reversal and a newer required reader version after creating self-consistent checksums. (low confidence) [source 1] [source 2]
- Leaf delta unpacking decodes the key and value with that delta image header and extracts flags plus actual value bytes from the custom delta value format. Internal delta unpacking instead resolves cell values against the base header while deciding transaction-ID cleanup from the delta header write generation. The source explains that using an older base generation for this cleanup would incorrectly clear current-run delta transaction IDs. (medium confidence) [source 1] [source 2] [source 3]
Children
- No children recorded.