# Pinned pipelined oplog applier multikey catalog publication ordering > Source-anchored research for Pinned pipelined oplog applier multikey catalog publication ordering. Run status: COMPLETED. Parent: [Pinned pipelined oplog applier in-flight backpressure and publication boundaries](https://llms-explorer.com/tree/pinned-pipelined-oplog-applier-in-flight-backpressure-and-publication-boundaries/) · 5 facets · 30 facts · page: https://llms-explorer.com/tree/pinned-pipelined-oplog-applier-multikey-catalog-publication-ordering/ ## Definitions - The explicit replicated setMultikeyMetadata command is a scoped exception to worker deferral: its fixture expects an empty worker metadata result and a directly updated multikey catalog path. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) ## Structure and components - Multikey identity consists of collection UUID, index name, and recorded earliest timestamp. Equal identities merge immediately; different timestamps for the same index remain separate. Merging adopts paths when the destination is empty, otherwise unions corresponding path components after checking equal vector lengths, and inserts metadata keys into the destination key set. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/multi_key_path_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) - The worker fixture creates a regular index on field a and a wildcard index. It expects both indexes to export metadata at the insert timestamp while their catalog multikey state remains false. It checks a regular-index path containing component zero and a wildcard metadata key for field a. A separate fixture checks wildcard field sets a and b.c at the first timestamp, and a and d at the second timestamp. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - SharedFlushMergesEqualTimestampsAndPreservesLaterPaths supplies out-of-order worker slots, a null timestamp and equal recorded timestamps. It expects three writes: the null contribution at the first timestamp, merged equal-timestamp paths at that same first timestamp, and the later path at the later timestamp; final catalog paths include all three components. Substituting the fallback timestamp does not coalesce its null-recorded contribution with the explicitly timestamped contribution. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The blocked-flush fixture sets stable timestamp and lastApplied to the same previous operation. In the flush hook it asserts a write timestamp above stable, unchanged lastApplied, unreplicated writes, disabled constraints, disabled multikey tracking, and exempt priority. While the hook is held, shutdown completion stays unready, lastApplied stays previous, and the catalog stays non-multikey. After release it expects one write and publication of the insert operation time and wall time. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The publication fixture helper assertPublished compares both lastApplied op time and wall time with the final entry. assertIndexMultikey requires the collection and index to exist, then compares both the catalog multikey flag and the exact path vector. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) ## How it works - flushMultikeyPathInfo moves every worker contribution through a local MultikeyPathTracker, returns OK immediately when the merged tracker is empty, and processes its stable sort by earliestTimestamp. It does not collapse all timestamps for an index into one earliest write. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/multi_key_path_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - For each sorted contribution, a null earliestTimestamp falls back to firstTimeInBatch only at flush time. The source comment limits this legacy case to prepared-transaction entries emitted while featureFlagReplicateMultikeynessInTransactions was off. The flush requires ts >= firstTimeInBatch, ts <= lastTimeInBatch and ts > lastApplied, then passes namespace, UUID, index name, metadata keys, paths and ts to setIndexIsMultikey. The comment says exact timestamps are required when that feature is enabled. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) ## Problems, failure modes and limitations - The worker requires an empty output and an empty operation-context tracker, starts tracking before batch application, copies tracked results to the output only after successful whole-slice application, and stops and clears tracking on scope exit. The failure fixture asserts that partially applied work exports no metadata and leaves tracking disabled and empty. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - A catalog failure throws through the publication status check before lastApplied advancement and oplog-waiter signaling; the advancer catches database exceptions with a fatal log. The catalog-failure death fixture injects IndexNotFound while asserting lastApplied is unchanged and expects the advancer-failed message. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_advancer.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) - Shutdown errors in consumeWorkItem call onWorkerAbandonment without reporting worker completion. The advancer shutdown waits for the idle-or-abandoned outcome, shuts down the tracker and joins its thread; the source explicitly leaves abandoned work unpublished. Applier teardown drains before stopping the advancer and leaves worker shutdown until last. The dispatch paths contain an unresolved shutdown-failure propagation TODO; these source paths do not establish that abandonment stops all further dispatcher activity. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_batch_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_advancer.cpp) - The WriteConflictRetryRecordsMultikeyAndCompletesBatchOnce fixture injects a write conflict after multikey discovery, expects two attempts and one timestamped contribution, one inserted document, and no catalog multikey change. GroupedInsertFallbackPreservesMultikeyPathsAndTimestamps injects a grouped failure and expects insert-group sizes {3, 1, 1, 1}, two distinct timestamped path contributions and three documents. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - SharedFlushSkipsEmptyMetadataAndStopsOnStorageError expects zero storage attempts for three empty slots. After injecting IndexNotFound into the first of two timestamped writes, it expects that error, one attempt, no recorded successful writes and unchanged catalog multikey paths. The shared flush returns that first storage failure immediately; the fixture does not test rollback of earlier successful writes. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) - The shared-flush death fixtures reject timestamps (2, 0) and (6, 0) outside batch [(3, 0), (5, 0)], and reject (3, 0) when lastApplied is also (3, 0). All match the unpublished-batch invariant message. CatalogWriteFailurePreventsPublication injects IndexNotFound while asserting unchanged lastApplied and expects the advancer fatal-error message. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) ## Facts and statements - The dispatcher allocates a shared vector of WorkerMultikeyPathInfo with one slot per worker and passes ownership into the in-flight batch. If addBatch returns no counter after shutdown, it returns without enqueuing work; otherwise each participating WorkItem receives a pointer to its own slot. consumeWorkItem exports the result before calling onWorkerCompletion; the completion decrement uses memory_order_acq_rel and the front-batch reader uses memory_order_acquire. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_batch_tracker.cpp) - Publication uses a fresh kill-ops-exempt operation context on the advancer Client, exempt admission priority and unreplicated writes. It disables constraint enforcement, configures ignored prepare conflicts with writes allowed, flushes against the batch first/last timestamps and current lastApplied, advances lastApplied, then calls signalOplogWaiters. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_advancer.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) - The tracker pops only the completed FIFO front and marks the advancer busy while holding its mutex. The advancer invokes publication before onBatchPublished clears that busy state. Successful idle requires both an empty FIFO and an idle advancer; abandonment wakes the idle wait but returns false. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_batch_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_advancer.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) - Serialized batches use drain, dispatch, drain. A successful drain waits for worker idleness and for publication side effects, including deferred catalog writes. The dispatcher returns from this batch dispatch path if either drain reports abandonment. The source names DDL, the new-primary noop and truncate as examples requiring serialization. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The TracksMultikeyWritesPerItemAtTheirOwnTimestamps fixture expects grouped array-insert and update metadata at their respective timestamps, an empty scalar-only item, and a later array contribution only in the later item; the catalog index remains non-multikey at the worker handoff. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The KeepsMultikeyResultsWithTheirBatchWhenALaterBatchFinishesFirst fixture holds the earlier batch open while the later counter reaches zero, rejects an early pop, then checks separate per-batch slot vectors. Its weak-pointer assertions retain both metadata objects until their respective popped batches are reset, even while a completed work item is still alive. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The shared flush moves worker contributions into a local tracker, then stable-sorts the merged records by recorded earliest timestamp before issuing catalog writes. The timestamp-order fixture deliberately visits worker slots opposite to timestamp order and expects two separate writes: the first path at the first insert timestamp and the second path at the second insert timestamp, followed by both paths in the catalog. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/multi_key_path_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - MergesSameTimestampPathsAndWildcardKeysAcrossWorkers sets replBatchLimitOperations to 1 and applies one applyOps entry containing two inserts. It expects exactly two catalog writes, one per index at the applyOps timestamp: compound paths {{0U}, {0U}} and wildcard metadata keys decoding to {a, b}, followed by publication and both catalog multikey flags true. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/oplog_applier_dispatch.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/multi_key_path_tracker.cpp) - Collection UUID prevents same-named indexes in different collections from sharing an accumulator identity. The collection-isolation fixture expects two writes for index name idx at the same operation timestamp, with exactly the two distinct collection UUIDs, and both catalogs marked multikey. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/multi_key_path_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - EmptyWorkerMetadataAndZeroWorkerBatchesDoNotWriteCatalog applies a scalar insert and an empty entry built by applyOpsOp({}) with replBatchLimitOperations set to 1; it expects no catalog writes, publication through the empty entry, and the index still non-multikey. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The later-completed-batch fixture holds an earlier worker and uses a later marker on another worker to establish later completion. Before release it expects no catalog writes and unchanged lastApplied. After release it expects two writes in earlier-then-later timestamp order and publication through the marker. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_batch_tracker.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_applier_advancer.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - LegacyNullTimestampUsesFirstTimestampOfItsBatch injects timestamp-less tracker metadata during two scalar inserts and expects one catalog write at the first insert timestamp with publication through the last insert. SerializedBatchFlushesBeforeFollowingDDL expects the catalog write at the serialized insert timestamp to succeed before a following collection drop and publication through that drop. The serialized-boundary fixture asserts that the insert into NamespaceString::kServerConfigurationNamespace requires serialization. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - ShutdownAbandonedSliceDoesNotRecordMultikeyWrites injects InterruptedAtShutdown for the multikey slice, expects the earlier counter to remain 1 while the later counter reaches 0, all metadata slots empty, and no FIFO pop past the abandoned front batch. (low confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The reused-operation-context fixture checks that the first array write exports its timestamped path, then a scalar write exports no metadata. After each call, the operation-context tracker must be empty and tracking must be disabled. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The whole-slice fixture holds a delete after an earlier array insert: while held, the remaining-worker counter stays at one and no completed batch can be popped. After release, it expects the insert timestamp and path in the completed metadata, a zero counter, and no remaining document. (medium confidence) — [source](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier.cpp) · [also](https://raw.githubusercontent.com/mongodb/mongo/e37ae17db503703515d2ec35f6793dee7651655f/src/mongo/db/repl/pipelined_oplog_applier_test.cpp) - The advancement-model task explicitly omits transaction handling and multikey flush ordering; that task description does not supply formal-model coverage of this catalog-publication boundary. (low confidence) — [source](https://jira.mongodb.org/browse/SERVER-133144)