Coordinator failure recovery
Parent: Distributed Sharded Transactions · Published reference · snapshot 2026-10-30
↓ Facts as markdownall context files
Source-anchored research for Coordinator failure recovery. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Cockroach Labs’ March 21, 2024 postmortem describes transaction-status recovery that lets another transaction encountering a STAGING transaction verify its recorded writes and establish COMMITTED or ABORTED when the original coordinator has not finished asynchronous finalization. (low confidence) [source]
Problems, failure modes and limitations
- A participant that has voted to commit cannot safely abort merely because the coordinator times out: a commit outcome may already exist or be delivered elsewhere. MongoDB likewise keeps prepared transactions dependent on the coordinator’s commit or abort decision. (high confidence) [source]
- Classic two-phase commit can block after coordinator failure; durable recovery state does not by itself guarantee progress while the coordinator remains unavailable. (high confidence) [source]
Comparisons and alternatives
- A 2024 Cockroach Labs bug investigation found recovery racing with retries after an ambiguous RPC failure. Replaying at a changed timestamp could violate idempotency; the reported fix returns SQL error 40003 statement_completion_unknown for unsafe ambiguous replays instead of blindly changing the transaction. (low confidence) [source]
Facts and statements
- MongoDB hands two-phase commit from its stateless router to a coordinator shard so lifecycle data is durable; coordinator recovery reads persisted coordinator documents and resumes their commit processing. (high confidence) [source]
- In the inspected MongoDB master implementation, primary recovery waits for the current system OpTime to become majority committed, reads coordinator documents, creates coordinators, and calls continueCommit(doc). That method restores participants and any recorded decision. (high confidence) (verified as of 2026-10-30) [source]
- In the inspected MongoDB coordinator implementation, recovery with a recorded decision skips preparing participants again. Without a recorded decision it sends prepare requests to collect votes. The decision path waits for majority durability before sending commit or abort to participants. (high confidence) (verified as of 2026-10-30) [source]
- MongoDB participant prepare state survives failover: a newly promoted primary reacquires prepared-transaction locks, and lost in-memory prepared state can be reconstructed from prepare oplog entries and the transactions table. (low confidence) [source]
- The inspected MongoDB coordinator recovery path explicitly bypasses admission ticket acquisition to avoid a possible deadlock with prepared participants; its comment references SERVER-82883 and SERVER-60682. This is an implementation-specific recovery dependency. (low confidence) (verified as of 2026-10-30) [source]
- Gray and Lamport’s Paxos Commit uses 2F+1 coordinators and can make progress with F+1 working coordinators under its protocol assumptions; ordinary two-phase commit is its F=0 special case. This quorum-based tolerance should not be read as progress despite arbitrary loss of a majority. (high confidence) [source]
- In MongoDB’s inspected coordinator implementation, the participant list is persisted and majority acknowledged before prepare requests; the commit or abort decision is persisted and majority acknowledged before decision messages. This ordering leaves recovery information available at the corresponding protocol boundaries. (low confidence) (verified as of 2026-10-30) [source]
Children
- No children recorded.