Pinned event-mongodb outbox relay failure accounting and retry exhaustion boundaries
Parent: Change Data Capture Patterns · Published reference · snapshot 2026-11-02
↓ Facts as markdownall context files
Source-anchored research for Pinned event-mongodb outbox relay failure accounting and retry exhaustion boundaries. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- RunStoreConformance asserts a non-nil empty batch that closes successfully, removal of an acknowledged event from a fresh claim, and re-claim of a failed event with RetryCount at least 1. The Mongo conformance adapter calls that suite through setupOutboxStore; its source comment says the setup skips when no MongoDB is reachable. These are source-defined fixture expectations, not evidence of a completed test run or retry-exhaustion terminality. (high confidence) [source 1] [source 2]
Problems, failure modes and limitations
- `Start` waits on cancellation, notification, polling, cleanup and stuck-recovery cases; it does not invoke the publish closure immediately on entry. MongoStore `Notifications` returns nil, so its notification case cannot trigger publishing. A store implementing `StuckRecoverer` is swept once at startup. (medium confidence) [source 1] [source 2]
- WithMaxRetries records only positive option values; its comment documents 0 as unlimited. The shouldSkip predicate is `r.maxRetries > 0 && msg.RetryCount >= r.maxRetries`. Its exhausted branch calls `batch.Fail(ctx, m, errExhausted)` and skips publication; neither exhaustion itself nor an error from that Fail call increments the drain failure total. (medium confidence) [source 1] [source 2] [source 3]
- `PublishOnce` returns a wrapped readiness error, otherwise calls `drainOnce` and returns `failed to publish %d message(s)` only when its total is positive. It can therefore return nil for a batch containing only exhausted messages, even if their Fail calls log errors, provided Close succeeds. (low confidence) [source]
- drainOnce passes context.WithoutCancel(ctx) to deferred Close, but publication, Ack and Fail use the original context. The source explicitly warns that this does not fully guarantee shutdown-safe commit because an earlier cancellable PostgreSQL claim transaction and Ack updates can still roll back. The Close method of mongoBatch simply returns nil; the MongoDB batch source describes its claim as already committed before ClaimPending returns. (high confidence) [source 1] [source 2]
- When relay metrics are configured, publishMessage records failure metrics only when `r.transport.Publish` errors and records successful-publication metrics after that transport call succeeds, before drainOnce calls Ack or Close. An Ack or Close failure can therefore coexist with a recorded successful transport publish. (low confidence) [source]
- The pinned fake-engine tests assert zero failures with two publishes and two Acks, one failure with a transport error and a failed-record entry, one failure for an injected Close error, an error return from PublishOnce on Close failure, and skipping at `RetryCount: 3` but not `RetryCount: 2` with `WithMaxRetries(3)`. These are fixture assertions, not a terminal-dequeue or exactly-once-publication guarantee. (high confidence) [source 1] [source 2]
- The pinned relay source explicitly describes its semantics as at-least-once in the Close/cancellation comment, and the backend-neutral outbox guidance advises idempotent handlers because messages may be delivered more than once. A publish followed by an Ack error is counted as failure by drainOnce; this evidence does not justify exactly-once publication. (high confidence) [source 1] [source 2] [source 3]
- As a source-level implication of successful calls to the Fail method of mongoBatch and the relay exhaustion branch, an exhausted record can be claimed repeatedly and have retry_count incremented repeatedly without a new transport publish. Such an exhaustion-only drain need not increase Start polling backoff, because the exhaustion path adds no drain failure. This implication is conditional on the store operations succeeding; it is not a fixture execution or production observation. (high confidence) [source 1] [source 2]
Comparisons and alternatives
- For a store implementing Starter, the Relay's ensureReady helper wraps setup in sync.Once, so Start and PublishOnce share one store EnsureReady attempt. The readyErr field retains the first setup result; an initial setup error is returned on later calls instead of another setup attempt. The EnsureReady method of MongoStore delegates to EnsureIndexes, and both relay entry points return a readiness error before claiming a batch. (high confidence) [source 1] [source 2]
- NewRelay copies configured retryBackoff and pollDelay values into the Relay fields. Start uses those stored settings while consuming drainOnce counts rather than returning each drain error. With retryBackoff configured, positive failures increment consecutiveFailures and reset the polling ticker through NextDelay(consecutiveFailures); a subsequent zero-failure drain resets it to the stored pollDelay. Both ticker and notification wakeups invoke the same publish closure; cancellation returns ctx.Err(). (low confidence) [source]
- In drainOnce, ClaimPending failure returns a count of 1. After a successful claim, a transport publish error adds one failure and calls Fail; an error from that Fail call is logged without adding another failure. An Ack error after successful publication adds one failure without calling Fail. The deferred Close adds one further failure if it errors, including for an empty batch. These counts include batch operations, rather than only distinct unsuccessfully published messages. (high confidence) [source 1] [source 2] [source 3]
- The WithMaxRetries documentation says exhausted messages remain StatusFailed and are no longer retried. NewRelay initializes the Relay's maxRetries field from the selected option before shouldSkip uses that stored field as a positive threshold against RetryCount. The exhausted branch calls ordinary Fail with errExhausted, then continues. The Fail method of mongoBatch sets StatusFailed and increments retry_count, while MongoStore's claimBatch selects pending and failed records without a retry-count ceiling. If that MongoDB Fail update succeeds, the record returns to a failed state eligible for another claim rather than a terminal excluded state. The exhausted relay branch itself skips transport publishing when its observed RetryCount meets the positive threshold. (high confidence) [source 1] [source 2]
- The WithRetryBackoff comment describes skipping an individual failed message until a delay based on its retry count expires. In the pinned implementation, Start instead delays its polling ticker by consecutive failing drains, while drainOnce contains no per-message backoff check and PublishOnce directly drains. This is an attributed documentation-versus-code difference. (low confidence) [source]
Children
- No children recorded.