Breaking-point / overload testing methodology (first bottleneck vs visible symptom, cascading failure, thundering herd, cache-eviction death spiral)
Parent: MongoDB Stress, Soak, and Chaos-Resilience Testing · Published reference · snapshot 2026-10-24 · skill mongodb-cohort-22db29b88ceb
↓ Facts as markdownall context files
Source-anchored research for Breaking-point / overload testing methodology (first bottleneck vs visible symptom, cascading failure, thundering herd, cache-eviction death spiral). Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Test components separately and end to end: reducing fleet capacity at constant traffic reveals per-instance limits, but does not reproduce the dependency load of increasing total traffic. (high confidence) [source]
- Exercise gradual ramps, sudden load steps, and the return to nominal load. Caching can change the breaking point, and overloaded components may require load to fall substantially before recovery. Stateful tests must also check correctness. (low confidence) [source]
- Timeouts can waste work already performed, while retries multiply offered load. Retrying at several layers can amplify a backend overload into a self-sustaining cascade, so include actual client retry behavior when evaluating overload protection. (high confidence) [source]
How it works
- WiredTiger eviction is a separate mechanism from application-cache regeneration. Its 11.3.1 documentation says application threads assist eviction when configured triggers are reached, increasing operation latency. MongoDB’s storage README also describes this recruitment; rising eviction work can therefore precede visible request timeouts. (high confidence) [source]
How-to and procedures
- Use gradual ramps and abrupt impulses, then reduce load and measure recovery. Warm caches can hide abrupt-load failures, and removing the triggering change may not end an established cascade. (high confidence) [source]
- Use application-thread eviction, failed eviction, memory and lock contention together when diagnosing the breaking point. CircleCI’s 2019 incident showed that ticket observations alone failed to explain degradation and that application-level metrics could appear recovered while the primary remained degraded. (medium confidence) [source]
Problems, failure modes and limitations
- Interpret queue growth, timeout storms and health-check failure as possible downstream symptoms. Correlate their onset with resource contention and cache behavior to identify the initiating bottleneck; the final visible failure need not identify its cause. (high confidence) [source]
- Exercise real retry behavior and dependency failure under load. Delayed responses waste work after deadlines, and retries amplify demand; backoff and jitter reduce synchronization but do not guarantee recovery from global overload. (high confidence) [source]
- A cache-miss feedback loop can sustain failure: misses increase database reads, overloaded reads time out, cache refill fails, and misses persist. Slack documented this mechanism in its 2022 incident; it is evidence for an application-cache test scenario, not proof of an intrinsic MongoDB eviction defect. (high confidence) [source]
- Measure client-perceived availability and latency alongside server metrics. Separate successful-request latency from rejection latency: cheap failures can make aggregate latency look healthy while useful requests remain slow. (low confidence) [source]
- As a diagnostic synthesis, correlate load and client latency with cache occupancy, eviction activity, application-thread eviction time, storage I/O, and commit progress. High cache occupancy alone does not demonstrate a death spiral: eviction is normal, while lost progress and falling goodput identify a more serious failure. (medium confidence) [source]
Comparisons and alternatives
- Treat cache-full or application-eviction thresholds as evidence to investigate rather than universal failure limits. The MongoDB development README describes workload-dependent detection using exceeded triggers, progress and application-thread eviction time together; this does not establish behavior for every released version. (medium confidence) (verified as of 2026-10-23) [source]
- Test components separately, then test the complete dependency path. Record the first resource or progress limit reached; later latency, memory pressure, backend failure, or failed health checks can be secondary symptoms rather than the initiating bottleneck. (high confidence) [source]
- As a test-design synthesis, compare bounded, jittered retries and early rejection against the unprotected baseline. Measure useful completions, retry volume, and recovery; protection should reduce wasted work rather than merely move failures between layers. (medium confidence) [source]
Facts and statements
- Increase offered load beyond the goodput plateau. Measure client-visible success within deadlines, latency, rejections and timeouts separately; fast rejection must not make successful-request latency look artificially healthy. (high confidence) [source]
- In Facebook’s 2013 memcache system, repeated invalidation of hot keys caused many readers to take the expensive backend path. Per-key leases limited concurrent regeneration. For a MongoDB-backed application cache, this motivates a hot-key invalidation or cold-cache test; it is not evidence of a native MongoDB lease feature. (medium confidence) [source]
- Distinguish application-cache stampedes from WiredTiger page eviction. WiredTiger normally evicts pages to free cache space; under pressure application threads can perform eviction, diverting time from operations and increasing latency. Eviction activity alone does not establish a self-sustaining death spiral. (high confidence) [source]
- Current manual guidance says low available execution tickets alone do not indicate overload; queued reads and writes are the relevant signal. Do not carry the fixed ticket defaults quoted in the 2019 CircleCI postmortem into a current test baseline. (medium confidence) (verified as of 2026-10-23) [source]
- Increase offered load beyond the capacity knee and plot goodput: successful responses completed before client deadlines. A healthy overload response approaches a plateau; collapsing goodput reveals overload instability even if request throughput stays high. (high confidence) [source]
- Cheap rejection has finite capacity. AWS reports that shedding itself eventually consumes enough resources to reduce goodput, and isolated fault injection does not replace an end-to-end load test that stresses dependencies. (low confidence) [source]
- Alex Bevilacqua’s incident describes MongoDB 3.0.15 and reports improvement after an upgrade. His June 2018 updates qualify the explanation: cache-management and checkpoint improvements were more likely contributors than the initially suggested write-ahead-log optimization. Treat this as historical diagnostic evidence, not a current-release defect or a controlled attribution. (low confidence) [source]
Children
- No children recorded.