Atlas Stream Processing throttle rate limits and backpressure
Parent: MongoDB Atlas Stream Processing · Published reference · snapshot 2026-10-27
↓ Facts as markdownall context files
Source-anchored research for Atlas Stream Processing throttle rate limits and backpressure. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Atlas Stream Processing has a documented $throttle stage that limits data passed to downstream stages in bytes or messages per second. It can protect downstream systems during initial snapshots, recovery, and historical replay. This is a pipeline flow-control feature; the reviewed planning premise that no ASP throttle concept exists is contradicted by the stage reference and supported-stage catalogue. (medium confidence) (verified as of 2026-10-14) [source]
How-to and procedures
- Use the verbose option of sp.processor.stats() to inspect stats.operatorStats, including per-operator throttle.throttledTimeMs and throttle.throttleEvents. Per-stage counters help locate the throttle responsible when a pipeline has multiple throttles. (medium confidence) (verified as of 2026-10-14) [source]
- Configure the limits for the traffic at the throttle placement: the reference limits data passed by that stage, and each throttle operates on its own flow. The reference does not establish an account-wide quota shared across separate processors or a guarantee that every downstream API request maps one-to-one to an input document. (low confidence) (verified as of 2026-10-14) [source]
Problems, failure modes and limitations
- The byte limit is not a strict maximum for every instantaneous transfer: if a single document exceeds bytesPerSec, $throttle passes that document downstream and then waits one second before passing more data. (low confidence) (verified as of 2026-10-14) [source]
- $throttle is supported only in the main pipeline and cannot be nested inside another stage. Multiple $throttle stages are permitted, with each limiting only the data flowing through it. (medium confidence) (verified as of 2026-10-14) [source]
- For operational diagnosis, compare input and output throughput with latency and processing-error signals. MongoDB documents input/output counters, latency, and DLQ metrics; an independent OneUptime monitoring tutorial also recommends these dimensions. The tutorial does not independently establish $throttle scheduling semantics. (medium confidence) (verified as of 2026-10-14) [source]
- Retention-budget implication (inference): growing change-stream lag from throttling warrants checking source oplog retention. Estuary independently documents that its own MongoDB capture connector cannot resume after required oplog events are evicted. That connector evidence supports the general retention risk; it does not establish ASP-specific recovery or backfill behavior. (medium confidence) [source]
Comparisons and alternatives
- When both limits are configured, each recovers capacity continuously and independently. Data advances only when both have capacity. If both are exhausted, the wait is the longer required recovery time, rather than the sum; processing can resume within a second. (low confidence) (verified as of 2026-10-14) [source]
- stats.changeStreamTimeDifferenceSecs measures the time difference between the event represented by the latest change-stream resume token and the latest oplog event. Kafka sources instead expose topic/partition offset lag in monitoring. Choose the lag signal appropriate to the source rather than treating the change-stream metric as universal. (low confidence) (verified as of 2026-10-14) [source]
Facts and statements
- $throttle accepts bytesPerSec and messagesPerSec as positive integers. At least one is required, and each document counts as one message. Omitting both produces: "$throttle requires at least one of 'bytesPerSec' or 'messagesPerSec'". (low confidence) (verified as of 2026-10-14) [source]
- Throttling creates backpressure on preceding stages. If sustained source production exceeds the configured throttle throughput, the processor falls behind and lag grows over time. Downstream protection therefore does not by itself ensure that the processor keeps up with its source. (low confidence) (verified as of 2026-10-14) [source]
- stats.throttle.throttledTimeMs measures cumulative active throttling time in milliseconds; stats.throttle.throttleEvents counts deliberate message delays. For multiple $throttle stages, processor-level values sum the per-stage values. These counters measure throttling activity, not a count of rejected operations. (medium confidence) (verified as of 2026-10-14) [source]
Children
- No children recorded.