Atlas Stream Processing partition idleness versus window idleness
Parent: MongoDB Atlas Stream Processing · Published reference · snapshot 2026-10-29
↓ Facts as markdownall context files
Source-anchored research for Atlas Stream Processing partition idleness versus window idleness. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- For $tumblingWindow and $hoppingWindow, idleTimeout controls closing windows when $source is idle. It belongs to the window stage; partitionIdleTimeout belongs to $source and governs individual Kafka partitions. (high confidence) [source]
- Both tumbling and hopping references prohibit idleTimeout and allowedLateness when boundary is processingTime. Their documented boundary default is eventTime. (high confidence) (verified as of 2026-10-14) [source]
- Independent conceptual context: Apache Flink and Confluent describe downstream watermarks as constrained by the slowest parallel input; Apache documents idleness detection to prevent inactive inputs from stalling progress. These sources explain the general mechanism, not Atlas timer implementation. (medium confidence) [source]
Facts and statements
- With a Kafka source, an idle partition can prevent watermark progress and leave a window without output. The architecture reference identifies partitionIdleTimeout as the mitigation; MongoDB diagnostic guidance also lists this condition for a STARTED processor with zero windowed output. (high confidence) [source]
- The $source reference documents partitionIdleTimeout as disabled by default. Its duration uses size and unit, supporting ms, second, minute, hour, or day. This default was verified in the 2026-10-14 documentation snapshot. (low confidence) (verified as of 2026-10-14) [source]
- Both window-stage references say the idle timer starts as soon as $source becomes idle, and describe closing after the greater of the remaining window time and idleTimeout. Their example closes a 12:00–13:00 window last active at 12:02 at 14:02 for a two-hour timeout, or at 13:00 for a ten-minute timeout. Treat this as reference behavior pending resolution of the architecture-page discrepancy. (medium confidence) [source]
- A published MongoDB pipeline example combines a five-second partitionIdleTimeout on Kafka $source with a ten-second idleTimeout on a thirty-second tumbling window. These durations are settings in that example, not universal recommendations. (medium confidence) [source]
- Forced window progress has a lateness consequence: events arriving after their window closes are late, and Atlas can route them to a configured dead letter queue. Choosing an idleness timeout therefore needs to account for expected event delays. (medium confidence) [source]
- Diagnostic inference: distinguish one silent Kafka partition while other partitions produce data from source-wide silence. Inspect partitionIdleTimeout for the former and window idleTimeout for the latter; the settings address different progress conditions. (medium confidence) [source]
Children
- Kafka partition watermarks (frontier)
- Event-time window closure (frontier)
- Allowed lateness and idle-triggered closure (frontier)
Frontier under this node: Allowed lateness and idle-triggered closure, Event-time window closure, Kafka partition watermarks