MongoDB Atlas Stream Processing
Parent: MongoDB Atlas · Published reference · snapshot 2026-09-08 · skill mongodb-atlas-expert/references/mongodb-atlas-stream-processing.md
↓ Facts as markdownall context files
Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax — without operating separat
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Overview
- Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax - without operating separate Kafka Streams or Flink infrastructure. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Real-time alerting, IoT analytics, fraud detection [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Near-real-time materialized views [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- CDC fan-out from MongoDB Atlas [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Event-driven enrichment and transformation [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- When NOT to use ASP: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Batch re-processing of finite historical datasets → scheduled aggregation pipelines or Atlas Data Federation [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Kafka Connect-only pipelines with no ASP involvement → mongodb-kafka-connector [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Spark Structured Streaming → mongodb-spark-connector [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
- Complex stateful ML inference requiring Flink [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview]
Architecture
- Each Stream Processor is a named pipeline with exactly one $source and one $emit. Processors run continuously in the background. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#architecture]
Connection Registry
- Before writing processors, register connections to data sources/sinks: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#connection-registry]
Tumbling Window
- Fixed, non-overlapping intervals. Good for periodic summaries. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#tumbling-window]
Hopping Window
- Overlapping intervals. Good for rolling metrics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#hopping-window]
Session Window
- Groups events by inactivity gap. Good for user session analytics. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#session-window]
SPI Tier Selection
- Stream Processing Instances (SPIs) are priced per instance-hour: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection]
- Measure peak message rate × average message size [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection]
- Add 2-3x headroom for burst [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection]
- Windowed aggregations require more memory → prefer SP10+ for windowed pipelines [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection]
- Multiple simultaneous processors share the SPI's capacity [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection]
Watermarks and Late Event Handling
- ASP uses event-time watermarks for windowed processing: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#watermarks-and-late-event-handling]
- allowedLateness: grace period for late-arriving events. Events arriving after the watermark + lateness are dropped to the DLQ. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#watermarks-and-late-event-handling]
Monitoring
- Key metrics in Atlas UI: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring]
- Consumer lag (Kafka) - growing lag = processor can't keep up with source throughput [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring]
- Error rate - check DLQ for failed events [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring]
- Processing latency - end-to-end from source to emit [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring]
ASP vs Kafka Connector vs Flink
- Decision rule: If you're already on Atlas and need real-time processing without operating infrastructure, use ASP. Use Kafka Connector when you need MongoDB as a source/sink in an existing Kafka ecosystem. Use Flink for complex stateful computation. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#asp-vs-kafka-connector-vs-flink]
Anti-Patterns
- Single SPI for all processing: Separate high-priority from low-priority processors across different SPIs [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns]
- Windowed aggregation without watermarks: Late events cause incorrect window results [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns]
- No DLQ configured: Failed events are silently dropped without $validate + allowedLateness [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns]
- Growing consumer lag left unchecked: Indicates SPI undersized for throughput; upgrade tier [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns]
- Using ASP for batch re-processing: ASP is for continuous streams - use Data Federation for historical batch [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns]
References
- Atlas Stream Processing Documentation [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references]
- Stream Processing Operators Reference [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references]
- SPI Tiers and Pricing [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references]
- ASP Window Types [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references]
Where this helps
- Building real-time alerting, fraud detection, or IoT analytics on data flowing through Atlas or Kafka without standing up separate Kafka Streams or Flink infrastructure. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Fanning out change-data-capture events from an Atlas cluster into enriched, transformed downstream streams. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Maintaining a near-real-time materialized view that needs to stay current with a continuous stream rather than a scheduled batch job. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding Atlas Stream Processing is the wrong tool because the workload is actually batch reprocessing of a finite historical dataset, or needs complex stateful ML inference that only Flink supports. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Project ideas
- Build a Stream Processor with a tumbling window for periodic summaries and a separate one with a hopping window for rolling metrics, to compare their output shapes on the same source. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Build a real-time alerting pipeline using MQL-compatible aggregation syntax that emits to a Time Series Collection, and monitor consumer lag and DLQ error rate as it runs. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Configure watermarks and allowedLateness for a windowed aggregation, then intentionally send late events to verify they land in the dead-letter queue instead of corrupting results. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Size a Stream Processing Instance (SPI) tier by measuring peak message rate times average message size with 2–3x headroom, then validate under a simulated burst. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Running all processing on a single SPI instead of separating high-priority from low-priority processors across different SPIs. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Running windowed aggregation without watermarks configured, which lets late-arriving events produce incorrect window results. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Leaving the dead-letter queue unconfigured, so failed events, without $validate plus allowedLateness, are silently dropped instead of surfaced. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Ignoring growing consumer lag instead of treating it as a signal the SPI tier is undersized for current throughput. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- Atlas Stream Processing is built for continuous streams, not batch reprocessing of finite historical datasets — that workload belongs on scheduled aggregation pipelines or Atlas Data Federation instead. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Windowed aggregations require more memory than simple pipelines, and the guidance is to prefer SP10+ tiers for windowed processing rather than the smallest available tier. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Multiple simultaneous processors share the same SPI's capacity, so an undersized SPI can degrade every processor running on it, not just the busiest one. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Late events that arrive after the watermark plus allowedLateness grace period are dropped to the DLQ rather than being incorporated, which is a deliberate tradeoff that has to be sized correctly for the data's actual lateness distribution. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- $source Operators
- $emit Operators
- mongodb-kafka-connector
- Stream Processing Instance (SPI) (frontier, viability 3.65 / 5)
- Stream Processor Pipeline (frontier, viability 3.30 / 5)
- Tumbling Windows (frontier, viability 3.48 / 5)
- Hopping Windows (frontier, viability 3.50 / 5)
- Session Windows (frontier, viability 3.38 / 5)
- Watermarks and Late-Event Tolerance (frontier, viability 4.22 / 5)
- Connection Registry (frontier, viability 3.75 / 5)
- Dead Letter Queue (DLQ) (frontier, viability 4.05 / 5)
- Stream Processor Monitoring (frontier, viability 3.67 / 5)
- ASP Pricing Model
Frontier under this node: Connection Registry (3.75), Dead Letter Queue (DLQ) (4.05), Hopping Windows (3.50), Session Windows (3.38), Stream Processing Instance (SPI) (3.65), Stream Processor Monitoring (3.67), Stream Processor Pipeline (3.30), Tumbling Windows (3.48), Watermarks and Late-Event Tolerance (4.22)