MongoDB Atlas Stream Processing
Parent: MongoDB Atlas · researched 2026-05-28T15:31:37.734Z· 4 sources · 12 concepts · skill mongodb-atlas-stream-processing
Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax — without operating separat
Overview
- Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax - without operating separate Kafka Streams or Flink infrastructure. [source]
- Real-time alerting, IoT analytics, fraud detection [source]
- Near-real-time materialized views [source]
- CDC fan-out from MongoDB Atlas [source]
- Event-driven enrichment and transformation [source]
- When NOT to use ASP: [source]
- Batch re-processing of finite historical datasets → scheduled aggregation pipelines or Atlas Data Federation [source]
- Kafka Connect-only pipelines with no ASP involvement → mongodb-kafka-connector [source]
- Spark Structured Streaming → mongodb-spark-connector [source]
- Complex stateful ML inference requiring Flink [source]
Architecture
- Each Stream Processor is a named pipeline with exactly one $source and one $emit. Processors run continuously in the background. [source]
Connection Registry
- Before writing processors, register connections to data sources/sinks: [source]
Tumbling Window
- Fixed, non-overlapping intervals. Good for periodic summaries. [source]
Hopping Window
- Overlapping intervals. Good for rolling metrics. [source]
Session Window
- Groups events by inactivity gap. Good for user session analytics. [source]
SPI Tier Selection
- Stream Processing Instances (SPIs) are priced per instance-hour: [source]
- Measure peak message rate × average message size [source]
- Add 2-3x headroom for burst [source]
- Windowed aggregations require more memory → prefer SP10+ for windowed pipelines [source]
- Multiple simultaneous processors share the SPI's capacity [source]
Watermarks and Late Event Handling
Monitoring
ASP vs Kafka Connector vs Flink
- Decision rule: If you're already on Atlas and need real-time processing without operating infrastructure, use ASP. Use Kafka Connector when you need MongoDB as a source/sink in an existing Kafka ecosystem. Use Flink for complex stateful computation. [source]
Anti-Patterns
- Single SPI for all processing: Separate high-priority from low-priority processors across different SPIs [source]
- Windowed aggregation without watermarks: Late events cause incorrect window results [source]
- No DLQ configured: Failed events are silently dropped without $validate + allowedLateness [source]
- Growing consumer lag left unchecked: Indicates SPI undersized for throughput; upgrade tier [source]
- Using ASP for batch re-processing: ASP is for continuous streams - use Data Federation for historical batch [source]
References
Children
- Stream Processing Instance (SPI) (frontier)
- Stream Processor Pipeline (frontier)
- $source Operators (frontier)
- $emit Operators (frontier)
- Tumbling Windows (frontier)
- Hopping Windows (frontier)
- Session Windows (frontier)
- Watermarks and Late-Event Tolerance (frontier)
- Connection Registry (frontier)
- Dead Letter Queue (DLQ) (frontier)
- Stream Processor Monitoring (frontier)
- ASP Pricing Model (frontier)
Frontier under this node: $emit Operators, $source Operators, ASP Pricing Model, Connection Registry, Dead Letter Queue (DLQ), Hopping Windows, Session Windows, Stream Processing Instance (SPI), Stream Processor Monitoring, Stream Processor Pipeline, Tumbling Windows, Watermarks and Late-Event Tolerance