# MongoDB Atlas Stream Processing > Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax — without operating separat Parent: [MongoDB Atlas](https://llms-explorer.com/tree/mongodb-atlas/) · 16 facets · 52 facts · page: https://llms-explorer.com/tree/mongodb-atlas-stream-processing/ ## Overview - Atlas Stream Processing (ASP) is a fully managed, Atlas-native stream processing engine that lets you build real-time data pipelines using MQL-compatible aggregation syntax - without operating separate Kafka Streams or Flink infrastructure. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Real-time alerting, IoT analytics, fraud detection — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Near-real-time materialized views — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - CDC fan-out from MongoDB Atlas — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Event-driven enrichment and transformation — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - When NOT to use ASP: — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Batch re-processing of finite historical datasets → scheduled aggregation pipelines or Atlas Data Federation — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Kafka Connect-only pipelines with no ASP involvement → mongodb-kafka-connector — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Spark Structured Streaming → mongodb-spark-connector — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` - Complex stateful ML inference requiring Flink — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#overview` ## Architecture - Each Stream Processor is a named pipeline with exactly one $source and one $emit. Processors run continuously in the background. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#architecture` ## Connection Registry - Before writing processors, register connections to data sources/sinks: — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#connection-registry` ## Tumbling Window - Fixed, non-overlapping intervals. Good for periodic summaries. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#tumbling-window` ## Hopping Window - Overlapping intervals. Good for rolling metrics. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#hopping-window` ## Session Window - Groups events by inactivity gap. Good for user session analytics. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#session-window` ## SPI Tier Selection - Stream Processing Instances (SPIs) are priced per instance-hour: — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection` - Measure peak message rate × average message size — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection` - Add 2-3x headroom for burst — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection` - Windowed aggregations require more memory → prefer SP10+ for windowed pipelines — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection` - Multiple simultaneous processors share the SPI's capacity — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#spi-tier-selection` ## Watermarks and Late Event Handling - ASP uses event-time watermarks for windowed processing: — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#watermarks-and-late-event-handling` - allowedLateness: grace period for late-arriving events. Events arriving after the watermark + lateness are dropped to the DLQ. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#watermarks-and-late-event-handling` ## Monitoring - Key metrics in Atlas UI: — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring` - Consumer lag (Kafka) - growing lag = processor can't keep up with source throughput — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring` - Error rate - check DLQ for failed events — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring` - Processing latency - end-to-end from source to emit — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#monitoring` ## ASP vs Kafka Connector vs Flink - Decision rule: If you're already on Atlas and need real-time processing without operating infrastructure, use ASP. Use Kafka Connector when you need MongoDB as a source/sink in an existing Kafka ecosystem. Use Flink for complex stateful computation. — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#asp-vs-kafka-connector-vs-flink` ## Anti-Patterns - Single SPI for all processing: Separate high-priority from low-priority processors across different SPIs — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns` - Windowed aggregation without watermarks: Late events cause incorrect window results — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns` - No DLQ configured: Failed events are silently dropped without $validate + allowedLateness — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns` - Growing consumer lag left unchecked: Indicates SPI undersized for throughput; upgrade tier — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns` - Using ASP for batch re-processing: ASP is for continuous streams - use Data Federation for historical batch — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#anti-patterns` ## References - Atlas Stream Processing Documentation — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references` - Stream Processing Operators Reference — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references` - SPI Tiers and Pricing — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references` - ASP Window Types — source: `https://llms-explorer.com/sources/mdb-context-hub/mongodb-atlas-stream-processing/#references` ## Where this helps - Building real-time alerting, fraud detection, or IoT analytics on data flowing through Atlas or Kafka without standing up separate Kafka Streams or Flink infrastructure. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Fanning out change-data-capture events from an Atlas cluster into enriched, transformed downstream streams. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Maintaining a near-real-time materialized view that needs to stay current with a continuous stream rather than a scheduled batch job. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Deciding Atlas Stream Processing is the wrong tool because the workload is actually batch reprocessing of a finite historical dataset, or needs complex stateful ML inference that only Flink supports. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Project ideas - Build a Stream Processor with a tumbling window for periodic summaries and a separate one with a hopping window for rolling metrics, to compare their output shapes on the same source. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Build a real-time alerting pipeline using MQL-compatible aggregation syntax that emits to a Time Series Collection, and monitor consumer lag and DLQ error rate as it runs. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Configure watermarks and allowedLateness for a windowed aggregation, then intentionally send late events to verify they land in the dead-letter queue instead of corrupting results. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Size a Stream Processing Instance (SPI) tier by measuring peak message rate times average message size with 2–3x headroom, then validate under a simulated burst. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Common mistakes - Running all processing on a single SPI instead of separating high-priority from low-priority processors across different SPIs. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Running windowed aggregation without watermarks configured, which lets late-arriving events produce incorrect window results. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Leaving the dead-letter queue unconfigured, so failed events, without $validate plus allowedLateness, are silently dropped instead of surfaced. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Ignoring growing consumer lag instead of treating it as a signal the SPI tier is undersized for current throughput. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* ## Known issues - Atlas Stream Processing is built for continuous streams, not batch reprocessing of finite historical datasets — that workload belongs on scheduled aggregation pipelines or Atlas Data Federation instead. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Windowed aggregations require more memory than simple pipelines, and the guidance is to prefer SP10+ tiers for windowed processing rather than the smallest available tier. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Multiple simultaneous processors share the same SPI's capacity, so an undersized SPI can degrade every processor running on it, not just the busiest one. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)* - Late events that arrive after the watermark plus allowedLateness grace period are dropped to the DLQ rather than being incorporated, which is a deliberate tradeoff that has to be sized correctly for the data's actual lateness distribution. — [source](https://llms-explorer.com/tree/mongodb-atlas-stream-processing/) *(AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.)*