Atlas Stream Processing Lambda external functions and error semantics
Parent: MongoDB Atlas Stream Processing · Published reference · snapshot 2026-10-27
↓ Facts as markdownall context files
Source-anchored research for Atlas Stream Processing Lambda external functions and error semantics. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Datadog distinguishes invocation errors from function errors and explains that Lambda Errors counts function failures, excluding invocation errors. Monitor invocation failures separately when diagnosing ASP-to-Lambda execution. (low confidence) [source]
Problems, failure modes and limitations
- onError defaults to dlq; ignore drops the affected document; fail terminates the processor. (medium confidence) (verified as of 2026-10-14) [source]
- Payload JSON conversion failures or invalid final BSON objects go to the ASP DLQ regardless of onError. Invalid operator expressions instead fail the processor. (low confidence) [source]
- After asynchronous acceptance, Lambda owns its event queue. Its documented default is two additional attempts for function errors, including timeouts; duplicates can occur even without an error. (medium confidence) (verified as of 2026-10-27) [source]
- Configure Lambda-side failure capture for asynchronous execution: a Lambda DLQ captures discarded events, and an on-failure destination captures failed-invocation records. ASP onError does not catch those later function failures. (high confidence) [source]
- Lambda asynchronous delivery can duplicate events even without function errors. Make external side effects idempotent; stream processing completion cannot establish exactly-once Lambda side effects. The latter is an inference from the documented invocation boundary. (medium confidence) [source]
Comparisons and alternatives
- ASP onError covers invocation API errors in both modes and Lambda function errors only in synchronous mode. AWS async invocation returns acceptance rather than function results. (high confidence) [source]
- ASP retries potentially transient request failures; its reference specifies neither retry count nor backoff. This differs from Lambda-managed asynchronous execution retries. (medium confidence) [source]
- The payload subpipeline supports $project, $addFields, $replaceRoot and $set. Lambda responses must be buffered rather than streamed. (low confidence) (verified as of 2026-10-14) [source]
Facts and statements
- $externalFunction invokes AWS Lambda through a registered connection. Synchronous execution returns enrichment results; asynchronous execution continues without awaiting function completion. (high confidence) [source]
- execution defaults to sync. as is required for synchronous nonterminal stages, forbidden for async, and unset for a zero-byte response. (low confidence) (verified as of 2026-10-14) [source]
- Lambda must share the ASP AWS region. IAM needs lambda:InvokeFunction and lambda:GetFunctionConfiguration; ASP reads the configured timeout to wait through execution. Only buffered responses are supported. (low confidence) (verified as of 2026-10-14) [source]
- Lambda Invoke HTTP success does not imply function success: synchronous function failures use FunctionError and the response payload. An accepted asynchronous invocation returns HTTP 202. (low confidence) [source]
- Datadog’s monitoring guidance recommends watching Lambda Errors and dead-letter delivery errors and correlating invocation metrics with logs. This supplies Lambda-side diagnostic guidance, not evidence of an ASP incident. (low confidence) [source]
- MongoDB’s changelog dates $externalFunction support to 26 March 2025, while its introductory blog is dated 3 July 2025; the blog date should not be substituted for the documented release date. (medium confidence) [source]
- The changelog records externalFunction parallelism on 5 August 2026 and Lambda Private Link on 19 August 2026. The Private Link guide requires its endpoint in the processors’ region. (medium confidence) [source]
- For asynchronous function errors, Lambda defaults to two additional attempts, waiting one minute then two minutes. These defaults are configurable; they are not an ASP retry policy. (high confidence) (verified as of 2026-10-27) [source]
Children
- Lambda invocation acceptance versus function failure (frontier)
- External-function payload conversion and DLQ routing (frontier)
- Idempotent Lambda side effects (frontier)
- Lambda asynchronous failure destinations (frontier)
Frontier under this node: External-function payload conversion and DLQ routing, Idempotent Lambda side effects, Lambda asynchronous failure destinations, Lambda invocation acceptance versus function failure