Canary and Shadow-Traffic Comparison
Parent: MongoDB Performance Regression Detection and Testing Methodology · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a0a420b316d1
↓ Facts as markdownall context files
Source-anchored research for Canary and Shadow-Traffic Comparison. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- Shadow traffic is not automatically side-effect-free. A state-changing request may execute on both versions, and shared state such as a cache can alter or contaminate performance measurements. Isolate candidate data and downstream effects, or make replayed operations safe to repeat. (high confidence) [source]
How it works
- A canary routes a controlled share of production traffic to a changed version while the control serves the rest. Users in the canary slice receive candidate responses, so defects can affect that slice; teams compare candidate and control signals and can stop or roll back when the candidate regresses. (high confidence) [source]
- AWS VPC Traffic Mirroring is not universally zero-impact: AWS documentation says mirrored packets count toward source-instance bandwidth and may be dropped when bandwidth or packet-rate limits are exceeded. The AWS blog's no-performance-impact statement is specific to its out-of-band architecture and should not be generalized to every mirroring setup. (medium confidence) (verified as of 2026-10-23) [source]
Problems, failure modes and limitations
- Shadow runs exercise the candidate on copied production inputs but cannot show how users respond to candidate answers because only the incumbent response reaches them. Replayed state-changing requests can duplicate effects; AWS calls out idempotency, and InfoQ recommends read-only replicas or staging data for downstream state. (high confidence) [source]
Comparisons and alternatives
- In Envoy v1.36.11, RequestMirrorPolicy mirrors all requests when runtime_fraction is unset and can restrict mirroring to a configured fraction when it is set. Sampling can limit candidate load, with the trade-off that fewer requests are measured. (low confidence) (verified as of 2026-10-01) [source]
Facts and statements
- A canary routes a portion of live requests to a candidate, so some users receive candidate responses; shadow traffic copies production requests to the candidate while the incumbent serves the client and the shadow response is discarded. (high confidence) [source]
- A useful canary comparison uses a baseline running the production version and configuration, created alongside the candidate; match deployment time, instance count, traffic type, and traffic amount. Comparing against long-running production can confound results through cache warmup, heap, or load-balancing differences. (high confidence) [source]
- Canary analysis should compare configured baseline and candidate time-series metrics that represent service health, including latency, errors, and saturation. Netflix's Kayenta description validates and cleans metric data, classifies individual metrics, and aggregates results; it describes Mann–Whitney U confidence intervals as its primary comparison algorithm. (high confidence) (verified as of 2018-04-10) [source]
- Google and Waze recommend validating a canary analysis configuration with known-good and known-bad app versions, and running an A/A check with the same version on both sides that should pass. (low confidence) [source]
- Google and Waze's Spinnaker guidance recommends at least 50 time-series data points per metric for statistically relevant canary analysis; depending on monitoring granularity, this can require several hours of analysis. (low confidence) (verified as of 2019-01-14) [source]
- Shadowing leaves the stable version on the client response path and sends a copy of each selected request to the candidate; the candidate response is discarded and does not delay the primary response. This exercises the candidate on real request inputs without exposing its output to users. (high confidence) [source]
- Canary metrics should be broken down by candidate and control population, and their aggregation intervals should fit within the canary duration. Service-wide metrics can dilute a regression affecting a small traffic slice, while a time window longer than the canary can mix unrelated events into the comparison. (high confidence) [source]
Children
- No children recorded.