Coefficient of Variation / Run-to-Run Noise Control
Parent: MongoDB Performance Regression Detection and Testing Methodology · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-803bb5221982
↓ Facts as markdownall context files
Source-anchored research for Coefficient of Variation / Run-to-Run Noise Control. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Separate within-trial benchmark variability across iterations, variability between trials on the same instance, and variability between different instances of the same type. The 2019 study reports total CV alongside these components, showing that one overall CV can obscure which experimental layer needs better control. (low confidence) [source]
How it works
- Distinguish inner-loop iterations from benchmark repetitions. Repetitions re-execute setup and teardown and sample additional process states; replacing one long run with many shorter runs can change cache warmup and the experiment being measured. (high confidence) [source]
- Preserve sufficient warmup and measurement time when increasing repetitions. Google Benchmark's interleaving discussion warns that shortening each repetition can leave caches cold and increase variance; its user guide supports excluding warmup measurements. (high confidence) [source]
- Separate warmup from measured repetitions when the target is steady-state performance. Google Benchmark can discard a configured warmup interval; the 2019 Java microbenchmark study cautions that ten one-second warmup iterations may be insufficient for some benchmarks. (high confidence) [source]
- Google Benchmark's documented repeated-run output includes mean, median, standard deviation and CV. --benchmark_repetitions requests repeated runs, while --benchmark_display_aggregates_only can retain individual results in file output. (low confidence) (verified as of 2026-10-23) [source]
Measurements and reference values
- There is no evidence here for a universal acceptable CV for MongoDB. A 2019 Java/Go microbenchmark study reported CVs from 0.03% to above 100%, whereas a 2025 Dynatrace stream-processing study reported CV below 3.7%. These are different workload and environment populations, not interchangeable thresholds. (high confidence) [source]
- MongoDB's January 2023 retrospective says its original greater-than-10% pairwise comparison missed small regressions and raised false positives on noisy tests. Raising thresholds for noisier tests improved the system but did not resolve the underlying detection problems. (low confidence) [source]
- MongoDB's January 2023 account reports that run-to-run noise persists even after extensive controls. Its historical 10% comparison rule missed smaller regressions and generated false positives on noisy tests. (low confidence) [source]
- Henning and colleagues' 2025 study reports CV below 3.7% for its stream-processing application experiments, with small daily and weekly patterns. This scopes claims that cloud execution necessarily creates large run-to-run variation; it does not establish MongoDB's CV. (low confidence) [source]
Problems, failure modes and limitations
- Use repeated runs of unchanged code as an A/A control to characterize the harness before interpreting A/B results. This is a methodological inference from Heisler's same-version experiment and Google Benchmark's warning that a single result may be unrepresentative. (medium confidence) [source]
Comparisons and alternatives
- For MongoDB testing, interpreting a measured percentage change requires evidence about background noise rather than a fixed percentage alone. MongoDB's historical false alarms illustrate this risk; a 2019 cloud microbenchmark study found false positives in unchanged-code A/A comparisons, especially at low sample sizes. (high confidence) [source]
- CV measures noise, not the statistical significance of a baseline-versus-candidate difference. Combine dispersion reporting with a comparison of repeated samples; Google Benchmark documents a Mann–Whitney U test, and the cloud microbenchmark study evaluates rank-sum tests and bootstrapped confidence intervals. (high confidence) [source]
Facts and statements
- The coefficient of variation (CV) is standard deviation divided by mean. For repeated benchmark measurements it describes relative dispersion; expressing the ratio as a percentage makes variability comparable across measurement scales. (high confidence) [source]
- Repeat candidate and baseline measurements on the same instances and randomize execution order to reduce confounding from machine differences and changing system state. Google Benchmark also supports random interleaving of repetitions for this purpose. (high confidence) [source]
- Google Benchmark documents --benchmark_repetitions and reports mean, median, standard deviation and CV across repeated runs; --benchmark_enable_random_interleaving randomizes repetition order. These are harness capabilities, not evidence that MongoDB's system uses these exact flags. (low confidence) (verified as of 2026-10-23) [source]
Children
- No children recorded.