CI Performance Regression Gating
Parent: MongoDB Performance Regression Detection and Testing Methodology · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a0a420b316d1
↓ Facts as markdownall context files
Source-anchored research for CI Performance Regression Gating. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- MongoDB's server benchmark guidance separates function-level microbenchmarks from full-server or cluster performance tests. For Evergreen, it recommends a feature-specific resmoke suite for faster feedback, requires a system resource canary and the CombineBenchmarkResults hook, and says thresholds apply per build variant because machines have different characteristics. (low confidence) [source]
- A pull-request comparison is more interpretable when baseline and candidate use a matched environment and workload. GlueSQL's proposal dated July 21, 2026 measures the base and head sequentially in the same CI job with the same runner, toolchain, configuration, sample size, warm-up, duration, and input. Conbench's documented history comparison matches repository, hardware, case permutation, and context. (high confidence) [source]
- Benchmark smoke checks and performance gates answer different questions. GlueSQL notes that its cargo test --benches command only compiles and executes benchmark targets without statistical comparison. Apache Arrow Rust recommends targeted cargo bench runs and saved baselines for performance-change evidence because the full benchmark suite takes longer. (high confidence) [source]
Measurements and reference values
- DuckDB nanoarrow documents a pull-request workflow that compares base and candidate benchmark runners, fails on benchmark errors or wrong results, and applies performance limits: a 10% slowdown in the suite geometric mean or a 10% slowdown in one benchmark on two runs in a row to filter runner noise. (low confidence) (verified as of 2026-10-23) [source]
- In its April 2025 account of MongoDB 8.0, MongoDB Engineering said server CI runs YCSB, Linkbench, TPCC, TPCH, and custom customer-workload benchmarks. It said a commit that regressed one of its most important workloads by more than 5% would be reverted, while that threshold would miss smaller regressions such as 0.1% changes accumulating across thousands of commits. (low confidence) (verified as of 2025-04-09) [source]
- MongoDB's April 2025 writeup says latency and throughput are noisy, so its simple ping CI test counts Linux instructions and uses a hard-coded 0.2% tolerance; the article warns that thresholds set too tightly on noisy measurements create flaky tests developers may ignore. (low confidence) (verified as of 2025-04-09) [source]
Comparisons and alternatives
- Thresholds should reflect comparable machine and workload histories rather than one universal percentage. MongoDB's wiki says thresholds vary by build variant; Conbench's lookback method compares only results with the same repository, hardware, case permutation, and context, then uses historical variance to judge a regression. (high confidence) [source]
Facts and statements
- Cloud-hosted performance CI can issue false alarms. A 2018 paired-run study found much greater variation on Travis CI than on a local desktop. A React on Rails issue opened on 2026-06-17 reported five false-positive regression issues over about 3.5 hours, including non-runtime commits, and said confirmation runs on shared runners repeated the noise. (medium confidence) (verified as of 2026-06-17) [source]
- RocksDB separates open-source pull-request unit and integration checks from continuously run internal stress and performance tests. Its testing page says teams triage detected regressions before deciding to patch or revert the responsible pull request. (low confidence) [source]
- Regression tooling makes benchmark selection and repeat counts explicit: Apache Arrow's `archery benchmark diff` filters for benchmark names matching `^Regression` by default and runs configured repetitions for statistical significance, while Chromium documents separate overall and subtest thresholds and chooses Pinpoint run counts through statistical analysis. (medium confidence) [source]
- OpenSSL's performance-check documentation describes 10 interleaved repeats by default, compares candidate and baseline medians and spread, and classifies under-sampled or known-noisy rows as inconclusive or low confidence; it says the check never blocks merging. (low confidence) (verified as of 2026-10-07) [source]
- A June 2026 React on Rails issue reports five automatic benchmark-regression issues in about three and a half hours from shared GitHub-hosted runners, with no causal code regression. Maintainers disabled cloud auto-alert filing while moving the signal to a dedicated M1 runner, and corrected an earlier suggestion that an effect-size threshold would solve the large noise swings. (low confidence) [source]
Children
- No children recorded.