Change Point Detection
Parent: MongoDB Performance Regression Detection and Testing Methodology · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a0a420b316d1
↓ Facts as markdownall context files
Source-anchored research for Change Point Detection. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- A published MongoDB performance-testing workflow applied E-Divisive means to benchmark-result histories and sent detected points to people for review and triage. The Hunter study also analyzed MongoDB's public performance-test dataset. (high confidence) [source]
How it works
- Treat a reported boundary as a candidate to investigate, not proof that a specific software change caused a regression. MongoDB's workflow has an engineer review each point for a real change, an insignificant change, or noise, then rerun missing commit tests when exact attribution is needed; Hunter also presents detected changes for engineering review. (high confidence) [source]
- Pair statistical detection thresholds with a practical effect threshold and review step. Hunter filters changes below a relative-magnitude threshold because small shifts may not be actionable; MongoDB's reported process also reviewed detected changes before triage. (high confidence) [source]
- Offline CPD may need additional measurements after a change before it can identify the newest point. MongoDB's system recomputes points as later test results arrive, and Hunter notes that algorithms need special handling at the end of a series. When tests skip commits, a detected point narrows the suspect range; rerunning the omitted commits is needed to isolate the exact change. (high confidence) [source]
How-to and procedures
- Check run-to-run stability when a detector's significance test uses random permutations. MongoDB's E-Divisive implementation uses permutation sampling, and DataStax's Hunter authors found that permutation-based significance produced unstable detections on fixed performance data, so they replaced it with a Student's t-test. (high confidence) [source]
Measurements and reference values
- Meta's FBDetect paper reports that typical change-point algorithms would yield a 99.7% false-positive rate in its transient-heavy production environment. This single-environment result warns that generic CPD can overwhelm an investigation queue, but it does not establish the expected false-positive rate for MongoDB benchmark histories. (low confidence) (verified as of 2024-11-04) [source]
Comparisons and alternatives
- Do not assume CPD removes measurement noise. Hunter documents false detections caused by unstable benchmark hardware, while the Mozilla evaluation reports a recall-versus-precision tradeoff for offline and hybrid CPD methods. (high confidence) [source]
- CPD accuracy varies with time-series characteristics: a cross-system evaluation found series length and autocorrelation informative, while MongoDB's E-Divisive deployment reports that correlated noise is harder to handle than independent noise. Taken together, these findings support validating and tuning detectors on each benchmark's own history instead of carrying an algorithm score from another workload. (medium confidence) [source]
Facts and statements
- CPD treats a benchmark's ordered metric history as a signal and locates boundaries where its statistical distribution changes. MongoDB's E-Divisive implementation recursively segments historical series at statistically significant changes, and Hunter describes using CPD to identify both regressions and improvements. (high confidence) [source]
- Evaluate CPD methods on representative software-performance histories with known or engineer-reviewed changes. A 2026 Mozilla study used engineer-annotated series and found offline and hybrid methods improved recall at a precision cost; a separate 2026 study evaluated methods on real-world software-performance datasets. (high confidence) [source]
- In the Hunter study's MongoDB benchmark analysis, the original permutation-based E-Divisive implementation returned different numbers of change points across repeated runs, and the paper describes quickly reverted regressions that whole-series segmentation could miss. Validate detector repeatability and short-lived-change sensitivity on the target workload history. (low confidence) [source]
Children
- Offline versus online change point detection (frontier)
- False-alarm and effect-size calibration (frontier)
- Ground-truth benchmark evaluation for CPD (frontier)
Frontier under this node: False-alarm and effect-size calibration, Ground-truth benchmark evaluation for CPD, Offline versus online change point detection