Conformal Prediction and Uncertainty Quantification

Parent: data analysis · researched 2026-05-30T22:38:51.606Z· 17 sources · 14 concepts · skill da-31-conformal-prediction-uq

Conformal prediction (CP) turns any point predictor into a set predictor with a

Overview

Conformal prediction (CP) turns any point predictor into a set predictor with a [source]
finite-sample, distribution-free marginal coverage guarantee. Given a target [source]
miscoverage rate alpha, CP outputs a prediction set C(X) such that [source]
P(Y in C(X)) >= 1 - alpha - holding for any underlying model, any data [source]
distribution, and any sample size, requiring only that the calibration and test [source]
data are exchangeable. It is a wrapper, not a model. You keep your XGBoost, [source]
neural net, or random forest and bolt CP on top to get honest error bars. [source]
CP is the dominant frequentist answer to "how do I get valid uncertainty without [source]
trusting my model's probabilities." Compared to Bayesian credible intervals [source]
(which require a correct prior/likelihood) and the bootstrap (asymptotic, can [source]
under-cover), CP's guarantee is exact in finite samples and model-agnostic. The [source]
modern reference is Angelopoulos & Bates, "A Gentle Introduction to Conformal [source]
Prediction and Distribution-Free Uncertainty Quantification" (arXiv 2107.07511, [source]
v-series 2021-2023). The theoretical foundation is Vovk, Gammerman & Shafer, [source]
"Algorithmic Learning in a Random World" (Springer, 2005; 2nd ed. 2022). [source]
Use this skill when someone needs **calibrated intervals/sets on top of an [source]
existing model**. For full Bayesian modeling use da-25; for point forecasting use [source]

1. The coverage guarantee (marginal validity)

Split-conformal gives the exact two-sided bound [source]
1 - alpha <= P(Y_{n+1} in C(X_{n+1})) <= 1 - alpha + 1/(n+1), where n is the [source]
calibration set size. The guarantee is marginal (averaged over the random [source]
draw of calibration + test points), not conditional on a specific X or a fixed [source]
calibration set - coverage fluctuates around the target for any one fixed [source]
calibration set, and the spread shrinks as n grows. Distribution-free and [source]
finite-sample, no asymptotics. Source: Angelopoulos & Bates 2023 [source]
(arxiv.org/abs/2107.07511); Lei et al. "Distribution-Free Predictive Inference [source]
for Regression" JASA 2018; Vovk et al. 2005. [source]

2. Exchangeability — the one assumption

CP requires only that (X_1,Y_1)...(X_{n+1},Y_{n+1}) are exchangeable: their [source]
joint distribution is invariant to permutation. This is weaker than i.i.d. but is [source]
violated by distribution shift and temporal/serial dependence - the two main ways [source]
CP breaks in practice. Source: Vovk et al. 2005; Barber, Candès, Ramdas & [source]
Tibshirani "Conformal Prediction Beyond Exchangeability" Annals of Statistics [source]
2023 (stat.cmu.edu/~ryantibs/papers/nexcp.pdf). [source]

3. Nonconformity (conformity) score

A function s(x,y) measuring how "strange" a label y is for input x given the [source]
model - e.g. residual |y - f(x)| for regression, or 1 - softmax(true class) for [source]
classification. CP is entirely defined by your choice of score: the score [source]
controls adaptivity and set shape, while the coverage guarantee holds for [source]
any score. Designing a good score (heteroscedastic-aware, class-adaptive) is [source]
the main lever for sharpness. Source: Angelopoulos & Bates 2023. [source]

4. Split / Inductive Conformal Prediction (ICP)

The workhorse. (a) Split data into proper-training + calibration. (b) Fit the [source]
model on training. (c) Compute calibration scores s_i. (d) Set qhat = the [source]
ceil((n+1)(1-alpha))/n empirical quantile of the calibration scores (the same [source]
finite-sample-corrected quantile used in Methodology step 5). (e) For a [source]
new x, output C(x) = { y : s(x,y) <= qhat }. One model fit, O(n log n) [source]
calibration - cheap and the default for deep learning. The finite-sample [source]
quantile correction (the +1) is what delivers exact validity. Source: [source]
Papadopoulos et al. 2002 (inductive CP); Angelopoulos & Bates 2023; [source]
github.com/aangelopoulos/conformal-prediction. [source]

5. Full / Transductive Conformal Prediction

The original formulation: for each candidate label y, refit (or re-score) the [source]
model with (x_{n+1}, y) appended and test that point's conformity rank against [source]
all others. Uses all data (no calibration split, so more statistically [source]
efficient on small data) but costs one refit per candidate label per test point — [source]
usually intractable except with closed-form/leave-one-out shortcuts. First [source]
proposed by Gammerman, Vovk & Vapnik 1998. Source: Vovk et al. 2005; Vovk [source]
"Transductive Conformal Predictors" 2013 (alrw.net/articles/08.pdf). [source]

6. Conformalized Quantile Regression (CQR)

Wraps a quantile regressor (estimating lower/upper conditional quantiles [source]
q_lo, q_hi) with conformal calibration. Score is the signed exceedance [source]
E_i = max(q_lo(x_i) - y_i, y_i - q_hi(x_i)); conformalize E to get qhat and output [source]
[q_lo(x) - qhat, q_hi(x) + qhat]. Inherits heteroscedastic adaptivity from [source]
quantile regression and valid coverage from CP - intervals widen where the [source]
data is noisy, producing shorter intervals than residual-based CP. Source: Romano, [source]
Patterson & Candès, "Conformalized Quantile Regression" NeurIPS 2019 [source]
(arxiv.org/abs/1905.03222); github.com/yromano/cqr. [source]

7. Classification sets — APS and RAPS

APS (Adaptive Prediction Sets): score accumulates sorted softmax mass until [source]
the true class is included; produces sets that adapt to difficulty (bigger sets [source]
on hard examples). Source: Romano, Sesia & Candès, "Classification with Valid [source]
and Adaptive Coverage" NeurIPS 2020. [source]
RAPS (Regularized APS): adds a regularization penalty that discourages [source]
including low-probability tail classes, yielding 5-10x smaller, more stable sets [source]
than APS while keeping the coverage guarantee. Source: Angelopoulos, Bates, [source]
Malik & Jordan, "Uncertainty Sets for Image Classifiers using Conformal [source]
Prediction" ICLR 2021 (arxiv.org/abs/2009.14193); [source]
github.com/aangelopoulos/conformal_classification. [source]
LAC / naive softmax score (1 - p_true) gives the smallest sets but worse [source]
conditional coverage; APS/RAPS trade size for adaptivity. [source]

8. Mondrian / group-conditional conformal

Partition (X x Y) into disjoint categories (e.g. by class, sex, region) and run a [source]
separate conformal calibration per category, giving group-conditional coverage [source]
P(Y in C(X) | group=g) >= 1 - alpha for each g. Only requires exchangeability [source]
within each category. Limitation: groups must be non-overlapping; overlapping [source]
or continuous attributes need newer methods (Kandinsky CP 2025; Gibbs et al. [source]
conditional-guarantee CP 2023). Source: Vovk et al. 2005 (Mondrian CP); [source]
MAPIE Mondrian docs (mapie.readthedocs.io); Ding et al. "Class-Conditional [source]
Conformal Prediction with Many Classes" NeurIPS 2023. [source]

9. Weighted conformal & covariate shift

Under covariate shift (P_test(X) != P_train(X) but P(Y|X) unchanged), [source]
exchangeability fails and CP under/over-covers. Weighted conformal reweights [source]
calibration scores by the likelihood ratio w(x) = dP_test(x)/dP_train(x) (the [source]
"weighted exchangeability" notion) to restore validity when w is known or [source]
estimable from unlabeled test data. Source: Tibshirani, Barber, Candès & Ramdas, [source]
"Conformal Prediction Under Covariate Shift" NeurIPS 2019 [source]
(arxiv.org/abs/1904.06019); extended to feedback shift in Fannjiang et al. PNAS [source]

10. Time series — EnbPI and ACI

Serial dependence breaks exchangeability, so two main adaptations exist: [source]
EnbPI (Ensemble batch Prediction Intervals): Xu & Xie, ICML 2021 [source]
(proceedings.mlr.press/v139/xu21h). Uses leave-one-out ensemble residuals, [source]
no data split, assumes stationary strongly-mixing errors; gives approximate [source]
marginal coverage asymptotically. [source]
ACI (Adaptive Conformal Inference): Gibbs & Candès, NeurIPS 2021 [source]
(arxiv.org/abs/2106.00170). Online update of the effective miscoverage level [source]
alpha_t after each observation; stable under arbitrary distribution shift and [source]
guarantees long-run coverage regardless of dependence. Variants: AgACI / DtACI, [source]
Zaffran et al. ICML 2022 (arxiv.org/abs/2202.07282); SPCI, Xu & Xie 2023. [source]

11. Calibration vs sharpness

Two orthogonal quality axes. Calibration/validity = the set actually covers at [source]
the nominal rate (a 90% interval contains Y ~90% of the time). **Sharpness/ [source]
efficiency** = sets are as small/tight as possible. CP guarantees marginal [source]
calibration by construction; sharpness depends on the model and score and is the [source]
thing you optimize. Gneiting's maxim: maximize sharpness subject to calibration. [source]
Report average set size / interval width alongside empirical coverage. Source: [source]
Gneiting et al. JRSS-B 2007; Angelopoulos & Bates 2023. [source]

12. Conformal Risk Control (beyond coverage)

Generalizes CP from miscoverage to any monotone bounded loss (false-negative [source]
rate, F1, recall), controlling E[loss] <= alpha. Useful for multilabel, [source]
segmentation, and structured outputs. Source: Angelopoulos, Bates et al. [source]
"Conformal Risk Control" 2022/ICLR 2024 [source]
(people.eecs.berkeley.edu/~angelopoulos/publications/downloads/conformal-risk.pdf). [source]

Tools / Frameworks

MAPIE (Model Agnostic Prediction Interval Estimator) - scikit-learn-native, [source]
the standard for tabular regression/classification; split/CV+/jackknife+, CQR, [source]
APS/RAPS, Mondrian, conformal risk control. mapie.readthedocs.io. Use this [source]
first for sklearn workflows. [source]
crepes - lightweight, clean conformal regressors & predictive systems [source]
(Boström); good for normalized/Mondrian regression and conformal predictive [source]
distributions. github.com/henrikbostrom/crepes. [source]
TorchCP - PyTorch-native, GPU-accelerated; deep classifiers, regressors, [source]
GNNs, LLMs; APS/RAPS/SAPS, CQR, time-series CP. JMLR 2025 [source]
(arxiv.org/abs/2402.12683). Use for deep learning. [source]
PUNCC (Deel), Fortuna (AWS), nonconformist (the original, now [source]
largely superseded) - alternatives; nonconformist is NumPy-only and unmaintained. [source]
Reference implementations: github.com/aangelopoulos/conformal-prediction [source]
(notebooks for every method), yromano/cqr, aangelopoulos/conformal_classification. [source]

Methodology (split-conformal recipe)

Define the task & alpha. Pick target coverage 1-alpha (e.g. 0.9). Decide [source]
regression (intervals) vs classification (sets), and whether you need marginal [source]
or group-conditional coverage. [source]
Three-way split. proper-train / calibration / test. Calibration n>=~1000 [source]
for stable 90% intervals; n>=~ a few hundred minimum. Never reuse training [source]
data for calibration. [source]
Fit the base model on proper-train only. [source]
Choose a score. Regression: residual (simple) or CQR (heteroscedastic). [source]
Classification: LAC (small sets) or APS/RAPS (adaptive). The score is your [source]
Calibrate. Compute scores on calibration set; take the finite-sample- [source]
corrected quantile qhat = quantile(scores, ceil((n+1)(1-alpha))/n). [source]
Predict. C(x) = {y : s(x,y) <= qhat}. [source]
Evaluate on test: empirical marginal coverage (should ~ 1-alpha), average [source]
set size / interval width (sharpness), and size-stratified / group coverage [source]
to expose conditional-coverage failures. [source]
Handle violations. Covariate shift -> weighted CP. Time series -> EnbPI/ACI. [source]
Heterogeneous subgroups -> Mondrian/CQR. [source]

Practical Patterns

Default stack: sklearn model + MAPIE split or CV+ with CQR is the 80% case [source]
for tabular regression. For deep classifiers use TorchCP + RAPS. [source]
CV+ / jackknife+ (Barber et al. Annals 2021) when data is scarce and you [source]
can't afford a calibration split - gives slightly weaker (1-2alpha) guarantees [source]
Always report coverage AND average width/size - coverage alone hides a [source]
useless predictor that returns the whole label space. [source]
Stratify coverage checks by feature bins / class / group; marginal coverage [source]
can be 90% while a subgroup sits at 60%. [source]
For online/streaming, use ACI and monitor the running coverage; let alpha_t [source]
Use a fixed random seed for the split and, where possible, average over [source]
multiple splits (or use CV+) to reduce calibration-set variance. [source]

Anti-Patterns

Calibrating on training data. Reusing fit data destroys validity - scores [source]
are optimistically small and you under-cover. Always hold out calibration. [source]
Trusting marginal coverage as conditional coverage. CP guarantees marginal, [source]
not P(Y in C | X=x); a model can be 90% overall and badly miscalibrated per [source]
subgroup. Use Mondrian / size-stratified checks. [source]
Ignoring exchangeability. Applying vanilla CP to time series or shifted test [source]
data silently breaks the guarantee. Use ACI/EnbPI or weighted CP. [source]
Optimizing the score for coverage. Coverage is guaranteed regardless - tune [source]
the score for sharpness, not coverage. [source]
Tiny calibration sets. n in the tens makes the realized coverage swing wildly [source]
around the target; the +1/(n+1) slack and quantile granularity dominate. [source]
Confusing CP with calibrated probabilities. CP gives valid sets, not [source]
calibrated softmax scores; Platt/temperature scaling is a different (and [source]
Reusing the calibration set to also select alpha or the model. That is [source]
double-dipping; it invalidates the guarantee. [source]

Troubleshooting

Empirical coverage below target: check for data leakage (calibration overlaps [source]
training), non-exchangeability (shift/time), or too-small n. Verify the quantile [source]
used the (n+1) finite-sample correction. [source]
Coverage fine but intervals huge: the base model is weak or the score is [source]
non-adaptive; switch residual->CQR, LAC->APS/RAPS, or improve the model. Width [source]
is a model problem, not a CP problem. [source]
Good marginal coverage, bad subgroup coverage: move to Mondrian/ [source]
group-conditional CP or CQR; report stratified coverage. [source]
Time-series coverage drifts over time: exchangeability violated - use ACI [source]
(online alpha_t update) or EnbPI; plot rolling coverage. [source]
Coverage degrades after deployment: likely covariate shift - estimate the [source]
likelihood ratio and apply weighted conformal, or recalibrate on fresh data. [source]
Classification sets sometimes empty or all-classes: empty sets are valid [source]
under some scores (force-inclusion of the top class if a non-empty set is [source]
required); full-label sets signal an uninformative model or too-small alpha. [source]

References

Angelopoulos & Bates, "A Gentle Introduction to Conformal Prediction and [source]
Distribution-Free Uncertainty Quantification," arXiv:2107.07511 (2021-2023) — [source]
arxiv.org/abs/2107.07511. The canonical practical intro. [source]
Vovk, Gammerman & Shafer, "Algorithmic Learning in a Random World," Springer [source]
(2005; 2nd ed. 2022). Theoretical foundation. [source]
Lei, G'Sell, Rinaldo, Tibshirani & Wasserman, "Distribution-Free Predictive [source]
Inference for Regression," JASA (2018) - split conformal for regression. [source]
Romano, Patterson & Candès, "Conformalized Quantile Regression," NeurIPS (2019) [source]
— arxiv.org/abs/1905.03222. [source]
Romano, Sesia & Candès, "Classification with Valid and Adaptive Coverage" (APS), [source]
Angelopoulos, Bates, Malik & Jordan, "Uncertainty Sets for Image Classifiers [source]
using Conformal Prediction" (RAPS), ICLR (2021) - arxiv.org/abs/2009.14193. [source]
Tibshirani, Barber, Candès & Ramdas, "Conformal Prediction Under Covariate [source]
Shift," NeurIPS (2019) - arxiv.org/abs/1904.06019. [source]
Barber, Candès, Ramdas & Tibshirani, "Predictive inference with the jackknife+," [source]
Annals of Statistics (2021); and "Conformal Prediction Beyond Exchangeability," [source]
Annals of Statistics (2023). [source]
Xu & Xie, "Conformal Prediction Interval for Dynamic Time-Series" (EnbPI), ICML [source]
(2021) - proceedings.mlr.press/v139/xu21h. [source]
Gibbs & Candès, "Adaptive Conformal Inference Under Distribution Shift" (ACI), [source]
NeurIPS (2021) - arxiv.org/abs/2106.00170. [source]
Zaffran et al., "Adaptive Conformal Predictions for Time Series" (AgACI), ICML [source]
(2022) - arxiv.org/abs/2202.07282. [source]
Angelopoulos, Bates et al., "Conformal Risk Control," ICLR (2024). [source]
Tooling: MAPIE (mapie.readthedocs.io), crepes (github.com/henrikbostrom/crepes), [source]
TorchCP (arXiv:2402.12683, JMLR 2025), aangelopoulos/conformal-prediction. [source]

Children

Marginal coverage guarantee (frontier)
Exchangeability assumption (frontier)
Nonconformity scores (frontier)
Split/inductive conformal prediction (ICP) (frontier)
Full/transductive conformal prediction (frontier)
Conformalized quantile regression (CQR) (frontier)
Adaptive prediction sets (APS/RAPS) (frontier)
Mondrian/group-conditional conformal (frontier)
Weighted conformal under covariate shift (frontier)
Conformal for time series (EnbPI, ACI) (frontier)
Calibration vs sharpness (frontier)
Conformal risk control (frontier)
Conformal vs Bayesian credible intervals and bootstrap (frontier)
Tooling (MAPIE, crepes, TorchCP) (frontier)

Frontier under this node: Adaptive prediction sets (APS/RAPS), Calibration vs sharpness, Conformal for time series (EnbPI, ACI), Conformal risk control, Conformal vs Bayesian credible intervals and bootstrap, Conformalized quantile regression (CQR), Exchangeability assumption, Full/transductive conformal prediction, Marginal coverage guarantee, Mondrian/group-conditional conformal, Nonconformity scores, Split/inductive conformal prediction (ICP), Tooling (MAPIE, crepes, TorchCP), Weighted conformal under covariate shift

← the whole tree · 3D view· how to read this page