Bayesian Data Analysis and Probabilistic Programming

Parent: Data Analysis · Published reference · snapshot 2026-09-08 · skill da-analytical-methods/references/da-25-bayesian-data-analysis.md

↓ Facts as markdown all context files

Applied Bayesian modeling: specify a generative model, fit the posterior with a probabilistic programming language (PPL), interrogate it with predictive checks and diagnostics, and compare alternative

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Bayesian Data Analysis & Probabilistic Programming

Applied Bayesian modeling: specify a generative model, fit the posterior with a probabilistic programming language (PPL), interrogate it with predictive checks and diagnostics, and compare alternatives. This skill is the workflow and tooling layer - it assumes Bayes' theorem and the frequentist/Bayesian contrast are already understood (see da-1-3-5, da-1-4-3). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#bayesian-data-analysis-probabilistic-programming]

Overview

A Bayesian model combines a prior p(θ) and a likelihood p(y|θ) into a posterior p(θ|y) ∝ p(y|θ) p(θ). For all but trivial models the posterior is intractable analytically, so we approximate it by sampling (MCMC) or optimization (variational inference). The discipline is an iterative loop - model, fit, check, expand - formalized as the Bayesian workflow (Gelman, Vehtari, Simpson et al. 2020, arXiv:2011.01808). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#overview]
Use Bayesian methods when you want: full uncertainty quantification (posteriors, not just point estimates), principled regularization via priors, partial pooling across groups (hierarchical models), the ability to incorporate domain knowledge, and propagation of uncertainty into predictions/decisions. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#overview]

Core Concepts

The Bayesian workflow (iterative). Build → simulate from priors → fit → diagnose computation → check against data → compare models → expand/simplify → repeat. Treats computational failure (divergences, bad R-hat) and model misfit (failed PPCs) as distinct problems (Gelman et al. 2020; Betancourt "Principled Bayesian Workflow" 2020; Gabry et al. JRSS-A 2019). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Priors & prior predictive checks. Favor weakly-informative priors (Normal(0,1) on standardized predictors, HalfNormal/Exponential on scales, LKJ on correlations) over flat/diffuse priors. Validate via prior predictive check: draw θ from prior, simulate y, confirm plausibility (Stan Prior Choice wiki 2024; PyMC v5 docs 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
MCMC: HMC & NUTS. HMC uses gradients for distant, high-acceptance moves; NUTS auto-tunes trajectory length + step size and is the default in every modern PPL. Continuous params only - marginalize discrete ones (Hoffman & Gelman JMLR 2014; Betancourt arXiv:1701.02434 2017; Stan Reference Manual 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Variational inference (VI/ADVI). Approximates the posterior by maximizing the ELBO - fast, scalable, but underestimates variance and can miss multimodality. ADVI automates it; normalizing flows enrich it. Validate against MCMC (Kucukelbir et al. JMLR 2017; PyMC VI docs 2024; NumPyro SVI docs 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Hierarchical/multilevel models. Group-level params drawn from a shared population distribution → partial pooling (data-determined shrinkage toward the global mean). The biggest practical reason to go Bayesian (Gelman & Hill 2006; McElreath Statistical Rethinking 2020; PyMC multilevel primer 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Posterior predictive checks (PPC). Simulate y_rep from the posterior predictive, compare to observed y - overlaid densities, test statistics, LOO-PIT calibration. A model that can't reproduce key data features is misspecified (Gabry et al. JRSS-A 2019; ArviZ API 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Model comparison: LOO-CV (PSIS), WAIC. Compare predictive accuracy with PSIS-LOO (elpd_loo); the Pareto-k diagnostic flags unreliable points (k > 0.7 problematic). LOO preferred over WAIC for its self-diagnostics; prefer LOO over Bayes factors (Vehtari et al. Stat&Computing 2017, arXiv:1507.04544; Vehtari et al. PSIS, JMLR 2024, arXiv:1507.02646). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Convergence diagnostics. Use rank-normalized split-R-hat (target < 1.01) and bulk-ESS/tail-ESS. HMC adds divergent transitions (curvature the sampler can't resolve - reparameterize/prior fix), max-treedepth (efficiency), and E-BFMI < 0.3 (poor energy exploration) (Vehtari et al. Bayesian Analysis 2021, arXiv:1903.08008; Stan Warnings 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
PPLs. PyMC (Python v5+, PyTensor backend, pluggable NUTS: default/nutpie/numpyro/blackjax); Stan (reference HMC, via CmdStanPy/cmdstanr); NumPyro (JAX, fastest NUTS, GPU); Bambi/brms (formula GLMM interface y ~ x + (1|g)); ArviZ (backend-agnostic diagnostics on InferenceData) (PyMC; CmdStanPy; NumPyro; Bambi; ArviZ, all 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
Bayesian regression & GLMs. GLM link families carry over: Gaussian/Bernoulli/Poisson/Negative-Binomial; coefficient priors regularize (Normal≈ridge, Laplace≈LASSO, horseshoe for sparsity). Student-t likelihood for robust regression. Standardize predictors (McElreath 2020; Bambi examples 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]
ArviZ - diagnostics & plotting hub. az.summary (R-hat/ESS/HDI), plot_trace, plot_ppc, loo/compare/plot_compare, plot_forest, plot_energy (BFMI), plot_pair (divergences) (ArviZ API 2024). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#core-concepts]

Methodology — the loop in practice

Scope & generative story. Write the model as a data-generating process; pick the likelihood family from the outcome type first. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Priors + prior predictive check. Weakly-informative priors on standardized variables; sample_prior_predictive; reject absurd priors. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Fit. NUTS, ≥4 chains. target_accept=0.8 default; raise to 0.9–0.99 on divergences. Try NumPyro/nutpie for large continuous models. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Diagnose computation. R-hat < 1.01, bulk/tail-ESS, zero divergences, E-BFMI > 0.3, no treedepth saturation. Fix here before interpreting. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Posterior predictive check. sample_posterior_predictive → az.plot_ppc, test statistics, LOO-PIT. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Compare. az.loo per model, az.compare; inspect Pareto-k. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]
Iterate. Expand (hierarchy/interactions/robust likelihood) or simplify; re-run. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#methodology-the-loop-in-practice]

Practical Patterns

Standardize continuous predictors (mean 0, sd 1) so default priors behave; back-transform for interpretation. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]
Non-centered parameterization for hierarchical models. Replace θ_g ~ Normal(μ, σ) with θ_g = μ + σ·z_g, z_g ~ Normal(0,1) (the "Matt trick") - removes the funnel that causes divergences with sparse groups (Stan User's Guide 2024; Betancourt & Girolami 2015, arXiv:1312.0906). [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]
≥4 chains, multiple seeds, inspect trace plots - R-hat alone misses problems. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]
Marginalize discrete parameters so NUTS can run. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]
Use Bambi/brms for standard GLMMs; drop to raw PyMC/Stan only for custom structure. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]
Save full InferenceData (posterior + predictive + log-likelihood + sample stats) so LOO/PPCs are reproducible. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#practical-patterns]

Anti-Patterns

Ignoring divergences - they bias the posterior. Reparameterize, raise target_accept, or tighten priors. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Flat/diffuse "uninformative" priors as default - rarely uninformative on quantities of interest; prefer weakly-informative. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Trusting VI/ADVI without MCMC comparison - VI understates variance. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Old R-hat < 1.1 as the bar - use rank-normalized R-hat < 1.01 and check ESS. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Comparing on in-sample fit / DIC / raw likelihood - use LOO/WAIC; read Pareto-k first. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Centered hierarchical parameterization with sparse groups - the classic funnel generator; go non-centered. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]
Reading the posterior mean only - report HDIs, propagate the full posterior. [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#anti-patterns]

References

Gelman et al. - Bayesian Workflow (2020). https://arxiv.org/abs/2011.01808 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Vehtari, Gelman, Gabry - LOO-CV and WAIC (2017). https://arxiv.org/abs/1507.04544 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Vehtari et al. - Pareto Smoothed Importance Sampling (JMLR 2024). https://arxiv.org/abs/1507.02646 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Vehtari et al. - Improved R-hat (Bayesian Analysis 2021). https://arxiv.org/abs/1903.08008 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Hoffman & Gelman - The No-U-Turn Sampler (JMLR 2014). https://jmlr.org/papers/v15/hoffman14a.html [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Betancourt - Conceptual Introduction to HMC (2017). https://arxiv.org/abs/1701.02434 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Betancourt - Towards a Principled Bayesian Workflow (2020). https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Betancourt & Girolami - HMC for Hierarchical Models (2015). https://arxiv.org/abs/1312.0906 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Kucukelbir et al. - ADVI (JMLR 2017). https://jmlr.org/papers/v18/16-107.html [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Gabry et al. - Visualization in Bayesian Workflow (JRSS-A 2019). https://academic.oup.com/jrsssa/article/182/2/389/7070184 [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Gelman et al. - Bayesian Data Analysis 3rd ed. (BDA3, 2013). http://www.stat.columbia.edu/~gelman/book/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
McElreath - Statistical Rethinking 2nd ed. (2020). https://xcelab.net/rm/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Gelman & Hill - Regression and Multilevel/Hierarchical Models (2006). http://www.stat.columbia.edu/~gelman/arm/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Stan - Prior Choice Recommendations (2024). https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Stan - Reference Manual / Runtime Warnings (2024). https://mc-stan.org/docs/reference-manual/mcmc.html [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
PyMC docs (2024). https://www.pymc.io/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
NumPyro docs (2024). https://num.pyro.ai/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
Bambi docs (2024). https://bambinos.github.io/bambi/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
CmdStanPy docs (2024). https://mc-stan.org/cmdstanpy/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]
ArviZ docs (2024). https://python.arviz.org/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/da-25-bayesian-data-analysis/#references]

Where this helps

Full uncertainty quantification is needed — a posterior distribution rather than a single point estimate — such as reporting a range on a small-sample effect instead of just its mean. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
A hierarchical or multilevel structure exists in the data, with per-group effects nested in an overall population, and partial pooling is preferable to treating every group independently or ignoring group structure entirely. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Genuine prior knowledge exists, such as a previous study's effect size or a domain constraint, and should inform the estimate rather than starting from a blank slate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Multiple candidate models need a principled out-of-sample comparison via LOO-CV or WAIC, rather than picking whichever model fits best in-sample. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Project ideas

Fit a hierarchical model with a non-centered, “Matt trick” parameterization on a small multi-group dataset and confirm it removes the divergences a centered parameterization produces. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Run the full Bayesian workflow loop — prior predictive check, fit with NUTS across at least four chains, posterior predictive check, and model comparison via LOO-CV — on one applied dataset end to end. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Compare a weakly-informative-prior model against a flat-prior model on the same data to see how much the prior actually shifts the posterior when data is sparse. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build a small model-comparison pipeline that reports Pareto-smoothed importance sampling diagnostics alongside LOO-CV, so unreliable comparisons get caught instead of trusted at face value. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Ignoring MCMC divergences instead of treating them as a sign the posterior geometry needs reparameterization, a higher target_accept, or tighter priors — divergences bias the resulting posterior. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Treating a flat or maximally diffuse prior as automatically uninformative, when on the actual quantity of interest after transformation it often concentrates prior mass in an unintended place. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Trusting variational inference results without ever cross-checking against MCMC — VI is known to systematically understate posterior variance. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Checking only R-hat and declaring convergence, instead of also inspecting trace plots across multiple chains and seeds, which catches problems R-hat alone can miss. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

MCMC sampling via HMC or NUTS is computationally expensive relative to point-estimate methods, and hierarchical models with many groups can be slow enough that VI or a data subset gets used for early iteration. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Model comparison metrics like LOO-CV degrade when the posterior has a few highly influential observations; a bad Pareto k diagnostic signals the LOO estimate itself is unreliable, not just the model. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
The Bayesian workflow is genuinely iterative — build, fit, diagnose, check, expand — rather than a one-shot fit, which takes real analyst time compared to running a single frequentist regression. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Weakly-informative priors are only weak relative to standardized predictors; without standardizing first, a prior tuned for one scale can be accidentally informative or accidentally uninformative on another. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

Bayesian workflow (Gelman/Vehtari) (frontier)
Priors and prior predictive checks (frontier)
MCMC (NUTS/HMC) (frontier)
Variational inference (ADVI) (frontier)
Hierarchical/multilevel models (partial pooling, non-centered parameterization) (frontier)
Posterior predictive checks (frontier)
Model comparison (LOO-CV/PSIS, WAIC) (frontier)
Convergence diagnostics (R-hat, ESS, divergences, BFMI) (frontier)
Probabilistic programming languages (PyMC, Stan, NumPyro, Bambi) (frontier)
Bayesian regression and GLMs (frontier)
ArviZ diagnostics and plotting (frontier)

Frontier under this node: ArviZ diagnostics and plotting, Bayesian regression and GLMs, Bayesian workflow (Gelman/Vehtari), Convergence diagnostics (R-hat, ESS, divergences, BFMI), Hierarchical/multilevel models (partial pooling, non-centered parameterization), MCMC (NUTS/HMC), Model comparison (LOO-CV/PSIS, WAIC), Posterior predictive checks, Priors and prior predictive checks, Probabilistic programming languages (PyMC, Stan, NumPyro, Bambi), Variational inference (ADVI)

← the whole tree · 3D view· how to read this page