Machine Learning

Machine Learning (Data Analysis Curriculum, Section 7)

Machine learning is the curriculum step where the analyst stops merely describing a sample and starts building a function that generalizes from data to unseen inputs. Section 6 (da-6-statistical-modeling) covered parametric statistical models grounded in explicit probabilistic assumptions. This section widens the lens to algorithms that learn flexible, often non-parametric mappings from data — and to the engineering scaffolding (tuning, evaluation, deployment, monitoring) that turns a trained model into a system that keeps working.

This skill is the curriculum reference for the seventh section of the data analysis path. It is intentionally broad: it sketches the territory and points to the deeper skills you should pull in for any specific build.

Part 1 — Machine Learning Taxonomy

1.1 Three (now four) classical paradigms

• Supervised learning. Input–output pairs (x, y). Sub-shapes: classification (discrete y) and regression (continuous y).
• Unsupervised learning. No labels. Discovers clusters (k-means, DBSCAN, HDBSCAN), topics (LDA, NMF), manifolds (PCA, t-SNE, UMAP, autoencoders), density (KDE, GMM), or anomalies (isolation forest, one-class SVM).
• Reinforcement learning. Agent → environment → reward; optimizes policy pi(a | s). Modern flavors: PPO, GRPO, DQN. RLHF/RLAIF align LLMs.
• Self-supervised learning. Labels constructed from the data: next-token prediction (GPT/Claude/Gemini), masked LM (BERT), masked patch (DINO/MAE), contrastive (CLIP). The engine behind every foundation model.

Edge shapes: semi-supervised (small labeled + large unlabeled), active learning (model picks next labels).

1.2 Bias-variance tradeoff

E[(y - f_hat(x))^2] = Bias[f_hat(x)]^2 + Var[f_hat(x)] + sigma^2

Bias (underfitting) drops with capacity; variance (overfitting) rises. Regularization, more data, and ensembling trade variance for some bias. The classical U-curve is an idealization — in over-parameterized regimes (most modern DL) test error follows “double descent”: rises near the interpolation threshold, then falls again as capacity grows.

1.3 Regularization toolbox

L1 (Lasso, feature selection), L2 (Ridge / weight decay, default in DL), ElasticNet, Dropout (ensemble averaging), early stopping, data augmentation (often the strongest), label smoothing, normalization layers.

1.4 Classical model zoo

Linear/logistic regression, decision trees, random forests, gradient-boosted trees (XGBoost/LightGBM/CatBoost still win most tabular competitions in 2026), SVM, kNN, Naive Bayes. On tabular: start with a GBT.

Part 2 — Deep Learning Fundamentals

2.1 CNNs

Grid-structured data. Convolution (weight sharing, translation equivariance), pooling, hierarchical feature learning. LeNet → AlexNet → VGG → ResNet (residual connections) → EfficientNet → ConvNeXt. Still competitive on edge devices, limited-data medical imaging, and as hybrid backbones.

2.2 RNNs

Sequential state h_t = f(x_t, h_{t-1}). Vanilla RNNs (vanishing gradients), LSTM (gated cell, workhorse 2014–2018), GRU (simpler, comparable), bidirectional. Mostly displaced by Transformers; still useful for streaming inference, tiny edge time-series, and as the conceptual ancestor of state-space models (Mamba, S4, S6) that scale linearly in sequence length.

2.3 Architecture choice

Transformer is the 2026 default, but consider inductive bias (CNNs encode locality), sequence length (attention is quadratic — sliding window/sparse/linear/SSMs win on long contexts), and latency budget (a 7B Transformer is overkill for keyword-spotting on a watch).

Part 3 — Transformers and Foundation Models

3.1 The Transformer

Attention Is All You Need (Vaswani et al., 2017). Per token: query Q, key K, value V. Attention = softmax(QK^T / sqrt(d_k)) V. Multi-head runs several attention ops in parallel. Parallelizable training, one-hop long-range dependencies, modality-agnostic (text, images via ViT, audio via Whisper, proteins via AlphaFold).

3.2 Foundation model paradigm

Pretrain large on broad data with self-supervision, adapt via: zero-shot prompting, few-shot in-context learning, fine-tuning (full, LoRA/QLoRA, DPO/KTO, RLHF/RLAIF), RAG, tool use / function calling. 2024–2026 era is agent-shaped systems built around pretrained foundations.

Part 4 — Frontier LLM Landscape (May 2026)

Selection heuristics: agentic coding → Claude Opus 4.7; general reasoning → GPT-5.5; multimodal → Gemini 3.1 Pro; self-hosted → Llama 4; budget/high-volume → DeepSeek V3.2 or smaller specialists. Use LiteLLM/OpenRouter for per-request routing. Treat the LLM as replaceable, not a vendor commitment.

Part 5 — Hyperparameter Tuning

5.1 Search strategies

Manual, grid (only ≤3 hyperparams), random (dominates grid when few hyperparams matter), Bayesian optimization (GP or TPE + EI/UCB acquisition, default for 10–100 trials), HyperBand (successive halving), BOHB (BO + HyperBand, production default for DL), PBT (population-based, learns schedules).

5.2 Tooling (2026)

Optuna (Python-native, TPE + pruning, individual default), Ray Tune (distributed, all algorithms above), Weights & Biases Sweeps (if W&B is your tracker), KerasTuner/AutoGluon/FLAML (AutoML), Vizier/SigOpt (hosted).

5.3 Cheatsheet

Tune on validation, never on test. Nested CV when small. Log-uniform LR/weight decay. Tune learning rate first. Cap wall-clock per trial. Re-tune on dataset/architecture/optimizer change. Save trial history for get_param_importances.

Part 6 — Evaluation Methodology

6.1 Splits

Train/val/test (test touched once). k-fold CV, stratified k-fold (imbalanced), group k-fold (leakage by group ID), time-series splits (never shuffle), nested CV (small data + many hyperparams).

6.2 Classification metrics

Accuracy (misleading on imbalance). Precision TP/(TP+FP) — optimize when false positives expensive. Recall TP/(TP+FN) — optimize when false negatives expensive. F1 harmonic mean. F-beta. ROC-AUC (insensitive to class balance). PR-AUC (preferred when positive class rare). Log-loss / cross-entropy (proper scoring rule). Calibration plots + Brier score. Macro/weighted/micro F1 for multi-class.

6.3 Regression metrics

MAE (robust), RMSE (penalizes outliers), MAPE / sMAPE, R^2, quantile / pinball loss.

6.4 Ranking metrics

MAP@k, NDCG@k, MRR, Hit-rate@k.

6.5 LLM evaluation

MMLU (saturated — frontier >90%), MT-Bench (multi-turn, LLM-as-judge), HELM (Stanford CRFM; accuracy, calibration, robustness, fairness, bias, toxicity, efficiency), Chatbot Arena (Elo from pairwise preference), SWE-bench / SWE-bench Verified / SWE-bench Pro (real GitHub issues; Claude Opus 4.7 leads Verified at 87.6%), HumanEval / MBPP / LiveCodeBench, GPQA / ARC-AGI / FrontierMath (not yet saturated), domain-specific (MedQA, LegalBench, FinanceBench, MMMU).

6.6 LLM-as-judge

Pros: fast, scalable. Cons: position bias, self-preference, verbosity bias, rubric drift. Mitigations: randomize position; use different judge family than model under test; calibrate against small human-labeled gold set; prefer pairwise to absolute scoring. Tools: G-Eval, DeepEval, Patronus, Braintrust.

Part 7 — MLOps

7.1 Experiment tracking

MLflow (OSS default), Weights & Biases (commercial), Neptune.ai / Comet, DVC + Git (small teams). Log: code hash, dataset hash, hyperparams, environment, train/val metrics, artifact, evaluation report.

7.2 Drift

Covariate shift P(x) changes, concept drift P(y|x) changes, label drift P(y) changes. Detection: PSI, JS divergence, KL divergence, KS test, chi-squared, Wasserstein. When labels arrive late, monitor proxies (prediction distribution, confidence). Tools: Evidently AI (best OSS), WhyLabs, Arize, Fiddler. W&B 2023 data: 62% of orgs see meaningful degradation within 12 months without monitoring.

7.3 Train-serve skew

Distinct from drift (drift = world changes; skew = code inconsistency). Feature parity (use a feature store: Feast, Tecton, Hopsworks). Schema parity (TFX SchemaGen, Great Expectations, Pandera). Lookup parity. Time-leakage (use as-of timestamps).

7.4 Deployment

Shadow, canary, A/B (tied to business metric), multi-armed bandit, champion/challenger.

7.5 Retraining triggers

Scheduled, drift-triggered, performance-triggered, continuous.

7.6 Reproducibility

Pinned deps (uv.lock/poetry.lock/conda-lock), hashed datasets (DVC, LakeFS, Delta), fixed seeds (not bit-identical on GPU), containerized training, MLflow run recording.

Anti-Patterns

• Tuning on the test set
• Mean accuracy on imbalanced data
• Different splits across compared models
• Shuffling a time series
• Ignoring calibration
• Validating the model in isolation rather than the pipeline
• Vibes-only LLM eval
• Treating the LLM as a fixed dependency
• Skipping monitoring because “it works in dev”

Related Skills

da-1-foundations-theory, da-1-3-probability-theory, da-1-4-statistical-inference-foundations, da-1-5-information-theory, da-1-6-epistemology-of-data, da-4-data-cleaning-preparation, da-6-statistical-modeling, da-8-data-visualization, da-9-reporting-communication, prompt-engineering, llm-context-engineering, rag-architecture, mongodb-atlas-vector-search, mongodb-search-ai, ai-datastores, ai-languages, llm-models, mongodb-atlas-stream-processing.

References

• Bergstra & Bengio (2012). Random Search for Hyperparameter Optimization. JMLR.
• Vaswani et al. (2017). Attention Is All You Need. NeurIPS.
• Liang et al. (2022). HELM. Stanford CRFM.
• Chiang et al. (2023). Chatbot Arena.
• Belkin et al. (2019). Double descent. PNAS.
• Hendrycks et al. (2021). MMLU. ICLR.
• Jimenez et al. (2024). SWE-bench. ICLR.
• LM Council, Vellum, Artificial Analysis — live LLM leaderboards (May 2026).
• Anthropic / OpenAI / Google DeepMind / Meta / DeepSeek / xAI / Zhipu — model release notes, April–May 2026.
• Evidently AI, Weights & Biases — MLOps drift industry reports (2023–2026).