Machine Learning
researched 2026-05-30T13:06:27.166Z· 6 sources · 33 concepts · skill da-7-machine-learning
Machine learning is the curriculum step where the analyst stops merely describing a sample and starts building a function that generalizes from data to unseen inputs. Section 6 (da-6-statistical-model
Machine Learning (Data Analysis Curriculum, Section 7)
- Machine learning is the curriculum step where the analyst stops merely describing a sample and starts building a function that generalizes from data to unseen inputs. Section 6 (da-6-statistical-modeling) covered parametric statistical models grounded in explicit probabilistic assumptions. This section widens the lens to algorithms that learn flexible, often non-parametric mappings from data - and to the engineering scaffolding (tuning, evaluation, deployment, monitoring) that turns a trained model into a system that keeps working. [source]
- This skill is the curriculum reference for the seventh section of the data analysis path. It is intentionally broad: it sketches the territory and points to the deeper skills you should pull in for any specific build. [source]
1.1 Three (now four) classical paradigms
- Supervised learning. Input–output pairs (x, y). Sub-shapes: classification (discrete y) and regression (continuous y). [source]
- Unsupervised learning. No labels. Discovers clusters (k-means, DBSCAN, HDBSCAN), topics (LDA, NMF), manifolds (PCA, t-SNE, UMAP, autoencoders), density (KDE, GMM), or anomalies (isolation forest, one-class SVM). [source]
- Reinforcement learning. Agent → environment → reward; optimizes policy pi(a | s). Modern flavors: PPO, GRPO, DQN. RLHF/RLAIF align LLMs. [source]
- Self-supervised learning. Labels constructed from the data: next-token prediction (GPT/Claude/Gemini), masked LM (BERT), masked patch (DINO/MAE), contrastive (CLIP). The engine behind every foundation model. [source]
- Edge shapes: semi-supervised (small labeled + large unlabeled), active learning (model picks next labels). [source]
1.2 Bias-variance tradeoff
- E[(y - f_hat(x))^2] = Bias[f_hat(x)]^2 + Var[f_hat(x)] + sigma^2 [source]
- Bias (underfitting) drops with capacity; variance (overfitting) rises. Regularization, more data, and ensembling trade variance for some bias. The classical U-curve is an idealization - in over-parameterized regimes (most modern DL) test error follows "double descent": rises near the interpolation threshold, then falls again as capacity grows. [source]
1.3 Regularization toolbox
- L1 (Lasso, feature selection), L2 (Ridge / weight decay, default in DL), ElasticNet, Dropout (ensemble averaging), early stopping, data augmentation (often the strongest), label smoothing, normalization layers. [source]
1.4 Classical model zoo
- Linear/logistic regression, decision trees, random forests, gradient-boosted trees (XGBoost/LightGBM/CatBoost still win most tabular competitions in 2026), SVM, kNN, Naive Bayes. On tabular: start with a GBT. [source]
2.1 CNNs
- Grid-structured data. Convolution (weight sharing, translation equivariance), pooling, hierarchical feature learning. LeNet → AlexNet → VGG → ResNet (residual connections) → EfficientNet → ConvNeXt. Still competitive on edge devices, limited-data medical imaging, and as hybrid backbones. [source]
2.2 RNNs
- Sequential state h_t = f(x_t, h_{t-1}). Vanilla RNNs (vanishing gradients), LSTM (gated cell, workhorse 2014–2018), GRU (simpler, comparable), bidirectional. Mostly displaced by Transformers; still useful for streaming inference, tiny edge time-series, and as the conceptual ancestor of state-space models (Mamba, S4, S6) that scale linearly in sequence length. [source]
2.3 Architecture choice
- Transformer is the 2026 default, but consider inductive bias (CNNs encode locality), sequence length (attention is quadratic - sliding window/sparse/linear/SSMs win on long contexts), and latency budget (a 7B Transformer is overkill for keyword-spotting on a watch). [source]
3.1 The Transformer
- Attention Is All You Need (Vaswani et al., 2017). Per token: query Q, key K, value V. Attention = softmax(QK^T / sqrt(d_k)) V. Multi-head runs several attention ops in parallel. Parallelizable training, one-hop long-range dependencies, modality-agnostic (text, images via ViT, audio via Whisper, proteins via AlphaFold). [source]
3.2 Foundation model paradigm
- Pretrain large on broad data with self-supervision, adapt via: zero-shot prompting, few-shot in-context learning, fine-tuning (full, LoRA/QLoRA, DPO/KTO, RLHF/RLAIF), RAG, tool use / function calling. 2024–2026 era is agent-shaped systems built around pretrained foundations. [source]
Part 4 — Frontier LLM Landscape (May 2026)
- Selection heuristics: agentic coding → Claude Opus 4.7; general reasoning → GPT-5.5; multimodal → Gemini 3.1 Pro; self-hosted → Llama 4; budget/high-volume → DeepSeek V3.2 or smaller specialists. Use LiteLLM/OpenRouter for per-request routing. Treat the LLM as replaceable, not a vendor commitment. [source]
5.1 Search strategies
- Manual, grid (only ≤3 hyperparams), random (dominates grid when few hyperparams matter), Bayesian optimization (GP or TPE + EI/UCB acquisition, default for 10–100 trials), HyperBand (successive halving), BOHB (BO + HyperBand, production default for DL), PBT (population-based, learns schedules). [source]
5.2 Tooling (2026)
- Optuna (Python-native, TPE + pruning, individual default), Ray Tune (distributed, all algorithms above), Weights & Biases Sweeps (if W&B is your tracker), KerasTuner/AutoGluon/FLAML (AutoML), Vizier/SigOpt (hosted). [source]
5.3 Cheatsheet
- Tune on validation, never on test. Nested CV when small. Log-uniform LR/weight decay. Tune learning rate first. Cap wall-clock per trial. Re-tune on dataset/architecture/optimizer change. Save trial history for get_param_importances. [source]
6.1 Splits
- Train/val/test (test touched once). k-fold CV, stratified k-fold (imbalanced), group k-fold (leakage by group ID), time-series splits (never shuffle), nested CV (small data + many hyperparams). [source]
6.2 Classification metrics
- Accuracy (misleading on imbalance). Precision TP/(TP+FP) - optimize when false positives expensive. Recall TP/(TP+FN) - optimize when false negatives expensive. F1 harmonic mean. F-beta. ROC-AUC (insensitive to class balance). PR-AUC (preferred when positive class rare). Log-loss / cross-entropy (proper scoring rule). Calibration plots + Brier score. Macro/weighted/micro F1 for multi-class. [source]
6.3 Regression metrics
- MAE (robust), RMSE (penalizes outliers), MAPE / sMAPE, R^2, quantile / pinball loss. [source]
6.4 Ranking metrics
- MAP@k, NDCG@k, MRR, Hit-rate@k. [source]
6.5 LLM evaluation
- MMLU (saturated - frontier >90%), MT-Bench (multi-turn, LLM-as-judge), HELM (Stanford CRFM; accuracy, calibration, robustness, fairness, bias, toxicity, efficiency), Chatbot Arena (Elo from pairwise preference), SWE-bench / SWE-bench Verified / SWE-bench Pro (real GitHub issues; Claude Opus 4.7 leads Verified at 87.6%), HumanEval / MBPP / LiveCodeBench, GPQA / ARC-AGI / FrontierMath (not yet saturated), domain-specific (MedQA, LegalBench, FinanceBench, MMMU). [source]
6.6 LLM-as-judge
- Pros: fast, scalable. Cons: position bias, self-preference, verbosity bias, rubric drift. Mitigations: randomize position; use different judge family than model under test; calibrate against small human-labeled gold set; prefer pairwise to absolute scoring. Tools: G-Eval, DeepEval, Patronus, Braintrust. [source]
7.1 Experiment tracking
- MLflow (OSS default), Weights & Biases (commercial), Neptune.ai / Comet, DVC + Git (small teams). Log: code hash, dataset hash, hyperparams, environment, train/val metrics, artifact, evaluation report. [source]
7.2 Drift
- Covariate shift P(x) changes, concept drift P(y|x) changes, label drift P(y) changes. Detection: PSI, JS divergence, KL divergence, KS test, chi-squared, Wasserstein. When labels arrive late, monitor proxies (prediction distribution, confidence). Tools: Evidently AI (best OSS), WhyLabs, Arize, Fiddler. W&B 2023 data: 62% of orgs see meaningful degradation within 12 months without monitoring. [source]
7.3 Train-serve skew
- Distinct from drift (drift = world changes; skew = code inconsistency). Feature parity (use a feature store: Feast, Tecton, Hopsworks). Schema parity (TFX SchemaGen, Great Expectations, Pandera). Lookup parity. Time-leakage (use as-of timestamps). [source]
7.4 Deployment
- Shadow, canary, A/B (tied to business metric), multi-armed bandit, champion/challenger. [source]
7.5 Retraining triggers
- Scheduled, drift-triggered, performance-triggered, continuous. [source]
7.6 Reproducibility
- Pinned deps (uv.lock/poetry.lock/conda-lock), hashed datasets (DVC, LakeFS, Delta), fixed seeds (not bit-identical on GPU), containerized training, MLflow run recording. [source]
Anti-Patterns
- Tuning on the test set [source]
- Mean accuracy on imbalanced data [source]
- Different splits across compared models [source]
- Shuffling a time series [source]
- Ignoring calibration [source]
- Validating the model in isolation rather than the pipeline [source]
- Vibes-only LLM eval [source]
- Treating the LLM as a fixed dependency [source]
- Skipping monitoring because "it works in dev" [source]
Related Skills
- da-1-foundations-theory, da-1-3-probability-theory, da-1-4-statistical-inference-foundations, da-1-5-information-theory, da-1-6-epistemology-of-data, da-4-data-cleaning-preparation, da-6-statistical-modeling, da-8-data-visualization, da-9-reporting-communication, prompt-engineering, llm-context-engineering, rag-architecture, mongodb-atlas-vector-search, mongodb-search-ai, ai-datastores, ai-languages, llm-models, mongodb-atlas-stream-processing. [source]
References
- Bergstra & Bengio (2012). Random Search for Hyperparameter Optimization. JMLR. [source]
- Vaswani et al. (2017). Attention Is All You Need. NeurIPS. [source]
- Liang et al. (2022). HELM. Stanford CRFM. [source]
- Chiang et al. (2023). Chatbot Arena. [source]
- Belkin et al. (2019). Double descent. PNAS. [source]
- Hendrycks et al. (2021). MMLU. ICLR. [source]
- Jimenez et al. (2024). SWE-bench. ICLR. [source]
- LM Council, Vellum, Artificial Analysis - live LLM leaderboards (May 2026). [source]
- Anthropic / OpenAI / Google DeepMind / Meta / DeepSeek / xAI / Zhipu - model release notes, April–May 2026. [source]
- Evidently AI, Weights & Biases - MLOps drift industry reports (2023–2026). [source]
Children
- ML Taxonomy (frontier)
- Bias-Variance Tradeoff (frontier)
- Regularization (frontier)
- Deep Learning (frontier)
- CNNs (frontier)
- RNNs (frontier)
- Transformers (frontier)
- Foundation Models (frontier)
- LLM Landscape 2026 (frontier)
- Hyperparameter Tuning (frontier)
- Bayesian Optimization (frontier)
- Optuna (frontier)
- Ray Tune (frontier)
- Model Evaluation (frontier)
- Precision Recall F1 (frontier)
- ROC AUC (frontier)
- Regression Metrics (frontier)
- LLM Evaluation (frontier)
- HELM (frontier)
- MMLU (frontier)
- MT-Bench (frontier)
- Chatbot Arena (frontier)
- SWE-bench (frontier)
- LLM-as-Judge (frontier)
- MLOps (frontier)
- Drift Detection (frontier)
- Train-Serve Skew (frontier)
- MLflow (frontier)
- Weights and Biases (frontier)
- Feature Stores (frontier)
- Deployment Patterns (frontier)
- Retraining Triggers (frontier)
- Reproducibility (frontier)
- Deep Reinforcement Learning Foundations
Frontier under this node: Bayesian Optimization, Bias-Variance Tradeoff, CNNs, Chatbot Arena, Deep Learning, Deployment Patterns, Drift Detection, Feature Stores, Foundation Models, HELM, Hyperparameter Tuning, LLM Evaluation, LLM Landscape 2026, LLM-as-Judge, ML Taxonomy, MLOps, MLflow, MMLU, MT-Bench, Model Evaluation, Optuna, Precision Recall F1, RNNs, ROC AUC, Ray Tune, Regression Metrics, Regularization, Reproducibility, Retraining Triggers, SWE-bench, Train-Serve Skew, Transformers, Weights and Biases