Recommender Systems and Learning-to-Rank Analytics

Parent: data analysis · researched 2026-05-31T04:50:26.503Z· 25 sources · 11 concepts · skill da-38-recommender-systems-and-ranking

A recommender system predicts, for each user, which items from a (often huge) catalog they are most likely to engage with, then orders a small slate to show. As a data-analysis discipline it sits at t

Overview

A recommender system predicts, for each user, which items from a (often huge) catalog they are most likely to engage with, then orders a small slate to show. As a data-analysis discipline it sits at the intersection of three problems: (1) modeling preference from sparse interaction data, (2) ranking candidates under a relevance objective, and (3) evaluating both offline and online while fighting the bias the system itself creates. [source]
Scope boundaries (read first): [source]
vs da-7 (machine learning): da-7 covers generic supervised/unsupervised model training, regularization, and hyperparameter tuning. da-38 covers the recommendation- and ranking-specific objectives (BPR/WRMF losses, NDCG-aware LambdaMART), the retrieve-then-rank funnel, and recsys evaluation. Use da-7 for "train a classifier"; use da-38 for "rank items for a user." [source]
vs da-27 (network/graph analytics): da-27 owns graph structure metrics (centrality, community detection, link prediction as graph analysis, GNNs as graph models). da-38 references GNN-for-recsys and bipartite user-item graphs only at the recommendation level; graph-structure questions belong to da-27. [source]
vs da-12 (A/B testing & causal inference): da-12 owns general experiment design and causal estimators. da-38 owns the recsys-specific online-eval wrinkles: interleaving, feedback-loop confounding, and off-policy/counterfactual evaluation (IPS, doubly-robust) of a ranking policy. [source]
vs rag-architecture / vector search: semantic retrieval with no personalization or ranking-quality objective is RAG; ranked personalization is da-38. [source]

1. Problem framing: feedback, the feedback loop, and cold-start

Explicit vs implicit feedback. Explicit = ratings/likes (sparse, signed). Implicit = clicks, views, dwell, purchases (abundant but positive-only and ambiguous - a non-click is not a dislike). Implicit dominates production and forces positive-unlabeled modeling: model confidence that an interaction is a preference, not a rating value. [source]
Popularity bias & the feedback loop. A deployed recommender only logs feedback on items it showed, chosen because they scored high - so the next training set over-represents popular/previously-recommended items (exposure bias). Unchecked, a self-reinforcing loop narrows the catalog and creates filter bubbles. Breaking it needs exploration and/or de-biasing (propensity weighting). [source]
Cold-start, three flavors. User (new user), item (new item), system (brand-new product). Mitigations: content/side features, hybrid models, factorization machines, popularity fallbacks, bandit exploration, and (2025-26) LLM/semantic-ID content grounding for long-tail items. [source]

2. Collaborative filtering (CF) — neighborhood methods

CF predicts preference purely from the user-item interaction matrix, no content. [source]
User-user CF: find similar users (cosine/Pearson on co-rated items). Sensitive to sparsity; similarities go stale. [source]
Item-item CF: precompute item-item similarity ("users who interacted with X also interacted with Y"). More stable than user-user, scales better, Amazon's production workhorse, still a strong baseline. [source]
Limitations: cold-start, sparsity, popularity skew, no side features - motivating latent-factor models. [source]

3. Matrix factorization (MF)

Factor R ≈ P·Qᵀ into low-rank user (P) and item (Q) latent factors; prediction = pᵤ·qᵢ. [source]
SVD / funkSVD. Not literal SVD - learns factors by SGD on observed entries only with L2 regularization plus user/item bias terms. SVD++ adds implicit signal. [source]
ALS. Fix P, solve Q in closed form, alternate. Parallel → standard for large distributed (Spark) training. [source]
Implicit-feedback MF / WRMF (Hu-Koren-Volinsky 2008). Treat all entries; weight observed interactions by confidence cᵤᵢ = 1 + α·rᵤᵢ; fit preference (0/1) with weighted ALS. What implicit's AlternatingLeastSquares does. [source]
BPR - Bayesian Personalized Ranking (Rendle 2009). Pairwise ranking objective for implicit feedback: maximize σ(x̂ᵤᵢ − x̂ᵤⱼ) over sampled (user, pos, neg) triples. Optimizes ranking (AUC), not rating error - better top-N fit than pointwise MF. [source]

4. Content-based, hybrid, and factorization machines

Content-based: recommend items similar to liked ones via item features → good for item cold-start, over-specializes (no serendipity). [source]
Hybrid: combine CF + content (weighted, switching, feature-augmented, cascade). Solves cold-start, keeps collaborative signal. [source]
Factorization Machines (FM, Rendle 2010): model all pairwise feature interactions with low-rank factorized weights - generalizes MF to arbitrary side features, natively handles cold-start. FFM (field-aware) gives each feature a latent vector per field - strong for CTR. DeepFM shares an embedding between a wide FM (low-order) and a deep DNN (high-order), no manual crosses - standard CTR/ranking model. [source]

5. Modern deep recommenders

Two-tower retrieval. Separate user/query tower and item tower into a shared embedding space; relevance = dot product. Item embeddings precomputed and indexed in an ANN/vector store → sub-linear nearest-neighbor lookup → dominant candidate-generation architecture (TorchRec's headline target). [source]
Neural CF (NCF): replace the MF dot product with an MLP - more expressive (though a tuned dot product is a strong baseline). [source]
Sequential / session-based: GRU4Rec (RNN over session events); SASRec (causal left-to-right self-attention, next-item); BERT4Rec (bidirectional self-attention, masked-item cloze training). Caveat: with the same loss, SASRec generally matches or beats BERT4Rec at lower cost - BERT4Rec's edge often came from its training objective, not bidirectionality. [source]
LLM-augmented & generative recommenders (2025-2026). Semantic IDs: quantize an item's content embedding (RQ-VAE) into a short code that reflects content → generalizes to cold-start/long-tail and slots into an LLM vocabulary (YouTube, Spotify gains). Generative retrieval: an LLM generates the next item's semantic ID instead of scoring a candidate set. LLMs as data augmenters / rerankers and knowledge-guided RAG (ColdRAG) for cold-start - watch LLM-reranker exposure/coverage issues. [source]

6. Learning-to-rank (LTR)

Directly optimize the order of a candidate list given relevance labels/features. [source]
Pointwise - predict each item's relevance independently; ignores list context. Weakest ranking fidelity. [source]
Pairwise - learn from item pairs. RankNet (neural, pairwise cross-entropy) is the archetype; its loss correlates only loosely with NDCG. [source]
Listwise - optimize the whole list. ListNet uses Plackett-Luce permutation probability; LambdaRank/LambdaMART weight pairwise gradients by ΔNDCG, directly targeting NDCG. LambdaMART (LambdaRank gradients + gradient-boosted trees) is the workhorse - XGBoost (rank:ndcg), LightGBM, RankLib. Pairwise-vs-listwise is how the loss treats the list, not the model family. [source]
Retrieval → ranking → re-ranking funnel. (1) candidate generation - cheap, high-recall, millions→hundreds (two-tower+ANN, item-item, popularity); (2) ranking - expensive, high-precision scorer (DeepFM/LambdaMART/DLRM); (3) re-ranking - business rules, diversity (MMR), freshness, fairness, exploration on the top slate. Each stage trades recall for precision. [source]

7. Offline evaluation

Evaluate on held-out interactions (time-based split more honest than random). [source]
Accuracy @k: Precision@k, Recall@k, Hit Rate@k, MAP (order-aware), MRR (rank of first relevant - best when one correct answer, e.g. next-item), NDCG@k (graded, position-discounted - the headline metric). Empirically cluster: {Recall}, {MRR, NDCG, HR}, {Precision, MAP}. [source]
Beyond-accuracy: Coverage, Diversity (intra-list dissimilarity), Novelty (non-popularity), Serendipity (relevant and surprising). Optimizing accuracy alone degrades these. [source]
Offline/online gap. Offline metrics measure fit to logged (biased) behavior. Higher offline NDCG does not guarantee online lift; too much novelty can hurt novice users. Treat offline as a filter, beware sampled-negative distortion and random-split leakage. [source]

8. Online evaluation & off-policy estimation

A/B testing is the gold standard, but recommenders create feedback loops that contaminate populations over time - keep tests short, watch novelty/interference. [source]
Interleaving mixes two rankers into one list per user and attributes clicks → far more sensitive than A/B (Airbnb search). [source]
Counterfactual / off-policy evaluation (OPE): IPS (weight by 1/propensity - unbiased but high variance, clip it); Direct Method (reward model - low variance, biased if misspecified); Doubly Robust (DM+IPS - unbiased if either model is correct, lower variance; standard for large action spaces). [source]

9. Production concerns

Two-stage serving (candidate gen vs ranking); embeddings in a vector/ANN store, features in a feature store (Feast) shared across train/serve to prevent skew. [source]
Real-time features & embeddings: session/recency features computed at request time; huge embedding tables sharded (TorchRec). [source]
Exploration vs exploitation: contextual bandits (LinUCB, Thompson sampling) inject controlled exploration to break feedback loops; ε-greedy/random injection in re-ranking mitigates filter bubbles. Caveat (RecSys 2025): in pure offline eval, greedy models often appear to beat bandits - a structural eval bias. [source]
Fairness & filter bubbles: monitor provider-side exposure fairness, popularity bias, diversity; randomization/fairness constraints in re-ranking. [source]

Methodology

Frame the problem. Implicit/explicit? Top-N, CTR, or next-item? Decides loss (BPR/WRMF vs pointwise vs LambdaMART) and metric (Recall@k/NDCG vs MRR). [source]
Baseline first. Popularity + item-item CF + WRMF/BPR - many "deep" wins vanish against a tuned MF baseline. [source]
Split honestly. Time-based (leave-last-out per user); avoid leakage; beware sampled-negative metrics. [source]
Add structure as needed. Side features → FM/LightFM/DeepFM; sequence → SASRec; cold-start → content + semantic IDs. [source]
Build the funnel at scale: two-tower retrieval → DeepFM/LambdaMART ranker → diversity/fairness/exploration re-rank. [source]
Evaluate in layers: offline → off-policy (IPS/DR) → interleaving → A/B. Never ship on offline alone. [source]
Close the loop safely: log propensities, add exploration, monitor popularity bias and exposure fairness. [source]

Anti-Patterns

Trusting offline NDCG as ground truth - confirm online. [source]
Treating implicit non-interactions as negatives - they're unlabeled; use confidence weighting (WRMF) or sampled pairwise negatives (BPR). [source]
Random split for sequential/temporal data - leaks the future. [source]
Assuming BERT4Rec > SASRec - loss, not bidirectionality, drove the gap. [source]
Optimizing accuracy only - tanks coverage/diversity/serendipity and feeds the feedback loop. [source]
Raw IPS with no clipping - variance explodes; clip or use doubly-robust. [source]
Deep model with no MF/CF baseline - can't claim a win without the bar. [source]

References

Hu/Koren/Volinsky (2008), CF for Implicit Feedback Datasets (WRMF). https://dl.acm.org/doi/10.1145/1864708.1864726 [source]
Rendle et al. (2009), BPR. https://arxiv.org/pdf/1205.2618 [source]
He et al. (2016), Fast MF for Online Recommendation with Implicit Feedback. https://dl.acm.org/doi/10.1145/2911451.2911489 [source]
Guo et al. (2017), DeepFM. https://arxiv.org/pdf/1703.04247 [source]
Two-Tower Model for Recommendation (Shaped). https://www.shaped.ai/blog/the-two-tower-model-for-recommendation-systems-a-deep-dive [source]
Sun et al. (2019), BERT4Rec. https://arxiv.org/pdf/1904.06690 [source]
Petrov & Macdonald (2023), is BERT4Rec really better than SASRec? https://arxiv.org/pdf/2309.07602 [source]
Spotify Research (2025), Semantic IDs for Generative Search and Recommendation. https://research.atspotify.com/2025/9/semantic-ids-for-generative-search-and-recommendation [source]
Semantic IDs for Joint Generative Search and Recommendation (RecSys 2025). https://dl.acm.org/doi/10.1145/3705328.3759300 [source]
ColdRAG (2025). https://arxiv.org/html/2505.20773v2 [source]
From RankNet to LambdaMART. https://en.heth.ink/Ranking/ [source]
XGBoost Learning to Rank. https://xgboost.readthedocs.io/en/latest/tutorials/learning_to_rank.html [source]
Evaluating Recommender Models: Offline vs. Online (Shaped). https://www.shaped.ai/blog/evaluating-recommender-models-offline-vs-online-evaluation [source]
10 metrics to evaluate recommender and ranking systems (Evidently). https://www.evidentlyai.com/ranking-metrics/evaluating-recommender-systems [source]
Widespread Flaws in Offline Evaluation of Recommender Systems (2023). https://arxiv.org/pdf/2307.14951 [source]
Interleaving and Counterfactual Evaluation for Airbnb Search Ranking (KDD 2025). https://arxiv.org/html/2508.00751v1 [source]
Doubly Robust OPE with Large Action Spaces. https://www.researchgate.net/publication/372961616 [source]
Introducing TorchRec (PyTorch). https://pytorch.org/blog/introducing-torchrec/ [source]
Recommender Systems: Lessons From Building and Deployment (Neptune). https://neptune.ai/blog/recommender-systems-lessons-from-building-and-deployment [source]
Exploitation Over Exploration (RecSys 2025). https://dl.acm.org/doi/10.1145/3705328.3748166 [source]

Children

Problem Framing (Explicit vs Implicit Feedback, Feedback Loop, Cold-Start) (frontier)
Collaborative Filtering (User-User, Item-Item, Neighborhood) (frontier)
Matrix Factorization (SVD, funkSVD, ALS, WRMF, BPR) (frontier)
Content-Based and Hybrid Recommenders (frontier)
Factorization Machines (FM, FFM, DeepFM) (frontier)
Modern Deep Recommenders (Two-Tower, Neural CF, Sequential SASRec/BERT4Rec/GRU4Rec) (frontier)
LLM-Augmented and Generative Recommenders (Semantic IDs) (frontier)
Learning-to-Rank (Pointwise, Pairwise RankNet, Listwise ListNet/LambdaMART) (frontier)
Offline Evaluation Metrics (NDCG, MAP, MRR, Recall@k, Coverage, Diversity, Serendipity) (frontier)
Online and Off-Policy Evaluation (A/B, Interleaving, IPS, Doubly-Robust) (frontier)
Production Concerns (Candidate Generation, Feature/Embedding Stores, Bandits, Fairness) (frontier)

Frontier under this node: Collaborative Filtering (User-User, Item-Item, Neighborhood), Content-Based and Hybrid Recommenders, Factorization Machines (FM, FFM, DeepFM), LLM-Augmented and Generative Recommenders (Semantic IDs), Learning-to-Rank (Pointwise, Pairwise RankNet, Listwise ListNet/LambdaMART), Matrix Factorization (SVD, funkSVD, ALS, WRMF, BPR), Modern Deep Recommenders (Two-Tower, Neural CF, Sequential SASRec/BERT4Rec/GRU4Rec), Offline Evaluation Metrics (NDCG, MAP, MRR, Recall@k, Coverage, Diversity, Serendipity), Online and Off-Policy Evaluation (A/B, Interleaving, IPS, Doubly-Robust), Problem Framing (Explicit vs Implicit Feedback, Feedback Loop, Cold-Start), Production Concerns (Candidate Generation, Feature/Embedding Stores, Bandits, Fairness)

← the whole tree · 3D view· how to read this page