Task-stratified retrieval evaluation
Parent: Concept knowledge architecture strategies · Published reference · snapshot 2026-10-22
↓ Facts as markdownall context files
Source-anchored research for Task-stratified retrieval evaluation. Run status: BUDGET_EXHAUSTED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- TREC test collections contain a document corpus, information needs, and relevance judgments. (low confidence) [source]
How it works
- BEIR selected nDCG because it accommodates binary and graded relevance while respecting rank. (low confidence) [source]
- NIST requires the document collection and relevance-judgment set to match when evaluating a run. (low confidence) [source]
- Bruch et al. reduce evaluation cutoff for small corpora because recall at 1000 approaches one across methods. (low confidence) [source]
Comparisons and alternatives
- BEIR evaluates retrieval over heterogeneous tasks and domains rather than one homogeneous question-answering setting. (low confidence) [source]
Facts and statements
- BEIR found no retrieval approach consistently strongest on every included dataset. (low confidence) [source]
- TREC pooling combines top results from multiple runs for relevance assessment. (low confidence) [source]
- Traditional pooled evaluation often scores unjudged documents as nonrelevant. (low confidence) [source]
- BEIR's manual TREC-COVID hole annotation increased measured scores for non-lexical approaches. (low confidence) [source]
- A TREC-8 study found near-identical system rankings after adding relevance judgments for modern neural retrieval results. (low confidence) [source]
- trec_eval reciprocal rank measures the first relevant retrieved document and suits a user needing one result. (low confidence) [source]
- TREC separates the development period from evaluation data released without relevance labels. (low confidence) [source]
- Recall divides retrieved relevant items by all relevant items; precision and recall ignore their order. (low confidence) [source]
- BEIR's TREC-COVID hole assessors did not know which retrieval system supplied each judged result. (low confidence) [source]
- Upadhyay et al. report successful LLM hole filling in simulations that remove relevance judgments from three TREC DL datasets. (low confidence) [source]
- Soboroff argues that model-generated relevance labels make the model's judgments the evaluation target, limiting the evaluation's validity. (low confidence) [source]
- The TREC 2024 RAG assessment study found UMBRELA-based system rankings highly correlated with manual rankings across 77 submitted runs. (low confidence) [source]
- ALCE evaluates generated text on separate fluency, correctness, and citation-quality dimensions. (low confidence) [source]
- RAGAS separates relevant focused retrieval, faithful use of context, and generation quality as distinct evaluation dimensions. (low confidence) [source]
- MultiHop-RAG includes queries, answers, and supporting evidence so evidence retrieval and reasoning given evidence can be evaluated separately. (low confidence) [source]
- The original BEIR neural retrieval experiments use only the first 512 word pieces of each document. (low confidence) [source]
Children
- Temporal relevance evaluation (frontier)
- Evidence-set completeness (frontier)
- Judge calibration and disagreement (frontier)
Frontier under this node: Evidence-set completeness, Judge calibration and disagreement, Temporal relevance evaluation