# Task-stratified retrieval evaluation > Source-anchored research for Task-stratified retrieval evaluation. Run status: BUDGET_EXHAUSTED. Parent: [Concept knowledge architecture strategies](https://llms-explorer.com/tree/concept-knowledge-architecture-strategies/) · 4 facets · 21 facts · page: https://llms-explorer.com/tree/task-stratified-retrieval-evaluation/ ## Structure and components - TREC test collections contain a document corpus, information needs, and relevance judgments. (low confidence) — [source](https://trec.nist.gov/howto.html) ## How it works - BEIR selected nDCG because it accommodates binary and graded relevance while respecting rank. (low confidence) — [source](https://arxiv.org/html/2104.08663) - NIST requires the document collection and relevance-judgment set to match when evaluating a run. (low confidence) — [source](https://trec.nist.gov/data/reljudge_eng.html) - Bruch et al. reduce evaluation cutoff for small corpora because recall at 1000 approaches one across methods. (low confidence) — [source](https://arxiv.org/html/2210.11934v2) ## Comparisons and alternatives - BEIR evaluates retrieval over heterogeneous tasks and domains rather than one homogeneous question-answering setting. (low confidence) — [source](https://arxiv.org/html/2104.08663) ## Facts and statements - BEIR found no retrieval approach consistently strongest on every included dataset. (low confidence) — [source](https://arxiv.org/html/2104.08663) - TREC pooling combines top results from multiple runs for relevance assessment. (low confidence) — [source](https://trec.nist.gov/howto.html) - Traditional pooled evaluation often scores unjudged documents as nonrelevant. (low confidence) — [source](https://trec.nist.gov/pubs/trec30/papers/Overview-2021.pdf) - BEIR's manual TREC-COVID hole annotation increased measured scores for non-lexical approaches. (low confidence) — [source](https://arxiv.org/html/2104.08663) - A TREC-8 study found near-identical system rankings after adding relevance judgments for modern neural retrieval results. (low confidence) — [source](https://arxiv.org/abs/2201.11086) - trec_eval reciprocal rank measures the first relevant retrieved document and suits a user needing one result. (low confidence) — [source](https://github.com/usnistgov/trec_eval/blob/main/m_recip_rank.c) - TREC separates the development period from evaluation data released without relevance labels. (low confidence) — [source](https://trec.nist.gov/howto.html) - Recall divides retrieved relevant items by all relevant items; precision and recall ignore their order. (low confidence) — [source](https://trec.nist.gov/pubs/trec15/appendices/CE.MEASURES06.pdf) - BEIR's TREC-COVID hole assessors did not know which retrieval system supplied each judged result. (low confidence) — [source](https://arxiv.org/html/2104.08663) - Upadhyay et al. report successful LLM hole filling in simulations that remove relevance judgments from three TREC DL datasets. (low confidence) — [source](https://arxiv.org/abs/2405.04727) - Soboroff argues that model-generated relevance labels make the model's judgments the evaluation target, limiting the evaluation's validity. (low confidence) — [source](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=958495) - The TREC 2024 RAG assessment study found UMBRELA-based system rankings highly correlated with manual rankings across 77 submitted runs. (low confidence) — [source](https://arxiv.org/abs/2411.08275) - ALCE evaluates generated text on separate fluency, correctness, and citation-quality dimensions. (low confidence) — [source](https://arxiv.org/abs/2305.14627) - RAGAS separates relevant focused retrieval, faithful use of context, and generation quality as distinct evaluation dimensions. (low confidence) — [source](https://arxiv.org/abs/2309.15217) - MultiHop-RAG includes queries, answers, and supporting evidence so evidence retrieval and reasoning given evidence can be evaluated separately. (low confidence) — [source](https://arxiv.org/abs/2401.15391) - The original BEIR neural retrieval experiments use only the first 512 word pieces of each document. (low confidence) — [source](https://arxiv.org/html/2104.08663)