# BM25 IDF corpus statistics > Depth-first rabbithole dossier for BM25 IDF corpus statistics; source-anchored research pack. Parent: [Atlas Hybrid Fusion and BM25 Diagnostics](https://llms-explorer.com/tree/mongodb-atlas-hybrid-fusion-bm25-frontier-20261001/) · 3 facets · 10 facts · page: https://llms-explorer.com/tree/bm25-idf-corpus-statistics/ ## Structure and components - [low] In a specific example, the IDF component was calculated as 1.8971199989318848 using `N` (total documents with field) of 9 and `n` (documents containing term) of 1. Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [low] Elasticsearch allows adding `?search_type=dfs_query_then_fetch` to gather distributed term frequencies before computing scores, adding an extra round trip to ensure consistent statistics across shards. Sources: https://www.elastic.co/blog/practical-bm25-part-1-how-shards-affect-relevance-scoring-in-elasticsearch — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [high] In Elasticsearch, the IDF component is implemented as `log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5))`, where `docCount` is the total documents with the field and `docFreq` is the number of documents containing the term. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [low] The IDF component penalizes common terms, meaning queries containing rarer terms receive a higher multiplier and contribute more to the final score. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [low] Elasticsearch switched to Okapi BM25 as the default similarity algorithm in version 5.0. (verified as of 2026-10-15) Sources: https://www.elastic.co/blog/practical-bm25-part-1-how-shards-affect-relevance-scoring-in-elasticsearch — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` ## How it works - [low] Because each MongoDB Search node builds indexes and performs updates independently, the document corpus may vary between nodes, causing `bm25` scores for the same documents to differ depending on which node handles the query. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#troubleshooting` - [low] Weaviate refers to keyword search as BM25 search because it is based on the BM25F scoring algorithm. Sources: https://weaviate.io/blog/hybrid-search-fusion-algorithms — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` ## Facts and statements - [high] MongoDB Search computes the inverse document frequency (IDF) using the formula `idf = log(1 + (N - n + 0.5) / (n + 0.5))`, where `N` is the total number of documents with the query field and `n` is the number of documents containing the term. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.mongodb.com/community/forums/t/score-varies-by-length/236529, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java, https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calcul — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [low] Atlas Search uses the same BM25 scoring implementation as Lucene, specifically referencing the BM25Similarity class in Lucene 10.3.2. (verified as of 2026-10-15) Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#core` - [low] A specific BM25 variant gives a negative term weight if a term occurs in over half the documents in the collection, which is considered undesirable. Sources: https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html — source: `~/.global-ai-hub/research/.pipeline/mongodb-2026-10-14/frb/bm25-idf-corpus-statistics-3e51baf90f/rabbithole-synthesis.md#antipatterns` ## Related concepts - BM25 — is a part of BM25 IDF corpus statistics - IDF — is a part of BM25 IDF corpus statistics - corpus — is a part of BM25 IDF corpus statistics - statistics — is a part of BM25 IDF corpus statistics