BM25 IDF corpus statistics
Parent: Atlas Hybrid Fusion and BM25 Diagnostics · Published reference · snapshot 2026-10-15 · viability 3.45 / 5 (research)
↓ Facts as markdownall context files
Depth-first rabbithole dossier for BM25 IDF corpus statistics; source-anchored research pack.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- [low] In a specific example, the IDF component was calculated as 1.8971199989318848 using `N` (total documents with field) of 9 and `n` (documents containing term) of 1. Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 [source]
- [low] Elasticsearch allows adding `?search_type=dfs_query_then_fetch` to gather distributed term frequencies before computing scores, adding an extra round trip to ensure consistent statistics across shards. Sources: https://www.elastic.co/blog/practical-bm25-part-1-how-shards-affect-relevance-scoring-in-elasticsearch [source]
- [high] In Elasticsearch, the IDF component is implemented as `log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5))`, where `docCount` is the total documents with the field and `docFreq` is the number of documents containing the term. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
- [low] The IDF component penalizes common terms, meaning queries containing rarer terms receive a higher multiplier and contribute more to the final score. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
- [low] Elasticsearch switched to Okapi BM25 as the default similarity algorithm in version 5.0. (verified as of 2026-10-15) Sources: https://www.elastic.co/blog/practical-bm25-part-1-how-shards-affect-relevance-scoring-in-elasticsearch [source]
How it works
- [low] Because each MongoDB Search node builds indexes and performs updates independently, the document corpus may vary between nodes, causing `bm25` scores for the same documents to differ depending on which node handles the query. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md [source]
- [low] Weaviate refers to keyword search as BM25 search because it is based on the BM25F scoring algorithm. Sources: https://weaviate.io/blog/hybrid-search-fusion-algorithms [source]
Facts and statements
- [high] MongoDB Search computes the inverse document frequency (IDF) using the formula `idf = log(1 + (N - n + 0.5) / (n + 0.5))`, where `N` is the total number of documents with the query field and `n` is the number of documents containing the term. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.mongodb.com/community/forums/t/score-varies-by-length/236529, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java, https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calcul [source]
- [low] Atlas Search uses the same BM25 scoring implementation as Lucene, specifically referencing the BM25Similarity class in Lucene 10.3.2. (verified as of 2026-10-15) Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 [source]
- [low] A specific BM25 variant gives a negative term weight if a term occurs in over half the documents in the collection, which is considered undesirable. Sources: https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html [source]
Related concepts
- BM25 — is a part of BM25 IDF corpus statistics
- IDF — is a part of BM25 IDF corpus statistics
- corpus — is a part of BM25 IDF corpus statistics
- statistics — is a part of BM25 IDF corpus statistics
Children
- No children recorded.