BM25 term frequency saturation and field length normalization

Parent: Atlas Hybrid Fusion and BM25 Diagnostics · Published reference · snapshot 2026-10-15 · viability 3.45 / 5 (research)

↓ Facts as markdown all context files

Depth-first rabbithole dossier for BM25 term frequency saturation and field length normalization; source-anchored research pack.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

[low] In the `stableTfl` model, `p` is a probability function based on Zipf's law computed as `p = 1 - (1 - m * 2 ^ (-c * tl)) ^ dl`, where `m` is a multiplicative constant, `c` is a decaying constant, `tl` is the query term length in unicode codepoints, and `dl` is the document length in tokens. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md [source]
[low] Weaviate refers to keyword search as BM25 search, as it is based on the BM25F scoring algorithm. Sources: https://weaviate.io/blog/hybrid-search-fusion-algorithms [source]

Structure and components

[low] For documents of average length, `k1` represents the term frequency value that yields half the maximum score for the considered term, with the score impact growing quickly when tf ≤ k1 and slower when tf > k1. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
[high] A `k1` value of 0 cancels out all factors except IDF, while a large `k1` value corresponds to using raw term frequency, and smaller `k1` values cause term scores to saturate much faster. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html [source]
[low] In BM25 scoring, the score difference between adding a second occurrence of a search term and the first is 0.061756645, and the difference between the third and second occurrence is 0.05225561, demonstrating diminishing returns. (verified as of 2026-10-15) Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
[low] With `k1 = 0.01`, the score influence of each additional occurrence drops off much faster than with `k1 = 5` or `k1 = 10`, indicating that smaller `k1` values lead to faster term frequency saturation. (verified as of 2026-10-15) Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
[high] The `k1` parameter is an internally specified term saturation parameter that affects how much the score increases with each reoccurrence of the term, limiting how much a single query term can affect the document score. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
[high] The `b` parameter is an internally specified length normalization parameter multiplied by the ratio of `dl` to `avgdl`, amplifying the effects of the length ratio as `b` increases. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables [source]
[high] In Elasticsearch, the default value for `b` is 0.75; setting `b` to 0 nullifies the effect of the length ratio, while a value of 1 corresponds to fully scaling the term weight by document length. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html [source]
[high] In Elasticsearch, the default value for `k1` is 1.2, and experiments suggest reasonable values range between 1.2 and 2. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html [source]

How it works

[low] The `discountOverlaps` flag, when true, discounts overlap tokens (tokens with a position increment of zero) from the document's length, and the `b` parameter must be between 0 and 1 to avoid illegal value exceptions. Sources: https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]

Comparisons and alternatives

[low] The `k3` parameter controls query-side term frequency saturation, computing duplicate term boosts as `((k3 + 1) * qtf) / (k3 + qtf)` instead of linear summing, with common values of 7 or 8 and a negative value disabling saturation. Sources: https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
[high] The `dl` field length in Elasticsearch is based on the number of terms rather than character length, and the `scoreDetails` description may indicate the length is approximate if the norm value exceeds 39. Sources: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
[low] The default implementation computes `avgdl` as `sumTotalTermFreq / docCount`. For the IDF statistics, Lucene uses `FieldStats#docCount()` instead of `IndexReader#numDocs()` because `TermStats#docFreq()` is used and, when it is inaccurate, so is `docCount()` in the same direction; `docCount()` also does not skew when fields are sparse. Sources: https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
[low] Lucene removed the `(k1 + 1)` factor in LUCENE-8563, reducing scores by a factor of 2.2 for `k1 = 1.2` compared to older versions, while Tantivy and Elasticsearch still use the old formula, explaining scoring differences with Atlas Search. (verified as of 2026-10-15) Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 [source]

Facts and statements

[high] MongoDB Search computes the BM25 term frequency (tf) using the formula `tf = freq / (freq + k1 * (1 - b + b * dl / avgdl))`, where `freq` is the number of occurrences of the term within the document. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
[medium] The `dl` variable represents the length of the field in the document, determined by the number of tokens produced by the analyzer, while `avgdl` is the average length of the field across all documents. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.mongodb.com/docs/search/indexes/field-types/string-type.md [source]
[low] The `stableTfl` scoring model computes the score as `boost * tr * tf`, where `tf` is a decaying function `freq / (freq + k1)` and `tr` is term rarity calculated as `log(1 + (1 - p + 0.05)/(p + 0.05))`. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md [source]
[low] The BM25 similarity constructor accepts `k1`, `b`, `discountOverlaps`, and `k3` parameters, and the score calculation uses `normInverse = 1f / (k1 * ((1 - b) + b * doclen / avgdl))` to compute `weight - weight / (1f + freq * normInverse)`. Sources: https://raw.githubusercontent.com/apache/lucene/main/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java [source]
Tantivy and Elasticsearch use the old BM25 formula with the `(k1 + 1)` factor, while Atlas Search uses the updated formula without it, leading to different score values. Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 [source]
[medium] The `scoreDetails` option, which defaults to `false`, breaks down scoring factors to analyze why a document matched, providing a `value` for the score contribution, a `description` of the formula subset, and recursive `details` for each match. Sources: https://www.mongodb.com/docs/search/query/score/get-details.md, https://www.mongodb.com/docs/manual/reference/operator/aggregation/score.md [source]
[low] Example score details show `tf` computed as 0.5398772954940796 with `k1` at 1.2000000476837158, `b` at 0.75, `dl` at 3, and `avgdl` at 4.888888835906982. Sources: https://dev.to/franckpachot/atlas-search-score-details-the-bm25-calculation-2g55 [source]

Related concepts

and — is a part of BM25 term frequency saturation and field length normalization
term — is a part of BM25 term frequency saturation and field length normalization
length — is a part of BM25 term frequency saturation and field length normalization
BM25 — is a part of BM25 term frequency saturation and field length normalization
frequency — is a part of BM25 term frequency saturation and field length normalization
saturation — is a part of BM25 term frequency saturation and field length normalization
field — is a part of BM25 term frequency saturation and field length normalization
normalization — is a part of BM25 term frequency saturation and field length normalization

Children

No children recorded.

← the whole tree · 3D view· how to read this page