High IDF doesn't always mean relevant search

Rare terms have high inverse document frequency (IDF). BM25 scoring treats high IDF terms as more relevant.

Why?

We assume if a term occurs rarely in the corpus, it must unambiguously point to what the user wants. It’s specific.

But that’s not always true. Not all text is created equal. Corpuses violate this assumption frequently. Why?

• No need to use a common term - Book titles may rarely mention the word “book”, but clearly “book” in a book index has low specificity.
• Language gaps between corpus + users - a medical corpus may not mention “heart attack” much. It may use the technical term “myocardial infarction” frequently. Yet a novice searching for “heart attack” still has a fairly generic intent.
• Tags + editorial controls - some fields don’t have consistent editorial controls. For example, search teams might shove tags into search indices and hope for the best. Document frequency becomes a phenomenon of human inconsistencies, not natural language.

Teams currently work around these problems by:

• ​Cross field or BM25F search: using a BM25 formulation that combines field statistics, importantly - the doc freq. So a more natural doc frequency for “book” might not be in a book’s title field, but in its body
• Merging text into an “all field”: using features like copy fields in lucene to just create a “text_all” for baseline matching. With everything merged we get more accurate stats
• Manual overrides - it’s pretty cool that Vespa lets you manually specify term significance. I even once built a Solr plugin for managed stats to let you directly manipulate document frequency.

AI Powered Search training starts TOMORROW! Signup here:

​http://maven.com/search-school/ai-powered-search​

-Doug

​

​

This is part of Doug’s Daily Search tips - subscribe here