RRF is Not Enough

Hybrid search means combining lexical and vector search results into one result listing.

“We’ll just use Reciprocal Rank Fusion” I’m sure I’ve said from time to time.

As if RRF is kind of “a miracle occurs”. You get the best of both worlds, and suddenly your search looks incredible.

Take the query hello to the planet. Let’s say we start with reasonable results from a vector search system (follow along in this notebook)

These are not bad. All the top results have to do with greeting ‘the planet’. Primarilly the planet earth.

We might notice a minor improvement we could make.

What if the user actually remembers text beginning with the phrase “hello to the planet”… Specifically they want the document beginning with “hello to the planet where I keep my stuff…”. If we added some lexical search, we might promote this to the top.

To perform RRF, we just also need the rank of the BM25 scores, and then we can merge the ranking with

RRF_score = 1/vector_rank + 1/bm25_rank

Easy enough.

We do this, and run our A/B test, only to see… 📉💥😢 Not Stonks.

What happened? We look under the hood at this specific query again, and…

Huh the results got WORSE!!

What happened!?

Well the BM25 results kind of suck for this query. Actually contradicting the already really good vector search results:

We’re getting the worst-case scenarios for bag of words results. The first result literally has nothing to do with.

RRF’ing bad search into good search will just drag down the good search. You actually have to give care that both sets of results deliver relevant search results to improve search.

How to use RRF

Use RRF, however, when you actually have distinct, disjoint sources of relevant search results. Each tuned to high precision.

If we change our BM25 solution to do phrase search instead of a bag of words query, we improve the precision of those results, and improve the overall experience.

When we have different retrieval sources, of very different technologies, we increase the likelihood of disjoint results. Now if we bias BOTH to give their highest degree of precision, and intentionally remove weird results, and let each focus in on a different, plausible use-case, we improve recall AND can trust the RRF score to reflect true definitions of overall relevance.

This is a bit counter to the conventional wisdom when combining retrieval sources. We usually say we want to cast a wide net at these early retrieval layers. But maybe, in the end, RRF is a great way to combine two precise retrieval sources into two precise result sets with a bit higher recall?

In this way, RRF improves recall and not precision?

Instead of RRF, first understand intent, then choose the best solution

In my opinion, a better path is to redefine the problem.

What’s the users intent with this query? Do they

(a) Want text similar to the “hello world” text? (b) Lookup a piece of text that uses this phrase?

Based on historical data, it’d be better to probabilistically decide which intent is more likely, then route the query accordingly to the best system to handle that query.

Perhaps we decide it’s 80% (a) vs 20% (b). Then we dedicate roughly 80% of our screen space to (a) and 20% to the other. We can now weight RRF accordingly

RRF_score = (80/vector_rank) + (20/bm25_rank)

We can keep going, why should we think of “vector search” and “bm25 search”? We ought to think in terms of intent:

RRF_score = (80/user_wants_semantically_similar_text) + (20/user_wants_to_closely_match_the_words)

That, we might think to generalize AWAY from thinking in terms of vector search and lexical search to systems solving the user’s specific problems. Towards query understanding. And we’ve always done this in search. Perhaps hybrid search simply means ‘choosing the right ranking solution for the job’.

Perhaps the REAL hybrid search has been inside of us all along ❤️