Doug Turnbull Doug Turnbull's personal blog and essays http://softwaredoug.com/ Can agents replace the search stack? Instead of deploying the traditional query understanding + reranking combo, can we let an agent do all the work? Tue, 28 Apr 2026 00:00:00 +0000 http://softwaredoug.com/blog/2026/04/28/search-apis-replaced-by-agents.html http://softwaredoug.com/blog/2026/04/28/search-apis-replaced-by-agents.html Metadata: the 3rd kind of retrieval In search we talk about lexical embedding retrieval But we miss another retrieval philosophy metadata And with LLMs its never... Tue, 21 Apr 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/04/21/metadata-the-3rd-kind-of-retrieval.html http://softwaredoug.com/blog/2026/04/21/metadata-the-3rd-kind-of-retrieval.html What is psuedo-relevance feedback? After retrieving BM25 or any ranked search results you might not realize it but you have new information about the... Mon, 13 Apr 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/04/13/what-is-psuedo-relevance-feedback.html http://softwaredoug.com/blog/2026/04/13/what-is-psuedo-relevance-feedback.html Is grep all you need for RAG? With enough work you can build a search system with just grep. But it’s not for the faint of heart. Mon, 06 Apr 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/04/06/agentic-search-is-having-a-grep-moment.html http://softwaredoug.com/blog/2026/04/06/agentic-search-is-having-a-grep-moment.html Stop evaluating search with queries To evaluate search we typically build a judgment list We transform clickstream data into evaluation data This labels a result... Mon, 30 Mar 2026 13:56:40 +0000 http://softwaredoug.com/blog/2026/03/30/session-vs-query-based-search-evals.html http://softwaredoug.com/blog/2026/03/30/session-vs-query-based-search-evals.html Agents put the Resilient in RAG Classic RAG expects a linear process 1 An LLM translates the user’s prompt into a query 2 We search on... Thu, 26 Mar 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/03/26/classic-rag-s-achilles-heel-lack-of-resilience.html http://softwaredoug.com/blog/2026/03/26/classic-rag-s-achilles-heel-lack-of-resilience.html Why tiny late interaction models win Late interaction is having a moment The team at LightOn including superstar developer Antoine Chaffin has demonstrated how a 150M... Mon, 23 Mar 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/03/23/we-can-learn-from-this-late-interaction-win.html http://softwaredoug.com/blog/2026/03/23/we-can-learn-from-this-late-interaction-win.html How to actually choose a retrieval engine How do teams choose vector databases search engines People wrack their brains between Elasticsearch OpenSearch Solr Vespa Pinecone Turbopuffer Weaviate... Fri, 20 Mar 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/03/20/how-to-actually-choose-a-retrieval-engine.html http://softwaredoug.com/blog/2026/03/20/how-to-actually-choose-a-retrieval-engine.html Content understanding IS Query understanding A user searches for red shoes they click on some products Now you have a set of relevant products Great... Wed, 18 Mar 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/03/18/content-understanding-is-query-understanding.html http://softwaredoug.com/blog/2026/03/18/content-understanding-is-query-understanding.html Bayesian BM25 is cool Look at this math and grasp at its majesty P(R) = prob(embeddings) * prod(lexical) Mon, 16 Mar 2026 12:15:00 +0000 http://softwaredoug.com/blog/2026/03/16/bayesian-bm25-is-cool.html http://softwaredoug.com/blog/2026/03/16/bayesian-bm25-is-cool.html