I fixed Alamo Drafthouse's Search

At boutique movie-theater chain, Alamo Drafthouse, you order food using their QR Code app. You search for menu items (burger, popcorn, M&Ms) and submit an order.

The search leaves a lot to be desired. Typing in ‘burger’ shows nachos. It’s rather confusing.

As an annoyed, but loyal customer, I rebuilt and improved their search.

I’m going to walk through why Alamo search breaks then begin to fix it. You can see the original search here and my updated implementation here.

Today we’ll focus on signals. That’s the idea of storing historical queries and clicks, aggregating to learn what’s frequently clicked, and exposing that as a search-time dataset to use to improve search quality.

Why focus on this first? Alamo has a small corpus, decent amount of user traffic, and I assume most search traffic dominated by a handful of queries (i.e. a fat head). In these situations, learning from frequently repeated queries ranks high in interventions. And it’s far simpler for most teams than trying to fine-tune rerankers / embeddings to solve the same problem.

Alamo’s search expects you to navigate

Confusingly, Alamo’s site seems built like Apple’s Spotlight or Microsoft’s start menu search. It’s navigational search. Alamo treats its menu like a list of files to filter with the search terms.

Specifically, as you type, the results shrink to match to items that begin with your search string. Type “bur” and any menu item with a word starting with “bur” (ie burger) stays in the result set. As if you were typing ‘bur’ into a file-system search, filtering down to items with text beginning with ‘bur*’.

Problem 1: Irrelevant results outranking relevant ones

As you might expect, the prefix searching strategy might surface many false positives. There’s basically no ranking. It’s simply including / excluding menu items based on the text prefix match.

But even more glaring: there’s a sneaky text attribute that adds to the noise: customizations

A search of ‘bur’ matches on the ‘burger sauce’ customization in Chips & Queso. That doesn’t sound appetizing to me, it at least explains our many false positive results.

For today, we’re NOT going to directly solve the problem of what text to search. Instead, we’re going to indirectly solve this by searching for historical queries that start with the users search terms. If the user searches ‘bur’ and someone in the past searched for ‘burger’, we’ll rank higher ‘burger’ clicked results.

Many zero-result searches

Because it’s all just prefix matching, many reasonable searches produce return no menu items. Search for hamburger, get zero-results:

Same for kids pizza and kids meal. Some queries produce results, but omit many irrelevant ones like cheeseburger.

Again, we won’t solve this directly by improving corpus retrieval. What we’ll do instead is notice when the user types a prefix of a popular query (ie hamb -> hamburger). We’ll then notice, through an embedding, the semantic similarity between hamburger and burger, and use burger’s clickstream data to inform what should be shown for hamburger.

Seeding signals with agent traffic

As I mentioned, the most important thing for Alamo to do is to leverage historical search traffic to improve search. Not to train a model, but instead, build out a corpus of historical query to product click telemetry.

Obviously, I don’t have access to Alamo’s search telemetry. I don’t know what’s clicked / not clicked. I do have an agent and puppeteer that could seed search traffic.

Now massive disclaimer - agent traffic is not human traffic. Humans teach us different lessons then an agent clicking on a website would. If you’re working for Alamo Drafthouse and reading this, I wouldn’t (just) do what I’m about to do. Because you have actual human traffic to use.

Disclaimer out of the way, I used my favorite coding agent (OpenCode) along with an MCP to drive browsers (Puppeteer). Delegating use-cases to agents, I synthesize traffic on my Alamo Drafthouse search interface.

Specifically, I instruct an orchestrator agent with one or more use cases (i.e. “patron wants a burger and beer”). The orchestrator delegates to menu-searcher subagents, each with variations on the use case. Those subagents drive a headless browser, inspect results, and click relevant results. Instead of an LLM as a judge, I use LLM-as-a-user.

How do I track this?

On the backend, a Postgres database for tracks user telemetry to capture the clicks, specifically tracking:

• SERPs (search result pages) - essentially a session created once search results were rendered
• Impressions - what products were shown in the viewport of this SERP
• Clicks - impressions that received clicks

One problem that came up: lack of stochasticity. If a sub-agent was told to search for a burger, it would always click the “double cheeseburger”. That wasn’t super helpful. As a hack, I had the orchestrator instruct the subagent to click the Nth most relevant item, not the absolute most relevant. N would then be a random value (1-3) drawn from a python code snippet.

With a sufficiently seeded database, the orchestrator became more autonomous. I ask the orchestrator “based on the corpus, what doesn’t seem well covered in the search traffic?”. It would select missing use-cases and iterate. Recursive Self Improvement!

Eventually with a combination of automated and manual traffic generation, I had a decently seeded set of 2300 search sessions.

Boosting the most relevant carousel

My goal today isn’t to completely revamp Alamo’s experience. Instead, we’re going to build query-specific intelligence to mitigate problems and improve ranking incrementally.

We note, for example, a common use-case of searching for a topic (burger) and seeing the appetizers first. A low-hanging fruit then is to show the best category / subcategory based on frequently clicked products for that query.

I built a script that aggregates SERPs + clicks to compute the query’s most clicked category / sub_category. The script computes a queries table in Postgres from raw SERPs, impressions, and clicks. The computed queries table holds query-specific metadata useful to improve ranking. For now, I store this in Postgres, but it’s arguable whether a full-text search engine or redis cache would be preferable.

Our queries table looks like this:

Importantly, the table impacts more queries than those directly stored. When a user searches bur the goal stops being just ‘match items in the corpus with words beginning with bur’ but instead: “what’s the most popular query that begins with ‘bur’?”. That’s burger. We assume the user must be beginning to type this common query. So we use its category / sub_category metadata. As the user types, they see “Entrees / Sandwiches” carousel first.

Obvious limitations here and room for improvement:

• We’re not exactly ranking carousels for a query, we’re instead aggregating to get the most relevant category and sub_category. We could rank carousels based on clickstream data to get decreasing relevance. That might matter for a query like shake where shakes, sweet treats, and boozy milkshake cocktails span many different category / sub categories.
• We’re not yet ranking documents themselves yet. Within carousel, we rely just on the prefix matching. So we note the “Fried Chicken Sandwich” outranks the Garden Burger. We ought to extend our queries information to remember query / document traffic.
• Nothing here improves overall result accuracy. We’re ranking accurate carousels higher, but we still show Boneless Wings.

As always, search is never ‘solved’ - we have tremendous room for improvement. In future posts, we can continue to tackle in-carousel ranking.

Using query to query similarity to eliminate zero-results

The other goal for today: make a significant improvement to reduce the zero-result rate.

Again, we turn to the queries table. There we record our query hamburger with zero results, and therefore we store NULL in best_category or best_subcategory.

I’m a big fan of embeddings for use-cases besides direct retrieval. In this case, we can use them to find similar queries to ‘hamburger’. When building the queries table, we scan the non-zero result queries to find the most similar query, and add that to the table:

Given this linkage, we can rewrite the query hamburger -> burger and use historical search data about burger.

Now a search for hamb goes from zero results to results with Entrees / Sandwiches prioritized in ranking. As in the demo app and below.

This won’t capture every zero-result case. And importantly: it’s highly dependent on the ability of the embedding model to perform query-to-query similarity. I didn’t take much time to compare / analyze the embedding model here. I’m just using BAAI/bge-base-en-v1.5. Some misses were noted. Like a search for ‘spicy’ mapping to ‘spicy burger’. Almost certainly, a model focused on short e-commerce / food queries would work better here.

Another use-cases I haven’t solved: misspellings. For a mobile app, dependent on type-ahead, handling misspellings could help reduce zero result rates further. However, being a mobile app, perhaps the spell correction of the phone itself resolves this?

Past vibes to evals

Let’s look at an evaluation using the simulated search traffic. Keep in mind it’s the same search traffic also used to build the queries table. So we should expect an improvement.

I’m evaluating at the session level, NOT at the query level. I measure, for each search session (i.e. SERP table above), can we say this strategy (original vs signals) reduces the clicked document’s position? I also measure total number of sessions with zero results.

This differs from classic search evaluation where you measure at the query grain. In this methodology, many SERPs may search for ‘burger’ but click different items. The benefits of this approach: I’m not trying to model query-document relevance. I’m not making grand claims about what’s most relevant globally in the corpus for ‘burger’. I’m making a tighter and narrower claim: within search sessions, did I improve the click position?

Anyway, out of 2327 simulated sessions:

## Original
Zero-result sessions: 322
MRR (sessions with clicks): 0.6345
MRR (all sessions): 0.0913

## Signals
Zero-result sessions: 89
MRR (all sessions): 0.1081
MRR (sessions with clicks): 0.7506

Another thing I tell people when evaluating: don’t just look at the means, but the distribution. How many queries were improved / harmed? In this case: 19.6% of queries with clicks improved. None were harmed.

So a solid improvement. With clear room for improvement when we tackle direct product ranking.

When not to use a signals-first strategy

Why not use the user sessions as training data? Why are we directly aggregating / memorizing per-query metadata?

For Alamo, the corpus stays fairly static. We have voluminous search traffic over a handful of extremely common queries. So it makes sense to try to focus less on enriching and improving the corpus, and instead on what the queries tell us about ranking and retrieval.

This approach might be less suited for a corpus constantly in flux. Documents come and go from the corpus constantly. Even there, however, you can learn general patterns. Like this popular query tends to click on items in this category.

What about search that’s not dominated by such a fat head? But with a long tail? Again, the usage diminishes, but not entirely. You might still note with long tail, natural language searches that you can learn from traffic on similar queries. IE “What is the Capital of France?” vs “Where is France’s Capital?” basically are identical. They could arguably be aggregated together and treated as the same query. But this puts tremendous pressure on an embedding model to find query to query similarity.

When do you want to consider training a model? When you want to lift the torso and tail of the query distribution. Here you’re lifting queries NOT prevent in the training set. You want to lift queries you see infrequently. In Alamo, there’s almost no torso+tail, it’s all common queries. But once we get past that, we might prefer using this data to train a generalized model that can use the head of the distribution to learn deeper patterns and rank queries we haven’t seen before.

If you want to explore a project with me - this is fairly indicative of my work. I use agents to recreate how your search works, and I improve it. Relatively quickly! It’s a lot of fun! Get in touch, I’d love to discuss a project.