One big reason search teams fail

What prevents search orgs from being successful? Is it technology? Lacking a new LLM thingy? Not implementing that new conference talk you saw? Is it having that one genius developer? Not having the right vector database? Hiring more people? Lack of machine learning?

No.

The thing that more often than not gets in the way is “politics”. Or more concretely: costly, unnecessary coordination between teams. Trying to convince other teams to unblock your feature.

Search orgs fail because of bad team organization leading to meetings, friction, disengagement, escalation, poor implementations, fighting, heroism, and burnout. In today’s obsession with software efficiency, and high user expectations, a poorly running search org, is simply fatal.

From functional silos to kernels

Orgs sometimes demarcate internal functional territory into teams. Nobody touches indexing code but the indexing team. To make a change, even a well trod one, that team does the work. Woe be unto you if you tried yourself, You’d be mired down in tribal knowledge and stuck in a slime pit of confusing deep specialist spaghetti code. You’d be like a foreigner in a strange land without a map. Likely you’d not be welcome too - how dare you tread on our territory!

You know the feature would take you a few measly hours if you just did it. But getting the indexing, etc team to understand the work, execute it the right way will take ten times that. And still probably be wrong. More infuriating is trying to get them to prioritize the work in their backlog: good luck getting them to care! Why should they be on call for more of your crap? Why should they add more maintenance burden to their codebase?

So much waste. Such a headache to build anything.

Yet functional specialization needs to exist. We need people that JUST build the important indexing parts, or manage the search infra, or build backend services. It’s important for teams in a non trivial search org to indeed focus on parts of the puzzle.

How do we solve this?

We absolutely have to, as leaders, ask our functional teams not just to build but to empower. We have to create structures that prevent escalation and politics, not invite them. Hire smart people that get shit done empower others to get shit done.

Devs must fully own their own feature soup-to-nuts, regardless of what repo it lives in. They should just be able to “do the work” without asking permission from a dozen teams. Without escalating to directors and VPs.

The role of specialist functional teams must be to eliminate themselves as scheduling dependencies, to get themselves out of the way, and to empower the feature dev with guardrails, training, and creating an “Operating System” where they own the kernel, not the apps. Let the feature devs own their destiny. NOT to own their “app”s themselves.

When you ship a new relevance feature: You should build and own your features indexing pipeline, the search engines config, search UI, and whatever else needed to build your functionality. You should also be on call for that, fix bugs, and ultimately be responsible for it working.

Kernels are not “platform”: they’re about shared code stewardship

Problem Solved. Life would be simpler if you just dove into the indexing codebase and built your indexing pipeline yourself, right? You’d build exactly what he needed, probably in half a day, and just get on with it.

Here’s the problem: you usually can’t easily dive into someone else’s functional area:

• Their code is built to be understood internally by their own teams, not by an “outsider”
• The code doesn’t clearly delineate “userland” code from that teams “kernel” code that empowers the user-space code
• There’s no examples of common “userland” implementation patterns done in a way the feature team understands.
• The specialist team doesn’t train or support you on how to work in their area
• They may be territorial and not open about their codebase.
• It’s not clearly defined who is on call for such “user land” code (its assumed the functional team - must be on call and that’s a mistake)

In short these teams keep their own ship in order and are distrustful of anyone coming to upset the applecart.

We actually need, in some ways, not strong lines, but co-ownership in every repo. It’s NOT an invitation for a complex, theoretical, platform full of speculative generality. It’s about inviting others into your specialist team’s codebase, with clear guardrails of what’s possible, and what should be avoided. Otherwise, teams will work around you, and you’ll have a case of Layerinitis. Teams will put code where its convenient, not where it belongs.

It all sounds AMAZING but it’s easier said than done. The big problem is how we get there.

This is hard. We’re setting a very high bar for our teams. They can’t just be smart and get shit done. They have to be smart and empower others to get shit done.

Empathy leads to good kernels: hire carefully

Hire people that care more about empowering and taking a back seat than taking credit. Creating a culture shared code stewardship starts with empathy, listening, and wanting to establish healthy boundaries. It starts with the factoring out dumbest obviously, useful common utilities across teams, and grows gradually to something more useful - NOT by spending months on up-front speculative generality.

• Hire really great generalists who can learn and readily shift roles between teams. IE “inverted T people”
• Hiring specialists is great, but ones that care more about empowering than doing. Hire professors, enablers, coaches, etc that work themselves out of a job.
• Hire for communication skills. Communication skills obviously include writing and speaking. But transfer to software engineering and building.
• Avoid status seekers. Willingness to be low status is a predictor of success.
• Hire for a growth mindset, not a fixed mindset. People that refuse to grow will “squat” in their comfort zone, get defensive about that area, and not want to let others in.
• Hire for empathy - people who really want to understand what others need and participate in building together.
• Careful about rewarding “direct impact” - this is tricky - the feature dev can get all the credit, but was standing on the shoulders of giants!

Embrace turnover

If you fear turnover, you have a problem. View turnover and switching teams as an opportunity to onboard more efficiently.

• Perhaps the #1 metric to pursue is “how quickly can somebody ship code in my codebase?”.
• Encourage team switching. Being new on a team is a great way to discover tribal knowledge blind spots and document them
• Don’t get stuck in a “hero” trap - avoid that amazing specialist that rides to the rescue constantly instead of empowering others to do that
• Get rid of squatters - team members who set up shop in one functional area and refuse to leave. Your best will eventually leave. And that’s OK.
• Expect your best to leave for other opportunities (internally or otherwise). If they’re really great they’ll leave behind a better situation than when they came
• Build an amazing developer experience that consistently enables people to get going without half a dozen bits of undocumented tribal knowledge

Build the kernel as if its open source

The “kernel” parts should feel like an open source project, general infrastructure that can serve multiple clients, but open to contributions and extensions.

• Treat your “kernel” as similar to an “internal open source tool” and the specialist team akin to committers.
• Have very clear lines between the kernel code and the feature code.
• Put the feature devs on call for their stuff, it’s not part of the core “kernel” open source
• Partner with feature devs on how to extend the kernel with new functionality. Oh you want to add a new vector search engine for your feature?
• Don’t speculatively generalize the kernel - build what works NOW. Refactor / rework as needed.
• Teams / functional areas that are constantly “in the way” rather than truly empowering will eventually be worked around! That’s OK! We should expect and encourage that in the marketplace of ideas in our company.
• Heck, why not open source the kernel? As I mention in my Haystack EU keynote, this is a great way to make your thing the center of the community’s mindshare, helping manage turnover and maintain a “kernel” even easier.

This is the really hard work of managing and building search (and any software org). It’s not doing the work, it’s how you structure the doing of the work to get out of people’s way. Yet it’s what you need to do to succeed and ship efficiently. Have really high standards here.

Additional Resources

• Amazon’s notion of tollbooths instead of guardrails in functional areas.
• The “Away Team” model of Amazon
• My talk with Chen Karako on how Data Science and Eng need to partner for relevance - A Search Divided Cannot Stand
• Platform Ecosystems Book - thanks Jesse Clark!
• Inner Sourcing
• BuildRightSide - book by former Shopify CTO Jean-Michel Lemieux

Shoutouts

Throughout my career, I’ve learned these lessons painfully and with coaching, encouragement, and modeling from lots of great leaders. Shoutout to Delaney Manders, Simon Eskildsen, Hannes Moser, Jesjit Birak, John Woodell, Tito Sierra, Max Irwin, René Kriegler, Eric Pugh, and many more I’m definitely forgetting!