Search this site:
Enterprise Search Blog

Do you really need a new search platform?

By: Miles Kehoe

So you have an existing enterprise search solution - either serving intranet or customer-facing content. You had great expectations for the application, but it hasn't worked out the way you had hoped. Your users are unhappy, and complain about not being able to find the information they need. They tell you “search sucks”. Maybe customers continue to call your support group for answers since they cannot find help on your web site. Or your sales remained flat after the roll-out of your new search. And management cannot help but feel the pain. What can you do?

We know of companies that, faced with these kinds of problems, went out and bought a new search engine. Sadly, they often find themselves in the same mess a year later, after wasting lots of time and money on the problem.

Most of these companies have a search technology that industry analysts have ranked as a top platform. You may have one of them yourself. But instead of giving up and trying your luck with the magic beans from another highly ranked vendor before you understand the nature of the problems you’re really having? You've got a good platform; now what you need is better methodology. The good news is that the methodologies of great enterprise search are not expensive, and they are not difficult to do. But you need to know what they are and be persistent.

Search applications are not like most enterprise applications - they need to be monitored and tuned as things change. Database applications are easy compared to search. Sure, the content may change every day, but the structure of the data - and the types of searches - are always of a known format. With search the content is changing all the time. But in addition, the queries and the types of queries are different every day. It seems to be a race between the search vendors to make a better, smarter search platform, and the universe to make a better dummy to do the searching. And the universe seems to be winning!

Do your users employ the same vocabulary your content authors use? Have you tried the searches yourself to identify the documents that come back? Have you looked for your products on your site? Do you know your top search queries? Many companies find that a histogram of query terms and frequencies shows a typical "long tail" graph. Some queries occur often, while others are outliers. Identify those common terms and build a methodology that enables them to work on improving the results for those queries. We call this the "Top 100" problem: if your top 100 queries return great results, you've made a good start. Then you move on to the next 100, and so on.

You also want to look at the "long tail". Terms there often indicate vocabulary mismatches, spelling errors, or a user who doesn't understand the nature of your content. Fix the former with thesaurus entries, synonym terms, or a good spelling solution. For the latter, you're on your own! When we’re brought into projects, we like to start by reviewing the requirements and the implementation, but our preference is to help our clients determine whether their existing search engine work the way it should. We start by recommending tools for our clients to monitor search activity and results, and by tuning results over time; we want to understand what the problem is. Usually, better monitoring and actively working to improve results will solve the problem.

Only when it’s clear that the current platform is beyond hope will we get involved in gathering and identifying requirements, interviewing stakeholders, and reviewing content creation and curation. At that point, then will we recommend alternative solutions; and ideally participate in a side-by-side onsite evaluation of the likely top vendors. Search is a technology that comes with high user expectations; and our job is to teach our clients how to deliver to those expectations!