As we continue this series on search and how it works, we have to address accuracy. First, how are we are going to measure accuracy?

  • Relevance
  • Recall
  • Precision
  • Accuracy – Hits, misses, noise
  • Ranking
  • Linguistics
  • Query processing
  • Results processing
  • Display
  • Search refinement
  • Usability
  • Business rules

As you can see from this list, there are a whole lot of different ways to do it. Relevance has become extraordinarily popular, so I put that at the top, but there are lots of other ways to measure it. One is recall:  Did you get everything in the database having to do with your question? And precision:  So, did you get everything but not a lot of junk as well? If you got everything, you might have gotten a lot of junk that was not really responsive or appropriate to your question. Taxonomies help an extraordinary amount with recall and precision. They don’t help much with relevance.

Another way to measure accuracy is with statistics on hits, misses, and noise. That is when you look at the records retrieved – and you can use this for the indexing accuracy, as well. (Indexing and search go hand in hand and have done so since the first computer databases for text were developed in the mid-1960s.) A hit is something that a human thinks is exactly right and the computer also thought was right. A miss is something that the human would have suggested but the computer did not suggest. Noise is something the computer suggested, about which the human says, “That isn’t really right for my query.” So, hit, miss and noise statistics are another way to measure accuracy.

Search results are often presented in rank order – those that are most appropriate are first, and those that match fewer and fewer parts of the query are closer to the bottom.

Linguistic analysis or general linguistic applications using natural language processing are frequently applied in search and, there again, we measure them either by recall and precision or hit, miss, and noise statistics. 

Query Processing

In measuring accuracy in search or measuring how well search works, we also talk about query processing. In particular, we are interested in how fast the results are returned. So, there are two parts to that. One part is the query. I’ve asked a query; now, how fast is it going to come back to me with an answer? That often depends on how much of that query is going to be held in cache memory and how much is going to be accessed through a hard drive of some kind.

The second part is the results processing. We are looking at the results that are coming to the user. How fast can I see my results? How fast are they returned to the user? A query is where we are getting the pieces of the answer, but once the query has given me those pieces, I need to process those into a result – into the answer that you see. I got 55 hits and I want to see them. I want to click on something and get the actual document presented to me. Well, that is the display processing. Most systems do not assemble the entire record on the fly, or as you ask for it, from a lot of different pieces. They have what is called a display server, and they will show you the results from that as you indicate your approval by clicking the URL or the path that will take you to the full document. So, it pops up as a full document in all its original glory. The easy way is just to store the full document somewhere out there and let you go retrieve it. In a system like Google, obviously you are going to the original web page or whatever is referenced in the URL link.

There are lots of refinements on that. How long does it take you to narrow down your search?  Can you do a search within a search, also known as a recursive search? You have a general set. Can you just keep that set and throw out the junk and narrow in to what is precisely what you want, or do you have to start the search over with more words so that you can get an answer?

Usability is another way that we measure search results to see how good the search system is. This is when we say – “This was really easy to use.”  “It was very user-friendly.”  “I like the user interface.” All of those user experience kinds of questions are, of course, big in the end user’s mind. They may not be particularly important to the IT people because they are looking at the things we have already talked about. When you get down to the customer service and user experience interface, then you want to know how usable the system is.

Finally, there’s a whole set of business rules. How is the security of the system? Am I able to limit it so that people can search only in a couple of ways?

That takes us to relevance and we’ll pick up there next week.

Marjorie M.K. Hlava
President, Access Innovations