User Feedback on Search Relevance
When writing the retrieval & ranking code that’s at the heart of any search engine — code that answers the question what results should be returned for this search, and what order should they be in? — engineers have a plethora of algorithms to choose from. And given the critical importance of getting relevance right, one might reasonably wonder whether, by now, the search industry has settled on a standard automated measurement of relevancy. (That would certainly help us choose the right search algorithms!) This is not the case. While some work has been done to analyze server logs to report proxy measures such as click-through rates and page dwell time, automated tools have yet to replace human raters. This is largely because relevancy is highly subjective.
Many local search engine firms have specialized in-house teams whose job it is to judge the relevance of the results produced by their engine in response to real-world queries from users. But this raises some questions: Are the scores from a group of ‘professional judges’ applicable to the general population? Without local knowledge, can the testers determine whether an ambiguously named business is properly categorized or whether the most popular restaurant in a neighborhood is ranking appropriately? Are the testers aware of colloquial differences in names for businesses or products? Do the testers understand local preferences around how far people are willing to travel for a particular type of business?
One way to address such concerns is simply by asking for direct feedback from real users. So today, yp.com is launching a tool to gather relevancy feedback on individual search results. Next to any business listing show on a search results page on yp.com, you’ll notice a pair of Yes/No buttons under the words “Listing relevant?” We hope you’ll take the time to click a few of them — especially when we return the wrong stuff, but also on those occasions when we manage to get it right, too!
