Thanks so much for the question.
One of the great things about working with quantitative numbers is that they allow you to summarize a complex problem in a very concise way. While search is a qualitative experience, the methods I describe here explain it in clear, simple numbers.
So for example, after completing the evaluation you can present the results like this:
- Our mean relevancy score is currently 5.7, and we want to bring that up to 2.5.
- Currently, 11% of the best matches fall below the 10th position. We want to reduce that to 5%.
- By the loose standard, our precision score is currently 63%. We want to bring that up to 75%.
- By the permissive standard, our precision score is currently 89%. We want to bring that up to 98%.
These metrics create a compelling case for further work to improve the quality of the search experience, and suggest the type of work that needs to be done. For example, solutions like engine tuning, thesaurus, and spellcheck improve the quality of all searches, while optimization and bets bets fix stubborn outliers that remain problematic.
In the past, I’ve used these methods to set objectives and create improvement plans in just this way. Not only has it been effective, but it’s often significantly overshot the improvement target resulting in a screamingly great search experience.