How do we evaluate Search?

Three approaches that drive optimizations of search quality:

• Interleaving or A/B testing
• Search quality rating
• User Research

Interleaving is a popular and important technique for evaluating information retrieval systems based on implicit user feedback, we could look at metrics like click and duration, etc.
To evaluate whether our systems do, in fact, provide information that people searching find relevant and reliable as intended, we solicit feedback on proposed improvements from Search Quality Raters who use our guidelines to review the changes. These search quality raters collectively perform millions of sample searches and rate the quality of the results according to the signals we previously established.

Query = 沈安成飞行员

How do we monitor Search Quality?

• Online experience quality monitor:
  • Daily quality rating. (sample searches 40 in-app + 40 from competitor)
  • User Behaviors: clicked query ratio A/A (clicked queries/ all queries)
• Improvements evaluation process：
  • Quality rating: 200-400 sample queries
  • Live experiments: analyze metircs

Why search relies on quality rating?

• Feasibility：Search is with intention. Humans could tell how well the results satisfy the query (If it provides information that people searching find relevant and reliable as intended)
• Advantages: Interpretable. Data from quality rating is interpretable. We could identify the improvements and problems through case review.
• Disadvantages: low bias, high variance.