User Model-Based Intent-Aware Metrics for Multilingual Search Evaluation
  Despite the growing importance of multilingual aspect of web search, no
appropriate offline metrics to evaluate its quality are proposed so far. At the
same time, personal language preferences can be regarded as intents of a query.
This approach translates the multilingual search problem into a particular task
of search diversification. Furthermore, the standard intent-aware approach
could be adopted to build a diversified metric for multilingual search on the
basis of a classical IR metric such as ERR. The intent-aware approach estimates
user satisfaction under a user behavior model. We show however that the
underlying user behavior models is not realistic in the multilingual case, and
the produced intent-aware metric do not appropriately estimate the user
satisfaction. We develop a novel approach to build intent-aware user behavior
models, which overcome these limitations and convert to quality metrics that
better correlate with standard online metrics of user satisfaction.
