On Design of Problem Token Questions in Quality of Experience Surveys
  User surveys for Quality of Experience (QoE) are a critical source of
information. In addition to the common "star rating" used to estimate Mean
Opinion Score (MOS), more detailed survey questions (problem tokens) about
specific areas provide valuable insight into the factors impacting QoE. This
paper explores two aspects of the problem token questionnaire design. First, we
study the bias introduced by fixed question order, and second, we study the
challenge of selecting a subset of questions to keep the token set small. Based
on 900,000 calls gathered using a randomized controlled experiment from a live
system, we find that the order bias can be significantly reduced by randomizing
the display order of tokens. The difference in response rate varies based on
token position and display design. It is worth noting that the users respond to
the randomized-order variant at levels that are comparable to the fixed-order
variant. The effective selection of a subset of token questions is achieved by
extracting tokens that provide the highest information gain over user ratings.
This selection is known to be in the class of NP-hard problems. We apply a
well-known greedy submodular maximization method on our dataset to capture 94%
of the information using just 30% of the questions.
