CrowDEA: Multi-view Idea Prioritization with Crowds
  Given a set of ideas collected from crowds with regard to an open-ended
question, how can we organize and prioritize them in order to determine the
preferred ones based on preference comparisons by crowd evaluators? As there
are diverse latent criteria for the value of an idea, multiple ideas can be
considered as "the best". In addition, evaluators can have different preference
criteria, and their comparison results often disagree.
  In this paper, we propose an analysis method for obtaining a subset of ideas,
which we call frontier ideas, that are the best in terms of at least one latent
evaluation criterion. We propose an approach, called CrowDEA, which estimates
the embeddings of the ideas in the multiple-criteria preference space, the best
viewpoint for each idea, and preference criterion for each evaluator, to obtain
a set of frontier ideas. Experimental results using real datasets containing
numerous ideas or designs demonstrate that the proposed approach can
effectively prioritize ideas from multiple viewpoints, thereby detecting
frontier ideas. The embeddings of ideas learned by the proposed approach
provide a visualization that facilitates observation of the frontier ideas. In
addition, the proposed approach prioritizes ideas from a wider variety of
viewpoints, whereas the baselines tend to use to the same viewpoints; it can
also handle various viewpoints and prioritize ideas in situations where only a
limited number of evaluators or labels are available.
