I'm providing the user with the option to rate news articles 1-5 stars. I'm trying to determine the quality and interest of articles based on the user ratings. My first thought was:

  • Interest = total number of ratings
  • Quality = average rating

I.e. a user rates articles that it is interested in and gives higher rating for quality articles. The problem is that users may give an article a positive rating as a sign of interest or a negative as a sign of disinterest - regardless of the quality of the article. So positive/negative ratings may indicate interest and/or quality, or it may not depending on the user's intention. The metrics overlap and it gets confusing.

One solution to this is to provide the user with two ways to rate the content:

"Does the subject of this article interest you?" (YES/NO)
"What do you think of the quality of this article?" 1-5 stars

It makes things more clear in terms of how I can use the data, but the user may be confused by it. Do you think this is a good idea or not? Is there a better way to determine quality and interest of content?

  • What is your audience, number of users? – Austin French Jul 23 '13 at 13:58
  • Its in development but its target audience is basically anyone world-wide looking for localized news aggregated by quality and popularity. – Pking Jul 23 '13 at 14:20

TL;DR version: Don't do it. A simple rating works and gives you some limited data of good quality. If you want more data, there are approaches which work, but are vastly more complex than adding a second question. Homebrew methods like a second question will give you a large amount of bad quality data which looks misleadingly like the real thing, and besides they will make for a worse user experience.

Measuring user satisfaction and related concepts is a very complex topic, despite the fact that it looks superficially easy ("Just ask them").

A five-star-rating does not measure product quality. It measures the valence and strength of the consumer's feelings for the product. This is an important difference. The first is an objective characteristic of the product which must be assessed with cognitive functions. The second is part of the affective domain of the consumer. (If you are unclear on the differences between cognition, affect and conation, the Wikipedia articles provide a decent starting point).

Measuring affect by self-reporting is easy. Humans know their own emotions, at least at the basic level of saying "I like this" or "I dislike this". When you ask them to give you a few stars, they can do it very accurately. Also, their own effort for this is rather low. They just report something they already know. In the worst case your scale is not fine enough for their feeling and they will spend some time wondering whether to give 3 or 4 stars, because their evaluation falls between them and neither feels right.

Requiring people to make an analytical assessment of something is hard on them. They dislike doing it, because it is work. There comes the first argument against doing it - the user experience will degrade significantly. But what is probably even worse, they make mistakes. The harder the task you give them, the more likely they are to make a mistake. And the task you are trying to give them here is really hard. What they have immediate access to is their own feeling. What you are asking them to provide are the drivers of this feeling, the reasons why they "decided" to like or dislike a product. And people are really lousy at reasoning about their own affective states. When you try to force them to do it anyway, they report wrong results, without being aware of being wrong. Any conclusions you make from data collected this way are likely to be wrong as well.

There are many examples of people not being able to explain their feelings. Interesting ones include the phenomenon of affective prediction (the research is mostly driven by Gilbert IIRC, but the Wikipedia article covers the most important points) and the feeling of certainty — see Burton, Robert, On being certain: Believing you are right even when you're not, (Macmillan, 2009) for a very nice pop-science text. Another one, closer to your case, happens right here on the Stack exchange network. The difference between downvoting and casting a "very low quality" flag is well documented and explained in the UI, but users keep confusing them. What I think is happening is that people are using both as an expression of their affect, instead of reflecting on the reasons for disliking an answer and downvoting for "not useful" and flagging for "so hard to understand that I cannot even extract a meaning out of it".

This is what happened to marketing researchers in the 70s too. They did exactly what you are trying to do here, creating the concept of importance performance analysis (Reference publication: Martilla, John A., and John C. James, "Importance-performance analysis." The Journal of Marketing (1977): 77-79.). There, consumers are asked to rate a product on how well it performs in quality in a given dimension (which probably corresponds to what you want to know with your quality question, only you are not distinguishing between dimensions), and how important this dimension is to them (which bears some resemblance to your interest question, even though you are asking it on a more abstract level). After decades of research, there have been studies trying to evaluate the usefulness of importance performance analysis as a tool, and they found some significant weaknesses. The worst one was that the scales were not independent - people tended to rate everything either high or low on both scales. This is a sign that they don't make a clear distinction between the two concepts when answering the questions. A comprehensive list of the weaknesses of the importance performance analysis can be found in the book Satisfaction (Oliver, Richard L., Satisfaction: A behavioral perspective on the consumer, ME Sharpe, 2010.).

The results of decades of research in marketing and psychology are clear. Using a simple approach, you can gain some limited information (solving the simple problem of learning what your users like), which can nevertheless be very useful for you. As you see, many companies are using it with good results. If you really need the deeper information (solving the hard problem of learning why your users like something), you need to use the proper approach, which is highly complex. It would involve a standalone marketing study, which should be performed by itself and not integrated in the product rating tools of a website (because it requires some experience on the part of the person performing it and lots of attention from the participants). But there is no simple approach which is capable of solving the hard problem. If you manage to find one, you will probably get the marketing analogue of a Nobel price.

  • "users will continue to confuse interest and quality, even when you ask them to rate them independently" damn users always making things difficult :) You have me half-convinced, but I'm pretty stubborn - if only there was a way to get users accurately distinguish between interest/quality. You said there where complicated approaches that do work, got any examples? – Pking Jul 23 '13 at 14:54
  • 1
    @Pking please read the new version, it is very different from the old one and hopefully explains it better. As for better approaches for measuring satisfaction, the best explanation of the theory I know of is the Oliver book I cite. I have not looked into practical instruments too closely, because the theory convinced me to use the simple approach for my case. – Rumi P. Jul 23 '13 at 15:25
  • The problem is very clear now, thank you. I'm still curious about maybe turning difficulty and "friction" in trying to get a better assessment from the user can be turned into a positive experience - I'm looking into gamification concepts, where difficulty, conflict, failure, challenge etc. play an important role. Perhaps there are way to encouraging the user to better reason about their affective state through gamification concepts. – Pking Jul 23 '13 at 15:46

If a user is on the news feed page and sees uninteresting article, he should be in position to hide this article (by clicking a cross for example), like you can do for advertisements on a lot of website today. It's like answer

Does the subject of this article interest you ? NO

For a better content, think also about tags (culture, technology, politic, etc.) that already filter the news for a better personal user experience.

In order to get more specific rating you can offer a different (but intuitive) rating system. Look at buzzfeed.com, they don't have this classical 5 stars rating or a simple thump up / thump down system but users can choose between different reactions (and also like/dislike the post).

More generally, users don't want to spend more than 2 seconds to rate a simple article, especially if they read 10 or more news each days. So you should let them hide what they dislike and don't ask too much from them.

  • They perhaps want to spend more time if you reward them for their effort - I have ideas about giving the user points for rating content, and providing certain mechanism to ensure the user has "consumed" the content before rating. – Pking Jul 23 '13 at 15:49

Your Answer

