6

To help me focus on which pages on a site to improve, I'm looking at user feedback to the question "Was this page helpful?" (Answers are "Yes" or "No".)

The response rate (responses divided by unique pageviews) varies across different pages. And in fact the pages with low response rate also tend to have a higher ratio of negative ratings. This suggests to me that many users who don't find a page helpful just leave without rating it. (I have looked at other factors that could have affected response rate but the correlations are very weak.)

I understand that it's problematic to generalize the result for pages that have a low response rate. But I'm reluctant to exclude these pages, since it seems that they may be the ones most in need of attention. In view of this, is it reasonable to ignore response rate per page and simply look at the positive to negative ratings ratio? Or is there a way I could account for response rate and still include these pages?

Joe Pairman
  • 205
  • 1
  • 4
  • What is your mental model of decisions made by people visiting your site? Is there any difference between first-timers and people using your site regularly? Will regular visitors think of spending a second answering the question? Do you have any extra variables from analytics (G$$gle's or others')? Non-response may be correlated with those variables... – Deer Hunter Dec 22 '12 at 18:27
  • Thanks for the tips, Deer Hunter. Some interesting avenues to explore there. Currently I just have the general analytics, not directly linked to responses and response rates. For example, I can look at things like average time on page for the pages that got lower response rates (not much correlation for that one), but I don't have the specific time on page numbers for only those visitors who didn't respond. – Joe Pairman Dec 23 '12 at 03:32
  • 1
    You could also look into experimental design with such pages: TRy to solve problems with some, better others, wait and see how the response changes ... – kjetil b halvorsen Oct 20 '20 at 00:24

2 Answers2

7

In answer to your question, you should take response rate into account as it is giving you extra information.

The point is that if there are pages with low response rates and low ratings, then it is possible that many people were looking for different information, rather than judging the quality of the page, and that some of those who did rate were say that the information did not help their particular situation, rather than being low quality. You need to investigate what question they actually wanted to ask which led them to that particular page to see what might be missing.

Henry
  • 30,848
  • 1
  • 63
  • 107
  • 1
    Henry, that's very helpful, thanks. Yes, it does seem that some users rate a page negatively (and others just don't respond) simply because the page wasn't what they were looking for, rather than it being poor quality in itself. Other data such as search keywords bears this out. So not only should I not exclude these pages; I should pay extra attention to them and why people might be erroneously landing on them. – Joe Pairman Dec 22 '12 at 14:13
3

Requests of this type often get answers from two groups: The extremely pleased and extremely displeased, while people who are sort of generally satisfied don't bother. Although I don't know of research into page ratings specifically, I've seen this with response cards in other situations (e.g. rating of hotel service).

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • Peter, that's interesting. In that case we may be delighting a lot of people! Though I think it's easier to click a single button on a web page compared to filling in a response card which takes a bit more commitment and may filter out those less determined to express their views. What would concern me a little is if the response rates are biased towards certain pages because of this. However, my main goal at the moment is to identify the pages which are most in need of attention, so hopefully this is still covered even if some generally satisfied users aren't responding. – Joe Pairman Dec 22 '12 at 14:30