Correct statistical test when people could appear in multiple groups

Question

Thank you in advance for your help.

I ran a survey. People first answered a multiple selection question - they could select as many of the choices as they wanted. The question asked them their purpose for visiting a website (for example, purchase an item, read reviews on an item, compare prices, browse for items, etc).

They were then asked to numerically rate, between 1 and 7, how easy it was to achieve their goal(s).

What I want to be able to do is establish whether certain goals are statistically easier / harder on this website. If the purposes were mutually exclusive, I would be confident. the problem is that people may have selected multiple goals. So I have $x$ people with purpose 1, $y$ with purpose 2, but I also have z with purpose 1 and 2, etc.

What I want to know is if there is a way that considers not only those in one category or another, but also those that may be in multiple categories.

Any feedback would be greatly appreciated.

Answer 1

I presume that when users were asked to rate the ease of website use, it was their overall impression, and ease was asked only once per user rather than specifically for each task. Therefore, if I said I used the website to update my profile & post a blog and I rated my ease of use 1 (presumably difficult), you would not know if updating the profile or posting the blog was difficult, or both.

I typically conduct two analyses for data of these nature. First off, I define an outer-product-variable which is a unique identifier corresponding to each possible combination of task or response. So if there were 3 possible tasks which users might endorse, I would have $2^3=8$ possible combination levels: those who endorse no tasks, the first task only, the second task only, the third task only, the first and second tasks, the first and third tasks, and all tasks. With 8 levels, we can inspect average difficulty ratings and error bar charts. If this leads to many possible levels, it can be useful to sort such an error bar based on a meaningful metric which combines reported difficulty as well as representation. For instance, you might consider response categories that have at least 5 or 10 respondents and sort them from most problematic to least problematic to see which particular tasks have been the worst for users.

Second: A linear regression model gives a way of exploring a similar issue. Treating ease as an outcome, 0/1 indicator variables can be used for each possible task to be used as covariates in the model. In addition, product terms can be created between two or more specific tasks. This leads to a very high dimensional model and techniques of model selection can be applied to identify heterogeneity. Doing this in two passes, first for only the main effects and second for the product terms, gives many useful hypotheses for process improvement.