I'm currently setting up a questionnaire for a target group of people. The questionnaire will ask users to rate how relevant a topic is to a given sentence. Since each user might have different amount of familiarity with a given topic, I expect that the answers given by some users with little experience with a specific topic will increase the variance of my data for that topic. Since I have a limited number of users and I'm not going to be using my final results to compare one user to another, I was thinking of prioritizing which topics I asked a user to rate based on a weight. This weight would be determined by how their answers compare with those of other users and how many answers they have already submitted for that topic.
How I was hoping to do this is by using a two sample t-test to determine the probability Pf that there is no difference between their answers for a topic and the groups answers. I would use this probability as one of my weights when determining which topic I should ask them evaluate next? The other weight would be calculated by giving priority to topics they have submitted the least amount of answers to Pn. Given by the formula Pn = 1 - [topic answers]/[total answers]
The weight of a topic Wt would be equal to Wt = [Pn] * [Pf] and the probability P of showing a specific topic to a user would be P = Wt / [Total weights]
The reason why I'm doing this is that I don't think that my users will have a chance to give an answer for every question in my questionnaire and I would want to determine what would be the best way to choosing which questions they answer.
(ie: Not knowing ahead of time that a user is a structural engineer, I would want to find a way to show them more questions about stress and pressure rather than showing them questions about the economic impact of a grain shortage)
If this is not the correct approach, has there been research done as to how to do this properly?
