2

I'm a new guy here. Hopefully, I'm asking this question to right forum.

Problem:

We have data of a group of people (P1, P2, P3). They rank their expertise (1-10, where higher number is better) in a list of components (G1, G2, G3).

    P1  P2  P3
--------------
G1 | 8   4   7
G2 | 7   3   7
G3 | 9   6   5

Also, we have some data regarding work done by each person in each component. Example:

For P1,

     W  WD
----------
G1 | 0   0
G2 | 2   0
G3 | 8   2

where W is total work allotted to user P1, and WD is actual work done. W >= WD >= 0. We have similar data for P2, and P3 users.

Point to note: The user might have some level of expertise regardless of work done in a component. Example: P1 has ranked himself 8/10 even though he has not been given any task in G1 component (W = 0). P1 also has ranked himself 7/10 for G2 even though he has not finished any task in that component (WD = 0).

Now, we want to calculate effective rating of all users relative to the group of users, not self-ranking, considering their work data and self-ranking.

Can anyone suggest some mechanism to achieve this?

Thanks much in advance!

Nayan
  • 129
  • 6
  • 1
    What exactly do you mean by "effective rating"? If it reflects expertise, why would this have anything to do with an amount of work allotted, which has little or nothing to do with expertise? What would it mean to rate a user "relative to the group"? – whuber Dec 11 '10 at 20:32
  • I was with you up to, but not including, the sentence starting "Now, we want to...". Could you try explaining what you want to achieve again? – onestop Dec 11 '10 at 20:54
  • Ok, I'll try to explain. Users think they are expert to some level in every component. They can rate themselves pretty good numbers. But not everyone has actually the same level of knowledge. This is what we have to find out with the help of amount of actual work done data. The ranking of a user should be higher if he has done some work in the component than the one who has not - even though the latter might have rated himself higher. This is a comparison of expertise of users based on their perception of possession of skill and actual work done. Is this clear now? Please write back if not. – Nayan Dec 12 '10 at 04:01
  • I cannot find any information in your question that relates "work" to "rating". As far as I can tell, "work" is something "allotted" to a worker and "work done" could mean--well, anything. Without such a connection it's impossible to justify any mechanism to use the "W" and "WD" values to modify the self-rating data. – whuber Dec 20 '10 at 15:23
  • @Whuber: Though I think I explained it the best I could, I'll try again to define relationship between W and WD. W is work given to a team member, WD is the work done by him. W >= WD >= 0 (in a given component). I'm trying to formulate a method which can take this info in account and use his self-rating to find out his (normalized) rating in the team. – Nayan Dec 21 '10 at 10:14

3 Answers3

3

It sounds like you have multiple sources of information about measurements of interest. The aim is to combine the multiple sources so as to maximise measurement accuracy. In general you could adopt a formula that weights the different sources of information based on the confidence you ascribe to the source.

It also seems that the confidence in the source might vary. E.g., the more work data you have on someone, perhaps the more weight this information would be given for that individual. You may also have evidence on the accuracy of an individual's self ratings.

Thus, a simple strategy would to develop a weighted composite of available information where the size of the weight is relative to the informativeness of the source.

This question on Bayesian rating systems might also be relevant.

Community
  • 1
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
  • I thought of using Bayesian methods, but I'm very new to such techniques and still trying to figure out factors and variables. But your suggestions conforms that Bayesian methods may hold the answer. I'm trying :-/ – Nayan Dec 12 '10 at 18:56
1

I'm going to give a stab at an answer, if I misunderstand your question, please comment here or revise your question and I'll do my best to revise my answer.

If I understand you correctly you are trying to find a users 'true level of expertise'. An elusive quantity to be sure. Given what you have written, I think you expect that true expertise has some relationship with actual amount of work done.

There are some problems meeting the assumption of the technique I am going to suggest first, but I hope the violations will be tolerable. A first step could be setting up a regression equation to predict work completed given work assigned for each component (G1, G2, G3, etc). Residuals from this regression equation are deviations from the normal amount of productivity in that component. Deviations from normal productivity (if you assume those who are more expert get a greater proportion of their work done) may be viewed as a measure of 'true expertise'. It also sets each person's productivity in comparison to others (a stated goal). Of course, I strongly suggest you view a scatter-plot of these data (and compare model fits) as I suspect that the relationship will not be linear and that you'll have have to fit a quadratic.

I'm not sure what you want to do with the self-ratings. Of course, you should eye-ball the self ratings, if someone is giving themselves all high scores, that participant may need to be dropped. Another thing you might consider doing prior to analysis is subtracting each subject's mean rating of expertise from each of their ratings of expertise. Though in this approach you'll lose information about people who are overall more skilled than others. To validate that the self-ratings are meaningful you want to correlate them with work completed. They should be somewhat positive given your assumption that those who complete more work on a given component tend to be (but are not necessarily) more expert. If the self-ratings are shown to be a somewhat valid measure, then you can move on to correlating the self ratings with the above mentioned residual score. Again, the correlation ought to be somewhat positive.

Edit 1: You say you want to calculate the "effective rating" of all users relative to the group of users, not self-ranking, considering their work data and self-ranking. I'm not sure how you can both consider self-ranking and not consider it at the same time. If you mean that you want to create a measure of skill based on something other than self-rankings, but then combine it with the self-ranking data, you can simply sum the Z-score of some measure of skill (e.g. work done or the method I describe above) and the Z-score of self ratings.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
  • About self-rating, they can not be completely ignored. A person might not get much work, but has great knowledge of component will lose the ranks significantly if self-rating is not considered. – Nayan Dec 12 '10 at 19:01
  • @Nayan: I'm afraid you're in a corner then. You can't exclude the bad variance from the self-rating and keep just the good variance. In absence of actual performance information you have no source of data other than their self-rating and you say you don't trust the self-rating. The method I outline above doesn't penalize those who don't get much work, but it does somewhat penalize those who get very low amounts of work or no work at all. One thing you might do is just acknowledge you don't have sufficient evidence to judge some of the people. – russellpierce Dec 12 '10 at 20:33
  • @drknexus: Hmm.. you are right. Still we can experiment with the different models until we reach the one which suits best. Frankly, since I'm not into statistics and maths, I find it bit hard to formulate your idea. Sorry, but can you help by showing the formula or pointing to some system that I can study? Thx! – Nayan Dec 13 '10 at 03:18
  • @Nayan: I need to determine where we can start in terms of common tools/knowledge. Do you use SPSS or R? Do you know how to do regression? Do you know how to calculate a Z-score? – russellpierce Dec 13 '10 at 05:22
  • I'm so afraid that I do not know much. But I'm willing to learn. Any guidance will be highly appreciated. I've just started studying R. – Nayan Dec 14 '10 at 02:52
  • To Z score a variable use the scale function. If you scale G1 G2 etc... you'll get a sense of how people rate their expertise compared to others. If you also scale WD then both stated expertise and work done will be put on the same scale and you can simply sum the scores to get some combined metric. – russellpierce Dec 15 '10 at 16:11
  • My approach is something like: scale(resid(lm(WD.G1 ~ W.G1)))+scale(G1). You'll (of course) have to restructure the data some to make that all work. Good luck. If you want clarification on any of these techniques, consider asking a new question. – russellpierce Dec 15 '10 at 16:13
  • Thanks for the comments! I am studying it. But please also have a look at my answer! – Nayan Dec 20 '10 at 07:27
0

I worked on it and thought this Bayesian equation will be useful.

RatingTM = SRTM/(1 + AWD/WDTM) + MSRT/(1 + WDTM/AWD)

The variables are:
SR = Team member's self rating
AWD = Average work done by team
WD = Work done by team member
MSRT = Mean self rating of team

(TM: TeamMember)

Please comment if you think this is not right. Thanks!

Nayan
  • 129
  • 6
  • 1
    I cannot tell whether it's right because I find it meaningless: although you call this formula "Bayesian" you haven't provided a probability model, you haven't defined a "team", and you haven't (yet) formulated a clear question. – whuber Dec 20 '10 at 15:24
  • Then can you suggest how to clearly declare such things properly and what is missing in my question which is not clear? Thx! – Nayan Dec 21 '10 at 10:11
  • Yes: please provide a probability model and define terms like "team" and "effective rating." If we have to guess at the meanings our chances of answering the intended question go down. (Normally it's too much to ask for an explicit probability model, but since you've already done a probability calculation you must have one!) – whuber Dec 21 '10 at 18:31