4

Let me introduce the context: I need to evaluate how my TAs tend to grade students.

Ideally I would like to pinpoint whether they show some sort of internal consistency (e.g., a TA usually grades almost always above/below a reference grade, or is inconsistent and the grades fluctuate). I would like also to identify a way to assess the goodness of each TA grading w.r.t. to a reference grade I would give or w.r.t. the others TAs grades.

For the first part of my question I was thinking about just variance analysis, whereas the second half I'm not sure.

I'm not really well versed in statistics and data analysis, and I could not manage to identify the right keywords to answer these doubts on my own, so here I am asking for help.

As a side note: grades are usually a single value in a well defined range, such as "78 out of 100" or (less likely) an interval within a range, such as "50-55 out of 100"

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Alexis
  • 41
  • 2
  • 1
    Approximately how many assignments do TA's grade? The sample size is important for deciding what test is appropriate – Hugh Feb 24 '17 at 19:36
  • I would say 25/30 instances of the same assignment each (4 TAs, classes of about 100 students). – Alexis Feb 24 '17 at 19:39
  • If either of the provided answers answer your question to your satisfaction, you can accept it by clicking the check mark under the voting arrows. – Kodiologist Mar 06 '17 at 17:44

2 Answers2

1

Ideally I would like to pinpoint whether they show some sort of internal consistency (e.g., a TA usually grades almost always above/below a reference grade, or is inconsistent and the grades fluctuate).

Plot it. For example, you could make a dot plot or density plot of all a given TA's grades on a given assignment.

I would like also to identify a way to assess the goodness of each TA grading w.r.t. to a reference grade I would give or w.r.t. the others TAs grades.

A simple way to compare a TA's grade to a reference grade is by computing the absolute difference. And you could compare the TA's grade to the grades of other TAs for the same assignment and student by computing the absolute difference from the mean grade.

I'm not really well versed in statistics and data analysis

In that case, I would urge you not to make any important decisions on the basis of these computations, such as deciding whether to fire a TA. Education, government, and business are overrun with supervisors who have a weak understanding of psychometrics judging workers using hamfisted applications of numbers.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68
  • Thank you for your response. I just wanted to reassure you that it is primarily in the interest of the students (I want to be 'sure' they are graded in a fair manner) and to provide some sort of feedback to my TAs, that sometime lack confidence in their grading skills. – Alexis Feb 24 '17 at 21:14
  • @Alexis Sounds good. – Kodiologist Feb 24 '17 at 22:29
0

You need to make an experiment where the same student papers is given to two different TAs. Then you can compare results, maybe with kappa or Krippendorffs alpha (which are coefficients of agreement). Also make plots, a good plot will be of differences in grade (between the two TAs) against the average. Such a plot is often called Tukey mean-difference plot or a Bland-Altman plot. See Agreement between methods with multiple observations per individual for some examples, or search this site.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467