6

In signal detection theory, people often use $d'$ to assess performance. Apart from the fact that $d'$ is in $z$ units (units of measurement transformed to standard deviation units, i.e., $z$ scores), making it comparable regardless of the original units of measurement. I can't see what the advantage in analysing $d'$ instead of proportion correct is.

Don't both account for bias? Would they follow the same shaped ROC curves?

amoeba
  • 93,463
  • 28
  • 275
  • 317
user41270
  • 215
  • 1
  • 3
  • 8
  • 4
    Could you please clarify 'follow the same shaped ROC curves'? – Trisoloriansunscreen Mar 09 '14 at 11:40
  • 2
    There is precisely no context here. Also, it is rare that notation like this is constant across disciplines, so explaining what the notation means would increase the possible value of this question to others. – Nick Cox Mar 09 '14 at 11:41

1 Answers1

6

D' is a measure of sensitivity, whereas proportion correct is affected by both sensitivity and bias.

In the special case of two balanced classes (same number of signal and noise trials) and zero bias, D' is monotonically mapped to proportion correct, offering no additional insight. However, if the two classes are not completely balanced or the bias isn't zero, the two measures can considerably depart. Consider these two examples:

  1. A dataset with 70% signal trials and 30% noise trials. An observer / classifier always responding 'signal' will have 0.7 proportion correct but zero D'.

  2. A dataset with balanced classes and a classifier with D'=1. A zero bias would produce a maximal proportion correct and any increase / decrease in the bias are excepted to decrease the proportion correct (think of the case of extreme biases).

Trisoloriansunscreen
  • 1,669
  • 12
  • 25
  • That's really helpful. I now realise that D prime is necessary where there are uneven amounts of signal and noise trials....previously I was only thinking about equal signal and noise trial situations. e.g. with 50% signal and 50% noise, an observer always responding signal will have .5 correct and .5 incorrect. proportion correct = .5 and d=zero. But if there are 70% signal trials and 30% noise trials, you start to get advantages by increasing your willingness to response 'signal'. proportion correct = .7 But d' is still comes out at zero: d = zhits - zFA = 0.524 - 0.524 = 0 – user41270 Mar 09 '14 at 12:18
  • i guess in this situation for an observer to increase their d' they would need to start responding above 70% correct. – user41270 Mar 09 '14 at 12:24
  • consider also the second example - even amounts of signal and noise trials. If there's zero sensitivity, bias won't change the proportion correct, but if sensitivity>0, bias does have an effect. – Trisoloriansunscreen Mar 09 '14 at 12:46
  • Regarding the 70/30 example, you can have D'>0 and proportion correct<0.7 if there's some sensitivity but the bias is suboptimal. – Trisoloriansunscreen Mar 09 '14 at 12:48
  • Thanks again. Re your first comment (equal sample sizes), i think understand the first part of your comment - when both noise and signal distributions overlap entirely, moving the criterion to the right would result in fewer hits but fewer FA. and moving it to the left more hits but more FA. You say that 'if D>0, bias does have an effect'....is this to do with the likihood ratio changing: LR(x)= f(x|noise)/f(x|signal).. pg 33, Macmillan and Creelman. Sorry i'm very new to this and am struggling to understand the underlying relationships – user41270 Mar 09 '14 at 14:30
  • Beware: Any statistic based on dichotomizing class membership probabilities is an improper scoring rule, i.e., is an accuracy measure that is optimized by selecting the wrong features or fitting a bogus model. It is far better to use proper scoring rules (e.g., Brier score or logarithmic probability accuracy score such as log-likelihood for a binary logistic model). – Frank Harrell Mar 09 '14 at 14:55
  • @user41270: assume that you have a classifier that can tell signal from noise, but not with perfect accuracy and you have equal amount of signal and noise trials. What bias will result with the best accuracy? – Trisoloriansunscreen Mar 09 '14 at 15:04
  • I'm imagining the noise and signal distributions overlapping with seperation in means (d prime >0). I would guess that best accuracy is to position criterion where the two intersect? Assuming payoff for hits and misses equal. Does this optimal position have anything to do with the liklihood ratio i mentioned above? Or is this something else? – user41270 Mar 09 '14 at 17:55
  • actually i think that the best place for criterion would be at the mean of the noise distribution - after this point false alarms begin to decrease quickly and but hits are rising (until the mean of the signal distribution is reached). – user41270 Mar 09 '14 at 18:08
  • I think that the optimal criterion is the one that will produce 50% 'signal' classification, matching the real distribution of signal and noise trials. Anyway, my point is that proportion correct and D' measure different things even when the occurances of signal and noise are equal. – Trisoloriansunscreen Mar 10 '14 at 09:43
  • I believe the example you are giving is wrong: "A dataset with 70% signal trials and 30% noise trials. An observer / classifier always responding 'signal' will have 0.7 proportion correct but zero D' ". This is not true. D' = z(0.7)-z(0.3) = 0.52-(-0.52) = 1.04 ! Lambda bias is 0.52; PC = 70% D' are both lambda bias is zero when it's a 50/50 split and the observer is saying signal present all the time. PC = 50% – Mecasickle Nov 21 '16 at 00:16
  • 1
    @Mecasickle, thank you for your attention to this answer! In my 70/30 example, Hit rate would be 1 and false alarm rate would be 1, hence d'=0. False alarm rate is p(responding 'signal'|no signal). – Trisoloriansunscreen Nov 21 '16 at 07:19
  • Oops! My mistake! Thank you for letting me know of this terrible blunder! – Mecasickle Nov 21 '16 at 17:27