2

I recently got into a debate with a friend of mine because I claimed that the 9th circuit court tends to have its rulings overturned at a greater rate than other courts (this is about statistics trust me!)

Looking at the data, from face value he rejected my claim because "the ninth circuit is only reversed at 10% higher than the rest of the courts".

To keep it short, things got personal and my intelligence was insulted. He asked to see a t-test and ANOVA w/ post hoc tests (even though we both acknowledge its not best suited here) in order for me to make my case.

In my pain I wrote it all up in a document (I wasn't able to attach a file here) but I wanted to share it with all of you for some opinions on the subject.

We also both agreed that "Reversal_Rate" (# of reversals / # of hearings) is the variable of interest here, not the quantity of reversals (since courts with higher number of hearings will have higher reversals by design).

The Problem:

1) So my question is, what are some tests I can use to see if ninth court's Reversal Rate is higher than the rest? In other words, what kind of statistical test is best suited to test proportional data such as Reversal_rt?

2) If someone would be so kind, can you check out my results so far (especially in the second half) and let me know their thoughts?

My results: https://imgur.com/a/vy3lhz9

Data Source: https://ballotpedia.org/SCOTUS_case_reversal_rates_(2007_-_Present)?fbclid=IwAR1rOpTKLg4JRWvZ1Je95Ma0PBURaz5iqJvBu9HpuR3ckjR7WuX2IaB0qu8

TAEHSAEN
  • 21
  • 1
  • 2
    In order to do a proper test you'd need the _numbers_ of decisions and of reversals for the 9th circuit court and also numbers of decisions and reversals for comparison court(s). _Percentages alone_ are almost useless for testing: 10 reversals for 50 decisions and 100 reversals for 500 decisions carry very different amounts of information. // Answer of @jros is a good place to start. – BruceET May 14 '20 at 07:06
  • Is independence of the rulings a reasonable assumption here? – jaradniemi May 21 '20 at 21:19

1 Answers1

3

the simplest method for finding evidence of a difference in proportions would be a Chi-Square test of independence. To compare a difference in proportions between two groups, construct a confidence interval in this manner:
$\hat{p_1}$ = proportion for group 1, $n_1$ is sample size of group 1
$\hat{p_2}$ = proportion for group 2, $n_2$ is sample size of group 2
$z^*$ is our critical value, ie for a 95% Confidence Interval, $z^*$ = 1.96

$(\hat{p_1}-\hat{p_2}) \pm z_{1-\frac{\alpha}{2}}^*\sqrt{\frac{\hat{p_1}*(1-\hat{p_1})}{n_1}+\frac{\hat{p_2}*(1-\hat{p_2)}}{n_2}}$

jros
  • 565
  • 2
  • 15
  • How would one set that up on a statistical software like Stata? I hesitated trying it because I was worried about the difference in sample size. For instance consider the following: n1: 9th circuit court (20 reversals; 5 affirmations) n2: all other courts (100 reversals 9 affirmations) If I group all other courts into n2, will the difference in sample size matter? Also I would assume that there is no way to do this test using actual proportions (Rate of Reversals). Rather I would have to do it by breaking it down by Reversals vs. Affirmations right? – TAEHSAEN May 14 '20 at 16:49
  • Can you expand on what you mean by actual proportions? For the 9th circuit court, your p-hat would be 20/25 = 0.8; if you have a rate for a really large n, you may be able to treat that as a population standard deviation. – jros May 14 '20 at 21:01
  • 1
    Some posts with other/better ways to form CI: https://stats.stackexchange.com/questions/28934/confidence-interval-about-the-difference-between-false-positive-and-false-negati, https://stats.stackexchange.com/questions/400729/two-binomial-distributions-of-widely-different-size-confidence-intervals-p-va, https://stats.stackexchange.com/questions/513390/finding-a-confidence-interval-for-difference-of-proportions – kjetil b halvorsen Nov 11 '21 at 00:38