Palantir's Asian discrimination case: how were the probabilities computed?

Question

I read this article about Palantir's case where Deparment of Labor is accusing them in discrimination against Asians. Does anyone know where did they get these probability estimates from?

I'm not getting 1/741 in item (a).

(a) For the QA Engineer position, from a pool of more than 730 qualified applicants—approximately 77% of whom were Asian—Palantir hired six non-Asian applicants and only one Asian applicant. The adverse impact calculated by OFCCP exceeds three standard deviations. The likelihood that this result occurred according to chance is approximately one in 741.

(b) For the Software Engineer position, from a pool of more than 1,160 qualified applicants—approximately 85% of whom were Asian—Palantir hired 14 non-Asian applicants and only 11 Asian applicants. The adverse impact calculated by OFCCP exceeds five standard deviations. The likelihood that this result occurred according to chance is approximately one in 3.4 million.

(c) For the QA Engineer Intern position, from a pool of more than 130 qualified applicants—approximately 73% of whom were Asian—Palantir hired 17 non-Asian applicants and only four Asian applicants. The adverse impact calculated by OFCCP exceeds six standard deviations. The likelihood that this result occurred according to chance is approximately one in a billion.

Answer 1

I am going to reverse-engineer this from experience with discrimination cases. I can definitely establish where the values of "one in 741," etc, came from. However, so much information was lost in translation that the rest of my reconstruction relies on having seen how people do statistics in courtroom settings. I can only guess at some of the details.

---

Since the time anti-discrimination laws were passed in the 1960's (Title VI), the courts in the United States have learned to look at p-values and compare them to thresholds of $0.05$ and $0.01$. They have also learned to look at standardized effects, typically referred to as "standard deviations," and compare them to a threshold of "two to three standard deviations." In order to establish a prima facie case for a discrimination suit, plaintiffs typically attempt a statistical calculation showing a "disparate impact" that exceeds these thresholds. If such a calculation cannot be supported, the case usually cannot advance.

Statistical experts for plaintiffs often attempt to phrase their results in these familiar terms. Some of the experts conduct a statistical test in which the null hypothesis expresses "no adverse impact," assuming employment decisions were purely random and ungoverned by any other characteristics of the employees. (Whether it is a one-tailed or two-tailed alternative may depend on the expert and the circumstances.) They then convert the p-value of this test into a number of "standard deviations" by referring it to the standard Normal distribution--even when the standard Normal is irrelevant to the original test. In this roundabout way they hope to communicate their conclusions clearly to the judge.

The favored test for data that can be summarized in contingency tables is Fisher's Exact Test. The occurrence of "Exact" in its name is particularly pleasing to the plaintiffs, because it connotes a statistical determination that has been made without error (whatever that might be!).

Here, then, is my (speculative reconstruction) of the Department of Labor's calculations.

1. They ran Fisher's Exact Test, or something like it (such as a $\chi^2$ test with a p-value determined via randomization). This test assumes a hypergeometric distribution as described in Matthew Gunn's answer. (For the small numbers of people involved in this complaint, the hypergeometric distribution is not well approximated by a Normal distribution.)

2. They converted its p-value to a normal Z score ("number of standard deviations").

3. They rounded the Z score to the nearest integer: "exceeds three standard deviations," "exceeds five standard deviations," and "exceeds six standard deviations." (Because some of these Z-scores rounded the up to more standard deviations, I cannot justify the "exceeds"; all I can do is quote it.)

4. In the complaint these integral Z scores were converted back to p-values! Again the standard Normal distribution was used.

5. These p-values are described (arguably in a misleading way) as "the likelihood that this result occurred according to chance."

To support this speculation, note that the p-values for Fisher's Exact Test in the three instances are approximately $1/1280$, $1/565000$, and $1/58000000$. These are based on assuming pools of $730$, $1160$, and $130$ corresponding to "more than" $730$, $1160$, and $130$, respectively. These numbers have normal Z scores of $-3.16$, $-4.64$, and $-5.52$, respectively, which when rounded are three, five, and six standard deviations, exactly the numbers appearing in the complaint. They correspond to (one-tailed) normal p-values of $1/741$, $1/3500000$, and $1/1000000000$: precisely the values cited in the complaint.

---

Here is some R code used to perform these calculations.

f <- function(total, percent.asian, hired.asian, hired.non.asian) {
  asian <- round(percent.asian/100 * total)
  non.asian <- total-asian
  x <- matrix(c(asian-hired.asian, non.asian-hired.non.asian, hired.asian, hired.non.asian),
              nrow = 2,
              dimnames=list(Race=c("Asian", "non-Asian"),
                            Status=c("Not hired", "Hired")))
  s <- fisher.test(x)
  s$p.value
}
1/pnorm(round(qnorm(f(730, 77, 1, 6))))
1/pnorm(round(qnorm(f(1160, 85, 11, 14))))
1/pnorm(round(qnorm(f(130, 73, 4, 17))))

Answer 2

How to calculate pvals properly using the hypergeometric distribution:

Drawing $k$ successes in $n$ trials without replacement from a set with $K$ successes amid $N$ total items will follow the hypergeometric distribution.

For a one-sided test, in MATLAB, you can call pval = hygecdf(k, N, K, n); or in this case pval = hygecdf(1, 730, 562, 7) which is about .0007839.

Mean and standard deviation are given by:

$$ \mu = n \frac{K}{N} \quad \quad \quad s = \sqrt{n \frac{K}{N} \frac{N - K}{N} \frac{N - n}{N-1}}$$ Thus we're -3.957 standard deviations outside the mean.

I've tried various things to replicate the p-values (eg. hypergeometric cdf, $\chi^2$ test, z-test), but I can't get an exact match. (Update: WHuber's answer has an algorithm that produces an exact match... it's scary stuff!)

Looking for formulas the OFCCP might use, this site I saw may perhaps be helpful: http://www.hr-software.net/EmploymentStatistics/DisparateImpact.htm

Summary of some calculations:

$$ \begin{array}{rrrr} \text{Number and method} & \text{Part A} & \text{Part B} & \text{Part C} \\ \text{PVal from hypergeometric CDF} & \text{7.839e-04} & \text{1.77e-06} & \text{1.72e-08}\\ \chi^2 \text{ stat} & 15.68 & 33.68 & 37.16\\ \chi^2 \text{ pval} & \text{7.49e-05} & \text{6.47e-09} & \text{1.09e-09} \\ \text{Pval from above document} & .00135 & \text{2.94e-07} & \text{1.00e-09} \end{array} $$

For $\chi^2$ stat I used the standard $\sum \frac{(\text{expected} - \text{actual})^2}{\text{expected}}$ over the four cells.