4

This is more of a theoretical question. Super large sample sizes will almost always show a significance when a $\chi^2$ test is done. Is there any other statistical test of significance (an alternative to $\chi^2$) which is good for testing independence when the sample size is very large?

This is the context of my problem: I have 2 large datasets of phrases. Set1 corresponds to the Google n-grams set and set2 is a smaller set corresponding to the phrases found in one single website. Now consider a phrase: say ('Technology') found in Set2. I want to test and see if this phrase is specific to this website (it could be if it is a Technology website) or if it is a general phrase. So I am performing a $\chi^2$ test between the frequency of phrases in the two sets as follows:

                        Set1                        Set2
Not_Technology   (set1) 2,674,797,869,255    (set2) 46,168,477.00 
Technology       (set1) 1710231              (set2) 1991

I understand that this might not be the best method to test whether a phrase is a general phrase or not, so if you have any suggestions or criticisms I am happy to listen to them.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
tan
  • 527
  • 2
  • 5
  • 13
  • Explain the context of your problem. Are you looking for independence in a contingency table? – Michael R. Chernick Aug 31 '12 at 20:39
  • In general any useful test will find significant differences in very large samples because when comparing two populations there will always be at least some small difference and small differences will be detected in very large samples. – Michael R. Chernick Aug 31 '12 at 20:41
  • @MichaelChernick right. So finding a statistical difference does not necessarily mean that the difference is large or important. In the case of a very large sample we will almost always find small differences. So is there a way to measure that the difference is important? I have mentioned the context of my problem in the question above. – tan Aug 31 '12 at 21:00
  • Look up resampling, or bootstrap, or jackknife techniques to assist is finding actual significant differences in large samples. All three are supported in R. – R. Schumacher Aug 31 '12 at 21:03
  • 5
    My advice is to stop thinking about statistical significance and start thinking about effect size. In a 2x2 table such as you seem to have, the odds ratio is a good (and easy to calculate) measure of effect size, just find the cross-product. Most software will have tools for confidence intervals around these, if you want them. – Peter Flom Aug 31 '12 at 21:12
  • 1
    @R.Schumacher How would that help? I don't see how any of those methods help with this particular issue. – Peter Flom Aug 31 '12 at 21:13

3 Answers3

3

Take a look at this paper by the late Jack Good: http://fitelson.org/probability/good_bnbc.pdf ; At section 4.3, his "Bayes/Non-Bayes Compromise" leads to the definition of a "standardized" p-value which tries to address the "Huge $n$ $\Rightarrow$ Highly Probable to Reject Null, Whatever Data" effect.

Zen
  • 21,786
  • 3
  • 72
  • 114
2

If your sample is large enough then it seems to me that a statistical test is not needed. You have characterised the effect. Is the effect that you have characterised large enough to be interesting? If so, then make a reasoned and principled argument about the observations without recourse to a testing procedure.

Michael Lew
  • 10,995
  • 2
  • 29
  • 47
  • 1
    Testing could still be relevant if the estimated difference is close to what is defined as a meaningful difference. Testing is a formal way of assessing how sure you are of the result. – Michael R. Chernick Sep 01 '12 at 11:36
  • @MichaelChernick Most people, and probably all scientists, are interested in the evidence. The idea of a 'meaningful difference' in the context of testing comes from the Neyman-Pearson approach where it stands as the alternative hypothesis. That approach eschews any evidential basis of interpretation in favour of 'behavioural inference'. Not as useful as looking at the data as evidence, in my opinion. – Michael Lew Sep 02 '12 at 22:52
  • Neyman-Pearson hypothesis testing doesn't define a meaningful difference. That is determined by the investigator and really determines which alternative hypotheses we really care about. I don't share your pessimistic view about hypothesis testing. – Michael R. Chernick Sep 03 '12 at 00:56
1

When we do sample size determination for clinical trials we define a clinically significant (or clinically important) difference. That is a difference that is large enough to be worth detecting. The definition is given by the clinician. It is not a statistical issue. It depends on the clinical problem and requires a clinical judgement. Once the clinician has decided on that we pick the smallest sample size required to have high power (80% or more) for detecting a difference that large.

In your case where you already have millions of samples what you can do is rephrase the question. Instead of the standard null hypothesis that the difference is 0 which you reject if you can determine that it is any size different from 0, define a delta that represents what you think is an important distance. Then you reject the null hpyothesis only if the test indicates that the difference is greater than delta.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • 1
    I can follow how that could be done, but is that what is wanted? If delta is meaningful, then you would want to detect not "is the difference greater than delta?" but "is the difference at least delta?" Perhaps a 1 sided test? Or maybe make delta the "maximum meaningless difference" that is, the largest difference no one would care about. – Peter Flom Aug 31 '12 at 22:13
  • 1)I suspect, in some cases, the difference that is “different enough” is really unknown (maybe until a certain level). Would it be appropriate to make a table of [difference, significance]? Would it help? 2) Isn't [equivalence testing](http://stats.stackexchange.com/questions/23617/effect-size-as-the-hypothesis-for-significance-testing) related to this topic? – if it is performed with (say on some acceptable delta), would the large sample have the opposite problem ( because of the reversed null hypothesis) ? –  Sep 01 '12 at 00:07
  • @PeterFlom Usually we are dealing with continuous variables,so greater than delta or greater than or equal to delta is a moot point. Anyway the choice of delta is a rough choice. – Michael R. Chernick Sep 01 '12 at 11:05
  • @6thwhy The choice of delta is not something that can be characterized as known or unknown, it is a subjective choice for the investigator. If the investigator is not sure what he wants to use we do look at tables for sample sizes based on a set of reasoanable choices. Even if you are not sure of what is a large enough difference, you probably have a good idea of what is too close to be meaningful. Equivalence testing is a different problem. There you want to show that two groups are practically the same. So the null and alternative hypotheses are switched around. – Michael R. Chernick Sep 01 '12 at 11:12
  • But delta does represent a difference that matters. So the interpretation is the same. But the choice could involve different considerations. – Michael R. Chernick Sep 01 '12 at 11:13