0

Let's say I have results of a measurement for a large number of treatments (p-value, beta, t-stat) on samples from a large number of different populations. I want to compare how different the effect of a treatment was in different populations, of which there are 30, and then look at which treatments were most sensitive to population differences. Populations that have effects in opposite directions are a definite possibility and would be of particular interest. Unfortunately, I don't have the raw data so I can't just go back and do a test that includes population as a factor.

My thought is to get an estimate of this by taking beta divided by the p-value. By doing this, I can factor in both the absolute difference between beta coefficient values, and my certainty that the coefficient is significant. In other words, I would get the biggest difference in scores by maximizing the absolute difference between their beta-coefficient values and minimizing each of their p-values. Either increasing p-value or decreasing the absolute difference between beta-coefficients will lower the score.

The problem that I am anticipating is that increasing p-value is not considered a valid way to claim increasing likelihood of the null hypothesis being true. I'm hoping that I can get away with it by thinking of p >0.95 as "significant non-significance" as opposed to "accepting the null". Is there something wrong with how I am thinking about this? Would this give me any different information than just looking at the effect size (t-stat)?

I've looked at this thread: Why are 0.05 < p < 0.95 results called false positives? and this: https://www.reddit.com/r/askscience/comments/2yiqjq/why_can_we_not_accept_the_null_hypothesis_if_p095/

Avraham
  • 3,182
  • 21
  • 40
Stonecraft
  • 101
  • 3
  • Is it that you want to know the standard error, but don't have access to it? Do you know how many degrees of freedom are associated w/ your betas? – gung - Reinstate Monica Aug 14 '16 at 02:59
  • I hadn't thought about factoring in standard error, but I don't have access to it. I'm pretty sure I can get the sample size though (df = n-1, right?). What I want is to identify which groups responded most differently and most similarly from one another. – Stonecraft Aug 14 '16 at 03:10
  • This sounds a bit like a meta-analysis (MA) or meta-regression (MR) problem. If I understand you correctly you have a number of regression coefficients and enough information to back-calculate their standard errors. That would be enough material for MA or MR but perhaps you could clarify your exact scientific question as I do not want to send you off on a wild-goose chase. – mdewey Aug 14 '16 at 10:56
  • I'm not doing a meta analysis, but it could be that meta-analysis methods would work. I am looking at eQTL data from a bunch of different tissues, so what I have is a really big file with a list of genetic variants and the results of statistical tests of their effect in each tissue. What I want to do is compare the effects of variants across tissues. Ultimately, I want to identify which variants are the most and least tissue-specific (as well as other stuff). – Stonecraft Aug 14 '16 at 18:06

1 Answers1

0

Have you considered looking into the literature for references? I have never heard "significant non-significance" and that phrase made me chuckle. If you have a t-statistic and N, I would look up Jeff Rouder Bayes factor calculator online bayes factor calculator. This way you can compute evidence for the null. However, be sure to read a few of his papers so that you understand the approach and what you are doing.

D_Williams
  • 685
  • 1
  • 5
  • 13
  • Yes, I just made that up. But I honestly don't understand why it's wrong to think of it that way. – Stonecraft Aug 14 '16 at 06:48
  • Are you saying that if I compute Bayes factor and then multiply it by the beta coefficient, it will do what I was trying to do? Inorder to read up on ways that people combine effect direction and effect strength with evidence strength into a single number, what terms should I search for? As you can probably tell from my question, I don't know what the word for what I'm trying to do is. – Stonecraft Aug 14 '16 at 06:58