3

I am analyzing Stack Overflow Posts. So I have a database with 1000000 questions, their current score (upvote or downvote) and a flag, that there is a source code part in the question (or not).

So I want to test. Is there a correlation between the presence of source code and the votes. So are posts with code have a higher score than posts without.

So I created a cross table like this:

enter image description here

Now I will do an Chi-Square test with R with these values. The Result is like this:

enter image description here

As you can see, the result is significant (p-value). So I am right, that there is a correlation between the score of the vote and the presence of code?

I am not sure, that I am doing things right here. And another question: How do I know the direction of the correlation. Does the result mean, that the presence of code will result in a higher score, or in a lower?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Chris
  • 142
  • 1
  • 8

2 Answers2

3

Your results suggest there is an association between the presence of code and a post's vote category. Because you / a chi-squared test treats your variables as unordered categories, it is not generally meaningful to ask about "the direction of the correlation". The confusion is due to the fact that net number of votes is not actually an unordered (nominal) categorical variable. If you have access to the actually net vote count for each post, you can correlate a vector of 0's (no code) and 1's (with code) with a vector of net votes. Then you could specify the strength and direction of the association. You could also refer to the raw mean difference in votes as an intuitive measure of how the distributions differ.

Regarding causality, statistical techniques (such as the chi-squared test) don't generally help you with that. (It may help you to read this excellent CV thread: Statistics and causal inference?) To identify if adding code has a causal impact on net votes, you need a true experiment (i.e., randomly assign posts to have code or not). In lieu of that, you can try to search for an instrumental variable (also see: How do instrumental variables address selection bias?) that would give you some leverage, but wanting a good instrument doesn't mean it's easy to find—or even that one exists.

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • So would it be a good solution, to do a chi-squared like this, to see a possible correlation. Then do a regression with the raw score values (non categorized) to proof the correlation and direction? It is all about correlation, I am not talking about causality. – Chris Aug 28 '15 at 06:50
  • @Chris, if you have the raw scores, I would skip the chi-squared test altogether. Also, "the presence of code will result in a higher score..." is an inherently causal claim. There may be something else that leads people to include code & that causes higher scores, but no direct connection between code & scores themselves. – gung - Reinstate Monica Aug 28 '15 at 12:49
  • you are right. There may be a lot of other influence on the score. I do not want to show that there is a causality. I am doing my master thesis. I have a lot of literature, that point to the fact, that in software project documentation the presence of source code is important for the quality of the documentation. So all I want to say is, that the data on Stack Overflow not argue against my litarature. So I can not say, that on SO the presence of code will lead to a higher score, but I can say, that the data on SO do not show, that my literature is wrong. – Chris Aug 29 '15 at 18:58
1

You are testing a simple hypothesis with a large sample size. It is very reasonable that the two variables in question are not truly statistically independent so you're almost guaranteed to reject.

See this question for more details on hypothesis testing with large samples.

As for the direction of the correlation: if you have already rejected the null hypothesis that the two variables are independent then you can simply compute the sample covariance (using the numerical number of votes, not the binned one) and note the sign. There may be better ways to get at this but that's what I'd do. If you're more interested in assessing how the presence of source code relates to the resulting question score, then you could fit a regression with the predictor being the indicator for source code and the response being the number of votes. Of course if the only predictor you have is the indicator for source code this regression will just give you the mean of votes conditioned upon the presence of source code. But this is a more general framework and you can then add other variables if you have any. Note that this does not tell you anything about causation (as mentioned in gung's answer).

Community
  • 1
jld
  • 18,405
  • 2
  • 52
  • 65
  • regarding the first part of your answer, it seems that the accepted answer on the thread you linked goes against what you say: `If the null hypothesis is true then it will not be rejected more frequently at large sample sizes than small. There is an erroneous rejection rate that's usually set to 0.05 (alpha) but it is independent of sample size` – Antoine Aug 29 '15 at 19:29
  • Yes, if the null is true. But with real data of the sort that OP has I'm sure that the null isn't exactly true (because we have a simple null) so it's the power, not the type I error rate, that matters. To me this question then is more an issue of practical significance rather than statistical significance. – jld Aug 29 '15 at 19:47