2

I would like to measure the change of a policy in a community, that is, check whether there was a change after the new policy was applied. The old policy required that people had to give away a right in order to participate in a community (similarly as academic journals work, where authors usually are asked to give away their copyright). The new policy dropped that requirement.

The old policy could have refrained some people to contribute (or participate) to the community.

The hypothesis would be:

  • $H_0$: The change in the policy does not produce any effect in contribution (participation)
  • $H_1$: The change in the policy produces an effect in contribution (participation)

It seems to me I have to use Chi-square (with a contingency matrix) because I would like to measure changes in the population (contributions or participants).

The following aspects makes me hesitate on how to approach the problem:

  • There are participants that contribute before and after the policy change. Therefore, the groups before and after are not independents. (Does this matter for the research design?)
  • The participation level is not homogeneous, neither before nor after. Because these contributions happen in a period of time (years), before the change there were participants who contributed for a short period of time, others for a longer period and other that still contribute.

Questions:

  • ¿Is Chi-Square the most appropriate approach for this problem? If not, what would be an adequate approach?
  • Does it makes sense to be concerned about the previous aspects listed? If so, how should they be approached?
  • Is there anything else I should consider? (something that could invalidate my analysis or something that could improve it)

For what it is worth, after analyzing one community and if the null hypothesis ($H_0$) were rejected, I would try to analyze if there are changes in the level of contributions of the active participants in both before and after periods. After that, I would analyze other communities with similar policy changes and compare them (if applicable).

Finally, it is an ex post facto study.

I have looked into another questions, but they seem different:

Community
  • 1
gpoo
  • 209
  • 1
  • 4
  • 13
  • How is your response measured? – Glen_b Apr 20 '13 at 03:23
  • There is no questionnaire. All contributions are recorded (person, date, contribution, etc.). Contributions are tipically in source code. However, participants give away their rights only once, before making the first contribution (the waiver is the any further contribution.) – gpoo Apr 20 '13 at 03:34
  • You misunderstand. WHat is being compared before and after? *Don't* just say the name of the variable (like 'participation'), *tell me how you measure it*. Is it a count? A rate or proportion? – Glen_b Apr 20 '13 at 03:43
  • Contributions are usually focused, so they could be counted as times a participant has submitted a contribution or the size/complexity of the contribution. The records allow both. – gpoo Apr 20 '13 at 03:48
  • When I say *participants* is because that also is counted. The change in the policy is to attract more participants (increase de population of the community). – gpoo Apr 20 '13 at 03:50
  • What we have here is a failure to communicate. Please *precisely* define how contribution and participation are *measured*. – Glen_b Apr 20 '13 at 06:13
  • Or maybe it is something I do not even have clear because of the problems I mentioned (sorry). A person either participate or not (binary), but the universe is unknown (anybody can participate). So, participation is measured as: 1) the number of people who have contributed, 2) # of contributions person / total of contributions, 3) size of contributions person (in lines of source code) / total size of contributions. A contribution is an event and it is measured as: 1) by count: # of events in a period of time (before, after), 2) by size: lines of source code in each event. – gpoo Apr 20 '13 at 07:24

0 Answers0