StackExchange fires a moderator, and now in response hundreds of moderators resign: is the increase in resignations statistically significant?

Question

I am doing a study on StackExchange. The management of StackExchange has demodded (for unclear reasons) a moderator, and now the network is on fire.

Currently many moderators resign or suspend their activities because they are dissatisfied. I wish to gather and analyse data about these resignations. I like to find out whether there is an increase or decrease of dissatisfaction and whether this is statistically significant.

• What kind of test can I perform to find this out? In particular I need some guidance on how to analyze/model/define this increase (the problem is that I have no simple linear model that I can fit to the time of events, it might be non-linear, so how to deal with that).

• I am planning to use this petition letter and this list of resignations to define events. How can I combine all this into a single model?

  For data stamps I am thinking about using the posts on meta-sites rather than to look for it in the text.

  The event types I wish to collect because possibly more data might allow me to have more power in my test?

  I am thinking about creating something like a table that looks like:

  Id     Moderator          Event-Type            Date-stamp       
  
  1      Monica Cellio      Fired                 Sep 27 
  2      Glen_b             diamond removed       Oct 9 at 0:53
  3      Gung               suspending activity   Oct 18 at 1:32
  4      whuber             weekly strike         Oct 18, 25, ...
  

  Ideally I am not making the table complete because that's a lot of work for the hundreds of events but instead do something like a random sampling (e.g. digging through posts like Gung's or GlenB's or comments like Whuber's). So this must be a consideration for the model/test that I am gonna apply.

---

Partial result/work

Based on the comments I did some initial parsing of the petition letter which results in the following image:

library(XML)
u <- 'https://dearstackexchange.com/'
html = htmlTreeParse(readLines(u), useInternal = TRUE)
dates = unlist(xpathApply(html, '//small', xmlValue))
dates <- text[-length(text)]  # remove final value
times <- 5+(as.numeric(strptime(dates, "%b %d")) - as.numeric(strptime("Oct 5", "%b %d")) )/24/3600
t <- table(times)
plot(t, xlab = "date (month October)", ylab = "number of signatures")

We see this peak of signatures on the 7th oktober and then a decrease. This is no surprise and relates to what gridAlien describes in his/her/their post as an intital firing. But there is still a remaining number of signatures toward the end of the month. Is this number increasing or decreasing?

Answer 1

This is an interesting investigation because of the flash-pan nature of the event. It's not the same as, say, installing a fence and trying to see if the number of trespassers was reduced. In that case, after the fence was installed, we would expect to see a permanent impact (if there was any) on the rate of trespassers.

In this case, a bunch of mods will resign/be fired/be suspended over the issue over a course of a few days, and then the rate of these will die down. There are only so many moderators willing to/forced to do these things, and once they do them it's done. We would expect the rate of leavings to die down eventually.

Graphically, you could represent this with a line chart. If you take the number of moderators leaving per day and plot it, you'd expect to see relatively consistent leavings up until the firing (lets call it $D_0$) after which you expect to see an increase, and a fall back to the original rate.

Numerically, if you wanted to show that this spike is not within the normal variation of the process, I'd try to treat this almost as if it were quality control. Take some data from before $D_0$. Calculate an estimate of the mean leavings per day, an estimate of the variance, and then construct a confidence interval for that mean at your preferred level of significance. If $D_0$ (and other days in the aftermath) are outside this interval, then you can conclude that these points represent a shift in the average leavings per day.

Anyway, that's my approach. I'm sure there are others.

Answer 2

The analysis you are proposing sounds interesting, but the data collection process will be quite complicated. There are a few main issues you are going to have to deal with:

1. Determine the scope of events of interest: Ideally you should determine the scope of events of interest to you (even in just a broad way) before you begin collecting the data. This could be a broad stipulation of all events involving an intentional diminution in activity.

2. Determine sampling frame and sampling method: You will need to determine your "sampling frame" to ensure you have a proper sampling method. The simplest way would be to stipulate some user criteria at a particular point in time (e.g., all moderators, all users with 5000+ rep, etc.). You will then need to decide how you will sample --- e.g., simple random sampling, or weighted sampling (e.g., by user reputation).

3. Find a baseline for comparison: Obviously the other important element will be to get baseline data on how users were behaving before the issue arose. I would recommend that you examine each of your sampled users and get some metrics of their activities prior to the scandal (e.g., activity over previous year).

May I offer a suggestion for a simpler analysis you could do with a lot less effort at data collection. Right now, a large number of users have converted their pictures to the "Reinstate Monica" picture, and many have also changed their user names. It should not be too onerous to go through the leader-boards of each site, and make a list of all users above some level (e.g., top 10, 20, 50) and list whether the user has converted their name and badge, and get a measure of the user's level of activity since the initial scandal. You could then do a "survival analysis" estimation of the rate of conversion on the different sites. Of course, this would only show that there is some symbolic solidarity on display, but it would be a simpler analysis than your proposal.