4

I have a dataset of 380 samples of 6 variables. These variables are counts of different types of events in each of the 380 defined regions. These counts are per month, which means that I have several of these datasets (for now, I only have four months).

When looking at the data, I can clearly see that there is some missing (or incomplete) data. For instance, the counts for one given region are about the same for all months, except one (where it's close to zero). However, I does not seem likely that there were actually that few events during this period.

What I would like is to be able, given data for a few months, to detect missing or incomplete values, and possibly to correct/complete them. Detecting missing values is not as obvious as looking for zeros, because it may happen that no event occurred in some regions.

I read a few things about matrix factorization, but I'm not sure it would apply to my case. It seems suited for the cases where you know what data is missing.

I assume this kind of problem is be quite common, for instance in biology for population estimation.

Wookai
  • 505
  • 3
  • 10

4 Answers4

3

I think you are really looking for outliers in your data (small values where most 'similar' values, i.e. values where a set of covariates hold the same values, are 'bigger').

You could look at outliers in MV data for this for starters, although you may be able to utilize some particularities of your situation (you are only interested in outliers in one variable, and only in one direction).

If your sample size is big enough for that, you can simply use the outlier definition as used in most boxplots (1.5 IQR away from the outer quartiles). If not, you should apply some truly parametric way of detecting outliers (e.g. residuals in a regression).

Community
  • 1
Nick Sabbe
  • 12,119
  • 2
  • 35
  • 43
1

Dealing with missing observation is a never ending problem. There are a few approaches i will mention, but I am sure others will add more.

  1. Make sure there is no any problems in recording the data, such as systematic omitting certain values. Given you have no control for it, then other options need to be considered.

  2. Based on the proportion of missing values, you have different options.

If there are less than 5% missing observations:

2.1 I would not care, and use either average or median based on the shape of the sample's distribution. Some use this approach even when there are more than 5% missing observations. It is a very quick solution when time pressed.

If there are more than 5% missing observations, some advanced techniques can used:

2.2 Data imputation methods is a wide class of techniques to impute missing values based on the observed values of the variable/or other variables.

For example, in time series framework, some values could be forecast based on the observed time series data.

In regression framework, knowledge of other variables as well can be used to impute missing variables, i.e., build a regression model for the variable of interest (the one with missing values) as a function of other variables available (probably excluding the original dependent variable Y) and then predict the missing values using the model.

A more advanced technique is Expectation Maximization method.

Finally, a dummy variable can be used just to control for the missing values.

M

yeveee
  • 164
  • 4
  • I cannot recommend using "dummy variable adjustment" (aka "a dummy variable can be used just to control for the missing values."). It is known that "this method generally produces biased estimates of the coefficients" ([Allison, Missing Data, p. 10](http://bit.ly/lEyBj8)). – Bernd Weiss May 22 '11 at 19:32
0

What could be done is:

  1. Apply some outlier detection
  2. Delete the detected outliers
  3. Perform a imputation algorithm

This could be one possible approach for this problem. But of course you would have to check if this actually improves the results or just makes things worse.

Steffen Moritz
  • 1,564
  • 2
  • 15
  • 22
0

In addition to the answers above, I think that you have a couple of issues:

First, from your post it does not seem that you have any real way of distinguishing an incomplete response from a valid response that happens to be 0. I would check for published literature from which you can find reference rates of incomplete responses, typical counts for your variables in similar regions, and so on. I do not recommend going by what "feels true" as a method for identifying anomalous responses unless you have a lot of experience with the research topic. If you are unable to find plausible reference data, I would run the analyses multiple times with different thresholds for what counts as a potential "incomplete 0" rather than a "complete-observation-that-happens-to-be-zero".

As for imputing missing data the method I'm most familiar with is a hot deck imputation, which may or may not suit your needs. There is almost certainly a hot deck imputation package available for whatever statistical software you use. Of course, this method (and any imputation method I've heard of) requires you to be clear about what data requires imputation-- impute haphazardly and your data set becomes more and more like pseudorandom number generator output.

Upper_Case
  • 517
  • 2
  • 12