0

enter image description here

Hi,

I was wondering if anyone had advice on removing outliers. In a practical experiment relative telomere length in DNA samples was measured in duplicate. Expected values are around 1, and the peak of samples between 0.0-0.5 are fairly certain failed samples. Since this data isn't normally distributed and the samples are likely fails, I was looking for a valid method to get rid of obvious outliers. Any advice how to do this for this data? I have tried Median Absolute Deviation already, but the peak remains.

Thanks!

Kim
  • 1
  • 1
  • When binned, the pdf (probability density function) of your data appears quite lumpy and anything but normally distributed. It may be that parametric models and assumptions aren't met and won't work. One option is to revert to nonparametric methods and models that are robust to outliers, e.g., estimators like the median (not the mean), quantile or kernel (not OLS) regression as well as assuming underlying extreme value probability distributions such as the Weibull, Cauchy or Levy (not the normal). Adopting these approaches should, for the most part, eliminate the need to delete outliers. – Mike Hunter Jul 26 '17 at 11:13

2 Answers2

1

The "failed" samples are not really "outliers", they are clearly a normal part of your experimental process. What you need to do is correctly model this process.

For example, it may be appropriate to assume a mixture of normals distribution. You would let $X_i \sim^{\text{iid}} \text{Bern}(p)$ be 0 if sample $i$ is a "failure", 1 otherwise, and then assume different conditional normal distributions for each outcome:

$$Y_i|(X_i = 0) \sim N(\mu_f,\sigma_f^2)$$ $$Y_i|(X_i = 1) \sim N(\mu_s,\sigma_s^2)$$

You only observe the $Y_i$ but you can estimate the parameter(s) of interest (which seems to be $\mu_s$, the mean of successful samples) using an algorithm like EM.

Chris Haug
  • 4,893
  • 1
  • 17
  • 24
0

In addition to the presented solution by @Kim I would transform data in order to normalize it which is a good solution instead of deleting outliers. See these answers to know more about the used transformations link.

BUT before I think you should see these comments link

Noah16
  • 159
  • 11
  • Which transformation would you suggest to transform a bimodal distribution, such as the one shown above, into a unimodal one like the normal? – Chris Haug Jul 26 '17 at 15:13
  • The Most important thing is to study the sample because in the general case it is not preferred to transform a bimodal variable into normal because it will be incoherent. Maybe the data comes from two different groups like in this example about DNA may we have adults &children, males&females,...etc So in this case the we have a bimodal distribution which can not be normal but it could be analyzed – Noah16 Jul 26 '17 at 15:47
  • Thanks for your replies! I am not familiar with modelling processes but you are correct about the mean of successful samples being our . However, I am guessing we first need to come up with a valid cut-off/reasoning when and why to regard samples as failed samples (extremely low values that are biologically impossible)? @Noah, the data comes from one and the same group and I checked the sex/age distributions and they were equally in samples with expected results and too low/likely failed results. – Kim Jul 31 '17 at 11:41