0

I work with some data that includes some "extreme outliers". E.g. timestamps that are totally unreasonable (surgery took 20 days when most take 1 hour). Is there a set of principles one can use to deal with this kind of issues and perhaps find some signal in the reasonable registrations?

Also is there a package in R dealing with outliers? I am looking for a systematic approach and a set of principles or a set of rules.

xhr489
  • 173
  • 4
  • Outliers should be treated separately. You may remove them from your initial dataset and apply to them a specific treatment (finding why such values, and decide what to do given the reason you find to explain this values) – Manu H Apr 28 '19 at 10:34
  • 1
    How to deal with outliers depends on what causes them to be outliers. They might very well be your most valuable data points. That makes this a difficult thing to automate. Any package you use will depend on assumptions that might not be appropriate for your problem. – Frans Rodenburg Apr 28 '19 at 10:54
  • @ Frans Rodenburg. I am looking for a systematic approach, maybe not to automate but based on some principles or a set of rules... – xhr489 Apr 28 '19 at 10:57
  • 1
    There are about 1000 questions here on outliers. I don't think this thread raises any new issues not covered previously. Look at highly upvoted threads under the tag. Note that although you naturally want a systematic approach you can automate, some element of craft and judgment is hard to avoid. – Nick Cox Apr 28 '19 at 11:41

2 Answers2

3

With outliers, only one thing is really straightforward: If a value is clearly impossible, fix it or delete it. So, your 20 day surgery would be an example of that. If you can't find out what the right value is, then discard it.

After that, though, there is no good prepackaged set of rules. It's going to depend on the specific application, your goals and objectives and so on. It will require substantive knowledge and judgement.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
2

Outliers of this kind are usually examples of measurement error, and these commonly occur when a variable is recorded on the wrong scale (e.g., a 20 minute surgery being recorded as 20 days). Observations that are so extreme that they are highly dubious are generally dealt with either by removing them or (if possible) by making further investigations to determine the true value of the observation. In the case of medical data on surgery times, it should be possible to go back to the data source (e.g., hospital records) and determine what happened in the outlying observation.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • @ Ben. I have added a little bit to my question. I calculate the duration from two timestamps. But thanks. I will wait a bit and see if there are more answers... – xhr489 Apr 28 '19 at 11:00
  • Also I have thousands of observations and perhaps 100 that seem very wrong. I cannot investigate so many observations. I think the correct thing is to to discard them but I want to rely on some principles. – xhr489 Apr 28 '19 at 11:04
  • The only real "principle" at work here is that we try our best to get good-quality data, time permitting. Assuming that these are, in fact, incorrectly recorded observations, it would be worth reporting this back to the data source. If you have the time, investigate it further and see if it is possible to get corrected data. If time is limited, and the observations are obviously wrong, you may have to discard them --- that is not ideal. – Ben Apr 28 '19 at 11:06
  • Even if the observations are correct if 95% are very different in scale it makes it hard to do sensible things with the data. Maybe split the data or formulate the question in a different way? Usually there is one that is 20 days, then 19 and 18 and so on... – xhr489 Apr 28 '19 at 11:10
  • I disagree - the data (and reality more generally) are not required to conform to your opinions of what is "sensible" (see more [here](https://stats.stackexchange.com/questions/15536/is-it-reasonable-to-delete-a-large-number-of-outliers-from-a-dataset/337187#337187)). If you have correct data occuring on two different scales of magnitude then that just means that outcomes occur on two different scales of magnitude, and should be modelled as such. – Ben Apr 28 '19 at 22:03