Find data points that deviate the most in 2D graph

Question

I have a set of data points where X axis represents the percentage of disk space, and Y axis represents the memory used by an application. In general, higher the disk space, the higher memory it'll use.

Sample data points (disk space percentage, memory used): (10%, 1MB), (20%, 1.5MB), (25%, 1.2MB), (60%,10MB)

From this data, I'm interested in finding out top N outlier samples which are using less disk space but more memory, when compared to other data points.

What statistics are suitable for this purpose? Is it even possible to apply a model without knowing what's the normal memory usage for a given disk space on an average?

It *sounds* like you might be interested in the thread at https://stats.stackexchange.com/questions/213 — whuber, Jan 08 '19 at 20:39
will look at it. Edited the question a bit. Is there any other information I can add to the question to make it more clear? — RandomQuestion, Jan 08 '19 at 21:27
Either an illustration (as a scatterplot) or a clearer description of what it looks like could help, because there are various different ways to approach this situation, depending on what that plot looks like and what kind of "outliers" you mean. (There are some subtle distinctions when you view one variable as being "independent" and the other as "responding" to its value.) — whuber, Jan 08 '19 at 23:48

score 2 · Accepted Answer · answered Jan 08 '19 at 23:42

There are a number of statistics and functions for defining outliers.

Assuming a normal distribution, you can standardize your data according to the sample mean and sample standard deviation. Assign a z-score to each observation.

Given that you have a bivariate data set, and that you want to know the outliers conditional on disk space, I would recommend doing a linear regression.

If you have access to the data, I would recommend doing a residual analysis in R.

Build a linear model using memory as a function of disk space. The linear model works best under assumptions of normality of variables. Then, you're looking for large positive residuals. (Positive because you're looking for outliers that use a lot more memory.)

library(car)
set.seed(1234)

#generating data
disk <- c(rep(.01, 25), rep(.05, 5), rep(.1, 5))
memory <- rnorm(35, 5, 1)
mydata <- as.data.frame(cbind(disk, memory))
model <- lm(memory ~ disk, data=mydata)

###residual plot
par(mfrow=c(2,2)) #better ordering of graphs
plot(model, which = 1:4)

###Order data by largest residual
residuals <- model$residuals
mydata2 <- as.data.frame(cbind(disk, memory, model$residuals))
colnames(mydata2) <- c("disk", "memory", "residuals")
mydata2[order(-residuals),]

Here is the residual plot, qqplot, cooks distance and scale-location.

And here is the output after sorting the data based on descending residual value (so the top values are the largest outliers.

If you're looking for specific statistics, I would consider using Shapiro-Wilk for normality of the distribution, or the outlierTest() function from the car package.

Find data points that deviate the most in 2D graph

1 Answers1