1

I have pandas data frame which has some related columns, like let's say I have a column that showing housing sale prices and other column shows total area of that house. If I look for outliers alone it's showing really expensive houses but some of them makes sense because they have huge area, what I want to find is cheap houses with huge areas.

I can spot them by eye if I plot scatterplot and filter them out manually, but is there any way to do this without visual analysis?

retardo
  • 43
  • 5

3 Answers3

1

If you want to remove outliers based on the assumption of a linear relationship between both variables, you can fit a robust linear regression.

Results will be less influenced by outliers than in the case of using traditional OLS. After that you can check the distribution of errors, outliers are those points with unusual big errors.

javierazcoiti
  • 613
  • 4
  • 11
1

Here are roughly associated $(x,y)$ pairs. The R code below identifies the outliers among the $x$'s and then finds the corresponding $y$'s.

So i think the orange points in the scatterplot (below) are the points of interest. Note that $x$'s (areas) rule, the largest $y$ may not correspond to any $x$ outlier. (You could make a list if you don't want to look at plots.)

set.seed(1234)
x = rexp(1000, .01) + 2000
y = x + rnorm(1000, 0, 50)
boxplot(x, horizontal=T, col="skyblue2", pch=20)

enter image description here

x.out   = boxplot.stats(x)$out
y.corsp = y[which(x %in% x.out)]
plot(x,y)
 points(x.out,y.corsp, col="orange", pch=20)

enter image description here

First six orange points in the list.

head(cbind(x.out, y.corsp))
        x.out  y.corsp
[1,] 2305.246 2207.574
[2,] 2432.456 2500.806
[3,] 2342.528 2359.964
[4,] 2368.037 2329.798
[5,] 2622.573 2645.349
[6,] 2365.179 2354.905
BruceET
  • 47,896
  • 2
  • 28
  • 76
  • This doesn't help solve his question. The orange points are not outliers for him. He wants mispriced houses, not big and expensive houses. – javierazcoiti Apr 28 '20 at 10:45
  • He says he wants " to find is cheap houses with huge [expensive] areas." I'll be happy to let OP decide if this is helpful. And delete it if not. – BruceET Apr 28 '20 at 10:50
1

One way to do this is to define intervals on x (e.g., house areas). After separating the x's intervals, iterate over them. In each interval iteration, select only the observations that belongs to the current interval and use the interquartile rule on y (e.g., house prices). With that, you can identify which values of y are incoherent with values of x in each interval.

I think you must do this twice, first using x to generate intervals and, after that, do it with y.

One more thing: For x and y, generate the intervals using pandas.qcut. Why? With pandas.qcut, you avoid missing that outlier "alone" in the corner.

Gabriel Fritz Sluzala
  • 11
  • 3