Clustering in data

Question

I have a set of data points $(x,y)$ where $y = f(x)$. My goal is to fit the function $f$ using ols. The choice of function $f$ is quadratic due to domain know-how.The independent variable $x$ exhibits clustering, i.e. a lot of observations in a particular region and relatively sparse in others (shown is a plot of $y$ versus $x$).

I am looking to address this clustering because I think it is biasing the results. One way to solve this problem is to try robust regression, as I think y it is a weighting problem: in other words, the clustered data points are weighted more than they should be. I am curious if there are best practices on how to address this problem and what should I be aware of.

Answer 1

The distribution of $X$ is not an assumption in OLS regression. Thus the 'clustering' in $X$ is not necessarily a problem.

In addition, data tend to have less leverage* over the fitted regression line the closer they are to the mean of $X$. I note that the high density cluster is roughly where the mean would have been if $X$ had been uniformly distributed. Therefore, if you are worried that the cluster will have too much influence, I suspect those worries are misplaced and no weighting need be done.

On the other hand, if you are worried the cluster won't have enough influence (I don't interpret this as your concern), then you could use WLS (weighted least squares). You would develop a weighting scheme from some prior theoretical understanding that would make the clustered points more influential than the more sparse points to the sides.

For example, if you repeatedly sample the same x-value ($x_j$), you should get an ever better approximation of the vertical position of $f(x_j)$. As a general rule of study design, you should try to oversample locations in $X$ where you are most interested in knowing $f(X)$. However, note that if you oversample in an area you don't care about and undersample where you are primarily interested, and your functional form is misspecified (e.g., you include only a linear term, when a quadratic is required), then you could end up with biased estimates of $f(X)$ in your locations of interest.

---

If you are worried that the nature of the function $f(X)$ might be different inside the cluster than outside it, @user777's suggestion to use splines will take care of that.

_{* Also see my answer here: Interpreting plot.lm().}

Answer 2

Or is the shape more complicated? Restricted cubic splines fit flexible functions to different sections of the data; cubic spline design matrices can be estimated with OLS. MARS accomplishes a similar effect (different models for different subsets of points), but discovers the subset locations on its own.

Splines build up from this idea: a data set can be partitioned into different intervals, and each of those intervals can be modeled with some degree of accuracy as a constant. Alternatively, you can model each interval as a linear function. Still more elaborately, you can make the linear functions continuous at the interval boundaries. These boundaries are "knots." This solution is a piece-wise linear model. Restricted cubic splines further enforce that the function have several orders of differentiability at the knots, and that the function be linear in the intervals $(-\infty, k_1],[k_n,\infty)$, i.e. "outside" of the last knots. This linearity requirement is motivated by a desire to not over-extrapolate in very data-sparse regions. Restricted cubic splines allow for non-monotonic functions, e.g. the function can increase and then decrease and then increase again.

How to select knot locations is a sticky wicket, since it will influence what the ultimate model looks like. Frank Harrell's book Regression Modeling Strategies has some recommendations based on quantiles.