3

I have the following challenge: The dataset has one dependent and one independent variable which are connected in a non-linear fashion. I am trying to give a more qualitative picture here because I am unsure how to handle the dataset.

Basically there are five regions from "one end to the other":

  1. "Unstable" relationship with "too few" data points
  2. Negative relationship with the dependent variable in negative territory
  3. Negative relationship with the dependent variable around "0" (seems also "unstable")
  4. Negative relationship with the dependent variable in positive territory
  5. "Unstable" relationship with "too few" data points

I differentiated regions 2-4 because it makes a difference concerning the actions to be taken for the different regions.

My question
I need an algorithm (best an implementation/tool) with which I can make the above mentioned regions exact (it is ok to stay with 5 regions). The width of the regions is not equal so the cutoff-points have to be optimized on basis of "distinctiveness". I thought of piecewise regression or optimal width histogram or even some more elusive clustering algorithms but I haven't gotten very far with these approaches (might be my fault). Perhaps you can share some ideas where to start.

JohnRos
  • 5,336
  • 26
  • 56
vonjd
  • 5,886
  • 4
  • 47
  • 59
  • You'll want to check out Frank Harrell's comments on this site about splines. – rolando2 Oct 05 '11 at 20:25
  • If it's adequate to think of the relationship *within each region* as being approximately linear, then this becomes a problem of changepoint detection (for slope and residual variance). You can find lots of good references in the questions at http://stats.stackexchange.com/questions/5700/finding-the-change-point-in-data-from-a-piecewise-linear-function and http://stats.stackexchange.com/questions/2432/loess-that-allows-discontinuities/2445#2445 – whuber Oct 05 '11 at 20:46
  • 2
    @rolando Which [comments](http://stats.stackexchange.com/search?q=user%3A4253+%2Bspline&submit=search) did you have in mind? None of the ones I can find seem to relate to this question, but maybe I'm not searching for the right keys. Even if you do make a good spline fit, how do you go from the coefficients to estimating the locations of the regions (which is not the same as getting the spline coefficients)? – whuber Oct 05 '11 at 20:49
  • It would be extremely difficult, and would require tens of thousands of observations, to demonstrate the existence of such cutpoints. – Frank Harrell Dec 22 '11 at 14:23

0 Answers0