2

I ran an experiment and obtained the points shown in black below.

I would like to smooth the curve or fit (something like the red curve) in order to identify the elbow.

The problem is that the experiment is too expensive and there is a large amount of variance in the obtained points. I tried fitting a second degree polynomial and an exponential curve without sucess (the curve is too smooth and the elbow is lost). Total precision is not necessary.

What is the best way to fit a curve in order to identify the elbow?

enter image description here

rolando2
  • 11,645
  • 1
  • 39
  • 60
test
  • 51
  • 2
  • 1
    What's wrong with using a loess curve? – gung - Reinstate Monica Mar 21 '12 at 03:02
  • You surprise me, @gung, since often you (along with cardinal and Frank H) seem to argue against using such opportunistic strategies and instead advocate using approaches that achieve nominal p-values and preserve nominal Type I and Type II error rates. – rolando2 Mar 21 '12 at 03:09
  • Heyya, @rolando2. Good to hear from you again. As far as I know, there wouldn't be an analogous problem with locating a bend via loess, if that was the point all along. I could be wrong, though, and if I am, I would be happy to retract the suggestion. I'm mostly trying to elicit more information so that the question can better be answered. Do you know of a problem with using loess here? – gung - Reinstate Monica Mar 21 '12 at 03:16
  • I will try using a LOESS – test Mar 21 '12 at 12:45
  • hi @gung - I guess "if that was the point all along" is the important distinction. Am I right that you would disagree with an approach that started with loess in order to determine whether a linear, quadratic, or cubic line fit better? And if so, that the problem could be described as something like "inappropriately avoiding the expenditure of degrees of freedom"? I'm thinking especially of a related thread at http://stats.stackexchange.com/questions/9334/determining-best-fitting-curve-fitting-function-out-of-linear-exponential-and . Cheers. – rolando2 Mar 21 '12 at 13:16
  • 4
    @test - I'm unsure of what you are ultimately trying to do. You wrote, "My objective is to identify the curve elbow." But isn't it plain to see--among your sample data at least? Maybe what you really want to do is to generalize from your sample to the larger population. If that's the case, loess smoothing won't help you as it's totally dependent on the data at hand--it will provide no equation with which to predict where the sharp drop in tb$V4 will occur with out-of-sample data. While regression splines is not a technique I can explain, you may want to look into it. – rolando2 Mar 21 '12 at 13:23
  • 1
    @rolando2, you're right, if you fit a loess, then chose a polynomial of order $p$ based on what's seen, but didn't originally intend to / wouldn't have if you'd seen something else, then that model would be biased & CI's, eg, would be inaccurate. I don't think that's the case here, although I'm not sure, & I was mostly trying to elicit more info. I make a strong distinction in my head b/t exploratory & confirmatory data analysis. I don't have a problem w/ that stuff *as EDA*, but that means you think about it differently (eg, as biased), & use it to plan another study for unbiased CDA. – gung - Reinstate Monica Mar 21 '12 at 14:34

3 Answers3

2

I used the LOESS regression and it seems to do the work. Since I have a big variance for each point I wanted to smooth this curve in order to get a similar 'elbow' for each execution.

The loess seems to do this job just fine.

test
  • 51
  • 2
2

Smoothing will tend to "smooth over" the elbow in the data. If your view, based on domain knowledge, is that the true model has a major breakpoint or elbow, then you should consider fitting a model that is not smooth.

I suggest considering regression trees or hinge functions. There are many methods for fitting these, e.g. gradient boosting and MARS. These are sometimes thought of as black box machine learning methods. That said, interpreting a model with only one independent variable is usually pretty simple.

MichaelJ
  • 636
  • 6
  • 6
1

What you have are two time series that are to be impacted by unspecified deterministic series e.g. local time trends and/or level shifts and/or pulses . If you post your actual data ( i.e. pairs of Y AND X's ) I will try and help you . The composite equation could include the predictor , lags of the predictor , lags of Y and some dummies (0/1) series to provide you with the needed equation/parameters.

IrishStat
  • 27,906
  • 5
  • 29
  • 55