0

I have a data-set, a model (single variable) and a loss function. I can collect more data but each data point requires significant analysis to obtain. Hence how can I target the data collection to minimize the loss function? In other words, at what value of independent variable would additional data most reduce the uncertainty in my model?

Currently my approach is to add a data point (the value predicted by the model) at increments of the independent variable, and each time evaluate the impact of that additional point on reducing the overall uncertainty of the model. Is there another (better) way to do this?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Uncertainmod
  • 1
  • Is your model a straight line? Is it some particular curving shape / nonlinear function? Minimizing the loss function just identifies the parameters (eg, slope & intercept). Are you trying to test if the parameters are equal to some null? – gung - Reinstate Monica Jan 21 '16 at 01:24
  • This is an experiment design problem. I your setting is a lab experiment where you can choose your future runs, that is, your predictor value settings, you could use something like D-optimality, see https://stats.stackexchange.com/questions/114039/why-people-often-optimize-the-determinant-of-x-sigma-x-1/114048#114048 – kjetil b halvorsen Aug 30 '21 at 15:16

2 Answers2

1

ANSWER:
You may want to set a preference for selecting observations at points where your model is currently underperforming (taking a lesson from boosted models).

OVERLY LITERAL ANSWER:
If by 'reduce uncertainty' you mean 'increase confidence' or 'explain a higher proportion of the variance in the sample'... in the case of linear regression, choosing an arbitrarily extreme or high-leverage value will increase the amount of variance explained by your model and increase any significance tests the model might produce. Though doing this means your model becomes highly dependent on the extreme point(s), violates IID, and may not really be trusted.

For example, say you have 5 random observations from a larger population:

       x      y
1 -7.390 -9.380
2 -4.580 -4.610
3 -0.723 -2.400
4  0.827  0.463
5  3.470  6.710

A linear model using least squares with the above 5 points explains ~92% of the variance and has a t-value for x of ~6.9.

Let's say you have the power to substitute one of your observations with an arbitrarily large x value from your population. Say x > |45|. Your new observation might come out to be:

     x      y
6 45.30 41.80

If you now build a model with this point and any four of the previous points, the variance explained will be >98% and the t-value for x will be >15.0. Though this seems to have substantially improved things, for the reasons mentioned previously, this is not really a satisfying answer...

Bryan Shalloway
  • 499
  • 5
  • 11
0

Choose the wide range of independent variable values.

If you're running a regression type of analysis, then the intuition comes from the OLS estimator equation. Here's OLS model in vector notation: $$y=X\beta+\varepsilon$$ It is solved by minimizing residuals, $\bar\varepsilon$, to the following estimator: $$b=(X'X)^{-1}X'y$$ and the variance-covariance matrix of estimators is $$\sigma_b^2=\sigma_\varepsilon^2(X'X)^{-1}$$

Hence, to reduce the variance of estimators (uncertainty) you can either reduce the error ($\sigma_\varepsilon^2$) or increase the variance of explanatory variables. You probably can't reduce the error, so you're left with the second option.

Aksakal
  • 55,939
  • 5
  • 90
  • 176