0

Curve fitting, using conventional measures of error, such as LSE (Least Squares Estimation), cannot include the complexity of the formula because the complexity of the formula must be measured in bits and is therefore incommensurable. However, if one can measure the total error in units of bits, it can be added to the complexity of the formula in bits to approximate the Kolmogorov Complexity of the data which, in turn, provides approximate Solomonoff Induction.

There must be some measure of error in units of bits. What is it?

James Bowery
  • 261
  • 1
  • 8
  • LSE can be interpreted as using a cross entropy loss and a normal distribution assumption, so it is already is already in bits. – Neil G Dec 23 '17 at 15:54
  • So the sum of formula complexity and squared error of its predictions could be a model selection criterion. Is there a name for such a selection criterion so I can look up the literature on it? – James Bowery Dec 23 '17 at 16:34
  • If you haven't already, you should probably look at Rissanen's work on stochastic complexity. – Neil G Dec 23 '17 at 16:38
  • Yes, models selection using AIC as below in my answer below. Also, BIC. – Carl Dec 23 '17 at 16:40

1 Answers1

4

Your bits imply Shannon entropy, a type of self-information, is applied in the context of Kullback-Leibler Information to generate AIC (Akaike Information Criterion) values under the assumption of maximum likelihood conditions. Those AIC values are then used for regression model selection under certain conditions.

Carl
  • 11,532
  • 7
  • 45
  • 102
  • 1
    +1 One of the most efficient, simple, and well annotated answers I've seen on this site. – EdM Dec 23 '17 at 15:25
  • @EdM I'm blushing. Took me a while to understand AIC, I guess the work paid off. – Carl Dec 23 '17 at 15:32