4

Most of the recent famous methods coming out of the machine learning, are supervised learning methods like Decision Trees, Random Forests, Deep Learning, SVMs.

The more traditional supervised learning methods, like linear and logistic regression, with or without regularization, have had a long history of analysis of their nuances (eg assumptions for reliable use like normality, confidence intervals, hypothesis tests, optimal estimators).

Though the traditional stats models and the more modern ML ones come out of different disciplines (for statistics theoretically associated with mathematics departments and practically agronomy, medicine, social science, and econometrics, and for machine learning out of computer science with applications in vision, NLP, and AI), they have the same ends.

It seems like the ML models, as wildly successful as they seem, also seem to have very little theoretical support.

In contrast, linear regression can have a p-value analysis of each variable, F-test for the entire fit, has (the classic five assumptions). I've never seen such analysis of the more complicated ML tests.

There doesn't seem to be a treatment of machine learning models with the rigor of analysis of the statistical models. http://www.fharrell.com/post/stat-ml/

Is there any attempt to apply classic statistical analysis techniques to assessing the newer ML regression models?

Mitch
  • 1,691
  • 2
  • 18
  • 33
  • 1
    There are four standard regression assumptions. The author mentions no or low multicollinearity which is not an assumption for regression, although people commonly and incorrectly say it is an assumption. I would re evaluate that reference. – LSC Apr 21 '19 at 17:12
  • 1
    I am not well-versed to answer that but in contrast to Asymptotic theory that usually pertains statistics I would say that ML is attacking a lot of its methods through generalisation bounds. – usεr11852 Apr 21 '19 at 17:23
  • I think I've added some detail to address the close voters. Surely part of the cause of lack of clarity on my part is lack of knowledge. My motivation for this question is that I feel like success in the complicated methods of ML is offset by lack of statistical rigor (and dually the great rigor in statistics is offset by lack of progress in new more successful methods). Or is it just that historically, methods were devised first and justifications and analysis came way later, and that is just as much the case for Random Forests now as it was for Logistic Regression in the thirties? – Mitch Apr 21 '19 at 18:39
  • 2
    I've read your edit and comments but it's not clear what properties of machine learning you want proved. It seems you wish to reason by analogy about $p$-values in some manner. But what would that mean for a random forest model? A $p$-value for regression coefs tests the hypothesis that the coefficient is not statistically different from zero. Random forests don't estimate a coefficient for each variable, and it's not clear what hypothesis you are interested in testing. "Machine learning" usually cares more about making good predictions, which is why @usεr11852 mentions generalization bounds. – Sycorax Apr 21 '19 at 18:51
  • We have this related thread, which might be of interest https://stats.stackexchange.com/questions/321851/what-are-the-main-theorems-in-machine-deep-learning – Sycorax Apr 24 '19 at 16:38
  • @Sycorax: Somehow I think that theoretical ML people care more for "*concentration inequalities*" exactly because the focus more in "bounds" but this is a highly subjective assessment. – usεr11852 Apr 28 '19 at 21:19
  • You seem to be asking two things: 1) "are we applying stats techniques to assess ML", and 2) "ML seems to have little theoretical support/basis". #1 is very unclear and so you will need to edit this to clarify what you mean. #2 is clearer but likely a duplicate of the question linked by Sycorax above. – mkt Apr 29 '19 at 05:13

1 Answers1

6

I guess the main part of an answer depends on what, precisely, you mean by "classical statistical analysis" but if we interpret it broadly to mean applying theorems and results from probability and statistics, then we can come up with a good bibliography.

Three references off the top of my head:

Aside: It's worth remarking that the difference between machine learning and statistics has more to do with marketing rather than any underlying mathematical principles.

For example, random forests were first proposed by Leo Breiman, who was a statistics professor at University of California, Berkeley.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • Thank you for those references. I am well aware of the first two. I suppose I wasn't clear about what I don't think has been done for DL, SVM, RF, for example what kind of distribution assumptions are necessary for robust results. – Mitch Apr 21 '19 at 17:00
  • 1
    Perhaps you could edit your question to clarify what you want to know in specific terms and why the resources you've consulted don't answer your question. – Sycorax Apr 21 '19 at 17:02
  • 2
    To answer your question -- It's rarely necessary to assume a specific distribution (e.g. normal) for the data. The whole point of these advanced methods is that they're robust to a wide variety of input data types -- so in a sense they are "generic" to a very large class of problems. For example, the chapter in *ESL* about random forest demonstrates that it has some nice properties without positing a particular distribution for the inputs. Probably the biggest assumption is that the input data is *iid*, with *independence* of inputs being the biggest potential problem. – Sycorax Apr 21 '19 at 17:12
  • 4
    This is a very accurate point that a lot of the “difference” is marketing a hot “new field” or that many of the “ML” methods are statistical methods with the appropriate brakes disabled or ignored. – LSC Apr 21 '19 at 17:18