9

I'm an experimental physicist by training and have used standard statistical methods to analyze data, and the design of experiments (DOE) framework to develop models of systems by varying inputs and measurement outputs.

Recently, I've been looking into the use of machine learning and I'm trying to figure if there's any utility/benefit over DOE.

I'm hoping someone on this forum can either validate the way I'm thinking about supervised machine learning, or point out what I'm missing.

I've basically come to the conclusion that supervised machine learning is a method to compute a transfer function of a system given the training data is a set of data that connects the set of inputs with what the output truth should be.

Notwithstanding the machinery that figures out the transfer function based on the training set, what is the difference between DOE and supervised machine learning in terms of the accuracy or other performance measure of the transfer function?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
JQK
  • 145
  • 2
  • 12
  • 3
    The basic idea behind design of experiments is that, by systematically manipulating the values of several variables of interest while controlling all other variables (by holding them constant or randomizing), our statistical models will give us *causal* knowledge of the relationship between the manipulated and dependent variables. Machine learning methods are focused purely on making accurate predictions, using whatever data are available and without taking into account how they were produced. – Dan Hicks Dec 18 '17 at 02:04
  • 1
    Most machine learning "basis engines", such as neural networks, svm, or CART ensembles, are universal function aproximators. My experience with DOE, even RSM, is that the bases are fairly limited and are often polynomial. As a physics guy, you can dislike that, and I think your "transfer function" is looking toward frequency domain models in that sense. Think multivariate, high-interactions, and perhaps neural model-adaptive control. I dispute the "prediction-only" assertion, but accede prediction-dominant use in the market. – EngrStudent Dec 18 '17 at 02:23
  • sorry if I missed it, but what exactly is the main difference that is confusing you? – Charlie Parker Dec 18 '17 at 02:27
  • Pinocchio What I'm trying to understand is whether or not there is a difference between DOE and supervised machine learning. From the previous comment it would appear that the only difference is the set of basis functions that the transfer function is decomposed upon. Do you agree? – JQK Dec 18 '17 at 12:38
  • @JQK I won't lie to you, but I am not familiar (yet) with DOE. I plan to read it in the next days...but supervised learning can be done with any basis functions. Supervised learning is just a paradigm of ML for learning a target function when samples of x and f(x) are known (i.e. the right labels). – Charlie Parker Dec 19 '17 at 02:18
  • @Pinocchio - DOE does the same thing, you provide x and f(x) and it gives you a model for the system that mapped x to f(x). If it turns out that that machine learning provides for a more general set of basis functions that would be great to know. This, of course, would beg the question, are those basis functions in some way superior to the basis functions used in DOE? – JQK Dec 19 '17 at 14:42

4 Answers4

6

Your question is difficult to answer because there is no "supervised ML algorithm". There are a large number of different ML algorithms that can be optimized in a supervised fashion, each with their strengths and weaknesses.

On a very abstract level, you can define Machine Learning (ML) as a search through some space $P$ for a parameterization ($\theta$) of a given model $M$ such that $M(x;\theta)$ gives a minimal value (though not always globally) for the cost function ($\mathcal{C}$) and input ($x$). More formally:

$$\arg\min_{\theta\in P} \mathcal{C}(M(\theta))$$

For supervised learning, one form the cost function can take is (given $y$ as ground truth):

$$\mathcal{C}(M(x; \theta)) = ||M(x;\theta) - y||_2$$

Any search through $P$ that minimizes the cost function can fit in this framework, and thus you can claim DOE as a ML algorithm if you want. Specifically, an ML algorithm is defined by: the optimization technique employed, the model used, and your cost function. If you fill those in for DOE, you can start to compare it against other ML algorithms.

Ranon
  • 328
  • 1
  • 5
  • My question relates to the output of the ML algorithm and not the method that produced it. This, of course, presumes that any supervised ML algorithm creates the similar functionality. That said, my friend Abe tells me that the output of the ML algo is, in fact, a model of the transfer function that maps x to f(x); however, the basis set can be very complicated and isn't necessarily a linear combination of orthonormal functions. – JQK Dec 19 '17 at 18:28
  • That's correct. It's generally considered ML if you optimize your model to some data giving you an M_{opt} (M_{opt}(x) = M(x; \theta), there \theta are the parameters you've optimized over). – Ranon Dec 21 '17 at 16:14
5

Substitute "supervised ML" with "regression analysis", and you'll see that your question is difficult to answer. Depending on how narrowly you define regression analysis, DOE could be part of the term.

Suppose, you're planning to use regression analysis to determine efficiency of a certain herbicide in the field. One of the very first step would be to design the experiment. In this case I'd argue that DOE is inseparable from your regression analysis, it's part of it.

In many applications of ML you're given the data, and can't do much about planning the experiments, in fact, there is no experiment. For instance, in Kaggle competitions everything's usually set in advance, the training dataset is pre-defined and given, the test data set will be given at evaluation step, and you can't do anything about these things. That is why DOE is not mentioned al ot in the field.

However, it doesn't have to be this way. Suppose, you're building self-driving cars. Yes, as usual, you can employ all the available datasets to train your ML vision components, all the images with traffic signs and road situations, nicely tagged etc. No DOE involved here, really. Yet, once you're out of the initial phase, and get into the field tests, things change. All the typical concerns that DOE addresses show up, e.g. would you need 10 test cars or 1,000 to get reliable results? Would you need 1,000 miles or 1,000,000 miles on the empty roads before trying the test car on a real street? etc.

ML may not necessarily call DOE what they do to plan the development, but in essence it is DOE. Therefore, the answer to your question lies in how narrowly or broadly you define ML term. Is it just fitting the function to the data using the cost (loss)? Or is it a more general definition of building the reliable machine that replaces humans, which would include more than just fitting/optimization but at least some aspects of DOE.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
2

I just finished a graduate-level course in experimental design and am starting to learn machine learning... I bet there are people on this website who can answer this better than I can, but hopefully this answer will do.

At their cores, experimental design (ED) and machine learning (ML) have different goals. The primary goal of ED is to assess the influences of treatments and, if applicable, compare the influences of different treatments. The primary goal of ML is to give accurate predictions.

These different cores thus influence how each topic is developed:

  • In ED, emphasis is placed on good design so that the variability (of treatment parameter estimates) is reduced, sometimes with the need to meet budgetary constraints. My ED professor once said something to the tune of "Statisticians are always criticized for demanding a large sample size. If you were a statistician working on missiles, firing a missile is a few million bucks down the drain." Fractional factorial designs, from what I understand, are particularly popular due to budgetary constraints. The primary goal of ED is statistical inference of treatment parameters.
  • In ML, emphasis is placed on using predictive algorithms and the issues behind them (e.g., computational compexity, computer software/hardware issues, etc.).

Given my experience in both, I would not try to compare the two subjects. It is like comparing apples to oranges.

Clarinetist
  • 3,761
  • 3
  • 25
  • 70
1

I think that there is a fundamental problem in the way the question is posed. DoE does not per se model a system, the scope of DoE is to efficiently excite the system to gain maximum amount of information with a limited set of resources (time and experiments).

Machine Learning(ML) supervised or unsupervised on the other and is a modelling technique that aims at finding relations within huge amount of observational data already collected. Much like simple regression is the step after you collected the data.

There might be some similarities between DoE and Re-inforced Learning methods, were the ML itself is able to move paramters in the system in order to find an optimal fit or minimize a predifined objective function. In that context the ML model is performin some kind of DoE to gain more information and improve itself.

Mattia Vallerio
  • 11
  • 1
  • While it’s true that DOE excited the system, when it does so you’re creating a data set (output, input) that represents the transfer function of the system that is being excited. Isn’t this measured transfer function similar to supervised learning? – JQK Jun 20 '21 at 11:59