Is it correct to say the Neural Networks are an alternative way of performing Maximum Likelihood Estimation? if not, why?

Question

We often say that minimizing the (negative) cross-entropy error is the same as maximizing the likelihood. So can we say that NN are just an alternative way of performing Maximum Likelihood Estimation? if not, why?

Answer 1

There seems to be a misunderstanding concerning the actual question behind. There are two questions that OP possibly wants to ask:

1. Given a fixed other parametrized model class that are formulated in a probabilistic way, can we somehow use NNs to very concretely optimize the Likelihood of the parameters? Then as @Cliff AB posted: This seems strange and unnatural for me. NNs are there for approximizing functions. However, I strongly believe that this was not the question.

2. Given a concrete dataset consisting of 'real' answers $y^{(i)}$ and real $d$-dimensional data vectors $x^{(i)} = (x^{(i)}_1, ..., x^{(i)}_d)$, and given a fixed architecture of a NN, we can use the cross entropy function in order to find the best parameters. Question: Is this the same as maximizing the likelihood of some probabilistic model (this is the question in the post linked in the comments by @Sycorax).

Since the answer in the linked thread is also somewhat missing insight let me try to answer that again. We are going to consider the following very simple neural network with just one node and sigmoid activation function (and no bias term), i.e. the weights $w = (w_1, ..., w_d)$ are the parameters and the function is: $$f_w(x) = \sigma\left(\sum_{j=1}^d w_j x_j\right)$$ The cross entropy loss function is $$l(\hat{y}, y) = -[y \log(\hat{y}) + (1-y) \log(1-\hat{y})] $$

So given the dataset $y^{(i)}, x^{(i)}$ as above, we form $$\sum_{i=1}^n l(y^{(i)}, f_w(x^{(i)}))$$ and minimize that in order to find the parameters $w$ for the neural network. Let us put that aside for a moment and go for a completely different model.

We assume that there are random variables $(X^{(i)}, Y^{(i)})_{i=1,...,n}$ such that $(X^{(i)}, Y^{(i)})$ are iid. and such that $$P[Y^{(i)}=1|X^{(i)}=x^{(i)}] = f_w(x^{(i)})$$ where again, $\theta=w=(w_1,...,w_d)$ are the parameters of the model. Let us setup the likelihood: Put $Y = (Y^{(1)}, ..., Y^{(n)})$ and $X = (X^{(1)}, ..., X^{(n)})$ and $y = (y^{(1)}, ..., y^{(n)})$ and $x = (x^{(1)}, ..., x^{(n)})$. Since the $Z^{(i)} = (X^{(i)}, Y^{(i)})$ are independent, \begin{align*} P[Y=y|X=x] &= \prod_{i=1}^n P[Y^{(i)}=y^{(i)}|X^{(i)}=x^{(i)}] \\ &= \prod_{\{i : y^{(i)}=1\}} P[Y^{(i)}=1|X^{(i)}=x^{(i)}] \prod_{\{i:y^{(i)}=0\}} (1 - P[Y^{(i)}=1|X^{(i)}=x^{(i)}]) \\ &= \prod_{\{i : y^{(i)}=1\}} f_w(x^{(i)}) \prod_{\{i:y^{(i)}=0\}} (1 - f_w(x^{(i)})) \\ &= \prod_{i=1}^n \left(f_w(x^{(i)})\right)^{y^{(i)}} \left(1 - f_w(x^{(i)})\right)^{1 - y^{(i)}} \end{align*} So this is the likelihood. We would need to maximize that, i.e. most probably we need to compute some gradients of that expression with respect to $w$. Uuuh, there is an ugly product in front... The rule $(fg)' = f'g + fg'$ does not look very appealing. Hence we do the following (usual) trick: We do not maximize the likelihood but we compute the log of it and maximize this instead. For technical reasons we actually compute $-\log(\text{likelihood})$ and minimize that... So let us compute $-\log(\text{likelihood})$: Using $\log(ab) = \log(a) + \log(b)$ and $\log(a^b) = b\log(a)$ we obtain

\begin{align*} -\log(\text{likelihood}) &= -\log \left( \prod_{i=1}^n \left(f_w(x^{(i)})\right)^{y^{(i)}} \left(1 - f_w(x^{(i)})\right)^{1 - y^{(i)}} \right) \\ &= - \sum_{i=1}^n y^{(i)} \log(f_w(x^{(i)})) + (1-y^{(i)}) \log(1-f_w(x^{(i)})) \end{align*}

and if you now compare carefully to the NN model above you will see that this is actually nothing else than $\sum_{i=1}^n l(y^{(i)}, f_w(x^{(i)}))$.

So yes, in this case these two concepts (maximizing a likelihood of a probabilistic model and minimizing the loss function w.r.t. a model parameter) actually coincide. This is a more general pattern that occurs with other models as well. The connection is always

$$-\log(\text{likelihood}) = \text{loss function}$$ and $$e^{-\text{loss function}} = \text{likelihood}$$

In that sense, statistics and machine learning are the same thing, just reformulated in a quirky way. Another example would be linear regression: There also exists a precise mathematical description of the probabilistic model behind it, see for example Likelihood in Linear Regression.

Notice that it may be pretty hard to figure out a natural explanation for the probabilistic version of a model. For example: in case of SVMs, the probabilistic description seems to be Gaussian Processes: see here.

The case above however was simple and what I have basically shown you is logistic regression (because a NN with one node and sigmoid output function is exactly logistic regression!). It may be a lot harder to interpret complicated architectures (with tweaks like CNNs etc) as a probabilistic model.

Answer 2

In abstract terms, neural networks are models, or if you prefer, functions with unknown parameters, where we try to learn the parameter by minimizing loss function (not just cross entropy, there are many other possibilities). In general, minimizing loss is in most cases equivalent to maximizing some likelihood function, but as discussed in this thread, it's not that simple.

You cannot say that they are equivalent, because minimizing loss, or maximizing likelihood is a method of finding the parameters, while neural network is the function defined in terms of those parameters.

Answer 3

These are fairly orthogonal topics.

Neural networks are a type of model which has a very large number of parameters. Maximum Likelihood Estimation is a very common method for estimating parameters from a given model and data. Typically, a model will allow you to compute a likelihood function from a model, data and parameter values. Since we don't know what the actual parameter values are, one way of estimating them is to use the value that maximizes the given likelihood. Neural networks are our model, maximum likelihood estimation is one method for estimating the parameters of our model.

One slightly technical note is that often, Maximum Likelihood Estimation is not exactly used in Neural Networks. That is, there are a lot of regularization methods used that imply we're not actually maximizing a likelihood function. These include:

(1) Penalized maximum likelihood. This one is a bit of a cop-out, as it doesn't actually take too much effort to think of Penalized likelihoods as actually just a different likelihood (i.e., one with priors) that one is maximizing.

(2) Random drop out. In especially a lot of the newer architectures, parameter values will randomly be set to 0 during training. This procedure is more definitely outside the realm of maximum likelihood estimation.

(3) Early stopping. It's not the most popular method at all, but one way to prevent overfitting is just to stop the optimization algorithm before it converges. Again, this is technically not maximum likelihood estimation, it's really just an ad-hoc solution to overfitting.

(4) Bayesian methods, probably the most common alternative to Maximum Likelihood Estimation in the statistics world, are also used for estimating the parameter values of a neural network. However, this is often too computationally intensive for large networks.