25

I received preliminary materials on deep learning in my class. It was written as follows. This raised me the question of the basic meaning of the word deep learning.

Deep learning is a machine learning method using a multi-layer neural network.

  1. Is a neural network essential for deep learning?
  2. Isn't it possible to do deep learning without a neural network by using PCA? (Example: PCANet)

I'm confused by similar terms like deep learning and deep neural networks. I have a month to start school and I can't contact my teacher during that time. I would appreciate it if you could tell me.

Arya McCarthy
  • 6,390
  • 1
  • 16
  • 47
desertpureolive
  • 359
  • 2
  • 3
  • 4
    I think this question is almost impossible to answer, due to [this one](https://stats.stackexchange.com/q/362425/60613). Anything that's a deep hierarchical computational graph that is modeling something will be called a deep neural network, by one account or another. It does not even need to be differentiable to get that treatment. – Firebug Apr 04 '21 at 15:04

5 Answers5

29

This is a good question.

Is a neural network essential for deep learning?

Yes, your teacher provided you with a correct definition of deep learning. You can still do machine learning (a broader category) without neural networks, but you need a neural network for it to qualify as 'deep learning'.

Isn't it possible to do deep learning without a neural network by using PCA? (Example: PCANet)

Based on the answer to the last part, no. It by definition wouldn't be 'deep' anymore. PCANet is actually a neural network, by the way.

PCA, on the other hand, isn't 'deep'. If you stack several layers of PCA on top of each other, then there is an equivalent single-layer PCA you could have done, because composing those linear transforms will just give you another linear transform.

Arya McCarthy
  • 6,390
  • 1
  • 16
  • 47
  • 9
    There is a technique called 'deep forests' that stacks random-forest classifiers, and which would have a reasonable claim to be 'deep learning': https://arxiv.org/abs/1702.08835 – Thomas Lumley Apr 03 '21 at 05:54
  • 2
    But what for @ThomasLumley? To bring more hype to using random forest? – Tim Apr 03 '21 at 15:41
  • 4
    @Tim One could just as well ask "what for" about "deep neural networks". What makes "deep" neural networks substantively different from non-deep ones to the extent it justifies a special label? Is it just hype, or is there something else? -- My understanding is that "deep forests" are predicated on the opinion that there _is_ something to "deep", and are an attempt to see if those same qualities can also be implemented with with a decision tree based methods. Whether they succeed or not :shrug:. – R.M. Apr 03 '21 at 17:54
  • 2
    "What makes 'deep' networks substantively different from non-deep ones?" For starters, the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem). More info in [these lecture slides](https://deeplearning.cs.cmu.edu/F20/document/slides/lec2.universal.pdf). – Arya McCarthy Apr 03 '21 at 18:02
  • 2
    @Tim partly to find out what properties of deep neural networks are due to the 'deep' aspect, and partly because they wanted something like a neural network that didn't have to be continuous – Thomas Lumley Apr 03 '21 at 22:26
  • 5
    @AryaMcCarthy Universal approximation theorems apply to both "shallow" (one hidden layer) and deep neural networks, so the UAT cannot explain why the two should be substantively different. One important difference (which is mentioned in your link) is that the number of nodes required to approximate certain functions is exponentially larger for shallow networks compared to deep networks. – Eric Perkerson Apr 05 '21 at 00:32
  • you can reach "universal approximation" with many other machine learning methods too, anyway. not just neural networks. – carlo Apr 05 '21 at 13:24
  • 1
    @R.M. The term is mostly hype, but the techniques aren't. It *is* true that large neural networks can be much more effective than small ones, but there's no meaningful distinction between "deep learning" and "neural nets with lots of layers". (Although I wouldn't be terribly surprised if the meaning evolves to cover other large models that work at multiple levels of feature granularity.) – Ray Apr 05 '21 at 19:52
19

I'm going to disagree with the other answers. Fundamentally, I would say that deep learning is defined by a hierarchy of learned representations, and not by which particular model is used to define these representations. Indeed, this is how Goodfellow et al define it in the introductory section of their text Deep Learning (neural networks are not mentioned until later).

In other words, the key with deep learning is that we are effectively learning a series of transformations of our data. Typically, these transformations define a neural network, with the activations of each layer serving as the transformed input data. However, this need not be the case. Deep Gaussian processes, for instance, have been gaining some attention in the research community.

However, if I were teaching an introductory class, I would feel perfectly comfortable using your professor's definition: in practice, people overwhelmingly use neural networks for deep learning.

You can do deep learning with basically any nonlinear model. PCA, being linear, does not qualify (though nonlinear analogues of PCA, such as kernel PCA, can qualify, see this article).

John Madden
  • 1,752
  • 9
  • 22
  • I am a bit torn with this is answer because it is a very good point but probably easy to misinterpret. Even in the example given the whole idea is that we have "layers" and for all intended purposes, "Deep GPs" could be called "Stacked GPs". But we would not really call "stacking" (in the context of ML) an application of Deep Learning because it would be misleading. The concept of "Depth" is pretty loosely defined. The use of the term "Deep" carries some buzz around it. I don't think this post disagrees with the other answers but rather that it critically complements them... You get my +1. – usεr11852 Apr 03 '21 at 22:38
  • 1
    Agreed, multi-layer Bayesian networks could also qualify as "deep". – madprogramer Apr 04 '21 at 15:07
  • Stacked normalizing flows could be considered deep as well. So you do not necessarily need a neural network, though it is the most convinient and commonly used. – Marcel Braasch Jun 01 '21 at 11:40
6

This answer depends on the definition of artificial neural network (ANN) you take to be true. See my question here: What *is* an Artificial Neural Network?. Therefore, no objective answer can be given since:

  1. To accommodate all definitions of ANNs, they are simply defined as arbitrary computational graphs, with tunable parameters (even if they are not tuned, random neural networks are still ANNs)
  2. Differentiability is not required (for example, genetic algorithms and other metaheuristics have been used to learn ANNs)

I will however, propose a counterfactual approach here that, in my experience, at least makes some people to conclude that deep learning is not solely based on ANNs:

  1. Do you consider hierarchical hidden markov models (HHMM)/hierarchical Bayesian networks deep learning?
  2. Do you consider them to be neural networks?

If you answered 'yes' and 'no', then the overall answer is definitely 'no', for you there is a deep learning model that is not a neural network. Other combinations of answer will lead to undefined overall answers.

Firebug
  • 15,262
  • 5
  • 60
  • 127
4

Deep learning is machine learning done using “deep” neural networks, i.e. such that have multiple (>2) layers. So you cannot do it without neural networks. For using other kinds of machine learning just use “machine learning” term, that includes neural networks as well.

Tim
  • 108,699
  • 20
  • 212
  • 390
  • 1
    By definition, then, deep learning requires neural networks. However, a neural network is a fairly straightforward thing to implement in software and there's nothing magic about it. It would also be straightforward to implement something different in software that performed a similar job, and we could have a lively debate as to whether it was a neural network or not :-) In other words, deep learning *only* requires neural networks because that's the definition of the term, not because that behaviour can only be achieved using neural networks. – Frog Apr 04 '21 at 01:46
  • @Frog No reason to re-use the “deep learning” name for something else if there’s already a term for it. By same logic, NNs are general approximators, so a non-deep NN would be considered as “deep” if it approximated the deep one? – Tim Apr 04 '21 at 09:41
0

Deep learning is a subset of machine learning, which is a field dedicated to the study and development of machines that can learn, and the goal is of deep learning is to achieve eventually attain general artificial intelligence. Neural network is just one of the biological inspired model. In simple terms I would define deep learning a subset method in machine learning primarily using artificial neural networks which are inspired by human brain. See the following diagram.Artificial Intelligence Figure

Classical machine learning uses classical statistics/mathematical modelling while deep learning use biological models aka neural networks.

Hamza
  • 25
  • 1