38

I have seen people have put a lot of efforts on SVM and Kernels, and they look pretty interesting as a starter in Machine Learning. But if we expect that almost-always we could find outperforming solution in terms of (deep) Neural Network, what is the meaning of trying other methods in this era?

Here is my constraint on this topic.

  1. We think of only Supervised-Learnings; Regression, and Classification.
  2. Readability of the Result is not counted; only the Accuracy on the Supervised-Learning Problem counts.
  3. Computational-Cost is not in consideration.
  4. I am not saying that any other methods are useless.
Community
  • 1
Robin
  • 585
  • 1
  • 6
  • 9
  • 3
    Any constraints on the amount of training data available? – Jake Westfall Feb 20 '17 at 00:53
  • 1
    I haven't done it, but I expect you'd have one hell of a hard time training a neural network to do e.g. a singular-value decomposition on a non-trivially-sized matrix (say, rank > 10). – user541686 Feb 20 '17 at 03:21
  • 1
    [Google Translate now uses neural networks](https://research.googleblog.com/2016/11/zero-shot-translation-with-googles.html), and now produces [more curious fails](https://translate.google.co.uk/#pt/en/S%C3%A3o%20Miguel%20das%20Miss%C3%B5es) for a Brazilian town name where a word-for-word use of a dictionary would have been much better – Henry Feb 20 '17 at 15:21
  • I haven't ever heard of deep learning for use of matrix completion (although using matrix completion before deep learning is common practice). You could argue this could be a computational-cost issue, but it's also worth noting that I don't know if all the computers in the world could do deep learning matrix completion with, say, the netflix problem. – Cliff AB Feb 20 '17 at 20:58
  • @CliffAB: (tongue halfway in cheek...) it's probably worth noting that they might not be able to, but I'm not sure it's worth noting that you don't know if they are able to ;) – user541686 Feb 21 '17 at 03:35

4 Answers4

34

Here is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach.

  1. The No Free Lunch Theorem from Wolpert and Macready says

    We have dubbed the associated results NFL theorems because they demonstrate that if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems.

    In other words, no single algorithm rules them all; you've got to benchmark.

    The obvious rebuttal here is that you usually don't care about all possible problems, and deep learning seems to work well on several classes of problems that people do care about (e.g., object recognition), and so it's a reasonable first/only choice for other applications in those domains.

  2. Many of these very deep networks require tons of data, as well as tons of computation, to fit. If you have (say) 500 examples, a twenty layer network is never going to learn well, while it might be possible to fit a much simpler model. There are a surprising number of problems where it's not feasible to collect a ton of data. On the other hand, one might try learning to solve a related problem (where more data is available), use something like transfer learning to adapt it to the specific low-data-availability-task.

  3. Deep neural networks can also have unusual failure modes. There are some papers showing that barely-human-perceptible changes can cause a network to flip from correctly classifying an image to confidently misclassifying it. (See here and the accompanying paper by Szegedy et al.) Other approaches may be more robust against this: there are poisoning attacks against SVMs (e.g., this by Biggio, Nelson, and Laskov), but those happen at train, rather than test time. At the opposite extreme, there are known (but not great) performance bounds for the nearest-neighbor algorithm. In some situations, you might happier with lower overall performance with less chance of catastrophe.

amoeba
  • 93,463
  • 28
  • 275
  • 317
Matt Krause
  • 19,089
  • 3
  • 60
  • 101
  • I agreed everything you say. But the problem is about "computational issues ignored". That means the OP assumes, you'd have infinite samples and infinite computing resources. – SmallChess Feb 20 '17 at 01:51
  • 18
    Infinite computation != infinite samples. For example, I have access to an astonishingly large cluster for processing data. However, the lab experiments we do to actually acquire some of that data are difficult, slow, and time-consuming (on the order of hours to days for a single data point) and all the computation in the world won't help that along. – Matt Krause Feb 20 '17 at 01:54
  • 2
    SVMs with any given feature extractor are likely just as vulnerable to adversarial inputs as CNNs – it's just harder to find them, because we don't have easily available gradients of the feature extraction layers. – Danica Feb 20 '17 at 10:41
  • 1
    A recent and interesting real-life example of the problem @MattKrause gives and an attempt to circumnavigate it using transfer learning is presented in [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://arxiv.org/abs/1610.04286) – HBeel Feb 20 '17 at 11:10
  • @Dougal, I also wonder if it matters that the DNN-feature-extractors are learned, while the SVM ones are (usually) hand-made and correspond to features that humans notice. Part of what makes the panda example so insidious is the imperceptible difference between the adversarial example and the regular one. – Matt Krause Feb 20 '17 at 16:57
  • @MattKrause I don't think that's the case. See e.g. [this blog post](http://karpathy.github.io/2015/03/30/breaking-convnets/) for similar adversarial examples on a linear classifier on 64x64 imagenet, and [this paper](http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=6130416) for a similar study on HoG features (though it doesn't do quite the same thing and is on low-res grayscale images). Goodfellow makes a similar point in [this summary of misconceptions about adversarial examples](http://www.kdnuggets.com/2015/07/deep-learning-adversarial-examples-misconceptions.html). – Danica Feb 20 '17 at 17:18
  • @Dougal, thanks for the links! I think the edits are a little more noticeable for some of the linear examples (e.g., obviously, the colored patches, but also adding the orange splotch to the field in order to push it towards "goldfish") but it's hard to tell. – Matt Krause Feb 20 '17 at 18:37
27

Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems:

  1. Problems where noise is the key feature,
  2. Problems where signal is the key feature.

I remember the explanation that while neural nets thrive in this latter space, traditional statistical methods are often better suited to the former. Analyzing high-res digital photographs of actual things in the world, a place where deep convolutional nets excel, clearly constitutes the latter.

On the other hand, when noise is the dominant feature, for example, in a medical case-control study with 50 cases and 50 controls, traditional statistical methods may be better suited to the problem.

If anybody finds that video, please comment and I'll update.

Ben Ogorek
  • 4,629
  • 1
  • 21
  • 41
  • Excellent answer. Exactly why we turn to deep learning for things we can already do (like recognize images and write text) but may turn to other models for things that may be intuitively difficult. – Mustafa Eisa Feb 21 '17 at 20:45
  • I personally take this answer in my greatest interest. Thank you so much for the answer. – Robin Feb 22 '17 at 09:01
14

Two linearly perfected correlated variables. Can deep-network with 1 million hidden layers and 2 trillion neutrons beat a simple linear regression?

EDITED

In my experience, sample collection is more expensive than computation. I mean, we can just hire some Amazon instances, run deep learning training and then come back a few days later. The cost in my field is about $200 USD. The cost is minimal. My colleagues earn more than that in a day.

Sample collection generally requires domain knowledge and specialized equipments. Deep learning is only suitable for problems with cheap and easy access data set, such as natural language processing, image processing and anything that you can scrape off from the Internet.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
SmallChess
  • 6,764
  • 4
  • 27
  • 48
  • 1
    Of course any MLE method will out-perform deep learning **conditional on the generating model meeting the assumptions of the MLE**. However, this is never happens on real data, or at least for any interesting problems (i.e. not predicting outcome of coin flip). So I think the OP is asking for examples involving real questions of interest with real data. – Cliff AB Feb 20 '17 at 20:48
  • It is a very nice answer. You have offered a very intuitive, and realistic viewpoint. Thank you so much. – Robin Feb 22 '17 at 08:53
-1

To be honest, it is not possible for a deep-learning method to outperform a kernel methods. Why ? It is very simple, because any network, be it deep or shallow, can be described by a kernel. Thus a kernel can reproduce any result coming from deep learning. However, a kernel method have access to other, more powerful methods, than deep learning ones. Indeed, today, kernel machines obtains results that are far better than any deep learning approach.

EDIT : As I received a warning concerning this answer, please let me detail it. I reference this paper : https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3769804.

  1. Every neural network (NN) can be input in a kernel machine, it is known since more than a decade now. For instance, in our kernel machines, we can input any NNs, there is an interface to it, see section 2.3.7.
  2. A kernel machine have access to other methods than NNs. See for instance section 3, that allows any PDE approach.
  3. We benchmarked both approaches. I am sorry, but our kernel machines always retrieved better results than NN ones, as for instance the MNIST test in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3766451
JohnLeM
  • 1
  • 2
  • 2
    Welcome to CV. "Indeed, today, kernel machines obtains results that are far better than any deep learning approach." Could you provide any example? Also, what about the following limitations of kernels: (1) obtaining the pairwise similarity matrix is very expensive (intractable in some cases); (2) a model has to be used downstream (e.g., SVM classifier), in other words, representation and learning are two independent steps (whereas NNs learn custom features as they learn the task). – Antoine Jan 31 '21 at 13:43
  • Antoine, thank you for your answer. Concerning example, I could provide tons of them. However, an emblematic one for machine learning is the MNIST problem. I provide an example of benchmark of neural network vs kernel in our basic tutorial : https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3766451. In particular, you shall find also an answer to your other questions in the same link. – JohnLeM Jan 31 '21 at 13:52
  • 2
    The state of the art on MNIST has been held by deep learning models for a long time: https://paperswithcode.com/sota/image-classification-on-mnist. They significantly outperform SVMs. The benchmarking in your paper has low value: you do not describe the architecture that you use (probably fully connected?), and the scores that you report are ridiculously low. Could you provide any other example? I am afraid the view that "Today, kernel machines obtains results that are far better than any deep learning approach." will be tough to defend. – Antoine Jan 31 '21 at 14:37
  • Finally, I want to add that computing the pairwise similarity matrix is quadratic wrt the nb of samples, so Kernel methods **do not scale well** (if not at all). In today's world where big datasets are everywhere, this is a major disadvantage. On the other hand, NNs scale linearly with the size of the dataset. – Antoine Jan 31 '21 at 14:42
  • Antoine. Did you really understood the test ? It is a benchmark of two standard methods. A standard ANN method, that is the goodle flagship ANN method of Tensorflow, and a standard kernel method, a very straightforward implementation. I agree with you: scores of ANNS are ridicously low. I am afraid that this test is enough for illustration purposes. – JohnLeM Jan 31 '21 at 14:44
  • Antoine, I think that you should really have a closer look to the link that I sent. Pairwise similarity computations are linear in the number of samples, as well as in the size of test set. – JohnLeM Jan 31 '21 at 14:46