5

There are lots of machine learning toolkits nowadays, such as weka, sklearn, R libs. If we choose to use these toolkits, besides that it is convenient, sometimes we might lose control of what is really happening. For example, in some learning methods the value of features should be scaled to $[0,1]$, but there may be lots of different methods which can achieve this. When we use a scaling function provided by a specific toolkit, we just don't know what's really happening, and this will cause trouble, say, when we want to check if different scaling would affect the performance of learning algorithms.

If the learning system is entirely implemented by ourselves, there could not be that much confusion, since we know exactly what happens. So what is the right attitude toward open source machine learning toolkits? Just use them for the first phase, and then implement manually without the help of learning toolkits in order to have more control over algorithms, or can we just rely on such kinds of learning toolkits?

Nick Stauner
  • 11,558
  • 5
  • 47
  • 105
crazyminer
  • 79
  • 5
  • My view is that good OS libraries should have sane, override-able, defaults. Scikit-Learn is a good example of this (Weka is poorer example). That way the programs are explicit only when required. To program your own just to have more control can be overkill in 90% of cases -- better is to fork to OS library and added in the pieces you want. – Cam.Davidson.Pilon Mar 29 '14 at 20:37

1 Answers1

6

Related answer: https://stats.stackexchange.com/a/66420/25433

If we choose to use these toolkits, besides that it is convenient, sometimes we might lose control of what is really happening.

This is not at all specific to open source toolkits. In fact, in open source programs it's easier to make changes where and when necessary. Additionally, if you add fundamental functionality you can become a contributor for the package to get a win-win situation.

If the learning system is entirely implemented by ourselves, there could not be that much confusion, since we know exactly what happens.

Probably. On the other hand, implementing from scratch takes a lot of knowledge, time and effort, especially if you want to do proper unit testing and what not (which most popular open source software solutions do provide). Also note that machine learning practitioners are often not software engineers, which makes (re)implementation far from trivial.

So what is the right attitude towards open source machine learning toolkits Just use them for the first phase, and then implement manually, without the help of learning toolkits in order to have more control over algorithms, or we can just rely on such kinds of learning toolkits?

The most important thing in my opinion is to be aware of what exists (for example here and here). More often than not, you can find what you need in open source (maybe after some fiddling here and there). Don't reinvent the hot water.

Never assume that open source software (or any software for that matter) is bug-free. If you get weird results, question the tools you used. The larger the userbase of a given package, the safer you are from critical bugs in the software. Users are inevitably the best testing bed for any software system. You can assume to be fairly safe from major issues in popular packages like weka, shogun, opencv, ...

This is another related question that might be particularly relevant for machine learning. A lot of academic open-source software is released as a proof-of-concept. Such implementations -- often at the cutting edge of algorithms with fairly few users -- should be treated with care. The smaller the userbase, the less scrutiny software has received.

Community
  • 1
Marc Claesen
  • 17,399
  • 1
  • 49
  • 70