Significance of single precision floating point

Question

I've been looking at some of the packages from the High perf task view dealing with GPU computations, and given that most GPU seem to be an order of magnitude stronger at performing single precision arithmetics than DP ones, I was wondering:

1. why none of the packages gives more control to the user on the type of precision required? I can see many applications in statistics where SP arithmetics (i.e., number coded with 7 digit accuracy) are good enough for practical use (if I am overestimating the gains involved, let me know).
2. is Python more flexible on this? If so, why? I don't see why the absence of a 'single' type in R would make including such an option (together with a warning) in say GPUtools or magma impossible (though I'll be happy to be shown wrong).

PS: I'm specifically thinking of applications where the numbers are already dimension-wise scaled and centered (so that Chebychev's inequality is binding) dimension-wise.

Answer 1

1. Because before GPUs there was no practical sense of using single reals; you never have too much accuracy and memory is usually not a problem. And supporting only doubles made R design simpler. (Although R supports reading/writing single reals.)
2. Yes, because Python is aimed to be more compatible with compiled languages. Yet you are right that it is possible for R libraries' wrappers to do in-fly conversion (this of course takes time but this is a minor problem); you can try e-mailing GPU packages' maintainers requesting such changes.

Answer 2

From the GPUtools help file, it seems that useSingle=TRUE is the default for the functions.

Answer 3

I presume that by GPU programming, you mean programming nvidia cards? In which case the underlying code calls from R and python are to C/CUDA.

---

The simple reason that only single precision is offered is because that is what most GPU cards support.

However, the new nvidia Fermi architecture does support double precision. If you bought a nvidia graphics card this year, then it's probably a Fermi. Even here things aren't simple:

• You get a slight performance hit if you compile with double precision (a factor of two if I remember correctly).
• On the cheaper cards Fermi cards, nvidia intentionally disabled double precision. However, it is possible to get round this and run double precision programs. I managed to do this on my GeForce GTX 465 under linux.

To answer the question in your title, "Is single precision OK?", it depends on your application (sorry crap answer!). I suppose everyone now uses double precision because it no longer gives a performance hit.

When I dabbled with GPUs, programming suddenly became far more complicated. You have to worry about things like:

• warpsize and arranging your memory properly.
• #threads per kernel.
• debugging is horrible - there's no print statement in the GPU kernel statements
• lack of random number generators
• Single precision.

Answer 4

OK, a new answer to an old question but even more relevant now. The question you're asking has to do with finite precision, normally the domain of signal analysis and experimental mathematics.

Double precision (DP) floats let us pretend that finite precision problems don't exist, the same as we do with most real-world mathematical problems. In experimental math there is no pretending.

Single precision (SP) floats force us to consider quantization noise. If our machine learning models inherently reject noise, such as neural nets (NN), convolutional nets (CNN), residual nets (ResN), etc, then SP most often gives similar results to DP.

Half precision (HP) floats (now supported in cuda toolkit 7.5) require that quantization effects (noise and rounding) be considered. Most likely we'll soon see HP floats in the common machine learning toolkits.

There is recent work to create lower precision computations in floats as well as fixed precision numbers. Stochastic rounding has enabled convergence to procede with CNNs whereas the solution diverges without it. These papers will help you to improve your understanding of the problems with the use of finite precision numbers in machine learning.

To address your questions:

SP is not so bad. As you point out it's twice as fast, but it also allows you to put more layers into memory. A bonus is in saving overhead getting data on and off the gpu. The faster computations and the lower overhead result in lower convergence times. That said, HP, for some problems, will be better in some parts of the network and not in others.

1. It seems to me that many of the machine learning toolkits handle SPs and DPs. Perhaps someone else with a wider range of experience with the toolkits will add their nickle.
2. Python will support what the gpu toolkit supports. You don't want to use python data types because then you'll be running an interpreted script on the cpu.

Note that the trend in neural networks now is to go with very deep layers, with runs of more than a few days common on the fastest gpu clusters.

Answer 5

The vast majority of GPUs in circulation only support single precision floating point.

As far as the title question, you need to look at the data you'll be handling to determine if single precision is enough for you. Often, you'll find that singles are perfectly acceptable for >90% of the data you handle, but will fail spectacularly for that last 10%; unless you have an easy way of determining whether your particular data set will fail or not, you're stuck using double precision for everything.