1

There is a statement in this quora answer:

Layer depth is usually a power of 2 because it is convenient for the GPU.

Also, in fully connected layers number of neurons in every hidden layer corresponds to a power of 2.

But, why is the power of 2 convenient for GPU? If I have everything for the power of 3, why would it be inconvenient for GPU? I thought one would use GPU in deep learning only because one can parallelize batches/or convolution computation.

Alina
  • 915
  • 2
  • 10
  • 21
  • 1
    Is your computer a binary computer or a ternary one? – whuber Dec 07 '18 at 23:05
  • How do I understand it? Each GPU will have 2 or 3 cores? – Alina Dec 07 '18 at 23:19
  • It's more basic than that: ultimately, data are represented on your computer in a unit like a [bit.](https://en.wikipedia.org/wiki/Bit) How many states do the bits of your computer have? Two? Three? Ten? – whuber Dec 08 '18 at 14:58
  • Read the answer by Dmitriy Genzel in Quora for a similar question: https://www.quora.com/Why-do-people-tend-to-use-powers-of-2-when-specifying-the-number-of-nodes-in-a-layer-of-a-neural-net-and-for-the-size-of-mini-batches – Hari Sep 01 '20 at 11:12

0 Answers0