2

I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my resources are not fully used (CPU and RAM) ? Thank you

Sycorax
  • 76,417
  • 20
  • 189
  • 313
ob21
  • 21
  • 1
  • 1
    I'm not sure this is more thoroughly-developed than guess & check – Sycorax May 07 '18 at 23:20
  • 1 is a bit small. usually somewhere between 8 and 128 is reasonable. – shimao May 07 '18 at 23:59
  • Value 1 will give you very erratic gradients. Might take too long time to converge (if ever) to a useful local maximum. Too large batch might be taxing computationally (and memory-wise on a GPU) and also lead to overfitting. I believe there were works showing that SGD with a good batch size pick acts as a regularizer itself! – Vladislavs Dovgalecs May 08 '18 at 00:08
  • Probably has an answer already in this thread: https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network. Also, training deep nets on a CPU is not the efficient way of doing things. – Jan Kukacka May 08 '18 at 06:32
  • JanKukacka I am not training on CPU I ment is there any way to exploit all the CPU during GPU training , – ob21 May 08 '18 at 13:59
  • beside I am having mini batch size set to 1024 when batch size is 1 – ob21 May 08 '18 at 13:59

0 Answers0