4

I saw some related question regarding to the fact is one should use sampling with resampling or without when using minibatch.

However my question is different.

Let's assume that I use sampling witouth replacement. Meaning that for each epoch one training example can only belong one 'batch size' group.

I want to know if one should only resample the training data (into the different batch size) only 1 time or before to start to update the parameters before each epoch.

For instance suppose I have 20 'batch size' group. Suppose at the first iteration I have sampled picture A in group 2 and picture B in group 4.

If so should picture A always stay in group 2 and picture B in group 4. Or should I resample it before to start a new epoch (e.g. at epoch 2 put picture B in group 6 and picture B in group 20).

Glen_b
  • 257,508
  • 32
  • 553
  • 939
S12000
  • 528
  • 1
  • 4
  • 14
  • I don't understand what you're asking. Are you asking whether mini-batches are sampled with or without replacement? Or are you asking something else? – Sycorax Dec 19 '18 at 16:17
  • @Sycorax I edited the post and added an extra assumption. Hope it helps – S12000 Dec 19 '18 at 16:23

1 Answers1

6

The typical usage of SGD is that each minibatch is constructed completely at random without replacement. (There are other ways to construct minibatches; for some comparison of alternatives, see Why do neural network researchers care about epochs?)

Suppose that you have 6 samples: $S=\{A, B, C, D, E, F\}$ and minibatch size 2. The first minibatch could be $\{A, C\}$ and the second $\{D, F\}$ and the third $\{B, E\}$. So what's happening is that at the first minibatch is that you're sampling 2 examples without replacement from the set $S$. At the second minibatch, you're sampling 2 examples without replacement from $S\setminus\{A,C\}=\{B,D,E,F\}$. At the third minibatch, you're sampling 2 examples without replacement from $S\setminus\{A,C,D,F\}=\{B,E\}$. At this final step, there is only a single possible minibatch because you have a minibatch size of 2, you're sampling without replacement, and only 2 examples remain.

Now you've exhausted all of your training samples, so the next epoch starts. Epochs are constructed completely at random, so a valid sequence of minibatches is first $\{D, F\}$, second $\{A, B\}$ and third $\{C,E\}$. It's ok that one of the minibatches with $\{D,F\}$ appear in both epoch 1 and epoch 2 -- this happened purely due to randomness. (And in a more realistic usage where more training data is available, the less likely it is that consecutive epochs will contain one or more mini-batches that are identical, unless the mini-batch size is 1.)

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • Thanks, just one question you say 'and batch size 2' but then you give first second and third mini-batch. Thus, I think it is an erratum you have to replace size 2 by size 3 – S12000 Dec 19 '18 at 16:48
  • No, the batch size refers to how many samples area in each batch. All of the minibatches have 2 samples in them, so the minibatch size is 2. – Sycorax Dec 19 '18 at 16:49
  • @Syncorax Ahhhhhhh. I got it. Thanks so much for your answer. It was very very helpful. Thanks. – S12000 Dec 19 '18 at 16:51