4

I am training a system that combines two sub-systems: one for classification and another for reconstruction. Can anyone suggestion what are the common practice for weight selection for combining two losses? The numerical values of two losses for first ten iterations are as follows:

Loss 1 (Cross entropy):

Epoch 1: 1.163

Epoch 2: 0.307

Epoch 3: 0.208

Epoch 4: 0.158

Epoch 5: 0.128

Epoch 6: 0.108

Epoch 7: 0.092

Epoch 8: 0.082

Epoch 9: 0.073

Epoch 10: 0.067

Loss 2 (MSE):

Epoch 1: 26.433

Epoch 2: 4.691

Epoch 3: 0.306

Epoch 4: 0.043

Epoch 5: 0.039

Epoch 6: 0.036

Epoch 7: 0.035

Epoch 8: 0.034

Epoch 9: 0.034

Epoch 10: 0.033

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
talk2speech
  • 518
  • 1
  • 6
  • 14

2 Answers2

2

Long story short, there is no algorithm to do that for you, you have to utilize a heuristic approach. How you could do it without getting deep into the domain knowledge and desired loss dynamics:

  1. Running the evaluation on the untrained model to get the mean value of sub-losses.
  2. Normalizing the sub-losses according to the means you got from the previous step.

This has worked for me for a variety of problems when I wanted a quick solution.

Lugi
  • 586
  • 2
  • 11
0

This is a very interesting question and I had a search and found some methods:

  1. uniform combination of losses from different tasks
  2. dynamic weight average
  3. uncertainty weighting methods with various amounts of training data per-task

For the details please refer to this paper: A comparison of loss weighting strategies for multi-task learning in deepneural networks and some more up-to-date papers under the term multi-task learning loss weighting strategies or refer to this open course: CS 330: Deep Multi-Task and Meta Learning.

Hope that helps.

Lerner Zhang
  • 5,017
  • 1
  • 31
  • 52