0

I want to use a mixture density network to make predictions with NLL loss (which is working) but my dataset is unbalanced so my network tends to make estimations that look like the distribution of data of the training y data. I am looking for an analogue to using weights with cross entropy loss to correct for an unbalanced classification dataset but I haven't been able to find any information on possible approaches for regression. I tried the somewhat naive idea to make a density estimate and then reweight each point by 1/density but it didn't seem to make much a difference (though I'll admit I haven't done much testing-maybe a large batch size or something is important)

Sam H
  • 11
  • 5
  • The point of NLL is that it seeks out the correct probability values. – Dave Aug 05 '21 at 10:36
  • I am looking for an equivalent to "weights" for cross entropy loss (which similarly tries to find the "correct probability values" for a classification problem). Cross entropy loss is often reweighted when minority classes are important. However, since I am using an MDN and my problem is a regression problem, its a bit more complicated (I think). I hope my question was clear enough on this-I can edit if needed. – Sam H Aug 05 '21 at 10:42
  • The point of NLL is that it seeks out the correct probability values. [Class imbalance is not a problem.](https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he) – Dave Aug 05 '21 at 20:47
  • Wait, you're doing a regression task or a classification? – Dave Aug 05 '21 at 21:05
  • Regression task with MDN. So my model produces a probability distribution as its output and then the loss is NLL Loss. But my data isn't evenly distributed so I end up with the model almost always predicting unreasonable spikes. – Sam H Aug 05 '21 at 23:12
  • Also I'd like to respectfully disagree with your claim that class imbalance is not a problem (and you can correct me if I am wrong). As I understand it, class imbalance isn't a problem if you don't care about the rare classes. As the link you sent says in "caveat", it is problematic if you care more about the positive identification for minority classes/region than false positives. But also the question would always remain what should be done if I have a train dataset with answers sampled from some distribution and a test dataset samples from another distribution – Sam H Aug 05 '21 at 23:17
  • The caveat has to do with biasing the class representation because of difficulties in obtaining data, nothing to do with a natural class imbalance. If you bias the prior probability, of course the posterior probability is affected, perhaps dramatically. If you care more about the rare classes, perhaps you need to have some utility function to handle that, but that is a separate issue from finding the posterior probabilities. // This really is a separate topic from what you’ve posted here. If you have questions about the caveat, please do post about one of my favorite topics (class imbalance). – Dave Aug 05 '21 at 23:27
  • So your claim is that if you have a dataset with 99% dogs and 1% cats which causes your model to always predict dogs and (importantly) the ultimate goal of your algorithm is to ensure that you predict cats correctly you should not rebalance the importance of each prediction in the loss function (say by setting weight in binary cross entropy loss- https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html)? I mean, cross entropy is the same as NLL loss for classification with the appropriate final layer. – Sam H Aug 06 '21 at 02:09
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/128283/discussion-between-sam-h-and-dave). – Sam H Aug 06 '21 at 02:09

0 Answers0