1

I have time sequences of 2D heat maps. For different people I have heat maps over time. For each person I have around 720 heat maps and in total around 50'000 heat maps.

Now, I would like to train an autoencoder on these heat maps to learn a meaningful low dimensional representation. I will use Keras in Python for the implementation.

I thought about using a CNN for encoder and decoder (but I'm unsure about the number of layers and the size of the layers). Another option could be to use a cycleGAN (pix2pix) or also taking time into account using LSTM but I don't know how to use LSTM for 2D image data. I have also heard that attention layers might be interesting.

What architectures and configurations are reasonable to use? And what loss function would you use?

machinery
  • 1,474
  • 4
  • 18
  • 30
  • Without actual details on what you plan on doing with the representations, there's no single valid answer here. Rather than asking which is best, just try one and see if it fits your use case. – Alex R. Jul 13 '19 at 17:12
  • @AlexR.I would like to use the low dimensional representation of the autoencoder as feature for a SVM classification. – machinery Jul 13 '19 at 18:49
  • 1
    If you already have class labels, why not just train the model end-to-end, with a CNN followed by a classification layer? – Alex R. Jul 13 '19 at 18:53
  • @AlexR.That is not possible because I have too few labels (only around 700 per class for 2 classes). – machinery Jul 13 '19 at 21:57
  • That’s not “few” necessarily. Either try a smaller model or gently fine tune a pretrained image model. – Alex R. Jul 13 '19 at 22:14
  • @AlexR Thank you. I will give it a try. What do you mean by smaller model? Do you know a good pretrained image model which could be a good fit for heat maps? – machinery Jul 14 '19 at 00:27
  • @AlexR.My above example heat map has 3 color channels (RGB). Would you use as input to the CNN these 3 color channels or would you use just one color channel (black white)? I have created the color image by using imshow() and the jet colormap on a 2D matrix... so the data is not truly with three color channels but the color channels are just created by the colormap. – machinery Jul 14 '19 at 00:29
  • 1
    It doesn't make a lot of sense to use 3 colors, since that's just a visual aid to communicate different levels of heat. Heat can be reduced to a single dimension, like temperature. Are you able to infer this from the raw data? – Alex R. Jul 14 '19 at 02:52
  • It's hard to recommend a model because it's not at all clear what your goal is. Could you explain what the heat maps are of, and what you're trying to classify? For example, are the heat maps all on a single fixed webpage? Or does the content change? If the content stays the same, then it's probably enough to just train a model on the heat map ,but if the content changes, you might need to train on the image itself. – Alex R. Jul 14 '19 at 02:53

0 Answers0