1

Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to understand the model generally and not specifically to this topic. I have a question about attention layer but before that I would like to share what I understood.

The things I understood so that you can correct me if I made some mistakes

For this, I have referred an article online which I am sharing here - http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/

It gave basic intuition of what the attention layer does - It asks the model the model to focus on a specific part of the input in order to generate the output. Here's an image of it - http://www.wildml.com/wp-content/uploads/2015/12/Screen-Shot-2015-12-30-at-1.51.19-PM.png

The pink shades are defined by attention variables 'a(i)'. These variables are actually probability distribution that shows 'the probability that the model should focus on the area represented by that variable'. So, a general figure on any attention based model is as - http://www.wildml.com/wp-content/uploads/2015/12/Screen-Shot-2015-12-30-at-1.16.08-PM.png - where 'a's are the attention variables (weights). These variables are trained by the model.

The question

I have a question in mind. Are the attention weights same for every inputs (in machine comprehension, input = 'comprehension'). If so, then independent of the comprehension, the model will focus on a very small part of every comprehension to find an answer. And it isn't necessary that the answer lies in same part of all comprehensions.

If the weights are different for different inputs, then how will we train them. Because, as per my knowledge, in every neural network, the weights are trained for all the input data irrelevant of the input identifier i.e the final weight values at the end of training is same for all the inputs. And after training, these weights are the final values which are used in real-time application.

Kadam Parikh
  • 115
  • 5

1 Answers1

2

The attention "weights" are not weights in the sense that the word is usually used -- they aren't trainable parameters. They are usually computed as a function of the input and some other parameters, so for question answering, the weight might be $a_t = \text{softmax}(h_t^T W q)$, where $h$ is the hidden state, $W$ is a parameter, and $q$ is the question you're trying to answer. So $a_t$ is different for different inputs, but the weight matrix $W$ is learnable and can be fixed.

shimao
  • 22,706
  • 2
  • 42
  • 81
  • Can you elaborate please? And how will this 'W' be related to "a(t)"? Can you provide an intuitive understanding for this? – Kadam Parikh Sep 16 '18 at 19:38
  • @KadamParikh $W$ is the parameter responsible for directing attention. I'm not sure what you're confused about. See [this post](https://stats.stackexchange.com/questions/344508/what-are-attention-mechanisms-exactly) for more info – shimao Sep 17 '18 at 05:11
  • Ok.. I understood what you said. But let me make my question clear. Here, we will find 'W' such that for all different inputs, it will produce different output which tells the model where to focus. Right? But I doubt that it is efficient. As to train such model we will need a large dataset which can train the parameter 'W' on (nearly) all the inputs. So, can we do this really? In short, I understood the way attention works but I doubt that it will work on all the inputs correctly. – Kadam Parikh Sep 17 '18 at 06:31
  • @KadamParikh "So, can we do this really?". Yes, SGD is quite adequate for training nueral networks, and W is just another parameter of the network. – shimao Sep 17 '18 at 14:36