30

Applying the softmax function on a vector will produce "probabilities" and values between $0$ and $1$.

But we can also divide each value by the sum of the vector and that will produce probabilities and values between $0$ and $1$.

I read the answer on here but it says that the reason is because it's differentiable, although Both functions are differentiable.

floyd
  • 1,240
  • 13
  • 24
  • 1
    I think its better if you first look at logistic regression. your 'goal' is to monotonically transform $(-\infty, \infty)$ to (0,1). This is what the logistic function does. Note that any cumulative (probability) distribution function on the real line also works - see probit regression which uses the normal distribution function. – seanv507 Jul 30 '19 at 20:09
  • See also: [StackOverflow](https://stackoverflow.com/q/17187507/562769) and [ai.SE](https://ai.stackexchange.com/q/17325/3217) – Martin Thoma Jun 08 '20 at 05:14

3 Answers3

49

The function you propose has a singularity whenever the sum of the elements is zero.

Suppose your vector is $[-1, \frac{1}{3}, \frac{2}{3}]$. This vector has a sum of 0, so division is not defined. The function is not differentiable here.

Additionally, if one or more of the elements of the vector is negative but the sum is nonzero, your result is not a probability.

Suppose your vector is $[-1, 0, 2]$. This has a sum of 1, so applying your function results in $[-1, 0, 2]$, which is not a probability vector because it has negative elements, and elements exceeding 1.

Taking a wider view, we can motivate the specific form of the softmax function from the perspective of extending binary logistic regression to the case of three or more categorical outcomes.

Doing things like taking absolute values or squares, as suggested in comments, means that $-x$ and $x$ have the same predicted probability; this means the model is not identified. By contrast, $\exp(x)$ is monotonic and positive for all real $x$, so the softmax result is (1) a probability vector and (2) the multinomial logistic model is identified.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • Thank you so much. We can solve both issues by dividing by the sum of absolute values, Right? – floyd Jul 30 '19 at 01:19
  • 2
    No. What happens if you sum the absolute values of both of my examples and then divide by that sum? – Sycorax Jul 30 '19 at 01:20
  • really thank you. I get it now. but we can solve this issue by taking the absolute value of the numerator or maybe calculating $x_i^2/sum(X^2)$ for each value in the vector. I am not trying to be stubborn, I just find it weird that people invented a complex function although there are simpler ones to calculate probabilities. I don't know a lot of math so maybe there are other mathematical properties – floyd Jul 30 '19 at 01:34
  • 9
    Your proposal still fails for $[0,0,0]$. Additional reasons for the softmax function relate to its properties as a generalization of binary logistic regression to the case of multiple outcomes. We have a number of threads about this such as https://stats.stackexchange.com/questions/349418/definition-of-softmax-function – Sycorax Jul 30 '19 at 01:52
  • 9
    In addition to Sycorax's point, applying $x_i^2 / \sum_j x_j^2$ or $|x_i| / \sum_j |x_j|$ does not have the desired property that reducing a vector element will always reduce its probability contribution. Reducing negative elements would increase their contribution. $\exp(x)$ has the nice property that its output is positive for all real inputs, and is monotonic on the whole real line. – Bridgeburners Jul 30 '19 at 13:21
4

Softmax has two components:

  1. Transform the components to e^x. This allows the neural network to work with logarithmic probabilities, instead of ordinary probabilities. This turns the common operation of multiplying probabilities into addition, which is far more natural for the linear algebra based structure of neural networks.

  2. Normalize their sum to 1, since that's the total probability we need.

One important consequence of this is that bayes' theorem is very natural to such a network, since it's just multiplication of probabilities normalized by the denominator.

The trivial case of a single layer network with softmax activation is equivalent to logistic regression.

The special case of two component softmax is equivalent to sigmoid activation, which is thus popular when there are only two classes. In multi class classification softmax is used if the classes are mutually exclusive and component-wise sigmoid is used if they are independent.

CodesInChaos
  • 157
  • 4
0

In addition to previous suggestion, the softmax function allows for an additional parameter $\beta$, often named temperature $t=1/\beta$ from statistical mechanics, that allows to modulate how much the output probability distribution is concentrated around the positions with larger input value versus smaller ones. $$ \sigma(\mathbf{z})_i = \frac{e^{\beta z_i}}{\sum_{j=1}^K e^{\beta z_j}} \text{ or } \sigma(\mathbf{z})_i = \frac{e^{-\beta z_i}}{\sum_{j=1}^K e^{-\beta z_j}} \text{ for } i = 1,\dotsc , K $$ With this formulation it is also difficult to get extremely unbalanced probabilities, e.g. [1,0,0,..,0], and the system will be allowed a bit of uncertainty in its estimation. To obtain these extreme probability values very low temperatures or very high inputs are necessary. For example in a decision system one may assume temperature that decreases with the number of samples, avoiding having high certainty with very little data

Also softmax does not consider only the relative value of two numbers but their absolute value. This may be important when each input is generated aggregating data from multiple sources and having overall low values for each dimension may just intuitively mean that there is not much information about this situation and so the difference between the output probabilities should be small. While when all the input are quite high, this may mean that more information has been aggregate over time and there is more certainty. If the absolute values are higher, in softmax with the same proportion of the input a higher difference in the output probabilities will be generated. Lower input values may be generated for example when the input is generated by a NN that had fewer samples similar to current input or with contrasting outputs.

Dimitri Ognibene
  • 31
  • 5