I can't seem to understand why when approaching a K-class classification problem using a Neural Network we take a different approach than any other classification model.
I understand that in the logistic regression case, we split the problem into K separate binary classification problems, using K different logistic regressions, one for each class. Then the class corresponding to the logistic regression that outputs the highest probability, is the output of the classification.
It seems to me obvious that for a Neural Network we would take the same approach, however it looks that the established approach is to instead increase the number of output neurons from 1 (in the binary classification case) to K, where each neuron is doing a binary classification for each class.
It seems to me that using K separate Neural Networks with a single output neuron, could only increase the predictive power, as they would not have to share any more the same parameters for all the layers before the output layers (as in the single Neural Network with K neurons in ouput layer case) The only downside I can see in this approach is being more computationally expensive. Any other ideas as to why this might not be a good idea?
Extra question: Would you think that in case it is much more computationally expensive, it would be worth consider K separate simpler neural networks vs a single neural network with K output neurons?
