37

Neural networks are often treated as "black boxes" due to their complex structure. This is not ideal, as it is often beneficial to have an intuitive grasp of how a model is working internally. What are methods of visualizing how a trained neural network is working? Alternatively, how can we extract easily digestible descriptions of the network (e.g. this hidden node is primarily working with these inputs)?

I am primarily interested in two layer feed-forward networks, but would also like to hear solutions for deeper networks. The input data can either be visual or non-visual in nature.

rm999
  • 748
  • 5
  • 10

5 Answers5

13

Estimate feature importance by randomly bumping every value of a single feature, and recording how your overall fitness function degrades.

So if your first feature $x_{1,i}$ is continuously-valued and scaled to $[0,1]$, then you might add $rand(0,1)-0.5$ to each training example's value for the first feature. Then look for how much your $R^2$ decreases. This effectively excludes a feature from your training data, but deals with cross-interactions better than literally deleting the feature.

Then rank your features by fitness function degradation, and make a pretty bar chart. At least some of the most important features should pass a gut-check, given your knowledge of the problem domain. And this also lets you be nicely surprised by informative features that you may not have expected.

This sort of feature importance test works for all black-box models, including neural networks and large CART ensembles. In my experience, feature importance is the first step in understanding what a model is really doing.

someben
  • 738
  • 1
  • 6
  • 11
  • 1
    Thanks for the answer; this is actually something I have done and I find it very useful. In this question though I'm looking more for something that looks inside the network rather than treats it as a black box. – rm999 Jun 11 '11 at 15:04
12

Neural networks are sometimes called "differentiable function approximators". So what you can do is to differentiate any unit with respect to any other unit to see what their relationshsip is.

You can check how sensitive the error of the network is wrt to a specific input as well with this.

Then, there is something called "receptive fields", which is just the visualization of the connections going into a hidden unit. This makes it easy to understand what particular units do for image data, for example. This can be done for higher levels as well. See Visualizing Higher-Level Features of a Deep Network.

bayerj
  • 12,735
  • 3
  • 35
  • 56
2

Here's a graphical intuition for a particular kind of neural networks. At the end of that post, there's a link to R code that shows a visualization for a particular problem. Here's what that looks like:

enter image description here

dimitriy
  • 31,081
  • 5
  • 63
  • 138
1

Fall 2011 I took the free online Machine Learning course from Standford taught by Andrew Ng, and we visualized a neural network which was a face detector. The output was a generic face. I want to mention this for completeness, but you didn't mention this kind of application, so I am not going to dig up the details. :)

Andrew
  • 656
  • 5
  • 11
  • The University is Stanford. I can find Andrew Ng course on the web. I heard Dick DeVeaux give a lecture on neural networks claiming that the neurons part of it sort of hides what is really going on. it is just another type of nonliner classifier and if instead of looking at it from the perspective of the diagram they are best understood by writing out what they do algebraically. I hope I am remembering this right. – Michael R. Chernick May 08 '12 at 23:30
  • De Veaux and Ungar, [A brief introduction to neural nets](http://people.orie.cornell.edu/davidr/or474/deveaux.pdf) 1997: "neural nets seem to be everywhere these days, and at least in their advertising are able to do everything that statistics can do without all the fuss and bother of having to do anything except buy a piece of software." – denis Aug 02 '15 at 13:43
0

The below mentioned method is taken from this link, visit the site for more details.

Start with a random image, i.e., arbitrarily provide values to the pixels. "Next, we do a forward pass using this image x as input to the network to compute the activation a_i(x) caused by x at some neuron i somewhere in the middle of the network. Then we do a backward pass (performing backprop) to compute the gradient of a_i(x) with respect to earlier activations in the network. At the end of the backward pass we are left with the gradient ∂a_i(x)/∂x, or how to change the color of each pixel to increase the activation of neuron i. We do exactly that by adding a little fraction αα of that gradient to the image:

x ← x + α⋅∂a_i(x)/∂x

We keep doing that repeatedly until we have an image x' that causes high activation of the neuron in question."

LKB
  • 1
  • 1