I am doing some experiments with LDA (Linear Discriminant Analysis), in python.
Now I am at the point in which I would like to display the separation planes in the 3-dimensional feature space.
I tough that a solution could be, initially, querying the feature space so that for each coordinate ("voxel"), the inferred category can be attached. This category is the argmax of the posterior probabilities inferred for each category.
Till here everything went fine, and the operation, although computationally expensive, was successful.
At this point I would like a reliable method to extract a decision boundary from the queried feature space. In other words, I would like to get a 3D matrix filled with zeroes except for the occurrences of the separation plane, which should be ones.
In order to use existing methods and not to re-invent the wheel, I decided to use the well-known Sobel filter in order to give some values at the points that belong to the decision boundary and zeroes elsewhere.
I will illustrate what I am doing with a simple example, using numpy. Let's assume that the following forged matrix represents a 2D feature space that has been queried with a classifier:
>>> m = np.fliplr(np.triu(np.ones([10,10]))) + np.triu(np.ones([10,10]))
>>> m
array([[ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
[ 1., 2., 2., 2., 2., 2., 2., 2., 2., 1.],
[ 1., 1., 2., 2., 2., 2., 2., 2., 1., 1.],
[ 1., 1., 1., 2., 2., 2., 2., 1., 1., 1.],
[ 1., 1., 1., 1., 2., 2., 1., 1., 1., 1.],
[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[ 1., 1., 1., 1., 0., 0., 1., 1., 1., 1.],
[ 1., 1., 1., 0., 0., 0., 0., 1., 1., 1.],
[ 1., 1., 0., 0., 0., 0., 0., 0., 1., 1.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 1.]])
As you can see, the matrix m could well represent a bi-dimensional space partitioned in several categories by a classifier.
Now, if I convolve that matrix with the Sobel filter, I obtain something that is promising:
>>> from scipy.ndimage.filters import sobel
>>> sobel(m)
array([[ 1., 1., 0., 0., 0., 0., 0., 0., -1., -1.],
[ 2., 3., 1., 0., 0., 0., 0., -1., -3., -2.],
[ 1., 3., 3., 1., 0., 0., -1., -3., -3., -1.],
[ 0., 1., 3., 3., 1., -1., -3., -3., -1., 0.],
[ 0., 0., 1., 3., 2., -2., -3., -1., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., -1., -3., -2., 2., 3., 1., 0., 0.],
[ 0., -1., -3., -3., -1., 1., 3., 3., 1., 0.],
[-1., -3., -3., -1., 0., 0., 1., 3., 3., 1.],
[-3., -4., -1., 0., 0., 0., 0., 1., 4., 3.]])
Since I want to use only zeroes and ones in the representation of the decision boundary, I can do the following:
>>> (np.abs(sobel(m)) > 0).astype(int)
array([[1, 1, 0, 0, 0, 0, 0, 0, 1, 1],
[1, 1, 1, 0, 0, 0, 0, 1, 1, 1],
[1, 1, 1, 1, 0, 0, 1, 1, 1, 1],
[0, 1, 1, 1, 1, 1, 1, 1, 1, 0],
[0, 0, 1, 1, 1, 1, 1, 1, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 1, 1, 1, 1, 1, 0],
[1, 1, 1, 1, 0, 0, 1, 1, 1, 1],
[1, 1, 1, 0, 0, 0, 0, 1, 1, 1]])
As you can see, now I have something that resembles what I would like to obtain, but still not: the decision boundary is too thick, almost three times that what is necessary.
I could attempt to solve the problem by setting a different threshold value:
>>> (np.abs(sobel(m)) > 2).astype(int)
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 1, 0],
[0, 1, 1, 0, 0, 0, 0, 1, 1, 0],
[0, 0, 1, 1, 0, 0, 1, 1, 0, 0],
[0, 0, 0, 1, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 1, 0, 0, 0],
[0, 0, 1, 1, 0, 0, 1, 1, 0, 0],
[0, 1, 1, 0, 0, 0, 0, 1, 1, 0],
[1, 1, 0, 0, 0, 0, 0, 0, 1, 1]])
The problem is that this threshold value ($2$) is dependent on the context. What if the categories are in larger number, or differently numbered, or if I have an arbitrary number of dimensions? This is clearly not working.
Do you have any ideas on how to solve this problem? Maybe another filter for the convolution or a different algorithm?
thank you! :)
