I was trying to make sure that BN worked the way I thought so I wanted to check how MatConvNet computed the forward pass of the batch normalization transform. I wrote a super duper simple script and checked BN manually but I can't get match MatConvNet's answer.
I simply compute 1D conv net and then batch normalized right after the convolutional layer:
clc;clear;clc;clear;
%% prepare Data
M = 5;
x = zeros(1,1,1,M); % (1 1 1 2) = (1 1 1 M)
for m=1:M,
x(:,:,:,m) = m;
end
%%
L1 = 3;
%% Forward Pass
w1 = zeros(1,1,1,L1); % (1 1 1 L1) = (1 1 1 3)
w1(:,:,:,1) = 1;
w1(:,:,:,2) = 2;
w1(:,:,:,3) = 3;
b1 = [];
z1 = vl_nnconv(x,w1,b1); % (1 1 3 2) = (1 1 L1 M)
Z1 = squeeze(z1)
disp('---');
% BN scale, one per dimension
G1 = ones(1,1,1,L1); % (1 1 1 3) = (1 1 1 L1)
% BN shift, one per dimension
B1 = zeros(1,1,1,L1); % (1 1 1 3) = (1 1 1 L1)
b1 = vl_nnbnorm(z1,G1,B1,'EPSILON',1e-10); % (1 1 3 2) = (1 1 L1 M)
B1 = squeeze(z1)
to check if it worked the way I thought it should I did:
( Z1(1,1) - mean(Z1(1,:)) ) / sqrt(var(Z1(1,:)) + 1e-10)
which returns the "wrong" answer (note that ( z1(1,1,k,m) - mean(z1(1,1,k,:)) ) / sqrt(var(z1(1,1,k,:)) + 1e-10) also returns the wrong answer for m=1,k=1):
-1.2649
instead of what MatConvNet thinks is the right answer:
-1.4142
I have read the documentation from MatConvNet of BN but I still can't get them to agree. Anyone has some ideas?
