13

I want to use BIC for HMM model selection:

BIC = -2*logLike + num_of_params * log(num_of_data)

So how do I count the number of parameters in the HMM model. Consider a simple 2-state HMM, where we have the following data:

data = [1 2 1 1 2 2 2 1 2 3 3 2 3 2 1 2 2 3 4 5 5 3 3 2 6 6 5 6 4 3 4 4 4 4 4 4 3 3 2 2];
model = hmmFit(data, 2, 'discrete');
model.pi = 0.6661    0.3339;
model.A = 
    0.8849    0.1151
    0.1201    0.8799
model.emission.T = 
    0.2355    0.5232    0.2259    0.0052    0.0049    0.0053
    0.0053    0.0449    0.2204    0.4135    0.1582    0.1578
logLike = hmmLogprob(model,data);
logLike =  -55.8382

So I think:

Nparams = size(model.A,2)*(size(model.A,2)-1) + 
          size(model.pi,2)-1) + 
          size(model.emission.T,1)*(size(model.emission.T,2)-1)
Nparams = 13

So at the end we have:

BIC = -2*logLike + num_of_params*log(length(x))
BIC = 159.6319

I've found a solution where the formula for num_of_params (for simple Markov model) looks like:

Nparams = Num_of_states*(Num_of_States-1) - Nbzeros_in_transition_matrix

So what's the right solution? Do I have to take into account some zero probabilities in transition or emission matrices?

====Updated since 07.15.2011====

I think I can provide some clarification on the impact of data dimension (using “Gaussian mixture distribution” example)

X is an n-by-d matrix where (n-rows correspond to observations; d-columns correspond to variables (Ndimensions).

X=[3,17 3,43
   1,69 2,94
   3,92 5,04
   1,65 1,79
   1,59 3,92
   2,53 3,73
   2,26 3,60
   3,87 5,01
   3,71 4,83
   1,89 3,30 ];
[n d] = size(X); 
n = 10; d =2;

The model will have the following number of parameters for GMM:

nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * Ndimensions  %for daigonal covariance matrices
nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * NDimensions * (NDimensions+1)/2; %for full covariance matrices

If we treat X as 1-dimensional data, than we have num_of_data = (n*d), so for the 2-dimensional data we have num_of_data = n.

2-dimensional data: nParam = 11 ; logLike = -11.8197; BIC = 1.689

1-dimensional data: nParam = 5 ; logLike = -24.8753; BIC = -34.7720

I have a very little practice with HMM. Is it normal to have HMM with (5000, 6000 and more parameters)?

Community
  • 1
Sergey
  • 606
  • 6
  • 10
  • do you have a justification for using BIC? It can give horribly wrong results if not with the appropriate assumptions. – suncoolsu Jul 15 '11 at 12:07
  • @suncoolsu , What do you mean about justification? I've found some examples on K-clusters (GMM models) selection based on BIC scoring. Probably I've provided wrong example with comparing two models with different input data (dimensions). – Sergey Jul 18 '11 at 08:41
  • 1
    I meant using BIC only if the assumption, **the true model is in the model space**, is justified. May be it is justified in your case. I agree with you that people use BIC like AIC, but both are very different things! – suncoolsu Jul 18 '11 at 08:44
  • Hi, this is a late comment and hope you are still active, but what is the best way to get the number of parameters in a model? – masfenix Sep 29 '14 at 23:02

2 Answers2

7

The question is whether some of your parameters in the transition matrix and / or emission matrix are fixed to begin with. Your computations (of the number of parameters) look correct. If you for some reason want a 3 state model instead of a 2 state model and decide upfront that transitions from state 1 to 3 and 3 to 1 are not allowed (have 0 probability) you would have to take this into account in the computation of the number of parameters.

NRH
  • 16,580
  • 56
  • 68
  • Do I have to take data dimension into account? What if `size(data)` will be 2x100 – Sergey Jul 12 '11 at 04:52
  • @Sergey, I only looked at the computation of the number of parameters in my answer. For completeness I should have considered how to compute the "size" of the data set too, as you are interested in BIC. To be honest, I have only seen a derivation of BIC for independent data, but my guess is that the correct "size" of the data is the length of the data series. I am not sure what you mean by the size being 2x100. If you want to comment on that, you have to give a more precise description of how the data are encoded as a 2 by 100 matrix. – NRH Jul 14 '11 at 07:55
  • Thanks for the answer, I think I understand the impact of the data dimension on BIC, please look at my update – Sergey Jul 15 '11 at 08:10
  • Do I have to decrease number of parameters in left-right (Bakis) HMM model (with triangular Transition matrix)?Is it normal to have HMM with (5000, 6000 and more parameters)? – Sergey Jul 18 '11 at 09:31
4

When we're calculating the number of free parameters in the model selection BIC, this means it's simply the number of zeros in the transtion and emission matrices. For example when there's a zero in the transition matrix-this means that there's no probability that a certain state moves to the next (as defined by the transition matrix). That's how BIC selects the optimum of states for an HMM. However, obtaining the no of free parameters just using the size of the intial, transition and emission matrices is confusing

jebsam
  • 41
  • 1