Questions tagged [gaussian-mixture-distribution]

A type of mixed distribution or model which assumes subpopulations follow Gaussian distributions.

Gaussian mixture can refer to distributions or models which assumes subpopulations follow Gaussian distributions. A mixed Gaussian distribution $\mathcal{P}(x)$ can be generally written as a weighted sum of individual Gaussians $\mathcal{N}$:

$$\mathcal{P}(x) = \sum_{i=1}^n w_i \mathcal{N}(\mu_i,\sigma_i^2)$$

where $\sum_{i=1}^{n}w_i = 1$

GMMs are often used in unsupervised learning where we don't know from which subpopulation a data point belongs. In this case we seek to maximize the likelihood function (which assumes the data is independent):

$$p(X|w,\mu,\sigma) = \prod_{i=1}^{n} p(x_i|w,\mu,\sigma)$$

The parameters $\{w,\mu,\sigma^2\}$ can be solved using either Expectation-Maximization (EM) or MAP estimation.

It can also be used to simulate outliers and thereby test outlier detection and robust statistical methods

579 questions
38
votes
5 answers

Clustering a dataset with both discrete and continuous variables

I have a dataset X which has 10 dimensions, 4 of which are discrete values. In fact, those 4 discrete variables are ordinal, i.e. a higher value implies a higher/better semantic. 2 of these discrete variables are categorical in the sense that for…
26
votes
1 answer

Different covariance types for Gaussian Mixture Models

While trying Gaussian Mixture Models here, I found these 4 types of covariances. 'full' (each component has its own general covariance matrix), 'tied' (all components share the same general covariance matrix), 'diag' (each component has its own…
Bee
  • 423
  • 1
  • 4
  • 5
25
votes
2 answers

If k-means clustering is a form of Gaussian mixture modeling, can it be used when the data are not normal?

I'm reading Bishop on EM algorithm for GMM and the relationship between GMM and k-means. In this book it says that k-means is a hard assign version of GMM. I'm wondering does that imply that if the data I'm trying to cluster are not Gaussian, I…
22
votes
2 answers

EM algorithm manually implemented

I want to implement the EM algorithm manually and then compare it to the results of the normalmixEM of mixtools package. Of course, I would be happy if they both lead to the same results. The main reference is Geoffrey McLachlan (2000), Finite…
20
votes
5 answers

Singularity issues in Gaussian mixture model

In chapter 9 of the book Pattern recognition and machine learning, there is this part about Gaussian mixture model: To be honest I don't really understand why this would create a singularity. Can anyone explain this to me? I'm sorry but I'm just…
Dang Manh Truong
  • 327
  • 1
  • 3
  • 9
18
votes
2 answers

Why is optimizing a mixture of Gaussian directly computationally hard?

Consider the log likelihood of a mixture of Gaussians: $$l(S_n; \theta) = \sum^n_{t=1}\log f(x^{(t)}|\theta) = \sum^n_{t=1}\log\left\{\sum^k_{i=1}p_i f(x^{(t)}|\mu^{(i)}, \sigma^2_i)\right\}$$ I was wondering why it was computationally hard to…
17
votes
1 answer

Quantiles from the combination of normal distributions

I have information on the distributions of anthropometric dimensions (like shoulder span) for children of different ages. For each age and dimension, I have mean, standard deviation. (I also have eight quantiles, but I don't think I'll be able to…
16
votes
2 answers

Why Expectation Maximization is important for mixture models?

There are many literature emphasize Expectation Maximization method on mixture models (Mixture of Gaussian, Hidden Markov Model, etc.). Why EM is important? EM is just a way to do optimization and is not widely used as gradient based method…
15
votes
2 answers

Finding category with maximum likelihood method

Let's say that we had an information for men and women heights. R code: set.seed(1) Women=rnorm(80, mean=168, sd=6) Men=rnorm(120, mean=182, sd=7) par(mfrow=c(2,1)) hist(Men, xlim=c(150, 210), col="skyblue") hist(Women, xlim=c(150, 210),…
15
votes
4 answers

In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?

I understand how main difference between K-mean and Gaussian mixture model (GMM) is that K-Mean only detects spherical clusters and GMM can adjust its self to elliptic shape cluster. However, how do they differ when GMM has spherical covariance…
daisybeats
  • 173
  • 1
  • 6
15
votes
2 answers

How to fit mixture model for clustering

I have two variables - X and Y and I need to make cluster maximum (and optimal) = 5. Let's ideal plot of variables is like following: I would like to make 5 clusters of this. Something like this: Thus I think this is mixture model with 5 clusters.…
rdorlearn
  • 3,493
  • 6
  • 26
  • 29
14
votes
2 answers

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and not another one ? This is maybe linked with…
14
votes
3 answers

References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much complexity. There are a few analytic properties we should…
14
votes
3 answers

Relation between sum of Gaussian RVs and Gaussian Mixture

I know that a sum of Gaussians is Gaussian. So, how is a mixture of Gaussians different? I mean, a mixture of Gaussians is just a sum of Gaussians (where each Gaussian is multiplied by the respective mixing coefficient) right?
13
votes
3 answers

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in cluster.stats() in R), but I assume those are best used…
dmartin
  • 3,010
  • 3
  • 22
  • 27
1
2 3
38 39