EM Algorithm is the short form of Expectation-Maximization Algorithm, a methodology for algorithm construction. It can be seen as an unsupervised clustering method on mixture models.
Unsupervised Machine Learning is a type of Machine Learning. To get to know more about it, check out this link.
The EM algorithm follows an iterative approach, which tries to find the probability distribution parameters with the maximum possibility of its attributes in the presence of missing or latent data.
In this article, we will be taking a deep dive into the EM algorithm and covering almost all of its important concepts extensively. We will first begin with understanding how the basic functioning of the algorithm. And then, take a look at various pointers, such as the applications, advantages, and disadvantages of the EM algorithm.
Understanding the EM Algorithm
How does it work?
It is an iterative approach; hence, given a set of data, consider a set of starting parameters. The EM algorithm involves two major steps: Expectation and Maximization.
The algorithm continuously iterates over these two steps until a convergence takes place. Convergence is assured as the algorithm includes the likelihood at each iteration until it reaches the maximum.
Now we will understand some important concepts and terms that have been mentioned above, and understanding these concepts will give us a better understanding of how the algorithm works.
Expectation Step
As we know, the EM algorithm works on missing or latent data, in this Expectation Step, the algorithm uses whatever knowledge it has gained from the datasets to fill in the values of the missing data. In short, it tries to guess all the missing values so that there so a vacant place at the end of this step.
Maximization Step
The parameters are updated in this stage with the help of the completed data produced at the end of the expectation step.
Convergence
Convergence is a situation where there is a negligible difference in the probability of two random variables. In other words, when the value of two variables gets close enough to each other, that situation is called convergence.
Goals of the EM Algorithm
After understanding the above points, it can be clearly stated that the EM algorithm is primarily suited to use cases in which there exists missing data.
EM algorithm majorly focuses on assuming the missing values based on the knowledge it gains from the data that is available in that data set. It performs this in an iterative manner till it reaches a desired position.
Gaussian Mixture Model (GMM)
The Gaussian mixture model is another estimation model. There are several techniques that can be used to implement this model, Maximum Likelihood Estimation is among the top techniques to do so.
The EM algorithm is the best technique that helps to estimate the parameters of Gaussian distribution. To understand Gaussian Distribution in-depth, follow along with the article in this link.
How does it work?
E-Step:
Guesses values for the latent values.
M-Step:
Improves the estimates produced in the E-Step using the MLE(Maximum Likelihood Estimation).
Convergence:
The model iterates over the E and M steps until the maximum likelihood is achieved and a significant amount of latent values have been found.
Conclusion
Advantages
There are several advantages of using the EM algorithm. Some of them are, that there is always a guarantee that the likelihood will increase. The Estimation Step and the Maximisation Step are very easy for many problems in terms of implementation, one of which we have seen in the GMM.
Disadvantages
Every element to be studied has its own pros and cons. After discussing the advantages or pros, we can now move on to discuss the disadvantages or cons.
Firstly it has a slow convergence which means the algorithm takes time to reach the convergence point. Secondly, there is a requirement for both forward and backward probability.
Applications
EM Algorithm has a wide range of applications as mentioned below:
- Calculating Gaussian Density
- Filling missing Data
- Natural Language Processing
- Medicine
- Engineering
Summary
To summarize the EM algorithm, all we need to understand is that it is majorly used with missing or latent data. It is a very easy-to-implement algorithm that has its own pros and cons.
One can easily identify the problem that they are working on and eventually use the EM algorithm if there is a need to predict missing values. It is a simple concept consisting of only two major steps after getting the parameters and before hitting the convergence point.