Baum-Welch Algorithm
Baum-Welch learns HMM transition and emission parameters from unlabeled sequences by expectation-maximization.
Learning an HMM without labels
When we observe sequences but never see the hidden states, we cannot count transitions and emissions directly. Baum-Welch is the special case of the EM algorithm that estimates HMM parameters from these unlabeled sequences by iterating between inferring the hidden states softly and re-estimating the parameters.
E-step: expected counts
Using forward-backward, the E-step computes gamma_t(i), the posterior probability of state i at time t, and xi_t(i,j), the posterior probability of transitioning from i to j between t and t+1. These are the soft equivalents of the counts we would tally if the states were observed.
M-step: re-estimation
The M-step turns expected counts into new parameters. The transition A_ij becomes the expected number of i-to-j transitions divided by the expected number of times in i. The initial distribution becomes gamma_1. For discrete emissions, B_j(v) becomes the expected time in j while emitting symbol v, normalized. Each M-step provably does not decrease the sequence likelihood.
- Iterate E and M until the log-likelihood change falls below a tolerance
- Converges to a local optimum; use multiple random initializations
- Add smoothing so unseen transitions or symbols keep small nonzero probability
- For continuous emissions the M-step updates Gaussian means and covariances
Practical notes
Baum-Welch inherits EM local-optimum behavior, so initialization matters; a common trick is to seed emissions with a clustering of the observations. Training on many sequences aggregates the expected counts across all of them before the M-step. Because the likelihood surface is multimodal, reported results should note how many restarts were tried and how the best was selected.
When some sequences are labeled, a supervised count-and-normalize estimate can initialize Baum-Welch, blending labeled and unlabeled data.