I think the big difference is that the Bayesian approach gives you an actual posterior distribution, the likelihood itself is not a distribution. The mode of the posterior can be similar to the maximum likelihood estimate (when the prior does not affect the posterior much), with the posterior you can also interpret the median, mean, and other summaries of the distribution. With the actual posterior you can answer questions like what is the probability that the parameter lies in this range.
If you have co-linearity (multiple regression for example) then the maximum likelihood estimate can be very unstable (things change quite a bit based on including/excluding a single point). Things like ridge regression, lasso, or elasticnet can help stabilize the results, but they also move away from a closed form solution with simple inference. Placing priors on the parameters and doing a Bayesian analysis can give gains equivalent (or superior) to ridge regression or lasso, and still allow inference without being any more complicated.
One example where a Maximum likelihood approach fails, but the Bayesian does not is the Hauck-Donner effect. This happens sometimes with logistic regression or other binomial based models (and could happen in other cases) when a very strong effect looks non-significant because the Wald variance approximation is inflated by the likelihood being fairly flat.
Also, just because a problem has a closed for solution does not mean that it is easy to find or that the computer can do all the calculations in our lifetimes. A Bayesian approach with estimating the posterior with McMC can sometimes be quicker (realistic) compared to finding the exact closed form solution.
And never under estimate the value of being able to incorporate prior information.
Hopefully others will have additional examples.
