Gibbs sampling is a statistical technique related to Monte Carlo Markov Chain sampling. It is used to search a solution space for an optimal (or at least locally optimal solution). It is an iterative technique. Basically, a single parameter is chosen at random and the value of it is set to a random value (or one chosen from a distribution).
Recently via Twitter I came across “ Gibbs Sampling for the Uninitiated ” by Philip Resnik and Eric Hardisty , a tutorial that shows how to use Gibbs sampling of a Naive Bayes model to estimate the labels on a set of documents. This paper goes through the algebra in great detail and concludes with pseudocode. Resnik and Hardisty do such a good job of making it look easy that I decided to write my own Gibbs sampler. It was, in fact, pretty easy.