Reversible Jump MCMC Simulated Annealing for Neural Networks
Christophe Andrieu, Nando de Freitas, Arnaud Doucet
We propose a novel reversible jump Markov chain Monte Carlo (MCMC) simulated annealing algorithm to optimize radial basis function (RBF) networks. This algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis functions. It performs a global search in the joint space of the parameters and number of parameters, thereby surmounting the problem of local minima. We also show that by calibrating a Bayesian model, we can obtain the classical AIC, BIC and MDL model selection criteria within a penalized likelihood framework. Finally, we show theoretically and empirically that the algorithm converges to the modes of the full posterior distribution in an efficient way.
Keywords: Model selection, Model averaging, MCMC, Reversible jump MCMC, MDL, AIC, BIC, Bayesian
PS Link: http://www.cs.berkeley.edu/~jfgf/rjpaper.ps
PDF Link: /papers/00/p11-andrieu.pdf
AUTHOR = "Christophe Andrieu
and Nando de Freitas and Arnaud Doucet",
TITLE = "Reversible Jump MCMC Simulated Annealing for Neural Networks",
BOOKTITLE = "Proceedings of the Sixteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-00)",
PUBLISHER = "Morgan Kaufmann",
ADDRESS = "San Francisco, CA",
YEAR = "2000",
PAGES = "11--18"