The generalisation behaviour of statistical learning algorithms relies essentially on the correctness of the implemented a priori information. While Gaussian processes and the related regularisation approaches have, on one hand, the very important advantage of being able to formulate a priori information explicitly in terms of the function of interest (mainly in the form of smoothness priors which have a long tradition in density estimation and regression problems [18,17,5]) they implement, on the other hand, only simple concave prior densities corresponding to quadratic errors. Especially complex tasks would require typically more general prior densities. Choosing mixtures of Gaussian process priors combines the advantage of an explicit formulation of priors with the possibility of constructing general non-concave prior densities.
While mixtures of Gaussian processes are technically a relatively straightforward extension of Gaussian processes, which turns out to be a computational advantage, practically they are much more flexible and are able to produce in principle, i.e., in the limit of infinite number of components, any arbitrary prior density.
As example, consider an image completion task, where an image have to be completed, given a subset of pixels (`training data'). Simply requiring smoothness of grey level values would obviously not be sufficient if we expect, say, the image of a face. In that case the prior density should reflect that a face has specific constituents (e.g., eyes, mouth, nose) and relations (e.g., typical distances between eyes) which may appear in various variations (scaled, translated, deformed, varying lightening conditions).
While ways how prior mixtures can be used in such situations have already been outlined in [6,7,8,9,10] this paper concentrates on the general formalism and technical aspects of mixture models and aims in showing their computational feasibility. Sections 2-4 provide the necessary formulae while Section 5 exemplifies the approach for an image completion task.
Finally, we remark that mixtures of Gaussian process priors do usually not result in a (finite) mixture of Gaussians [3] for the function of interest. Indeed, in density estimation, for example, arbitrary densities not restricted to a (finite) mixture of Gaussians can be produced by a mixture of Gaussian prior processes.