Automatic relevance detection

A useful application of hyperparameters is the identification of sensible directions within the space of $x$ and $y$ variables. Consider the general case of an inverse covariance, decomposed into components ${\bf K}_0$ = $\sum_i \theta_i {\bf K}_i$. Treating the coefficient vector $\theta$ (with components $\theta_i$) as hyperparameter with hyperprior $p(\theta)$ results in a prior energy (error) functional

$$\frac{1}{2} \big(\phi-t,\; (\sum_i \theta_i {\bf K}_i)(\phi-t)\,\big) -\ln p(\theta) +\ln Z_\phi (\theta) .$$ (463)

The $\theta$-dependent normalization $\ln Z_\phi(\theta)$ has to be included to obtain the correct stationarity condition for $\theta$. The components ${\bf K}_i$ can be the components of a negative Laplacian, for example, ${\bf K}_i$ = $-\partial_{x_i}^2$ or ${\bf K}_i$ = $-\partial_{y_i}^2$. In that case adapting the hyperparameters means searching for sensible directions in the space of $x$ or $y$ variables. This technique has been called Automatic Relevance Determination by MacKay and Neal [170]. The positivity constraint for $a$ can be implemented explicitly, for example by using ${\bf K}_0$ = $\sum_i \theta_i^2 {\bf K}_i$ or ${\bf K}_0$ = $\sum_i \exp(\theta_i) {\bf K}_i$.