General formalism

Decomposed into components the posterior density becomes

$$p(h,\beta\vert D) \propto\! \int\!d\theta\,\sum_j^m p(y_T\vert x_T,h,\beta)$$ (12)

$$\times\; p(h\vert\beta,\theta,j,D_0) \,p(\beta,\theta,j\vert D_0).$$

Writing probabilities in terms of energies, including parameter dependent normalisation factors and skipping parameter independent factors yields

$$p(y_T\vert x_T,h,\beta ) \propto e^{-\beta E_T+\frac{n}{2}\ln \beta}$$

$$p(h\vert\beta,\theta,j,D_0) = e^{-\beta E_{0,j}+\frac{d}{2}\ln \beta}$$ (13)

$$\times e^{\frac{1}{2} \ln \det {\bf K}_j (\theta)}$$

$$p(\beta,\theta,j\vert D_0) \propto e^{-E_{\theta,\beta,j}} .$$

This defines hyperprior energies $E_{\theta,\beta,j}$, prior energies $E_{0,j}$ (`quadratic concepts')

$$E_{0,j} = \frac{1}{2} \mbox{$\Big( h-t_j (\theta) ,\, {\bf K}_j (\theta) (h-t_j (\theta,j) ) \Big)$} ,$$ (14)

(the generalisation to a sum of quadratic terms $E_{0,j} =\sum_k E_{0,k,j}$ is straightforward) and training or likelihood energy (training error)

$$E_T = \frac{1}{2} \sum_i^n (h(x_i)-y_i)^2$$ (15)

$$= \frac{1}{2} \left( \mbox{$\Big( h-t_T ,\, {\bf K_T} (h-t_T) \Big)$} +\sum_i^{n} V_{T}(x_i) \right) .$$

The second line is a `bias-variance' decomposition where

$$t_T (x_i) = \sum_k^{n_{x_i}} \frac{y_k(x)}{n_{x_i}} ,$$ (16)

is the mean of the $n_{x_i}$ training data available for $x_i$, and

$$V_T(x_i) = \sum_k^{n_{x_i}} \frac{y^2_k(x)}{n_{x_i}} -t_T^2 (x_i) ,$$ (17)

is the variance of $y_i$ values at $x_i$. ($V_i$ vanishes if every $x_i$ appears only once.) The diagonal matrix ${\bf K}_T$ is restricted to the space of $x$ for which training data are available and has matrix elements $n_x$.