The Bayesian model

Let us consider the following random variables:

1.

$x$, representing (a vector of) independent, visible variables (`measurement situations'),

2.

$y$, being (a vector of) dependent, visible variables (`measurement results'), and

3.

$h$, being the hidden variables (`possible states of Nature').

A Bayesian approach is based on two model inputs [1,11,4,12]:

1.

A likelihood model $p(y\vert x,h)$, describing the density of observing $y$ given $x$ and $h$. Regarded as function of $h$, for fixed $y$ and $x$, the density $p(y\vert x,h)$ is also known as the ($x$-conditional) likelihood of $h$.

2.

A prior model $p(h\vert D_0)$, specifying the a priori density of $h$ given some a priori information denoted by $D_0$ (but before training data $D_T$ have been taken into account).

Furthermore, to decompose a possibly complicated prior density into simpler components, we introduce continuous hyperparameters $\theta$ and discrete hyperparameters $j$ (extending the set of hidden variables to $\tilde h$ = $(h,\theta,j)$),

$$p(h\vert D_0) = \int \!d\theta \sum_j p(h,\theta,j\vert D_0) .$$ (1)

In the following, the summation over $j$ will be treated exactly, while the $\theta$-integral will be approximated. A Bayesian approach aims in calculating the predictive density for outcomes $y$ in test situations $x$

$$p(y\vert x,D) = \int \!dh\, p(y\vert x,h)\, p(h\vert D) ,$$ (2)

given data $D$ = $\{D_T,D_0\}$ consisting of a priori data $D_0$ and i.i.d. training data $D_T$ = $\{(x_i,y_i)\vert 1\le i\le n\}$. The vector of all $x_i$ ($y_i$) will be denoted $x_T$ $(y_T)$. Fig.1 shows a graphical representation of the considered probabilistic model.

In saddle point approximation (maximum a posteriori approximation) the $h$-integral becomes

$$p(y\vert x,D) \approx p(y\vert x,h^*) ,$$ (3)

$$h^*=\,{\rm argmax}_{h\in{\cal H}} p(h\vert D) ,$$ (4)

assuming $p(y\vert x,h)$ to be slowly varying at the stationary point. The posterior density is related to ($x_T$-conditional) likelihood and prior according to Bayes' theorem

$$p(h\vert D) =\frac{p(y_T\vert x_T,h)\,p(h\vert D_0)}{p(y_T\vert x_T,D_0)} ,$$ (5)

where the $h$-independent denominator (evidence) can be skipped when maximising with respect to $h$. Treating the $\theta$-integral within $p(h\vert D)$ also in saddle point approximation the posterior must be maximised with respect to $h$ and $\theta$ simultaneously .

Figure 1: Graphical representation of the considered probabilistic model, factorising according to $p(x_T,y_T,x,y,h,\theta ,j,(\beta )\vert D)$ = $p(x_T)$ $p(x)$ $p(y_T\vert x_T,h,(\beta ))$ $p(y\vert x,h,(\beta ))$ $p(h\vert\theta ,j,D_0,(\beta ))$ $p(\theta ,j,(\beta )\vert D_0)$. (The variable $\beta$ is introduced in Section 3.) Circles indicate visible variables.