next up previous contents
Next: Equal covariances Up: Prior mixtures for regression Previous: Prior mixtures for regression   Contents

High and low temperature limits

It are the limits of large and small $\beta $ which make the introduction of this additional parameter useful. The reason being that the high temperature limit $\beta\rightarrow 0$ gives the convex case, and statistical mechanics provides us with high and low temperature expansions. Hence, we study the high temperature and low temperature limits of Eq. (551).

In the high temperature limit $\beta\rightarrow 0$ the exponential factors $a_j$ become ${h}$-independent

\begin{displaymath}
a_j
\stackrel{\beta\rightarrow0}{\longrightarrow}
a^0_j =
\...
...um_k^me^{-E_{\theta,\beta,k}+\frac{1}{2}\ln\det{{\bf K}}_k}}
.
\end{displaymath} (553)

In case one chooses $E_{\theta,\beta,j}$ = $E_{\beta,j} + \beta E_{\theta}$ one has to replace $E_{\theta,\beta,j}$ by $E_{\beta,j}$. The high temperature solution becomes
\begin{displaymath}
{h} = \bar t
\end{displaymath} (554)

with (generalized) `complete template average'
\begin{displaymath}
\bar t =
\left( {{\bf K}}_D + \sum_j^m a^0_j {{\bf K}}_j \ri...
...ft( {{\bf K}}_D t_D + \sum_l^m a^0_j {{\bf K}}_j t_j \right)
.
\end{displaymath} (555)

Notice that $\bar t$ corresponds to the minimum of the quadratic functional
\begin{displaymath}
E_{(\beta = \infty)} =
\Big( {h} - t_D,\,{{\bf K}}_D ({h}-t...
...sum_j^m a^0_j
\Big( {h} - t_j,\,{{\bf K}}_j ({h}-t_j)\Big)
.
\end{displaymath} (556)

Thus, in the infinite temperature limit a combination of quadratic priors by OR is effectively replaced by a combination by AND.

In the low temperature limit $\beta\rightarrow \infty$ we have, assuming $E_{\theta,\beta,j}$ = $E_{\beta}+E_{j} + \beta E_{\theta}$,

\begin{displaymath}
\sum_j e^{-\beta (E_{0,j}+E_\theta) - E_{\beta}-E_{j}}
=
e^{...
...eta)-E_{\beta}}
\sum_j e^{-\beta (E_{0,j}-E_{0,{j^*}})-E_{j}}
\end{displaymath} (557)


\begin{displaymath}
\stackrel{\beta\rightarrow\infty}{\longrightarrow}
e^{-\beta...
...
E_{0,{j^*}}<E_{0,j}, \; \forall j\ne j^*,
\; p(j^*) \ne 0,
,
\end{displaymath} (558)

meaning that
\begin{displaymath}
a_j \stackrel{\beta\rightarrow\infty}{\longrightarrow}
a^\in...
...min}_j E_{0,j}= {\rm argmin}_j E_{{h},j}
\end{array} \right.
.
\end{displaymath} (559)

Henceforth, all (generalized) `component averages' $\bar t_j$ become solutions
\begin{displaymath}
{h} =
\bar t_{j}
,
\end{displaymath} (560)

with
\begin{displaymath}
\bar t_{j}
=
\left( {{\bf K}}_D + {{\bf K}}_{j} \right)^{-1}
\left( {{\bf K}}_D t_D + {{\bf K}}_j t_{j} \right)
,
\end{displaymath} (561)

provided the $\bar t_{j}$ fulfill the stability condition
\begin{displaymath}
E_{{h},j} ({h}=\bar t_j)
<E_{{h},j^\prime} ({h}=\bar t_j)
, \quad \forall j^\prime \ne j
,
\end{displaymath} (562)

i.e.,
\begin{displaymath}
V_{j}
<
\frac{1}{2} \Big( \bar t_{j} - \bar t_{j^\prime},\,...
...\prime})\Big)
+ V_{j^\prime},
\quad \forall j^\prime \ne j
,
\end{displaymath} (563)

where
\begin{displaymath}
V_j =
\frac{1}{2} \Bigg(
\Big(t_D,\,{{\bf K}}_D\,t_D\Big)
+...
...(\bar t_j,\,({{\bf K}}_D+{{\bf K}}_j) \,\bar t_j\Big)
\Bigg)
.
\end{displaymath} (564)

That means single components become solutions at zero temperature $1/\beta$ in case their (generalized) `template variance' $V_j$, measuring the discrepancy between data and prior term, is not too large. Eq. (551) for $h$ can also be expressed by the (potential) low temperature solutions $\bar t_j$
\begin{displaymath}
h = \left( \sum_j^m a_j ({\bf K}_D +{\bf K}_j)\right)^{-1}
\sum_j^m a_j \,({\bf K}_D +{\bf K}_j)\,\bar t_j
.
\end{displaymath} (565)

Summarizing, in the high temperature limit the stationarity equation (548) becomes linear with a single solution being essentially a (generalized) average of all template functions. In the low temperature limit the single component solutions become stable provided their (generalized) variance corresponding to their minimal error is small enough.


next up previous contents
Next: Equal covariances Up: Prior mixtures for regression Previous: Prior mixtures for regression   Contents
Joerg_Lemm 2001-01-21