The general case

In the previous sections we studied priors consisting of a factor (the specific prior) which was Gaussian with respect to $P$ or $L=\ln P$ and additional normalization (and non-negativity) conditions. In this section we consider the situation where the probability $p(y\vert x,h)$ is expressed in terms of a function $\phi(x,y)$. That means, we assume a, possibly non-linear, operator $P$ = $P(\phi)$ which maps the function $\phi$ to a probability. We can then formulate a learning problem in terms of the function $\phi$, meaning that $\phi$ now represents the hidden variables or unknown state of Nature $h$.³Consider the case of a specific prior which is Gaussian in $\phi$, i.e., which has a specific prior energy quadratic in $\phi$

$$\frac{1}{2} (\,\phi\,, \,{{\bf K}}\,\phi\, ) .$$ (186)

This means we are lead to error functionals of the form

$$E_\phi = -(\,\ln P(\phi)\,,\, N \,) +\frac{1}{2} (\,\phi\,,\,{{\bf K}}\,\phi \,) +(\,P(\phi)\,,\,\Lambda_X\,),$$ (187)

where we have skipped the $\phi$-independent part of the $\Lambda_X$-terms. In general cases also the non-negativity constraint has to be implemented.

To express the functional derivative of functional (187) with respect to $\phi$ we define besides the diagonal matrix ${\bf P}$ = ${\bf P}(\phi)$ the Jacobian, i.e., the matrix of derivatives ${\bf P}^\prime$ = ${\bf P}^\prime(\phi)$ with matrix elements

$${\bf P}^\prime (x,y;x^\prime,y^\prime;\phi)= \frac{\delta P (x^\prime,y^\prime;\phi)}{\delta \phi (x,y)}.$$ (188)

The matrix ${\bf P}^\prime$ is diagonal for point-wise transformations, i.e., for $P(x,y;\phi)$ = $P(\,\phi(x,y)\,)$. In such cases we use $P^\prime$ to denote the vector of diagonal elements of ${\bf P}^\prime$. An example is the previously discussed transformation $L=\ln P$ for which ${\bf P}^\prime$ = ${\bf P}$. The stationarity equation for functional (187) becomes

$$0 = {\bf P}^\prime (\phi) {\bf P}^{-1}(\phi) N - {{\bf K}}\phi -{\bf P}^\prime (\phi) \Lambda_X ,$$ (189)

and for existing ${\bf P} {{\bf P}^\prime}^{-1}$ = $({\bf P}^\prime {\bf P}^{-1})^{-1}$ (for nonexisting inverse see Section 4.1),

$$0 = N - {\bf P} {{\bf P}^\prime}^{-1}{{\bf K}}\, \phi - {\bf P} \Lambda_X .$$ (190)

From Eq. (190) the Lagrange multiplier function can be found by integration, using the normalization condition ${\bf I}_X P$ = $I$, in the form ${\bf I}_X {\bf P} \Lambda_X$ = $\Lambda_X$ for $\Lambda_X(x) \ne 0$. Thus, multiplying Eq. (190) by ${\bf I}_X$ yields

$$\Lambda_X = {\bf I}_X \left( N - {\bf P} {{\bf P}^\prime}^{... ... - {\bf I}_X {\bf P} {{\bf P}^\prime}^{-1} {{\bf K}}\, \phi .$$ (191)

$\Lambda_X$ is now eliminated by inserting Eq. (191) into Eq. (190)

$$0 = \left( {\bf I} - {\bf P}{\bf I}_X\right) \left( N - {\bf P} {{\bf P}^\prime}^{-1}{{\bf K}}\, \phi \right).$$ (192)

A simple iteration procedure, provided ${{\bf K}}^{-1}$ exists, is suggested by writing Eq. (189) in the form

$${{\bf K}} \phi = T_\phi ,\quad \phi^{i+1} = {{\bf K}}^{-1} T_\phi (\phi^i) ,$$ (193)

with

$$T_\phi (\phi) = {\bf P}^\prime {\bf P}^{-1}N -{\bf P}^\prime \Lambda_X .$$ (194)

Table 2 lists constraints to be implemented explicitly for some choices of $\phi$.

Table 2: Constraints for specific choices of $\phi$

$\phi$	$P(\phi)$	constraints
$P(x,y)$	$P=P$	norm	non-negativity
$z(x,y)$	$P=z/\int\!z\,dy$	--	non-negativity
$L(x,y)=\ln P$	$P=e^L$	norm	--
$g(x,y)$	$P={e^g}/{\int\!e^g\,dy}$	--	--
$\Phi=\int^y dy^\prime \,P$	$P={d\Phi}/{dy}$	boundary	monotony