3 Combining quadratic concepts

Having defined the fundamentals of prior concepts in a bit more detail we now give a short and rather compact outline of combinations of concepts. A more detailed paper is in preparation. We also include some comments on learning algorithms.

Commonly one wants $h$ to approximate training data AND an additional regularization (e.g. smoothness) constraint. Interpreting the functional $E[h]$ up to a constant as posterior log-probability, an AND of independent events is realized by a sum. A sum of quadratic terms $d_i^2$ = $<\!h -t_i\vert O_i \vert h-t_i\!\!>$ is again quadratic and one finds easily

Proposition 1 (probabilistic AND) AND-ing quadratic concepts by adding their square distances $d_i^2$ results in $E=\sum_i^N d_{i}^2[h]$ = $d^2 [h]$ + $E_0$ with square distance $d^2 [h]$ = $<\!h -\overline{t}\vert O \vert h-\overline{t}\!\!>$ from template average $\overline{t} = O^{-1} \tilde t$ where $\tilde t = \sum_{i}^N O_i t_i$, $O = \sum_{i} O_i$ and $h$-independent minimal component energy $E_0$ = $\sum_{i}^N <\!t_{i}\vert O_i\vert t_i\!\!> -$ $<\!\overline{t} \vert O \vert \,\overline{t}\!\!>$, which has, up to a factor $N$, the structure of a variance. The linear stationarity equation for a functional $E= d^2[h]/2 +E_0$ reads $0 = O(\overline{t} - h) = \tilde t-Oh$.

The inverse exists for positive definite $O$ giving the solution $h=\overline{t}=O^{-1}\tilde t$. This case includes for example the classical regularization approaches of Radial Basis Functions (RBF) and various spline methods [3,4,5,6].

As example for OR-like combinations. consider an old, only partly conserved image. Parts of it might be trees OR persons. Having available a collection of example images of trees and persons one may wish to use this information to improve the reconstruction of the image. One has to be careful not to reconstruct a tree as person. Also we do not want to simply replace objects by their prototypes, but to reconstruct the original image as good as possible.

A probabilistic OR for disjunct events appears for probabilities as sum, i.e. as mixture model, $p(h)$ = $\sum_i^{N_i} p(h, i)$ = $\sum_i^{N_i} p_i \, p(h\vert i)$. Assuming $p(h\vert i)$ to be easier to specify than $p(h,i)$ and using the notations $p (h\vert i) = e^{- \beta E_i}/ Z_i$, $Z_i=\int\! dh\, e^{- \beta E_i}$, $p_i= e^{- b e_i}/z$, $z=\sum_i e^{- b e_i}$ and $E_i = (1/2)\sum_{j_i}^{n_{j_i}} d_{ij_i}^2$ we obtain

Proposition 2 (probabilistic OR) Modeling a probabilistic OR for concepts by a mixture (or finite temperature) model

$$E^M[h] =\ln \left( \sum_i^{N_i} e^{-\beta E_i[h]- b e_i - \ln Z_i (\beta)}\right),$$ (7)

the stationarity condition becomes $0 = t^M[h] - O^M [h] h,$ with $O^M = \sum_i p(h,i) O_i$ and $t^M$ = $\sum_i p(h,i) O_i \overline{t}_i$ = $\sum_i p(h,i) \tilde t_i$. It is in general nonlinear.

Here we introduced a convexity parameter $\beta$ (inverse temperature, Lagrange multiplier to determine the average error) controlling the number of local minima of $E^M$ and an analogous $b$ for mixture probabilities $p_i$. (See for example [7] and, similarly, for clustering [8].)

Instead of a mixture of Gaussian (``free'') models one may specify non-concave (``interacting'') log-probabilities by fuzzy logical combinations. Choosing the product as a simple realization of a fuzzy logical OR for distances we can consider polynomial interactions:

Proposition 3 (``a fuzzy OR'') A product implementation of OR gives the polynomial model

$$E^F = \frac{1}{2} \sum_{i=1}^{n_i} \beta_F^{n_{j_i}-1} \prod_{j_i=1}^{n_{j_i}} d_{ij_i}^2,$$ (8)

with stationarity equation $O^F [h] h = t^F[h]$. Here $O^F[h]$ = $\sum_i\sum_{j_i} M_{ij_i}[h] O^{ij_i}$ and ${t}^F[h]$ = $\! \sum_i\sum_{j_i} M_{ij_i}[h] O^{ij_i} t^{ij_i}$ with $M_{ij_i} [h] = \!\!\beta_F^{n_{j_i}-1} \prod_{k_i\ne j_i} d_{ik_i}^2[h]$.

The error surface becomes convex in the limit $\beta_F\rightarrow 0$. This parallelizes the phenomenological Landau-Ginzburg treatment of phase transitions in statistical physics.