Independent, dependent, and hidden variables

Constructing theories means introducing concepts which are not directly observable. They should, however, explain empirical findings and thus have to be related to observations. Hence, it is useful and common to distinguish observable (visible) from non-observable (hidden) variables. Furthermore, it is often convenient to separate visible variables into dependent variables, representing results of measurements the theory is aiming to explain, and independent variables which are under explicit control and specify the kind of measurements performed.

Hence, we will consider the following three groups of variables

1.

observable (visible) independent variables $x$,

2.

observable (visible) dependent variables $y$,

3.

not directly observable (hidden, latent) variables ${h}$.

This characterization of variables translates to the following factorization property, defining the model we will study,

$$p(x,y,{h}) = p(y\vert x,{h}) \, p(x) \, p({h}).$$ (1)

In particular, we will be interested in scenarios where $x$ = $(x_1,x_2,\cdots )$ and analogously $y$ = $(y_1,y_2,\cdots )$ are decomposed into independent components, meaning that $p(y\vert x,{h})$ = $\prod_i p(y_i\vert x_i,{h})$ and $p(x)$ = $\prod_i p(x_i)$ factorize. Then,

$$p(x,y,{h}) = p({h})\, \prod_i p(y_i\vert x_i,{h}) \,p(x_i) .$$ (2)

Fig.1 shows a graphical representation of the factorization model (2) as a directed acyclic graph [185,125,107,200]. The $x_i$ and/or $y_i$ itself can also be vectors.

The interpretation will be as follows: Variables ${h}\in {H}$ represent possible states of (the model of) Nature, being the invisible conditions for dependent variables $y$. The set ${H}$ defines the space of all possible states of Nature for the model under study. We assume that states ${h}$ are not directly observable and all information about $p({h})$ comes from observed variables (data) $y$, $x$. A given set of observed data results in a state of knowledge $f$ numerically represented by the posterior density $p({h}\vert f)$ over states of Nature.

Independent variables $x\in X$ describe the visible conditions (measurement situation, measurement device) under which dependent variables (measurement results) $y$ have been observed (measured). According to Eq. (1) they are independent of ${h}$, i.e., $p(x\vert{h})$ = $p(x)$. The conditional density $p(y\vert x,{h})$ of the dependent variables $y$ is also known as likelihood of $h$ (under $y$ given $x$). Vector-valued $y$ can be treated as a collection of one-dimensional $y$ with the vector index being part of the $x$ variable, i.e., $y_\alpha (x) = y(x,\alpha) = y(\tilde x)$ with $\tilde x = (x,\alpha)$.

In the setting of empirical learning available knowledge is usually separated into a finite number of training data $D$ = $\{(x_i,y_i)\vert 1\le i\le n\}$ =$\{(x_D,y_D)$ and, to make the problem well defined, additional a priori information $D_0$. For data $D\cup D_0$ we write $p({h}\vert f) = p({h}\vert D,D_0)$. Hypotheses $h$ represent in this setting functions $h(x,y)$ = $p(y\vert x,{h})$ of two (possibly multidimensional) variables $y$, $x$. In density estimation $y$ is a continuous variable (the variable $x$ may be constant and thus be skipped), while in classification problems $y$ takes only discrete values. In regression problems on assumes $p(y\vert x,{h})$ to be Gaussian with fixed variance, so the function of interest becomes the regression function ${h} (x) = \int \,dy\, y p(y\vert x,{h})$.

Figure 1: Directed acyclic graph for the factorization model (2).