Suppose that the function 
is not specified and we wish to find the
variable 
such that the mutual information between y (with
additive noise) and x (which is bounded) is
maximized, according to the ``infomax'' principle first suggested by
Linsker (1988). In order to use the probability analogy,
define the
noise variable 
to have probability density equal to the shape of the
receptive fields 
. Then we can interpret 
as the
probability density of 
given x.
can be interpreted as sampled values of the probability
density 
.
The mutual information between 
and x is given by
where H is the entropy of the density of its argument. Since 
which is a constant determined by the shape of the receptive fields
, maximization of mutual information is achieved by maximizing
. This maximum occurs when 
is a constant. In
this case, we also have that 
is equal for all j.
This gives us a principle for choosing an optimal linear transformation kernel K between densities, and thereby an optimal nonlinear transformation f. Effectively, the solution performs ``histogram equalization'' of the input density (see [Nadal and Parga 1994] for a detailed theoretical analysis). In the context of population codes, this is equivalent to adjusting cortical maps and cellular resources in such a way that all cells are used at the same average frequency.
Note that maintaining constant output expectation does not uniquely define the function f. However, if we add the requirement that the output units continue to have localized receptive fields we can obtain a unique solution. In particular, in the continuous case the kernel K needs to have the form shown in equation 2. In the discrete (sampled) case we use equation 3.
Figure: Synaptic diffusion algorithm structure and results. The mapping
input density is chosen randomly, and the learned mapping is computed
iteratively.
We can use a form of ``synaptic diffusion'' to approximate the optimal
solution through an iterative algorithm. Consider the structure shown in
figure 2. The ``sensory receptive fields'' are the
functions 
. These outputs are remapped onto a large vector
via a stochastic matrix B whose column sums are all 1 and for
which each row contains exactly one nonzero value. B is chosen so that
the expected values 
are all equal. This means that the
connections from 
to 
spread out the activity of each
over a region of the vector 
proportional to the expected
value 
. A subsequent step of calculating the 
's as a
smoothed and subsampled version of 
yields the new population code.
The receptive field shapes for a randomly chosen input density are shown at the top of figure 2. Note that in terms of the original input coordinates the output receptive fields are wider in regions of low data density and narrower in regions of high data density. This is appropriate, since the narrower fields will convey more information about the input data and thus should be allocated to more likely regions. A similar result is shown in more detail in [Linsker 1989].
The algorithm for finding B simply involves allowing the synapses from
to 
to
``diffuse'' down a gradient determined by the product of the local synaptic
density 
and the expected value 
. The one-dimensional
example
shown required 1.2 million diffusion steps (approximately 15 seconds on a
100Mhz 80486). Many other algorithms to solve this problem are possible
and might perform faster.
The extension of this type of algorithm to high-dimensional input is
theoretically straightforward but computationally difficult. Suppose that
the input variable x is a 2-dimensional vector. Then the optimal y
will also be 2-dimensional, with 
a constant. If y is bounded in a
square region, then 
is a separable function that can be written as
and thus the two components 
and 
of y are
independent. To achieve this, let 
and choose a monotonic
such that 
is constant. Now for each value of 
compute
and choose a function 
such that 
has
constant. Then 
which
is a constant as desired. The computational difficulty lies in the
computation of 
when the number of dimensions n
becomes large. Future work will investigate efficient approximations for
this procedure.