\documentstyle[11pt]{article}
\pagestyle{plain}
\begin{document}
\begin{center}
{\Large Relation Between Brockett's Algorithm and GHA}\\
Terry Sanger, 11/16/93
\end{center}

Brockett's Algorithm for finding principal components
can be written
\begin{eqnarray}
	y & = & Wx \\
	\Delta W &= & \gamma(Nyx^T - yy^TNW) \label{brockett}
\end{eqnarray}
where $W$ is the matrix of feedforward weights, $\gamma$ is the learning
rate, and $N$ is a diagonal matrix
with strictly decreasing entries.

Sanger's Generalized Hebbian Algorithm (GHA) can be written
\begin{eqnarray}
	\Delta W & = & \gamma(yx - {\rm LT}[yy^T]W) 
\end{eqnarray}
where LT[] sets the above diagonal elements of its matrix argument to zero.

To see the relation between the two algorithms, multiply both sides of
equation \ref{brockett} by $N^{-1}$ to obtain
\begin{equation}
	N^{-1}\Delta W = \gamma(yx^T - N^{-1}yy^TNW) 
\end{equation}
and consider the second term  $A=N^{-1}yy^TN$.  This is a matrix with
elements given by 
\[	a_{ij} = \left(\frac{n_j}{n_i}\right)y_iy_j. \]
Since $n_0 > n_1 > \ldots $, we see that 
\[  a_{ij} = \left\{ \begin{array}{ll} 
			y_i^2 & i=j \\
			<y_iy_j & i<j \\
			>y_iy_j & i>j \\
			\end{array}
		\right.
	\]
which means that the elements of the matrix $A$ are less than the elements
of $yy^T$ above the diagonal, greater
below the diagonal, and equal along the diagonal. 

So GHA can be thought of as a rectified form of Brockett's algorithm, in
which the matrix $A$ is given by
\[  a_{ij} = \left\{ \begin{array}{ll} 
			y_i^2 & i=j \\
			0 & i<j \\
			y_iy_j & i>j \\
			\end{array}
		\right.
	\]
and it is not surprising that both algorithms converge to the same
solutions at approximately the same rate.

\end{document}