\documentstyle[11pt]{article}
\pagestyle{plain}
\begin{document}
\begin{center}
{\Large Relation Between the APEX Algorithm and GHA}\\
Terry Sanger, 11/16/93
\end{center}

Kung and Diamantaras's APEX (Adaptive Principal Components Extraction)
algorithm can be summarized with the following equations:
\begin{eqnarray}
y_j & = & w_j^Tx \label{gha-1} \\
\tilde{y}_j & = & y_j + a_j^TY_{j-1} \label{apex-2} \\
Y_{j-1} & = & [\tilde{y}_0,\ldots,\tilde{y}_{j-1}]^T \label{apex-1} \\
\Delta w_j &=& \gamma ( \tilde{y}_jx - \tilde{y}_j^2w_j) \label{apex-3}\\
\Delta a_j &=& -\gamma (y_jY_{j-1} + y_j^2a_j) \label{apex-4} 
\end{eqnarray}
where $x$ is the input data, $a_j$ are lateral interaction 
terms affecting the outputs, the usual outputs are given by $y_j$, $w_j$
are the feedforward weights to $y_j$, the
outputs including lateral interactions are $\tilde{y}_j$, the
vector of the first $j-1$ outputs is given by $Y_{j-1}$, 
and $\gamma$ is a rate term.

Sanger's GHA (Generalized Hebbian Algorithm) can be summarized:
\begin{eqnarray}
\tilde{x} & = & x - \sum_{k=0}^{j-1} w_k^Ty_k \label{gha-0} \\
\Delta w_j & = & \gamma (y_j \tilde{x} - y_j^2 w_j) \label{gha-2}\\
	 & = & \gamma ( w_j^Tx \tilde{x} - y_j^2 w_j ) \label{gha-3}
\end{eqnarray}
where in this case the outputs do not include lateral
interaction terms.  The other variables are as in APEX.

First, note that equation \ref{apex-4} for the lateral interactions in APEX is
equivalent to the Widrow-Hoff LMS rule for estimating $Y_{j-1}$ from $y_j$
using the linear coefficients $-a_j$:
\begin{eqnarray}
\hat{Y}_{j-1} & = & -a_jy_j \\
-\Delta a_j & = & \gamma (Y_{j-1} - \hat{Y}_{j-1})y_j
			\label{delta-a}
\end{eqnarray}
so at convergence $-a_jy_j$ will be the best linear estimate of $Y_{j-1}$
from $y_j$.

Now define
\[	W_{j-1} = [w_0,...,w_{j-1}] \]
and since Kung and Diamantaras show that $a_k \rightarrow 0$ for $k<j$,
we can write
\begin{equation}
	Y_{j-1} \rightarrow W_{j-1}^Tx .
\label{multi-output}
\end{equation}
This implies that the
best estimate of $Y_{j-1}$ from $y_j = w_j^Tx$ is given by
\[ \hat{Y}_{j-1} = W_{j-1}^Tw_j \|w_j\|^{-1} y_j \]
so that equation \ref{apex-4} converges to
\begin{equation}
	a_j = -W_{j-1}^Tw_j\|w_j\|^{-1}
\label{a-convergence}
\end{equation}
and note that APEX causes $\|w_j\| \rightarrow 1$.

Next, substitute back into equation \ref{apex-2} to obtain:
\begin{eqnarray}
 \tilde{y}_j  &=&  w_j^Tx - w_j^TW_{j-1}Y_{j-1} \\
	& = & w_j^T( x - \sum_{k=0}^{j-1} w_k^Ty_k ) \\
	& = & w_j^T \tilde{x}
\end{eqnarray}
and substitute this into equation \ref{apex-3} to obtain:
\begin{equation}
\Delta w_j = \gamma ( w_j^T\tilde{x}x - \tilde{y}_j^2w) 
\end{equation}
which is almost exactly equivalent to equation \ref{gha-3} except that the
normalization term uses $\tilde{y}_j$ instead of $y_j$, and the order of
terms is exchanged in the first multiplication.  But we know
that $a_j$ converges to 0, so $\tilde{y}_j$ approaches $y_j$, and the
stable points of GHA and APEX must be the same.
\end{document}