%* Last edited: Sep  5 19:34 1996 (misha)
% -*- mode: LaTeX; TeX-master: "main" -*-

\chapter{Abacus-1 System-Level Design}

This section contains a detailed design of the entire system, including
sequencer, PE array, and I/O interfaces.  The topics of this section have
largely been ignored by architecture researchers, with most of the
attention going to the more theoretically interesting processing element
and interconnection network design. However, system interfaces are
precisely those elements most responsible for bottlenecks and therefore
most likely to cripple the performance of the carefully designed PEs and
network.  This preliminary design is therefore essential to the viability
of the Abacus chip in a realistic environment.  Although the system
description in this chapter uses present tense, the system has not been
built. However, various subsystems, such as the I/O boards, were designed
to the point of specifying the chips and completing timing diagrams for
typical modes of operation.

The Abacus-1 system consists of the sequencer and instruction memory, the
PE array, the input and output boards, and the video I/O boards.  They are
connected by a shared 32-bit bus and several direct interfaces.

\mypsfig{systemarch.epsi}{systemarch}{System Block Diagram}{Abacus-1 System
  Block Diagram}


\section{Sequencer}


In conventional SIMD systems, low-level microinstructions that drive the
processor array are generated at runtime from high-level macroinstructions.
For example, an 8-bit integer addition operation expands to eight
bit-serial add-with-carry instructions.  Since the PE array can
operate much faster than the host computer, this instruction expansion
places high demand on the host's computation and bandwidth. Even if the
host's cycle time is comparable to that of the array, it must typically
execute multiple instructions to interpret the macroinstruction.

The typical solution is to place a circuit called a \emph{sequencer} or
\emph{microcontroller} betwen the host and the array.  The sequencer is
responsible for interpreting macroinstructions sent from the host and
sending the microinstructions to the array.  The sequencer thus acts as a
bandwidth amplifier between the host and PE array.

In this model, the host computer executes scalar code on its own. When it
is time to perform a parallel operation, the host sends a request to the
sequencer to execute a block of instructions.

Although the simplest implementation of the sequencer is little more than
an address counter and a microcode dispatcher, its complexity invariably
grows.  The first step down the slippery slope of complexity is the
observation that loops are common and the extra latency of repeated block
issues by the host is substantial. Thus, a loop counter is added.
Unconditional loops become too restrictive, so sequencer gains the ability
to sample the global OR flag from the array.  Conditional loops often
require a single scalar value obtained from the array (such as the active
pixel count) to be compared with some other quantity, and the overhead of
transferring this value to the host for comparison is too high.  Now
registers are added to the sequencer, as well as simple program structure.
The sequencer begins to resemble a conventional microcontroller.

This process is of course familiar as the wheel of reincarnation, as more
and more functionality migrates between the main computing system and the
formerly dumb controller. This migration occurs as a result of
technological changes as more silicon area becomes available, as new
algorithms are developed, or as subsystem speeds change. It has occurred in
systems ranging from disk controllers to graphics renderers.


\mypsfigx{\textwidth}{seqarch.ps}{seqarch}{Sequencer Block Diagram}{Sequencer and instruction delivery path.}



The Abacus-1 sequencer design is a new point in the evolutionary cycle. It
is based on recognition of the following technological factors:
\begin{itemize}

\item Memory density has ben growing far faster than code size, especially
  the highly repetitive code of image processing. There is no need to
  interpret instructions at runtime when the microinstructions themselves
  can be stored.

\item Contemporary microprocessors already internally run faster than the
  PE array, and their external bus speeds are within a factor of two of
  array speeds.

\end{itemize}

The Abacus-1 sequencer integrates the conventional roles of the host and
sequencer.  It executes scalar code intermixed with parallel code at a much
finer level than before.  Specifically, it executes parallel instructions
by performing a read operation to a set of addresses, as shown in
\fig{blockcode}. A specialized circuit watches the address bus, detects the
read, and converts it to a parallel instruction address (PIADDR). The
PIADDR is applied to the parallel instruction memory (PIMEM), which is then
sent to the PE array. If no read request is issued during a particular
cycle, the buswatcher sends a NOP (no operation) instruction to the array.
This ability allows variable timing on the part of the sequencer (due to
cache effects, for example).



\begin{Figure}{blockcode}{Interleaved scalar and parallel code}{
    Example of interleaved scalar and parallel code. The parallel code
    implements a 2-point horizontal boxcar filter. Notice that the issued
    addresses are identical for different microinstructions. This reduces
    instruction memory requirements. In actual code, instruction reuse will
    not be as frequent, since instructions are grouped in pairs.}
\begin{verbatim}
ADD  R3, R1, R2              ;  some scalar operation
MULT R4, R4, R5              ;  another scalar operation
LD   R0, 0xF1000000          ;  NETSOURCE := EAST
LD   R0, 0xF1000001          ;  NET_OUT   := P1
LD   R0, 0xF1000002          ;  P2        := P2 + NET_IN
LD   R0, 0xF1000003          ;  NETSOURCE := WEST
LD   R0, 0xF1000001          ;  NET_OUT   := P1
LD   R0, 0xF1000002          ;  P2        := P2 + NET_IN
ADD  R5, R3, R2              ;  back to scalar code
\end{verbatim}
\end{Figure}


Bandwidth matching between the sequencer's slow bus speed and the array's fast
instruction speed is accomplished with the straightforward approach of
grouping microinstructions together into blocks.  Each read request causes
several consecutive instructions to be fetched at once and then delivered
to the array in sequence.  The disadvantage of blocking is that when
control flow changes, NOP instructions have to be inserted to fill the
block.

A number of interesting optimizations are possible with this approach.
Since the instructions are referenced indirectly, with the read address
functioning as a pointer, identical microinstructions need only be stored
once despite being referenced from multiple program locations.


The Abacus sequencer consists of a conventional medium-speed DSP
chip, the Texas Instruments C40.  It has uniform cycle times as it does
not use caches.  The DSP chip is limited to clock rates of 40~MHz, leading
to 3:1 blocking: for every instruction address, three instructions are
fetched from the memory.  Controller versions of modern fast
microprocessors such as the PowerPC or the Alpha support 66 MHz bus speeds,
and will allow 2:1 blocking.


\section{PE Board}

% Overview: chip arrangement
% Clock distribution
% Off-board interfaces
% IO circuits
% Physical issues

The processor board contains 16 PE chips, their associated memories, and
instruction distribution circuitry.  A floorplan is shown in \fig{peboard}.

\paragraph{Instruction Distribution}
The processor chips are organized in a 4 by 4 array. Each column of PE chips
shares a bus of 30 instruction wires. These wires are driven by
single-ended ECL drivers at 250~MHz.  The bus is parallel-terminated to a
3V power plane (-2 in unoffset ECL) by 50 ohm resistors. The resistors,
housed in a SIP with integrated bypass capacitors, serve both as pulldowns
and as line terminations.  Each set of column buffers is in turn driven by
differential line receivers connected to the instruction distribution
ribbon cable. The termination for the initial bus is identical to the
column bus.

\mypsfigy{3in}{peboard.epsi}{peboard}{PE Board}{PE Board}

\paragraph{Data Distribution}
Image data is delivered to the board at 125 MHz, over a 32-wire
differentially-driven cable. It is retimed by a set of registers. The
retiming allows for different time of flight from the IO boards to
different PE boards.  Similarly, output data is driven differentially from
the PE chips to retiming registers. This retiming compensates for the
varying time of flight on the PCB board itself, and eliminates the need to
snake traces from nearby PE chips to equalize delays.  The registers drive
the retimed data differentially off-board.  Global OR differential signals
from the chips are registered before being presented to an OR gate.

\begin{Table}{partcount}{Board Level Parts Count}{
Board Level SSI ECL Parts Count}
\begin{tabular}{|l|l|r|l|}
\hline
Part & Desc & Qty  & Comment\\
\hline
E122 & 9-bit buffer              & 16 & 4 chips/instruction bus \\
E451 & 6-bit D register, diff-in &  6 & 1 register/ chip (GOR and DATAI)   \\
E452 & 5-bit D register, diff    &  4 & 1 register/ chip (DATAO)  \\
E101 & Quad 4-input OR/NOR       &  5 & 1 gate/chip, 1 to sum (GOR) \\
E116 & Quint diff receivers      &  6 & 30 instruction bits \\
E111 & Clock repeater           &  3 & 1 / 8 clocked chips \\
\hline
Total &                          & 37 & \\
\hline
\end{tabular}
\end{Table}

Although the Abacus-1 chip was designed for glueless integration with
memory, the SSI chips lower the integration level.  In production, a two
custom VLSI chips, each with 164 signal pins, can replace the 37 SSI chips
listed in \tab{partcount}.



\paragraph{Inter-Chip Network Wiring.}
Mesh inter-board wiring uses conventional ribbon cable in a GSG
(ground-signal-ground configuration). Although this is a surprisingly
low-tech approach for 250 MHz signalling, calculcations show that it is
very effective over short distances \cite{fastpcb}. Each ribbon
cable is wrapped in a ground shield to reduce interference between cables
in the rack.


\paragraph{Power And Thermal Considerations.}

Each PE chip dissipates 18~W at 125~MHz. In addition, EDRAM parts are
estimated to have a static column read current of 105~mA per chip at 16~ns
cycle times, or about 500~mW.  Sixteen of these chips, forming two SIMMS,
consume 8~W.  Termination resistor power is negligible, amounting to 24~mW
per resistor, or 2.9~W for the system.

Neglecting termination power, a fully populated PE board requires
416 W and 8 A. Large heat sinks are used on the chip to reduce thermal
resistance, and forced air cooling is clearly required.  As a result, the
boards cannot be spaced closely together. This limitation increases the
worst-case signal travel time on \mesh\ wires. Although the cooling and power
delivery systems have not been designed, arrays of small fans have been
successfully used to cool large dense TTL boards dissipating 3000 W.



\section{I/O Boards \seclabel{ioboard}}

A key design decision in the array I/O subsystem is the access of the
sequencer to the array data. Direct access to the PE chip internals
requires additional data pathways. This was judged to be too pin-intensive.
Instead, the access model is that an entire plane is dumped from the array
to a memory bank close to the PE. This approach supports dense data
transfers such as that of image data directly to the frame buffer.  In a
sparse access, the sequencer accesses a small subset of the computed
results. A block transfer of one data plane requires approximately
1000~cycles, or 8~microseconds. We assume that this is not a substantial
overhead compared to the overall algorithm time.

Block transfer is used in both the input and output boards. Since the
boards are fairly symmetrical in design, only the output (array to
sequencer) board is described.

The first challenge is to receive the large number of ECL level signals
from the PE boards. There are 16 signals per board and 16 boards in the
system, for a total of 256 signals at a data rate of 125 Mbits/second.  The
received signals must then be converted from a bit-serial to bit-parallel
form. This is more complicated than a simple serial-to-parallel conversion
since a single data word is scattered across several serial transmissions
(four in the case of a 16-bit four by four arrangement). After the pixels
are reformatted, they must be stored in memory as a linearly accessible
frame buffer. This organization allows the sequencer to operate on the
image efficiently, with loops and auto-incrementing of pointers.


All IO from a group of 16 chips comprising a board is processed by a
collection of circuitry called a reformat unit.  A reformat unit is composed
of a 32-word, 16-bit ECL FIFO, two ECL/CMOS level converters, a SHMUX
unit, and two SRAMs.  The SHMUX is composed of two FPGAs and is effectively
a chain of shift registers and multiplexers.  The multiplexer outputs are
connected to two SRAM chips.  The reformatting proceeds as follows:

\begin{enumerate}

\item The FIFO is enabled for input and reads in the 32-bit
  burst.

\item The FIFO is enabled for output. The ECL signals are converted to CMOS
  levels and entered into the SHMUX.

\item The controller sequences the multiplexers inside the SHMUX units to
  select the appropriate bit groups for output. At the same time,
  appropriate addresses are presented to the SRAM so that the bitwise
  reformatted data are image-wise reformatted.
\end{enumerate}
Notice that the data transfer is pipelined, as the FIFO can be loaded at
the same time as the SHMUX is trasferring data to the SRAMs.

\mypsfigx{\textwidth}{reformat.epsi}{reformat}{Reformatting Circuitry Block Diagram}{Reformatting
    Circuitry Block Diagram.}



The 16 reformat units are controlled by the format controller. The
controller consists of a small state machine, a counter, and an address
lookup table. The table contains addresses to be presented to the SHMUX
units for appropriate reformatting.

The sequencer accesses the board memory by presenting the desired address
to the format controller, which maps the address to the appropriate SRAM
chip based on the preprogrammed image size table. This address translation
adds two cycles of latency to sequencer memory accesses, but  allows fast
random access.

The input board symmetrically reverses the reformatting process by loading
the SHMUX chips from the SRAMs, loading the FIFO from the SHMUX, and
then executing burst transfer operations. If need be, the functionality of
the two boards can be combined into one with a few multiplexers and extra
wiring demands.


\begin{Table}{fmtpartcount}{Reformatter Part Count}{Reformatting subsystem
    part count}
\begin{tabular}{|l|r|}
\hline
Chip & Quantity \\
\hline
Differential receivers & 4 \\
FIFO &  1     \\
Level converters & 3 \\
FPGA & 2    \\
SRAM & 2 \\
\hline
Total & 12 \\
\hline
\end{tabular}
\end{Table}


The total chip count for the board is 192 plus the 8 or so chips
implementing the format controller for a total of 200. Assuming 1~inch chip
pitch, and a double-sided surface mount design, each board is about ten
inches on a side, which is quite reasonable.  We estimate that a single
custom VLSI chip can replace two reformat units, for a substantial
improvement in integration and cost. The major limitation is the number of
pins required: 16 input pins and 32 output pins.



% Local Variables: 
% mode: latex
% TeX-master: "main"
% End: 
