%* Last edited: Sep 5 19:34 1996 (misha) % -*- mode: LaTeX; TeX-master: "main" -*- \chapter{Abacus-1 System-Level Design} This section contains a detailed design of the entire system, including sequencer, PE array, and I/O interfaces. The topics of this section have largely been ignored by architecture researchers, with most of the attention going to the more theoretically interesting processing element and interconnection network design. However, system interfaces are precisely those elements most responsible for bottlenecks and therefore most likely to cripple the performance of the carefully designed PEs and network. This preliminary design is therefore essential to the viability of the Abacus chip in a realistic environment. Although the system description in this chapter uses present tense, the system has not been built. However, various subsystems, such as the I/O boards, were designed to the point of specifying the chips and completing timing diagrams for typical modes of operation. The Abacus-1 system consists of the sequencer and instruction memory, the PE array, the input and output boards, and the video I/O boards. They are connected by a shared 32-bit bus and several direct interfaces. \mypsfig{systemarch.epsi}{systemarch}{System Block Diagram}{Abacus-1 System Block Diagram} \section{Sequencer} In conventional SIMD systems, low-level microinstructions that drive the processor array are generated at runtime from high-level macroinstructions. For example, an 8-bit integer addition operation expands to eight bit-serial add-with-carry instructions. Since the PE array can operate much faster than the host computer, this instruction expansion places high demand on the host's computation and bandwidth. Even if the host's cycle time is comparable to that of the array, it must typically execute multiple instructions to interpret the macroinstruction. The typical solution is to place a circuit called a \emph{sequencer} or \emph{microcontroller} betwen the host and the array. The sequencer is responsible for interpreting macroinstructions sent from the host and sending the microinstructions to the array. The sequencer thus acts as a bandwidth amplifier between the host and PE array. In this model, the host computer executes scalar code on its own. When it is time to perform a parallel operation, the host sends a request to the sequencer to execute a block of instructions. Although the simplest implementation of the sequencer is little more than an address counter and a microcode dispatcher, its complexity invariably grows. The first step down the slippery slope of complexity is the observation that loops are common and the extra latency of repeated block issues by the host is substantial. Thus, a loop counter is added. Unconditional loops become too restrictive, so sequencer gains the ability to sample the global OR flag from the array. Conditional loops often require a single scalar value obtained from the array (such as the active pixel count) to be compared with some other quantity, and the overhead of transferring this value to the host for comparison is too high. Now registers are added to the sequencer, as well as simple program structure. The sequencer begins to resemble a conventional microcontroller. This process is of course familiar as the wheel of reincarnation, as more and more functionality migrates between the main computing system and the formerly dumb controller. This migration occurs as a result of technological changes as more silicon area becomes available, as new algorithms are developed, or as subsystem speeds change. It has occurred in systems ranging from disk controllers to graphics renderers. \mypsfigx{\textwidth}{seqarch.ps}{seqarch}{Sequencer Block Diagram}{Sequencer and instruction delivery path.} The Abacus-1 sequencer design is a new point in the evolutionary cycle. It is based on recognition of the following technological factors: \begin{itemize} \item Memory density has ben growing far faster than code size, especially the highly repetitive code of image processing. There is no need to interpret instructions at runtime when the microinstructions themselves can be stored. \item Contemporary microprocessors already internally run faster than the PE array, and their external bus speeds are within a factor of two of array speeds. \end{itemize} The Abacus-1 sequencer integrates the conventional roles of the host and sequencer. It executes scalar code intermixed with parallel code at a much finer level than before. Specifically, it executes parallel instructions by performing a read operation to a set of addresses, as shown in \fig{blockcode}. A specialized circuit watches the address bus, detects the read, and converts it to a parallel instruction address (PIADDR). The PIADDR is applied to the parallel instruction memory (PIMEM), which is then sent to the PE array. If no read request is issued during a particular cycle, the buswatcher sends a NOP (no operation) instruction to the array. This ability allows variable timing on the part of the sequencer (due to cache effects, for example). \begin{Figure}{blockcode}{Interleaved scalar and parallel code}{ Example of interleaved scalar and parallel code. The parallel code implements a 2-point horizontal boxcar filter. Notice that the issued addresses are identical for different microinstructions. This reduces instruction memory requirements. In actual code, instruction reuse will not be as frequent, since instructions are grouped in pairs.} \begin{verbatim} ADD R3, R1, R2 ; some scalar operation MULT R4, R4, R5 ; another scalar operation LD R0, 0xF1000000 ; NETSOURCE := EAST LD R0, 0xF1000001 ; NET_OUT := P1 LD R0, 0xF1000002 ; P2 := P2 + NET_IN LD R0, 0xF1000003 ; NETSOURCE := WEST LD R0, 0xF1000001 ; NET_OUT := P1 LD R0, 0xF1000002 ; P2 := P2 + NET_IN ADD R5, R3, R2 ; back to scalar code \end{verbatim} \end{Figure} Bandwidth matching between the sequencer's slow bus speed and the array's fast instruction speed is accomplished with the straightforward approach of grouping microinstructions together into blocks. Each read request causes several consecutive instructions to be fetched at once and then delivered to the array in sequence. The disadvantage of blocking is that when control flow changes, NOP instructions have to be inserted to fill the block. A number of interesting optimizations are possible with this approach. Since the instructions are referenced indirectly, with the read address functioning as a pointer, identical microinstructions need only be stored once despite being referenced from multiple program locations. The Abacus sequencer consists of a conventional medium-speed DSP chip, the Texas Instruments C40. It has uniform cycle times as it does not use caches. The DSP chip is limited to clock rates of 40~MHz, leading to 3:1 blocking: for every instruction address, three instructions are fetched from the memory. Controller versions of modern fast microprocessors such as the PowerPC or the Alpha support 66 MHz bus speeds, and will allow 2:1 blocking. \section{PE Board} % Overview: chip arrangement % Clock distribution % Off-board interfaces % IO circuits % Physical issues The processor board contains 16 PE chips, their associated memories, and instruction distribution circuitry. A floorplan is shown in \fig{peboard}. \paragraph{Instruction Distribution} The processor chips are organized in a 4 by 4 array. Each column of PE chips shares a bus of 30 instruction wires. These wires are driven by single-ended ECL drivers at 250~MHz. The bus is parallel-terminated to a 3V power plane (-2 in unoffset ECL) by 50 ohm resistors. The resistors, housed in a SIP with integrated bypass capacitors, serve both as pulldowns and as line terminations. Each set of column buffers is in turn driven by differential line receivers connected to the instruction distribution ribbon cable. The termination for the initial bus is identical to the column bus. \mypsfigy{3in}{peboard.epsi}{peboard}{PE Board}{PE Board} \paragraph{Data Distribution} Image data is delivered to the board at 125 MHz, over a 32-wire differentially-driven cable. It is retimed by a set of registers. The retiming allows for different time of flight from the IO boards to different PE boards. Similarly, output data is driven differentially from the PE chips to retiming registers. This retiming compensates for the varying time of flight on the PCB board itself, and eliminates the need to snake traces from nearby PE chips to equalize delays. The registers drive the retimed data differentially off-board. Global OR differential signals from the chips are registered before being presented to an OR gate. \begin{Table}{partcount}{Board Level Parts Count}{ Board Level SSI ECL Parts Count} \begin{tabular}{|l|l|r|l|} \hline Part & Desc & Qty & Comment\\ \hline E122 & 9-bit buffer & 16 & 4 chips/instruction bus \\ E451 & 6-bit D register, diff-in & 6 & 1 register/ chip (GOR and DATAI) \\ E452 & 5-bit D register, diff & 4 & 1 register/ chip (DATAO) \\ E101 & Quad 4-input OR/NOR & 5 & 1 gate/chip, 1 to sum (GOR) \\ E116 & Quint diff receivers & 6 & 30 instruction bits \\ E111 & Clock repeater & 3 & 1 / 8 clocked chips \\ \hline Total & & 37 & \\ \hline \end{tabular} \end{Table} Although the Abacus-1 chip was designed for glueless integration with memory, the SSI chips lower the integration level. In production, a two custom VLSI chips, each with 164 signal pins, can replace the 37 SSI chips listed in \tab{partcount}. \paragraph{Inter-Chip Network Wiring.} Mesh inter-board wiring uses conventional ribbon cable in a GSG (ground-signal-ground configuration). Although this is a surprisingly low-tech approach for 250 MHz signalling, calculcations show that it is very effective over short distances \cite{fastpcb}. Each ribbon cable is wrapped in a ground shield to reduce interference between cables in the rack. \paragraph{Power And Thermal Considerations.} Each PE chip dissipates 18~W at 125~MHz. In addition, EDRAM parts are estimated to have a static column read current of 105~mA per chip at 16~ns cycle times, or about 500~mW. Sixteen of these chips, forming two SIMMS, consume 8~W. Termination resistor power is negligible, amounting to 24~mW per resistor, or 2.9~W for the system. Neglecting termination power, a fully populated PE board requires 416 W and 8 A. Large heat sinks are used on the chip to reduce thermal resistance, and forced air cooling is clearly required. As a result, the boards cannot be spaced closely together. This limitation increases the worst-case signal travel time on \mesh\ wires. Although the cooling and power delivery systems have not been designed, arrays of small fans have been successfully used to cool large dense TTL boards dissipating 3000 W. \section{I/O Boards \seclabel{ioboard}} A key design decision in the array I/O subsystem is the access of the sequencer to the array data. Direct access to the PE chip internals requires additional data pathways. This was judged to be too pin-intensive. Instead, the access model is that an entire plane is dumped from the array to a memory bank close to the PE. This approach supports dense data transfers such as that of image data directly to the frame buffer. In a sparse access, the sequencer accesses a small subset of the computed results. A block transfer of one data plane requires approximately 1000~cycles, or 8~microseconds. We assume that this is not a substantial overhead compared to the overall algorithm time. Block transfer is used in both the input and output boards. Since the boards are fairly symmetrical in design, only the output (array to sequencer) board is described. The first challenge is to receive the large number of ECL level signals from the PE boards. There are 16 signals per board and 16 boards in the system, for a total of 256 signals at a data rate of 125 Mbits/second. The received signals must then be converted from a bit-serial to bit-parallel form. This is more complicated than a simple serial-to-parallel conversion since a single data word is scattered across several serial transmissions (four in the case of a 16-bit four by four arrangement). After the pixels are reformatted, they must be stored in memory as a linearly accessible frame buffer. This organization allows the sequencer to operate on the image efficiently, with loops and auto-incrementing of pointers. All IO from a group of 16 chips comprising a board is processed by a collection of circuitry called a reformat unit. A reformat unit is composed of a 32-word, 16-bit ECL FIFO, two ECL/CMOS level converters, a SHMUX unit, and two SRAMs. The SHMUX is composed of two FPGAs and is effectively a chain of shift registers and multiplexers. The multiplexer outputs are connected to two SRAM chips. The reformatting proceeds as follows: \begin{enumerate} \item The FIFO is enabled for input and reads in the 32-bit burst. \item The FIFO is enabled for output. The ECL signals are converted to CMOS levels and entered into the SHMUX. \item The controller sequences the multiplexers inside the SHMUX units to select the appropriate bit groups for output. At the same time, appropriate addresses are presented to the SRAM so that the bitwise reformatted data are image-wise reformatted. \end{enumerate} Notice that the data transfer is pipelined, as the FIFO can be loaded at the same time as the SHMUX is trasferring data to the SRAMs. \mypsfigx{\textwidth}{reformat.epsi}{reformat}{Reformatting Circuitry Block Diagram}{Reformatting Circuitry Block Diagram.} The 16 reformat units are controlled by the format controller. The controller consists of a small state machine, a counter, and an address lookup table. The table contains addresses to be presented to the SHMUX units for appropriate reformatting. The sequencer accesses the board memory by presenting the desired address to the format controller, which maps the address to the appropriate SRAM chip based on the preprogrammed image size table. This address translation adds two cycles of latency to sequencer memory accesses, but allows fast random access. The input board symmetrically reverses the reformatting process by loading the SHMUX chips from the SRAMs, loading the FIFO from the SHMUX, and then executing burst transfer operations. If need be, the functionality of the two boards can be combined into one with a few multiplexers and extra wiring demands. \begin{Table}{fmtpartcount}{Reformatter Part Count}{Reformatting subsystem part count} \begin{tabular}{|l|r|} \hline Chip & Quantity \\ \hline Differential receivers & 4 \\ FIFO & 1 \\ Level converters & 3 \\ FPGA & 2 \\ SRAM & 2 \\ \hline Total & 12 \\ \hline \end{tabular} \end{Table} The total chip count for the board is 192 plus the 8 or so chips implementing the format controller for a total of 200. Assuming 1~inch chip pitch, and a double-sided surface mount design, each board is about ten inches on a side, which is quite reasonable. We estimate that a single custom VLSI chip can replace two reformat units, for a substantial improvement in integration and cost. The major limitation is the number of pins required: 16 input pins and 32 output pins. % Local Variables: % mode: latex % TeX-master: "main" % End: