\documentclass{article}[11pt]
\usepackage{times,epsfig,subfigure}

\def\figref#1{Figure \ref{#1}}

\setlength{\oddsidemargin}{0.25in}
\setlength{\evensidemargin}{0.25in} \setlength{\parindent}{0pt}
\setlength{\topmargin}{0.0in} \setlength{\textwidth}{6.0in}
\setlength{\textheight}{9in}

\begin{document}

\title{An Outline for Engineering Visual Cognition}
\author{Giulio Sandini and Satyajit Rao}
\date{\today}
\maketitle

\section{Introduction}
We choose to
\begin{itemize}
\item {\em Explore visual cognition through a humanoid robot}, because we
believe that cognition is by definition about being in and interacting
with the real world, and therefore a physical body is
essential. Furthermore a human-like body besides being easier to
interact with would give insights into our own cognition.

\item {\em Explore visual cognition via a developmental program}. We believe
that full blown adult human visual cognition is far too complex to be
directly programmed into a robot. It is fundamentally a result of a
developmental process that is intimately tied to the interaction of
the robot with its environment, and can therefore only be engineered by a
developmental process.
\end{itemize}

For a developmental program one must be be able to specify
\begin{itemize}
\item {\bf What is built in apriori:} The hardware, the pre-programmed
vision and motor architecture, and an initial set of pre-programmed behaviors.

\item {\bf What develops and How:} What kinds of structures get
learned because of the initial set of behaviors? How do these
structures help bootstrap the learning of new behaviors?
\end{itemize}

Naturally, we do not yet have the complete answers to these questions. 
However, in the next few sections we describe our current thinking/approach.

\section{What is built in apriori}

In this section we describe the hardware, the visual routine
machinery which we believe is crucial for the flexibility of visual
cognition, the motor routine machinery, and an initial set of
pre-programmed behaviors.

\subsection{Hardware}

\fbox{\bf Giulio: Please fill in according to BabyBot's specs:} 

\begin{figure}
\centering{\epsfig{figure=arch.eps,width=6in}}
\caption{An overview of the architecture of the system}
\label{fig:arch}
\end{figure}

\subsection{Vision Architecture}

The architecture focuses on two problems are at the core of
visual cognition: One is the ability to extract spatial relations on
demand (eg if a blob is bigger or above another blob), and the second
is the ability to learn spatiotemporal regularities (eg if an object
is released it is likely to move downwards).

\vspace{0.2in}
{\bf Early and Preattentive vision}
\vspace{0.2in}

\figref{fig:arch} is an overview of the proposed visual architecture.
The system can be divided into four distinct levels. At the
first level (starting at the bottom) early-visual are computed.
The defining characteristic of this level is
that the image properties are computed independent of the task at
hand. The sample of these properties follows
\begin{enumerate}
\item Binocular disparity.
\item Optical Flow 
\item The responses of spatial filters as described by Freeman \&
Adelson \cite{freeman}. Five spatial filters (2 first order
derivatives and 3 second order derivates of the Gaussian) across 4
different scales resulting in a 20 dimensional vector at every point
in the image.
\item A color saliency map - indicating regions of high color contrast.
\end{enumerate}

There has been considerable work on {\em perceptual grouping}
\cite{shashua-ullman}\cite{tao-basri}\cite{Mahoney:thesis} - the
grouping of low-level features like a bunch of line terminators that
are aligned, into a high-order feature.  One or more of such line or
region based groupings (not shown in the figure) may be used to
produce a richer early-visual representation.

\vspace{0.2in}
{\bf Visual Routines and Attention}
\vspace{0.2in}

The second level, shows the machinery for visual routines.
The problem of extracting spatial relations on demand (eg finding the
larger of two objects, or the countries that the equator passes
through on a map, etc) is closely linked to visual attention and the
topic of ``visual routines''. Shimon Ullman \cite{Ullman:vis-routines}
initially proposed the problem of finding a versatile spatial analysis
mechanism and also described the framework of a solution. The essence
of his solution is that there exists a set of elementary operations
that when combined in different ways produce different visual routines
for doing various spatial tasks. The elementary operations therefore
form a kind of basis set for visual routines.

Ullman suggests that visual processing is divided into two stages. The
bottom-up, spatially uniform, viewer-centered computation of the base
representation (like the 2 1/2 D sketch) followed by the extraction of
abstract spatial properties by visual routines. Visual
routines define objects and parts, their shapes and spatial
relations. The formation and application of visual routines is not
determined by visual input alone but also by the specific task at
hand. The elementary operations are not all of the same type, some of
them operate in parallel across the entire image, others can be
applied only at a single location at a time. It is suggested that
these characteristics of the operators reflect constraints inherent to
the computation they perform, not because of a shortage of resources.
The structures computed as a result of the application of various
visual routines are incrementally pieced together so that subsequent
processing of the same image can benefit from the intermediate results
of previous computations. For example when asked to count the number
of red objects in a scene, the intermediate result of the locations of
the red objects is maintained to help answer a later question about
the biggest red object.

Ullman suggests the following as plausible elementary
operations:
\begin{enumerate}
\item {\em Shift of Processing focus:} A process that controls where an
operation is applied.

\item {\em Indexing:} Locations that are the odd-man-out in the base
representation (e.g an island of blue in a sea of red), attract the
processing focus directly. They are called indexable locations and
serve as starting points for further
processing.

\item {\em Bounded activation or Coloring:} The spreading of activation in the
base representation from a location or locations. The activation is
stopped by boundaries in the base
representation.

\item {\em Boundary Tracing:} Moving the processing focus along one or more
contours.

\item {\em Marking:} Remembering a particular location so that processing can
ignore it or return to it in the future.
\end{enumerate}

To Ullman's primitives we add the the very important notion of
{\bf attentional state}, i.e.  a set of short-term properties (eg, size
of an object, or a set of marker positions, or the relative position
of one region with respect to another) that are constantly accessed
and modified by the primitive operations.

Composing a basis set of elemental spatial operations to solve spatial
tasks is an attractive idea. Clearly this kind of explanation is far
more plausible than having specialized feature detectors like ``the
smallest object inside the big circle''. However in order to flesh out
the details of the visual routines proposal we will have to to address two
key issues:

\begin{enumerate}
\item Choice of a set of primitives: What is a good set of elementary
operations? 

\item Composition: Given a set of primitives how do they get strung
together to perform some spatial task?
\end{enumerate}

\vspace{0.2in}
{\bf Exploratory behaviors}
\vspace{0.2in}

The third level in \figref{fig:arch} shows the behaviors, some of them
pre-programmed and some dynamically generated. These behaviors are
constructed from the visual routine primitives as well as from some
motor routine primitives (not shown in the figure). Some of the
behaviors we intend to implement are:

\begin{itemize}
\item Saccading to a sudden motion or salient color region.
\item Saccading to a sudden sound.
\item Saccading to a human face (the assumption here is that human face
      detectors are pre-built and not learned)
\item Smooth pursuit tracking (the parameters are learned)
\item Grasp reflex, i.e. closing the fingers around something when it 
comes brushes against the palm of the hand.
\item Random reaching arm movements
\end{itemize}

The approach that we take is that pre-programmed behaviors cause the
robot to interact with and explore the environment. During free
exploration however, i.e. when the system is not trying to solve any
particular problem, it {\em must} have biases about where to look
next, {\em and} what to do at the focus of attention.

\begin{figure}
\centering
\mbox{\subfigure[]{\epsfig{figure=behaviors.eps,height=1.8in}}\quad
\subfigure[]{\epsfig{figure=foabiases.eps,height=1.8in}}}
\caption{Biases about where to look and what to do at the focus of
attention are necessary for exploration.}
\label{fig:biases}
\end{figure}

\begin{figure}
\centering \mbox{\subfigure[A ball is tracked, while its attributes
and the local spatial context around it are attended
to.]{\epsfig{figure=fall2_79lcon.eps,width=1.6in}}\quad
\subfigure[A trace of the local context for the entire event, only
this trace is remembered]{\epsfig{figure=allfall2.eps,width=4.2in}}}
\caption{The attentional trace for an event generated by the bottom-up
exploratory behaviors of \figref{fig:biases}. Only the tracking
behavior was active}
\label{fig:balltrack}
\end{figure}

\begin{figure}
\centering{\epsfig{figure=attpatterns.eps,width=5in}}
\caption{Regularities in the world and attentional biases repeatedly
drive the attentional state through patterns. High frequency patterns
become expectations. Expectations generate visual routines to check
for the predicted part of the pattern.}
\label{fig:attpatterns}
\end{figure}

\vspace{0.2in}
{\bf Learning spatial regularities and generating novel behavior}
\vspace{0.2in}

Exploration of the environment leaves a ``trace'', ie a time sequence
of attentional states. \figref{fig:balltrack} shows an example of an
attentional trace. Recurring spatial relationships and events thus
leave repeating traces in attentional state, and these traces are a
rich source for capturing an abstract description of the event.  For
example when watching a ball roll over the edge of a table, the
pre-programmed behaviors for tracking the ball are engaged. Over many
separate instances of balls rolling over tables the intersection of
the traces generated in attentional state is a good representation of
what's spatially common to balls rolling of tables.

Furthermore, because the pattern learned is a ``trace in attentional
state space'', it can be easily matched to the description generated as
it views a new instance of a ball rolling of a table. The match
between the initial trajectories of the learned pattern and the
ongoing pattern of activity in attentional state enable prediction of
what's going to happen next (the ball falling down and hitting the
floor). This expectation is enough to generate a visual routine
causing the robot to look down at where the ball is going to
fall. \figref{fig:attpatterns} shows a schematic of this idea.

\section{What develops and how}

\bibliography{sajit_viscog}
\bibliographystyle{plain}

\end{document}