1.1	Thrust 1: Modeling and analysis

CISST systems of the future will serve a central role in many scenarios:
•	visualization of surgical settings, in which a surgeon’s perception of the procedure is augmented with complementary data about the patient’s internal anatomy;
•	navigation of surgical scopes and instruments, in which a surgeon’s ability to reach deeply buried structures is enhanced by passive guidance from the system; and 
•	locally assistive robotic devices, in which the system actively aids in the surgical procedure.  

Such scenarios share a core requirement: the system must construct an accurate representation of the surgical setting. This includes creating a detailed model of the patient, then using it to plan specific procedures, e.g., by simulation or optimization.  By connecting the model to the actual surgery, we can then execute robotically assisted or perceptually augmented procedures [Moesges, 1996 #357; Ayache, 1996 #356; R#11]. These activities of modeling, planning and executing require several capabilities:
•	We need accurate models of the patient, of surgical instruments and devices, and of the surgical procedure. These models include geometric information, e.g., the shape and disposition of anatomical parts. They include mechanical properties, e.g., the tissue deformation parameters of an anatomical structure. They may include functional information, e.g., the boundaries of motor cortex. And the model of the procedure will include information coupling patient and instrument, e.g., how tissue will deform under action from an instrument, or the standard sequence of tissue changes in a procedure. We thus need to build accurate, detailed representations of geometry, mechanics and function.
•	Not only must we construct such models, we need to connect them to the patient in the actual surgery.  Thus, we need methods for registration of computer models with their physical counterparts.
•	As surgery involves interaction with tissue, our models must adapt to changes induced by surgical instruments, e.g., tissue retraction, cutting, and removal.  This includes incorporating intra-operative imagery, and providing predictive capabilities about how tissue will change during interaction. 
•	In many CISST applications, it is essential to provide the surgeon with access to the computational models constructed by the system.  Thus, we need visualization methods that will efficiently enable the surgeon to perceive the full context of the model together with live views of the surgical setting.
•	We need to optimize performance.  This is not just for efficiency.  Since the systems are interacting with a patient, we need to find the best way to execute a procedure, while minimizing adverse effects.
A series of increasingly difficult surgical procedures will guide our development of these capabilities. We will initially focus on procedures that require simple registration and localized deformations, such as neurosurgical tumor resections, and skull base surgeries.  A second stage will consider procedures that involve articulating structures, such as spinal surgery or orthopedic surgery.  A third stage would consider procedures that involve deformable tissue, primary examples including kidney, liver and prostate.
1.1.1	Sub-Thrust Area 1.1: Modeling the Patient

Building, registering and adapting patient-specific models is the key activity of the modeling thrust area.  The resulting models form a central foundation for all aspects of CISST systems, and thus we will devote considerable effort to this area.

Milestone:  Because building models is ubiquitous in CISST, we need a generic framework that supports rapid development of models for new anatomical areas and new surgical procedures, with minimal duplication of methods.  In Year 1, we will merge prior work from BWH, MIT and JHU into a uniform software architecture for modeling and registration, to be tested in a range of surgical testbeds. 
1.1.1.1	Model construction:

The first stage of modeling is the analysis of preoperatively acquired imagery, focusing on segmentation (e.g., [Xu, 1997 #390; Prince, 1995 #385; R#1; R#2; R#3; R#4; R#10]), on geometric modeling of anatomical structures (e.g., [Chandrasekhar, 1995 #339; Chandrasekhar, 1996 #325; Williams, 1995 #326; Williams, 1997 #327; R#1; R#6]) and on anatomical registration (e.g., [Xu, 1997 #390; Prince, 1995 #385; Boes, 1994 #41; Davatzikos, 1995 #365; Davatzikos, 1996 #368; R#5; R#7; R#9]).

Segmentation of medical imagery is the process of assigning to each element of the image a tissue label.  Current common approaches include statistical classifiers [R#2; R#10; R#13], deformable surface models [R#12; R#14], and variations thereof.   Our goal is to build on the substantive body of existing work, especially the extensive prior work at MIT, BWH and JHU, to create initial prototype systems.  Merging efforts into a common framework will enable us to quickly develop a prototype testbed on which to test refinements, extensions and interactions with interfacial devices.

Milestone:  We will develop, integrate and test algorithms for segmenting anatomical structures from MRI neuroimagery, in Year 1.  This segmentation infrastructure will combine methods from statistical classification, deformable models, and morphology into a common framework.

Milestone:  We will develop, integrate and test algorithms for segmenting spinal imagery, in Year 3.  This testbed will directly extend methods from the above neuroimaging case.

Milestone:  We will develop, integrate and test algorithms for segmentation of soft tissue imagery, such as for liver, kidney and prostate, in Year 4.

Of course, integrating existing techniques is only a starting point.  We need to dramatically extend the capabilities of our segmentation algorithms:
•	Segmentation needs anatomical knowledge. Registering anatomical atlases (e.g., [R#1, R#6]) to the imagery provides a context in which to initialize deformable models, or to establish priors for probabilistic classifiers.  We will leverage our existing anatomical atlases to create new segmentation methods that utilize this anatomical knowledge to guide the local refinement of segmentation.

Milestone:  In Year 2, we will extend our segmentation methods to include atlas driven segmentation.  This will include use in statistical and probabilistic segmenters and in guiding elastically deformable models.

•	We must fuse information from many modalities; across sensor type (e.g. MRI, MRA, CT, PET, and SPECT) and across dimension (e.g. merging 2D ultrasound or intra-operative MR with full 3D data). Typically one uses multiple modalities because they accentuate different data.  This makes fusion difficult, however, as distinctive features are not preserved across sensors; thus rendering fiducial based registration ineffective. We have pioneered Mutual Information methods (e.g., [R#9, R#5]) for multimodal fusion, and we will extend this approach to other segmentation problems.

Milestone:  In year 3, we will incorporate multi-modal segmentation into our core framework.

Accurate patient-specific models are crucial for CISST. Although any medical image can be treated as a model, it is too impoverished.  While tissue labeling enhances this simple model, general models should also encompass representations of anatomical, physiological, mechanical or other physical patient information.  A CISST system should deal with 3D volumetric models in which information is associated with every voxel, and boundary representations in which the surfaces of entire structures are described. 

We have developed many techniques for modeling the geometry of anatomical structures.  These models explicitly represent surface and volume information about connected anatomical components, and support such tasks as visualization, registration and shape analysis. We have developed methods that generate models of the brain, head and neck region, chest (trachea and bronchi, lungs, heart), vascular structures, abdominal cavity and the musculoskeletal system (shoulder, pelvis, knee and foot).  We have methods that create anatomical models from a single modality, and we have begun to integrate information from different modalities (MR, MRA, fMRI, CT, SPECT and TMS) into single individual displays.

Milestone:  We will integrate these of modeling tools into the core computational framework, in Year 2.  These modeling tools will be tested on several hard and soft tissue structures.

Given these tools as a foundation, we can then extend our computational infrastructure to include:
•	Better methods for representing geometry, especially of deformable tissues.
•	Methods that merge geometry, function and tissue mechanics into a common representation.  

These methods will then relate image-derived anatomical models to tissue properties for use in surgical simulations or model-based control of robotic devices.  We will also develop methods for combining purely geometric descriptions of tissue structures with functional organ descriptions and other information needed for surgical planning.  Our approach focuses on registering 3D images to atlases of nominal anatomy.  Anatomical atlases accurately registered with a patient’s images not only assist the surgeon in navigating through complex structures, but also provide important information to computational surgical planning techniques [Vaillant, 1997 #402].  The majority of atlas registration methods have focused on maximizing the image similarity between an atlas and an individual 3D image [Miller, 1993 #381; Thirion, 1995 #387; Thirion, 1996 #388; Mazziotta, 1995 #380]; hence the application of these methods across modalities is limited.  We will instead develop elastic transformation algorithms that use geometric models of anatomy, such as parametric representations of boundaries or cortical folds [Miller, 1993 #381; Thirion, 1995 #387; Thirion, 1996 #388; Mazziotta, 1995 #380].  We have demonstrated the use of these techniques in atlas-based neurosurgical planning algorithms [Vaillant, 1997 #402]. 

The key role of geometric and functional modeling is demonstrated by our work in developing a finite element model of the brain, focusing on modeling tissue deformation resulting from the growth of tumors, and on adapting neuro-anatomical models of a normal brain to abnormal anatomy [R#15]. We will extend this work to develop more realistic models of brain anatomy, which will reflect the different elastic properties of soft-tissue structures, and to model non-homeomorphic deformations involved in tumor growth.   We will also develop substantially faster methods for solving the finite-element equations, by using locally adaptive meshing schemes and multi-grid methods. Warfield [R#16] has also developed elastic warping methods for segmentation, and we will unify these approaches into a common framework.

Milestone:  We will develop, integrate and test methods for elastic deformation of canonical models to novel imagery.  We will use such methods to segment abnormal tissue and to integrate functional and mechanical properties into our representations.  This will be completed by Year 4.
1.1.1.2	Model registration:
Having constructed patient models, we must relate them to the actual patient in the OR.  For example, accurate stereotaxy is necessary to position robotic devices. Similarly, anatomical registration is needed for surgical simulation, to transfer mechanically appropriate tissue parameters to the segmented body, and for visualization of “invisible” structures such as motor cortex during surgery.  

As with segmentation, our team already has considerable experience in registration: rigid, affine, locally affine and deformable/elastic/flexible (e.g. [R#5; R#7; R#9; R#16; R#17]). This includes work on rigid and affine registration of segmented preoperative scans to neurosurgical patients for intra-operative guidance, change detection analysis of MS lesions, and robotic alignment in orthopedics.  Our team also has extensive expertise in the area of motion analysis for non-rigid anatomy (e.g., heart) [R#18; R#19] as well as deformable registration. 

In addition to incorporating these existing methods into fully integrated CISST systems, we also plan to significantly extend the capabilities of registration methods.  We will register multiple modalities to a common framework, and to deal with a hierarchy of registration transformations, i.e. rigid, affine, locally affine, deformable, flexible, and elastic.  We will also incorporate biomechanical properties into the registration algorithm, i.e., using modes of deformation of a tissue to help guide the registration process.

To deal with multiple modalities, we will leverage our work on Mutual Information registration.  This approach is ideally suited to registration of information in which no common distinctive landmarks exist.  We have already used such methods to register MRI, MRA, CT, and PET imagery.  Current goals are to enhance this framework to support registration of intra-operative MR slices, or US or video images (all inherently 2D) to fused 3D models.  Such registration methods will allow us to constantly maintain a registration of a model to a patient, and to track surgical instruments relative to the patient and the model.

Milestone:  By Year 2, we will incorporate a suite of registration algorithms into our common infrastructure.  These algorithms will cover a range of transformations, and multiple modalities, and will provide a foundation on which to extend work in segmentation and adaptation. 

A second key goal will be to develop registration methods for different classes of transformations.  Of particular interest is flexible or deformable registration.  One approach is to use elastic deformation methods. We will continue our current focus on the brain, heart, and lungs and expand it to include organs, such as liver, prostate, and kidney, which are likely to be early targets of percutaneous therapy delivery. Our deformable registration research will be extended to incorporate more exotic deformations including nonlinear elasticity and viscoelastic and fluid flow, and to incorporate abnormal anatomies using transformations that are not homeomorphic.  Our research on motion estimation will be extended to address the real-time imaging systems likely to be found in the surgical environment, such as fluoroscopy and ultrasound, and to address issues of 3-D registration using 2-D projection or cross-sectional images.

Milestone:  We will demonstrate flexible registration algorithms in the application of multi-modal imagery of the spine, by Year 3.
1.1.1.3	Model adaptation:
The previous areas focus on building models from pre-operative imagery and registering them to the patient.  This is often sufficient to support CISST, for example in augmented reality visualization in neurosurgery, in augmented reality guidance for biopsies, and in tracking of surgical probes that minimally interact with adjacent tissue. There are applications, however, in which we need to update the model to reflect changes in the patient, either because of natural deformations or because of surgical intervention.  Thus, we will extend our framework to include model adaptation, primarily focusing on acquiring new sensor data and re-registering it to the current model. Possible methods of acquisition include using the intra-operative MR scanner at BWH, using X-ray or ultrasound imagery, or using video images.  In all cases, we will need the same novel deformable registration methods described above.

For example, X-ray images or ultrasound provide much of the current image guidance in the OR.  We will continue work in these areas, especially on intra-operative registration of 2D x-ray images to 3D models for orthopedics [Joskowicz, 1995 #311], vascular anatomy registration, and soft tissue puncture applications [Schreiner, 1997 #233; Bzostek, 1997 #337].  But these techniques have the distinct disadvantage that the delineation of soft tissues is very poor, and the image is a volume projection. MRI imaging provides excellent soft tissue contrast and can depict changes in the conformation of the organs during the surgery.  Also, changing tissue contrast can be used to monitor the immediate response to the application of some forms of therapy, such as hyperthermy. These principles have been used effectively by Jolesz et al. at the Brigham and Woman’s Hospital [Schenck, 1995 #401].  

Milestone: We will develop, integrate and test adaptation methods that update pre-surgical models to reflect observed tissue changes during surgery, to be completed by Year 4.
1.1.1.4	Visualization:
Finally, we need efficient means for communicating information contained in the model to humans, primarily through visualization. Our work here will focus on developing a general framework for real-time visualization that combines multiple model representations. Real time updating of visualizations incorporating sophisticated models is crucial in surgical simulation and many forms of interactive planning. For interventions, there is often the need to combine model visualizations with live images.  

To meet these goals, we will concentrate on an interrelated set of visualization tasks.  First, we need to build representations of our patient models that are appropriate for visualization.  We plan to focus on parametric representations, that both speed up image generation, and support rapid visualization of variations in tissue deformation.  For example, temporary changes to soft tissues could be represented by a parametric variance-map.  We will also design extensible skeleton formats to represent anatomical atlases. By modifying coefficients of a small number of basis functions (that may be obtained by imaging techniques) we plan to automatically generate models fitting specific patients. We will also investigate representing specific patients as variance-maps to a few canonical of human anatomy. Tears and wounds would be represented as trimming in such a scheme.

Second, we need novel displays to present the visualization to the surgeon.  Examples include superimposed displays of surgical plan information (tumor, major vessels, planned cut lines, etc.) with video [Grimson, 1994 #107] or intra-operative MRI [Schenck, 1995 #401; Atalar, 1997 #362; Atalar, 1996 #361].   From the viewpoint of generation, our approach will be based on extensions of our current work on model reductions [Kalvin, 1996 #372; Kumar, 1996 #375], fast interactive visualizations of complex 3D objects [Kumar, 1995 #373; Kumar, 1995 #374; Kumar, 1996 #376], and rapid computation of visibility information [Kumar, 1996 #377; Kumar, 1996 #378].    For example, we will develop interactive visualizations combining deformable registrations with image data, e.g., vascular data with fluoroscopic x-rays and later 3D brains with MRI.  From the viewpoint of hardware, we will continue our development of augmented reality displays, in which a semi-transparent panel is placed between the patient and the surgeon.  The surgeon may then see overlays of internal anatomy exactly aligned with the patient, in a viewpoint that coincides with his working volume.  In our view, for both effective surgery and surgical simulation, and for mentoring and collaboration, it is crucial for the surgeon to be able to “see” the visually occluded innards of human anatomy and how surgical tools interact with them. To enable such visualization we need to devise the appropriate representation for complex models of human anatomy and its real-time rendering. In addition, we need to allow intuitive and efficient navigation of the user. To this end, it is crucial that the visualization be easily customizable, since different users want and often need to obtain different views of the same intrinsic model.

In order to facilitate these operations, we propose to build a configurable visualization infrastructure that will incorporate and extend our research in parametric representations and image generation. To enable real-time rendering, the infrastructure will incorporate incremental algorithms that will exploit the inherent coherence in surgical applications. In addition, the system will be designed in terms of independent threads that can be easily parallelized. 
1.1.2	Sub-Thrust Area 1.2: Surgical Planning and Simulation
Building accurate models of the patient and aligning them to the surgical setting provides a fundamental basis on which to erect CISST systems.  For some applications, e.g., augmented reality visualization, or tracking of surgical instruments, these stages are sufficient.  In applications in which there is considerable interaction between patient and surgical devices, however, we can extend the capabilities of our systems to both reason about how particular procedures are likely to effect the anatomical model and how the model should adapt to reflect changes induced by the procedure.  This requires both a modeling of the devices and instruments, and a modeling of the procedure itself.  
One extension deals primarily with simulation issues—letting a surgeon rehearse or plan by providing biologically realistic interactions with a model.  This requires creating patient-specific models that can adapt to conform to observed changes.  Here, one needs ways of observing the actions of the surgical procedure and incorporating those effects into the reconstructed, flexible model.  In this case, we must include mechanical properties as part of our model, to ensure that tissue deforms in a realistic manner upon interaction with a simulated tool.  These properties can be obtained by registration of a segmented model with anatomical atlases, as in subthrust 1.1.  We have already built a knee simulator that couples haptic and visual displays, and allows exploration of articulated joints.

Milestone:  We will build on our prior experience to build a simulator for another articulated structure, in this case the spine.  As part of this process, we will use our tools from subthrust 1.1 to segment imagery, augment it with tissue properties and then support haptic and visual interaction with a variety of surgical procedures.
A second extension deals with planning and optimization issue.  If our models are sufficiently accurate in detail and sufficiently realistic in tissue mechanical properties, then they can be used to support optimization of surgical plans.  For example, we will extend our work on atlas based neurosurgical planning to the more complex problem of atlas-based radiotherapy plan optimization.   We will also build on our earlier work on adaptive segmentation and registration of vascular anatomy for validation and eventual intra-operative use of a catheterization simulator.  We will focus on combining central-axis representations and accurate boundary representations for efficient simulations and rapid adaptive methods for 2D-3D registration to track small motions.  Finally, we will build on our earlier work on configuration space planning with uncertainty [Taylor, 1976 #254; Taylor, 1988 #256; Lozano-Perez, 1989 #174] and adaptive simplification of boundary representations while constraining simplification error [Kalvin, 1996 #372]. 

For example, our team has extensive experience in deformable neuroanatomical models and in transformations adapting these models to an individual anatomy [R#20; R#21; R#22]. We have previously used such models in optimization schemes for pre-operative surgical path planning [R#23]. The advent of highly flexible surgical instruments, such as magnetically guided catheters or seeds  [R#24], opens up exciting new opportunities for the further development of minimally invasive and very precise surgical systems. We will extend our current work on path planning, by developing algorithms that find highly nonlinear surgical paths that wind their way through the cortical gyrations and around blood vessels, or that approach deep seated targets through the ventricular system. We will continue to improve the accuracy of the registration of neuroanatomical atlases with a patient’s anatomy, especially in cortical regions. Other NIH-funded research in our group focuses on mapping the brain function and correlating it with brain morphology. This research will provide the substrate for developing probabilistic models that reflect the variability of cortical functional areas and which will be used in our optimization framework for preoperative planning, which will include measures of functional localization uncertainties.

Our team has also completed preliminary work on a system that finds optimal beam therapy plans utilizing information from a patient’s images and from atlases adapted to the individual morphology of the patient’s brain. On-going work in collaboration with the Applied Physics Laboratory and with the division of Radiosurgery is toward the development of a system for atlas-based radiotherapy planning. This work will continue in the next several years, with emphasis on two issues. First, the extension of our current optimization algorithms to include nonlinear cost functions reflecting inherent differences in the functional organization of various brain regions. For example, the destruction of even the smallest part of the primary motor cortex, which has a highly focal character, will result in partial paralysis, while destruction of much larger parts of the less focal pre-frontal cortex might have little or no side-effects. Second the development of fast numerical procedures, which will allow a surgeon to experiment extensively with the parameters of our optimization algorithms before deciding on a therapy plan.

1.1.3	Sub-thrust area 1.3: Surgical Execution

The final component of the modeling area deals with coupling the constructed models and the selected plan together with the test-bed system to actively assist in performing a surgical procedure.  The central issue here is creating systems that take as input a plan and a current model of the surgical setting, and then use this information to improve surgical performance, by leveraging available interfacial technologies, such as those developed in Thrust Area 2.

Initial efforts will focus on relatively passive systems.  For example, our neurosurgical collaborators already use enhanced reality displays.  In these systems, segmented patient specific models are aligned with the actual patient, then used to provide visualization of surgical instruments.  This supports rapid navigation and localization of targets, while aiding in avoiding critical nearby structures.  A next step is to extend these systems to incorporate planning information, such as the best trajectory for a biopsy needle, or insertion of a catheter.  Ideally, such enhanced reality displays would provide real-time adaptation of the underlying anatomical model, as well as providing feedback and guidance to the surgeon based on the precomputed plans.

More general systems would involve active interaction with interfacial systems.  Examples include the control of a robot for positioning of instruments or for tissue removal (such as is currently executed in Robodoc); passive or semi-active constraints (steady hand manipulation) with task-optimized gains; supervised autonomy activities (such as tissue retraction).  These activities will clearly be tightly coupled with the efforts in the other thrusts and most particularly with the integration into the designated test-beds.