Image Understanding in Medical Domains

E. Grimson, T. Kanade, N. Ahuja, P. Allen, R. Bajscy, B. Bhanu, T. Binford, T. Boult, R. Haralick, C. Kohl, G. Medioni, R. Nevatia

November 4, 1993

1 Medical Applications of Image Understanding

1.1 Goals

The goal of this proposed initiative is to merge current and emerging techniques from the image understanding community with existing opportunities in surgical, diagnostic and clinical problems in medical imagery. Advances in health care/medicine are clearly of importance to the general populace. Aspects of medical applications are of particular relevance to the military, especially techniques that enable rapid deployment of diagnosis, triage, and trauma care to the battlefield, without jeopardizing highly trained personel. Image Understanding is directly revelant to many such applications, including visualization, diagnosis, image compression (for effective transmission of data to/from the combat zone), and telepresence feedback. Applications of image understanding methods to medical problems can also have a clear “dual use” impact as well. Many of the same situations arise in cases of natural disasters, remote expert second opinions, and related problems. For example, registration methods that support visualization of 3D structures could be used for remote diagnosis and trauma care, and for telepresence surgury, applications that are relevant both to handling medical emergencies during natural disasters and in battle situations.

1.2 Possible application areas

There has been an impressive surge in the capabilities of medical sensors in the past few years, especially in 3D sensors (e.g. MRI, CT, PET, SPECT). These sensors are providing practitioners with a wealth of accurate 3D data that often carries additional information about underlying material properties. Concurrently, noninvasive techniques (e.g. laparoscopy, endoscopy, arthroscopy) are also rapidly increasing in use and sophistication. Both of these developments are examples of emerging opportunities for IU techniques, in large part because the combination of data sources and restricted operating environments directly requires methods for interpreting, manipulating and highlighting 3D data in a manner that effectively enhances the surgeon’s field of view.

Thus, there are exciting opportunities (that include the possibility of making an impact on a very broad scale) for using IU methods to leverage current capabilities in the following applications:

• 3D visualization of a patient to support minimally invasive surgical procedures, remote diagnosis and telepresence surgery;

• realistic 3D simulation of anatomical structures, to support surgical simulations and planning of procedures, as well as the training of medical students;

• image guided surgical procedures, such as laser disc fusion, image guided biopsies, and focused therapeutic procedures;

• compression of medical data for transmission;

• enhanced clinical studies, by enabling automatic registration and change detection of 3D medical imagery acquired over time, and by tracking anatomical structure through time sequences of imagery;

• medical database manipulation, especially retrieval and correlation of relevant data by image analysis and association;

• enhancing 3D medical data with traditional IU 3D data (stereo, shape from X, motion);

• 3D data fusion, especially multi-sensor fusion, such as among MRI, CT, SPECT, or PET, to support diagnosis and surgical planning;

Applying IU techniques to problems in the medical domain does more than just provide benefits for particular applications. It also provides a focused means of pushing out the performance and capabilities envelop of existing IU technology. For example, current medical sensors are capable of delivering very rich, and accurate information, in scenarios with minimal clutter. This fits nicely within the operating characteristics of current IU recognition/registration systems. At the same time, the objects that must be handled are often flexible, and of complex shape, so that we need to extend our methods to deal with new shape representations, and a broader class of transformations. Thus the medical domain may well provide the IU community with a rich source of new problems, which will help stretch our techniques to the next generation. At the same time, application of current techniques may well lead to near term successes, because of the advantages of constrained problem domains and rich data sources.

2 Technical Issues

2.1 IU technologies needed

Because the range of medical applications is so broad, techniques from almost all areas of IU are relevant to problems in this area.

For example, many of the applications involve the registration of 3D medical data, such as MRI or CT, either to other medical data sets, or to 2d or 3D visual data of the actual position of the patient. Such applications clearly benefit from robust and efficient registration/recognition methods, and the IU community has considerable experience in developing such methods.

Similarly, clinical and surgical uses of medical imagery often require the segmentation of the data into distinctive tissue types. Again, IU methods for image segmentation are clearly relevant, especially those that use deformable models and those that can incorporate atlas information into the process.

Other IU techniques that apply include:

• the representation of 3D shapes (especially deformable models) and generation and presentation of graphical displays of such shapes, especially for visualization tasks;

• motion tracking, in order to update the relative position of a 3D model with a surgeon’s current viewpoint, and to update visualization capabilities, especially in minimally invasive techniques where the surgeon needs systems that can automatically expand the surgeon’s limited field of view;

• change detection for clinical studies and to support diagnostic capabilities;

• shape extraction techniques like stereo vision and shape from shading, to augment medical 3D sensors;

• motion analysis for cardiology and other applications, especially using medical imagery acquired at rapid rates relative to such anatomical motion;

• learning techniques, especially as applied to automatic segmentation;

• as well as other methods.

2.2 Special concerns and constraints for IU in medicine

Perhaps the most critical factor for IU to have an impact in the medical arena is to ensure that the IU community establishes meaningful collaborations with medical practitioners. While IU clearly has potential contributions to make to medicine, they will have a realistic impact only with carefully coupled with actual medical practice.

2.3 Examples of IU accomplishments in medicine

A number of techniques and systems have already been demonstrated in the medical domain. For example, a collaborative effort by MIT, Brigham and Women’s Hospital, TASC and GE have developed and demonstrated a registration system that matches 3D data sets against one another. This system has been demonstrated as a frameless stereotaxy system for neurosurgery, by enabling an enhanced reality visualization of a video view of the patient from the perspective of the surgeon with an aligned view of an MRI or CT reconstruction of the patient’s 3D anatomy. The system has also been used to register MRI data sets taken at different times of the same patient, as part of a clinical study of MS. A similar frameless stereotaxy system, using X-ray views and MRI reconstructions has been demonstrated by Stanford. An example of the MIT system applied to an artificial skull is shown in Figure 1.

Various groups are developing tools for surgical and therapeutic planning, based on the accurate and registered 3D models that can be built from medical imagery. For example, Amerinex A.I. is developing a system that segments and displays brain structures from MRI data (similar systems exist at MIT, Stanford and other sites), which is illustrated in Figure 2.

*** I really need other examples, especially pictures ***

Figure 1: Example of registered laser range data (shown as large dots) overlaid on CT model. Such registration enables the visualization of 3D medical models, especially visual overlay of such models from any perspective.

Figure 2: Left half demonstrates a rendered reconstruction of MRI data, and segmented structures in it. Right half illustrates an interactive tool for planning trajectories through the model.

2.4 Benchmarking plans

The issue of appropriate benchmarking methods will depend on the specific applications. For example, for problems involving registration of data, benchmarking should include measures of accuracy of registration on standard data sets, range of conditions under which convergence to a correct solution occurs and with what probability, and measures of false local minima encountered. For problems involving segmentation, benchmarking can be done against data sets segmented by experts (many of which already exist), with different measuresof missegmentation rates being most appropriate.

3 Political Issues

3.1 Areas of impact/stakeholders

Clearly any successful application of IU technology to medical problems and applications can have an enormous potential impact in the political arena. This is especially true in surgical applications, such as frameless stereotaxy, or enhanced visualization for endoscopic procedures; and in diagnostic applications, such as change detection or volumetric segmentation for flagging possible tumors. Because these applications have both a military opportunity and a “dual use” opportunity, the potential stakeholders come from a range of areas. One such stakeholder would be ARPA itself, through Rick Satava and Don Jenkins, other military stakeholders could include the medical corps of the different branches of the services, and military medical institutions (e.g. Wltaer Reed). On the civilian side, possible stakeholders could include the Surgeon General’s office, again possibly through Don Jenkins, and presumably some aspect of the Clinton Health Care Initiative. More traditional medical funding agencies, such as NIH seem much more focused on diseases and less on surgical or clinical issues, so this may be a less likely avenue for backing.

3.2 Why now?

There are several factors that make this an excellent time to bring IU methods to the medical arena. First, ARPA has only recently been authorized to participate in medical funding. Thus the ARPA IU community now has access to a new funding opportunity, and perhaps more importantly, now has a direct opportunity to make a contribution in the medical area.

Second, the Clinton administration has placed a strong emphasis on “dual use” technologies, and clearly the medical arena is an excellent candidate for transferring technologies originally developed within the military environment to the commercial and domestic sector.

Third, the IU community could benefit from an application that has clear impact on a broad portion of the populace. If useful medical applications can be created using IU methods, such a high visibility success can be used to leverage the visibility and utility of IU in a more general sense. It seems that the convergence of well tested IU techniques with the developing medical sensor technology and the surge of interest in visualization leads to a very timely opportunity for merging IU and medicine.

3.3 Involvement with other communities

Such a initiative, to be successful, must connect with the traditional medical research community, and with components of the health delivery system, perhaps mostly through interactions with major medical research hospitals (e.g. Johns Hopkins, Mass General, Brigham and Women’s, Mayo Clinic, etc.).

3.4 Where do we go from here?

The IU community needs to coordinate with Satava and Jenkins, bogth to leverage interactions with the existing focus of their program, and to identify important, but orthogonal, directions for additional funding. Secondly, establishing some “bona fides” on the part of the IU community in this area could lend credence to the effort to establish a new initiative. To this end, ARPA sponsored pilot studies, in a small selected set of topics (especially those in which a “quick hit” can show the immediate relevance of the initiative) are one mechanism for leveraging funding in this area.