Knowledge Based 3D
 Medical Image Segmentation

	Tina Kapur
	MIT Artificial Intelligence Laboratory
	http://www.ai.mit.edu/~tkapur

Outline

	Goal of Segmentation
	Applications
	Why is segmentation difficult?
	My method for segmentation of MRI
	Future Work

The Goal of Segmentation

The Goal of Segmentation

Applications of Segmentation

	Image Guided Surgery

Applications of Segmentation

	Image Guided Surgery

Applications of Segmentation

	Image Guided Surgery
	Surgical Simulation

Applications of Segmentation

	Image Guided Surgery
	Surgical Simulation

Applications of Segmentation

	Image Guided Surgery
	Surgical Simulation
	Neuroscience Studies
	Therapy Evaluation

Limitations of Manual Segmentation

	slow (up to 60 hours per scan)
	variable (up to 15% between experts)
	[Warfield 95, Kaus98]

The Automatic Segmentation Challenge

	An automated segmentation method needs to reconcile
		Gray-level appearance of tissue
		Characteristics of imaging modality
		Geometry of anatomy

How to Segment? i.e. Issues in
Segmentation of Anatomy

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models
		Parametric [Vannier]
		Non-Parametric [Gerig]
		Point distribution Models [Cootes]
		Texture [Mumford]

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models
	Imaging Modality Models

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models
	Imaging Modality Models
		MRI inhomogeneity [Wells]

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models
	Imaging Modality Models
	Anatomy Models: Shape, Geometric/Spatial

How to Segment? i.e. Issues in
Segmentation of Anatomy

	Tissue Intensity Models
	Imaging Modality Models
	Anatomy Models: Shape, Geometric/Spatial
		PCA [Cootes and Taylor, Gerig, Duncan, Martin]
		Landmark Based [Evans]
		Atlas [Warfield]

Typical Pipeline for Segmentation of Brain MRI

Typical Pipeline for Segmentation of Brain MRI

Typical Pipeline for Segmentation of Brain MRI

Contributions of Thesis

	Developed an integrated Bayesian Segmentation Method for MRI that incorporates de-noising and global geometric knowledge using priors into EM-Segmentation
	Applied integrated Bayesian method to segmentation of Brain and Knee MRI.

Contributions of Thesis

	The Priors
		de-noising: novel use of a Mean-Field Approximation to a Gibbs random field in conjunction with EM-Segmentation (EM-MF)
		geometric: novel statistical description of global spatial relationships between structures, used as a spatially varying prior in EM-Segmentation

Background to My Work

	Expectation-Maximization Algorithm
	EM-Segmentation

Expectation-Maximization

	Relevant Literature:
		[Dempster, Laird, Rubin 1977]
		[Neal 1998]

Expectation-Maximization (what?)

	Search Algorithm
	for Parameters of a Model
	to Maximize Likelihood of Data
	Data: some observed, some unobserved

Expectation-Maximization (how?)

	Initial Guess of Model Parameters
	Re-estimate Model Parameters:
		E Step: compute PDF for hidden variables, given observations and current model parameters
		M Step: compute ML model parameters assuming pdf for hidden variables is correct

Expectation-Maximization (how exactly?)

	Notation
		Observed Variables:
		Hidden Variables :
		Model Parameters:

Expectation-Maximization (how exactly?)

	Initial Guess:
	Successive Estimation of
		E Step:
		M Step:

Expectation-Maximization

	Summary/Intuition:
		If we had complete data, maximize likelihood
		Since some data is missing, approximate likelihood with its expectation
		Converges to local maximum of likelihood

EM-Segmentation [Wells 1994]

	Observed Signal is modeled as a product of the true signal and a corrupting gain field due to the imaging equipment
	Expectation-Maximization is used on log-transformed observations for iterative estimation of
		tissue classification
		corrupting bias field (inhomogeneity correction)

Slide 32

Slide 33

EM-Segmentation [Wells 1994]

	Observed Variables
				log transformed intensities in image
	Hidden Variables
				indicator variables for classification
	Model Parameters
				the slowly varying corrupting bias field
		(                      refer to variables at voxel s in image)

EM-Segmentation [Wells 1994]

	Initial Guess:
	Successive Estimation of
		E Step:
		M Step:

EM-Segmentation [Wells 1994]

	Initial Guess:
	Successive Estimation of
		E Step:
		M Step:

Situating My Work

	Prior in EM-Segmentation:
		Independent and Spatially Stationary
	My contribution is addition of two priors:
		a spatially stationary Gibbs prior to model local interactions between neighbors (thermal noise)
		spatially varying prior to model global relationships between geometry of structures

The Gibbs Prior

	Gibbs Random Field (GRF)
		natural way to model piecewise homogeneous phenomena
		used in image restoration [Geman&Geman 84]
		Probability Model on a lattice
		Partially Relaxes independence assumption to allow interactions between neighbors

EM-MF Segmentation: EM + Gibbs Prior

	We model tissue classification W as a Gibbs random field:

EM-MF Segmentation: Gibbs Prior on Classification

	We model tissue classification W as a Gibbs random field:

EM-MF Segmentation: Gibbs Prior on Classification

	To fully specify the Gibbs model:
		define neighborhood system         as a first order neighborhood system i.e. 6 closest voxels
		use       to define

EM-MF Segmentation: Gibbs form of Posterior

	Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior:

EM-MF Segmentation: Gibbs form of Posterior

	Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior:

EM-MF Segmentation

	For E-Step: Need values for

EM-MF Segmentation

	For E-Step: Need values for
	Cannot compute directly from Gibbs form

EM-MF Segmentation

	For E-Step: Need values for
	Cannot compute directly from Gibbs form
	Note

EM-MF Segmentation

	For E-Step: Need values for
	Cannot compute directly from Gibbs form
	Note
	Can approximate
		Mean-Field Approximation to GRF

Mean-Field Approximation

	Deterministic Approximation to GRF [Parisi84]
		the mean/expected value of a GRF is obtained as a solution to a set of consistency equations
	Update Equation is obtained using derivative of partition function with respect to the external field g. [Elfadel 93]
	Used in image reconstruction [Geiger, Yuille, Girosi 91]

Mean-Field Approximation to Posterior GRF

Summary of EM-MF Segmentation

	Modeled piecewise homogeneity of tissue using a Gibbs prior on classification
	Lead to Gibbs form for Posteriors
	Posterior Probabilities in E-Step are approximated as a Mean-Field solution

EM-MF Results

	Application: Brain MRI
		white matter, gray matter, fluid/air, skin/scalp
	Results
	Comparison with Manual Segmentation

Some Results

Some Results

More Results

Posterior Probabilities (EM)

Posterior Probabilities (EM-MF)

Results

Modeling Global Geometric Relationships between Structures

Modeling Global Geometric Relationships between Structures

	Relative Geometry Models
	Motivate Using Knee MRI
	Brain MRI Example

Segmented Knee MRI

Motivation

	Primary Structures
				image well
				easy to segment
	Secondary Structures
				image poorly
				relative to primary

Relative Geometric Prior Approach

	Select primary/secondary structures
	Measure geometric relation between primary and secondary structures from training data
	Given novel image
		segment primary structures
		use geometric relation as prior on secondary structure in EM-MF Segmentation

Segment Primary Structures:
 Femur, Tibia

Status

	Have Bone
	Want Cartilage

Measure Geometric Relationship between Primary and Secondary Structures

	Using primitives such as
		distances between surfaces
		local normals of primary structures
		local curvature of primary structures
		etc.

Measure Geometric Relationship between Primary and Secondary Structures

Estimate of

Status

	Have Bone
	Have spatial relation between Bone and Cartilage
	Need Cartilage

Use Relative Geometric Prior in EM Segmentation

	Replace stationary prior with relative geometric prior:

Results: Segmentation of
Femoral & Tibial Cartilage

Relative Geometric Priors for Brain Tissue

	Prior Estimation
				Select primary structures (boundary of skin, ventricles)
				Estimate
	Using Prior in Segmentation
				Segment primary structures: skin, ventricles
				Use                                                       as geometric prior

Estimate

Resultant Segmentation

Posterior Probabilities

In Summary

	Incorporated robustness to thermal noise by using Mean-Field Approximation to Gibbs model in conjunction with EM Segmentation.  Applied to Brain MRI.
	Introduced Relative-Geometry Models and applied to Brain and Knee MRI.

Future Work

	Further development of Relative-Geometry Models:
		Automatic selection of primary/secondary structures
		Additional primitives for Spatial Relationships

STOP

Class Conditional Density

Unified Bayesian Segmentation Method

Rest of Talk:

	1. The Unified Segmentation Method
	2. Two Priors
	3. Results on Brain, Knee Segmentation
	4. Conclusions

The Gibbs Prior

	To fully specify the Gibbs model:
		define neighborhood system         as a first order neighborhood system i.e. 6 closest voxels
		use       to define

Components of EM Framework

	Measurement models for tissue
	Prior models for tissue
	Model for bias field
		piecewise smooth

Addition of Two Priors

Gibbs Model

Gibbs Model

	probability model on  a lattice
	independence assumption is partially relaxed
	spatial range of interaction is local neighborhood

Slide 86

Gibbs Model

	probability model on  a lattice
	independence assumption is partially relaxed
	spatial range of interaction is local neighborhood

Contributions of Thesis

Proposed MRI Segmentation Method

	Bayesian Statistical Classification Scheme that uses Expectation-Maximization
	Replaces pipeline with Priors on intensity and geometry

Proposed MRI Segmentation Method

	Previous Work [Wells 1994, 1996]
		derived as a special case
		spatially stationary, independent priors
		piecewise smooth inhomogeneity model
	This Work:
		locally interacting prior for intensity
		spatially varying prior for geometry

Next

	Background on Expectation-Maximization

Computation of

	1. Use Bayes’ rule, and independence between     and      to write:
	2. Specify Measurement Models                            as Gaussian

Computation of

	3. Specify Prior on Classification           as a Gibbs Random Field.
	4. Gibbs Prior + Gaussian Measurement Model imply                 is also a GRF.
	5. Approximate                         using Mean-Field Solution for the GRF.

Computation of

	Recap:
		1. Bayes’ Rule to rewrite …
		2. Gaussian Measurement Models
		Ö 3. Gibbs form of Prior
		Ö 4. Gibbs form of Posterior
		Ö 5. Mean-Field Approximation to Gibbs form

The Gibbs Prior

	Tissue-class interaction Matrix
	Stationary Prior

The Mean-Field Solution

	Gibbs models can be solved using
		[Metropolis 1953], [Geman and Geman 1984], [Besag 1986]  etc.
	We use Mean-Field Approximation to estimate the expected value of the posterior GRF as a solution to a set of consistency equations.

The Mean-Field Approximation

	Deterministic Approximation
	Update Equation is obtained using derivative of partition function with respect to the external field g.

The Mean-Field Solution

EM-MF Summary

	Initial Guess:
	Successive Estimation of
		E Step:  Estimate                                as
			Mean-Field Solution to a Gibbs Random Field
		M Step:  Compute           same as [Wells 1996]