1111111111111111 IIIIII IIIII 11111 1111111111 111111111111111 111111111111111 1111111111 11111111
`US 20070171220Al
`
`c19) United States
`c12) Patent Application Publication
`Kriveshko
`
`c10) Pub. No.: US 2007 /0171220 Al
`Jul. 26, 2007
`(43) Pub. Date:
`
`(54) THREE-DIMENSIONAL SCAN RECOVERY
`
`(52) U.S. Cl. .............................................................. 345/419
`
`(76)
`
`Inventor:
`
`Ilya A. Kriveshko, Littleton, MA (US)
`
`Correspondence Address:
`STRATEGIC PATENTS P.C ..
`C/O PORTFOLIOIP
`P.O. BOX 52050
`MINNEAPOLIS, MN 55402 (US)
`
`(21) Appl. No.:
`
`11/383,623
`
`(22) Filed:
`
`May 16, 2006
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 11/337,182, filed on
`Jan. 20, 2006.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G06T 15100
`
`(2006.01)
`
`,.,--110
`
`108~ □
`
`(57)
`
`ABSTRACT
`
`A scanning system that acquires three-dimensional images
`as an incremental series of fitted three-dimensional data sets
`is improved by testing for successful incremental fits in real
`time and providing a variety of visual user cues and process
`modifications depending upon the relationship of newly
`acquired data to previously acquired data. The system may
`be used to aid in error-free completion of three-dimensional
`scans. The methods and systems described herein may also
`usefully be employed to scan complex surfaces including
`occluded or obstructed surfaces by maintaining a continuous
`three-dimensional scan across separated subsections of the
`surface. In one useful dentistry application, a full three(cid:173)
`dimensional surface scan may be obtained for two dental
`arches in occlusion.
`
`100
`,,,.-..,.
`
`0001
`
`Exhibit 1106 page 1 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 1 of 7
`
`US 2007/0171220 Al
`
`\0
`0
`
`tj" _./.,.
`0
`
`.......
`LL
`
`\
`
`'i /.
`'/
`:
`:
`:
`
`I
`/ I
`I
`
`/ u
`
`(
`00
`0
`
`0002
`
`Exhibit 1106 page 2 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 2 of 7
`
`US 2007/0171220 Al
`
`200
`\'i,,...
`
`( BEGIN SCAN
`
`202
`
`ACQUIRE CANDIDATE
`FRAME FROi'V1 SENSOR
`
`\__ 2
`
`04
`
`DERIVE 3D DATA FROM
`CANDIDATE SENSED DATA
`208
`~_,.,.
`
`"·~206
`
`---~ NO
`. , ; ; ; , - - - - - - - - - - - - - ,
`
`2·10
`"-
`
`'
`/
`HAVE A
`'
`
`212
`
`CORRELATE CANDIDATE TO
`REFERENCE AND DERIVE
`INCREMENTAL CAMERA TRANSFORM
`
`YES
`
`STITCH
`RECOVERY
`MODE
`
`' 226
`
`APPEND INCREMENTAL CAMERA MOTION
`TO TOTAL CAMEF!A MOTION
`
`,
`
`21
`
`'---
`
`,-:..
`0
`
`TRANSFORM CANDIDATE 3D DATA
`BY TOTAL CAMEF~A MOTION
`
`"\__218
`
`RECORD CANDIDATE 30 DATA AND
`SET CANDIDATE AS NEW REFEF{Ef\lCE
`
`222
`--.. .... ___
`~.,.,.,,-_...,.,.,.-
`- IS SCAN FINISH E[)?..>--N_O ______ ___,
`
`YES
`\. END SCAN J
`
`f
`
`'
`
`,_228
`
`I
`
`Fig. 2
`
`0003
`
`Exhibit 1106 page 3 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 3 of 7
`
`US 2007/0171220 Al
`
`300
`
`ENTER STITCH
`RECOVERY MODE
`
`302
`
`LOCATE A SUIT/-\BLE
`RECENT REFERENCE
`FRAME
`
`ASSIGN IMAGE FROM
`NEW REFERENCE FRAME
`TO BE DISPLAYED
`ON SCREEN
`
`·304
`
`306
`
`ACQUIRE CANDIDATE
`FRAME FROM SENSOR
`
`'-.308
`
`DISPLAY IMAGE FROM
`CANDIDATE ON SCREEN
`COMBINED WITH
`REFERENCE IMAGE
`
`310
`
`DERIVE 3D DATA FHOM
`CANDIDATE SENSED DATA
`
`312
`
`/
`
`3·14
`NO
`
`----~
`
`CORRELATE CANDIDATE
`TO REFERENCE AND
`DEHIVE INCREMENTAL
`CAMERA TRANSFORM
`
`316
`
`NO
`
`LEAVE STITCH
`RECOVERY MODE
`
`Fig. 3
`
`0004
`
`Exhibit 1106 page 4 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 4 of 7
`
`US 2007/0171220 Al
`
`Fig~ 4
`
`0005
`
`Exhibit 1106 page 5 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 5 of 7
`
`US 2007/0171220 Al
`
`402
`r
`
`400
`
`.r
`
`404 _/
`
`" 602
`
`Fig~ 6
`
`400
`-~-✓
`
`0006
`
`Exhibit 1106 page 6 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 6 of 7
`
`US 2007/0171220 Al
`
`_402
`
`400
`
`"-
`702
`7
`
`-----~---.. ;~ .. .
`/.,.,,,.,.
`. .-, :.· · ....... ",
`·.·· .· .. •.: ...
`.. ·. :-. -:·.··--4 04 : :: :". ; ... '-\
`·.• ·. "-:;_",
`
`,
`
`\
`
`Fig~ 8
`
`_402
`
`0007
`
`Exhibit 1106 page 7 of 22
`DENTAL IMAGING
`
`

`

`Patent Application Publication Jul. 26, 2007 Sheet 7 of 7
`
`US 2007/0171220 Al
`
`Fig. 11
`
`Fig. 12
`
`0008
`
`Exhibit 1106 page 8 of 22
`DENTAL IMAGING
`
`

`

`US 2007 /0171220 Al
`
`Jul. 26, 2007
`
`1
`
`THREE-DIMENSIONAL SCAN RECOVERY
`
`BACKGROUND
`
`[0001] 1. Field of the Invention
`
`[0002] The invention relates to three-dimensional scan(cid:173)
`ning, and more particularly to techniques for resuming a
`three-dimensional scan after losing data acquisition.
`
`[0003] 2. Description of the Related Art
`
`[0004] A wide range of techniques exist for acqumng
`three-dimensional data from a subject. These techniques
`range from techniques using structured laser illumination or
`other controlled stimulus (such as x-rays, ultrasound, or
`magnetic resonance) to techniques that operate directly on
`video data captured from one or more cameras. While many
`criteria can be devised for sorting and comparing these
`techniques, one broad category employs continuous scan(cid:173)
`in which
`incremental
`three-dimensional data
`is
`ning
`acquired and assembled into a full three-dimensional model.
`
`[0005]
`In this latter category, regardless of the particular
`sensing technology, a scanning process can be divided into
`abstract steps of incremental data capture, incremental deri(cid:173)
`vation of three-dimensional data, and registration of the
`incremental data to a common coordinate system. The final
`registration step brings the incremental data together into a
`single three-dimensional model of a scan subject. Typical
`implementations separate acquisition from registration, with
`registration being performed in a post-processing step after
`completion of the data acquisition. While this permits rela(cid:173)
`tively exhaustive processing, it poses a significant disadvan(cid:173)
`tage because accuracy and completeness of the overall scan
`cannot be evaluated until after the scan has been terminated.
`Unrecoverable errors or gaps in incremental data cannot be
`identified and fixed without initiating a new scan-possibly
`a full scan to completely replace the defective results. In
`commercial applications, this may create delays and incon(cid:173)
`venience that increase the effective cost of scanning.
`
`[0006]
`In certain instances, recovery of lost or unusable
`scan segments may be addressed by using robotics, inde(cid:173)
`pendent references, or other techniques that can reliably
`position a scanning device within a global coordinate sys(cid:173)
`tem. In addition to imposing further equipment costs, this
`approach can only be used to continue a previous scan if the
`subject of the scan has also retained its position and orien(cid:173)
`tation within the same global coordinate system.
`
`[0007] There remains a need for processing techniques
`that identify and aid in recovery from errors in incremental
`scans. There also remains a need for real time feedback
`systems to support recovery from a lost scanning sequence
`while the scan is in progress.
`
`SUMMARY
`
`[0008] A scanning system that acquires three-dimensional
`images as an incremental series of fitted three-dimensional
`data sets is improved by testing for successful incremental
`fits in real time and providing a variety of visual user cues
`and process modifications depending upon the relationship
`of newly acquired data to previously acquired data. The
`system may be used to aid in error-free completion of
`three-dimensional
`scans. The methods and systems
`described herein may also usefully be employed to scan
`
`complex surfaces including occluded or obstructed surfaces
`by maintaining a continuous three-dimensional scan across
`separated subsections of the surface. In one useful dentistry
`application, a full three-dimensional surface scan may be
`obtained for two dental arches in occlusion.
`[0009]
`In one aspect, a method disclosed herein may
`include acquiring three-dimensional surface data from a
`subject as a sequence of three-dimensional images, each one
`of the sequence of three-dimensional images fitted to at least
`one previous one of the sequence of three-dimensional
`images, and acquiring an image set for a next three-dimen(cid:173)
`sional image. When the image set can be converted into the
`next three-dimensional image and the next three-dimen(cid:173)
`sional image can be fitted to the three-dimensional surface
`data, the three-dimensional surface data and the next three(cid:173)
`dimensional image are superimposed onto a two-dimen(cid:173)
`sional image of the subject in a display. Further, when the
`image set cannot be converted into the next three-dimen(cid:173)
`sional image or the next three-dimensional image cannot be
`fitted to the three-dimensional surface data, a recover mode
`is entered. The recover mode may include superimposing a
`current two-dimensional image of the subject onto a previ(cid:173)
`ous two-dimensional image of the subject in the display. The
`two-dimensional image represents a view of the subject
`from a position from which one of the sequence of three(cid:173)
`dimensional images was acquired. Additionally the current
`two-dimensional image represents a view of the subject
`from a position from which the image set was acquired. The
`recover mode further may include acquiring at least one
`subsequent three-dimensional image and test fitting it to one
`or more of the sequence of three-dimensional images.
`[0010] The method may further include adding the next
`three-dimensional image to the three-dimensional surface
`data when the image set can be converted into the next
`three-dimensional image and the next three-dimensional
`image can be fit to the three-dimensional surface data. In the
`method, the previous two-dimensional image and the current
`two-dimension image may include video frames. The
`method may further include manually entering the recover
`mode in response to a user input. Also, the method may
`further include automatically entering the recover mode in
`response to a detection of a lost acquisition. The detection of
`a lost acquisition may include, for example, a detection of
`motion blur. The detection of lost acquisition may also
`include a detection of an insufficient recovery of three(cid:173)
`dimensional data. In the method, the recover mode may
`include the step of highlighting a region on the subject for
`attempting a recovery. Also the highlighted region may
`include a plurality of previous ones of the sequence of
`three-dimensional images. The recover mode may include
`the step of test fitting at least one subsequent three-dimen(cid:173)
`sional image to one or more additional images acquired
`during the recover mode and test fitting may include select(cid:173)
`ing a reference image from the sequence of three-dimen(cid:173)
`sional images based on a suitability for test fitting. The
`suitability for test fitting may be based upon an entropy of
`image data in the previous three-dimensional image. Also,
`the suitability for test fitting may be based upon an amount
`of three-dimensional data in the previous three-dimensional
`image. Also, or instead, the suitability for test fitting may be
`based upon a placement of three-dimensional data within a
`range and a field of view of an imaging device. Determining
`the suitability for test fitting may be based upon a temporal
`proximity of the at least one subsequent three-dimensional
`
`0009
`
`Exhibit 1106 page 9 of 22
`DENTAL IMAGING
`
`

`

`US 2007 /0171220 Al
`
`Jul. 26, 2007
`
`2
`
`image to one or more of the sequence of three-dimensional
`images. Determining the suitability for test fitting may be
`based upon a spatial proximity of the at least one subsequent
`three-dimensional image to one or more of the sequence of
`three-dimensional images. The recover mode may include
`the step of manually selecting one or more of the sequence
`of three-dimensional images. The recover mode may also
`include selecting a reference image from a beginning or an
`end of the sequence of three-dimensional images.
`
`[0011]
`In another aspect, a method is disclosed herein may
`include providing three-dimensional surface data for a sub(cid:173)
`ject, and acquiring an image set for a three-dimensional
`image of the subject. When the image set cannot be fitted to
`the three-dimensional surface data, the method may enter a
`landing mode which may include superimposing a current
`two-dimensional image of the subject onto a previous two(cid:173)
`dimensional image of the subject in the display, the current
`two-dimensional image representing a view of the subject
`from a position from which the image set was acquired, and
`the previous two-dimensional image representing a view of
`the subject from which at least a portion of the three(cid:173)
`dimensional surface data was acquired; acquiring at least
`one subsequent three-dimensional image; fitting the at least
`one subsequent three-dimensional image to a second three(cid:173)
`dimensional surface reconstruction; and test fitting the at
`least one subsequent three-dimensional image to the three(cid:173)
`dimensional surface data. When the image set can be fitted
`to the three-dimensional surface data, the method superim(cid:173)
`poses the three-dimensional surface data and the next three(cid:173)
`dimensional image onto a two-dimensional image of the
`subject in a display and adds the second three-dimensional
`surface reconstruction to the three-dimensional surface data.
`
`[0012]
`In the method, the previous two-dimensional image
`and the current two-dimension image may include video
`frames. The method may further include manually entering
`the landing mode in response to a user input. In the landing
`mode, the method may include highlighting a region on the
`subject for attempting a landing. The highlighted region may
`include a portion of the three-dimensional surface data. Test
`fitting may include selecting a reference image from the
`three-dimensional surface data based on the reference
`image's suitability for test fitting. Suitability for testing may
`be based on any one or more of the following: entropy of
`image data in the previous three-dimensional image, amount
`of three-dimensional data in the previous three-dimensional
`image, placement of three-dimensional data within a range
`and a field of view of an imaging device, and spatial
`proximity of the image set to a portion of the three(cid:173)
`dimensional surface data. The method may further include
`manually selecting a region of the three-dimensional surface
`data for test fitting. Test fitting may include selecting a
`reference image from a beginning or an end of a sequence
`of three-dimensional images used to construct the three(cid:173)
`dimensional surface data. The previous two-dimensional
`image and the current two-dimension image may include
`video frames. The method may further include manually
`entering the recover mode in response to a user input.
`Additionally, the method may further include manually
`selecting a region from the three-dimensional surface data
`for test fitting.
`
`[0013]
`In another aspect, a system is disclosed herein may
`include a scanning device that acquires three-dimensional
`surface data from a subject as a sequence of three-dimen-
`
`sional images, each one of the sequence of three-dimen(cid:173)
`sional images fitted to at least one previous one of the
`sequence of three-dimensional images. The scanning device
`may include an acquisition mode in which one or more
`additional three-dimensional images are acquired and added
`to the three-dimensional surface data. Additionally the
`device may include a recovery mode in which one or more
`additional three-dimensional images are test fit to one or
`more of the sequence of three-dimensional images to
`recover the acquisition mode, and a display may be adapted
`to provide one or more visual indicators distinguishing
`between the acquisition mode and the recovery mode.
`
`[0014]
`In the system, the display may be further adapted
`to provide guidance to an operator in manipulating the
`subject in the recovery mode.
`
`[0015]
`In another aspect, a method is disclosed herein
`including performing a first acquisition of a first set of
`three-dimensional data from a first subject, placing a second
`subject in a fixed orientation relative to the first subject, and
`performing a second acquisition of a second set of three(cid:173)
`dimensional data starting at a location on a surface of the
`first subject and including at least a portion of the second
`subject. In the method, the first subject and the second
`subject each may be a rigid body.
`
`[0016]
`In the method, the location on the surface of the
`first subject may include a plurality of spatially proximate
`ones of a sequence of three-dimensional images used to
`derive the three-dimensional data from the first subject.
`Also, in the method, the location on the surface of the first
`subject may include a plurality of temporally proximate ones
`of a sequence of three-dimensional images used to derive the
`three-dimensional data from the first subject. Alternatively,
`in the method, the first subject may include a first dental
`arch. At least a portion of the second set of three-dimen(cid:173)
`sional data may be fitted to at least a portion of the first set
`of three-dimensional data to provide a unified set of three(cid:173)
`dimensional data.
`
`[0017] The method may further include terminating the
`second acquisition at a location on a surface of the second
`subject, removing the first subject, and performing a third
`acquisition of three-dimensional data starting at the location
`on the surface of the second subject. The location on the
`surface of the second subject may include a plurality of
`spatially proximate ones of a sequence of three-dimensional
`images used to derive the three-dimensional data from the
`second subject. Alternatively, the location on the surface of
`the second subject may include a plurality of temporally
`proximate ones of the sequence of three-dimensional images
`used to derive the three-dimensional data from the second
`subject.
`
`[0018] The method may further include combining the
`first set of three-dimensional data, the second set of three(cid:173)
`dimensional data, and the third set of three-dimensional data
`into a unified set of three-dimensional data. In the method,
`the second subject may include a second dental arch, and the
`fixed orientation may include the first dental arch and the
`second dental arch in occlusion.
`
`[0019]
`In another aspect, a system disclosed herein may
`include an acquisition means for performing a first acquisi(cid:173)
`tion of a first set of three-dimensional data from a first
`subject, and a positioning means for placing a second subject
`
`0010
`
`Exhibit 1106 page 10 of 22
`DENTAL IMAGING
`
`

`

`US 2007 /0171220 Al
`
`Jul. 26, 2007
`
`3
`
`in a fixed orientation relative to the first subject. In the
`system, the acquisition means may include a means for
`performing a second acquisition of a second set of three(cid:173)
`dimensional data starting at a location on a surface of the
`first subject and including at least a portion of the second
`subject. In the system, the first subject and the second
`subject each may be a rigid body.
`
`[0020]
`In the system, the location on the surface of the first
`subject may include a plurality of spatially proximate ones
`of a sequence of three-dimensional images used to derive the
`three-dimensional data from the first subject. Alternatively,
`the location on the surface of the first subject may include a
`plurality of temporally proximate ones of a sequence of
`three-dimensional images used to derive the three-dimen(cid:173)
`sional data from the first subject. The first subject may
`include a first dental arch. At least a portion of the second set
`of three-dimensional data may be fitted to at least a portion
`of the first set of three-dimensional data to provide a unified
`set of three-dimensional data. The system may further
`include a control means for terminating the second acqui(cid:173)
`sition at a location on a surface of the second subject, the
`positioning means may include a means for removing the
`first subject; and the acquisition means may include a means
`for performing a third acquisition of three-dimensional data
`starting at the location on the surface of the second subject.
`The location on the surface of the second subject may
`include a plurality of spatially proximate ones of a sequence
`of three-dimensional images used to derive the three-dimen(cid:173)
`sional data from the second subject. Alternatively, the loca(cid:173)
`tion on the surface of the second subject may include a
`plurality of temporally proximate ones of the sequence of
`three-dimensional images used to derive the three-dimen(cid:173)
`sional data from the second subject. The system may further
`include a computing means for combining the first set of
`three-dimensional data, the second set of three-dimensional
`data, and the third set of three-dimensional data into a
`unified set of three-dimensional data. In the system, the
`second subject may include a second dental arch, and the
`fixed orientation may include the first dental arch and the
`second dental arch in occlusion.
`
`[0021]
`In another aspect, a method disclosed herein may
`include placing a first subject in a fixed orientation relative
`to a second subject, acquiring a first set of three-dimensional
`data including a first portion of the first subject and a first
`portion of the second subject, acquiring a second set of
`three-dimensional data from the first subject in isolation
`starting at a location on a surface of the first subject and
`including a second portion of the first subject different from
`the first portion of the first subject, and combining the first
`set of three-dimensional data with the second set of three(cid:173)
`dimensional data.
`
`[0022]
`In the method, the first subject and the second
`subject may be a rigid body. Alternatively, the first subject
`may include a first dental arch. The location on the surface
`of the first subject may include a plurality of spatially
`proximate ones of a sequence of three-dimensional images
`used to derive the three-dimensional data from the first
`subject. Alternatively, the location on the surface of the first
`subject may include a plurality of temporally proximate ones
`of a sequence of three-dimensional images used to derive the
`three-dimensional data from the first subject. The method
`may further include acquiring a third set of three-dimen(cid:173)
`sional data from the second subject in isolation starting at a
`
`location on a surface of the second subject and including a
`second portion of the second subject, and combining the
`third set of three-dimensional data with the first and second
`sets of three-dimensional data. The location on the surface of
`the second subject may include a plurality of spatially
`proximate ones of a sequence of three-dimensional images
`used to derive the three-dimensional data from the second
`subject. Alternatively, the location on the surface of the
`second subject may include a plurality of temporally proxi(cid:173)
`mate ones of the sequence of three-dimensional images used
`to derive the three-dimensional data from the second sub(cid:173)
`ject. The second subject may include a second dental arch,
`and the fixed orientation may include the first dental arch
`and the second dental arch in occlusion.
`
`[0023]
`In another aspect, a method disclosed herein may
`include providing a first set of three-dimensional data from
`a subject, scanning the subject to acquire one or more
`additional three-dimensional images, test fitting each one of
`the one or more additional three-dimensional images to the
`set of three dimensional data, and upon a successful fit,
`adding the one or more additional three-dimensional images
`to the set of three-dimensional data.
`
`[0024] The method may include fitting each new one of
`the one or more additional three-dimensional images to one
`another to provide a second set of three-dimensional data,
`and upon a successful fit, adding the second set of three(cid:173)
`dimensional data to the first set of three-dimensional data.
`The three-dimensional data may include three-dimensional
`surface data. Alternatively, the one or more additional three(cid:173)
`dimensional images may include surface data derived from
`two-dimensional image sets of the subject. In the method,
`the set of three-dimensional data may include three-dimen(cid:173)
`sional surface data acquired from a subject as a sequence of
`three-dimensional images, each one of the sequence of
`three-dimensional images fitted to at least one previous one
`of the sequence of three-dimensional images.
`
`[0025]
`In another aspect, a system disclosed herein may
`include a storage means for providing a first set of three(cid:173)
`dimensional data from a subject, an acquisition means for
`scanning the subject to acquire one or more additional
`three-dimensional images, and a computing means for test
`fitting each one of the one or more additional three-dimen(cid:173)
`sional images to the set of three dimensional data. The
`computing means may further include a means for detecting
`a successful fit and, upon detecting a successful fit, may add
`the one or more additional three-dimensional images to the
`set of three-dimensional data.
`
`[0026]
`In the system, the computing means may include a
`means for fitting each new one of the one or more additional
`three-dimensional images to one another to provide a second
`set of three-dimensional data, and upon a successful fit, may
`add the second set of three-dimensional data to the first set
`of three-dimensional data. In the system, the three-dimen(cid:173)
`sional data may include three-dimensional surface data.
`Alternatively in the system, the one or more additional
`three-dimensional images may include surface data derived
`from two-dimensional image sets of the subject. The set of
`three-dimensional data may include three-dimensional sur(cid:173)
`face data acquired from a subject as a sequence of three(cid:173)
`dimensional images, each one of the sequence of three(cid:173)
`dimensional images may be fitted to at least one previous
`one of the sequence of three-dimensional images.
`
`0011
`
`Exhibit 1106 page 11 of 22
`DENTAL IMAGING
`
`

`

`US 2007 /0171220 Al
`
`Jul. 26, 2007
`
`4
`
`[0027]
`In another aspect, a computer program product
`disclosed herein may include computer executable code
`which may be embodied on a computer readable medium
`that, when executing on one or more computing devices,
`performs the steps of providing a first set of three-dimen(cid:173)
`sional data from a subject, scanning the subject to acquire
`one or more additional three-dimensional images, test fitting
`each one of the one or more additional three-dimensional
`images to the set of three dimensional data, and upon a
`successful fit, adding the one or more additional three(cid:173)
`dimensional images to the set of three-dimensional data.
`
`[0028] The computer program product may further
`include code to perform the steps of fitting each new one of
`the one or more additional three-dimensional images to one
`another to provide a second set of three-dimensional data,
`and upon a successful fit, adding the second set of three(cid:173)
`dimensional data to the first set of three-dimensional data.
`
`[0029]
`In the computer program product, the three-dimen(cid:173)
`sional data may include three-dimensional surface data.
`Alternatively, in the computer program product, the one or
`more additional three-dimensional images may include sur(cid:173)
`face data derived from two-dimensional image sets of the
`subject. The set of three-dimensional data may include
`three-dimensional surface data acquired from a subject as a
`sequence of three-dimensional images, each one of the
`sequence of three-dimensional images may be fitted to at
`least one previous one of the sequence of three-dimensional
`images.
`
`[0030]
`In another aspect, a method disclosed herein may
`include providing a first set of three-dimensional data from
`a subject, scanning the subject to acquire one or more
`additional three-dimensional images, test fitting each one of
`the one or more additional three-dimensional images to the
`first set of three-dimensional data in real time, and providing
`real time visual feedback to a user relating to the test fitting.
`
`[0031]
`In the method, providing real time visual feedback
`may include superimposing the first set of three-dimensional
`data and the one or more additional three-dimensional
`images on a video image of the subject when a test fit is
`successful. Providing real time visual feedback may include
`displaying one or more navigation cues when a test fit is
`unsuccessful. When the test fitting is unsuccessful, the
`method may further include fitting each new one of the one
`or more additional three-dimensional images to one another
`to provide a second set of three-dimensional data. Upon a
`successful fit, the method may further include adding the
`second set of three-dimensional data to the first set of
`three-dimensional data. The three-dimensional data may
`include three-dimensional surface data. The one or more
`additional three-dimensional images may include surface
`data derived from two-dimensional image sets of the sub(cid:173)
`ject. Additionally, the set of three-dimensional data may
`include three-dimensional surface data acquired from a
`subject as a sequence of three-dimensional images, each one
`of the sequence of three-dimensional images fitted to at least
`one previous one of the sequence of three-dimensional
`images.
`
`[0032]
`In another aspect, a system disclosed herein may
`include a storage means for providing a first set of three(cid:173)
`dimensional data from a subject, an acquisition means for
`scanning the subject to acquire one or more additional
`three-dimensional images, a computing means for test fitting
`
`each one of the one or more additional three-dimensional
`images to the first set of three-dimensional data in real time
`and a display means for providing real time visual feedback
`to a user relating to the test fitting.
`
`[0033]
`In the system, providing real time visual feedback
`may include superimposing the first set of three-dimensional
`data and the one or more additional three-dimensional
`images on a video image of the subject when a test fit is
`successful. Alternatively, providing real time visual feed(cid:173)
`back may include displaying one or more navigation cues
`when a test fit is unsuccessful. The computing means may
`include means for, when the test fitting is unsuccessful,
`fitting each new one of the one or more additional three(cid:173)
`dimensional images to one another to provide a second set
`of three-dimensional data, and upon a successful fit, adding
`the second set of three-dimensional data to the first set of
`three-dimensional data. Alternatively, the three-dimensional
`data may include three-dimensional surface data. The one or
`more additional three-dimensional images may include sur(cid:173)
`face data derived from two-dimensional image sets of the
`subject. The set of three-dimensional data may include
`three-dimensional surface data acquired from a subject as a
`sequence of three-dimensional images, each one of the
`sequence of three-dimensional images fitted to at least one
`previous one of the sequence of three-dimensional images.
`
`[0034]
`In another aspect, a computer program product
`disclosed herein may include computer executable code
`embodied on a computer readable medium that, when
`executing on one or more computing devices, performs the
`steps of providing a first set of three-dimensional data from
`a subject, scanning the subject to acquire one or more
`additional three-dimensional images, test fitting each one of
`the one or more additional three-dimensional images to the
`first set of three-dimensional data in real time, and providing
`real time visual feedback to a user relating to the test fitting.
`
`[0035]
`In the computer program product, providing real
`time visual feedback may include superimposing the first set
`of three-dimensional data and the one or more additional
`three-dimensional images on a video image of the subject
`when a test fit is successful. Alternatively, providing real
`time visual feedback may include displaying one or more
`navigation cues when a test fit is unsuccessful. The computer
`program product may further include computer code that
`performs the steps of, when the test fitting is unsuccessful,
`fitting each new one of the one or more additional three(cid:173)
`dimensional ima