as) United States
`a2) Patent Application Publication co) Pub. No.: US 2012/0062557 Al
`
` Dillon et al. (43) Pub. Date: Mar. 15, 2012
`
`
`US 20120062557A1
`
`(54) SYSTEMS AND METHODS FOR
`PROCESSING AND DISPLAYING
`INTRA-ORAL MEASUREMENTDATA
`
`(75)
`
`Inventors:
`
`(73) Assignee:
`
`Robert F. Dillon, Bedford, NH
`(US); Olaf N. Krohg, Topsfield,
`MA (US); Andrew F. Vesper,
`Townsend, MA (US); Timothy I.
`Fillion, Bedford, MA (US)
`
`DIMENSIONAL PHOTONICS
`INTERNATIONAL,INC.,
`Wilmington, MA (US)
`
`(21) Appl. No.:
`
`13/217,629
`
`(22)
`
`Filed:
`
`Aug. 25, 2011
`
`Related U.S. Application Data
`(60) Provisional application No. 61/381,731, filed on Sep.
`10, 2010.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2011.01)
`G06T 15/00
`(2006.01)
`G06K 9/00
`(52) US. CMe ceeccececccssseecsseesssseessssseesssssesssnseessseeeen 345/419
`(57)
`ABSTRACT
`Provided are a system and methodfor generating and display-
`ing intra-oral measurement data. A measurementfield ofview
`ofan intra-oral scanning deviceis directedat a first region of
`an object scene to acquire image data related to the first
`region. Theintra-oral scanning device is moved from thefirst
`region along a path proximal to one or more surfaces of the
`object scene to a second region of the object scene. The
`intra-oral scanning device acquires image data corresponding
`to the object scene along the path. A set of 3D data is pre-
`sented in a display. 3D data are generated from the image data
`acquiredfor the first region of the object scene to the second
`region of the object scene. Presented in a window of the
`display is a current video image ofacquired image data ofthe
`object scene in the measurementfield of view. The current
`video image overlays a respective portion of a graphical rep-
`resentation of accumulated data of the set of 3D data.
`
`18
`
`SS
`
`410
`
`14
`
`IMAGE OVERLAY
`SYSTEM
`
`DISPLAY
`16
`
`Align EX1027
`Align v. 3Shape
`IPR2022-00144
`
`Align EX1027
`Align v. 3Shape
`IPR2022-00144
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 1 of 9
`
`US 2012/0062557 Al
`
`14
`
`IMAGE OVERLAY
`SYSTEM
`
`FIG. 1
`
`DISPLAY
`16
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 2 of 9
`
`US 2012/0062557 Al
`
`SCAN DIRECTION
`—_—
`
`fo
`
`42
`
`<< 20
`
`IMAGER
`24
`
`VIDEO
`INTERFACE
`34
`
`3D
`PROCESSOR
`32
`
`MEMORY
`36
`
`OPACITY
`ADJUSTOR
`40
`
`OVERLAY
`ENGINE
`38
`
`DISPLAY
`16
`
`
`
`FIG. 2
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 3 of 9
`
`US 2012/0062557 Al
`
`105
`
`POSITION SCANNING DEVICE AT /
`
`FIRST REGION OF DENTAL ARCH
`
`100
`
`110
`
`MOVE SCANNING DEVICE TOA
`SECOND REGION OF DENTAL
`ARCH
`
`115
`
`ACQUIRE IMAGE DATA FROM
`FIRST REGION TO SECOND
`REGION
`
`120 \\_||GENERATE SET OF 3D DATA
`
`FROM ACQUIRED IMAGE DATA
`
`128\
`
`DISPLAY GRAPHICAL
`
`REPRESENTATION OF 3D DATA
`
`130
`
`OVERLAY“LIVE”’VIDEO OF
`ACQUIRED IMAGE DATA ON
`GRAPHICAL REPRESENTATION
`
`OF 3D DATA
`
`FIG. 3
`
`

`

`Patent Application Publication
`
`Mar.15, 2012 Sheet 4 of 9
`
`US 2012/0062557 Al
`
`zoalOrt)oeaZz<oOa
`
`FIG. 4A
`
`

`

`Patent Application Publication
`
`Mar.15, 2012 Sheet 5 of 9
`
`US 2012/0062557 Al
`
`42B
`
`SCAN DIRECTION
`
`48
`
`
`
`ae\POreyer,..&iragtei
`
`NN—
`
`FIG. 4B
`
`
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 6 of 9
`
`US 2012/0062557 Al
`
`16
`
`FIG. 4C
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 7 of 9
`
`US 2012/0062557 Al
`
`205
`
`DISPLAY 3D DATA FROM
`MEASUREMENT SCAN
`
`3D DATA
`
`210 OVERLAY VIDEO IMAGE ON
` 215
`
`
`ADJUST OPACITY OF VIDEO
`IMAGE IN DISPLAY WINDOW
`
`200
`
`
`
`
`220 NN
`
`VIEW 3D DATA AND VIDEO
`IMAGE IN DISPLAY WINDOW
`
`FIG. 5
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 8 of 9
`
`US 2012/0062557 Al
`
`f
`
`305
`
`310
`
`315
`
`320
`
`325
`
`330
`
`335
`
`340 300
`
`DETERMINE THAT FOV OF
`MEASUREMENT DEVICE
`REGISTERS TO GRAPHICAL
`REPRESENTATION OF 3D DATA IN
`DISPLAY WINDOW
`
`POSITION FOV OF MEASUREMENT
`DEVICE AT FIRST SCAN STARTING
`LOCATION OF DENTAL ARCH AND
`ACQUIRE IMAGE DATA
`
`MOVE FOV OF MEASUREMENT
`DEVICE TO FIRST SCAN END
`POINT OF DENTAL ARCH AND
`ACQUIRE IMAGE DATA
`
`GENERATEFIRST 3D DATA FROM
`ACQUIRED IMAGE DATA AND
`DISPLAY 3D DATA
`
`OVERLAY VIDEO IMAGE OF
`CURRENT FOV ON GRAPHICAL
`REPRESENTATION OF 3D DATA
`
`ADJUST OPACITY OF VIDEO IMAGE
`
`MOVE FOV OF MEASUREMENT
`DEVICE TO SECOND SCAN
`STARTING LOCATION
`
`JOIN SECOND 3D DATA TO
`FIRST 3D DATA
`
`FIG. 6
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 9 of 9
`
`US 2012/0062557 Al
`
`
`
`64
`66
`
`FIG. 7
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`SYSTEMS AND METHODS FOR
`PROCESSING AND DISPLAYING
`INTRA-ORAL MEASUREMENT DATA
`
`RELATED APPLICATIONS
`
`[0001] This application claims the benefit of the earlier
`filing date of U.S. Provisional Patent Application Ser. No.
`61/381,731, filed Sep. 10, 2010 andtitled “Method of Data
`Processing and Display for a Three-Dimensional Intra-Oral
`Scanner,” the contents of which are incorporated herein by
`reference in their entirety.
`
`FIELD OF THE INVENTION
`
`[0002] The invention relates generally to three-dimen-
`sional imaging (3D) of an object surface. Moreparticularly,
`the invention relates to an apparatus and a methodfor gener-
`ating and displaying a graphical representation of a set of 3D
`data acquired during a scanning operation of an intra-oral
`cavity and overlaying the displayed graphical representation
`with a two-dimensional (2D) video image ofthe intra-oral
`cavity.
`
`BACKGROUND
`
`[0003] A dental or medical 3D camera or scanner, when
`part of an imaging system, can capture a series of 2D intensity
`images of one or more object surfaces in an object scene. In
`some systems, this is achieved by projecting structured light
`patterns onto the surface. A light pattern can be generated by
`projecting a pair of coherent optical beams onto the object
`surface and the resulting fringe pattern varied between suc-
`cessive 2D images. Alternatively, the projected light pattern
`maybea series of projected parallel lines generated using an
`intensity mask and the projected pattern shifted in position
`between successive 2D images. In still other types of 3D
`imaging systems, confocal imaging techniques andthe like
`are employed.
`[0004] A typical imaging system includes a wandor other
`handheld scanning device that a user manually directs at the
`object scene. During measurement of the object scene, the
`wand can be used to acquire a set of 3D data related to the
`object scene while the wandis in motion. In some applica-
`tions, multiple object surfaces are measured by positioning
`the wand to be in close proximity to the object surfaces.
`However, when the wandis positioned at one location of the
`object scene, some sections of the object scene may be
`obscured from view of the wand. For example, in dental
`applications, the presence of teeth, gingiva, or other dental
`features in a particular static view can obscure the view of
`other teeth. Accordingly, a clinician may acquire 3D data sets
`from various scans of a dental arch. A processing unit can
`register the overlapped regions of all 3D data sets acquired
`from the various scans to obtain a full 3D data set represen-
`tation ofall surfaces observed during the measurementpro-
`cedure.
`
`BRIEF SUMMARY
`
`In one aspect, a computer-implemented methodis
`[0005]
`provided for displaying intra-oral measurement data. A mea-
`surement field of view of an intra-oral scanning device is
`directedat a first region of an object scene to acquire image
`data related to the first region. The intra-oral scanning device
`is movedfrom thefirst region along a path proximalto one or
`more surfaces of the object scene to a second region of the
`
`object scene. The intra-oral scanning device acquires image
`data correspondingto the object scene along the path. A set of
`3D data is presented in a display. 3D data are generated from
`the image data acquiredfor the first region ofthe object scene
`to the second region of the object scene. Presented in a win-
`dow ofthe display is a current video image of acquired image
`data ofthe object scene in the measurementfield ofview. The
`current video image overlays a respective portion of a graphi-
`cal representation of accumulated data of the set of 3D data.
`[0006]
`In another aspect, a methodis provided for display-
`ing intra-oral measurementdata related to a dental arch. An
`intra-oral measurement device is positioned at a first scan
`starting point proximalto a first region of a dental arch. A
`measurement field of view of the intra-oral measurement
`
`device is directed at the first region of the dental arch. The
`intra-oral measurement device is moved from thefirst region
`along a path proximal to a surface of the dental arch to a first
`scan end point proximalto a second regionof the dental arch
`to acquire image data from thefirst scan starting point to the
`first scan end point. A set of 3D data generated from the
`acquired image data is displayed at a display. A video image
`of the acquired image data is overlaid on a respective portion
`of a graphical representation of accumulateddataofthe set of
`3D data. The 3D data is displayed in the window byadjusting
`an opacity level of the current video image.
`[0007]
`In another aspect, an image overlay system com-
`prises a 3D processor, a video interface, and an overlay
`engine. The 3D processor generates three-dimensional (3D)
`data from imagedata acquired in an intra-oral scan procedure.
`The videointerface outputs a current video image in response
`to receiving the image data. The overlay engine generates a
`graphical representation of accumulated data of the 3D data
`and that overlays a respective portion of the graphical repre-
`sentation in a display window.
`[0008]
`In another aspect, an orthodontic analysis system
`comprises a scanning device, an image overlay processor, and
`a display device. The scanning device acquires image data
`related to an object scene of an intra-oral cavity. The image
`overlay processor generates at least one of a current video
`image and 3D data from the acquired image data, and con-
`figures the current video image for overlay on a graphical
`representation of accumulated data of the 3D data. The dis-
`play device includes a windowfor displaying the video image
`on a respective portion of the graphical representation of the
`3D data.
`
`In another aspect, a computer program productis
`[0009]
`provided for displaying intra-oral measurement data. The
`computer program product comprises a computer readable
`storage medium having computer readable program code
`embodied therewith. The computer readable program code
`comprises computer readable program code configured to
`direct a measurementfield of view of an intra-oral scanning
`device ata first region ofan object scene to acquire image data
`related to the first region. The computer readable program
`code further comprises computer readable program code con-
`figured to acquire image data corresponding to the object
`scene along a path between the first region and a second
`region of the object scene. The computer readable program
`code further comprises computer readable program code con-
`figured to present a graphical representation of a set of three-
`dimensional
`(3D) data generated from the image data
`acquiredfor the first region of the object scene to the second
`region of the object scene. The computer readable program
`code further comprises computer readable program code con-
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`figured to present a current video imageof the object scene in
`the measurement field of view, wherein the current video
`imageoverlays a respective portion ofthe graphical represen-
`tation of accumulated data of the set of 3D data.
`
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWINGS
`
`[0010] The above and further advantages of this invention
`may be better understood by referring to the following
`description in conjunction with the accompanying drawings,
`in which like numerals indicate like structural elements and
`
`features in various figures. The drawings are not necessarily
`to scale, emphasis instead being placed uponillustrating the
`principles of the invention.
`[0011] FIG.1isaschematic diagram ofan environmentfor
`acquiring image data related to dental structures during an
`intra-oral scanning operation and displaying images from the
`acquired image data, in accordance with an embodiment;
`[0012] FIG.2 isa block diagram ofthe scanning device and
`the image overlay system of FIG. 1, in accordance with an
`embodiment;
`[0013]
`FIG. 3 is a flowchart of a method for presenting
`dental structure image data acquired during a scanning opera-
`tion, in accordance with an embodiment;
`[0014]
`FIGS. 4A-4C show a measurementfield of view at
`various positions along an upper dental arch during a mea-
`surementscan ofa dental arch and also show displayed results
`from the measurement scan, in accordance with an embodi-
`ment;
`FIG. 5 is a flowchart of a method for displaying
`[0015]
`intra-oral measurementdata related to a dental arch, in accor-
`dance with an embodiment;
`[0016]
`FIG. 6 is a flowchart of a method for obtaining
`three-dimensional (3D) surface data ofa dental arch, in accor-
`dance with an embodiment; and
`[0017]
`FIG. 7 showsa set of intra-oral measurement data
`displayed in a window, in accordance with an embodiment.
`
`DETAILED DESCRIPTION
`
`[0018] The present teaching will now be described in more
`detail with reference to exemplary embodiments thereof as
`shown in the accompanying drawings. While the present
`teaching is described in conjunction with various embodi-
`ments and examples,it is not intendedthat the present teach-
`ing be limited to such embodiments. On the contrary, the
`present teaching encompassesvariousalternatives, modifica-
`tions and equivalents, as will be appreciated by those of skill
`in the art. Those of ordinary skill in the art having accessto the
`teaching herein will recognize additional implementations,
`modifications and embodiments, as well as otherfields ofuse,
`which are within the scope of the present disclosure as
`described herein.
`
`[0019] The methods of the present invention may include
`any of the described embodiments or combinations of the
`described embodiments in an operable manner. In brief over-
`view, systems and methodsofthe present inventive concepts
`produce a display of a graphical representation of 3D data as
`well as a video image overlaid on a portion of the graphical
`representation. The video image can bea real-time or near-
`real time 2D video stream that can correspond to a measure-
`mentfield of view of the clinician. The 3D data and the video
`imageare generated from a set of 2D imagedata taken during
`a measurementscan of an object scene, for example, a dental
`
`arch in an intra-oral cavity. A graphical representation of the
`acquired 3D data is generated during a 3D measurement scan
`of the object scene. As additional 3D data is acquired and
`displayed during the measurement scan, the graphical repre-
`sentation can grow. The video image is displayed in a window
`at a portion of the display andis overlaid on a portion of the
`graphical representation of the 3D data.
`[0020] As described above, a clinician such as a dentist
`typically performsdifferent scans of a set of teeth in order to
`obtain a full and “final” 3D data set representation ofall
`surfaces observed during the measurement procedure. To
`achieve this, the clinician maneuvers a scanner wand in the
`patient’s mouth and acquires the 3D data in a preferred
`sequenceso that the final 3D dataset resulting from all the 3D
`data more accurately represents the dental arch. In particular,
`a first 3D data set is generated and additional sequences of
`second 3D data are subsequently joined to the first 3D data
`set. Individual scan segments are used to acquire subsets of
`3D data for the final 3D data set, which can include a point
`cloud, a wireframe representation, or other 3D surface repre-
`sentations. For example, data acquisition starts by acquiring
`data from a measurementfield of view at the patient’s left
`back molar of the upper dental arch. The wandis then moved
`along the archto the right back molar. The clinician can then
`position the wand so that the measurement field of view
`includesa portionofthefirst 3D data set, and new 3D data are
`acquired that overlap a portion ofthefirst 3D data set. Pref-
`erably, the 3D measurement system provides an affirmative
`visual or audible indication to the clinician when the new 3D
`data for the real-time position of the measurementfield of
`view “locks on”to the display of the surface for the first 3D
`data set. The newly-acquired 3D data are then registered or
`joinedto the first 3D data and serve asthestart of a different
`scan segmentfor the arch. The wandis then rotated aboutits
`primary axis and movedsothat anew portion ofthe surface of
`the arch is within the measurementfield of view and 3D data
`are acquired. The wandis then maneuveredbythe clinician so
`that the measurementfield of view moves along a new seg-
`mentof the arch surface.
`
`[0021] Acclinician can have difficulty locking the measure-
`mentfield of view to the display of the surfaceforthe first 3D
`data set dueto difficulty interpreting a graphical display of 3D
`data, for example, due to a lack of shading, color, and other
`viewing characteristics. Thus, 3D data for the subsequent
`scan segment may not properly “register”to the existing 3D
`data in the commoncoordinate reference system. The acqui-
`sition of additional 3D data can be interrupted, for example,
`when switching between different scans, where the additional
`3D points cannot be joined.
`[0022] The present invention permits the scanning wand to
`be repositioned by the clinician to a position such that the
`current video imagesubstantially matches a portion ofthe 3D
`data displayed in the same display window as the video
`image. Once 3D data represented in the displays are deter-
`minedto be similarin their region of overlap, the acquisition
`of 3D measurement data resumes and subsequently deter-
`mined 3D data are joinedto the previously acquired 3D data.
`Providing a live video image for a current measurementfield
`of view can therefore facilitate the interpretation of the pre-
`viously acquired and displayed 3D data. Thus, an intra-oral
`measurement procedure can be performed moreefficiently,
`resulting in less discomfort to the patient and shorter acqui-
`sition times.
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`[0023] FIG.1 is a schematic diagram of an environment10
`for acquiring imagedatarelated to dental structures during an
`intra-oral scanning operation and displaying images from the
`acquired image data, in accordance with an embodiment. The
`environment 10 includes a scanning device 12, an image
`overlay system 14, and a display 16. The scanning device 12,
`the image overlay system 14, and the display 16 can each
`include a processor, a memory, and an I/O interface. The
`memory can include removable and/or non-removablestor-
`age media implemented in accordance with methods and
`technologies known to those of ordinary skill in the art for
`storing data. Program code, such as program code of an
`operating system, graphics, applications, and the like for
`execution by the processoris stored ina memory. Data related
`to 2D and/or 3D images can likewise be stored in a memory.
`[0024] The scanning device 12 is constructed to measure
`one or more object surfaces by scanning an object scene. In
`doing so, the scanning device 12 captures 2D imagedata that
`is used to generate 2D and/or 3D images for display. The
`scanning device 12 can be an intra-oral scanner such as a
`wand. When the scanning device 12 is inserted in the intra-
`oral cavity 20 of a patient 18, a dentist, a hygienist, or other
`clinician can conduct a 3D scan of a dental arch or other
`intra-oral structures.
`
`191, incorporated herein by reference in its entirety, where a
`technique referred to as Accordion Fringe Interferometry
`(AFI) can be usedfor high precision 3D measurements based
`on interferometric fringe projection.
`[0029] The imager 24 can include a charge-coupled device
`(CCD) camera or other imaging device that includes one or
`more image sensors, a photodetector array, or related elec-
`tronic components (not shown) that receive one or more
`beams28of optical radiation reflected or otherwise received
`from the surface of the illuminated dental arch 20. As is
`well-known to those of ordinary skill in the art, electrical
`signals can be generated by the imager 24, for example, an
`array of photodetectors or CCD readers (not shown),
`in
`responseto the received radiation. The imager 24 can capture
`the signals used to process a two dimensional image of the
`dental arch 20, and generate an image of the projection pat-
`tern after reflection of the pattern off the surface of the dental
`arch 20.
`
`[0030] The images acquired by the imager 24 include 3D
`information related to the surface of the object 20. The
`images, morespecifically, 2D imagedata includingthis infor-
`mation, are output to a3D processor 32. The 3D processor can
`generate 3D data from the received image data.
`[0031] The image overlay system 14 can include the 3D
`[0025] The acquired image data is output to the image
`processor 32, a video interface 34, a memory 36, an overlay
`overlay system 14, which converts the image data into a set of
`engine 38, and an opacity adjuster 40. All of these elements
`3D data. The image overlay system 14 processes the 3D data
`can execute entirely on the image overlay system 14. Alter-
`to generate one or more wireframe representations, point
`natively, some elements can execute on the image overlay
`clouds, or other 3D object surface representations.
`system 14 or other computer platform, while other elements
`execute on the scanning device 12, the display 16, or a remote
`[0026] The image overlay system 14 overlays a portion of
`computer. For example, the 3D processor 32 can be part ofthe
`the graphical representation ofthe 3D data with a real-time or
`image overlay system 14 as shown in FIG.2. Alternatively,
`near-real time 2D video imageofa section of a current object
`the 3D processor 32 can bepart of the scanning device 12. In
`scene in the measurementfield of view for a current position
`another example, the overlay engine 38 can be part of the
`of the scanning device 12. The video imageis presented in a
`windowofthe display 16. At any time during active scanning,
`image overlay system 14 as shownin FIG.2, or can alterna-
`
`the video image can showatrue grayscale or color image of tively be part of the display 16.
`the oral cavity within thefield of view of the scanning device
`[0032] The 3D processor 32 can receive signals related to
`12, while the 3D display shows accumulated surface data of
`one or more 2D images from the imager 24. For example, the
`the scanning operation. During operation, the point cloud or
`signals can includes information on the intensity ofthe light
`object surface representation appears to grow withinthe dis-
`received at each photodetector in the imager 24. In response,
`play 16 while the live video imageallowsthe clinician to see
`the 3D processor 32 can calculate the distance from the
`the portion of the oral cavity currently being measured.
`imager 24, for example, a detector array, of the scanning
`[0027] The display 16 preferably includes its own proces-
`device 12 to the surface of the dental arch 20 for each pixel
`sor and memory for providing a graphical user interface to
`based on the intensity values for the pixel in the series of
`display the graphical representation of the 3D data generated
`generated 2D images. Thus, the 3D processor 32 createsa set
`from the acquired image data. The display 16 can include a
`of 3D coordinates that can be displayed as a point cloud or a
`touchscreen or a monitor coupledto the image overlay system
`surface map that represents the object surface. The 3D pro-
`14 for receiving 2D and/or 3D image feeds from the image
`cessor 32 communicates with the memory 36 for storage of
`overlay system 14. The display 16 includes a window for
`3D data generated during a measurement procedure. A user
`displaying 2D video of an object scene overlaid on the 3D
`interface (not shown) allows an operator such asa clinician to
`representation of the object scene.
`provide operator commands andto observe the acquired 3D
`information in a near-real time manner. For example, the
`[0028]
`FIG.2 is a block diagram of the scanning device 12
`operator can observe a display of the growth of a graphical
`and the image overlay system 14 ofFIG.1, in accordance with
`representation of the point cloud as different regions of the
`an embodiment. The scanning device 12 includes a projector
`surface of the dental arch 20 are measured and additional 3D
`22 and an imager 24. The projector 22 includes a radiation
`source, for example, a light or laser source, for projecting an
`optical radiation pattern 26, for example, light, onto a dental
`arch in a patient’s mouth, which includesa set of teeth, gums,
`and the like. In an embodiment, the projector 18 is a fringe
`projectorthat emits optical radiation, for example, two diver-
`gent optical beams generated from a coherent light source
`(e.g. a laser diode), where they generate a fringe pattern. A
`surface of the dental arch is illuminated with the fringe pat-
`tern. A related approach is described in U.S. Pat. No. 5,870,
`
`measurement data are acquired.
`[0033] The video interface 34 can likewise receive 2D
`image data from the scanning device 12. The 2D image data
`can be the samedata as that received by the 3D processor 32,
`for example, from the imager 24. Optionally, the video inter-
`face 34 can receive 2D imagedata from a different source, for
`example, a video camera instead of the scanning device 12.
`The video interface 24 processes and outputs from the
`received 2D image data a real-time or“live” video image of
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`particular, a change in transparencies of the video image and
`3D data allowsthe clinician to maneuverthe scanning device
`to substantially matcha live video image with a portion ofthe
`displayed, previously generated 3D data. Once the twodis-
`plays are determinedto be substantially similarin their region
`of overlap, the acquisition of a 3D measurement scan can
`resume, and subsequently determined 3D datasets, e.g. point
`clouds or surface maps, can bestitched to an existing 3D data
`set.
`
`the surfaces being measured to the overlay engine 38. In
`particular, the image data received by the video interface 34
`correspondsto a portion ofthe dental arch in the measurement
`field of view 42 of the scanning device 12.
`[0034] As described above, the memory 36 can store the 3D
`data and/or 2D data. The memory 36 can also include
`machine executable instructions enabling the 3D processor
`32 to process the points in a point cloud and/or generate a
`single mesh surface configuration representing the scanned
`object, i.e., the dental arch 20 forthe display 16. The memory
`36 can include volatile memory, for example, RAM and the
`like, and/or non-volatile memory, for example, ROM,flash
`memory, and the like. The memory can include removable
`and/or non-removable storage media implemented in accor-
`dance with methods and technologies known to those of
`ordinary skill in the art for storing data. Stored in the memory
`can include program code, such as program codeof an oper-
`ating system executed by the image generator 34, the 3D
`processor32,or other processorsofthe image overlay system
`14.
`
`FIG. 3 isa flowchart of a method 100 for presenting
`[0038]
`dental structure image data acquired during a scanning opera-
`tion. In describing the method 100, reference is also made to
`FIGS. 1 and 2. The method 100 can be governed by instruc-
`tionsthatare stored in a memory and executed by a processor
`of the scanning device 12, the image overlay system 14,
`and/or the display 16. The method 100 is described herein as
`being performed on a dental arch. In other embodiments, the
`method 100 can be performed on virtually any object.
`[0039] Aclinician such asa dentist initiates the method 100
`by positioning (step 105) the scanning device 12 at a starting
`point of the dental arch so that a structured light pattern
`[0035] The overlay engine 38 can bepart of a display pro-
`generated from the scanning device 12 illuminatesafirst
`cessor or graphicaluser interface for displaying the 3D data as
`region of the dental arch, for example, a back portion of an
`a graphical representation on the display 16. The overlay
`occlusal surface of the dental arch at one end of the arch.
`engine 38 includesa first input that receives a 2D video feed
`from the video interface 34 and a second input that receives
`3D data from the 3D processor. The overlay engine 38 can
`overlay or superimpose real-time or near-real time video
`images of the 2D feed corresponding to the dentition within
`the field ofview ofthe imager 24 overlaid on at least a portion
`of the graphical representation of the 3D data, for example,
`one or more point clouds or object surface representation.
`[0036] During an operation, the 3D data and video images
`can be output from the overlay engine 38 to the display 16,
`and can be configured by the overlay engine 38 such that the
`3D data is displayed as a point cloud, object surface repre-
`sentation on the display, and the video images are displayed in
`a window on the display. In a preferred embodiment, the
`video imageis displayed in a windowcentered in the viewing
`area ofthe display 16. The window in which the video image
`is displayed can have a rectangular or square shape that is
`substantially smaller than the rectangular shape of the view-
`ing area of the display 16. The display 16 can include a user
`interface (not shown) for presenting the received images in
`grayscale, color, or other user-defined format, and for permit-
`ting a user to enter commands, view images, or other well-
`known functions for performing a scanning and/or display
`operation.
`[0037] A portion of the 3D data can also be available for
`presentation in the window. The opacity adjuster 40 can be
`configured to change the opacity level of the 3D data and/or
`the video image in the window.For example, the video image
`can be presented as being substantially opaque andthe por-
`tion of the graphical representation of the 3D data in the
`window canbetransparent to preventa display of the graphi-
`cal representation of the 3D data in a region of overlay iden-
`tified by the window.The opacity adjuster 40 can reduce the
`opacity of the video image and/or reduce the transparency of
`the graphical representationofthe set of 3D data in the region
`ofoverlay identified by the window.In this manner, the video
`image and the graphical representation ofthe set of 3D data in
`the region of overlay can be simultaneously viewed in the
`window. This feature can be beneficial when a clinician
`attemptsto “stitch” orjoin a 3D point cloud or surface map to
`a previously acquired 3D point cloud or surface map. In
`
`Image data for providing 2D and/or 3D images can be
`acquired for the illuminated portion of the surface of the
`dental arch atthefirst region. The scanning device can include
`a2D imager 24 with a small measurementfield ofview (FOV)
`(e.g., 13 mmx10 mm)relativeto the full arch. The imager 24
`can include a camera that captures 2D imagesofthe surface
`and displays them on the display 16. The camera can be a
`video camera and present the 2D imagesasa real time or near
`real time video stream.
`
`[0040] The clinician can move (step 110) the scanning
`device 12 along a path proximalto a surface ofthe dental arch
`to a secondregion ofthe dental arch. Here, the structuredlight
`pattern generated from the scanning device 12 can illuminate
`a remainderof the surface of the dental arch along the path,
`for example, the occlusal surface. Image data can therefore be
`acquired (step 115) for the remainder of the occlusal surface
`from the first region to the second region. A set of 3D data can
`be generated (step 120) from the acquired image data. A
`graphical representation ofthe 3D data can be displayed (step
`125) as a wireframe representation, an object map, and the
`like.
`
`[0041] The 2D video can be overlaid (step 130) on the
`graphical representation of 3D data. The 2D video can be
`provided from the acquired image data, or other 2D image
`data, for example, acquired from a CCD camera. The video
`image can correspondsto acurrent measurementfield ofview
`directed at a region of object scene for receiving image data
`related to that region.
`[0042]
`FIGS. 4A-4C show a measurementfield of view at
`various positions along an upper dental arch during a mea-
`surementscan ofa dental arch and also show displayedresults
`from the measurement scan according to the method of FIG.
`3, in accordance with an embodiment. The measurement scan
`can be performed using a handheld image-capturing device
`such as the scanning device 12 of FIGS. 1 and 2. During the
`measurementscan, a set of 2-D imagesof the dental arch 20
`can be acquired. In FIG. 4A, a measurement scan can be
`initiated by acquiring image data from within a measurement
`field of view 42A at the patient’s right back region of t