1111111111111111 IIIIII IIIII 111111111111111 11111 111111111111111 lllll 111111111111111 11111111
`US 20120062557Al
`
`c19) United States
`c12) Patent Application Publication
`Dillon et al.
`
`c10) Pub. No.: US 2012/0062557 Al
`Mar. 15, 2012
`(43) Pub. Date:
`
`(54) SYSTEMS AND METHODS FOR
`PROCESSING AND DISPLAYING
`INTRA-ORAL MEASUREMENT DATA
`
`(75)
`
`Inventors:
`
`(73) Assignee:
`
`Robert F. Dillon, Bedford, NH
`(US); OlafN. Krohg, Topsfield,
`MA (US); Andrew F. Vesper,
`Townsend, MA (US); Timothy I.
`Fillion, Bedford, MA (US)
`
`DIMENSIONAL PHOTONICS
`INTERNATIONAL, INC.,
`Wilmington, MA (US)
`
`(21) Appl. No.:
`
`13/217,629
`
`(22) Filed:
`
`Aug. 25, 2011
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/381,731, filed on Sep.
`10, 2010.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G06T 15100
`(2011.01)
`G06K 9/00
`(2006.01)
`(52) U.S. Cl. ........................................................ 345/419
`ABSTRACT
`(57)
`
`Provided are a system and method for generating and display(cid:173)
`ing intra-oral measurement data. A measurement field of view
`of an intra-oral scanning device is directed at a first region of
`an object scene to acquire image data related to the first
`region. The intra-oral scanning device is moved from the first
`region along a path proximal to one or more surfaces of the
`object scene to a second region of the object scene. The
`intra-oral scanning device acquires image data corresponding
`to the object scene along the path. A set of 3D data is pre(cid:173)
`sented in a display. 3D data are generated from the image data
`acquired for the first region of the object scene to the second
`region of the object scene. Presented in a window of the
`display is a current video image of acquired image data of the
`object scene in the measurement field of view. The current
`video image overlays a respective portion of a graphical rep(cid:173)
`resentation of accumulated data of the set of 3D data.
`
`1Q
`
`C_
`
`IMAGE OVERLAY
`SYSTEM
`H
`
`DISPLAY
`1.§.
`
`0001
`
`Exhibit 1030 page 1 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 1 of 9
`
`US 2012/0062557 Al
`
`10
`
`C_
`
`IMAGE OVERLAY
`SYSTEM
`14
`
`DISPLAY
`.1§
`
`FIG.1
`
`0002
`
`Exhibit 1030 page 2 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 2 of 9
`
`US 2012/0062557 Al
`
`SCAN DIRECTION
`
`42
`
`..,-/ 20
`
`26
`
`PROJECTOR
`22
`
`12.
`
`IMAGER
`24
`
`VIDEO
`INTERFACE
`34
`
`30
`PROCESSOR
`32
`
`MEMORY
`36
`
`OPACITY
`ADJUSTOR
`40
`
`OVERLAY
`ENGINE
`38
`
`14
`
`DISPLAY
`1§_
`
`FIG. 2
`
`0003
`
`Exhibit 1030 page 3 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 3 of 9
`
`US 2012/0062557 Al
`
`/100
`
`105
`
`110
`
`FIRST REGION OF DENTAL ARCH
`
`"- POSITION SCANNING DEVICE AT
`l
`"- SECOND REGION OF DENTAL
`l
`
`MOVE SCANNING DEVICE TO A
`
`ARCH
`
`115
`
`"-
`
`120"-
`
`125"-
`
`ACQUIRE IMAGE DATA FROM
`FIRST REGION TO SECOND
`REGION
`
`l
`
`GENERATE SET OF 3D DATA
`FROM ACQUIRED IMAGE DATA
`
`t
`
`DISPLAY GRAPHICAL
`REPRESENTATION OF 3D DATA
`
`l
`
`130
`
`"-
`
`OVERLAY "LIVE"VIDEO OF
`ACQUIRED IMAGE DATA ON
`GRAPHICAL REPRESENTATION
`OF 3D DATA
`
`FIG. 3
`
`0004
`
`Exhibit 1030 page 4 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 4 of 9
`
`US 2012/0062557 Al
`
`46
`
`SCAN DIRECTION
`
`20
`✓
`
`..-../16
`
`52
`
`FIG. 4A
`
`0005
`
`Exhibit 1030 page 5 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 5 of 9
`
`US 2012/0062557 Al
`
`42B
`
`SCAN DIRECTION
`
`20
`✓
`
`48
`
`✓ 16
`
`54,.__
`
`50
`
`FIG. 4B
`
`0006
`
`Exhibit 1030 page 6 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 6 of 9
`
`US 2012/0062557 Al
`
`~ 16
`
`50
`
`FIG. 4C
`
`0007
`
`Exhibit 1030 page 7 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 7 of 9
`
`US 2012/0062557 Al
`
`200
`
`205
`
`210
`
`215
`
`DISPLAY 3D DATA FROM
`MEASUREMENT SCAN
`
`OVERLAY VIDEO IMAGE ON
`3D DATA
`
`ADJUST OPACITY OF VIDEO
`IMAGE IN DISPLAY WINDOW
`
`220"-
`
`VIEW 3D DATA AND VIDEO
`IMAGE IN DISPLAY WINDOW
`
`FIG. 5
`
`0008
`
`Exhibit 1030 page 8 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 8 of 9
`
`US 2012/0062557 Al
`
`300
`
`/
`
`305
`
`310
`
`315
`
`320
`
`325
`
`330
`
`335
`
`340
`
`POSITION FOV OF MEASUREMENT
`DEVICE AT FIRST SCAN STARTING
`LOCATION OF DENTAL ARCH AND
`ACQUIRE IMAGE DATA
`
`MOVE FOV OF MEASUREMENT
`DEVICE TO FIRST SCAN END
`POINT OF DENTAL ARCH AND
`ACQUIRE IMAGE DATA
`
`GENERATE FIRST 3D DATA FROM
`ACQUIRED IMAGE DATA AND
`DISPLAY 3D DATA
`
`OVERLAY VIDEO IMAGE OF
`CURRENT FOV ON GRAPHICAL
`REPRESENTATION OF 3D DATA
`
`ADJUST OPACITY OF VIDEO IMAGE
`
`MOVE FOV OF MEASUREMENT
`DEVICE TO SECOND SCAN
`STARTING LOCATION
`
`DETERMINE THAT FOV OF
`MEASUREMENT DEVICE
`REGISTERS TO GRAPHICAL
`REPRESENTATION OF 3D DATA IN
`DISPLAY WINDOW
`
`JOIN SECOND 3D DATA TO
`FIRST 3D DATA
`
`FIG. 6
`
`0009
`
`Exhibit 1030 page 9 of 17
`DENTAL IMAGING
`
`

`

`Patent Application Publication Mar. 15, 2012 Sheet 9 of 9
`
`US 2012/0062557 Al
`
`16
`
`62
`
`/ 5 0
`
`/
`
`64
`66
`
`FIG. 7
`
`0010
`
`Exhibit 1030 page 10 of 17
`DENTAL IMAGING
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`1
`
`SYSTEMS AND METHODS FOR
`PROCESSING AND DISPLAYING
`INTRA-ORAL MEASUREMENT DATA
`
`RELATED APPLICATIONS
`
`[0001] This application claims the benefit of the earlier
`filing date of U.S. Provisional Patent Application Ser. No.
`61/381,731, filed Sep. 10, 2010 and titled "Method of Data
`Processing and Display for a Three-Dimensional Intra-Oral
`Scanner," the contents of which are incorporated herein by
`reference in their entirety.
`
`FIELD OF THE INVENTION
`
`[0002] The invention relates generally to three-dimen(cid:173)
`sional imaging (3D) of an object surface. More particularly,
`the invention relates to an apparatus and a method for gener(cid:173)
`ating and displaying a graphical representation of a set of 3D
`data acquired during a scanning operation of an intra-oral
`cavity and overlaying the displayed graphical representation
`with a two-dimensional (2D) video image of the intra-oral
`cavity.
`
`BACKGROUND
`
`[0003] A dental or medical 3D camera or scanner, when
`part of an imaging system, can capture a series of2D intensity
`images of one or more object surfaces in an object scene. In
`some systems, this is achieved by projecting structured light
`patterns onto the surface. A light pattern can be generated by
`projecting a pair of coherent optical beams onto the object
`surface and the resulting fringe pattern varied between suc(cid:173)
`cessive 2D images. Alternatively, the projected light pattern
`may be a series of projected parallel lines generated using an
`intensity mask and the projected pattern shifted in position
`between successive 2D images. In still other types of 3D
`imaging systems, confocal imaging techniques and the like
`are employed.
`[0004] A typical imaging system includes a wand or other
`handheld scanning device that a user manually directs at the
`object scene. During measurement of the object scene, the
`wand can be used to acquire a set of 3D data related to the
`object scene while the wand is in motion. In some applica(cid:173)
`tions, multiple object surfaces are measured by positioning
`the wand to be in close proximity to the object surfaces.
`However, when the wand is positioned at one location of the
`object scene, some sections of the object scene may be
`obscured from view of the wand. For example, in dental
`applications, the presence of teeth, gingiva, or other dental
`features in a particular static view can obscure the view of
`other teeth. Accordingly, a clinician may acquire 3 D data sets
`from various scans of a dental arch. A processing unit can
`register the overlapped regions of all 3D data sets acquired
`from the various scans to obtain a full 3D data set represen(cid:173)
`tation of all surfaces observed during the measurement pro(cid:173)
`cedure.
`
`BRIEF SUMMARY
`
`In one aspect, a computer-implemented method is
`[0005]
`provided for displaying intra-oral measurement data. A mea(cid:173)
`surement field of view of an intra-oral scanning device is
`directed at a first region of an object scene to acquire image
`data related to the first region. The intra-oral scanning device
`is moved from the first region along a path proximal to one or
`more surfaces of the object scene to a second region of the
`
`object scene. The intra-oral scanning device acquires image
`data corresponding to the object scene along the path. A set of
`3D data is presented in a display. 3D data are generated from
`the image data acquired for the first region of the object scene
`to the second region of the object scene. Presented in a win(cid:173)
`dow of the display is a current video image of acquired image
`data of the object scene in the measurement field of view. The
`current video image overlays a respective portion of a graphi(cid:173)
`cal representation of accumulated data of the set of3D data.
`In another aspect, a method is provided for display(cid:173)
`[0006]
`ing intra-oral measurement data related to a dental arch. An
`intra-oral measurement device is positioned at a first scan
`starting point proximal to a first region of a dental arch. A
`measurement field of view of the intra-oral measurement
`device is directed at the first region of the dental arch. The
`intra-oral measurement device is moved from the first region
`along a path proximal to a surface of the dental arch to a first
`scan end point proximal to a second region of the dental arch
`to acquire image data from the first scan starting point to the
`first scan end point. A set of 3D data generated from the
`acquired image data is displayed at a display. A video image
`of the acquired image data is overlaid on a respective portion
`of a graphical representation of accumulated data of the set of
`3D data. The 3D data is displayed in the window by adjusting
`an opacity level of the current video image.
`In another aspect, an image overlay system com(cid:173)
`[0007]
`prises a 3D processor, a video interface, and an overlay
`engine. The 3D processor generates three-dimensional (3D)
`data from image data acquired in an intra-oral scan procedure.
`The video interface outputs a current video image in response
`to receiving the image data. The overlay engine generates a
`graphical representation of accumulated data of the 3D data
`and that overlays a respective portion of the graphical repre(cid:173)
`sentation in a display window.
`In another aspect, an orthodontic analysis system
`[0008]
`comprises a scanning device, an image overlay processor, and
`a display device. The scanning device acquires image data
`related to an object scene of an intra-oral cavity. The image
`overlay processor generates at least one of a current video
`image and 3D data from the acquired image data, and con(cid:173)
`figures the current video image for overlay on a graphical
`representation of accumulated data of the 3D data. The dis(cid:173)
`play device includes a window for displaying the video image
`on a respective portion of the graphical representation of the
`3D data.
`In another aspect, a computer program product is
`[0009]
`provided for displaying intra-oral measurement data. The
`computer program product comprises a computer readable
`storage medium having computer readable program code
`embodied therewith. The computer readable program code
`comprises computer readable program code configured to
`direct a measurement field of view of an intra-oral scanning
`device at a first region ofan object scene to acquire image data
`related to the first region. The computer readable program
`code further comprises computer readable program code con(cid:173)
`figured to acquire image data corresponding to the object
`scene along a path between the first region and a second
`region of the object scene. The computer readable program
`code further comprises computer readable program code con(cid:173)
`figured to present a graphical representation of a set of three(cid:173)
`dimensional (3D) data generated from the image data
`acquired for the first region of the object scene to the second
`region of the object scene. The computer readable program
`code further comprises computer readable program code con-
`
`0011
`
`Exhibit 1030 page 11 of 17
`DENTAL IMAGING
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`2
`
`figured to present a current video image of the object scene in
`the measurement field of view, wherein the current video
`image overlays a respective portion of the graphical represen(cid:173)
`tation of accumulated data of the set of3D data.
`
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWINGS
`
`[0010] The above and further advantages of this invention
`may be better understood by referring to the following
`description in conjunction with the accompanying drawings,
`in which like numerals indicate like structural elements and
`features in various figures. The drawings are not necessarily
`to scale, emphasis instead being placed upon illustrating the
`principles of the invention.
`[0011] FIG. 1 is a schematic diagram ofan environment for
`acquiring image data related to dental structures during an
`intra-oral scanning operation and displaying images from the
`acquired image data, in accordance with an embodiment;
`[0012] FIG. 2 is a block diagram of the scanning device and
`the image overlay system of FIG. 1, in accordance with an
`embodiment;
`[0013] FIG. 3 is a flowchart of a method for presenting
`dental structure image data acquired during a scanning opera(cid:173)
`tion, in accordance with an embodiment;
`[0014] FIGS. 4A-4C show a measurement field of view at
`various positions along an upper dental arch during a mea(cid:173)
`surement scan of a dental arch and also show displayed results
`from the measurement scan, in accordance with an embodi(cid:173)
`ment;
`[0015] FIG. 5 is a flowchart of a method for displaying
`intra-oral measurement data related to a dental arch, in accor(cid:173)
`dance with an embodiment;
`[0016] FIG. 6 is a flowchart of a method for obtaining
`three-dimensional (3D) surface data of a dental arch, in accor(cid:173)
`dance with an embodiment; and
`[0017] FIG. 7 shows a set of intra-oral measurement data
`displayed in a window, in accordance with an embodiment.
`
`DETAILED DESCRIPTION
`
`[0018] The present teaching will now be described in more
`detail with reference to exemplary embodiments thereof as
`shown in the accompanying drawings. While the present
`teaching is described in conjunction with various embodi(cid:173)
`ments and examples, it is not intended that the present teach(cid:173)
`ing be limited to such embodiments. On the contrary, the
`present teaching encompasses various alternatives, modifica(cid:173)
`tions and equivalents, as will be appreciated by those of skill
`in the art. Those of ordinary skill in the art having access to the
`teaching herein will recognize additional implementations,
`modifications and embodiments, as well as other fields of use,
`which are within the scope of the present disclosure as
`described herein.
`[0019] The methods of the present invention may include
`any of the described embodiments or combinations of the
`described embodiments in an operable manner. In brief over(cid:173)
`view, systems and methods of the present inventive concepts
`produce a display of a graphical representation of3D data as
`well as a video image overlaid on a portion of the graphical
`representation. The video image can be a real-time or near(cid:173)
`real time 2D video stream that can correspond to a measure(cid:173)
`ment field of view of the clinician. The 3 D data and the video
`image are generated from a set of2D image data taken during
`a measurement scan of an object scene, for example, a dental
`
`arch in an intra-oral cavity. A graphical representation of the
`acquired 3D data is generated during a 3D measurement scan
`of the object scene. As additional 3D data is acquired and
`displayed during the measurement scan, the graphical repre(cid:173)
`sentation can grow. The video image is displayed in a window
`at a portion of the display and is overlaid on a portion of the
`graphical representation of the 3D data.
`[0020] As described above, a clinician such as a dentist
`typically performs different scans of a set of teeth in order to
`obtain a full and "final" 3D data set representation of all
`surfaces observed during the measurement procedure. To
`achieve this, the clinician maneuvers a scanner wand in the
`patient's mouth and acquires the 3D data in a preferred
`sequence so that the final 3D data set resulting from all the 3D
`data more accurately represents the dental arch. In particular,
`a first 3D data set is generated and additional sequences of
`second 3D data are subsequently joined to the first 3D data
`set. Individual scan segments are used to acquire subsets of
`3D data for the final 3D data set, which can include a point
`cloud, a wireframe representation, or other 3D surface repre(cid:173)
`sentations. For example, data acquisition starts by acquiring
`data from a measurement field of view at the patient's left
`back molar of the upper dental arch. The wand is then moved
`along the arch to the right back molar. The clinician can then
`position the wand so that the measurement field of view
`includes a portion of the first 3D data set, and new 3D data are
`acquired that overlap a portion of the first 3D data set. Pref(cid:173)
`erably, the 3D measurement system provides an affirmative
`visual or audible indication to the clinician when the new 3D
`data for the real-time position of the measurement field of
`view "locks on" to the display of the surface for the first 3D
`data set. The newly-acquired 3D data are then registered or
`joined to the first 3D data and serve as the start of a different
`scan segment for the arch. The wand is then rotated about its
`primary axis and moved so that a new portion of the surface of
`the arch is within the measurement field of view and 3D data
`are acquired. The wand is then maneuvered by the clinician so
`that the measurement field of view moves along a new seg(cid:173)
`ment of the arch surface.
`[0021] A clinician can have difficulty locking the measure(cid:173)
`ment field of view to the display of the surface for the first 3D
`data set due to difficulty interpreting a graphical display of3D
`data, for example, due to a lack of shading, color, and other
`viewing characteristics. Thus, 3D data for the subsequent
`scan segment may not properly "register" to the existing 3D
`data in the common coordinate reference system. The acqui(cid:173)
`sition of additional 3D data can be interrupted, for example,
`when switching between different scans, where the additional
`3D points cannot be joined.
`[0022] The present invention permits the scanning wand to
`be repositioned by the clinician to a position such that the
`current video image substantially matches a portion of the 3D
`data displayed in the same display window as the video
`image. Once 3D data represented in the displays are deter(cid:173)
`mined to be similar in their region of overlap, the acquisition
`of 3D measurement data resumes and subsequently deter(cid:173)
`mined 3D data are joined to the previously acquired 3D data.
`Providing a live video image for a current measurement field
`of view can therefore facilitate the interpretation of the pre(cid:173)
`viously acquired and displayed 3D data. Thus, an intra-oral
`measurement procedure can be performed more efficiently,
`resulting in less discomfort to the patient and shorter acqui(cid:173)
`sition times.
`
`0012
`
`Exhibit 1030 page 12 of 17
`DENTAL IMAGING
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`3
`
`[0023] FIG.1 is a schematic diagram of an environment 10
`for acquiring image data related to dental structures during an
`intra-oral scanning operation and displaying images from the
`acquired image data, in accordance with an embodiment. The
`environment 10 includes a scanning device 12, an image
`overlay system 14, and a display 16. The scanning device 12,
`the image overlay system 14, and the display 16 can each
`include a processor, a memory, and an I/O interface. The
`memory can include removable and/or non-removable stor(cid:173)
`age media implemented in accordance with methods and
`technologies known to those of ordinary skill in the art for
`storing data. Program code, such as program code of an
`operating system, graphics, applications, and the like for
`execution by the processor is stored in a memory. Data related
`to 2D and/or 3D images can likewise be stored in a memory.
`[0024] The scanning device 12 is constructed to measure
`one or more object surfaces by scanning an object scene. In
`doing so, the scanning device 12 captures 2D image data that
`is used to generate 2D and/or 3D images for display. The
`scanning device 12 can be an intra-oral scanner such as a
`wand. When the scanning device 12 is inserted in the intra(cid:173)
`oral cavity 20 of a patient 18, a dentist, a hygienist, or other
`clinician can conduct a 3D scan of a dental arch or other
`intra-oral structures.
`[0025] The acquired image data is output to the image
`overlay system 14, which converts the image data into a set of
`3D data. The image overlay system 14 processes the 3D data
`to generate one or more wireframe representations, point
`clouds, or other 3D object surface representations.
`[0026] The image overlay system 14 overlays a portion of
`the graphical representation of the 3D data with a real-time or
`near-real time 2D video image of a section of a current object
`scene in the measurement field of view for a current position
`of the scanning device 12. The video image is presented in a
`window of the display 16. At any time during active scanning,
`the video image can show a true grayscale or color image of
`the oral cavity within the field of view of the scanning device
`12, while the 3D display shows accumulated surface data of
`the scanning operation. During operation, the point cloud or
`object surface representation appears to grow within the dis(cid:173)
`play 16 while the live video image allows the clinician to see
`the portion of the oral cavity currently being measured.
`[0027] The display 16 preferably includes its own proces(cid:173)
`sor and memory for providing a graphical user interface to
`display the graphical representation of the 3D data generated
`from the acquired image data. The display 16 can include a
`touchscreen or a monitor coupled to the image overlay system
`14 for receiving 2D and/or 3D image feeds from the image
`overlay system 14. The display 16 includes a window for
`displaying 2D video of an object scene overlaid on the 3D
`representation of the object scene.
`[0028] FIG. 2 is a block diagram of the scanning device 12
`and the image overlay system 14 of FIG. 1, in accordance with
`an embodiment. The scanning device 12 includes a projector
`22 and an imager 24. The projector 22 includes a radiation
`source, for example, a light or laser source, for projecting an
`optical radiation pattern 26, for example, light, onto a dental
`arch in a patient's mouth, which includes a set of teeth, gums,
`and the like. In an embodiment, the projector 18 is a fringe
`projector that emits optical radiation, for example, two diver(cid:173)
`gent optical beams generated from a coherent light source
`(e.g. a laser diode), where they generate a fringe pattern. A
`surface of the dental arch is illuminated with the fringe pat(cid:173)
`tern. A related approach is described in U.S. Pat. No. 5,870,
`
`191, incorporated herein by reference in its entirety, where a
`technique referred to as Accordion Fringe Interferometry
`(AFI) can be used for high precision 3D measurements based
`on interferometric fringe projection.
`[0029] The imager 24 can include a charge-coupled device
`(CCD) camera or other imaging device that includes one or
`more image sensors, a photodetector array, or related elec(cid:173)
`tronic components (not shown) that receive one or more
`beams 28 of optical radiation reflected or otherwise received
`from the surface of the illuminated dental arch 20. As is
`well-known to those of ordinary skill in the art, electrical
`signals can be generated by the imager 24, for example, an
`array of photodetectors or CCD readers (not shown), in
`response to the received radiation. The imager 24 can capture
`the signals used to process a two dimensional image of the
`dental arch 20, and generate an image of the projection pat(cid:173)
`tern after reflection of the pattern off the surface of the dental
`arch 20.
`[0030] The images acquired by the imager 24 include 3D
`information related to the surface of the object 20. The
`images, more specifically, 2D image data including this infor(cid:173)
`mation, are output to a 3D processor 32. The 3D processor can
`generate 3D data from the received image data.
`[0031] The image overlay system 14 can include the 3D
`processor 32, a video interface 34, a memory 36, an overlay
`engine 38, and an opacity adjuster 40. All of these elements
`can execute entirely on the image overlay system 14. Alter(cid:173)
`natively, some elements can execute on the image overlay
`system 14 or other computer platform, while other elements
`execute on the scanning device 12, the display 16, or a remote
`computer. For example, the 3 D processor 3 2 can be part of the
`image overlay system 14 as shown in FIG. 2. Alternatively,
`the 3D processor 32 can be part of the scanning device 12. In
`another example, the overlay engine 38 can be part of the
`image overlay system 14 as shown in FIG. 2, or can alterna(cid:173)
`tively be part of the display 16.
`[0032] The 3D processor 32 can receive signals related to
`one or more 2D images from the imager 24. For example, the
`signals can includes information on the intensity of the light
`received at each photodetector in the imager 24. In response,
`the 3D processor 32 can calculate the distance from the
`imager 24, for example, a detector array, of the scanning
`device 12 to the surface of the dental arch 20 for each pixel
`based on the intensity values for the pixel in the series of
`generated 2D images. Thus, the 3D processor 32 creates a set
`of3D coordinates that can be displayed as a point cloud or a
`surface map that represents the object surface. The 3D pro(cid:173)
`cessor 32 communicates with the memory 36 for storage of
`3D data generated during a measurement procedure. A user
`interface (not shown) allows an operator such as a clinician to
`provide operator commands and to observe the acquired 3D
`information in a near-real time manner. For example, the
`operator can observe a display of the growth of a graphical
`representation of the point cloud as different regions of the
`surface of the dental arch 20 are measured and additional 3D
`measurement data are acquired.
`[0033] The video interface 34 can likewise receive 2D
`image data from the scanning device 12. The 2D image data
`can be the same data as that received by the 3D processor 32,
`for example, from the imager 24. Optionally, the video inter(cid:173)
`face 34 can receive 2D image data from a different source, for
`example, a video camera instead of the scanning device 12.
`The video interface 24 processes and outputs from the
`received 2D image data a real-time or "live" video image of
`
`0013
`
`Exhibit 1030 page 13 of 17
`DENTAL IMAGING
`
`

`

`US 2012/0062557 Al
`
`Mar. 15, 2012
`
`4
`
`the surfaces being measured to the overlay engine 38. In
`particular, the image data received by the video interface 34
`corresponds to a portion of the dental arch in the measurement
`field of view 42 of the scanning device 12.
`[0034] As described above, the memory 36 can store the 3D
`data and/or 2D data. The memory 36 can also include
`machine executable instructions enabling the 3D processor
`32 to process the points in a point cloud and/or generate a
`single mesh surface configuration representing the scanned
`object, i.e., the dental arch 20 for the display 16. The memory
`36 can include volatile memory, for example, RAM and the
`like, and/or non-volatile memory, for example, ROM, flash
`memory, and the like. The memory can include removable
`and/or non-removable storage media implemented in accor(cid:173)
`dance with methods and technologies known to those of
`ordinary skill in the art for storing data. Stored in the memory
`can include program code, such as program code of an oper(cid:173)
`ating system executed by the image generator 34, the 3D
`processor 32, or other processors of the image overlay system
`14.
`[0035] The overlay engine 38 can be part of a display pro(cid:173)
`cessor or graphical user interface for displaying the 3D data as
`a graphical representation on the display 16. The overlay
`engine 38 includes a first input that receives a 2D video feed
`from the video interface 34 and a second input that receives
`3D data from the 3D processor. The overlay engine 38 can
`overlay or superimpose real-time or near-real time video
`images of the 2D feed corresponding to the dentition within
`the field of view of the imager 24 overlaid on at least a portion
`of the graphical representation of the 3D data, for example,
`one or more point clouds or object surface representation.
`[0036] During an operation, the 3D data and video images
`can be output from the overlay engine 38 to the display 16,
`and can be configured by the overlay engine 38 such that the
`3D data is displayed as a point cloud, object surface repre(cid:173)
`sentation on the display, and the video images are displayed in
`a window on the display. In a preferred embodiment, the
`video image is displayed in a window centered in the viewing
`area of the display 16. The window in which the video image
`is displayed can have a rectangular or square shape that is
`substantially smaller than the rectangular shape of the view(cid:173)
`ing area of the display 16. The display 16 can include a user
`interface (not shown) for presenting the received images in
`grayscale, color, or other user-defined format, and for permit(cid:173)
`ting a user to enter commands, view images, or other well(cid:173)
`known functions for performing a scanning and/or display
`operation.
`[0037] A portion of the 3D data can also be available for
`presentation in the window. The opacity adjuster 40 can be
`configured to change the opacity level of the 3D data and/or
`the video image in the window. For example, the video image
`can be presented as being substantially opaque and the por(cid:173)
`tion of the graphical representation of the 3D data in the
`window can be transparent to prevent a display of the graphi(cid:173)
`cal representation of the 3D data in a region of overlay iden(cid:173)
`tified by the window. The opacity adjuster 40 can reduce the
`opacity of the video image and/or reduce the transparency of
`the graphical representation of the set of3D data in the region
`of overlay identified by the window. In this manner, the video
`image and the graphical representation of the set of3D data in
`the region of overlay can be simultaneously viewed in the
`window. This feature can be beneficial when a clinician
`attempts to "stitch" or join a 3D point cloud or surface map to
`a previously acquired 3D point cloud or surface map. In
`
`particular, a change in transparencies of the video image and
`3 D data allows the clinician to maneuver the scanning device
`to substantially match a live video image with a portion of the
`displayed, previously generated 3D data. Once the two dis(cid:173)
`plays are determined to be substantially similar in their region
`of overlap, the acquisition of a 3D measurement scan can
`resume, and subsequently determined 3D data sets, e.g. point
`clouds or surface maps, can be stitched to an existing 3D data
`set.
`[0038] FIG. 3 is a flowchart of a method 100 for presenting
`dental structure image data acquired during a scanning opera(cid:173)
`tion. In describing the method 100, reference is also made to
`FIGS. 1 and 2. The method 100 can be governed by instruc(cid:173)
`tions that are stored in a memory and executed by a processor
`of the scanning device 12, the image overlay system 14,
`and/or the display 16. The method 100 is described herein as
`being performed on a dental arch. In other embodiments, the
`method 100 can be performed on virtually any object.
`[0039] A clinician such as a dentist initiates the method 100
`by positioning (step 105) the scanning device 12 at a starting
`point of the dental arch so that a structured light pattern
`generated from the scanning device 12 illuminates a first
`region of the dental arch, for example, a back portion of an
`occlusal surface of the dental arch at one end of the arch.
`Image data for providing 2D and/or 3D images can be
`acquired for the illuminated portion