(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`OTATAA
`
`(19) World Intellectual Property Organization
`International Bureau
`
`27 January 2011 (27.01.2011)
`
`(43) International Publication Date
`
`(10) International Publication Number
`WO 2011/011193 Al
`
`(51) International Patent Classification:
`A61C 3/00 (2006.01)
`(21) International Application Number:
`.
`PCT/US2010/041045
`
`(22) International Filing Date:
`
`6 July 2010 (06.07.2010)
`.
`English
`English
`
`(25) Filing Language:
`(26) Publication Language:
`(30) Priority Data:
`61/227.255
`
`21 July 2009 (21.07.2009)
`
`(81) Designated States (unless otherwise indicated, for every
`kind of national protection available): AE, AG, AL, AM,
`AO, AT, AU,AZ, BA, BB, BG, BH, BR, BW, BY, BZ,
`CA, CH, CL, CN, CO, CR, CU, CZ, DE, DK, DM, DO,
`DZ, EC, EE, EG, ES, FL, GB, GD, GE, GH, GM, GT,
`HN, HR, HU,ID, IL, IN, IS, JP, KE, KG, KM, KN, KP,
`KR, KZ, LA, LC, LK, LR, LS, LT, LU, LY, MA, MD,
`ME, MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI,
`NO, NZ, OM,PE, PG, PH, PL, PT, RO, RS, RU, SC, SD,
`SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN, TR,
`TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW.
`
`us (84) Designated States (unless otherwise indicated, for every
`kind of regional protection available): ARIPO (BW, GH,
`GM,KE, LR, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG,
`ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU,TJ,
`TM), European (AL, AT, BE, BG, CH, CY, CZ, DE, DK,
`EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU,
`LV, MC, MK, MT, NL, NO, PL, PT, RO, SE, SL, SK,
`
`(for all designated States except US): DI-
`(71) Applicant
`MENSIONAL
`PHOTONICS
`INTERNATIONAL,
`INC.
`[US/US];
`187 Ballardvale Street, Suite A135,
`Wilmington, Massachusetts 01887 (US).
`(72) Inventors; and
`
`(75) Inventors/Applicants
`
`(for US
`
`only): WALLACE,
`
`oN Me MIR,NE-SN-TD.TG), CI, CM, GA, GN, GQ,
`
`>
`
`>
`
`>
`
`>
`
`>
`
`:
`
`(74) Agent: GUERIN, William G.; Guerin & Rodriguez,
`LLP, 5 Mount Royal Avenue, Mount Royal Office Park,
`Marlborough, MA 01752 (US).
`
`(54) Title: INTEGRATED DISPLAY IN A HAND-HELD THREE-DIMENSIONAL METROLOGY SYSTEM
`
`>
`Nathan E. [US/US]; 17 Steinbeck Street, Tyngsborough,
`Massachusetts 01879 (US). FILLION, Timothy I. Published:
`[US/US]; 44 Gould Road, Bedford, Massachusetts 01730
`—__with international search report (Art. 21(3))
`(US).
`
`
`
`
`
`wo2011/011193A.IMITINMININMTNIAAA
`
`
`
`
`PROCESSOR
`30
`
`2D IMAGE
`DATA
`
`CAMERA
`22
`
`
`
`
`
`3D
`MEASUREMENT
`DATA
`
`
`x
`
`FIG. 1
`
`(57) Abstract: Described is a user-manipulated imaging device for measuring a three-dimensional surface of an object. The device
`includes an imager configured for acquiring two-dimensional images of the surface and a device housing coupled to the imager
`and configured for manual positioning of the imager. The device also includes a processor in communication with the imager and
`configured to generate three-dimensional surface data based on the two-dimensional images. The device further includes a display
`coupled to the device housing and in communication with at least one of the imager and the processor. The display shows images
`of the surface and is observable within a field of view of the user while the device housing is manually positioned within the field
`ofview andrelative to the surface. In various embodiments, the display shows the two-dimensional images and representations of
`the three-dimensional surface data.
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`INTEGRATED DISPLAY IN A HAND-HELD THREE-
`
`DIMENSIONAL METROLOGY SYSTEM
`
`RELATED APPLICATION
`
`This application claims the benefit of the earlier filing date of U.S.
`
`Provisional Patent Application Serial No. 61/227,255, filed July 21, 2009,
`
`titled “Integrated Display in a Hand-Held Three-Dimensional Metrology
`
`System,” the entirety of which is incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The invention relates to the field of three-dimensional imaging and
`
`10
`
`more specifically to the field of displaying non-contact surface measurement
`
`data for dental and medical applications.
`
`BACKGROUND OF THE INVENTION
`
`A variety of precision non-contact three-dimensional (83D) metrology
`
`systems have been developed for dental and medical applications.
`
`15
`
`Conventional systems typically include a handheld camera or scanner
`
`connected to a processing unit that communicates with a display monitor.
`
`The display monitor presents a variety of information to the user. The
`
`information can include control options, acquired images, and operator
`
`assistance information such as an indication of an optimal focus condition.
`
`20
`
`This configuration requires the user to look in two directions, that is, to look
`
`at the position of the handheld device with respect to the patient and to look
`
`at the display monitor to determine that proper images are being acquired.
`
`Thus the time and effort to obtain the desired measurement data is
`
`adversely affected by the requirement for the user to alternately view the
`
`25
`
`position of the device and view the acquired images.
`
`

`

`WO 2011/011193
`
`SUMMARY
`
`PCT/US2010/041045
`
`DPI-019PC
`
`In one aspect, the invention features a methodof displaying
`
`information for a user-manipulated 3D imaging device. The method
`
`includes acquiring a plurality of two-dimensional (2D) images of a surface of
`
`an object with an imaging device manipulated by a user in position relative
`
`to the surface of the object and within a field of view of the user. The 2D
`
`images are processed to generate three-dimensional surface data for the
`
`surface of the object. Measurement data are displayed to the user within
`
`the field of view of the user during continued manipulation of the imaging
`
`10
`
`device.
`
`In one embodiment, the displayed measurement information
`
`includes the two-dimensional images acquired by the imaging device and, in
`
`another embodiment, the displayed information includes a representation of
`
`the 3D surface data.
`
`In another aspect, the invention features a user-manipulated imaging
`
`15
`
`device for measuring a 3D surface of an object. The imaging device includes
`
`an imager, a device housing, a processor and a display. The imageris
`
`configured for acquiring 2D images of a surface of the object. The device
`
`housing is coupled to the imager and configured for manipulation by a user
`
`to position the imager relative to the surface of the object. The processor
`
`20
`
`communicates with the imager and is configured to generate 3D surface
`
`data for the surface based on the 2D images. The display is coupled to the
`
`device housing and communicates with at least one of the imager and the
`
`processor. The display shows images of the surface observable within a field
`
`of view of the user while the device housing is manually positioned within
`
`25
`
`the field of view of the user relative to the surface.
`
`In one embodiment, the
`
`display shows the 2D images of the surface acquired by the imager and, in
`
`another embodiment, the display shows a representation of the 3D surface
`
`data generated by the processor.
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The above and further advantages of this invention may be better
`
`understood by referring to the following description in conjunction with the
`
`accompanying drawings, in which like numerals indicate like structural
`
`elements and features in the various figures. The drawings are not
`
`necessarily to scale, emphasis instead being placed uponillustrating the
`
`principles of the invention.
`
`FIG. 1 illustrates a 3D imaging device that projects a structured light
`
`pattern onto an object.
`
`10
`
`FIG. 2 is a flowchart representation of an embodiment ofa
`
`measurement procedure using a hand-held 3D imaging device according to
`
`the invention.
`
`FIG. 3 illustrates an embodiment of a user-manipulated imaging
`
`device according to the invention.
`
`15
`
`FIG. 4A illustrates an embodiment of a user-manipulated imaging
`
`device according to the invention and showing a display panel in an open
`
`position.
`
`FIG. 4B illustrates the user-manipulated imaging device of FIG. 4A
`
`showing the display panel in a closed position.
`
`20
`
`DETAILED DESCRIPTION
`
`In brief overview, the invention relates to a user-manipulated 3D
`
`metrology device such as a hand-held camera or scanning device. The
`
`device includes an integrated display monitor that provides the user with
`
`convenient access to control options, acquired images, and operator
`
`25
`
`assistance indications within a field of view of the user. Advantageously, the
`
`location of the operating tip of the device relative to the object being
`
`measured can be viewed without the need to redirect the view of the user to
`
`-3-
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`a display monitor. For medical and dental 3D metrology devices, the user
`
`positions and aligns the device to a patient while simultaneously viewing a
`
`display of the acquired images or data. As a result, measurement data are
`
`obtained with less time and operator effort than is required for conventional
`
`user-manipulated 3D metrology devices.
`
`The present teaching will now be described in more detail with
`
`reference to exemplary embodiments thereof as shown in the accompanying
`
`drawings. While the present teaching is described in conjunction with
`
`various embodiments and examples, it is not intended that the present
`
`10
`
`teaching be limited to such embodiments. On the contrary, the present
`
`teaching encompasses variousalternatives, modifications and equivalents,
`
`as will be appreciated by those of skill in the art. Those of ordinary skill in
`
`the art having access to the teaching herein will recognize additional
`
`implementations, modifications and embodiments, as well as otherfields of
`
`15
`
`use, which are within the scope of the present disclosure as described
`
`herein.
`
`In a typical dental or medical 3D camera or scanner imaging system, a
`
`series of 2D intensity images of an object surface is acquired where the
`
`illumination for each image can vary.
`
`In some systems, structured light
`
`20
`
`patterns are projected onto the surface and detected in each 2D intensity
`
`image. FIG. 1 shows an example of a 3D imaging system 10 in which the
`
`structured light pattern is generated by a projector 14 as a pair of
`
`overlapping coherent optical beams 16A and 16B that illuminate the object
`
`18. The 3D imaging system 10 may be constructed to operate in accordance
`
`25
`
`with the principles described in U.S. Patent No. 5,870,191, titled “Apparatus
`
`and Methods for Surface Contour Measurement,” incorporated herein by
`
`reference in its entirety. A CCD camera 22 is used to acquire images of the
`
`illuminated object 18. The fringe pattern 26 resulting from the interference
`
`of the two beams 16 is varied between successive 2D images acquired by the
`
`30
`
`camera 22. For example, the fringes in the fringe pattern 26 can be shifted
`
`by changing the phase difference between the two beams 16.
`
`-4-
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`A processor 30 calculates the distance from the camera 22 to the
`
`object surface for each image pixel based on the intensity values for the pixel
`
`in the 2D images. Thus the process creates a set of 3D coordinates, thatis,
`
`a “point cloud,” for the object surface.
`
`In a dynamic 3D imaging system, a series of point clouds is acquired
`
`while the camera or scanneris in motion relative to the object surface. For
`
`example, the imaging system can be a handheld device that a user manually
`
`positions relative to the object surface.
`
`In some applications, multiple
`
`objects surfaces are measured by moving the device relative to the objects so
`
`10
`
`that surfaces obscured from view of the device in one position are observable
`
`by the device in another position. A processor registers the overlapped
`
`region of adjacent point clouds, using a 3D correlation technique or other
`
`registration technique, to transform each successive point cloud into an
`
`initial coordinate space. The successive point clouds are thus “stitched” into
`
`15
`
`a common reference space.
`
`Referring to FIG. 2, at the start of an embodiment of a measurement
`
`procedure 100 according to the invention, the user aligns and positions
`
`(step 110) the hand-held imaging device relative to the patient while
`
`acquiring 2D images of a patient area of interest. The 2D images are
`
`20
`
`processed (step 120) to generate 3D surface data of the area of interest. The
`
`user simultaneously observes measurement images in a display while
`
`controlling (step 130) the positioning and motion of the handheld imaging
`
`device with respect to the patient. The images in the display can be the
`
`acquired 2D images. Alternatively, the displayed images can be 3D surface
`
`25
`
`representations generated by processing the acquired 2D images. By way of
`
`examples, the 3D surface representations can be 3D wire-mesh
`
`representations of point clouds orartificial surface displays that comprise
`
`simple geometrical shapes(e.g., triangles) between neighboring points in
`
`point clouds.
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`Providing a display that is in communication with the processor and
`
`mounted to or otherwise integrated with the 3D imaging device according to
`
`the principles of the invention permits the user to see the acquired 2D
`
`images, 3D surface representation, other display information or
`
`combinations of such images and information simultaneous with the
`
`observation and continued manipulation of the 3D imaging device relative to
`
`the patient. Thus the user can more easily and rapidly complete the
`
`measurement procedure than would be possible using a conventional
`
`handheld dental or medical imaging device. Other displayed information
`
`10
`
`can include operator assistance information such as a slide bar shown along
`
`the edge of the display to indicate measured position within a usable
`
`imaging range, the distance to a surface of the object being measured, anda
`
`color box to indicate the current modeof the device, such as idle, preview
`
`and scan modes.
`
`15
`
`In one embodiment, the display includes a touchscreen that permits
`
`the user to input selection data while maintaining the handheld device in
`
`proper position relative to the patient. Control options shown on the
`
`touchscreen display can include, by way of example, preview, scan and stop
`
`function activation “buttons;” save and redo buttons presented at the
`
`20
`
`completion of a scan, and input data buttons. For example, in dental
`
`applications, the input data buttons can be used to indicate the jaw to be
`
`imaged (upper or lower) or particular teeth to be imaged for a partial jaw
`scan.
`
`In another embodiment illustrated in FIG. 3, the imaging device 34
`
`25
`
`includes a miniature display 38 similar to the displays typically used in
`
`mass-produced cell phones for consumers. The miniature display 38 can be
`
`embedded in a side of the device housing 42 and optionally has a viewing
`
`surface that is flush with the housing 42. By way of a specific example, the
`
`miniature display 38 may have a 1.8 inch diagonal viewing area.
`
`In one
`
`30
`
`embodiment, the display is a compactliquid crystal display (LCD).
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`In an alternative embodiment, a display 46 is integral to a panel 50
`
`that is pivotally attached to a side of a device housing 54 for the 3D imaging
`
`device 58 shown in FIG. 4A and FIG. 4B. The panel 50 is small enough to
`
`be compatible with the overall dimensions of the device 58 and yet include a
`
`display 46 that is large enough to present detailed images to the user. By
`
`way of a specific example, the display 46 can have a four inch diagonal
`
`viewing area. FIG. 4A shows the panel 50 in an open position in which the
`
`user views the displayed images, 3D representations and information.
`
`FIG. 4B showsthe panel 50 in a closed position such that the panel 50 is
`
`10
`
`substantially parallel and adjacent to the side of the device housing 54. The
`
`closed position is intended for when the device 58 is stored or otherwise not
`
`in use for extended periods of time.
`
`In the embodiments described above, the device according to the
`
`invention is generally described as a handheld device; however, the
`
`15
`
`invention also contemplates that the device can be manually adjusted or
`
`manipulated by a user without being directly held by hand.
`
`While the invention has been shown and described with reference to
`
`specific embodiments, it should be understood by those skilled in the art
`
`that various changes in form and detail may be made therein without
`
`20
`
`departing from the spirit and scope of the invention.
`
`What is claimed is:
`
`

`

`WO 2011/011193
`
`CLAIMS
`
`PCT/US2010/041045
`
`DPI-019PC
`
`1.
`
`A method of displaying information for a user-manipulated three-
`
`dimensional imaging device, the method comprising:
`
`acquiring a plurality of two-dimensional images of a surface of an
`
`object with an imaging device manipulated by a user in position relative to
`
`the surface of the object and within a field of view of the user;
`
`processing the two-dimensional images to generate three-dimensional
`
`surface data for the surface of the object; and
`
`displaying measurement information to the user within thefield of
`
`view of the user during continued manual manipulation of the imaging
`
`device.
`
`2.
`
`The method of claim 1 wherein the displayed measurement
`
`information comprises the two-dimensional images acquired by the imaging
`
`device.
`
`3.
`
`The method of claim 1 wherein the displayed measurement
`
`information comprises a representation of the three-dimensional surface
`
`data.
`
`4.
`
`The method of claim 1 wherein the displayed measurement
`
`information comprises operator assistance information.
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`5.
`
`The method of claim 4 wherein the operator assistance information
`
`comprises a distance to the surface of the object.
`
`6.
`
`A user-manipulated imaging device for measuring a three-dimensional
`
`surface of an object, comprising:
`
`an imager configured for acquiring two-dimensional images of a
`
`surface of an object;
`
`a device housing coupled to the imager and configured for
`
`manipulation by a user to position the imagerrelative to the surface of the
`
`object;
`
`a processor in communication with the imager and configured to
`
`generate three-dimensional surface data for the surface based on the two-
`
`dimensional images; and
`
`a display coupled to the device housing and in communication with at
`
`least one of the imager and the processor, the display showing images of the
`
`surface observable within a field of view of the user while the device housing
`
`is manually positioned within the field of view of the user relative to the
`
`surface.
`
`7.
`
`The user-manipulated device of claim 6 wherein the images shown in
`
`the display are the two-dimensional images of the surface acquired by the
`
`imager.
`
`8.
`
`The user-manipulated device of claim 6 wherein the images shown in
`
`the display are representations of the three-dimensional surface data
`
`generated by the processor.
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`9.
`
`The user-manipulated device of claim 6 wherein the display shows
`
`operator assistance information.
`
`10.
`
`The user-manipulated device of claim 9 wherein the operator
`
`assistance information includes a distance to the surface of the object.
`
`11.
`
`The user-manipulated device of claim 6 wherein the display is a
`
`touchscreen display configured to receive data input from the user.
`
`12.
`
`The user-manipulated device of claim 6 wherein the display comprises
`
`a liquid crystal display (LCD).
`
`13.
`
`The user-manipulated device of claim 6 wherein the display comprises
`
`a display panel pivotably secured to a side of the device housing, the display
`
`panel extending away from a surface of the device housing while in an open
`
`position and extending substantially parallel to the surface of the device
`
`housing while in a closed position, and wherein images of the surface are
`
`observable to the user while the display panel is in the open position.
`
`14.
`
`The user-manipulated device of claim 6 wherein the display comprises
`
`a viewing surface that is substantially flush with a side of the device
`
`housing.
`
`-10-
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`DPI-019PC
`
`15.
`
`The user-manipulated device of claim 6 further comprising a projector
`
`in communication with the processor and configured for projecting a
`
`structured light pattern onto the surface of the object.
`
`16.
`
`The user-manipulated device of claim 15 wherein the projector
`
`comprises a source of coherent optical beams for illuminating the surface of
`
`the object with a fringe pattern.
`
`-ll-
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`1/3
`
`10
`
`
`
`PROJECTOR
`14
`
`
`
`PROCESSOR
`30 16B
`
`”
`—_
`
`”
`—_—
`ee
`=
`
` 2D IMAGE
`
`DATA
`
`
`3D
`MEASUREMENT
`DATA
`
`
`
`CAMERA
`22
`
`FIG. 1
`
`100
`
`CONTINUOUSLY ALIGN/POSITION
`USER-MANIPULATED IMAGING
`DEVICE RELATIVE TO PATIENT
`WHILE ACQUIRING 2D IMAGES
`
`PROCESS 2D IMAGES TO GENERATE
`
`
`
`110
`
`120
`
`
`
`3D SURFACE DATA FOR PATIENT 130
`
`DISPLAY MEASUREMENT DATA TO USER WITHIN
`FIELD OF VIEW OF USER DURING CONTINUED
`USER MANIPULATION OF IMAGING DEVICE
`
`FIG. 2
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`2/3
`
`
`
`Ss
`PROJECTED
`STRUCTURED
`LIGHT PATTERN
`
`FIG. 3
`
`58
`
`50
`
`4
`
`6
`
`54
`
`FIG. 4A
`
`

`

`WO 2011/011193
`
`PCT/US2010/041045
`
`3/3
`
`58
`
`!
`
`FIG. 4B
`
`

`

`PCT OSP: 871-272-7774
`
`[| Further documents are listed in the continuation of Box C.
`Special categories of cited documents:
`documentdefining the generalstate ofthe art which is not considered
`to be of particular relevance
`earlier application or patent but published on orafterthe international
`filing date
`document which may throw doubts on priority claim(s) or whichis
`cited to establish the publication date of another citation or other
`special reason (as specified)
`document referring to an oral disclosure, use, exhibition or other
`means
`
`-1
`
`3, 15-16
`
`US 7,046,286 B1 (KOBAYASHI etal.) 16 May 2006 (16.05.2006) Fig 4-6
`
`US 6,438,272 B1 (HUANGetal.) 20 August 2002 (20.08.2002) col 5, In 4-34
`
`13
`
`15-16
`
`“T”
`
`later documentpublishedafter the intemationalfiling date or priority
`date and not in conflict with the application but cited to understand
`the principle or theory underlying the invention
`document of particular relevance; the claimedinvention cannot be
`considered novel or cannot be considered to involve an inventive
`step when the documentis taken alone
`documentof particular relevance; the claimed invention cannot be
`considered to involve an inventive step when the document
`is
`combined with one or more other such documents, such combination
`being obvious to a person skilled in the art
`document memberof the samepatent family
`
`Date of mailing of the international search report
`27 SEP 2010
`Authorized officer:
`
`PCT Helpdesk: 571-272-4300
`
`Lee W. Young
`
`documentpublishedpriorto the international filing date but later than
`the priority date claimed
`
`Date of the actual completion of the international search
`16 September 2010 (16.09.2010)
`
`Nameand mailing address of the ISA/US
`Mail Stop PCT,Attn: ISA/US, Commissioner for Patents
`P.O. Box 1450, Alexandria, Virginia 22313-1450
`Facsimile No.
`571-273-3201
`
`Form PCT/ISA/210 (second sheet) (July 2009)
`
`INTERNATIONAL SEARCH REPORT
`
`International application No.
`PCT/US 10/41045
`
`CLASSIFICATION OF SUBJECT MATTER
`A.
`IPC(8) - :A61C 3/00 (2010.01)
`USPC - 433/29
`According to International Patent Classification (IPC) or to both national classification and IPC
`.
`FIELDS SEARCHED
`
`Minimum documentation searched (classification system followed by classification symbols)
`IPC - A61C 3/00 (2010.01)
`USPC - 433/29
`
`
`
`Documentation searched other than minimum documentation to the extent that such documents are includedin the fields searched
`USPC- 345/419,420,421,422,423, 424,426,427
`
`Electronic data base consulted during the international search (name of data base and, where practicable, search terms used)
`PubWEST(PGPB,USPT,USOC,EPAB,JPAB); Google Scholar
`Search terms- display, screen, touch screen,fringe patter, two dimension$, three dimension$, pivot$, hinge$, control$, distance,
`surface, contour
`
`C. DOCUMENTS CONSIDERED TO BE RELEVANT
`
`
`
`Citation of document, with indication, where appropriate, of the relevant passages Relevantto claim No.
`
`US 2005/0237581 A1 (KNIGHTONetal.) 27 October 2005 (27.10.2005) entire document,
`especially para [0025]-[0035,] [0047]-[0048}, [(0052]-[0054); Fig 1, 6
`
`1-12, 14
`
`