I 1111111111111111 11111 lllll lllll 111111111111111 1111111111 lll111111111111111
`US009329675B2
`
`c12) United States Patent
`Ojelund et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 9,329,675 B2
`May 3, 2016
`
`(54) SYSTEM WITH 3D USER INTERFACE
`INTEGRATION
`
`(75)
`
`Inventors: Henrik Ojelund, Lyngby (DK); David
`Fischer, Stenlose (DK); Karl-Josef
`Hollenbeck, Kobenhavn 0 (DK)
`
`(52) U.S. Cl.
`CPC . G06F 3101 (2013.01); A61C 9/004 (2013.01);
`G0JB 11124 (2013.01)
`(58) Field of Classification Search
`CPC combination set(s) only.
`See application file for complete search history.
`
`(73) Assignee: 3SHAPE A/S, Copenhagen K (DK)
`
`(56)
`
`References Cited
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 249 days.
`
`(21) Appl. No.:
`
`13/991,513
`
`(22) PCT Filed:
`
`Dec. 5, 2011
`
`(86) PCT No.:
`
`PCT/DK2011/050461
`
`§ 371 (c)(l),
`(2), ( 4) Date:
`
`Jun.4,2013
`
`(87) PCT Pub. No.: W02012/076013
`
`PCT Pub. Date: Jun. 14, 2012
`
`(65)
`
`Prior Publication Data
`
`US 2013/0257718 Al
`
`Oct. 3, 2013
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/420,138, filed on Dec.
`6, 2010.
`
`(30)
`
`Foreign Application Priority Data
`
`Dec. 6, 2010
`
`(DK) ................................. 2010 01104
`
`(51)
`
`Int. Cl.
`G06F 3101
`A61C9/00
`G0JB 11124
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`U.S. PATENT DOCUMENTS
`
`6,135,961 A
`6,967,644 Bl*
`7,141,020 B2
`7,813,591 B2 *
`7,831,292 B2 *
`8,035,637 B2 *
`
`10/2000 Pflugrath et al.
`11/2005 Kobayashi .................... 345/158
`11/2006 Poland et al.
`10/2010 Paley et al .................... 382/285
`11/2010 Quaid et al.
`.................. 600/424
`10/2011 Kriveshko ..................... 345/419
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`WO
`
`8/2009
`101513350
`8/2004
`WO 2004/066615 Al
`(Continued)
`OTHER PUBLICATIONS
`
`Three-Dimensional Virtual Reality Xia et al. Jun. 2001.*
`(Continued)
`
`Primary Examiner - Van Chow
`(74) Attorney, Agent, or Firm - Buchanan Ingersoll &
`Rooney PC
`
`ABSTRACT
`(57)
`Disclosed is a system comprising a handheld device and at
`least one display. Thehandheld device is adapted for perform(cid:173)
`ing at least one action in a physical 3D environment; wherein
`the at least one display is adapted for visually representing the
`physical 3D environment; and where the handheld device is
`adapted for remotely controlling the view with which the 3D
`environment is represented on the display.
`
`19 Claims, 5 Drawing Sheets
`
`0001
`
`Exhibit 1032 page 1 of 15
`DENTAL IMAGING
`
`

`

`US 9,329,675 B2
`Page 2
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`U.S. PATENT DOCUMENTS
`
`8,384,665 Bl *
`2003/0158482 Al
`2004/0204787 Al *
`2005/0057745 Al
`2006/0025684 Al
`2006/0092133 Al*
`2006/0146009 Al
`2009/0061381 Al
`2009/0217207 Al
`
`2/2013 Powers et al. ................. 345/156
`8/2003 Poland et al.
`10/2004 Kopelman et al. ............ 700/182
`3/2005 Bontje
`2/2006 Quistgaard et al.
`5/2006 Touma et al.
`................. 345/158
`7 /2006 Syrbe et al.
`3/2009 Durbin et al.
`8/2009 Kagermeier et al.
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`WO
`
`2007084727
`2009089126
`2001011193
`
`7/2007
`7/2009
`1/2011
`
`International Search Report (PCT /ISA/210) issued on Feb. 22, 2012,
`by the Danish Patent Office as the International Searching Authority
`for International Application No.PCT/DK/2011/050461.
`C. Graetzel et al., "A Non-Contact Mouse for Surgeon-Computer
`Interaction", Technology and Health Care, 12(3), 2004, pp. 1-19.
`Sebastian Vogt et al., "AnAR System With Intuitive User Interface for
`Manipulation and Visualization of 3D Medical Data", Stud. Health
`Technol. Inform., Medicine Meets Virtual Reality 12, 2004; vol. 98,
`pp. 397-403.
`First Office Action issued in corresponding Chinese Patent Applica(cid:173)
`tion No. 201180066956.6, issued Apr. 3, 2015. (13 pages).
`Second Office Action issued in corresponding Chinese Patent Appli(cid:173)
`cation No.201180066956.6, dated Nov. 18, 2015, with English trans(cid:173)
`lation (27 pages).
`
`* cited by examiner
`
`0002
`
`Exhibit 1032 page 2 of 15
`DENTAL IMAGING
`
`

`

`U.S. Patent
`
`May 3, 2016
`
`Sheet 1 of 5
`
`US 9,329,675 B2
`
`102
`
`Fig. 1
`
`0003
`
`Exhibit 1032 page 3 of 15
`DENTAL IMAGING
`
`

`

`U.S. Patent
`
`May 3, 2016
`
`Sheet 2 of 5
`
`US 9,329,675 B2
`
`100
`
`105
`
`102
`
`Fig. 2a)
`
`0004
`
`Exhibit 1032 page 4 of 15
`DENTAL IMAGING
`
`

`

`U.S. Patent
`
`May 3, 2016
`
`Sheet 3 of 5
`
`US 9,329,675 B2
`
`105
`
`100
`
`Fig. 2b)
`
`0005
`
`Exhibit 1032 page 5 of 15
`DENTAL IMAGING
`
`

`

`U.S. Patent
`
`May 3, 2016
`
`Sheet 4 of 5
`
`US 9,329,675 B2
`
`107
`
`106
`
`Fig. 3
`
`0006
`
`Exhibit 1032 page 6 of 15
`DENTAL IMAGING
`
`

`

`U.S. Patent
`
`May 3, 2016
`
`Sheet 5 of 5
`
`US 9,329,675 B2
`
`----
`I
`I
`I
`I
`
`I ,_
`
`I
`I
`I
`I
`
`I ,_
`
`101
`
`102
`
`103
`
`Fig. 4
`
`0007
`
`Exhibit 1032 page 7 of 15
`DENTAL IMAGING
`
`

`

`US 9,329,675 B2
`
`1
`SYSTEM WITH 3D USER INTERFACE
`INTEGRATION
`
`FIELD OF THE INVENTION
`
`This invention generally relates to a method and a system
`comprising a handheld device and at least one display.
`
`BACKGROUND OF THE INVENTION
`
`3D visualization is important in many fields ofindustry and
`medicine, where 3D information is becoming more and more
`predominant.
`Displaying and inspecting 3D information is inherently
`difficult. To fully understand a 3D object or entire environ(cid:173)
`ment on a screen, the user should generally be able to rotate
`the object or scene, such that many or preferentially all sur(cid:173)
`faces are displayed. This is true even for 3D displays, e.g.
`stereoscopic or holographic, where from a given viewing
`position and with a given viewing angle, the user will only see
`some surfaces of an arbitrary 3D environment. Often, the user
`will also want to zoom into details or zoom out for an over(cid:173)
`view.
`Various user interaction devices are in use for software that
`displays 3D data; these devices are: 3D mice, space balls, and
`touch screens. The operation of these current interaction
`devices requires physically touching them.
`Physically touching a user-interaction device can be a dis(cid:173)
`advantage in medical applications due to risks of cross-con(cid:173)
`tamination between patients or between patient and operator,
`or in industrial applications in dirty environments.
`Several non-touch user interfaces for 3D data viewing in
`medical applications have been described in the literature.
`Vogt et al (2004) describe a touchless interactive system for
`in-situ visualization of 3D medical imaging data. The user
`interface is based on tracking of reflective markers, where a
`camera is mounted on the physician's head. Graetzel et al
`(2004) describe a touchless system that interprets hand ges(cid:173)
`tures as mouse actions. It is based on stereo vision and
`intended for use in minimally invasive surgery.
`It remains a problem to improve systems that require user
`interfaces for view control, which for example can be used for
`clinical purposes.
`
`SUMMARY
`
`Disclosed is a system comprising a handheld device and at
`least one display, where the handheld device is adapted for
`performing at least one action in a physical 3D environment,
`where the at least one display is adapted for visually repre(cid:173)
`senting the physical 3D environment, and where the handheld
`device is adapted for remotely controlling the view with
`which said 3D environment is represented on the display.
`The system may be adapted for switching between per(cid:173)
`forming the at least one action in the physical 3D environ- 55
`ment, and remotely controlling the view with which the 3D
`environment is represented on the display.
`The system disclosed here performs the integration of 3D
`user interface functionality with any other handheld device
`with other operating functionality, such that the operator ide- 60
`ally only touches this latter device that is intended to be
`touched. A particular example of such a handheld device is
`one that records some 3D geometry, for example a handheld
`3D scanner.
`The handheld device is a multi-purpose device, such as a 65
`dual-purpose or two-purpose device, i.e. a device both for
`performing actions in the physical 3D environment, such as
`
`5
`
`45
`
`2
`measuring and manipulating, and for remotely controlling the
`view of the 3D environment on the display.
`Geometrically, a view is determined by the virtual observ(cid:173)
`er's/camera's position and orientation relative to the 3D envi-
`ronment or its visual representation. If the display is two(cid:173)
`dimensional, the view is also determined by the type of
`projection. A view may also be determined by a magnification
`factor.
`The virtual observer's and the 3D environment's position
`10 and orientation are always relative to each other. In terms of
`user experience in software systems with 3D input devices,
`the user may feel that for example, he/she is moving the 3D
`environment while remaining stationary himself/herself, but
`15 there is always an equivalent movement of the virtual
`observer/camera that gives the same results on the display.
`Often, descriptions of 3D software systems use the expres(cid:173)
`sion "pan" to indicate an apparent translational movement of
`the 3D environment, "rotate" to indicate a rotational move-
`20 ment of the 3D environment, and "zoom" to indicate a change
`in magnification factor.
`Graphically, a view can represent a 3D environment by
`means of photographs or as some kind of virtual representa(cid:173)
`tion such as a computer graphic, or similar. A computer
`25 graphic can be rendered for example with texture and/or
`shading and/or virtual light sources and/or light models for
`surface properties. A computer graphic can also be a simpli(cid:173)
`fied representation of the 3D environment, for example a
`mesh, an outline, or an otherwise simplified representation.
`30 All or parts of the 3D environment can also be rendered with
`some degree of transparency. A view may represent the 3D
`environment in total or only parts thereof.
`All of the touch-less prior art systems are 3D user interface
`35 devices only. In many prior art applications, the operator
`using such user interface device will also hold and work with
`another device that really is the central device in the overall
`application, e.g. a medical instrument.
`It is thus an advantage of the present system that the 3D
`40 user-interface functionality is integrated in the central device,
`which is used for performing some kind of action.
`In some embodiments the handheld device is adapted for
`remotely controlling the magnification with which the 3D
`environment is represented on the display.
`In some embodiments the handheld device is adapted for
`changing the rendering of the 3D environment on the display.
`In some embodiments the view is defined as viewing angle
`and/or viewing position.
`In some embodiments the at least one action comprises one
`50 or more of the actions of:
`measuring,
`recording,
`scanning,
`manipulating,
`modifying.
`In some embodiments the 3D environment comprises one
`or more 3D objects.
`In some embodiments the handheld device is adapted to be
`held in one hand by an operator.
`In some embodiments the display is adapted to represent
`the 3D environment from multiple views.
`In some embodiments the display is adapted to represent
`the 3D environment from different viewing angles and/or
`viewing positions.
`In some embodiments the view of the 3D environment in
`the at least one display is at least partly determined by the
`motion of the operator's hand holding said device.
`
`0008
`
`Exhibit 1032 page 8 of 15
`DENTAL IMAGING
`
`

`

`US 9,329,675 B2
`
`3
`In some embodiments the magnification represented in the
`at least one display is at least partly determined by the motion
`of the operator's hand holding said device.
`In some embodiments the handheld device is adapted to
`record the 3D geometry of the 3D environment.
`Thus the handheld device may be an intraoral dental scan(cid:173)
`ner, which records the 3D geometry of a patient's teeth. The
`operator may move the scanner along the teeth of the patient
`for capturing the 3D geometry of the relevant teeth, e.g. all
`teeth. The scanner may comprise motion sensors for taking 10
`the movement of the scanner into account while creating the
`3D model of the scanned teeth.
`The 3D model of the teeth may be shown on a display, and
`the display may for example be a PC screen and/or the like.
`The user interface functionality may comprise incorporat- 15
`ing motion sensors in the scanner to provide that the user can
`determine the view on the screen by moving the scanner.
`Pointing the scanner down can provide that the scanned teeth
`are shown given a downward viewing angle. Holding the
`scanner in a horizontal position can provide that the viewing 20
`angle is likewise horizontal.
`In some embodiments the handheld device comprises at
`least one user-interface element. A user-interface element is
`an element which the user may manipulate in order to activate
`a function on the user interface of the software. Typically the
`use interface is graphically presented on the display of the
`system.
`The handheld device may furthermore be provided with an
`actuator, which switches the handheld device between per(cid:173)
`forming the at least one action and remotely controlling the 30
`view. By providing such a manual switching function that
`enables the operator to switch between performing the at least
`one action and remotely controlling the view, the operator
`may easily control what is performed.
`Such an actuator can for example be in the form of a button, 35
`switch or contact. In other embodiments it could be a touch
`sensitive surface or element.
`In another embodiment the actuator could be a motion
`sensor provided in the handheld device that function as the
`actuator when it registers a specific type of movement, for
`example if the operator shakes the handheld device.
`Examples of such motion sensors will be described herein
`with respect to the user-interface element, however, the per(cid:173)
`son skilled in the art will based on the disclosure herein
`understand that such motion sensors may also be used as
`actuators as discussed.
`For example, the handheld device can in one embodiment
`be an intra-oral 3D scanner used by a dentist. The scanner is
`set to be performing the action of scanning a dental area when
`the actuator is in one position. When the actuator is switched 50
`into a second position the handheld is set to control the view
`with which the 3D environment is represented on the display.
`This could for example be that when the dentist have scanned
`a part of or the complete desired area of an dental arch he can
`activate the actuator which then allows the dentist to remotely 55
`control the view of the 3D representation of the scanned area
`on the display by using the handheld device.
`For example, the actuator could be a button. When the
`button is pressed quickly the handheld device is prepared for
`scanning, i.e. it is set for performing at least one action, the
`scanning procedure, in the physical 3D environment. The
`scanning is stopped when the button is pressed quickly a
`second time.
`While the scanning is performed a virtual 3D representa(cid:173)
`tion is visually built on the display.
`The user can now press and hold the button. This will put
`the handheld in a controller mode, where the handheld device
`
`4
`is adapted for remotely controlling the view with which the
`3D environment, such as scanned teeth, is represented on the
`display. While holding the button pressed the system will use
`signals from a motion sensor in the handheld device to deter-
`5 mine how to present the view of the virtual 3D environment.
`Thus, if the user turns or otherwise moves the hand that holds
`the handheld device the view of the virtual 3D environment
`on the display will change accordingly.
`Thus, the dentist may use the same handheld device for
`both scanning an area and subsequently verifying that the
`scan has been executed correctly without having to move
`away from the patient or touching any other equipment than
`already present in his hands.
`In one embodiment the user-interface element is the same
`as the actuator, or where several user-interface elements are
`present at least one also functions as an actuator.
`The system may be equipped with a button as an additional
`element providing the user-interface functionality.
`In an example the handheld device is a handheld intraoral
`scanner, and the display is a computer screen. The operator or
`user may be a dentist, an assistant and/or the like. The opera(cid:173)
`tion functionality of the device may be to record some
`intraoral 3D geometry, and the user interface functionality
`25 may be to rotate, pan, and zoom the scanned data on the
`computer screen.
`In some embodiments the at least one user-interface ele(cid:173)
`ment is at least one motion sensor.
`Thus the integration of the user interface functionality in
`the device may be provided by motion sensors, which can be
`accelerometers inside the scanner, whose readings determine
`the orientation of the display on the screen of the 3D model of
`the teeth acquired by the scanner. Additional functionality,
`e.g. to start/stop scanning, may be provided by a button. The
`button may be located where the operator's or user's index
`finger can reach it conveniently.
`Prior art intraoral scanners use a touch screen, a trackball,
`or a mouse to determine the view in the display. These prior
`40 art user interface devices can be inconvenient, awkward and
`difficult to use, and they can be labor-intensive, and thus
`costly to sterilize or disinfect. An intraoral scanner should
`always be disinfected between scanning different patients,
`because the scanner is in and may come in contact with the
`45 mouth or other parts of the patient being scanned.
`The operator or user, e.g. dentist, may use one hand or both
`hands to hold the intraoral scanner while scanning, and the
`scanner may be light enough and comfortable to be held with
`just one hand for a longer time while scanning.
`The device can also be held with one or two hands, while
`using the device as remote control for e.g. changing the view
`in the display. It is an advantage of the touchless user interface
`functionality that in clinical situations, the operator can main(cid:173)
`tain both hands clean, disinfected, or even sterile.
`An advantage of the system is that it allows an iterative
`process of working in a 3D environment without releasing the
`handheld device during said process. For the above intraoral
`scanning system example, the operator, e.g. dentist, can
`record some teeth surface geometry with a handheld device
`60 that is an intraoral scanner, inspect coverage of the surface
`recording by using that same handheld device to move, e.g.
`rotate, the recorded surface on the display, e.g. a computer
`screen, detect possible gaps or holes in the coverage of the
`scanned teeth, and then for example arrange the scanner in the
`65 region where the gaps were located and continue recording
`teeth surface geometry there. Over this entire iterative cycle,
`which can be repeated more than once, such as as many times
`
`0009
`
`Exhibit 1032 page 9 of 15
`DENTAL IMAGING
`
`

`

`US 9,329,675 B2
`
`5
`as required for obtaining a desired scan coverage of the teeth,
`the dentist does not have to lay the handheld intraoral scanner
`out of his or her hands.
`In some embodiments, the 3D user interface functionality
`is exploited in a separate location than the operation function(cid:173)
`ality. For the above intraoral scanning system example, the
`scanning operation is performed in the oral cavity of the
`patient, while the user interface functionality is more flexibly
`exploited when the scanner is outside the patient's mouth.
`The key characteristic and advantage of the system, again, is
`that the dentist can exploit the dual and integrated function(cid:173)
`ality, that is operation and user interface, of the scanner with(cid:173)
`out laying it out of his or her hands.
`The above intraoral scanning system is an example of an
`embodiment. Other examples for operation functionality or
`performing actions could be drilling, welding, grinding, cut(cid:173)
`ting, soldering, photographing, filming, measuring, execut(cid:173)
`ing some surgical procedure etc.
`The display of the system can be a 2D computer screen, a
`3D display that projects stereoscopic image pairs, a volumet- 20
`ric display creating a 3D effect, such as a swept-volume
`display, a static volume display, a parallax barrier display, a
`holographic display etc. Even with a 3D display, the operator
`has only one viewing position and viewing angle relative to
`the 3D environment at a time. The operator can move his/her
`head to assume another viewing position and/or viewing
`angle physically, but generally, it may be more convenient to
`use the handheld device with its built-in user interface func(cid:173)
`tionality, e.g. the remote controlling, to change the viewing
`position and/or viewing angle represented in the display.
`In some embodiments the system comprises multiple dis(cid:173)
`plays, or one or more displays that are divided into regions.
`For example, several sub-windows on a PC screen can repre(cid:173)
`sent different views of the 3D environment. The handheld
`device can be used to change the view in all of them, or only 35
`some of them.
`In some embodiments the user interface functionality com(cid:173)
`prises the use of gestures.
`Gestures made by e.g. the operator can be used to change,
`shift or toggle between sub-windows, and the user-interface 40
`functionality can be limited to an active sub-window or one of
`several displays.
`In some embodiments the gestures are adapted to be
`detected by the at least one motion sensor. Gestures can
`alternatively and/or additionally be detected by range sensors 45
`or other sensors that record body motion.
`The operator does not have to constantly watch the at least
`one display of the system. In many applications, the operator
`will shift between viewing and possible manipulating the
`display and performing another operation with the handheld 50
`device. Thus it is an advantage that the operator does not have
`to touch other user interface devices. However, in some cases
`it may not be possible for the operator to fully avoid touching
`other devices, and in these cases it is an advantage that fewer
`touches are required compared to a system where a handheld 55
`device does not provide any user interface functionality at all.
`In some embodiments the at least one display is arranged
`separate from the handheld device.
`In some embodiments the at least one display is defined as
`a first display, and where the system further comprises a 60
`second display.
`In some embodiments the second display is arranged on the
`handheld device.
`In some embodiments the second display is arranged on the
`handheld device in a position such that the display is adapted 65
`to be viewed by the operator, while the operator is operating
`the handheld device.
`
`6
`In some embodiments the second display indicates where
`the handheld device is positioned relative to the 3D environ(cid:173)
`ment.
`In some embodiments the first display and/or the second
`5 display provides instructions for the operator.
`The display(s) can be arranged in multiple ways. For
`example, they can be mounted on a wall, placed on some sort
`of stand or a cart, placed on a rack or desk, or other.
`In some embodiments at least one display is mounted on
`10 the device itself. It can be advantageous to have a display on
`the device itselfbecause with such an arrangement, the opera(cid:173)
`tor's eyes need not focus alternatingly between different dis(cid:173)
`tances. In some cases, the operating functionality may require
`a close look at the device and the vicinity of the 3D environ-
`15 ment it operates in, and this may be at a distance at most as far
`away as the operator's hand. Especially in crowded environ(cid:173)
`ments such as dentist's clinics, surgical operation theatres, or
`industrial workplaces, it may be difficult to place an external
`display closely to the device.
`In some embodiments visual information is provided to the
`operator on one or more means other than the first display.
`In some embodiments audible information to the operator
`is provided to the operator.
`Thus in some embodiments, the system provides addi-
`25 tional information to the operator. In some embodiments, the
`system includes other visual clues shown on means other than
`the display(s), such as LEDs on the device. In some embodi(cid:173)
`ments, the system provides audible information to the opera-
`tor, for example by different sounds and/or by speech.
`Said information provided to the operator can comprise
`instructions for use, warnings, and the like.
`The information can aid with improving the action perfor-
`mance or operation functionality of the device, for example
`by indicating how well an action or operation is being per(cid:173)
`formed, and/or instructions to the operator aimed at improv(cid:173)
`ing the ease of the action oroperation and/ or the quality of the
`action or operation's results. For example, a LED can change
`in color and/or flashing frequency. In a scanner, the informa(cid:173)
`tion can relate to how well the scanned 3D environment is in
`focus and/or to scan quality and/or to scan coverage. The
`information can comprise instructions on how best to position
`the scanner such as to attain good scan quality and/or scan
`coverage. The instructions can be used for planning and/or
`performing bracket placement. The instructions can be in the
`form of a messenger system to the operator.
`In some embodiments, some 3D user interface functional(cid:173)
`ity is provided by at least one motion sensor built into the
`device. Examples of motion sensors are accelerometers,
`gyros, and magnetometers and/or the like. These sensors can
`sense rotations, lateral motion, and/or combinations thereof.
`Other motion sensors use infrared sensing. For example, at
`least one infrared sensor can be mounted on the device and at
`least one infrared emitter can be mounted in the surroundings
`of the device. Conversely, the at least one emitter can be
`mounted on the device, and the at least one sensors in the
`surroundings. Yet another possibility is to use infrared reflec-
`tor(s) on the device and both sensor(s) and emitter(s) on the
`surroundings, or again conversely. Thus motion can be sensed
`by a variety of principles.
`Through proper signal processing, some sensors can rec(cid:173)
`ognize additional operator actions; for example gestures such
`as taps, waving, or shaking of the handheld device. Thus,
`these gestures can also be exploited in the 3D user interface
`functionality.
`In some embodiments the handheld device comprises at
`least two motion sensors providing sensor fusion. Sensor
`fusion can be used to achieve a better motion signal from for
`
`30
`
`0010
`
`Exhibit 1032 page 10 of 15
`DENTAL IMAGING
`
`

`

`US 9,329,675 B2
`
`7
`8
`In some embodiments the view of the 3D environment
`example raw gyro, accelerometer, and/or magnetometer data.
`comprises a viewing angle, a magnification factor, and/or a
`Sensor fusion can be implemented in ICs such as the
`viewing position.
`InvenSense MPU 3000.
`In some embodiments the view of the 3D environment
`In some embodiments the handheld device comprises at
`least one user-interface element other than the at least one 5
`comprises rendering of texture and/or shading.
`In some embodiments the at least one display is divided
`motion sensor.
`In some embodiments the at least one other user-interface
`into multiple regions, each showing the 3D environment with
`element is a touch-sensitive element.
`a different view.
`In some embodiments the at least one other user-interface
`Thus in some embodiments the user interface functionality
`10 comprises changing the view with which the 3D environment
`element is a button.
`In some embodiments the at least one other user-interface
`is displayed. Changes in view can comprise changes in view(cid:173)
`element is a scroll-wheel.
`ing angle, viewing position, magnification and/or the like. A
`change in viewing angle can naturally be effected by rotating
`In some embodiments, user interface functionality is pro(cid:173)
`the device. Rotation is naturally sensed by the aid of gyros
`vided through additional elements on the device. Thus these
`15 and/or relative to gravity sensed by an accelerometer. Zoom(cid:173)
`additional elements can for example be buttons, scroll
`ing, i.e. a change in magnification, can for example be
`wheels, touch-sensitive fields, proximity sensors and/or the
`like.
`achieved by pushing the handheld device forward and back(cid:173)
`The additional user interface elements can be exploited or
`ward, respectively. A translational change of the viewing
`position, i.e., panning, can for example be achieved by push-
`utilized in a workflow suitable for the field of application of
`20 ing the handheld device up/down and/or sideways.
`the device. The workflow may be implemented in some user
`software application that may also control the display and
`In some embodiments the user interface functionality com(cid:173)
`thus the view represented thereon. A given interface element
`prises selecting or choosing items on a display or any other
`can supply multiple user inputs to the software. For example,
`functionality provided by graphical user interfaces in com(cid:173)
`puters known in the art. The operator may perform the selec-
`a button can provide both a single click and a double click. For
`25 tion. The Lava C.O.S scanner marketed by 3M ESPE has
`example, a double click can mean to advance to a subsequent
`step in a workflow. For the example of intraoral scanning,
`additional buttons on the handheld device, but it is not pos(cid:173)
`sible to manipulate the view by these. Their only purpose is to
`three steps within the workflow can be to scan the lower
`allow navigation through a menu system, and to start/stop
`mouth, the upper mouth, and the bite. A touch-sensitive field
`can provide strokes in multiple directions each with a differ(cid:173)
`scanning.
`In some embodiments the user interface functionality com-
`ent effect, etc. Providing multiple user inputs from a user 30
`interface elements is advantageous because the number of
`prises manipulating the 3D environment displayed on the
`screen. For example, the operator may effect deformations or
`user interface elements on the device can be reduced relative
`change the position or orientation of objects in the 3D envi(cid:173)
`to a situation where each user interface element only provides
`ronment. Thus, in some embodiments the user interface func-
`one user input.
`The motion sensors can also be exploited in a workflow. 35 tionality comprises virtual user interface functionality, which
`For example, lifting the device, which can be sensed by an
`can be that the 3D data are manipulated, but the physical 3D
`environment in which the device operates may not be manipu(cid:173)
`accelerometer, can represent some type of user input, for
`lated.
`example to start some action. In a device that is a scanner, it
`may start scanning. Conversely, placing the device back in
`In some embodiments the handheld device is an intraoral
`scanner and/ or an in-the-ear scanner. If the scanner comprises
`some sort of holder, which can be sensed by an accelerometer 40
`as no acceleration occur over some period of time, can stop
`a tip, this tip may be exchanged whereby the scanner can
`become suitable for scanning in the mouth or in the ear. Since
`said action.
`If the action performed by the device is some kind of
`the ear is a smaller cavity than the mouth, the tip for fitting
`recording, for example scanning, for example 3D scanning,
`into an ear may be smaller than a tip for fitting in the mouth.
`the results of the recording can also be exploited as user 45
`In some embodiments the handheld device is a surgical
`instrument. In some embodiments, the surgical instrument
`inputs, possibly along with user inputs from other user inter(cid:173)
`face elements. For example, with a 3D scanner with a limited
`comprises at least one motion sensor, which is built-in in the
`depth offield, it may be possible to detect whether any objects
`instrument.
`within the 3D environments are present in the volume corre(cid:173)
`In some embodiments the handheld device is a mechanical
`sponding to this depth of field by detecting whether any 3D 50
`tool. In some embodiments, the tool has at least one motion
`sensor built in. In other embodiments, ot