(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`(19) World Intellectual Property
`Organization
`International Bureau
`
`
`
`IIIIII11111111011101010N OH IIIII II III
`
`(43) International Publication Date
`29 October 2015 (29.10.2015) WIPOI PCT
`
`(10) International Publication Number
`WO 2015/164402 Al
`
`(51) International Patent Classification:
`A61B 19/00 (2006.01)
`
`(21) International Application Number:
`
`(22) International Filing Date:
`
`(25) Filing Language:
`
`(26) Publication Language:
`
`PCT/US2015/026916
`
`21 April 2015 (21.04.2015)
`
`English
`
`English
`
`(81) Designated States (unless otherwise indicated, for every
`kind of national protection available): AE, AG, AL, AM,
`AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW, BY,
`BZ, CA, CH, CL, CN, CO, CR, CU, CZ, DE, DK, DM,
`DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT,
`HN, HR, HU, ID, IL, IN, IR, IS, JP, KE, KG, KN, KP, KR,
`KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, ME, MG,
`MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM,
`PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, SC,
`SD, SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN,
`TR, TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW.
`
`(30) Priority Data:
`61/982,787
`
`22 April 2014 (22.04.2014)
`
`US (84)
`
`(71) Applicant: SURGERATI, LLC [US/US]; 3439 Benjamin
`Ave, #427, Royal Oak, MI 48073 (US).
`
`(72) Inventors: DOO, Florence, X.; 3439 Benjamin Ave, #427,
`Royal Oak, MI 48067 (US). BLOOM, David, C.; 19400
`Sibley Road, Chelsea, MI 48118 (US).
`
`(74) Agent: SHACKELFORD, Jon, E.; Endurance Law
`Group, PLC, 180 W Michigan Ave., Ste 501, Jackson, MI
`49201 (US).
`
`Designated States (unless otherwise indicated, for every
`kind of regional protection available): ARIPO (BW, GH,
`GM, KE, LR, LS, MW, MZ, NA, RW, SD, SL, ST, SZ,
`TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU,
`TJ, TM), European (AL, AT, BE, BG, CH, CY, CZ, DE,
`DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU,
`LV, MC, MK, MT, NL, NO, PL, PT, RO, RS, SE, SI, SK,
`SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ,
`GW, KM, ML, MR, NE, SN, TD, TG).
`
`Published:
`
`— with international search report (Art. 21(3))
`
`(54) Title: INTRA-OPERATIVE MEDICAL IMAGE VIEWING SYSTEM AND METHOD
`
`FIG 2
`
`(57) Abstract: An intra-operative medical image viewing system can allow a surgeon to maintain a viewing perspective on the pa-
`tient while calling-up visual images on-the-fly. A digital image source has at least one image file representative of an anatomical or
`pathological feature of a patient. A display is worn by the surgeon or positioned between the surgeon and her patient during surgery.
`The display is selectively transparent, and exhibits to the surgeon an image derived from the image file. An image control unit re-
`trieves the image file from the image source and controls the display so that at least a portion of the image depiction can be exhibited
`and modified at will by the surgeon. A plurality of peripheral devices are each configured to receive an image control input from the
`surgeon and, in response, generate an image control signal. Each peripheral accepts a different user-interface modality.
`
`WO 2015/164402 Al
`
`1
`
`Medivis Exhibit 1008
`
`1
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`INTRA-OPERATIVE MEDICAL IMAGE VIEWING SYSTEM AND METHOD
`
`CROSS REFERENCE TO RELATED APPLICATIONS
`
`[0001] This application claims priority to Provisional Patent Application No. 61/982,787
`
`filed April 22, 2014, the entire disclosure of which is hereby incorporated by reference and
`
`relied upon.
`
`BACKGROUND OF THE INVENTION
`
`[0002] Field of the Invention. The invention relates generally to generating, processing,
`
`transmitting or transiently displaying images in a medical environment, in which the local
`
`light variations composing the images may change with time, and more particularly to subject
`
`matter in which the image includes portions indicating the three-dimensional nature of the
`
`original object.
`
`[0003] Description of Related Art. In a surgical environment, there are often many display
`
`screens each displaying different visual information that is of interest to the medical
`
`practitioner, such as a surgeon. In particular, the visual information may include images
`
`representing an anatomical or pathological feature of a patient, such as an X-ray, MRI,
`
`ultrasound, thermal image or the like. The term surgeon is used throughout this patent
`
`document in a broad sense to refer to any of the one or more specialized medical practitioners
`
`present in a surgical or interventional-procedural environment that provide critical personal
`
`treatment to a patient. In addition to practitioners and interventionalists, the term surgeon can
`
`also mean a medical student, as well as any other suitable person. The term surgical
`
`environment is also used broadly to refer to any surgical, interventional or procedural
`
`environment. Similarly, the term surgical procedure is chosen to broadly represent both
`
`interventional and non-interventional activities, i.e., including purely exploratory activities.
`
`Figure 1 is a simplified illustration of a surgical environment in which numerous display
`
`screens 20, 22, 24 compete for the attention of a surgeon 26 while the surgeon provides
`
`critical personal treatment to a patient 28. The display screens 20, 22, 24 are typically
`
`located in widely distributed locations within the operating room. Some of the displays 22,
`
`24 are suspended from boom-arms, others are mounted to the wall, and still others 20 can be
`
`mounted to mobile carts. An operating room that is filled with many display screens all
`
`presenting different relevant anatomical or pathological image data to the surgeon causes
`
`several problems in the medical community, which problems have proven particularly
`
`difficult to eradicate.
`
`1
`
`2
`
`Medivis Exhibit 1008
`
`2
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`[0004] A first problem relates to distraction of the surgeon's attention posed by the need to
`
`frequently look away from her patient in order to see the images on one or more display
`
`screens dispersed about the operating room. While surgeons are generally gifted with
`
`extraordinary eye-hand coordination, the surgical procedures they perform often depend on
`
`sub-millimeter-level control of their instruments. The risk of a tiny, unwanted hand
`
`movement rises each time a surgeon must consult an image on a screen that is located some
`
`distance away from the patient. The accidental nicking of an adjacent organ could perhaps in
`
`some cases be attributed to the surgeon's momentary head turn as she looks at an important
`
`anatomical or pathological image on a display screen on a nearby medical cart or suspended
`
`from a boom arm.
`
`[0005] A second problem that is provoked by the presence of multiple display screens in an
`
`operating room relates to compounding a surgeon's cognitive load. Cognitive load refers to
`
`the total amount of mental effort being used in the working memory of the surgeon. Surgeons
`
`are trained to function at high cognitive loading levels, yet every human has a limit.
`
`Biomedical research has confirmed that managing a surgeon's cognitive load level will allow
`
`her to perform at peak ability for a longer period of time. In operating room settings, one of
`
`the most intense contributors to the cognitive load of a surgeon is the mental act of image
`
`registration. Image registration is the process of transforming different sets of data into one
`
`coordinate system. For the surgeon in an operating environment, this means the ability to
`
`compare or integrate the data obtained from medical images presented on the display screens
`
`to the patient in front of them. For example, if the image on the display screen was taken (or
`
`is being rendered) from a perspective different than the instantaneous visual perspective of
`
`the surgeon, the surgeon automatically aligns the image to the patient by envisioning a
`
`rotation, pan, tilt, zoom or other manipulation of the displayed image to that of the live
`
`patient in front of them. While image registering a single static image to the patient may not
`
`be particularly taxing, the cognitive load quickly compounds when there are many display
`
`screens to be consulted, each exhibiting an image taken from yet a different perspective or
`
`presented in a different scale. Therefore, the multiplied act of image-registering a large
`
`number of images profoundly intensifies the cognitive loading imposed on a surgeon, which
`
`in turn produces an accelerated fatiguing effect.
`
`[0006] Yet another problem that is provoked by the presence of multiple display screens in
`
`an operating room relates to ergonomics. Namely, the occupational safety and health of a
`
`surgeon is directly compromised by the required use of many widely-dispersed images during
`
`a surgical procedure. During a surgical procedure, which can sometimes last for many hours,
`
`the surgeon 26 must often look up from the patient 28 in order to obtain information from the
`
`various display screens 20, 22, 24. In the exemplary illustration of Figure 1, if the surgeon 26
`
`2
`
`3
`
`Medivis Exhibit 1008
`
`3
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`is required to gaze intently at the display screen 20 for a long period of time, her head must
`
`be held steadily in an uncomfortable sideways-looking direction. Some surgical procedures,
`
`such as a laparoscopic procedure for example, require the surgeon to watch the real-time
`
`image feed from a remote camera. The surgeon's gaze may be intently directed to the real-
`
`time image on a display screen for an extended period of time. Surgery does not afford the
`
`practitioner with the ability to rest or change positions at will in order to combat muscle
`
`cramps or nerve aggravations. On a daily basis, this physical fatigue limits a surgeon's
`
`ability to perform at optimum ability during long shifts. Over time, the stresses placed on the
`
`surgeon accumulate to the point where the injuries accumulate/compound and become
`
`chronic and must either be remediated through medical intervention or the surgeon
`
`prematurely limits (or truncates) her service career.
`
`[0007] Furthermore, these problems can be inter-related. Issues associated with cognitive
`
`load and ergonomics compound each other to diminish a surgeon's working efficiency, which
`
`affect the patient by increasing the length of time they must undergo a surgical procedure.
`
`Naturally, increased procedure time impacts the surgeon's health but also the surgeon's
`
`productivity. That is, with more time in each surgery the surgeon can do fewer operations
`
`over the course of a year, which also then limits the surgeon's ability to gain experience.
`
`Increased procedure time impacts the patient in a number of ways also, including increased
`
`risks associated with prolonged time under anesthesia and its after-affects, increased risk for
`
`infections attributed to longer open incision times, longer hospital stays, increased medical
`
`costs, and the like.
`
`[0008] Finding a solution to these persistent image-related problems in the operating room
`
`has been elusive. One reason is that any proposed solution must itself have a practical chance
`
`of being adopted in the surgical community. That is to say, a solution that works only in the
`
`lab or only for a small sub-set of practitioners will not be genuinely viable as a marketable
`
`product. A real solution needs to be practical for the medical community as a whole.
`
`Therefore, understanding and accommodating the medical community, as a whole, is a
`
`critical step in assessing whether or not a particular solution will have authentic merit. As a
`
`group, surgeons tend to be somewhat unique in temperament. They are generally recognized
`
`as excessively driven toward achievement, decisive, well organized, hardworking, assertive,
`
`and aim to reduce uncertainty in their operations to reduce risk for their patient's outcomes.
`
`Any touted ergonomical or cognitive load benefit (and resultant benefit to patient outcomes)
`
`weighs against the heavy judgment of centuries of historic medical science and knowledge.
`
`Medical students, and the physicians they become, learn from their mentors the tried and true
`
`methods and techniques of their predecessors to ensure no patient harm. Thus, the point of
`
`mentioning this assessment is that surgeons by and large will tend not to accept into their
`
`3
`
`4
`
`Medivis Exhibit 1008
`
`4
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`practice a new technique or new technology unless that new technology is regarded as
`
`practical. But not all surgeons are alike, and what may be regarded by one surgeon as
`
`practical will be deemed unacceptably impractical by another. Therefore, any attempt to
`
`introduce a solution to the above-mentioned image issues must be instantly perceived as
`
`being practicable to all (or at least a substantial majority of) surgeons. It is predictable that a
`
`majority of surgeons will not adopt a solution if the solution is perceived to be overly
`
`complicated or as requiring a high degree of training to master.
`
`[0009] The reason why multiple display screens litter the typical operation room today is
`
`that display screens are universally intuitive. The mere act of looking at an image displayed
`
`on a screen requires no training for use. Therefore, if the surgeon needs to see more patient
`
`images during a surgical procedure, there is a tendency to add another display screen in the
`
`operating room. Adding more display screens, in turn, compounds the distraction, cognitive
`
`loading and ergonomic issues. A degenerative spiral results, because the current state of the
`
`art has no simpler, more intuitive option than adding more display screens to exhibit patient
`
`medical images in an operating room.
`
`[0010] There is therefore a need for an improved system in which the customary multitude
`
`of medical images needed to be viewed by a surgeon during an operation are better managed
`
`so that a surgeon is not required to look away from the patient, so that the surgeon does not
`
`have to sustain heavy cognitive loading in order to mentally register all of exhibited images,
`
`and so that the surgeon does not suffer unnecessary additional physical stresses. However,
`
`any an improved system to overcome these issues must be easily and intuitively implemented
`
`without the need for extensive training or practice.
`
`BRIEF SUMMARY OF THE INVENTION
`
`[0011]
`
`In summary, the invention is an intra-operative medical image viewing system that
`
`can allow the surgeon to maintain a viewing perspective on the patient while concurrently
`
`obtaining relevant information about the patient. The intra-operative medical image viewing
`
`system can include an image source having at least one image file representative of an
`
`anatomical or pathological feature of a patient. The intra-operative medical image viewing
`
`system can also include a display positionable between a surgeon and the patient during
`
`surgery. The display can be configured to exhibit and position at least one image to the
`
`surgeon overlaid on or above the patient. The intra-operative medical image viewing system
`
`can also include an image control unit configured to retrieve the image file from the image
`
`source and control the display so as to exhibit and modify at least a portion of the image. The
`
`intra-operative medical image viewing system can also include a plurality of peripheral
`
`devices. Each peripheral device may be configured to receive an image control input from
`
`4
`
`5
`
`Medivis Exhibit 1008
`
`5
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`the surgeon and, in response, generate an image control signal in a respective user-interface
`
`modality. The image control input can be representative of a desire by the surgeon to modify
`
`the at least one image exhibited by the display. Each peripheral device can define a different
`
`user interface modality.
`
`[0012]
`
`In another aspect of the invention, an intra-operative medical image viewing system
`
`can include an image source having at least one image file representative of an anatomical or
`
`pathological feature of a patient or of a surgical implementation, trajectory or plan. The
`
`intra-operative medical image viewing system can also include a display positionable
`
`between a surgeon and the patient during surgery. The display can be configured to exhibit
`
`the image to the surgeon overlaid on the patient. The intra-operative medical image viewing
`
`system can also include an image control unit configured to retrieve the image file from the
`
`image source and control the display to exhibit and modify at least a portion of the image.
`
`The intra-operative medical image viewing system can also include at least one peripheral
`
`device configured to receive an image control input from the surgeon and in response
`
`transmit an image control signal to the image control unit. The image control input can be
`
`representative of a desire by the surgeon to modify the image exhibited by the display. The
`
`image control unit can be configured to modify the image in response to the image control
`
`signal in any one of a plurality of different three-dimensional modalities.
`
`[0013]
`
`In another aspect of the invention, an intra-operative medical image viewing system
`
`can include
`
`an image source having an image file representative of an anatomical
`
`feature of a patient. The intra-operative medical image viewing system can also include a
`
`display wearable by a surgeon during surgery on the patient. The display can be selectively
`
`transparent and configured to exhibit an image to the surgeon overlaid on the patient. The
`
`intra-operative medical image viewing system can also include an image control unit
`
`configured to retrieve the image file from the image source and control the display to exhibit
`
`and modify the image. The image can be a visual representation of the anatomical feature of
`
`the patient. The image control unit can be responsive to inputs from the surgeon to modify
`
`the image to allow the surgeon to selectively position, size and orient the image exhibited on
`
`the display to a selectable first configuration. The intra-operative medical image viewing
`
`system can also include a station-keeping module. The station-keeping module can include a
`
`position module configured to detect a first position of the display when the first
`
`configuration can be selected and determine a change in position of the display from the first
`
`position. The station-keeping module can also include an orientation module configured to
`
`detect a first orientation of the display when the first configuration can be selected and
`
`determine a change in orientation of the display from the first orientation. The station-
`
`keeping module can also include a registration module configured to determine a registration
`
`5
`
`6
`
`Medivis Exhibit 1008
`
`6
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`default condition that can be defined by a frame of reference or a coordinate system; the first
`
`configuration, the first position, and the first orientation can also be defined the frame of
`
`reference or the coordinate system. The station-keeping module can also include an image
`
`recalibration module configured to determine one or more image modification commands to
`
`be applied by the display to change the image from the first configuration to a second
`
`configuration in response to at least one of the change in position and change in the
`
`orientation. The image recalibration module can be configured to transmit the one or more
`
`image modification commands to the image control unit and the image control unit to control
`
`the display in response to the one or more image modification commands and change the
`
`image to a second configuration. The second configuration can be different from the first
`
`configuration and consistent with the registration default condition.
`
`[0014] The present invention is particularly adapted to manage the multitude of medical
`
`images needed to be viewed by a surgeon during an operation so that a surgeon is not
`
`required to look away from the patient, so that the surgeon does not have to sustain heavy
`
`cognitive loading in order to mentally register all of exhibited images, and so that the surgeon
`
`does not suffer unnecessary additional physical stresses. In addition, the present invention
`
`can be easily and intuitively implemented without the need for extensive training or practice.
`
`By lowering distraction, cognitive loading, and concomitant fatigue, use of the present
`
`invention will lead to greater efficiency. That is to say, the surgeon can perform more
`
`procedures per shift, so that her productivity is improved. In addition, a surgeon executing a
`
`surgical procedure with the present invention will be more productive, learn faster and
`
`perform better, thereby leading to greater effectiveness.
`
`BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
`
`[0015] These and other features and advantages of the present invention will become more
`
`readily appreciated when considered in connection with the following detailed description
`
`and appended drawings, wherein:
`
`[0016] Figure 1 is a perspective view of a surgical environment according to the prior art;
`
`[0017] Figure 2 is a perspective view of an embodiment of the invention in a first surgical
`
`environment;
`
`[0018] Figure 3 is a schematic view of an embodiment of the invention in a second surgical
`
`environment;
`
`[0019] Figure 4 is another schematic view of the invention;
`
`[0020] Figure 5 is a perspective view of an embodiment of the invention in a third surgical
`
`environment;
`
`6
`
`7
`
`Medivis Exhibit 1008
`
`7
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`[0021] Figure 6 is a perspective view of an embodiment of the invention in a fourth
`
`surgical environment;
`
`[0022] Figure 7 is a perspective view of an embodiment of an embodiment of the invention
`
`in a fifth surgical environment;
`
`[0023] Figure 8 is a perspective view of a two-dimensional image in a planar configuration;
`
`[0024] Figure 9 is a perspective view of the two-dimensional image of Figure 8 in a
`
`wrapped configuration;
`
`[0025] Figure 10 is a perspective view of a two-dimensional image in a planar
`
`configuration in two different observable planes;
`
`[0026] Figure 11 is a series of three-dimensional tomographic slices of an anatomical
`
`feature of a patient;
`
`[0027] Figure 12 is a perspective view of an embodiment of the invention in a sixth
`
`surgical environment;
`
`[0028] Figure 13 is a perspective view of an embodiment of the invention in a seventh
`
`surgical environment; and
`
`[0029] Figure 14 is a perspective view of an embodiment of the invention in an eighth
`
`surgical environment.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`[0030] The exemplary embodiment can provide an intra-operative medical image viewing
`
`system 34 and method for displaying and interacting with two-dimensional, 2-1/2-
`
`Dimentional, or three-dimensional visual data in real-time and in perceived three-dimensional
`
`space. The system 34 can present a selectively or variably transparent image of an
`
`anatomical feature of a patient 28 to a surgeon 26 during surgery, as the surgeon 26 maintains
`
`a viewing perspective generally centered on the actual anatomical feature of the patient 28 or
`
`at least toward the patient 28 on whom some operation is being performed. The image as
`
`perceived by the surgeon 26 is selectively and/or variably transparent, in the sense that the
`
`surgeon 26 controls the image opacity throughout the range of fully transparent, e.g., when
`
`the image is not in use, to fully opaque, e.g., when high-contrast is desired, and through some
`
`if not all levels in-between. In most cases, the medical image appears to the surgeon to be
`
`located between herself, i.e., her eyes, and the patient 28. Typically, the image will appear to
`
`hover over (Figures 2, 12 and 13) or be overlaid on the skin of the patient 28 (Figures 7 and
`
`14), or have the appearance of being inside the patient's body volume (Figure 6). In other
`
`cases, the surgeon 26 may wish to locate the appearance of the image conveniently adjacent
`
`to the patient 28, such as hovering directly above them (Figure 5). The present invention is
`
`better able to manage the multitude of medical images needed to be viewed by a surgeon
`
`7
`
`8
`
`Medivis Exhibit 1008
`
`8
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`during a procedure by positioning the medical image between herself and her patient. Such
`
`positioning of the perceived appearance of the medical images (i.e., as perceived by the
`
`surgeon 26) can be accomplished via numerous techniques, including wearable devices,
`
`heads-up/teleprompter type devices, and projection devices. Any one or all of these device
`
`types, as well as any other suitable means, can be used to apply the concepts of this invention
`
`so that the medical image is positioned between the surgeon and her patient, or at least in a
`
`convenient adjacent location, so that a surgeon is not required to look away from the patient,
`
`so that the surgeon does not have to sustain heavy cognitive loading in order to mentally
`
`register all of exhibited images, and so that the surgeon does not suffer unnecessary
`
`additional physical stresses. It is noted that the term "surgeon" is not used in a limiting sense;
`
`the invention is not limited to systems that can only be used by a surgeon. It is also noted that
`
`patient data can be stored in an "upstream" image file and remain unchanged while a
`
`"downstream" image that is generated based on the image file is modified and manipulated.
`
`It is noted that while a human patient is illustrated in the Figures, one or more embodiments
`
`of the invention may be utilized in teaching or simulation environments, and/or in the care of
`
`a non-human.
`
`[0031] The exemplary embodiment can provide an intra-operative medical image viewing
`
`system 34 and method that allows the surgeon to self-manage the vital medical images she
`
`may wish to reference during a surgical procedure so that the instances in which her attention
`
`is shifted away from the patient are reduced, so that she can reduce the cognitive loading
`
`associated with mentally registering all of the displayed images, and so that she will suffer
`
`less physical stresses on her body. During surgery, the surgeon 26 can use the intra-operative
`
`medical image viewing system to self-modify the image as desired and on-the-fly.
`
`[0032] More specifically, the problem of distraction is attenuated by the present invention
`
`in that the images, as perceived by the surgeon, appear to overlay or hover in close proximity
`
`to the patient. As a direct result, the surgeon 26 will not need to frequently look away from
`
`her patient in order to see the desired images. A substantial benefit of mitigating distraction
`
`is that the risk of unwanted hand movements will decrease, and surgical accuracy will
`
`increase, when the surgeon is no longer required to turn her head to see important anatomical
`
`or pathological images. Additionally, cognitive load/cognitive distraction away from the
`
`surgical task can accumulate into increased productive surgical time and reduced (or even
`
`adverse) patient outcomes. Another potential benefit is reduced operating time, which may
`
`improve patient outcomes.
`
`[0033] The problem of excessive cognitive loading may also be mitigated by the present
`
`invention through its ability to position and scale a medical image relative to the patient 28
`
`from the perspective of the surgeon 26. That is to say, the present invention manipulates the
`
`8
`
`9
`
`Medivis Exhibit 1008
`
`9
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`way a medical image is exhibited so that it conforms to the surgeon's visual perspective. As
`
`a result, the surgeon 26 does not need to mentally correlate each medical image to her actual,
`
`natural view of the patient 28.
`
`In situations where a given medical image was taken (or is
`
`being rendered) from a perspective different than the instantaneous visual perspective of the
`
`surgeon 26, the invention adapts the presentation of the image (but not the image source data)
`
`through actions like panning, zooming, rotating and tilting, to better align with the patient
`
`thereby reducing the cognitive effort expended by the surgeon to make thoughtful use of the
`
`medical image. Considering the large number of medical images typically referenced by a
`
`surgeon during a medical procedure, the cumulative cognitive loading imposed on a surgeon
`
`will be greatly reduced and with it the mental fatigue will also be reduced..
`
`[0034] The system 34 can reduce physical demands on the surgeon 26 by placing the
`
`medical images over the patient 28, or in some embodiments the image will appear directly
`
`adjacent the patient 28 in a hovering manner. By strategically placing medical images over
`
`or directly adjacent the patient 28, as perceived by the surgeon 26, the need for the surgeon
`
`26 to frequently look away during surgery is substantially if not completely eliminated. As a
`
`result, the physical stresses of muscle, joint and eye strains will be mitigated. A surgeon
`
`using the present invention may experience a marked reduction in physical fatigue, thereby
`
`enabling her to perform at optimum ability during long shifts. Over time, the surgeon will be
`
`exposed to fewer workplace-related injuries thereby favorably extending her service career.
`
`In addition, a reduction in surgery time can directly benefit the patent and improve safety. In
`
`particular, faster surgical procedures mean reduced affects associated with anesthesia,
`
`reduced risk for infections, shorter hospital stays, reduced medical costs, and the like.
`
`[0035] The present invention will enjoy accelerated adoption in the medical field by
`
`overcoming the natural barriers associated with the stereotypical resistance to complicated
`
`technologies by surgeons by and large. This natural market resistance is addressed in the
`
`present invention by enabling the surgeon 26 to choose how to communicate image control
`
`inputs to the system from among many different user-interface modalities. Regardless of
`
`which user-interface modality the surgeon 26 selects, each image control input implements a
`
`desire by the surgeon 26 to modify the displayed image so that the position, pose, orientation,
`
`scale, and spatial (3D) structure of the image is adaptively changed in real-time and overlaid
`
`on the surgeon's view. The system can thus allow the surgeon 26 to communicate image
`
`control inputs in any of a plurality of different user-interface modalities. Each user-interface
`
`modality represents a different communication medium or command language, such as voice,
`
`touch, gesture, etc. Accordingly, the system 34 can be more intuitive for the surgeon 26 to
`
`use because the surgeon can choose the user-interface modality that is most intuitive to her.
`
`Said another way, the plurality of user-interface modalities allows the surgeon 26 to interact
`
`9
`
`10
`
`Medivis Exhibit 1008
`
`10
`
`

`

`WO 2015/164402
`
`PCT/US2015/026916
`
`with the system in the most comfortable manner to her, thereby obviating the need for the
`
`surgeon 26 to learn and/or maintain knowledge of just one particular user-interface modality.
`
`During surgery, the surgeon 26 can be freed to communicate with the system in the way most
`
`"natural" to the surgeon 26. As a result, the likelihood of ready adoption for this technology
`
`within the surgical field will be greatly increased.
`
`[0036] The exemplary embodiment can provide an intra-operative medical image viewing
`
`system 34 that increases the available viewing options for a surgeon 26 by providing the
`
`surgeon 26 with various approaches to three-dimensional viewing. As will be described in
`
`greater detail below, three-dimensional images can be defined in different formats. One
`
`surgeon 26 may find three-dimensional images in one particular format useful while another
`
`surgeon 26 may prefer images in a different format. The system 34 can allow the surgeon 26
`
`to choose the format in which three-dimensional images are displayed so that the information
`
`contained in the medical image will be most useful to the surgeon 26 at the particular moment
`
`needed and for a particular surgical procedure.
`
`[0037] The exemplary embodiment can provide an intra-operative medical image viewing
`
`system 34 that maintains the registration of an image to an actual anatomical feature of the
`
`patient 28 despite head movement by the surgeon 26. The system 34 can allow the surgeon
`
`26 to selectiv