Haptic feedback in robot-assisted minimally invasive surgery
`Allison M. Okamura
`
`Johns Hopkins University, Department of Mechanical
`Engineering, Laboratory for Computational Sensing
`and Robotics, Baltimore, Maryland, USA
`
`Correspondence to Allison M. Okamura, Department of
`Mechanical Engineering, Laboratory for Computational
`Sensing and Robotics, 3400 N. Charles St., Baltimore,
`MD 21218, USA
`Tel: +1 410 516 7266; fax: +1 410 516 7254;
`e-mail: aokamura@jhu.edu
`
`Current Opinion in Urology 2009, 19:102–107
`
`Purpose of review
`Robot-assisted minimally invasive surgery (RMIS) holds great promise for improving the
`accuracy and dexterity of a surgeon and minimizing trauma to the patient. However,
`widespread clinical success with RMIS has been marginal. It is hypothesized that
`the lack of haptic (force and tactile) feedback presented to the surgeon is a limiting
`factor. This review explains the technical challenges of creating haptic feedback for
`robot-assisted surgery and provides recent results that evaluate the effectiveness of
`haptic feedback in mock surgical tasks.
`Recent findings
`Haptic feedback systems for RMIS are still under development and evaluation. Most
`provide only force feedback, with limited fidelity. The major challenge at this time is
`sensing forces applied to the patient. A few tactile feedback systems for RMIS
`have been created, but their practicality for clinical implementation needs to be shown. It
`is particularly difficult to sense and display spatially distributed tactile information. The
`cost–benefit ratio for haptic feedback in RMIS has not been established.
`Summary
`The designs of existing commercial RMIS systems are not conducive for force feedback,
`and creative solutions are needed to create compelling tactile feedback systems.
`Surgeons, engineers, and neuroscientists should work together to develop effective
`solutions for haptic feedback in RMIS.
`
`Keywords
`force, haptics, minimally invasive surgery, robotics, tactile
`
`Curr Opin Urol 19:102–107
`ß 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins
`0963-0643
`
`Introduction
`Haptics generally describes touch feedback, which
`may include kinesthetic (force) and cutaneous (tactile)
`feedback. In manual minimally invasive surgery (MIS),
`surgeons feel the interaction of the instrument with the
`patient via a long shaft, which eliminates tactile cues
`and masks force cues. Some studies have linked the
`lack of significant haptic feedback in MIS to increased
`intraoperative injury [1]. In teleoperated robot-assisted
`minimally invasive surgery (RMIS), all natural haptic
`feedback is eliminated because the surgeon no longer
`manipulates the instrument directly. The lack of effec-
`tive haptic feedback is often reported by surgeons and
`robotics researchers alike to be a major limitation of
`current RMIS systems.
`
`Haptic technology
`In the robotics and virtual reality literature, haptics is
`broadly defined as real and simulated touch interactions
`between robots, humans, and real, remote, or simulated
`environments, in various combinations. The goal of hap-
`tic technology in robot-assisted minimally invasive
`surgery is to provide ‘transparency’, in which the surgeon
`
`does not feel as if he is operating a remote mechanism,
`but rather that his own hands are contacting the patient.
`This requires artificial haptic sensors on the patient-side
`robot to acquire haptic information, and haptic displays to
`convey the information to the surgeon (Fig. 1). We
`categorize haptics as kinesthetic (related to forces and
`positions of the muscles and joints) and/or cutaneous
`(tactile; related to the skin) in nature. Haptics includes
`force, distributed pressure, temperature, vibrations, and
`texture, which are in some cases difficult to model and
`quantify, let alone acquire and display. To provide myr-
`iad haptic information to the surgeon without sacrificing
`the maneuverability and dexterity afforded by the RMIS
`system is a major technical challenge. Simultaneously,
`the robot components, particularly disposable instru-
`ments, must remain low cost and robust.
`
`As a technical field, haptics research has been active
`for several decades. In the 1990s, haptics research
`expanded significantly with the availability of high-
`fidelity, commercially available force feedback systems
`from companies such as SensAble Technologies, Inc.
`(Woburn, Massachusetts, USA) and Immersion, Inc.
`(San Jose, California, USA). Currently, much of the
`force feedback research focuses on developing practical
`
`0963-0643 ß 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins
`
`DOI:10.1097/MOU.0b013e32831a478c
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Downloaded from https://journals.lww.com/co-urology by fU9RKm26ePeHR2F8RjaJx4QSgIZG2AtSyUuVsPJIeOEVnJPRDDTJe6RdHh3z3bhsOeuX2yb5Zr6ttGfhQkxHT5399oVF2FjhMwZTu/bJeV9nPqiMpAYG8OIHB9fI9lIfgJWOgVt/LqkDWMtvLBvdeLsiyPfPEDvECV2rs5LkSKFgRUgkDkpEqw== on 02/27/2019
`
`Ethicon Exhibit 2018.001
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Figure 1 The main components of a teleoperated robot for minimally invasive surgery with multimodal haptic feedback
`
`Haptic feedback in robot-assisted minimally invasive surgery Okamura 103
`
`Both force and tactile feedback are included in the model, and graphical display (one method of sensory substitution) is shown as a possible alternative
`to direct haptic feedback.
`
`systems for application in fields such as entertainment,
`education, training, medicine and dentistry, and rehabi-
`litation. Although researchers have studied tactile feed-
`back for many years, there is currently no commercially
`available tactile display system that provides distributed
`information to the skin in a compact package feasible for
`applications. One aspect of tactile feedback that has
`proven effective in applications such as video games is
`vibration feedback, in which a single vibrating actuator is
`typically used to provide information about events such
`as making and breaking contact. Further reading about
`[2,3,4],
`tutorials
`haptic technology includes books
`[5,6,7], and research reviews [8–10]. Recent reviews
`of haptics in surgery are [11,12].
`
`Force feedback
`Kinesthetic or force feedback systems for RMIS typically
`measure or estimate the forces applied to the patient by the
`
`surgical instrument, and provide resolved forces to the
`hand via a force feedback device. Commercially available
`force sensors are very effective for measuring forces and
`torques in many teleoperation applications, but the surgi-
`cal environment places severe constraints on size, geome-
`try, cost, biocompatibility, and sterilizability. Although it is
`difficult to add force sensors to existing robotic instruments
`that were not designed with force sensing in mind, some
`researchers have had success on this front by creating
`specialized grippers that can attach to the jaws of existing
`instruments [13,14]. Another approach is to rethink the
`design of surgical instruments. The design of the force
`sensor can be integrated with that of the dexterous instru-
`ment [15,16,17], as shown in Fig. 2. For reasons of cost,
`biocompatibility, and sterilizability, the forces applied to
`the patient would ideally be estimated without using force
`sensors. For patient-side robots designed with low inertia
`and friction, the difference between the desired and actual
`position of the patient-side robot (where the desired
`
`Figure 2 A robotic surgery system for two-hand manipulation with integrated force feedback and 3D vision, designed by researchers
`at DLR, Germany
`
`The system consists of a specially designed dexterous force-sensing instrument, robotic arms and teleoperation controller, and haptic device
`commercially available from Force Dimension, Inc. (Lausanne, Switzerland) as the master manipulator. Original figures used with permission from B.
`Kuebler, DLR.
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Ethicon Exhibit 2018.002
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`104 Robotics
`
`position is that of the master manipulator) is an indication
`for forces being applied to the environment. However, the
`fidelity of such systems are limited as there are dynamic
`forces present in most robots that are difficult to account for
`and often mask the relatively minute forces of interacting
`with the patient [18].
`
`Direct force feedback to the surgeon’s hands can use
`conventional force display technology, in which the motors
`of the master manipulator are programmed to recreate the
`forces sensed by the patient-side robot. A dexterous sur-
`gical robot typically has seven degrees of freedom of
`motion, including translational, rotational, and gripping.
`However, not all of those degrees of freedom are actuated
`on the master. That is, the system cannot provide force
`feedback in certain directions. The effects may be negli-
`gible or detrimental, depending on the directions of force
`feedback lost [19,20]. The dynamics of the master manip-
`ulator can also limit the accuracy of the force display, but a
`more fundamental limitation is the trade-off between
`system stability and transparency for force feedback. A
`perfectly transparent
`telemanipulator
`is not possible
`because it would require perfect models of the master
`and patient-side robot dynamics, zero time delays from
`computer processing and information transmission, and
`perfect environment force sensing or estimation. As one
`pushes toward the limit of transparency, small errors and
`delays in the system can cause uncontrollable oscillations
`in a ‘closed-loop’ teleoperator – this is known as instability
`and would be unacceptable in surgery. An alternative
`approach is to display force using sensory substitution to
`display force, including audio feedback [21], graphical
`feedback [22,23], or other forms of haptic feedback such
`as vibrotactile display [24]. Substantial information about
`environment properties and forces can be acquired by
`simply observing visually how the patient’s tissue and
`materials such as suture respond to motions of the surgical
`instruments. A design guideline is that sensory substi-
`tution through graphical overlays should not distract from
`the surgeon’s view of the patient via the endoscopic
`].
`camera(s) [25
`
`In the last few years, several research groups have used the
`force sensing and feedback techniques described earlier to
`test the effectiveness of haptic feedback on surgeon per-
`formance and ‘outcomes’ in phantom patients. All the
`experiments to date are preclinical. (Current commercial
`systems that use haptic feedback include those of Hansen
`Medical and MAKO Surgical Corp; however, no data exits
`demonstrating the relative effectiveness of those systems
`with and without haptic feedback) Ortmaier, et al. [17]
`found that haptic feedback reduced unintentional injuries
`during a dissection task. However, operating time was
`longer than that of a manual intervention. Wagner and
`Howe [13] found that force feedback reduces potential
`tissue damage (as measured by the level of applied force)
`
`for both surgeons and nonsurgeons, but only surgically
`trained individuals improve performance without a signifi-
`cant increase in trial time. They hypothesize that this is
`due to the interaction between visual-spatial motor abil-
`ities and the information contained in the mechanical
`interaction forces. Cao et al. [26] used a virtual environ-
`ment to demonstrate, the surgeons performed a Transfer
`Place task faster and more accurately with haptics than
`without, even under cognitive load.
`Mahvash et al. [27] used a modified da Vinci Surgical
`System to demonstrate that, in a palpation task, direct
`force feedback is superior to graphical force displays. Due
`to the limited fidelity of force feedback of their system
`(which did not use force sensors), users’ identification of
`hard lumps in artificial tissue was only correct for models
`with significantly different mechanical properties
`between the lumps and surrounding tissue. Zhou et al.
`[28] did a study of MIS showing that, with trocar friction,
`one of the undesirable forces that arise in RMIS, sur-
`geons’ force perception was degraded and the time to
`detect contact was longer. When friction was present,
`experienced surgeons detected contact with tissue faster
`than novices. Compared to no force feedback, Reiley et al.
`[25] found that graphical displays of applied force
`during a knot-tying task reduced suture breakage and
`overall applied forces, and increased consistency of
`applied forces for inexperienced robot-assisted surgeons.
`In contrast to the direct force feedback results from [26],
`the results of Reiley et al. [25] suggest that graphical
`force feedback primarily benefits novices, with diminish-
`ing benefits among experienced surgeons. In a simple
`grasping task, Tholey et al. [29] found that providing both
`vision and force feedback leads to better tissue charac-
`terization than only vision or force feedback alone.
`
`One would expect that better performance is achieved
`with direct force feedback than graphical feedback; sen-
`sory substitution systems are unnatural and thus have a
`longer learning curve, and direct force feedback provides
`physical constraints that helps a surgeon make the correct
`motions simply due to dynamic force balance [30]. There
`is an alternative to force feedback from the environment
`that provides such useful physical constraints: virtual fix-
`tures. These are software-generated force and position
`signals applied to human operators in order to improve
`the safety, accuracy, and speed of robot-assisted manip-
`ulation tasks [31]. For example, a virtual ‘wall’ may be
`placed around a delicate anatomical structure to keep the
`surgical instruments from contacting it.
`
`Although this article focuses on haptic feedback in actual
`surgeries, it is worth noting that the role of force feedback
`in training is a topic of current research. Some virtual
`reality simulators have proven effective in developing
`laparoscopic minimally invasive surgery (MIS) skills,
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Ethicon Exhibit 2018.003
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Haptic feedback in robot-assisted minimally invasive surgery Okamura 105
`
`especially when force feedback is provided in early
`training [32,33]. However, accurate modeling of tis-
`sue–instrument interaction that gives rise to motions
`and forces relevant to outcomes is not yet achievable
`at computation rates that allow real-time interaction.
`
`Tactile feedback
`Compared to force feedback, there has been relatively
`little work done in the area of tactile feedback for RMIS.
`In many surgical procedures, such as suture knot tying,
`force feedback is sufficient; the addition of contact
`location, finger pad deformation, and pressure distri-
`bution information may not be necessary [34]. However,
`palpation is one task for which deformation of the fin-
`gerpad seems to be particularly relevant [35,27], motiv-
`ating the need for tactile feedback.
`
`As in force feedback, tactile feedback systems require both
`a sensor and a display. The goal of tactile sensing in RMIS
`can be to detect local mechanical properties of tissue such
`as compliance, viscosity, and surface texture – all indica-
`tions of the health of the tissue – or to obtain information
`that can be used directly for feedback to a human operator,
`such as pressure distribution or deformation over a contact
`area [36]. Constraints in sensor design include cost, size,
`geometry (for example, to fit within a laparoscopic grasper),
`biocompatibility, and surface finish to allow grasping.
`Many sensors require some deformation of the sensor in
`order to measure distributed information; this necessitates
`flexible coverings, which also remove detailed, local infor-
`mation. In addition, recording data from tactile sensors is
`difficult because they often include many individual sen-
`sing elements; wireless communications are possible, but
`power must still be cabled to the instrument tip. Examples
`of tactile sensors include arrays of capacitive sensors [37]
`and force-sensitive resistors [38],
`instrumented mem-
`branes [39], and micromachined piezoelectric arrays
`[40]. Companies that sell tactile array systems include
`Pressure Profile Systems, Inc. (Los Angeles, California,
`USA) and TekScan, Inc. (South Boston, Massachusetts,
`USA). Data relevant to tactile information can also be
`obtained through other means, such as laparoscopic ultra-
`sound [41].
`
`Tactile displays attempt to create the perception that the
`surgeon’s fingertip is directly contacting the patient or
`surgical material such as suture. The most literal type of
`tactile display is an array of pins that are individually
`actuated (for example, [42]), so that their position com-
`mands are easily mapped from data from an array-type
`tactile sensor. Making such array-type displays for RMIS
`is very challenging due to size and weight constraints.
`The display must sit at the end of the master manipulator
`and not impede its maneuverability. Such pin displays
`developed for MIS and RMIS are actuated using shape-
`
`memory alloys [43], micromotors [44], and pneumatic
`,46]. The latter method allows the most
`systems [45
`lightweight display to be attached to the master manip-
`ulator, but requires infrastructure for air pressure, which
`can be noisy, and has limited resolution. Little work has
`been done to combine kinesthetic and tactile infor-
`mation for surgery, but one study demonstrates that
`the ability to maintain an appropriate force in the remote
`environment is necessary for the surgeon to take full
`advantage of the spatially distributed force information
`from the tactile sensor [47]. Graphical displays of tactile
`data can also be very compelling, especially for diagnosis
`applications [48,49,50]. Most of the tactile sensors and
`displays developed have not been tested in RMIS sys-
`tems. Due to the complexity of integrating distributed
`tactile information into RMIS, it may be useful in the
`future to consider clever ‘tactile illusions’ [51] and other
`display methods recently developed in the haptics
`research community.
`
`Conclusion
`Haptic feedback for RMIS is currently being developed
`and evaluated in engineering laboratories, and further
`development is required before these techniques are
`ready for clinical testing. Because the fidelity of haptic
`feedback and surgical scenario (e.g., degrees of freedom
`and type of surgery) of each research system is different,
`the results regarding the effectiveness of haptic feedback
`in the literature are not completely consistent. Contri-
`butions by neuroscientists to our understanding of how
`humans perceive force and tactile information affects how
`we design haptic displays. Promising new developments in
`the haptic technology and neuroscience fields may yield
`more efficient, practical force and tactile displays in the
`future. To accomplish these goals, surgeons, engineers,
`and neuroscientists need to work together to develop and
`test effective haptic displays for RMIS.
`
`Acknowledgements
`The author
`thanks current and former graduate students and
`postdoctoral fellows for their contributions to ideas presented in this
`review: J. Abbott, J. Gwilliam, K. Kuchenbecker, M. Mavash, C. Reiley,
`and L. Verner. This work was supported in part by National Institutes of
`Health grant R01 EB002004 and National Science Foundation grants
`IIS-0347464 and EEC-9731478.
`
`References and recommended reading
`Papers of particular interest, published within the annual period of review, have
`been highlighted as:
`
`of special interest
` of outstanding interest
`Additional references related to this topic can also be found in the Current
`World Literature section in this issue (p. 123).
`1
`Xin H, Zelek JS, Carnahan H. Laparoscopic surgery, perceptual limitations
`and force: a review. In: First Canadian Student Conference on Biomedical
`Computing, Kingston, Ontario, Canada; 2006. No. 144.
`
`2
`
`Burdea CG. Force and Touch Feedback for Virtual Reality. New York: Wiley
`Interscience; 1996.
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Ethicon Exhibit 2018.004
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`25 Reiley CE, Akinbiyi T, Burschka D, et al. Effects of visual force feedback on
`
`robot-assisted surgical task performance. J Thorac Cardiovasc Surg 2008;
`135:196–202.
`Authors showed that graphical displays of applied force during a knot-tying task
`reduced suture breakage and overall applied forces, and increased consistency of
`applied forces for inexperienced robot-assisted surgeons.
`
`26
`
`
`Cao CGL, Zhou M, Jones DB, Schwaitzberg SD. Can surgeons think and
`operate with haptics at the same time? J Gastrointest Surg 2007; 11:1564–
`1569.
`Authors use a virtual environment to show that force feedback increases accuracy.
`
`27
`
` Mahvash M, Gwilliam J, Agarwal R, et al. Force-feedback surgical teleopera-
`
`In: 16th Symposium on
`tor: controller design and palpation experiments.
`Haptic Interfaces for Virtual Environments and Teleoperator Systems 2008;
`pp. 465–471.
`Experimental results with a custom force feedback teleoperator based on da Vinci
`Surgical System hardware showed that direct force feedback is better for palpa-
`tion than graphical force feedback, but that attempting force feedback without
`using force sensors limits the system fidelity.
`
`28 Zhou M, Perreault J, Schwaitzberg SD, Cao CGL. Effects of experience on
`force perception threshold in minimally invasive surgery. Surg Endosc 2008;
`22:510–515.
`
`29 Tholey G, Desai JP, Castellanos AE. Force feedback plays a significant role in
`minimally invasive surgery: results and analysis. Annals of Surgery 2005;
`241:102–109.
`
`30
`
` Wagner CR, Stylopoulos N, Jackson PG, Howe RD. The benet of force
`
`feedback in surgery: examination of blunt dissection. Presence: teleoperators
`and virtual environments 2007; 16:252–262.
`Authors show that increased force feedback results in improved accuracy for a
`mock blunt dissection task. However, haptic feedback lengthens operation time for
`novice surgeons.
`
`31 Abbott JJ, Marayong P, Okamura AM. Haptic Virtual Fixtures for Robot-
`Assisted Manipulation.
`In: Thrun S, Durrant-Whyte H, Brooks R, editors.
`Robotics Research, Springer Tracts in Advanced Robotics, Vol. 28. 2007; pp.
`49–64.
`
`32 Basdogan C, De S, Kim J, Muniyandi M. Haptics in minimally invasive surgical
`simulation and training. IEEE Computer Graphics Applications 2004; 24:56–
`64.
`33 Stro¨ m P, Hedman L, Sa¨ rna˚ L, et al. Early exposure to haptic feedback enhances
`performance in surgical simulator training: a prospective randomized crossover
`study in surgical residents. Surg Endosc 2006; 20:1383–1388.
`
`34 Kitagawa M, Okamura AM, Bethea BT, et al. Analysis of suture manipulation
`forces for teleoperation with force feedback. In: Proceedings of the Fifth
`International Conference on Medical Image Computing and Computer As-
`sisted Intervention: MICCAI 2002, Lecture Notes in Computer Science, Vol.
`2488. Dohi T, Kikinis R, editors. 2002; pp. 155–162.
`
`35 Peine WJ, Howe RD. Do humans sense finger deformation or distributed
`pressure to detect lumps in soft tissue? In: Proc. of the ASME Dynamic
`Systems and Control Division, ASME International Mechanical Engineering
`Congress and Exposition, DSC-Vol. 64. 1998; pp. 273–278.
`
`36 Etlaib MEH, Hewit JR. Tactile sensing technology for minimal access surgery:
`a review. Mechatronics 2003; 13:1163–1177.
`37 Pawluk DTV, Son JS, Wellman PS, et al. A Distributed Pressure Sensor for
`Biomechanical Measurements. ASME J Biomech Eng 1998; 102:302–305.
`38 Schostek S, Ho CN, Kalanovic D, Schurr MO. Artificial tactile sensing in
`minimally invasive surgery: a new technical approach. Minim Invasive Ther
`Allied Technol 2006; 15:296–304.
`39 Dargahi J, Najarian S. A supported membrane type sensor for medical tactile
`mapping. Sensor Rev 2004; 24:284–297.
`40 Dargahi J, Parameswaran M, Payandeh S. A micromachined piezoelectric
`tactile sensor for an endoscopic grasper – theory, fabrication and experi-
`ments. J Microelectromechanical Syst 2000; 9:329–335.
`41 Leven J, Burschka D, Kumar R, et al. DaVinci Canvas: a telerobotic surgical
`system with integrated, robot-assisted, laparoscopic ultrasound capability. In:
`Proceedings of the Eighth International Conference on Medical Image Com-
`puting and Computer Assisted Intervention: MICCAI, Lecture Notes in
`Computer Science (Vol. 3750) 2005; pp. 811–818.
`42 Killebrew JH, Bensmaia SJ, Dammann JF, et al. A dense array stimulator to
`generate arbitrary spatio-temporal tactile stimuli. J Neurosci Methods 2007;
`161:62–74.
`43 Howe RD, Peine WJ, Kontarinis DA, Son JS. Remote palpation technology.
`IEEE Eng Med Biol 1995; 14:318–323.
`44 Ottermo MV, Stavdahl O, Johansen TA. Electromechanical design of a miniature
`tactile shape display for minimally invasive surgery. In: First Joint Eurohaptics
`Conference and Symposium on Haptic Interfaces for Virtual Environment and
`Teleoperator Systems (World Haptics) 2005; pp. 561–562.
`
`106 Robotics
`
`3
`
`Jones L, Lederman S. Human Hand Function. New York: Oxford University
`Press; 2006.
`
`4
`
`Lin MC, Otaduy MA, editors. Haptic Rendering: Foundations, Algorithms, and
`Applications. London: AK Peters, Ltd; 2008.
`Provides a current and in-depth review of the field of haptics, focusing on virtual
`environments. Medical applications are covered.
`
`5 Hayward V, MacLean KE. Do it yourself haptics, Part-I. IEEE Robot. Autom
`Mag 2007; 14:88–104.
`
`6 MacLean KE, Hayward V. Do It Yourself Haptics, Part-II. IEEE Robot Autom
`Mag 2008; 15:104–119.
`
`7
`
`In: Siciliano B, Khatib O,
`Hannaford B, Okamura AM. Chapter 30: Haptics.
`editors. Handbook of Robotics. New York: Springer; 2008.
`Provides a brief overview of the field of haptics suitable for a scientifically literate
`audience.
`
`8 Hayward V, Astley OR, Cruz-Hernandez M, et al. Haptic interfaces and
`devices. Sensor Rev 2004; 24:16–29.
`
`9
`
`Salisbury K, Conti F, Barbagli F. Haptic rendering: introductory concepts.
`IEEE Comput Graphics Applicat 2004; 24:24–32.
`
`10 Robles-De-La-Torre G. The importance of the sense of touch in virtual and real
`environments. IEEE Multimedia 2006; 13:24–30.
`
`11 Westebring - van der Putten EP, Goossens RHM, Jakimowicz JJ, Dankelman J.
`
`Haptics in minimally invasive surgery: a review. Minim Invasive Ther Allied
`Technol 2008; 17:3–16.
`Reviews the literature from 1985 to 2007 for haptics in both MIS and RMIS.
`
`12 Puangmali P, Althoefer K, Seneviratne LD, et al. State-of-the-art in force and
`
`tactile sensing for minimally invasive surgery. IEEE Sensors Journal 2008;
`8:371–381.
`Reviews recent developments of haptic sensing in MIS.
`
`13
`
` Wagner CR, Howe RD. Force Feedback Benet Depends on Experience in
`
`Multiple Degree of Freedom Robotic Surgery Task. IEEE Transactions on
`Robotics 2007; 23:1235–1240.
`Demonstrates a complete teleoperation system for force feedback RMIS and
`shows that, while all users improve accuracy with force feedback, only experienced
`surgeons do not take longer to perform a surgical task.
`
`14 Dargahi J, Sedaghati R, Singh H, Najarian S. Modeling and testing of an
`endoscopic piezoelectric-based tactile sensor. Mechatronics 2007; 17:
`462–467.
`
`15 Kuebler B, Seibold U, Hirzinger G. Development of actuated and sensor
`integrated forceps for minimally invasive robotic surgery. Int J Med Robotics
`Comput Assist Surg 2005; 1:96–107.
`
`16 Zemiti N, Morel G, Ortmaier T, Bonnet N. Mechatronic design of a new robot
`for force control in minimally invasive surgery. IEEE/ASME Transactions on
`Mechatronics 2007; 12:143–153.
`
`17 Ortmaier T, Deml B, Kuebler B, et al. Robot assisted force feedback surgery.
`
`In: Ferre M, Buss M, Aracil R, et al., editors. Advances in Telerobotics,
`Springer Tracts in Advanced Robotics (STAR) Vol. 31. New York: Springer;
`2007. pp. 341–358.
`With a custom teleoperation system designed for RMIS, the authors showed that
`haptic feedback reduced unintentional injuries during a dissection task. However,
`operating time was longer than that of a manual intervention.
`
`18 Mahvash M, Okamura AM. Friction compensation for enhancing transparency
`of a teleoperator with compliant transmission. IEEE Transactions on Robotics
`2007; 23:1240–1246.
`
`19 Semere W, Kitagawa M, Okamura, AM. Teleoperation with sensor/actuator
`asymmetry: task performance with partial force feedback. In: 12th Symposium
`on Haptic Interfaces for Virtual Environments and Teleoperator Systems
`2004; pp. 121–127.
`
`20 Verner LN, Okamura AM. Effects of translational and gripping force feedback
`are decoupled in a 4-degree-of-freedom telemanipulator. In: Second Joint
`Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual
`Environment and Teleoperator Systems (World Haptics); 2007, pages 286–
`291.
`
`21 Kitagawa M, Dokko D, Okamura AM, Yuh DD. Effect of Sensory Substitution
`on Suture Manipulation Forces for Robotic Surgical Systems. J Thorac
`Cardiovasc Surg 2005; 129:151–158.
`
`22 Okamura AM. Methods for haptic feedback in teleoperated robot-assisted
`surgery. Industrial Robot 2004; 31:499–508.
`
`23 Tavakoli M, Aziminejad A, Patel RV, Moallem M. Methods and Mechanisms for
`Contact Feedback in a Robot-Assisted Minimally Invasive Environment. Surg
`Endosc Other Interv Tech 2006; 10:1570–1579.
`
`24 Schoonmaker RE, Cao CGL. Vibrotactile force feedback system for minimally
`invasive surgical procedures. In: IEEE Conference on Systems, Man, and
`Cybernetics 2006; pp. 2464–2469.
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Ethicon Exhibit 2018.005
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Haptic feedback in robot-assisted minimally invasive surgery Okamura 107
`
`45
`
`Culjat MO, King CH, Franco ML, Bisley JW, Dutson E, Grundfest WS,
`‘Pneumatic balloon actuators for tactile feedback in robotic surgery,’ Industrial
`Robot 2008; 35: 449–455.
`The authors have developed a tactile display consisting of an array of small
`balloons that provide spatially distributed forces to the fingertip. The device
`was integrated with the da Vinci surgical system.
`46 Moy G, Wagner C, Fearing RS. A compliant tactile display for teletaction. In:
`IEEE International Conference on Robotics and Automation 2000; pp. 3409–
`3415.
`47 Feller RL, Lau CKL, Wagner CR, et al. The effect of force feedback on remote
`palpation. IEEE International Conference on Robotics and Automation 2004;
`pp. 782–788.
`48 Hosseini M, Najarian S, Motaghinasab S, Dargahi J. Detection of tumours
`using a computational tactile sensing approach. Int J Med Robotics Comput
`Assist Surg 2006; 2:333–340.
`
`49
`
`Dargahi J, Najarian S, Ramezanifard R. Graphical display of tactile sensing
`data with application in minimally invasive surgery. Can J Electrical Comput
`Eng 2007; 32:151–155.
`A laparoscopic grasper equipped with custom tactile array sensors was used to
`provide a graphical representation of the tactile data on a computer monitor.
`
`50 Miller AP, Hammoud Z, Son JS, Peine WJ. Tactile imaging system for
`
`localizing lung nodules during video assisted thoracoscopic surgery. IEEE
`International Conference on Robotics and Automation 2007; pp. 2996–
`3001.
`A graphical overlay on endoscopic images shows a color map on the surface of
`tissue. Results indicate that subjects could localize stiff lumps more accurately
`using the system.
`
`51 Hayward V. A Brief Taxonomy of Tactile Illusions and Demonstrations That
`Can Be Done In a Hardware Store. Brain Research Bulletin 2008; 75:742–
`752.
`
`Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
`
`Ethicon Exhibit 2018.006
`Intuitive v. Ethicon
`IPR2018-01254
`
`