Hermann Mayer*
`Department of Robotics and
`Embedded Systems
`Technische Universita¨t Mu¨nchen
`D-85748 Garching bei Mu¨nchen
`and Department of
`Cardiovascular Surgery
`German Heart Center Munich
`D-80636 Mu¨nchen
`
`Istvan Nagy
`Alois Knoll
`Department of Robotics and
`Embedded Systems
`Technische Universita¨t Mu¨nchen
`D-85748 Garching bei Mu¨nchen
`
`Eva U. Braun
`Robert Bauernschmitt
`Ru¨ diger Lange
`Department of Cardiovascular
`Surgery
`German Heart Center Munich
`D-80636 Mu¨nchen
`
`Haptic Feedback in a Telepresence
`System for Endoscopic Heart
`Surgery
`
`Abstract
`
`The implementation of telemanipulator systems for cardiac surgery enabled heart
`surgeons to perform delicate minimally invasive procedures with high precision un-
`der stereoscopic view. At present, commercially available systems do not provide
`force-feedback or Cartesian control for the operating surgeon. The lack of haptic
`feedback may cause damage to tissue and can cause breaks of suture material. In
`addition, minimally invasive procedures are very tiring for the surgeon due to the
`need for visual compensation for the missing force feedback. While a lack of Carte-
`sian control of the end effectors is acceptable for surgeons (because every move-
`ment is visually supervised), it prevents research on partial automation. In order to
`improve this situation, we have built an experimental telemanipulator for endo-
`scopic surgery that provides both force-feedback (in order to improve the feeling
`of immersion) and Cartesian control as a prerequisite for automation. In this article,
`we focus on the inclusion of force feedback and its evaluation. We completed our
`first bimanual system in early 2003 (EndoPAR Endoscopic Partial Autonomous Ro-
`bot). Each robot arm consists of a standard robot and a surgical instrument, hence
`providing eight DOF that enable free manipulation via trocar kinematics. Based on
`the experience with this system, we introduced an improved version in early 2005.
`The new ARAMIS system (Autonomous Robot Assisted Minimally Invasive Surgery)
`has four multi-purpose robotic arms mounted on a gantry above the working
`space. Again, the arms are controlled by two force-feedback devices, and 3D vision
`is provided. In addition, all surgical instruments have been equipped with strain
`gauge force sensors that can measure forces along all translational directions of the
`instrument’s shaft. Force-feedback of this system was evaluated in a scenario of ro-
`botic heart surgery, which offers an impression very similar to the standard, open
`procedures with high immersion. It enables the surgeon to palpate arteriosclerosis,
`to tie surgical knots with real suture material, and to feel the rupture of suture ma-
`terial. Therefore, the hypothesis that haptic feedback in the form of sensory substi-
`tution facilitates performance of surgical tasks was evaluated on the experimental
`platform described in the article (on the EndoPAR version). In addition, a further
`hypothesis was explored: The high fatigue of surgeons during and after robotic op-
`erations may be caused by visual compensation due to the lack of force-feedback
`(Thompson, J., Ottensmeier, M., & Sheridan, T. 1999. Human Factors in Telesurgery,
`Telmed Journal, 5 (2) 129 –137.).
`
`Presence, Vol. 16, No. 5, October 2007, 459 – 470
`© 2007 by the Massachusetts Institute of Technology
`
`*Correspondence to mayerh@in.tum.de
`
`Mayer et al. 459
`
`Ethicon Exhibit 2017.001
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`460 PRESENCE: VOLUME 16, NUMBER 5
`
`tering, and a stereo vision interface at the input console.
`Surgeons can now operate using a surgical mechatronic
`assistant in a comfortable, dextrous, and intuitive man-
`ner (Falk, Jacobs, Gummert, & Walther, 2003; Falk,
`Jacobs, Gummert, Walther, & Mohr, 2003; Falk,
`Mintz, Grunenfelder, Fann, & Burdon, 2001). Despite
`the obvious potential advantages of robot-assisted, en-
`doscopic surgery, most researchers and surgeons in this
`area agree that the lack of haptic feedback is the most
`important drawback of currently available systems (Mit-
`suishi, Tomisaki, Yoshidome, Hashizume, & Fujiwara,
`2000). The inability of the operator to sense the applied
`forces causes increased tissue trauma and frequent su-
`ture material damage. The systems are telemanipulators
`with no Cartesian position control (the control loop is
`implicitly closed by the visual surveilance of the surgeon).
`Both features are important in order to move the surgeon
`up in the control hierarchy, that is, to implement “shared
`control” or “partial autonomy.”
`In order to overcome these deficiencies, two crucial
`issues have to be solved. One is inclusion of force sen-
`sory and feedback, and the other is the implementation
`of full Cartesian control of the end effector. The latter is
`indispensable for calculating exact directions of forces in
`a known coordinate frame. Therefore, one of our main
`research interests is the construction and evaluation of
`force sensory/feedback in realistic scenarios of robotic
`surgery. In particular, we focus on instrumental suturing
`and knot-tying tasks, which are very time consuming if
`performed via telemanipulation. Our working hypothe-
`sis was that the handling of telemanipulated surgical
`systems can be significantly improved by the inclusion of
`force-feedback. Therefore, we focus below on hardware
`and software for force-feedback in endoscopic surgery
`and present an evaluation of its quality.
`
`2
`
`Related Work
`
`Telemanipulators for endoscopic surgery are al-
`ready commercially available. Systems, such as daVinci
`(Guthart & Salisbury, 2000), offer comfortable user
`interface and are used in daily practice to perform even
`delicate operations. However, they offer no force-
`
`Figure 1. Location of the instrument and camera port.
`
`1
`
`Introduction
`
`Recently, minimally invasive surgery (MIS) has
`become a promising option for a great number of surgi-
`cal interventions (such as heart surgery) and has had
`significant impact on both patients and surgeons. Mini-
`mally invasive and endoscopic cardiac surgery not only
`minimizes the collateral surgical trauma, but it also re-
`sults in quicker recovery. The length of the hospital stay
`and the infection rate can be reduced (Morgan et al.,
`2004). Therefore, patients can profit significantly from
`this treatment option. On the other hand, surgeons
`have to cope with increasingly complex working condi-
`tions. Since endoscopic surgery is performed through a
`small port or keyhole in the patient’s chest (Figure 1),
`surgeons must learn to operate with unfamiliar and of-
`ten awkward surgical instruments. Hence, the tech-
`niques of endoscopic surgery have been applied infre-
`quently, particularly in the field of heart surgery. An
`important step in developing this technology was the
`introduction of telemanipulation, which was especially
`designed to enable delicate interventions with high sur-
`gical precision. The surgeon no longer controls the in-
`struments directly, but they are controlled by a special
`device with a Cartesian user interface that surgeons can
`handle as usual, that is, like instruments for open sur-
`gery. They offer as much freedom of movement as the
`surgeon’s own hand would in conventional open sur-
`gery, thus providing 6 DOF instead of the 4 DOF that
`conventional endoscopic instruments have. In addition,
`they assist the surgeon with motion scaling, tremor fil-
`
`Ethicon Exhibit 2017.002
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Mayer et al. 461
`
`Figure 2. Master console and magnified view of a customized PHANToM stylus.
`
`feedback and their control loop is closed visually by the
`surgeon. In addition to these commercial systems, a
`variety of devices for robotic surgery do exist, imple-
`mented by other research groups. At the University of
`California, Berkeley, a robotic system was developed
`that has already been used to perform certain surgical
`tasks, such as suturing and knot-tying (Cavasoglu, Wil-
`liams, Tendick, & Sastry, 2003). The Korean Advanced
`Institute of Science and Technology developed a micro-
`telerobot system that provides force-feedback (Kwon,
`Woo, Song, Kim, & Cho, 1998). In Germany, two sys-
`tems for robotic surgery were built at the Research Fa-
`cility in Karlsruhe (Voges, Holler, Neisius, Schurr, &
`Vollmer, 1997) and at the DLR in Oberpfaffenhofen
`(Konietschke, Ortmaier, Weiss, & Hirzinger, 2004).
`While the first system provides no force-feedback, the
`latter is equipped with PHANToM devices for haptic
`display. There is also some work available dealing with
`analysis of knot-tying. At Johns Hopkins University,
`Kitagawa, Okamura, Bethea, Gott, and Baumgartner
`(2002) have evaluated forces occurring during knot-
`tying. They did not measure forces directly on the in-
`struments or during realistic operations, but with a con-
`trivance that was especially designed for these
`measurements. Wagner, Vasilyev, Perrin, del Nido, and
`Howe (2006) have proposed an instrument with force-
`feedback in order to provide additional modality for
`ultrasound-guided interventions. Tavakoli, Patel, and
`Moallem (2005) developed a novel endoscopic instru-
`ment with exchangeable head and force-feedback. Their
`
`evaluation showed that surgical tasks profit from force
`feedback, regarding performance time and reduction of
`fatigue. Cao, MacKenzie, and Payandeh (1996) ana-
`lyzed a variety of surgical tasks (among other things,
`knot-tying) and broke them down into subtasks. They
`did not include force measurement. However, the ne-
`cessity of haptic feedback in robotic surgery has been
`discussed controversially by surgeons and haptic engi-
`neers (Bethea et al., 2004; Fager, 2004; Hu, Tholey,
`Desai, & Castellanos, 2002; MacFarlane, Rosen, Han-
`naford, Pellegrini, & Sinanan, 1999). At the Rensselaer
`Polytechnic Institute, a robotic system capable of knot-
`tying was developed (Kang & Wen, 2001), but they
`mainly focused on force control and have used dedi-
`cated instruments for knot-tying. At the German Heart
`Center in Munich, a daVinci system has been installed.
`Surgeons often ponder the question of whether perfor-
`mance could be improved if force-feedback is included.
`Since this problem cannot be evaluated directly within
`the daVinci system, we decided to take the original in-
`struments and incorporate them into an experimental
`setup that can be used for evaluation.
`
`3 Materials and Methods
`
`Like typical systems for robotic surgery, our setup
`consists of an operator-side master console (Figure 2)
`for in-output and a patient-side robotic manipulator
`that directly interacts with the operating environment.
`
`Ethicon Exhibit 2017.003
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`462 PRESENCE: VOLUME 16, NUMBER 5
`
`Figure 3. New system ARAMIS with four ceiling mounted robots.
`
`3.1 Robotic Telemanipulator
`
`The ARAMIS system consists of four small robots
`mounted on the ceiling (Figure 3). The robots have 6
`DOF. Since the rotation of the robot’s flange and the
`rotation of the instrument share one axis, our system
`ultimately ends up having 8 DOF under the restriction
`of trocar kinematics (Mayer, Nagy, & Knoll, 2004).
`The crucial part of the master console is two PHANToM
`haptic interfaces. They are used to control the instruments
`and to feed back forces. Particular advantages of this
`setup with multipurpose robots are high precision and
`stiffness, moderate costs, and an advanced dynamic be-
`havior. The latter could be exploited to perform sophis-
`ticated tasks in motion compensation, for example, sup-
`port for beating heart surgery as it was proposed in
`Ortmaier, Groeger, Boehm, Falk, and Hirzinger (2005),
`or compensation for respiratory motion of the ribs.
`We have also developed adapters that are attached to
`the robotic arms and can be equipped with either an
`instrument or a stereo endoscope. For security reasons
`and better handling, we have equipped all flange adapt-
`ers with magnetic security couplings (Figure 4c). Those
`will disengage if forces beyond a certain level are exerted
`that might cause harm to the instruments or chest
`mockup. Each of the surgical instruments has 3 DOF. A
`microgripper at the distal end of the shaft can be ro-
`
`Figure 4. Sensorized instrument, magnetic adapter, and servos.
`
`tated, and adaptation of pitch and yaw angles is possi-
`ble. All movable parts of the gripper are driven by steel
`wires. Their motion is controlled by four driving wheels
`at the proximal end of the instrument, one for each de-
`gree of freedom (two for the yaw of the fingers). In or-
`der to control the joints of the instruments, we have
`flanged servomotors to each driving wheel by means of
`an Oldham coupling (Figure 4c). This guarantees in-
`strument movement that is free of jerk. The servocon-
`trollers can be connected via serial lines or a CANbus to
`a multiport card. The basic idea of endoscopic surgery is
`that only small openings have to be made into the sur-
`face of the patient’s body (so-called keyholes, Figure 1).
`Therefore, translational movements of the instruments
`are essentially restricted by shifts and rotations within
`these holes. In order to provide the surgeon with a
`comfortable user interface, it is desirable to map move-
`ments of the stylus of the input device directly onto in-
`strument motions. Considering these requirements, we
`have implemented the inverse kinematics of our system
`as shown in Figure 1.
`A desired position of the instrument is given by the
`position of the input stylus. Arbitrary positions of the
`
`Ethicon Exhibit 2017.004
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Mayer et al. 463
`
`Figure 5. Endoscopic stereo camera with customized magnetic adapter.
`
`instrument’s tip are mapped on corresponding positions
`of the motors that control the 8 DOF. The input data is
`represented by a homogenous transform matrix. Since
`the position of the instrument’s shaft is restricted by the
`port (the position of the keyhole), only one possibility
`for aligning the instrument exists. The angle of the cor-
`responding joints of the instrument can be found by
`geometric calculations, which are explained in detail in
`Mayer et al. (2004). In order to speed up this complex
`computation, the complete inverse kinematics of the
`robot can be performed on the graphical processing unit
`(GPU) of a graphics card. In order to prepare this com-
`putation, the desired Cartesian positions are transferred
`to a texture on the graphics card. A so-called fragment
`shader, which is written in the OpenGL Shading Lan-
`guage and contains the algorithm for inverse kinematics,
`is applied to the texture. The results (the joint angles of
`the robot) are transferred to the framebuffer of the card
`and can be moved back to the CPU. The calculated an-
`gles can be directly applied to the robot. Employment
`of a GPU (NVIDIA GeForce 5200 graphics card) leads
`to a computation time that is 20 times faster than a
`modern CPU (Athlon64 2200 MHz).
`As mentioned above, the master console (the work-
`station of the surgeon) consists of custom made modifi-
`cations of PHANToM force feedback devices (see below).
`
`3.2 Optical System
`
`Due to the extremely high-precision requirements
`in modern minimally-invasive surgery (arteries in heart
`surgery have a diameter between 1 and 2 mm), it is im-
`perative to provide an accurate 3D view.
`
`Most of the other systems mentioned above come
`with magnifying three-dimensional endoscopes and ap-
`propriate display devices. This is also an essential prereq-
`uisite for high accuracy machine vision. The ARAMIS
`system is equipped with a 3D endoscope providing two
`separate optical (fiberglass) channels and two synchro-
`nized CCD cameras. A magnetic coupling mechanism
`(Figure 5) allows quick mounting and dismounting
`without losing the calibration against the rest of the sys-
`tem. We have precisely determined the intrinsic and ex-
`trinsic parameters of the camera system in order to
`achieve the necessary calibration for scene reconstruc-
`tion. Like the instruments, the endoscopic camera can
`be moved by means of trocar kinematics and can either
`be actively controlled by the operator or automatically
`guided by the system (following the region of interest,
`i.e., where the instruments are placed). Images taken
`from the stereo camera system are displayed on a 3D
`device, which provide differentially-polarized images for
`right and left eye. Therefore, the surgeon has to wear
`polarized lenses in order to get a 3D view.
`
`3.3 Force Feedback
`
`Positions and orientations of the instruments are
`controlled by two PHANToM devices (Figure 2). This
`device is available in different versions with different
`capabilities. We have chosen the version PHANToM
`Premium 1.5. It has a working space of approximately
`20 ⫻ 25 ⫻ 40 cm, which is appropriate for performance
`of surgical procedures. The user controls a stylus pen
`that is equipped with a switch that can be used to open
`
`Ethicon Exhibit 2017.005
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`464 PRESENCE: VOLUME 16, NUMBER 5
`
`and close the microgrippers. We have extended the
`plain stylus with sleeves for thumb and forefinger in or-
`der to increase the user’s comfort (Figure 2).
`The most interesting feature of the employed
`PHANToM devices is the capability to display forces
`which are fed back by small servomotors incorporated
`into the device. They are used to steer the stylus pen in
`a certain direction. This creates the impression of occur-
`ring forces, while the user is holding the pen in a certain
`position. Our version of the PHANToMTM device can
`display forces in all translational directions, while no
`torque is fed back. In order to be able to display realistic
`forces during operation, we have equipped the instru-
`ments with force sensors.
`Since the shaft of the surgical instrument is made of
`carbon fiber, force sensors have to be very sensitive and
`reliable. Therefore, we decided to apply strain-gauge
`sensors, which are employed in industrial force registra-
`tion (Figure 4b).
`Strain-gauge sensors are usually made of silicon,
`which provides by far the best sensitivity with respect to
`elongations. Its piezoelectric resistance increases under
`dilation and decreases under compression. The resulting
`resistance R is linearly dependent on the axial stress
`(␴ ⫽ F/A), the relative elongation (␧ ⫽ ⌬l/l) of the re-
`sistor and its transversal contraction (⌬r ⫽ ⫺␮r␧),
`where ␮ is the Poisson ratio, a material-dependent con-
`stant. For small, reversible elongations (␧ ⬍ 10⫺3) the
`following equation holds:
`
`␴␧册␧ ⫽K 䡠 ␧
`
`⌬␴
`
`⌬R
`R
`
`⫽
`
`⌬l
`l
`
`⫺
`
`2⌬r
`r
`
`⫹
`
`⌬␴
`␴
`
`⫽冋1 ⫹ 2␮⫹
`
`(1)
`
`Therefore, resistance depends only on the elongation ␧
`and a material-dependent constant K, which denomi-
`nates the sensitivity of the material and accounts for a
`dimension of 100 for silicon. These strain gauges are
`very compact, cheap, and have fast reactions. We ap-
`plied the sensor gauges at the distal end of the instru-
`ment’s shaft, that is, near the gripper. This results in an
`inevitable disadvantage that sensor readings are flawed
`by the tension of the driving wires. This is particularly
`an issue of measurements in axial direction when the
`
`gripper is opened and closed. For that reason, we dis-
`abled this axis for our experiments and only took into
`account lateral movements. Due to their layout as full
`bridges (see below), they are not negatively impacted by
`the closing state of the gripper. However, since disturb-
`ing forces still can exceed an acceptable limit during
`extensive movements (e.g., knot-tying), we provided a
`switch to reduce/remove force levels in situations when
`advanced sensitivity is demanded. As mentioned above,
`the sensors were directly attached to the instruments by
`a heat-resistant, two-component adhesive and have been
`coated with shrink tubes. Therefore, this technology is
`waterproof and heat-resistant, in order to survive the
`medical autoclave. Since we did not know the exact ma-
`terial parameters of the instruments, we calibrated the
`sensor system with standard weights.
`In order to make sensor readings more stable, we em-
`ployed four sensor gauges interconnected as a Wheat-
`stone full bridge for each translational direction. Nor-
`mally, there are three possibilities to arrange the sensor
`gauges as a full bridge. We have chosen the layout de-
`picted in Figure 6 (left side) with all four gauges mea-
`suring elongations along the main axis. Sensor readings
`are independent of thermal fluctuations, since resulting
`changes of single sensors eliminate each other. In addi-
`tion, the chosen layout has the advantage that the re-
`sulting resistance is still linearly correlated with elonga-
`tions. Other possibilities presume that two of the
`sensors are arranged perpendicular to the axial direction
`in order to measure lateral strains. But they are either
`nonlinear or not resistant to changes in temperature.
`For measurement of x- and y-forces the bridge parts
`for dilation (1 and 3 in Figure 6) and compression (2
`and 4 in Figure 6) are attached on opposite areas of the
`shaft (Figure 7 left). Therefore the forces measured in
`these directions are independent from axial elongations
`as they may occur during the opening and closing of the
`gripper (induced by the steel wires inside the shaft). For
`digitizing the sensor readings, we have used GSV3CAN
`measurement amplifiers from ME-Messsysteme GmbH.
`The blank with the main components is very compact
`(Figure 7: blank size 30 ⫻ 55 mm) and can be attached
`directly onto the instruments. It also provides a CAN-
`bus interface which can transmit 1220 16 bit values per
`
`Ethicon Exhibit 2017.006
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Mayer et al. 465
`
`Figure 6. Layout of strain gauge sensors: four active strain gauges (left), two active and two oblique
`strain gauges in nonlinear arrangement (middle), two active and two oblique strain gauges in linear
`arrangement (right).
`
`Figure 7. Arrangement of the sensor gauges (left), measurement amplifier (right).
`
`second. This is necessary since haptic inter-action with
`stiff contacts requires a sampling rate of at least 1000
`Hz. For internal processing, the amplifier provides a
`sampling rate of 10 kHz. Therefore, it calculates the
`medium of 8 sensor evaluations and transmits them
`thereafter. Since we know the position and orientation
`of the grippers, we can transform the forces into the
`coordinate frame of the PHANToMTM devices. In addi-
`tion, forces can be arbitrarily amplified by our control soft-
`ware and recorded to data files for subsequent evaluation.
`
`3.4 Evaluation of Force-Feedback
`
`3.4.1 Human Participants and Haptic Levels.
`In order to test the quality of force-feedback and the
`overall handling of the system, we have conducted a
`comprehensive evaluation of our setup. The subjects of
`
`this study were 25 heart surgeons of varying levels of
`surgical training and age (Table 1). We divided them
`into three groups regarding their skill level.
`The study group consisted of one group of eight
`young surgeons, another group of twelve experienced
`surgeons, and a third group of five surgeons with ro-
`botic experience. The study included basic surgical and
`cardiac surgical procedures. A psychophysical scaling
`pretest of static force was performed beforehand, to
`evaluate the alignment of the implementation of force-
`feedback in the telepresent robotic system to human
`perception. This pretest demonstrated that due to indi-
`vidual differences, two different haptic levels—1:1 and
`1:2 haptics— had to be transferred to and used in the
`evaluation. The order of haptic conditions (type of tasks
`and haptic level) was completely balanced to avoid
`learning effects and was double blinded.
`
`Ethicon Exhibit 2017.007
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`466 PRESENCE: VOLUME 16, NUMBER 5
`
`Table 1. Random Sample of the Human Subjects
`
`All participants
`n⫽25
`
`Young surgeons
`n⫽8
`
`Experienced
`surgeons
`n⫽12
`
`Robotic surgeons
`n⫽5
`
`7.4 (⫾6.9)
`
`0.4 (⫾0.3)
`
`11.5 (⫾7.6)
`
`9.0 (⫾5.2)
`
`36 (⫾8)
`
`29 (⫾7)
`
`39 (⫾8)
`
`39 (⫾5)
`
`0
`
`0
`
`0
`
`2.4 (⫾2.0)
`
`Surgical experience (years)
`mean value ⫾ standard
`deviation
`Age (years) mean value ⫾
`standard deviation
`Robotic experience (years)
`mean value ⫾ standard
`deviation
`
`3.4.2 Surgical Tasks. 3.4.2.1 Knot-tying. The
`human subjects had to tie surgical knots with two surgi-
`cal instruments equipped with haptic feedback. The sur-
`geons had ten minutes to perform as many knots as pos-
`sible in alternating manner (left and right tied knots).
`The total number of knots, the applied forces, the break-
`age of suture material, and the speed of motion during
`knot-tying have been recorded.
`
`3.4.2.2 Breaking of Suture Material. The breaking
`of suture material represents the amount of telepresence
`and immersion of the robotic system for the surgeons,
`and was considered an indicator for the technical quality
`of force feedback. The surgeons had to strain the thread
`until an assumed breaking point was reached at which
`point they had to indicate imminent breaking. The dif-
`ference of force between the indicated point and the
`actual breaking point of the thread was measured in
`Newtons. The surgical suture material used, ProleneTM
`6-0 (Ethicon Inc., Somerville, NJ, USA), is a common
`and frequently used nonabsorbable thread used for
`heart surgery.
`
`3.4.2.3 Detection of Arteriosclerosis. The subjects
`had to detect a possible stenosis with one haptic instru-
`ment (dominant hand), in artificial arteries made from
`polymer (Figure 8), precisely and at the same time rap-
`idly. Detection error, regarding the existence or size of
`
`the stenoses, have been counted. In addition, the ap-
`plied forces during detection have been recorded and
`the time needed for detection in seconds.
`
`3.4.3 Workflow and Critical Flicker Fusion
`Frequency (CFF). The critical flicker fusion frequency
`(CFF) is an individual part of the Wiener Testsystem
`(Schuhfried GmbH, Mo¨dling, Austria) analyzing the
`progression of fatigue during the evaluation (Wiemeyer,
`2002). The CFF is regarded as an indicator of the cen-
`tral-nervous function capacity, the activation level, and
`the progression of fatigue during practical tasks (Johans-
`son & Sandstroem, 2003). The CFF is defined as me-
`dian of the fusion frequency in the ascending and de-
`scending process.
`
`3.4.4 Training Skills. Before performing the
`actual tasks (knot-tying, breaking, and detecting), the
`surgeons underwent a short training program to be-
`come familiar with the robotic system. They had to
`move small objects with the grippers and thread a
`rubber band through narrow eyelets (Figure 9). Af-
`terwards, all tasks were repeated with three different
`levels of haptic feedback: absence of feedback, feed-
`back with real measured forces, and amplified force-
`feedback (factor two). Each surgeon had to perform
`each task at all force levels (Table 2). Since tasks had
`to be performed in dedicated task-sequences with
`
`Ethicon Exhibit 2017.008
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Mayer et al. 467
`
`Figure 8. Mockup for arteriosclerosis detection.
`
`Figure 9. Workplate with eyelets for training.
`
`double blinding, the sequence of force levels for each
`task was chosen randomly.
`
`4
`
`Results
`
`With the setup described above, we had already
`performed automized procedures like knot-tying (Nagy,
`Mayer, Knoll, Schirmbeck, & Bauernschmitt, 2004;
`Bauernschmitt et al., 2004). The focus of this article is
`on the evaluation of the impact of force measurement
`and feedback on surgical procedures. Our hypothesis,
`which has been proven correct (see below), was that
`haptic feedback contributes to better performance of
`systems for robotic surgery by preventing force-induced
`damage.
`
`4.1 Evaluation of the System
`
`The experiments which were conducted with the
`surgeons from the German Heart Center in Munich
`have shown the following results: increasing the amplifi-
`cation of force feedback during knot-tying reduces the
`applied forces significantly (p ⬍ .05, Figure 10a), but
`does not cause significant change in the performance
`time.
`During breaking of suture material, the correspond-
`ing differences of forces were calculated as described
`above. Increasing the magnitude of haptic feedback
`leads to a decrease in this difference (p ⬍ .05, Figure
`10b). This militates in favor of the precision of the esti-
`mated force when the thread was breaking. And it
`
`Table 2. Workflow of the Evaluation: (*) Randomly Presented Sequence of Force-Feedback Levels
`
`*Force-feedback levels were absence of feedback, feedback with real measured forces, and amplified force-feedback.
`
`Ethicon Exhibit 2017.009
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`468 PRESENCE: VOLUME 16, NUMBER 5
`
`Figure 10. Forces while knot-tying (left); with increasing haptic feedback the applied forces decrease significantly (right: *p ⬍ .05). The
`difference of the assumed and the real force while breaking a surgical suture decreases significantly with haptic feedback (*p ⬍ .05).
`
`Figure 11. Comparison of surgeons with different skill levels in detection of arteriosclerosis and the surgeon’s fatigue.
`
`shows that the telemanipulator system provides a high
`level of immersion. Similar results have been found for
`detection of arteriosclerosis. Haptic feedback signifi-
`cantly influences the amount of applied forces: an in-
`crease in the amplification of force-feedback leads to a
`
`significant decrease of applied forces. However, haptic
`feedback has no significant influence on detection errors
`(p ⫽ .05, Figure 11a) and performance time. And lastly,
`with regard to fatigue, operating with haptic feedback
`leads to a significant decrease (p ⬍ .05, Figure 11b).
`
`Ethicon Exhibit 2017.010
`Intuitive v. Ethicon
`IPR2018-01254
`
`

`

`Mayer et al. 469
`
`5
`
`Discussion and Conclusion
`
`The goal of these experiments was to examine
`claims about the necessity of force-feedback for robot-
`assisted surgical procedures in cardiac surgery. We
`present an approach of evaluating haptic feedback with a
`novel robotic system for minimally invasive and endo-
`scopic surgery. Haptic feedback is needed for surgical
`tasks since less force is applied by the surgeon. The
`number of tied knots does not increase, but the experi-
`ments have shown that haptic feedback can be em-
`ployed to prevent the surgeon from potentially harmful
`mistakes, such as the breakage of suture material and
`the loss of the surgical needle. The fatigue level of the
`surgeon decreases and the perception of telepresence by
`the surgeons increases. The safety level of the patient
`increases when being operated on by a telemanipulator
`with integrated haptic. These results lead to the conclu-
`sion that haptic feedback is an indispensable feature for
`surgical telemanipulators, in particular for operations
`with delicate suture material, such as is used in heart
`surgery. To further research and confirm the findings of
`these experiments, the next step would be to examine
`an animal model in order to further define surgical quality.
`
`Acknowledgments
`
`This work is supported by the German Research Foundation
`(DFG) within the Collaborative Research Center SFB 453-I4
`on “High-Fidelity Telepresence and Teleaction” and by the
`German Heart Center Munich, Department of Cardiovascular
`Surgery.
`
`References
`
`Bauernschmitt, R., Schirmbeck, E. U., Knoll, A., Mayer, H.,
`Nagy, I., Lange, R., et al. (2004). Towards robotic heart
`surgery: Introduction of autonomous procedures into an
`experimental surgical telemanipulator system. The Interna-
`tional Journal of Medical Robotics and Computer Assisted
`Surgery, 1(3), 74 –79.
`Bethea, B. T., Okamura, A. M., Kitagawa, M., Fitton, T. P.,
`
`Cattaneo, S. M., Gott, V. L., et al. (2004). Application of
`haptic feedback to robotic surgery. Journal of Laparoendo-
`scopic Advances in Surgical Technique, 14(3), 191–195.
`Cao, C., MacKenzie, C., & Payandeh, S. (1996). Task and
`motion analyses in endoscopic surgery. Proceedings of ASME
`Dynamic Systems and Control Division (Fifth Annual Sympo-
`sium on Haptic Interfaces for Virtual Environment and Tele-
`operator Systems), 583–590.
`Cavasoglu, M., Williams, W., Tendick, F., & Sastry, S. S.
`(2003). Robotics for telesurgery: Second generation Berke-
`ley/UCSF laparoscopic telesurgical workstation and looking
`towards the future applications. Industrial Robot, Special
`Issues on Medical Robotics, 30(1), 22–29.
`Fager, P. J. (2004). The use of haptics in medical applications.
`International Journal of Medical Assisted Surgery, Robotic
`Computer 1(1), 36 – 42.
`Falk, V., Jacobs, S., Gummert, J., & Walther, T. (2003). Ro-
`botic coronary artery bypass grafting (CABG)—The Leipzig
`experience. Surgical Clinician North America, 83(6), 1381–
`1386.
`Falk, V., Jacobs, S., Gummert, J., Walther, T., & Mohr, F. W.
`(2003). Computer-enhanced endoscopic coronary artery
`bypass grafting: The da Vinci experience. Seminal Thoracic
`Cardiovascular Surgery, 15(2), 104 –111.
`Falk, V., Mintz, D., Grunenfelder, J., Fann, J. I., & Burdon,
`T. A. (2001). Influence of three-dimensional vision on sur-
`gical telemanipulator performance. Surgical Endoscopy,
`15(11), 1282–1288.
`Guthart, G. S., & Salisbu