0
Select Articles

Intelligent Surgical Robots with Situational Awareness PUBLIC ACCESS

From Good to Great Surgeons

[+] Author Notes
Nabil Simaan

Associate Professor, ARMA Laboratory, Department of Mechanical Engineering, Vanderbilt University

Rusell H. Taylor

Professor Laboratory for Computational Sensing + Robotics, Department of Computer Science, Johns Hopkins University

Howie Choset

Professor, Biorobotics Laboratory, Robotics Institute, Carnegie Mellon University

Mechanical Engineering 137(09), S3-S6 (Sep 01, 2015) (4 pages) Paper No: ME-15-SEP-6; doi: 10.1115/1.2015-Sep-6

This article discusses various challenges and puts forth the concepts of intelligent surgical robots and complementary situational awareness (CSA) as a means for achieving new surgical systems with unprecedented capabilities in terms of safety, ease of operation, and exact execution of pre-operative surgical plans. Situational awareness is defined in accordance with by the three stages of sensory acquisition, sensory comprehension, and projection. A robotic system with CSA assists the user not only in manipulation, but also in forming the situational awareness regarding the task at hand by using perception resources beyond the capabilities of the user. In the future, it is expected that CSA systems will increasingly be embedded within a larger framework of Computer-Integrated Interventional Medicine, in which patient-specific information such as images, lab results, and genomics are combined with general knowledge to model and diagnose the patients’ condition and to develop an optimized treatment plan.

During the past thirty years, surgeons gradually converted open surgical procedures to minimally invasive laparoscopy and then to robot-assisted multi-port minimally invasive surgery (MIS). This conversion from open to MIS surgical technique has been driven by the aim of decreasing patient trauma, wound site infection, risk of incisional hernia, and post-operative recovery time and scarring.

Surgeons use multiple incisions (typically three to six incisions) to access the anatomy during multi-port MIS. By using insufflation of the surgical site (e.g. the abdomen) the operational field is enlarged to facilitate visualization and operation of multiple tools. Typically, three tools are used (right and left arms for manipulation/ablation and a third arm for visualization). Other access ports may be used for organ retraction and auxiliary tasks of suction or delivery of tools such as blood vessel clips. For instance, the da-Vinci Si system (Figure 1) uses a quadruple-armed tele-manipulator allowing the operation of additional tools and collaboration among two surgeons. The dexterous tools of the da-Vinci slave robot are matched via a telemanipulation interface to the mechanical architecture of the wrists of the master user interface. A high level telemanipulation computer relays commands from the master user interface to the slave arms thus allowing motion replication of surgeon's hand movements with tremor filtering and motion scaling.

Figure 1 The da-Vinci Si system: (a) the patient side manipulator, (b) the surgeon master interface, (c) the correspondence between dexterous tool wrists and the master interface.

Grahic Jump LocationFigure 1 The da-Vinci Si system: (a) the patient side manipulator, (b) the surgeon master interface, (c) the correspondence between dexterous tool wrists and the master interface.

Robotic systems such as in Figure 1 have successfully enabled many surgical procedures using the multiport MIS approach. However, with the aim of further reducing trauma to the patient, the last decade has seen significant growth in works investigating MIS in confined spaces, single port access (SPA) and natural orifice trans-luminal endoscopic surgery (NOTES). These new surgical paradigms present surgeons with unprecedented challenges that require a new approach to robotic assistance. This paper discusses these challenges and puts forth the concepts of intelligent surgical robots and complementary situational awareness (CSA) as means for achieving new surgical systems with unprecedented capabilities in terms of safety, ease of operation, and exact execution of pre-operative surgical plans. Within the context of this paper, intelligent surgical robots are robots capable of sensing and regulating their interaction with the environment in order to assist the surgeon in achieving safe surgical intervention and to facilitate CSA. Situational awareness is defined in accordance with [1] by the three stages of sensory acquisition, sensory comprehension, and projection (projecting the interpretation of sensed data to decide on a future action). A robotic system with CSA assists the user not only in manipulation, but also in forming the situational awareness regarding the task at hand by using perception resources beyond the capabilities of the user.

In the following sections, we show that the emerging surgical paradigms such as NOTES require new robot designs and human-robot interaction framework that go beyond the use of robots and computer assistance to allow manipulation augmentation. We will show that, while the two approaches of haptics and sensory immersion through virtual reality help surgeons overcome the sensory acquisition step, they do not help surgeons with obtaining full situational awareness. We will put forth the concept of CSA as a natural progression beyond these two approaches thereby allowing robots to help surgeons In Interpreting the surgical scene and in projecting the perceived intraoperative sensory data to allow exact safe operation and the execution of pre-operative surgical plans.

The early drivers for robot surgical assistance stemmed from the desire to improve patient outcomes by achieving two goals: 1) offering patients the benefits while sparing surgeons the difficulties of laparoscopic MIS, 2) improving the accuracy of surgical execution of pre-operative surgical plans to ensure optimal outcomes. These two goals have driven most of the medical robotics research in the past three decades and resulted in several robotic systems reviewed in [2] and more recently in [3].

The concept of robotics for manipulation augmentation was introduced to overcome the challenges of manual laparoscopy. This radical move decreased the learning curve of surgeons who no longer had to contend with the reverse manipulation mapping typical of manual laparoscopy. Compared to laparoscopy robots provided increased dexterity, allowed the manipulation of multiple arms, improved precision and steadiness and lowered the physiological performance requirements of surgeons.

To improve surgical plan execution accuracy, computer assisted surgery was introduced in order to provide perception augmentation through intraoperative imaging and guidance. By using image registration a pre-operative surgical plan is matched to the intraoperative surgical reality. While image registration proved feasible for rigid anatomy (e.g. Orthopedics), it has been elusive in general surgery due to organ flexibility, deformation and gravitational shift and/or swelling.

The advantages of telemanipulation robot assistance came at the cost of removing the surgical tool from the surgeon's hand, thus, resulting in limited sensory perception and situational awareness. Surgeons cannot feel the tool Interaction with the anatomy and current commercial systems still do not provide force feedback. Surgeons are also challenged with interpreting and relating the surgical scene with pre-operative imaging.

Haptic feedback, augmented reality and assistive manipulation have been proposed to alleviate the loss of sensory presence and situational awareness. Haptic feedback aims at restoring sensory presence through force feedback or by sensory substitution (e.g. substituting force with sound feedback). Augmented reality partially restores situational awareness via image overlay allowing the surgeon to superimpose an intraoperative image or visual cues to anatomical structures on the image display of the telemanipulation master console.Assistive manipulation uses control laws to help surgeons achieve critical surgical tasks. These control laws include active constraints and virtual fixtures. Active constraints enforce safety barriers preventing the surgical tool from venturing into sensitive anatomy. Virtual fixtures generalize this concept to facilitate tracing of a target geometry such as an ablation path or an anatomical surface (for an up-to-date review see [5]). Assistive manipulation can be applied during cooperative manipulation or telemanipulation. During cooperative manipulation, the robot and surgeon hold the tool and admittance control allows the surgeon to move the tool while benefiting from tremor filtering and enforcement of assistive manipulation laws. Examples of cooperative manipulation robots are the REMS robot and the Mako Rio® shown in Figure 2.

Figure 2 Examples of cooperative robots: (a) The JHU REMS robot for head & Neck microsurgery [4], (b) the MakoRio® for orthopedic surgery.

Grahic Jump LocationFigure 2 Examples of cooperative robots: (a) The JHU REMS robot for head & Neck microsurgery [4], (b) the MakoRio® for orthopedic surgery.

By and large, the frameworks of assistive manipulation, augmented reality, and haptics have historically evolved with multiport MIS in mind as an application domain. Intraoperative information seamlessly gathered by the surgeon's hand (e.g. stiffness/constraint cues) during open surgery is not used. Also, existing frameworks for assistive manipulation typically assume single and known contact between the end effector and the environment. The newly emerging surgical

Multi-port MIS requires several small incisions that generally heal well, but can also be associated with pain, scarring and potential wound infection and/or hematoma. To ameliorate surgical outcomes, SPA and NOTES have been proposed to reduce or eliminate the number of surgical access incisions. During an SPA procedure a single access port is placed in the abdomen to provide surgical access to the necessary tools. Figure 3 shows an example of the insertable robot effectors platform (IREP) [6] developed to operate on the abdomen through a single Ø15 mm port. During NOTES procedures natural orifices are used to access internal anatomy. Examples of NOTES access routes include transurethral, trans-nasal, trans-oral, trans-esophgeal, trans-gastric, trans-anal, and trans-vaginal surgeries. Figure 4 shows an example of the highly articulated robotic platform (HARP) designed to provide deep access into the anatomy and recently used for trans-oral surgery [7]. Much of the understanding we gained regarding the limitations of traditional manipulation paradigms has been through experience in using these two systems.

Figure 3 (a) The IREP single port access system, (b) the stereovision head with two seven degrees of freedom arms, (c) the distal wrist and gripper, (d) example of tying a double-throw knot.

Grahic Jump LocationFigure 3 (a) The IREP single port access system, (b) the stereovision head with two seven degrees of freedom arms, (c) the distal wrist and gripper, (d) example of tying a double-throw knot.

Figure 4 The HARP robot: (a) steering around a heart, (b) performing trans-oral access into the airways.

Grahic Jump LocationFigure 4 The HARP robot: (a) steering around a heart, (b) performing trans-oral access into the airways.

Figure 5 illustrates the encumbered difficulty of NOTES/SPA compared to multi-port MIS. In addition to the challenges of MIS, NOTES adds the complexity of operating in constrained workspace and traversing anatomical passageways. Unlike multi-port MIS where contact with the anatomy occurs only at the dexterous wrists on rigid shaft tools, in NOTES contact occurs along the length of the robot as it is inserted in anatomical passageways, Figure 6-(a). Also, in procedures such as trans-gastric abdominal surgery there is the significant challenge of obtaining wound closure within the gastric wall after completing the procedure. And compared to MIS where generally there is a correspondence between the motion range and shape of the wristed surgical tools and the surgeon's hand (e.g. Figure 1), in NOTES the robots must have many degrees of freedom and arms and this correspondence become significantly more complex to learn. Finally, while situational awareness is limited in MIS, the limitation is exacerbated in NOTES due to the further limited perception of robot shape and its interaction with the anatomy. Figure 6-(b) illustrates the risks of operating in confined space subject to the limited perception of the endoscope: collisions between the robot and the anatomy can occur outside the perception range of the surgeon.

Figure 5 The challenges of MIS and NOTES.

Grahic Jump LocationFigure 5 The challenges of MIS and NOTES.

Figure 6 Example of perception limitations affecting situational awareness of the user: (a) invisible multiple contacts, (b) contacts outside field of view, (c) the field of view visible to the surgeon.

Grahic Jump LocationFigure 6 Example of perception limitations affecting situational awareness of the user: (a) invisible multiple contacts, (b) contacts outside field of view, (c) the field of view visible to the surgeon.

When designing systems for NOTES/SPA, the prerequisites of safe deployment into the anatomical passageways, distal dexterity and collaborative multiarm workspace must be satisfied. Depending on the mechanical embodiment of the robot, there are four ways of achieving deep access into the anatomy. The first approach tasks the surgeon with steering the front end of the robot using camera visualization and uses the robot controller to execute a follow-the-leader task. Such design requires a very large number of actuators to control the shape of the robot and can lead to slow deployment. The second approach implements passive compliance with a steerable end tip. Once the robot reaches the site the robot structure can be actively locked. The Transport® endoscopic access system by USGI Medical Inc. is an example of such an approach. The third approach is by steering the robot tip while allowing the back end of the robot to alternate motions of locking and relaxation of the robot backbone in order to match the anatomical passageway during insertion. The HARP robot in Figure 4 uses this approach. Finally, the fourth approach requires the robot to use its sensing capabilities to actively comply with the constraint forces of the anatomy while tasking the surgeon with advancing the robot. Figure 7-(a) shows an example of this approach; which has also been reported in [8] to facilitate rapid trans-nasal access into the upper airway.

Figure 7 Continuum robots with intrinsic sensing capabilities demonstrating (a) active compliance to facilitate insertion [12], (b) palpation [13], (c) contact detection and localization [11].

Grahic Jump LocationFigure 7 Continuum robots with intrinsic sensing capabilities demonstrating (a) active compliance to facilitate insertion [12], (b) palpation [13], (c) contact detection and localization [11].

The second design prerequisite is distal dexterity with a dual-arm dexterous workspace. To effectively achieve dual-arm tasks, the robot arms must be able to oppose each other as human arms can. This challenging task is called triangulation of tools. This calls for designs enabling multiple dexterous arms to operate with almost parallel shafts. For instance the IREP, shown in Figure 3, has been designed with the ability to control the distance between its two snake-like arms in order to facilitate dexterous dual-arm operations such as knot tying. Other example of NOTES systems have been reviewed in [3].

Even if the design prerequisites of NOTES are satisfied, the success and safe use of these systems hinges on implementing advanced sensory and control capabilities to overcome the challenges in Figure 5. As initially proposed in [9], smart surgical tools can facilitate manipulation augmentation. In our work, we propose that intelligent surgical slaves are a critical component for enabling CSA. For example, intelligent robots with sensing capabilities can act as both surgical intervention and diagnostic tools much in the same way the surgeon's hand manipulates anatomy and helps in identifying stiff nodules invisible to the naked eye. These robots can act as perception augmentation tools by deploying sensory modalities that extend the human perception (e.g. ultrasound, optical coherence tomography, or confocal microscopy). As an example, the continuum robot shown in Figure 7-(b) is able to estimate forces and moments acting on its tip and it has been shown in [10] to enable palpation and building a stiffness map of a model prostate. To discern invisible contacts (as in Figure 6) [11] proposed kinematics-based models for detecting and localizing the contact. Figure 7-(c) shows a continuum robot demonstrating contact detection.

Finally, one of the key benefits of intelligent surgical slaves is their ability to offload cognitively burdensome tasks. To achieve this, the low level controllers of these robots must support force and motion regulation. For example, hybrid force/motion control can be used to facilitate controlled ablation along a path while maintaining a fixed contact force between the ablation probe and the anatomy. Our team is also working on other advanced capabilities of these robots including exploration of an unknown flexible environment with the aim of localizing and mapping the environment shape and stiffness and using this information for registration to a pre-operative model.

Figure 8 shows our envisioned tele-robotic system with advanced features of computer-aided surgery, taking advantage of the fact that telesurgery systems essentially place a computer between the surgeon and the patient. In addition to basic telesurgery capabilities such as high quality stereo video, distal dexterity, motion scaling, and tremor filtering, these Complementary Situational Awareness (CSA) systems have the capability to combine sensing, imaging, and model information to provide the surgeon with significantly enhanced information and decision support, using augmented reality visualization, haptic feedback, and other means. These systems can also assist the surgeon in manipulating tissue through the use of virtual fixtures and other assistive methods. Further, haptic, imaging, and other intraoperative sensing during the procedure can update and refine the computers of the patient and surgical situation.

Figure 8 From basic telemanipulation to advanced computer-aided surgery via manipulation and information augmentation.

Grahic Jump LocationFigure 8 From basic telemanipulation to advanced computer-aided surgery via manipulation and information augmentation.

In the future, we expect that CSA systems will increasingly be embedded within a larger framework of Computer-Integrated Interventional Medicine (Figure 9), in which patient-specific information such as images, lab results, and genomics are combined with general knowledge to model and diagnose the patients’ condition and to develop an optimized treatment plan. All of this information will be available to a CSA system to help the physician carry out the treatment plan and assess the results. This closed-loop process (blue loops) will occur over multiple time scales, from an entire patient treatment cycle down to every second in the operating room or interventional suite. Further the CSA system will function as a flight data recorder enabling the creation of a much more complete record of what happened in the operating room. All this information can then be related to observed outcomes and statistical methods can be used to improve treatment processes for future patients (red loop).

Figure 9 Computer-Integrated Interventional Medicine will exploit technology and information for modeling, planning, surgical execution, and evaluation to help physicians to treat each patient in an optimal manner (blue loops). Further, information about each intervention can be saved, compared to outcomes, and used to improve treatment processes for future patients (red loop).

Grahic Jump LocationFigure 9 Computer-Integrated Interventional Medicine will exploit technology and information for modeling, planning, surgical execution, and evaluation to help physicians to treat each patient in an optimal manner (blue loops). Further, information about each intervention can be saved, compared to outcomes, and used to improve treatment processes for future patients (red loop).

Research reported in this paper has been supported in part by the National Robotics Initiative through NSF grants IIS-1327566, IIS-1426655 and IIS-1327657 and by university funds.

Endsley, M. R., 1995. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37 (1), pp. 32– 64 [CrossRef]
Taylor, R., and Stoianovici, D., 2003. Medical robotics in computer-integrated surgery IEEE Transactions on Robotics and Automation, 19 (5), Oct, pp. 765– 781 [CrossRef]
Vitiello, V., Lee, S.-L., Cundy, T., and Yang, G.-Z., 2013. Emerging robotic platforms for minimally invasive surgery. IEEE Reviews in Biomedical Engineering, 6 pp. 111– 126 [CrossRef] [PubMed]
Olds, K., Chalasani, P, Pacheco-Lopez, P, lordachita, I., Akst, L., and Taylor, R., 2014. Preliminary evaluation of a new microsurgical robotic system for head and neck surgery. In IEEE/ RSJ International Conference on Intelligent Robots and Systems, pp. 1276– 1281
Bowyer, S., Davies, B., and Rodriguez y Baena, F., 2014. Active constraints/virtual fixtures: A survey IEEE Transactions on Robotics, 30 (1), Feb, pp. 138– 157 [CrossRef]
Ding, J., Goldman, R. E., Xu, K., Allen, P.K., Fowler, D. L., and Simaan, N., 2013. Design and Coordination Kinematics of an Insertable Robotic Effectors Platform for Single-Port Access Surgery IEEE/ASME transactions on mechatronics, 18 (5), Oct, pp. 1612– 1624 [CrossRef]
“Rivera-Serrano, C. M., Johnson, P, Zubiate, B., Kuenzler, R., Choset, H., Zenati, M., Tully, S., and Duvvuri, U.” The Laryngoscope.
Groom, K., Wang, L., Simaan, N., and Netterville, J., 2015. Robot-assisted transnasal laryngoplasty in cadaveric models: Quantifying forces and identifying challenges. The Laryngoscope, 125 (5), pp. 1166– 1168 [CrossRef] [PubMed]
Dario, P, Hannaford, B., and Menciassi, A., 2003. Smart surgical tools and augmenting devices. IEEE Transactions on Robotics and Automation, 19 (5), Oct, pp. 782– 792 [CrossRef]
Xu, K., and Simaan, N., 2010. Intrinsic Wrench Estimation and Its Performance Index for Multisegment Continuum Robots. IEEE Transactions on Robotics, 26 (3), June, pp. 555– 561 [CrossRef]
Bajo, A., and Simaan, N., 2012. Kinematics-based detection and localization of contacts along multisegment continuum robots’. IEEE Transactions on Robotics, 28 (2), April pp. 291– 302 [CrossRef]
Goldman, R. E., Bajo, A., and Simaan, N., 2014. Compliant motion control for multisegment continuum robots with actuation force sensing. IEEE Transactions on Robotics, 30 (4), pp. 890– 902 [CrossRef]
Xu, K., and Simaan, N., 2008. An Investigation of the Intrinsic Force Sensing Capabilities of Continuum Robots. IEEE Transactions on Robotics, 24 (3), June pp. 576– 587 [CrossRef]
Copyright © 2015 by ASME
View article in PDF format.

References

Endsley, M. R., 1995. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37 (1), pp. 32– 64 [CrossRef]
Taylor, R., and Stoianovici, D., 2003. Medical robotics in computer-integrated surgery IEEE Transactions on Robotics and Automation, 19 (5), Oct, pp. 765– 781 [CrossRef]
Vitiello, V., Lee, S.-L., Cundy, T., and Yang, G.-Z., 2013. Emerging robotic platforms for minimally invasive surgery. IEEE Reviews in Biomedical Engineering, 6 pp. 111– 126 [CrossRef] [PubMed]
Olds, K., Chalasani, P, Pacheco-Lopez, P, lordachita, I., Akst, L., and Taylor, R., 2014. Preliminary evaluation of a new microsurgical robotic system for head and neck surgery. In IEEE/ RSJ International Conference on Intelligent Robots and Systems, pp. 1276– 1281
Bowyer, S., Davies, B., and Rodriguez y Baena, F., 2014. Active constraints/virtual fixtures: A survey IEEE Transactions on Robotics, 30 (1), Feb, pp. 138– 157 [CrossRef]
Ding, J., Goldman, R. E., Xu, K., Allen, P.K., Fowler, D. L., and Simaan, N., 2013. Design and Coordination Kinematics of an Insertable Robotic Effectors Platform for Single-Port Access Surgery IEEE/ASME transactions on mechatronics, 18 (5), Oct, pp. 1612– 1624 [CrossRef]
“Rivera-Serrano, C. M., Johnson, P, Zubiate, B., Kuenzler, R., Choset, H., Zenati, M., Tully, S., and Duvvuri, U.” The Laryngoscope.
Groom, K., Wang, L., Simaan, N., and Netterville, J., 2015. Robot-assisted transnasal laryngoplasty in cadaveric models: Quantifying forces and identifying challenges. The Laryngoscope, 125 (5), pp. 1166– 1168 [CrossRef] [PubMed]
Dario, P, Hannaford, B., and Menciassi, A., 2003. Smart surgical tools and augmenting devices. IEEE Transactions on Robotics and Automation, 19 (5), Oct, pp. 782– 792 [CrossRef]
Xu, K., and Simaan, N., 2010. Intrinsic Wrench Estimation and Its Performance Index for Multisegment Continuum Robots. IEEE Transactions on Robotics, 26 (3), June, pp. 555– 561 [CrossRef]
Bajo, A., and Simaan, N., 2012. Kinematics-based detection and localization of contacts along multisegment continuum robots’. IEEE Transactions on Robotics, 28 (2), April pp. 291– 302 [CrossRef]
Goldman, R. E., Bajo, A., and Simaan, N., 2014. Compliant motion control for multisegment continuum robots with actuation force sensing. IEEE Transactions on Robotics, 30 (4), pp. 890– 902 [CrossRef]
Xu, K., and Simaan, N., 2008. An Investigation of the Intrinsic Force Sensing Capabilities of Continuum Robots. IEEE Transactions on Robotics, 24 (3), June pp. 576– 587 [CrossRef]

Figures

Tables

Errata

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In