0
Select Articles

# A Passivity-Based System Design of Semi-Autonomous Cooperative Robotic SwarmOPEN ACCESS

[+] Author Notes
Takeshi Hatanaka

1School of Engineering Tokyo Institute of Technology

Nikhil Chopra

2Department of Mechanical Engineering University of Maryland

Junya Yamauchi

3Department of Mechanical and Control Engineering Tokyo Institute of Technology

Mamoru Doi

4Department of Systems and Control Engineering Tokyo Institute of Technology

Yasunori Kawai

5Department of Electrical Engineering National Institute of Technology Ishikawa College

Masayuki Fujita

6School of Engineering Tokyo Institute of Technology

Takeshi Hatanaka is an Associate Professor with School of Engineering at Tokyo Institute of Technology, Tokyo, Japan. He received Ph.D. degree in applied mathematics and physics from Kyoto University, Japan in 2007. His research interests include networked robotics and energy management systems.

Nikhil Chopra is an Associate Professor with the Department of Mechanical Engineering, University of Maryland, College Park, MD, USA. He received Ph.D. degree in systems and entrepreneurial engineering from the University of Illinois at Urbana-Champaign in 2006. His research interests include networked control systems, cooperative control of networked robots, and bilateral teleoperation.

Junya Yamauchi is a Ph.D. student of Department of Mechanical and Control Engineering, Graduate School of Science and Engineering, Tokyo Institute of Technology. His research interests include networked robotics.”

Mamoru Doi is a master student of Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology. His research interests include networked robotics.

Yasunori Kawai is an Associate Professor with the Department of Electrical Engineering at National Institute of Technology, Ishikawa College. He received Ph.D. in Department of Electrical Engineering from Kanazawa University, Japan, 2005. His research interests include model predictive control and bilateral teleoperation.

Masayuki Fujita is a Professor with School of Engineering at Tokyo Institute of Technology. He is also a Research Supervisor for Japan Science and Technology Agency Core Research for Evolutional Science and Technology. He received the Dr. of Eng. degree in Electrical Engineering from Waseda University, Tokyo, in 1987. His research interests include passivitybased control in robotics, distributed energy management systems and robust control.

Mechanical Engineering 139(06), S14-S18 (Jun 01, 2017) (5 pages) Paper No: ME-17-JUN6; doi: 10.1115/1.2017-Jun-6

## Abstract

This article presents a novel semiautonomous cooperative robotic system. Information flows in the architecture are designed based on the passivity property. The human passivity needed to guarantee the control goal is demonstrated using a human-in-the-loop simulator and system identification techniques. As a solution to stable coordination via bilateral human–swarm interactions, the research group built the system consisting of multiple planar robots, an operator and a computer whose role is to implement the distributed motion synchronization law. Experimental results on remote control of the robots are finally demonstrated. A typical approach to stabilization of an interconnected system that includes a highly uncertain component like a human operator is to assume passivity of the component block. Specifically, modeling of the operator has been addressed in this manner for several studies in bilateral teleoperation. It is observed from the figures provided in the article that the robots form and maintain the specified formation while they are maneuvered stably toward the desired references by the operator.

## Article

Over the past few decades, significant advancements have been accomplished in autonomous control of robotic systems. Nevertheless, in several critical applications, the inclusion of the human operator in the closed-loop system is necessary due to several reasons, the most compelling being that the autonomy of robotic systems has not advanced significantly to a level where they can be completely entrusted to operate autonomously in possibly unknown environments. Hence, complex robotic coordination tasks in highly uncertain environments require the system designer to take advantage of human operator's strengths like high-level decision-making and flexibility.

Motivated by these necessities, semi-autonomous operation of robots has gained increasing attention and has impacted several application areas. The human operator can receive various sensor feedback signals from the remote robot (termed the slave) contingent on the task at hand. For example, if the remote system is being used for manipulation tasks using single or multiple slave robots, then force or haptic feedback is useful [1]. On the other hand, if the objective is to control the center of mass of the remote robotic system via a tablet-like interface, then feedback may be based on the position or velocity mean of the robot ensemble [2,3].

Composability between heterogeneous human operators, multi-robot systems, and remote environments through unreliable communication links is an important desired characteristic of semi-autonomous robotic systems. Passivity emerged as such a property for bilateral teleoperation systems [1], where passivity of the teleoperator was guaranteed due to the passive interconnection of the master/slave robots, operator, environment, and the scattering/wave variable based communications. This paradigm has also been successfully utilized for bilateral control of networked robots where force feedback (using system states as proxies) was primarily transmitted back to the human operator through the master robot [4,5,6].

However, it is also important to study the human-robot-environment interconnection when system modalities apart from force feedback are utilized in coupling the operator to the remote environment. Recent work has been accomplished in this area [7,8]. Furthermore, it is important to couple the sensory feedback from an arbitrary ensemble of slave robots to the human operator without overwhelming the operator and while maintaining the desired stability and convergence goals [2,3]. In this article, based on the theoretical work in [2,3], and through remote experiments, a system and control architecture for achieving these goals is presented.

## Control Architecture for Human-Enabled Motion Synchronization

Imagine a scene in which a human operator maneuvers a robotic swarm toward a certain position or velocity based on visual feedback from the robots as in Figure 1.

FIGURE 1 Interaction between a robotic swarm and a human operator.

In general, the operator can hardly determine more than one signal in real time, and hence a single signal from the operator needs to be utilized to supervise the robotic swarm. In addition, the information feedback to the operator also needs to be simplified so as to prevent operator fatigue and reduce errors.

Distributed motion synchronization schemes developed during the last decade have the potential to provide a solution to the above issues since they allow one to virtually view the robotic swarm as a single robot. For example, a trivial solution is to take the architecture detailed in Figure 2 (left) [7,8], where only one robot interacts with the operator. However, this one-human-one-robot interaction is essentially fragile against communication failures. In this regard, the architecture in Figure 2 (right) is preferable, wherein the information from multiple robots is fused before being presented to the human operator.

FIGURE 2 One-human-one-robot interaction (left) and one-human-multi-robots interaction (right).

It is to be emphasized that the architecture in Figure 2 is similar to those for the leader following or reference tracking studies that have been accomplished in the literature. The distinguishing feature of the proposed work is the presence of feedback from the robots to the operator. This feedback allows for flexible human decision-making and ensures the viability of global situation awareness.

## Design of Semi-Autonomous Robotic Swarm

As a solution to stable coordination via bilateral human-swarm interactions, the authors’ group built the system in Figure 3 consisting of multiple planar robots, an operator and a computer whose role is to implement the distributed motion synchronization law. In the system, the operator can choose either the position or velocity variable to be synchronized between the robots. A tablet is chosen as an interface to interact with the robots.

FIGURE 3 Schematic of the present semi-autonomous multi-robot system.

The computer first extracts the positions and velocities of the robots from the image data acquired by an overhead camera via image processing, and then computes the input for motion synchronization. Among a variety of potential synchronization laws, the present system employs the so-called PI consensus estimator [9] whose benefit is that both position and velocity synchronization are achieved by a single law and the robots are not required to share the selected variable to be synchronized. Simultaneously, the computer sends to a tablet the positions and velocities of accessible robots through the Internet. Accessible robots are a set of swarming robots that can receive input from the operator. The router is connected not only to a tablet in the room but also to a tablet at a distance of about 290km to simulate remote control.

Once the tablet receives the information from the computer, it displays the average position or velocity on its monitor depending on the variable to be synchronized. The operator then determines a velocity command, by the position of a finger tapping the monitor, so that the displayed information leads to a desirable position or velocity. In this case, the operator feels as if he is manipulating a single virtual robot with position (velocity) equal to the average position (velocity) of the accessible robots, thereby mitigating his cognitive load. The tablet then sends the human command to the computer.

Subsequently, the computer adds the received operator command to the computed synchronization input for the accessible robots, and communicates the resulting signal to each robot. The robots then regard the received signal as a velocity reference, and move so as to follow the reference velocity.

A condensed description of the signal flows is illustrated in Figure 4. It is to be noted that the error between the reference and the feedback information is internally computed in the operator brain.

FIGURE 4 Block diagram of the present control architecture.

## Why this Architecture?

Consider a system from input uRP to output yRP. The system is then said to be passive if the integralDisplay Formula

$∫0τyTt ut dt$

is lower bounded for any time τ and input signal u. This concept generalizes physical systems such that the internal energy never increases more than the energy supplied to the system [1]. Passivity is known to be preserved with respect to feedback interconnections in Figure 5, and stability of the closed-loop system is also ensured under additional assumptions [1].

FIGURE 5 Feedback interconnection of two passive systems, where the resulting closed-loop system is also passive from input u to output y.

A typical approach to stabilization of an interconnected system that includes a highly uncertain component like a human operator is to assume passivity of the component block. Specifically, modeling of the operator has been addressed in this manner for several studies in bilateral teleoperation [1]. If such a human model would also be acceptable in the present system, interconnecting the operator and robotic swarm in the form of Figure 4 would guarantee closed-loop passivity as long as the robotic swarm is designed to be passive.

The above discussion inspires us to choose an appropriate yh to be fed back to the operator so that the robotic swarm is passive with the input-output pair as (yh, uh), where uh is the human operator input to the accessible robots. Fortunately, the property is guaranteed [2,3] if the output yh is chosen to be the average position/velocity of the accessible robots under the assumption that the inter-robot communication is bidirectional and robots can directly determine their own velocities. This is why the operator feeds back the average information in the present system. In other words, it is passivity theory that provides critical insights into interconnecting the human operator and robotic swarm. Indeed, synchronization of the intended variable to the reference has been demonstrated [2,3].

## Human Passivity Analysis

Despite the above desirable result, an issue remains open, that is, validity of the key assumption on passivity of the human operator. To analyze the property, a human-in-the-loop simulator is developed, where the robot motion is restricted to one dimension in order to simplify the analysis.

The picture displayed on the tablet is shown in Figure 6. The blue dot is the average position of accessible robots, and the red bar is the human velocity command uh to be sent to the accessible robots. The green dot is the externallyprovided reference position, denoted by r, which is used as the input for system identification.

FIGURE 6 Picture displayed on the tablet.

Since the human operator model needs to be identified through the process of the closed-loop system in Figure 4, a closed-loop system identification technique is employed. The detailed modeling procedure is described in [2,3] but, roughly speaking, two open-loop models from r to uh and to e=r-yh are identified and then a model of the decision process is computed from these models.

Experiments for the human modeling are conducted for the three different network structures in Figure 7. For these networks, the human operator's models are identified using trial data for a subject. Please refer to [2,3] for more details on the trials.

FIGURE 7 Network structures used for the human modeling, where the robots accessible from the human are colored red and the others blue. The left, middle, and right structures are called type 1, 2 and 3, respectively.

Figure 8 illustrates the Bode diagrams of the identified human operator models for each network, where the dotted and solid curves describe the models before and after 10-min training, respectively. Ignoring the effects of nonlinearity, the operator is known to be passive if the phase diagram lies from -90̊ to 90̊. In this regard, the models for type 3 provide an insight that the human approaches a passive system through training. However, for type 2, the subject does not attain passivity even after the training.

FIGURE 8 Bode diagrams of the identified human operator model without (Figure 8) and with the high-pass filter (Figure 9).

In previous work, the authors [2,3] experimentally demonstrated that a high-pass filter that modified the human command uh is useful for overcoming the shortage of passivity. In this case, the cascade system of the operator and the filter can be viewed as a human. Bode diagrams of the cascade system identified using the data of additional trials after inserting the filter for the same subject are shown in Figure 9, where the dotted and solid curves describe the human models without and with the filter, respectively. It is seen that the cascade system attains passivity for all of the three network types.

FIGURE 9 Bode diagrams of the identified human operator model without (Figure 8) and with the high-pass filter (Figure 9).

## Remote Control Experiment

Experiments are finally conducted using the system in Figure 3 with six mobile robots in Tokyo and a human operator with a tablet in Ishikawa. The inter-robot communication structure is illustrated in Figure 10, where robots 1, 3 and 4 are accessible from the operator. In order to avoid collisions, we aim at synchronizing, instead of the actual robot positions, virtual positions with biases, which are designed so that the robots eventually form a regular hexagon formation. In the experiments, the operator does not determine the reference position but it is determined by a computer to clearly observe the convergence. The reference is switched twice during the experiment.

FIGURE 10 Inter-robot communication network in the experiment.

Snapshots of the robots are presented in Figure 11. It is observed from the figures that the robots form and maintain the specified formation while they are maneuvered stably toward the desired references by the operator.

FIGURE 11 Snapshots of the experiments, where the red star describes the reference position shown to the operator. The figures in each row show the robots behavior for a common reference. The left figures are snapshots at the times of the reference switches, the right are those just before the switches and the middle show those at transient times.

## Conclusions

In this article, a novel semi-autonomous cooperative robotic system is presented. Information flows in the architecture are designed based on the passivity property. The human passivity needed to guarantee the control goal is demonstrated using a human-in-the-loop simulator and system identification techniques. Experimental results on remote control of the robots are finally demonstrated.

## Acknowledgements

This work was supported by JSPS KAKENHI Grant #15H04019.

## References

T. Hatanaka, N. Chopra, M. Fujita and M. W. Spong, “Passivity Based Control and Estimation in Networked Robotics”, Communications and Control Engineering Series, Springer Verlag, 2015.
T. Hatanaka, N. Chopra and M. Fujita, “Passivity-based Bilateral Human-Swarm-Interactions for Cooperative Robotic Networks and Human Passivity Analysis”, Proc. 54th IEEE Conference on Decision and Control, 2015, pp. 1033– 1039.
T. Hatanaka, N. Chopra, J. Yamauchi and M. Fujita, “A Passivity-Based Approach to Human-Swarm Collaborations and Passivity Analysis of Human Operators,” Trends in Control and Decision-Making for Human-Robot Collaboration Systems, Y. Wang and F. Zhang (eds.), Springer-Verlag, 2017, pp. 325– 355.
D. Lee, M.W. Spong, “Bilateral teleoperation of multiple cooperative robots over delayed communication network: theory”, Proc. IEEE International Conference on Robotics and Automation”, pp. 360– 365, 2005.
E.J. Rodriguez-Seda, J.J. Troy, C.A. Erignac, P. Murray, D.M. Stipanovic, M.W. Spong, “Bilateral teleoperation of multiple mobile agents: coordinated motion and collision avoidance”, IEEE Trans. Contr. Syst. Technol., 18 (4), 2010, pp. 984– 992.
A. Franchi, C. Secchi, M. Ryll, H.H. Bulthoff, P.R. Giordano, “Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs”, IEEE Robotics and Automation Magazine 19 (3), 2012, pp. 57– 68.
J.-P. de la Croix, M. Egerstedt, “Controllability characterizations of leader-based swarm interactions”, AAAI Symposium on Human Control of Bio-inspired Swarms, 2012.
M. Egerstedt, J.-P. de la Croix, H. Kawashima, P. Kingston, “Interacting with networks of mobile agents. Large-Scale Networks in Engineering and Life Sciences”, Springer-Verlag, 2014, pp. 199– 224.
R.A. Freeman, P. Yang and K.M. Lynch, “Stability and convergence, properties of dynamic average consensus estimators”, Proc. 45th IEEE Conf. Decision and Control, 2006, pp. 398– 403.
View article in PDF format.

## References

T. Hatanaka, N. Chopra, M. Fujita and M. W. Spong, “Passivity Based Control and Estimation in Networked Robotics”, Communications and Control Engineering Series, Springer Verlag, 2015.
T. Hatanaka, N. Chopra and M. Fujita, “Passivity-based Bilateral Human-Swarm-Interactions for Cooperative Robotic Networks and Human Passivity Analysis”, Proc. 54th IEEE Conference on Decision and Control, 2015, pp. 1033– 1039.
T. Hatanaka, N. Chopra, J. Yamauchi and M. Fujita, “A Passivity-Based Approach to Human-Swarm Collaborations and Passivity Analysis of Human Operators,” Trends in Control and Decision-Making for Human-Robot Collaboration Systems, Y. Wang and F. Zhang (eds.), Springer-Verlag, 2017, pp. 325– 355.
D. Lee, M.W. Spong, “Bilateral teleoperation of multiple cooperative robots over delayed communication network: theory”, Proc. IEEE International Conference on Robotics and Automation”, pp. 360– 365, 2005.
E.J. Rodriguez-Seda, J.J. Troy, C.A. Erignac, P. Murray, D.M. Stipanovic, M.W. Spong, “Bilateral teleoperation of multiple mobile agents: coordinated motion and collision avoidance”, IEEE Trans. Contr. Syst. Technol., 18 (4), 2010, pp. 984– 992.
A. Franchi, C. Secchi, M. Ryll, H.H. Bulthoff, P.R. Giordano, “Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs”, IEEE Robotics and Automation Magazine 19 (3), 2012, pp. 57– 68.
J.-P. de la Croix, M. Egerstedt, “Controllability characterizations of leader-based swarm interactions”, AAAI Symposium on Human Control of Bio-inspired Swarms, 2012.
M. Egerstedt, J.-P. de la Croix, H. Kawashima, P. Kingston, “Interacting with networks of mobile agents. Large-Scale Networks in Engineering and Life Sciences”, Springer-Verlag, 2014, pp. 199– 224.
R.A. Freeman, P. Yang and K.M. Lynch, “Stability and convergence, properties of dynamic average consensus estimators”, Proc. 45th IEEE Conf. Decision and Control, 2006, pp. 398– 403.

## Discussions

Some tools below are only available to our subscribers or users with an online account.

### Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related Proceedings Articles
Related eBook Content
Topic Collections