A review on multi-robot systems: current challenges for operators and new developments of interfaces

[ES] Los sistemas multi-robot están experimentando un gran desarrollo en los últimos tiempos, ya que mejoran el rendimiento de las misiones actuales y permiten realizar nuevos tipos de misiones. Este artículo analiza el estado del arte de los sistemas multi-robot, abordando un conjunto de temas relevantes: misiones, flotas, operadores, interacción humano-sistema e interfaces. La revisión se centra en los retos relacionados con factores humanos como la carga de trabajo o la conciencia de la situación, así como en las propuestas de interfaces adaptativas e inmersivas para solucionarlos.[EN] Multi-robot systems are experiencing great development in recent times, since they are improving the performance of current missions and allowing new types of missions. This article analyzes the state of the art of multi-robot systems, addressing a set of relevant topics: missions, fleets, operators, human-system interaction and interfaces. The review focuses on the challenges related to human factors such as workload and situational awareness, as well as the proposals of adaptive and immersive interfaces to solve them.Esta investigación ha recibido fondos de los proyectos SAVIER (Situational Awareness VIrtual EnviRonment) de Airbus; RoboCity2030-DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/ NMT-4331, financiado por los Programas de Actividades I+D de la Comunidad de Madrid y confinanciado por los Fondos Estructurales de la UE; y DPI2014-56985-R (Protección Robotizada de Infraestructuras Críticas) financiado por el ministerio de Economía y Competitividad del Gobierno de España.Roldan-Gómez, JJ.; De León Rivas, J.; Garcia-Aunon, P.; Barrientos, A. (2020). Una revisión de los sistemas multi-robot: desafíos actuales para los operadores y nuevos desarrollos de interfaces. Revista Iberoamericana de Automática e Informática industrial. 17(3). https://doi.org/10.4995/riai.2020.13100305173Abbadi, A. and Prenosil., 2015. Safe path planning using cell decomposition approximation. Distance Learning, Simulation and Communication, 8.Adams, B. and Suykens, F., 2013 Astute: Increased Situational Awareness through proactive decision support and adaptive map-centric user interfaces. 2013 IEEE European Intelligence and Security Informatics Conference (EISIC), 289-293. https://doi.org/10.1109/EISIC.2013.74Almeida, L., Menezes, P. and Dias, J., 2017. Improving robot teleoperation experience via immersive interfaces. In 2017 4th IEEE Experiment@ International Conference (exp. at'17), 87-92. https://doi.org/10.1109/EXPAT.2017.7984414Arnold, K. P., 2016. The UAV Ground Control Station: Types, Components, Safety, Redundancy, and Future Applications. International Journal of Unmanned Systems Engineering., 4(1), 37.Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B. and Onaral, B., 2012. Optical brain monitoring for operator training and mental workload assessment. Neuroimage, 59(1), 36-47. https://doi.org/10.1016/j.neuroimage.2011.06.023Beer, J., Fisk, A. D. and Rogers, W. A., 2014. Toward a framework for levels of robot autonomy in human-robot interactions. Journal of Human-Robot Interaction, 3(2), 74. https://doi.org/10.5898/JHRI.3.2.BeerBourguet, M. L., 2003. Designing and Prototyping Multimodal Commands. Interact, 3, 717-720.Brutschy, A., Pini, G., Pinciroli, C., Birattari, M. and Dorigo, M., 2014. Self-organized task allocation to sequentially interdependent tasks in swarm robotics. Autonomous agents and multi-agent systems, 28(1), 101-125. https://doi.org/10.1007/s10458-012-9212-yCantelli, L., Mangiameli, M., Melita, C. D. and Muscato, G., 2013. UAV/UGV cooperation for surveying operations in humanitarian demining. 2013 IEEE international symposium on safety, security, and rescue robotics (SSRR),1-6). https://doi.org/10.1109/SSRR.2013.6719363Chang, H. and Jin, T., 2013. Command fusion based fuzzy controller design for moving obstacle avoidance of mobile robot. Future information communication technology and applications, Springer, 905-913. https://doi.org/10.1007/978-94-007-6516-0_99Chen, J. Y. C., Haas, E. C. and Barnes, M. J., 2007. Human performance issues and user interface design for teleoperated robots. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(6), 1231-1245. https://doi.org/10.1109/TSMCC.2007.905819Chen, J. Y. C., Barnes, M. J. and Harper-Sciarini, M., 2011. Supervisory control of multiple robots: Human-performance issues and user-interface design. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications an Reviews), 41(4), 435-454. https://doi.org/10.1109/TSMCC.2010.2056682Clark, C. M., 2005. Probabilistic road map sampling strategies for multi-robot motion planning. Robotics and Autonomous Systems, 53(3), 244-264. https://doi.org/10.1016/j.robot.2005.09.002Cummings, M. L., Bruni, S., Mercier, S. and Mitchell, P. J., 2007. Automation architecture for single operator, multiple UAV command and control. Tech. Rep DTIC Document.Cummings, M. L. and Mitchell P. J., 2008. Predicting controller capacity in supervisory control of multiple UAVs. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 38(2), 451-460. https://doi.org/10.1109/TSMCA.2007.914757Cummings, M. L., Mastracchio, C., Thornburg, K. M. and Mkrtchyan, A., 2013. Boredom and distraction in multiple unmanned vehicle supervisory control. Interacting with computers, 25(1), 34-47. https://doi.org/10.1093/iwc/iws011De Cubber, G., Doroftei, D., Serrano, D., Chintamani, K., Sabino, R. and Ourevitch, S., 2013. The EU-ICARUS project: developing assistive robotic tools for search and rescue operations. 2013 IEEE international symposium on safety, security, and rescue robotics (SSRR), 1-4. https://doi.org/10.1109/SSRR.2013.6719323De Greeff, J., Hindriks, K., Neerincx, M. A. and Kruijff-Korbayova, I., 2015. Human-robot teamwork in USAR environments: the TRADR project. Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, pp. 151-152. https://doi.org/10.1145/2701973.2702031Dezfoulian, S. H., Wu, D. and Ahmad, I. S., 2013. A generalized neural network approach to mobile robot navigation and obstacle avoidance. Intelligent Autonomous Systems, Springer, 12, 25-52. https://doi.org/10.1007/978-3-642-33926-4_3Di, B., Zhou, R. and Duan, H., 2015. Potential field based receding horizon motion planning for centrality-aware multiple UAV cooperative surveillance. Aerospace Science and Technology, 46, 386-397. https://doi.org/10.1016/j.ast.2015.08.006Dixon, S. R., Wickens, C. D. and Chang, D, 2005. Mission control of multiple unmanned aerial vehicles: A workload analysis. Human Factors: The Journal of the Human Factors and Ergonomics Society 47(3), 479-487. https://doi.org/10.1518/001872005774860005Donmez, B., Nehme, C. and Cummings, M.L., 2010. Modeling workload impact in multiple unmanned vehicle supervisory control. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 40(6), 1180-1190. https://doi.org/10.1109/TSMCA.2010.2046731Drury, J. L., Scholtz, J. and Yanco, H.A., 2003. Awareness in human-robot interactions. IEEE International Conference on Systems, Man and Cybernetics, 1, 912-918.Drury, J. L., Riek, L. and Rackliffe, N., 2006A. A decomposition of UAV-related situation awareness. Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, 88-94. https://doi.org/10.1145/1121241.1121258Drury, J. L., Richer, J., Rackliffe, N. and Goodrich, M. A., 2006B. Comparing situation awareness for two unmanned aerial vehicle human interfaces approaches. Technical rpeort, Mitre Corp Bedford MA.Endsley, M. R., 1988A. Design and evaluation for situation awareness enhancement. Proceedings of the human factors and ergonomics society annual meeting, SAGE Publications, 32(2), 97-101. https://doi.org/10.1177/154193128803200221Endsley, M. R., 1988B. Situation awareness global assessment technique (SAGAT). Proceedings of the IEEE 1988 National Aerospace and Electronics Conference (NAECON), 789-795.Endsley, M. R. and Garland, D. J., 2000. Situation awareness analysis and measurement. CRC Press. https://doi.org/10.1201/b12461Endsley M. R., 1999. Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42(3), 462-492. https://doi.org/10.1080/001401399185595Flushing, E. F., Gambardella, L. and Di Caro, G. A., 2012. Gis-based mission support system for wilderness search and rescue with heterogeneous agents. Proc 2nd Workshop on robots and sensors integration in future rescue INformation system (ROSIN), IEEE/RSJ Int Conference on Intelligent Robots and Systems (IROS).Foit, K., 2014. Mixed reality as a tool supporting programming of the robot. Advanced Materials Research, Trans Tech Publications, 1036, 737-742. https://doi.org/10.4028/www.scientific.net/AMR.1036.737Frische, F. and Lüdtke, A., 2013. SA-tracer: A tool for assessment of UAV swarm operator SA during mission execution. 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA), 203-211. https://doi.org/10.1109/CogSIMA.2013.6523849Fuchs, C., Borst, C., de Croon, G. C., Van Paassen, M. M. and Mulder, M., 2014. An ecological approach to the supervisory control of UAV swarms. International Journal of Micro Air Vehicles, 6(4), 211-229. https://doi.org/10.1260/1756-8293.6.4.211Galceran, E. and Carreras, M., 2013. A survey on coverage path planning for robotics. Robotics and Autonomous Systems, 61(12), 1258-1276. https://doi.org/10.1016/j.robot.2013.09.004García, J. C., Patrão, B., Pérez, J., Seabra, J., Menezes, P., Dias, J. and Sanz, P.J., 2015. Towards an immersive and natural gesture controlled interface for intervention underwater robots. IEEE OCEANS 2015-Genova, 1-5. https://doi.org/10.1109/OCEANS-Genova.2015.7271501García, S. E., Slawiñski, E., Mut, V., and Penizzotto, F., 2018. Collision avoidance method for multi-operator multi-robot teleoperation system. Robotica, 36(1), 78-95. https://doi.org/10.1017/S0263574717000169Garzón, M., Valente, J., Zapata, D. and Barrientos, A., 2013. An aerial-ground robotic system for navigation and obstacle mapping in large outdoor areas. Sensors, 13(1), 1247-1267. https://doi.org/10.3390/s130101247Garzón, M., Valente, J., Roldán, J. J., Cancar, L., Barrientos, A. and Del Cerro, J., 2016. A multirobot system for distributed area coverage and signal searching in large outdoor scenarios. Journal of Field Robotics, 33(8), 1087-1106. https://doi.org/10.1002/rob.21636Garzón, M., Valente, J., Roldán, J. J., Garzón-Ramos, D., de León, J., Barrientos, A. and del Cerro, J., 2017. Using ros in multi-robot systems: Experiences and lessons learned from real-world field tests. In Robot Operating System (ROS), Springer, Cham, 449-483. https://doi.org/10.1007/978-3-319-54927-9_14Goerzen, C., Kong, Z. and Mettler, B., 2009. A survey of motion planning algorithms from the perspective of autonomous UAV guidance. Selected papers from the 2nd International Symposium on UAVs, Reno, Nevada, USA, Springer, 64-100. https://doi.org/10.1007/978-90-481-8764-5_5Goodrich, M. A. and Schultz, A. C., 2007. Human-robot interaction: a survey. Foundations and trends in human-computer interaction, 1(3), 203-275. https://doi.org/10.1561/1100000005Gregory, J., Fink, J., Stump, E., Twigg, J., Rogers, J., Baran, D., Fung, N. and Young, S., 2016. Application of multi-robot systems to disaster-relief scenarios with limited communication. In Field and Service Robotics, Springer, Cham, 639-653. https://doi.org/10.1007/978-3-319-27702-8_42Haas, E. C., Pillalamarri, K., Stachowiak, C. C. and Fields, M., 2011. Multimodal controls for soldier/swarm interaction. IEEE RO-MAN, 223-228. https://doi.org/10.1109/ROMAN.2011.6005227Hagiwara, Y., 2015. Cloud based VR system with immersive interfaces to collect multimodal data in human-robot interaction. 2015 IEEE 4th Global Conference on Consumer Electronics (GCCE), 256-259. https://doi.org/10.1109/GCCE.2015.7398709Hart, S. G. and Staveland E.L., 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology, 52, 139-183. https://doi.org/10.1016/S0166-4115(08)62386-9Hart, S. G., 2006. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the human factors and ergonomics society annual meeting, Sage Publications Sage CA, Los Angeles, CA, 50(9), 904-908. https://doi.org/10.1177/154193120605000909Hobbs, A. and Herwitz, S. R., 2014. Human factors in the maintenance of unmanned aerial vehicle (UAV) swarm management. Applied Ergonomics, 58, 66-80. https://doi.org/10.1016/j.apergo.2016.05.011Hocraffer, A. and Nam, C. S. A meta-analysis of human-system interfaces in unmanned aerial vehicle (UAV) swarm management. Applied Ergonomics, 58, 66-80. https://doi.org/10.1016/j.apergo.2016.05.011Hong, A., 2016. Human-Robot Interactions for Single Robots and Multi-Robot Teams. PhD thesis, Department of Mechanical and Industrial Engineering, University of Toronto.Janchiv, A., Batsaikhan, D., hwan Kim, G. and Lee, S. G., 2011. Complete coverage path planning for multi-robots based on. 2011 11th IEEE International Conference on Control, Automation and Systems, 824-827.Jasper, P., Sibley, C. and Coyne, J. Using Heart Rate Variability to Assess Operator Mental Workload in a Command and Control Simulation of Multiple Unmanned Aerial Vehicles. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Sage CA, Los Angeles, CA, 60(1), 1125-1129. https://doi.org/10.1177/1541931213601264Jia, X. and Meng, M. Q. H., 2013. A survey and analysis of task allocation algorithms in multi-robot systems. 2013 IEEE International Conference on Robotics and biomimetics (ROBIO), 2280-2285. https://doi.org/10.1109/ROBIO.2013.6739809Jiang, Y., 2016. A survey of task allocation and load balancing in distributed systems. IEEE Transactions on Parallel and Distributed Systems, 27(2), 585-599. https://doi.org/10.1109/TPDS.2015.2407900Johannsmeier, L. and Haddadin, S., 2016. A hierarchical human-robot interaction-planning framework for task allocation in collaborative industrial assembly processes. IEEE Robotics and Automation Letters, 2(1), 41-48. https://doi.org/10.1109/LRA.2016.2535907Kapoutsis, A. C., Chatzichristofis, S. A., Doitsidis, L., de Sousa, J. B., Pinto, J., Braga, J. and Kosmatopoulos, E. B., 2016. Real-time adaptive multi-robot exploration with application to underwater map construction. Autonomous robots, 40(6), 987-1015. https://doi.org/10.1007/s10514-015-9510-8Kavitha, S., Veena, S. and Kumaraswamy, R., 2015. Development of automatic speech recognition system for voice activated Ground Control system. 2015 IEEE International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15), pp. 1-5. https://doi.org/10.1109/ITACT.2015.7492684Khaleghi, A. M., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., Son, Y. J., Vo, C., Mousavian, A. and Lien, J. M., 2014. A comparative study of control architectures in UAV/UGV-based surveillance system. Proceedings of the IIE Annual Conference, Institute of Industrial and Systems Engineers (IISE), 3455.Khamis, A., Hussein, A. and Elmogy, A., 2015. Multi-robot Task Allocation: A Review of the State-of-the-Art. Cooperative Robots and Sensor Networks, Springer, 31-51. https://doi.org/10.1007/978-3-319-18299-5_2Kirchner, E. A., Kim, S. K., Tabie, M., Wöhrle, H., Maurus, M. and Kirchner, F., 2016. An intelligent man-machine interface - Multi-robot control adapted for task engagement based on single-trial detectability of P300. Frontiers in human neuroscience, 10, 291. https://doi.org/10.3389/fnhum.2016.00291Kolling, A., Nunnally, S. and Lewis, M., 2012. Towards human control of robot swarms. Proceedings of the seventh annual ACM/IEEE international conference on human-robot interaction, 89-96. https://doi.org/10.1145/2157689.2157704Kothari, M., Postlethwaite, I. and Gu, D. W. Multi-UAV path planning in obstacle rich environment using rapidly-exploring random trees. Proceedings of the 48 IEEE Conference on Decision and Control, 3069-3074.Kruijff-Korbayová, I., Colas, F., Gianni, M., Pirri, F., de Greeff, J., Hindriks, K., Neerincx, M., Ögren, P., Svoboda, T. and Worst, R., 2015. Tradr project: Long-term human-robot teaming for robot assisted disaster response. KI-Künstliche Intelligenz, 29(2), 193-201. https://doi.org/10.1007/s13218-015-0352-5Kurniawan, H., Maslov, A. V. and Pechenizkiy, M., 2013. Stress detection from speech and galvanic skin response signals. IEEE 26th International Symposium on Computer-Based Medical Systems (CBMS), 209-214. https://doi.org/10.1109/CBMS.2013.6627790Lang, R. G., da Silva, I. N., Romero, R. A. F., 2014. Development of distributed control architecture for multi-robot systems. IEEE 2014 Joint Conference on Robotics: SBR-LARS Robotics Symposium and Robocontrol, 163-168. https://doi.org/10.1109/SBR.LARS.Robocontrol.2014.49Larochelle, B., Kruijff, G. J. M., Smets, N., Mioch, T. and Groenewegen, P., 2011. Establishing human situation awareness using a multi-modal operator control unit in an urban search & rescue human-robot team, IEEE, 229-234. https://doi.org/10.1109/ROMAN.2011.6005237Lesire, C., Infantes, G., Gateau, T. and Barbier, M., 2016. A distributed architecture for supervision of autonomous multi-robot missions. Autonomous Robots, 40(7), 1343-1362. https://doi.org/10.1007/s10514-016-9603-zLindemuth, M., Murphy, R., Steimle, E., Armitage, W., Dreger, K., Elliot, T., Hall, M., Kalyadin, D., Kramer, J., Palankar, M. and Pratt, K., 2011. Sea robot-assisted inspection. IEEE robotics & automation magazine, 18(2), 96-107. https://doi.org/10.1109/MRA.2011.940994Lysaght, R. J., 1989. Operator workload: Comprehensive review and evaluation of operator workload methodologies. Tech. Rep. DTIC Document. https://doi.org/10.21236/ADA212879Mantecón, T., del Blanco, C. R., Jaureguizar, F. and García, N., 2014. New generation of human machine interfaces for controlling UAV through depth-based gesture recognition. Unmanned Systems Technology XVI, International Society for Optics and Photonics, 9084. https://doi.org/10.1117/12.2053244Marino, A., Parker, L. E., Antonelli, G. and Caccavale, F., 2013. A decentralized architecture for multi-robot systems based on the null-space-behavioral control with application to multi-robot border patrolling. Journal of Intelligent & Robotic Systems, 71(3-4), 423-444. https://doi.org/10.1007/s10846-012-9783-5Martín-Barrio, A., Terrile, S., Barrientos, A. and del Cerro, J., 2018. Hyper-Redundant Robots: Classification, State-of-the-Art and Issues. Revista Iberoamericana de Automática e Informática Industrial, 15(4), 351-362. https://doi.org/10.4995/riai.2018.9207Martín-Barrio, A., Roldán, J. J., Terrile, S., del Cerro, J. and Barrientos, A., 2019. Application of Immersive Technologies and Natural Language to Hyper-Redundant Robot Teleoperation. Virtual Reality, 1-15. https://doi.org/10.1007/s10055-019-00414-9Martins, H., Oakley, I. and Ventura, R., 2015. Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots. Robotica, 33(10), 2166-2185. https://doi.org/10.1017/S026357471400126XMatellán, V. and Borrajo, D., 2001. ABC2 an agenda based multi-agent model for robots control and cooperation. Journal of Intelligent & Robotic Systems, 32(1), 93-114. https://doi.org/10.1023/A:1012009429991McCarley, J.S. and Wickens, C. D., 2005. Human factors implications of UAVs in the national airspace. University of Illinois at Urbana-Champaign, Aviation Human Factors Division.McDuff, D., Gontarek, S. and Picard, R., 2014. Remote measurement of cognitive stress via heart rate variability. 36th Annual International Conference on the IEEE Engineering in Medicine and Biology Society (EMBC), 2957-2960. https://doi.org/10.1109/EMBC.2014.6944243Menda, J., Hing, J. T., Ayaz, H., Shewokis, P. A., Izzetoglu, K., Onaral, B. and Oh, P., 2011. Optical brain imaging to enhance UAV operator training, evaluation, and interface development. Journal of intelligent & robotic systems, 61(1-4), 423-443. https://doi.org/10.1007/s10846-010-9507-7Monajjemi, V. M., Pourmehr, S., Sadat, S. A., Zhan, F., Wawerla, J., Mori, G. and Vaughan, R., 2014. Integrating multi-modal interfaces to command UAVs. Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, 106-106. https://doi.org/10.1145/2559636.2559646Moore, J., Wolfe, K. C., Johannes, M. S., Katyal, K. D., Para, M. P., Murphy, R. J., Hatch, J., Taylor, C. J., Bamberger, R. J. and Tunstel, E., 2016. Nested marsupial robotic system for search and sampling in increasingly constrained environments. 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2279-2286 https://doi.org/10.1109/SMC.2016.7844578Mosteo, A. R. and Montano, L., 2010. A survey of multi-robot task allocation. Technical Report No. AMI-009-10-TEC, Instituto de Investigación en Ingeniería de Aragón, University of Zaragoza, Zaragoza, Spain.Murphy, R. R., 2004. Human-robot interaction in rescue robotics. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 34(2), 138-153. https://doi.org/10.1109/TSMCC.2004.826267Nagi, J., Giusti, A., Di Caro, G. A. and Gambardella, L. M., 2014. Human control of UAVs using face pose estimates and hand gestures. 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 1-2. https://doi.org/10.1145/2559636.2559644Nam, C. S., Johnson, S., Li, Y. and Seong, Y., 2009. Evaluation of human-agent user interfaces in multi-agent systems. International Journal of Industrial Ergonomics, 39(1), 192-201. https://doi.org/10.1016/j.ergon.2008.08.008Nestmeyer, T., Giordano, P. R., Bülthoff, H. H. and Franchi, A., 2017.