Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Development of ROS2-TMS: new software platform for informationally structured environment

Development of ROS2-TMS: new software platform for informationally structured environment This study proposes a new software platform, called ROS2-TMS, for an informationally structured environment. An informationally structured environment is vital for developing intelligent service robots by embedding various sensors in the environment to enhance the sensing capability and intelligence of robots. Thus far, we have been developing a software platform, named ROS-TMS, for an informationally structured environment, which connects various sensors and robots using ROS architecture. In recent years, ROS2, a next-generation version of ROS, has been released. ROS2 has many advantages, such as enhanced security, QoS control, and support for various platforms. ROS2-TMS, a new version of ROS-TMS, is developed not only by porting existing modules in ROS-TMS, such as the control system for a communication robot, but also by adding useful functions utilizing new features in ROS2. For instance, we added a voice user interface to control robots and various devices in the environment, such as lights or a bed. In addition, we implemented a new task scheduler that provides a cancelation function to stop running tasks and improve the security of the platform. Keywords: Service robot, Informationally structured environment, Internet of things, Cyber physical system, Ambient sensing Introduction the surrounding environment to collect, analyze, and In recent decades, with the decline in birthrate and aging retain environmental information. As a software platform of population, labor shortage has become a crucial issue for an ISE, we have been developing ROS-TMS [1], which in various fields such as medical and nursing care. As a realizes service robots that coexist with humans. ROS- solution to this problem, the realization of life support TMS connects various sensors embedded in the environ- services using artificial intelligence and service robots has ment and service robots using a robot operating system attracted considerable attention. However, unlike robots (ROS) [2], which is a general-purpose robot middleware. operated in factories, the environment in which service In recent years, ROS2  [3], a next-generation version of robots work for daily life support is diverse and dynami- ROS, has been released. ROS2 has many advantages, cally changing. Therefore, it is difficult to fully under - such as enhanced security, QoS control, support for vari- stand the surrounding situation based on the embedded ous platforms, and advanced navigation algorithms [4]. sensors of the robot alone. One of the key solutions to In this study, we propose a novel software platform for this problem is an informationally structured environ- an ISE, named ROS2-TMS. The characteristics of ROS2- ment (ISE), in which sensors are distributed throughout TMS are as follows. (1) Upgradation of the middleware from ROS to ROS2: Enhanced security, QoS control, and latest navigation algorithms are available. (2) Uni- *Correspondence: itsuka@irvs.ait.kyushu-u.ac.jp fied task management mechanism: Regardless of the Kyushu University, 744 Motooka, Nishi-ku, Fukuoka-shi, Fukuoka 819-0395, type of device (for example, robots, room lighting, and Japan © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 2 of 19 intelligent home appliances), service tasks can be man- Various research topics and references on IoRT are intro- aged uniformly as subtasks in the task manager. This duced in [13]. makes it easier to perform advanced coordination of vari- The robot operating system (ROS)  [2], an open source ous devices. (3) Cancelation function: By managing sub- platform, has contributed significantly to the develop - tasks in all devices using the ROS2 Action protocol, all ment and research of mobile and service robots. The service tasks can be terminated during execution regard- new generation of ROS, ROS2 [3], has enhanced security less of the complexity of the task, thus, increasing safety. and QoS control and supports the latest development environments, such as C++14 and Python3. The latest robot autonomous mobility package Navigation2 is also Related works available in ROS2. Navigation2 [4] consists of a behavior ISE enables the flexible operation of service robots by tree, which represents a management mechanism that is collecting, analyzing, and managing data for a complex higher than the autonomous movement of the robot, and environment by deploying distributed sensors not only it is easy to describe complex control such as temporary on the robot but also in the environment. Several systems avoidance behavior for environments wherein autono- have been developed on ISE thus far. In the “Robotic mous movement is difficult. Consequently, it has the Room”  [5] at The University of Tokyo, an approach to potential to cope with complex everyday environments. deploy various sensors in the environment was pro- One of the important updates in the internal design is the posed to observe the status of patients. In the “Intelligent action protocol  [14], which is useful for time-consum- Room”  [6] at MIT, AILab deployed multiple cameras in ing tasks, such as the moving task of a robot or a pick- a room to track the location of humans and determine ing up task of a manipulator. With the evolution of the their direction to provide services. “Intelligent Sweet design, an identifier is issued for each action protocol to Home”  [7] proposed a robotic platform that includes an distinguish each action. This action protocol is incorpo - intelligent bed with pressure sensors and an autonomous rated into the task scheduler proposed in this research mobile wheelchair in the context of ambient assisted liv- and plays a major role in connecting individual tasks and ing (AAL), which aims to promote independent living subtasks. for the elderly and disabled. These studies on ISEs have focused on two aspects: (1) deploying sensors in the From ROST ‑ MS to ROS2‑TMS environment to collect advanced information regarding The authors began to develop an ISE in the “Robot Town humans, robots, and objects, and (2) providing services Project” in 2005 and have been developing a software to assist humans through robots and intelligent home platform named Town Management System (TMS). appliances. In [1], we proposed a software platform, ROS-TMS 5.0 In recent years, various systems related to ISE have (Fig. 1), and a hardware platform, Big Sensor Box (Fig. 2), been proposed and commercialized, such as AAL, smart for an ISE. ROS-TMS 5.0 is developed as a core software houses, and the Internet of Robotic Things (IoRT). In platform for IoRT and has some novel functions, such as the field of smart homes, smart speakers, such as Ama - a care receiver-watching service and a voice control for zon’s “Alexa” [8] and Google’s “Google Assistant” [9], are service robots. widely used in households, and the technology to control However, there were four concerning points about home appliances, lighting, and other Internet of Things ROS-TMS: (IoT) devices through voice interfaces is widely adopted in households. In [10], it was stated that the presence 1. Using ROS as middleware: ROS initially supported of a central interface, such as a smart speaker, increases Python2; thus, the ROS-TMS executable used the quality of the robot service. IoRT is a field that incor - Python2. However, support for Python2 was dis- porates IoT technology, in which various devices in the home environment are connected to the Internet and robotics technology. As a method of component coor- dination using IoT technology, there is coordination between sensors to collect information using multiple sensors, as well as coordination on actuators to perform various services; for example, [11] identifies user activi - ties and habits by collecting information from a group of sensors installed in a smart house, such as a pressure sensor installed in a bed. In [12], a service robot was con- nected to an elevator control unit to move the service Fig. 1 ROS-TMS robot across multiple floors of an apartment building. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 3 of 19 which can be used in the C++14 / Python3 environ- ment. Also, advanced ROS2 technologies such as QoS control and Navigation2 are applicable in ROS2- TMS. 2. Redesign of task execution: In ROS-TMS, the micro- phone module was responsible not only for under- standing the user’s speech but also for task search and task execution. In addition, ROS2 does not have SMACH, the task execution machine used in ROS- TMS; therefore, the task execution had to be rebuilt. Therefore, we separated task search and task execu - tion from the roles of the microphone module and implemented task search in the task search node and Fig. 2 Big sensor box task execution and management in the task scheduler module. Furthermore, task execution was performed using ROS2 actions, and we were able to add func- continued in 2020. We believe that the platform tions for failure behavior and stopping during execu- should support Python3 and the latest development tion. resources. 3. Redesign of tasks and subtasks: In ROS-TMS, there 2. Difficulty in task implementation: The ROS-TMS were only three types of subtasks: moving the robot microphone module, which is responsible for voice and grasping and releasing by the robot hand. In input, transcribes the user’s speech, searches for ser- ROS2-TMS, the scope of subtasks was expanded to vice tasks, and executes tasks specific to those ser - include robots as well as robotic beds, room light- vices (sending commands to devices such as robots, ing, and speakers. As a result, all service tasks can lightings, and robotic beds using ROS topics, ser- be managed using a task scheduler. In addition, tasks vices, socket communication, etc.). Therefore, the linked to multiple devices can be added simply by dependency between task implementation and the entering the configuration information of these sub - microphone module is high, and it is necessary to tasks in the database. develop the microphone control node again when 4. Addition of cancel function: In ROS-TMS, a user tasks are added. cannot stop a task in the middle of the execution. 3. Limited subtasks: In ROS-TMS, the objects to be ROS2-TMS has a newly developed task scheduler managed by the task scheduler were limited to with ROS2 action, which can stop a task in the mid- robots only, and there were only three types of sub- dle of an execution according to the user’s request. tasks: robot movement, robot arm grasp, and release. Each subtask defines its behavior when canceled Together with the problems discussed in 2) above, such that adding a task does not need to define a new it was difficult to implement services that scheduled behavior. robot movements and other devices. 4. Cannot stop a task while it is running: For example, The rest of the paper is organized as follows: The ROS2- when a robot is moving and bumps into something TMS modules section describes the structure of the inadvertently, it is necessary to perform an emer- modules in ROS2-TMS. The Robots, Sensors, and User gency stop of the service task. However, ROS-TMS Request Devices section describes the modules related does not allow the user to perform an emergency to the devices in ROS2-TMS, such as robots and robotic stop while the task is in progress. beds. The Task execution flow section describes an overview of task execution. The Voice interfaces (TMS_ Therefore, we developed a new software platform for an UR) section describes the design of the voice user inter- ISE, named ROS2-TMS. This platform has the following face. The Database (TMS_DB) section describes the four features to solve these four problems: design of the database. The Task scheduler (TMS_TS) section describes the design of the task scheduler. The 1. Using ROS2 as middleware: While ROS uses C++03 Robot service experiment section describes the service / Python2 as its development environment, ROS2 tasks that can be provided by ROS2-TMS and provides can now use C++14 / Python3. Therefore, ROS2- examples of their execution. Finally, the section  Con- TMS ports the core modules of ROS-TMS to ROS2, clusions provides the conclusion. TMS_DB: Database Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 4 of 19 ROS2‑TMS modules Sensor system module (TMS_SS) Sensor  data are ROS2-TMS has a wide variety of functions as an ROS2 interpreted and converted into higher-level environ- node. ROS2-TMS is composed of modules connected mental information and stored in TMS_DB. in a hierarchical manner as shown in Fig.  3. In a ROS2- State analyzer module (TMS_SA) This module TMS service, various modules play roles by intercon- receives information from TMS_SS and estimates necting with each other, such as interpreting the user’s the state of the environment. For example, we plan speech, selecting a task to be executed, planning, and to estimate the user’s health status, but we are still in even the robot’s behavior. The contents of each module the concept stage. Details are given in the “Conclu- are described below. sions” section. Architecture Robots, sensors, and user request devices Database module (TMS_DB) Stores environmental In this study, we upgraded the middleware for each information managed by ROS2-TMS in a database. device from ROS to ROS2, improved the communica- The database is developed with mongoDB. tion quality of the Data Distributed Service (DDS), and User request module (TMS_UR) Receives task expanded the functions of ROS2, such as the cancelation requests from users and sends task execution function. Some sensors and robots used in Big Sensor requests to TMS_TS. Box and ROS2-TMS are shown in Fig. 4. Task scheduler module (TMS_TS) The requested ser - vice is executed by combining subtasks. In the pre- Robot controller module (TMS_RC) vious study (ROS_TMS), only robots were subject to subtask management, whereas in this study, we Robotic bed An electric bed (Rakusho Z KQ-7302, expanded the scope of task management to intelli- Paramount Bed) is controlled by RaspberryPi Zero, gent home appliances such as beds and speakers. which can raise the upper body and the height of the Robot planning module (TMS_RP) From some sub- bed. tasks of the robot commanded by TMS_TS, motion Communication robot (Double 2, Double 3) Double planning was performed to correctly execute the 2  [15] and Double 3  [16] are communication robots subtasks. manufactured by Double Robotics. Double 2 and Robot controller module (TMS_RC) This is a module Double 3 are controlled by attached iPad and Linux that executes the planned subtasks using the robot. PC, respectively. We implemented a moving task for A dedicated module is implemented for each robot. the robots to a specified position and angle using a Sensor driver module (TMS_SD) The system acti - motion capture system and the ROS2 Navigation2 vates various sensors embedded in the environment package. and publishes the acquired sensor data. Sensor system module (TMS_SS) User Wearable Heart rate sensor (WHS-1) Whs-1 is a wearable heart rate sensor developed by Union Tool Corporation that updates the heart rate information TMS_UR: User Request in the database at each timestep at which the user’s heart rate is measured. TMS_TS: Task Scheduler Motion capture system (VICON) VICON is a motion capture system that uses multiple motion-tracking TMS_SA TMS_SS TMS_RP cameras to recognize the position of an object. We State Analyzer Sensor System Robot Planning used it to estimate the position of the robot. TMS_SD: TMS_RC: Sensor Driver Robot Controller User request module (TMS_UR) Sensor Robot Microphone device A USB microphone is attached to Fig. 3 ROS2-TMS architecture Intel’s NUC to receive voice requests from users. User I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 5 of 19 Sensor System Robot Controller User Request Robot Intelligent appliances Collect sensor infomation Interface with users 0DUNHU Motion capture Robotic bed Mic Communication Speaker Heart rate sensor Robot Fig. 4 Sensors and robots in TMS_SS, TMS_RC, TMS_UR modules Speaker device A USB speaker is attached to the TMS_TS, TMS_DB, and the task search node in TMS_ Raspberry Pi 3 to play a conversation with the user UR were executed on a dedicated ROS2-TMS server in using the Google Assistant API, as well as announce- our experiments. ments and sound effects when the task starts using An overview of task execution from a user’s voice com- TMS_TS. mand to the robot and lighting control is presented in Fig.  5. The following bullet points correspond to the fol - lowing figure: Task execution flow To run the ROS2-TMS service, the modules shown in the 1. The user requests a command from the microphone. previous sections need to control corresponding devices 2. The microphone control node transcribes the voice and tasks coordinately. In TMS_RC, the robots and the and sends a string to the task search node. robotic bed are controlled. In TMS_RP, the robots are 3. The task search node searches for related task IDs connected to TMS_TS through Navigation2, and the and objects in the database. robotic bed is connected directly to TMS_TS. The micro - 4. The task search node passes the task IDs and objects phones and the speakers are managed in TMS_UR. to TMS_TS. TMS_DB: Database TMS_UR TMS_TS 3. 5. 9. TMS_RP Subtask (Navigation2) Task control node Subtask node TMS_RC 1. 2. 4. 6. Find a task Mic Manage tasks Thread Microphone Task search node Task management Robot control node node Thread Subtask Bed Task control node Subtask Lighting Fig. 5 Flowchart of processes from user request to task execution in ROS2-TMS Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 6 of 19 5. The task management node integrates task and service that is also installed in Google’s smart speakers, object information. to transcribe text and provide simple responses. 6. The task management node creates a task control Note that the Google Assistant API is only used for node to execute the task. speech recognition and transcription, weather forecast- 7. If necessary, more task control nodes are generated ing, and other general-purpose responses, and not for and coordinated by an additional number of tokens operating devices such as robotic beds. All device opera- for parallel execution. tions on ROS2-TMS were performed via ROS2. 8. Each task control node eventually requests one sub- The ROS-TMS also had a voice interface. However, a task node. microphone control node implemented the task search 9. Each subtask node executes its process. and execution functions, which had to be developed when a task was added (Fig.  6a). In ROS2-TMS, the task The subsequent sections on “Voice interfaces (TMS_ search and execution functions are outsourced to the UR),” “Task scheduler (TMS_TS),” and “Database (TMS_ other nodes. In particular, the task scheduler can manage DB)” describe the function of the individual modules in the execution of all tasks. As a result, the development of detail. a microphone module is no longer necessary when tasks are added or changed. In addition, tasks that combine multiple devices can be realized using only the task infor- Voice interfaces (TMS_UR) mation in the database (Fig. 6b). Currently, smart speakers are being released by various First, the wake word “ROS-TMS” is detected by companies, and home automation, in which home appli- Julius  [18]. Once the wake word is detected, Google ances are controlled by smart speakers, is also gaining Assistant transcribes the request. The transcribed text is popularity. In ROS2-TMS, we implemented a voice inter- separated into words by Janome  [19] and then matched face using the Google Assistant API [17], a voice assistant to tags in the database for task execution by ROS2-TMS. Fig. 6 Task execution flow of User Interface ( TMS_UR) and Task Scheduler ( TMS_TS) I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 7 of 19 If an appropriate task is found, TMS_TS schedules the ID: A unique ID in the database. execution of the task. If a proper task cannot be found, Name: A name that represents this object. the speaker outputs the response using Google Assistant. Type: The name of the type that the object repre - “Cancel” is also a wake word recognized by Julius. sents, such as “room_place.” The same type must When Julius recognizes “cancel,” TMS_TS is requested to have the same properties. terminate the current task immediately. Tags for searching: Store several related words for searching the object. In addition to this, there shold Database (TMS_DB) be properties for each type. The database stores tasks and environmental informa - tion, such as a map of the environment, task information, The information in the database can be referenced and robot, human, and object positions. from other information. For example, the robot move- The task information included an ID of the task, a ment task (ID: 9001) refers to the patrol points speci- sequence of subtasks that complied with the task, tags fied by the user and stored in the database; thus, the used for voice search, and the text that was announced robot moves autonomously along these points (Fig.  7). when the task was started. It is described in detail in the “Integrate tasks and data- In addition to these tasks, other objects can be added bases” subsection under the “Task Scheduler (TMS_ to the database. The following four pieces of information TS)” section. are required to work with the voice interface: “ROS-TMS, Double, go to the bed.” Search in the database Task Information Object (location) information “Robot movement task” “Nearly bed” (a) Searching for tasks andobjects in thedatabase Subtask composition of “Move” task “Nearly bed” data 9001${“position”: (room_place.position), type: “room_place” “orientation”: (room_place.orientation)} position: [8.65, 1.62, 0.0] orientation: [0.0, 0.0, 0.28, 0.96] Composition of subtasks to be executed 9001${“position”: [8.65, 1.62, 0.0], “orientation”:[0.0, 0.0, 0.28, 0.96]} (b) Linking movement tasks withlocation information Fig. 7 Task definition in database Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 8 of 19 Task scheduler (TMS_TS) consists of the following contents. All tasks are stored The task scheduler module (TMS_TS) receives the task as data structures in the database and thus have no sub- requested from the user request (TMS_UR), analyzes stance as programs. the request, and executes the task while interpolating the necessary information in cooperation with the database. • Tags for searching by a requested text In our previous study  [1], we adopted the approach • Set of subtasks named “task information structuring.” In this approach, • Text announced when starting a task a task is defined as a combination of subtasks, which are basic actions shared among different tasks. We imple - If a task needs to refer to environmental information in mented moving, grasping, and handling subtasks, all of the database, such as the destination of a robot in a mov- which are fundamental functions for service robots. In ing task, the type of necessary information is also speci- addition to these subtasks, we implemented new sub- fied in the data structure above. tasks in ROS2-TMS, which control not only robots but A subtask is implemented as an ROS2 node that can be also devices such as speakers, beds, and room lighting. In executed by robots or devices. A subtask is composed of addition, all tasks can be canceled using the ROS2 Action the following contents. protocol. Consequently, ROS2-TMS has the following advan- • Unique ID for each subtask tages compared to ROS-TMS. • Processing implementation • Implementation of the cancelation process • Controllable devices are increased. Various devices, such as speakers, room lighting, and beds, can be Because a task is defined as a set of subtasks, a new task controlled simultaneously to provide service tasks can be represented as a set of existing subtasks; thus, it is with robots (Fig. 8). not necessary to implement the execution code of robots • Since all operations are centrally controlled by the or devices for each task individually. task manager using the ROS2 Action protocol, any task can be terminated or canceled immediately. Method of adding a subtask It is implemented by inheriting the SubtaskNodeBase Two‑layer service structure: tasks and subtasks class, which inherits from the ROS2 Node class and then We defined and provided services using a two-layer overrides the following items: structure in ROS2-TMS, that is, the task and subtask layers. Name: Name of the ROS2 node. A task is defined by a one-to-one correspondence with The ID of the subtask: Assign a number between a service that can be requested in ROS2-TMS. The task 9000 and 9999. Robot subtask Bed subtask 9001 : The robot moves to "Nearly Bed" 9102 : Raise the head side of the bed 9900 : Wait 3 seconds 9300: Announce "Raised the bed" 9200 : Lighting on Connect 9300 : Announce "Turned on the lights " 9300 : Announce "Double has arrived" Lights subtask Speaker subtask 9300: Announce "Good morning" Subtasks “Good morning” task Fig. 8 “Good morning” task consisting of subtasks for various devices I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 9 of 19 Function at runtime: Define the process at runtime. • Raise (9100) or lower (9101) the head and leg sides of Because arguments assigned by the task scheduler the bed. are allocated, it is possible to change the behavior • Raise (9102) or lower (9103) the head side of the bed. according to the arguments. • Raise (9104) or lower (9105) the leg side of the bed. Function when canceled: The action to be taken can • Raise (9106) or lower (9107) the height of the bed. be defined when cancelation is requested. Other subtasks Implementation of subtasks • Turn on (9200) or off (9201) the lights in the room. Subtasks are implemented as the minimum processes • Play a text from the speaker (9300). executed by the robots and various devices. All subtasks • Wait for specified seconds (9900). were implemented as nodes in the ROS2. IDs from 9000 to 9999 were assigned to the subtasks. Currently, 13 sub- tasks are defined in the database. The detailed behaviors Connecting subtasks within a task when executing the subtasks are shown in Table 1. A task is defined as a set of subtasks and stored with string information in the database. The subtask execu - Subtask for communication robot (tms_rc_double) tion token is represented by “subtask_id” or “subtask_id $ ID 9001 is a subtask for a robot, such as the communica- json_format_argument,” and the task is specified by con - tion robot Double 2, to move to a destination. The desti - necting them with the sequential execution token “+” or nation is defined with a position and an orientation and the parallel execution token “|.” Examples of connecting specified when the subtask is called. This movement sub - multiple subtasks using this notation are shown in Fig. 9. task uses the Navigation2 package and a behavior tree to More complex tasks can be represented with the move. Backus-Naur Form (BNF) notation as follows: �task − structure� ::= �task� • Double 2 moves to the destination (9001). �task� ::= �task��task��operator� | �subtask� ′′ ′′ �operator� ::= “ + | “| Subtasks for robotic bed (tms_rc_bed) ′′ IDs in the 9100s are subtasks for a robotic bed. For each �subtask� ::= “subtask − id subtask, the execution time in seconds is defined as an ′′ | “subtask − id$json − arguments argument. Table 1 Definition of the implemented subtasks Name IDs Arguments Behavior Behavior on cancelation Robot movement subtask 9001 Goal position (x, y, z) Requests the Navigation2 package to autonomously move the Cancels the currently Goal angle (w, x, y, z) robot to the target location using ROS2 action communication. If executed request to Navi- the request fails, it informs the upper-level task control node that gation2 by ROS2 action. it has failed. Subtasks for robotic bed Execution seconds Requests the bed control node (websocket operation) in TMS_RC Sends a stop command to perform the respective operation via a websocket, such as rais- to the bed control node ing the bed. If another task has already made the bed request, this via a websocket. subtask informs the upper task control node of the failure. Subtasks for the lights 9200 None An HTTP request is sent to the lighting management server None 9201 located in the big sensor box to turn the lights on and off. Failure is reported to the upper task control node if no response is received from the server within the 5-second timeout period. A subtask for the speaker 9300 Text to be spoken Request the speaker control node in TMS_UR to speak using the None ROS2 service. A subtask for the speaker 9900 Execution seconds Wait for the specified time. None Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 10 of 19 Fig. 9 Task notations �task − structure� represents the connection between the and �property − name� is the property’s name of the data- ′′ subtasks. “subtask − id is the ID of the subtask to be base object. ′′ executed, and “subtask − id$json − arguments is the ID The integration of object information and the task and some arguments. For example, the string command is performed by the task search node in TMS_UR and for raising the height of the bed for 17 s is represented as the task management node in TMS_TS. For example, details of the operation from 3 to 5 shown in the “Task ′′ ′′ 9106${ sec : 17.0} execution flow” section are as follows: This command implies that the subtask to raise the height 1. The task search node in TMS_UR checks the string of the bed (ID: 9106) is executed for 17 s. against the task tag in the database and obtains the task ID and object. Integrate tasks and databases 2. The task search node sends the task ID and objects to The task description in the database can be linked with the task management node in TMS_TS. other objects. 3. The task management node retrieves the task from (�object − type�.�property − name�) the task ID. After that, it replaces the above syntaxes of the startup readout text and subtask structure with The syntax above can be replaced by writing it in the the information of the object. subtask structure or the startup readout property. 4 A task control node is generated, and the task is exe- �object − type� is the type of information in the database, cuted. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 11 of 19 Here is an example of a robot moving task. When a user • Raise or lower the leg side of the bed for a certain says “ROS-TMS, Double, go to the bed,” a task search period. node in TMS_UR retrieves the “Robot movement task” • Raise or lower the height of the bed for a certain and the “Nearly bed” location information from the tags period. in the database (Fig. 7a). Subsequentry, the task manage- ment node in TMS_TS replaces the information in the “Robot movement task” with the contents of the “Nearly Tasks related to the lights (2 types) bed” (Fig. 7b). It then executes the task. • Turn on or off the lights in the room. Method of adding a task Adding a task is performed by storing the following infor- Tasks that combine multiple devices (2 types) mation in the database. • “Good morning” task ID: Set a unique number that does not overlap with • “Good night” task any other object in the database. Name: Give a name to the task content. Type: Set the string “task” to distinguish it from other objects in the database. Task execution Subtask composition: Set the composition of the TMS_TS dynamically allocates a thread to execute the subtasks using the syntax described in the “Con- requested task when the execution of a task is requested necting subtasks within a task” subsection. by TMS_UR (Fig.  10a). The number of threads is set to Required tags and tags: If a word in a required tag n + 1 , where n is the number of tokens “| ” that represent is included in a sentence, this task is searched. If a the parallel execution. word in the tag is included in a sentence, the priority A control node is an ROS2 node that corresponds to of this task is increased in the searched tasks. each thread. Several control nodes are created when tasks Announcement text: An announcement at startup start, and these nodes are connected to each other by and when object integration fails (optional). Set the the ROS2 action protocol in the tree structure. The task text to be spoken by the speaker at startup. The text scheduler represents the root, the control nodes repre- can also be set when the integration with the data- sent the branches, and the subtask execution nodes rep- base fails; for example, when the target point of the resent the leaves. Action communication in ROS2  [14] robot’s movement task does not exist in the database. handles accept or reject requests, and if accepted, the status of the action is returned when the action is con- cluded, such as succeeded, failed, or canceled. In addi- tion, a cancelation request can be accepted while the Implementation of tasks action is being executed. Currently, there are 13 types of tasks. Tasks exist in a database, and their behavior can be based on a single Cancel function subtask or a combination of subtasks. In ROS-TMS, a user cannot stop a task in progress. To implement this cancelation function, an update to the A task related to the communication robot (1 type) action function in ROS2 is required. ROS action commu- nication consists of two types of entities: clients and serv- • Double 2 moves to the destination. ers. However, the server could not recognize which client requested it. In ROS2, when a client requests the server, Currently, there are 3 target locations in the database: the client generates a UUID [14]. This UUID is used as an near the entrance, near the kitchen, and the bed, and identifier of the request between the server and the cli - Double2 can move to them. ent to manage each request individually. This new feature was used to map tasks to subtasks in the task scheduler. Tasks related to the robotic bed (8 types) In ROS2-TMS, even when a task is running, the task can be stopped immediately by receiving a “cancel” voice • Raise or lower the head and leg sides of the bed for a request issued by the user or other cancelation com- certain period. mands. The cancel operation is realized by sending a • Raise or lower the head side of the bed for a certain cancel command to the subtasks currently running. It is period. an emergency stop function, and there is currently no Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 12 of 19 further serial operations of subtasks are terminated Run in thread-1 + (Fig. 10b, c). Error termination If the subtask is not completed or canceled, its subtask node informs the task search node in the TMS_UR of Run in thread-2 | + the failure through the requested task control node or its upper-level task management node. When the task Run in thread-3 search node is informed of the failure, it sends a mes- 9001 9300 sage to the speaker that the task has failed or has been canceled. 9102 9300 9300 + : sequential execution Tasks that combine subtasks 9900 9200 | : parallel execution The task scheduler in ROS2-TMS can manage not only (a) Threading robots but also IoT devices such as beds, lights, and microphones, and ROS2-TMS can provide various ser- vices by combining robots and intelligent home appli- Run in thread-1 ances. Examples of services that are newly added to the ROS2-TMS, such as the “Good morning” and “Good night” tasks, are shown in Fig.  11. Both tasks combine the subtasks for the communication robot Double 2, a robotic bed, lighting, and speaker. For example, the Run in thread-2 | + “Good morning” task is defined as shown in Fig. 12. As shown above, these tasks consist of sequential and Successful Run in thread-3 9001 9300 termination parallel executions of several subtasks and are highly In operation complicated. We confirmed that the cancel function is 9102 9300 9300 appropriately executed even for these complex tasks. In operation + : sequential execution 9900 9200 | : parallel execution Service experiments using ROS2‑TMS and big sensor box (b)Inoperation We conducted an experiment to verify the operation of services with a voice interface using ROS2-TMS in Big Run in thread-1 Sensor Box (Fig. 2). The big sensor box is a hardware platform for an infor - mationally structured environment. As described in [1], it is located on the second floor of the COI building at the Ito campus of Kyushu University and is a habitable Run in thread-2 | + space with a living room, bedroom, and kitchen. In the room, there is a service robot, such as Double 2 and Dou- Successful Run in thread-3 9001 9300 + termination ble 3, and a motion capture system that can acquire the Canceled robot’s position, a robotic bed, and a server that manages 9102 9300 9300 the room lighting. In this study, we interconnected them using ROS2 communication and experimented with pro- + : sequential execution 9900 9200 | : parallel execution viding services using ROS2-TMS. We confirmed that ROS2-TMS can launch proper tasks (c) Canceling according to a user’s voice request through the micro- Fig. 10 Thread allocation for “Good morning” task phone device and perform tasks such as user interaction (daily conversation), bed operations, lighting control, and robot motion by combining the proper information in function to resume a task that has been stopped in the the database. middle. The following subsections show how each task was per - Even if the execution time is specified when start - formed. Detailed timecharts of each task are shown in ing subtasks, the execution is stopped immediately, and Figs. 13 and 14. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 13 of 19 high is Mount Fuji?,” proper responses were provided by Google Assistant (Fig. 15). Lighting control 9001:The robot movesto"Nearly Bed" When a user requests tasks related to the lighting such as “ROS-TMS, turn on the lights” or “ROS-TMS, turn off the 9102:Raisethe head side of the bed 9900:Wait3 seconds lights,” ROS2-TMS chooses and executes the proper tasks by sending commands to the lighting management server in Big Sensor Box (Fig. 16). 9300:Announce "Raised thebed" 9200: Lighting on Robotic bed control 9300:Announce"Turned on thelights" 9300:Announce"Double hasarrived" When a user requests tasks related to a robotic bed, such as “ROS-TMS, raise the height of the bed,” ROS2-TMS exe- cutes the proper tasks to control the robotic bed (Fig. 17a). 9300:Announce "Goodmorning" Robot control When the user requests “ROS-TMS, Double, go to the kitchen,” TMS_TS in ROS2-TMS generates a motion plan for the communication robot Double 2 to move to the (a) “Good morning” task kitchen. The coordinates of the kitchen are stored in the database and interpolated to generate motion commands in TMS_TS. In addition, if the user requests “ROS-TMS, Double, go to the bed,” Double 2 moves to the vicinity of the bed by linking with the bed coordinates stored in the 9001:The robotmoves to "Kitchen" database. (Fig. 18a, b). 9300: Announce "Lay down thebed"9103: Lower theheadsideofthe bed Execution of complex tasks We implemented the “Good morning” and “Good night” 9300:Announce "Double hasreturned to thekitchen" tasks to confirm the sequential and parallel executions of subtasks. Both tasks consisted of several subtasks related 9300: "Turnoff the lights"9201:Lightingoff to robot motion, bed operation, and room lighting control connected sequentially or in parallel. To confirm the flex - ibility of the task representation, we implemented these tasks with several different connections of subtasks and 9300:Announce"Good night" confirmed that these tasks were executed appropriately even with complex task structures. “Good morning” task (b) “Good night” task The “Good morning” task performs the following three Fig. 11 Example of tasks combining subtasks subtasks in parallel. • Raise the upper body of the bed and announce “Raised Daily conversation the bed” from the speaker. When a user asks ROS2-TMS questions such as “ROS- • Wait for 3 s, turn on the lights, and announce “Turned TMS, What’s the weather today?” or “ROS-TMS, How on the lights.” 9102${“sec”:20.0} 9300${“announce”: “Raised the bed”} +9900${“wait sec”: 3.0} 9200 +9300${“announce”: “Turned on the lights”} + | 9001${“position”: [8.65,1.62, 0.0],“orientation”: [0.0, 0.0, 0.280, 0.960]} 9300${“announce”: “Doublehas arrived”} + | 9300${“announce”: “Goodmorning”} + Fig. 12 Task notation of “Good morning” task Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 14 of 19 (a) “Weather forecast” time chart. User 0: Askthe weatherto microphone.Speaker 1: Answer theweather. (b) “Switching off the light” time chart. User 0: “ROS-TMS,turnoffthe lights”. Speaker 1: Announcementat task startup. Light 2: Turned off. (c) “Lower the head side of thebed”timechart. User 0: “ROS-TMS,lower thehead sideofthe bed”. Speaker 1: Announcementat task startup. Bed 2: Lowered thehead side. (d) “Canceling bed control task”timechart. User 0: “ROS-TMS,lower thehead side of thebed”. Speaker 1: Announcementat task startup. Bed 2: Loweredthe head side. User 3: “Cancel”. Speaker 4: Announcement at task cancelation. (e) “Double goes to the kitchen”timechart. User 0: “ROS-TMS, Double, go to thekitchen”. Speaker 1: Announcementat task startup.Robot 2: Went to thekitchen. (f) “Double goes to thebed”time chart. User 0: “ROS-TMS, Double, go to thebed”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. (g) “Canceling move task” time chart. User 0: “ROS-TMS, Double,gotothe bed”.Speaker 1: Announcement at task startup. Robot 2: Went to thenearsideofthe bed. User 3: “Cancel”. Speaker 4: Announcementattask cancelation. Fig. 13 Time chart of task execution (1 of 2) I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 15 of 19 (a) Execute “Good morning” task time chart. User 0: “ROS-TMS,goodmorning”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. Speaker Robot 3: Announced“Doublehas arrived”.Bed 4: Raisethe head sideofthe bed. Speaker Bed 5: Announced “Raised thebed”. Light 6: Turned on.Speaker Light 7: Announced “Turnedonthe lights”. Speaker GoodMorning 8: Announced “Good morning”. (b) Canceling “Good morning” task time chart. User 0: “ROS-TMS,goodmorning”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. Bed 4: Raisethe head side of thebed.Light 6: Turned on.Speaker Light 7: Announced“Turned on the lights”. User Cancel 9: “Cancel”.Speaker Cancel 10: An- nouncementat task cancelation. (c) Execute “Good night” task time chart. User 0: “ROS-TMS,goodnight”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe kitchen.Speaker Robot 3: Announced“Doublehas returned to the kitchen”. Speaker Bed 4: Announced “Laydownthe bed”. Bed 5: Loweredthe head sideofthe bed. Speaker Light 6: Announced “Turnoffthe lights”. Light 7: Turned off. Speaker GoodNight 8: Announced “Goodnight”. (d) Canceling “Good night” task time chart. User 0: “ROS-TMS, good night”.Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe kitchen. Speaker Bed 4: Announced “Lay down thebed”. Bed 5: Lowered thehead sideofthe bed. User Cancel 9: “Cancel”. Speaker Cancel 10: Announcement at task cancelation. Fig. 14 Time chart of task execution (2 of 2) Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 16 of 19 User System User System Today in Fukuoka, it’ll be sunny, ROS-TMS, What’s the weather today? ROS-TMS, Double, go to the kitchen. Double goes to the kitchen. with a forecast high of 13 and a low of 2. 1 2 1 2 3 (a) Doublegoestothe kitchen Fig. 15 Weather forecast User System ROS-TMS, Double, go to the bed. Double goes to the bed. 1 2 3 User System ROS-TMS, turn off the lights. Turn off the lights. 1 2 (b) Doublegoestothe bed User System User System ROS-TMS, Double, Double goes to the Canceled the Cancel. go to the bed. bed. task. Fig. 16 Switching off the light 13 2 Stop moving midway. User System (c) Canceling move task ROS-TMS, lower the head side Lower the head side of the bed. of the bed. Fig. 18 Move task for communication robot 1 2 “Good night” task The “Good night” task executes two subtasks in parallel. In addition, there is a further parallel operation within the first operation. The following two operations are exe - (a) Lowerthe head sideofthe bed cuted in parallel. User System User System • A task lays the head side of the bed down and ROS-TMS, lower the Lower the head Canceled the Cancel. head side of the bed. side of the bed. task. announces “Lay down the bed” from the speaker and then turns off the lights in the room and announces 13 2 “Turn off the lights.” • After Double 2 returns to the kitchen, the speaker announces, “Double has returned to the kitchen.” Stop lowering the bed. After completing all the subtasks, “Good night” is (b) Cancelingbed control task announced from the speaker (Fig. 20a). Fig. 17 Bed control task Cancel tasks If the word “cancel” is included in the user’s command, TMS_TS sends a cancel command to the control nodes • Move the communication robot Double 2 close to in all subtasks executing currently. When the control the bed and announce “Double has arrived.” node in the subtask receives the cancel command, the control node sends the cancel command to the subtasks After completing all the subtasks, “Good morning” is and the control nodes of the child subtasks executing announced from the speaker (Fig. 19a). I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 17 of 19 User System System User System System ROS-TMS, good Start the “Good morning” Turned on the lights. ROS-TMS, good night. Start the “Good night” task. Lay down the bed. morning. task. 1 2 1 2 System System System System System System Double has returned Turn off the lights. Raised the bed. Double has arrived. Good morning! Good night! to the kitchen. 3 4 3 4 (a) Execute “Goodmorning” task (a) Execute “Goodnight”task User System System User System System ROS-TMS, good Start the “Good morning” Turned on the lights. ROS-TMS, good night. Start the “Good night” task. Lay down the bed. morning. task. 1 2 1 2 User System User System Cancel. Canceled the task. Cancel. Canceled the task. 3 4 3 4 (b) Canceling “Good morning” task (b)Canceling“Good night” task Fig. 19 “Good morning” task Fig. 20 “Good night” task in parallel. When all the control nodes in the sub- • The middleware was changed from ROS to ROS2, tasks receive the cancel command, the task is stopped and various modules were ported and newly devel- (Figs. 17b, c, 19b, and 20b). oped. Currently, ROS2-TMS does not actively use secu- Conclusions rity and QoS settings. However, as we continue our In this study, we presented a new software platform for research, QoS will used to configure sensor streams, an ISE, named ROS2-TMS, and demonstrated a few such as cameras, which have a large amount of data. robot services using ROS2-TMS. ROS2-TMS is publicly For the robot movement subtask, we used Naviga- available and can be downloaded from [20]. The differ - tion2 and the behavior tree. The application of ROS2- ences between ROS2-TMS and the previous ROS-TMS TMS with this behavior tree is more flexible than system are summarized as follows. ROS-TMS for moving in complex environments. Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 18 of 19 Acknowledgements • The interaction with a user was realized using the This work was partially supported by the Cabinet Office (CAO) Cross-Ministe - Google Assistant API. rial Strategic Innovation Promotion Program (SIP), “An intelligent knowledge • The task scheduler was extended to manage not only processing infrastructure, integrating physical and virtual domains” (funding agency: NEDO). robots but also various IoT devices such as intelligent beds, lighting, and speakers. This makes it possible to Authors’ contributions provide various services that combine robots and IoT TI drafted the manuscript. RK constructed the study concept. AK and RK man- aged the study. TI and MS developed the system and carried out the experi- devices. ments. All members verified the content of their contributions. All authors • To improve safety, we added a new function that read and approved the final manuscript. allows the user to stop running tasks by issuing a Funding cancelation request to the ROS nodes using the This work was partially supported by the Cabinet Office (CAO), Cross-Ministe - ROS2 Action protocol. rial Strategic Innovation Promotion Program (SIP), “An intelligent knowledge processing infrastructure, integrating physical and virtual domains” (funding agency: NEDO). Future works includes the following improvements. Availability of data and materials Not applicable. Low‑cost localization system In this study, an optical motion capture system was used Declarations to estimate the position of the robot. Although this sys- Competing interests tem can identify the robot position with high accuracy, The authors declare that they have no competing interests. it is considerably expensive. Currently, we are testing less expensive range-based localization systems such as Received: 25 September 2021 Accepted: 28 December 2021 Wi-Fi, RFID, UWB, and visible light  [21], and we believe that low-cost autonomous mobile services can be pro- vided by applying these technologies. References 1. Sakamoto J, Kiyoyama K, Matsumoto K, Pyo Y, Kawamura A, Kurazume R (2018) Development of ros-tms 5.0 for informationally structured environ- Services by health status ment. ROBOMECH J 5:24. https:// doi. org/ 10. 1186/ s40648- 018- 0123-9 In this study, we developed a module for a wearable heart 2. ROS.org | Powering the world’s robots. https:// www. ros. org/. Accessed 3 rate sensor. We will develop a service robot that recog- Aug 2021 3. ROS 2 Documentation: Foxy. https:// docs. ros. org/ en/ foxy/ index. html. nizes not only the human’s location but also the health Accessed 30 July 2021 condition using environmental sensors as a state analyzer 4. Macenski S, Martin F, White R, Clavero JG (2020) The Marathon 2: a naviga- module (TMS_SA, in Fig. 3) and provides proper services tion system. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 2718–2725. https:// doi. org/ 10. 1109/ IROS4 according to the health status of the individual. 5743. 2020. 93412 07. https:// github. com/ ros- plann ing/ navig ation2https:// ieeex plore. ieee. org/ docum ent/ 93412 07/ 5. Sato T, Nishida Y, Mizoguchi H (1996) Robotic room: symbiosis with Dynamic planning of tasks, such as autonomous robot human through behavior media. Robot Auton Syst 18(1–2):185–194. movement https:// doi. org/ 10. 1016/ 0921- 8890(96) 00004-8 6. Brooks RA (1997) The intelligent room project. In: Proceedings of the 2nd The autonomous movement of robots in large areas, International Conference on Cognitive Technology (CT ’97). CT ’97, p 271. including doors or elevators, requires coordination with IEEE Computer Society, USA surrounding actuators  [12, 22]. Therefore, even for the 7. Park KH, Bien Z, Lee JJ, Kim BK, Lim JT, Kim JO, Lee H, Stefanov DH, Kim DJ, Jung JW, Do JH, Seo KH, Kim CH, Song WG, Lee WJ (2007) Robotic same autonomous movement, the composition of the smart house to assist people with movement disabilities. Auton Robots subtasks differ significantly between movement in a 22(2):183–198. https:// doi. org/ 10. 1007/ s10514- 006- 9012-9 room and movement in a large area across rooms. For 8. Amazon Alexa Voice AI | Alexa Developer Official Site. https:// devel oper. amazon. com/ en- US/ alexa. Accessed 23 Aug 2021 such environment-dependent tasks, we will extend the 9. Google Assistant. https:// assis tant. google. com/. Accessed 23 Aug 2021 task scheduler so that the necessary functions can be 10. Kwak SS, San Kim J, Moon BJ, Kang D, Choi J (2020) Robots versus dynamically linked. speakers: what type of central smart home interface consumers prefer? In: 2020 IEEE/RSJ International Conference on Intelligent Robots and In addition, when several tasks are executed simultane- Systems (IROS), pp 11397–11404. https:// doi. org/ 10. 1109/ IROS4 5743. ously, it is necessary to monitor and appropriately control 2020. 93417 48. https:// ieeex plore. ieee. org/ docum ent/ 93417 48/ the progress of all the tasks currently running. Thus, we 11. Melo N, Lee J, Suzuki R (2018) Identification of the User’s Habits based on Activity Information. In: 2018 IEEE/RSJ International Conference on will design an efficient interface to present detailed and Intelligent Robots and Systems (IROS), pp 2014–2019. https:// doi. org/ 10. comprehensive information, including the task status, 1109/ IROS. 2018. 85938 73. https:// ieeex plore. ieee. org/ docum ent/ 85938 location of humans and robots, and human conditions. 73/ I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 19 of 19 12. Cavallo F, Limosani R, Manzi A, Bonaccorsi M, Esposito R, Di Rocco M, Pecora F, Teti G, Saffiotti A, Dario P (2014) Development of a socially believable multi-robot solution from town to home. Cognit Comput 6(4):954–967. https:// doi. org/ 10. 1007/ s12559- 014- 9290-z 13. Simoens P, Dragone M, Saffiotti A (2018) The internet of robotic things: a review of concept, added value and applications. Int J Adv Robot Syst 15(1):1–11. https:// doi. org/ 10. 1177/ 17298 81418 759424 14. ROS2 Design | Actions. http:// design. ros2. org/ artic les/ actio ns. html. Accessed 4 Feb 2020 15. Double Robotics - Telepresence Robot for Telecommuters. https:// www. doubl erobo tics. com/ doubl e2. html. Accessed 2 Feb 2020 16. Double Robotics—Telepresence Robot for the Hybrid Office. https:// www. doubl erobo tics. com/ doubl e3. html. Accessed 15 Sept 2021 17. Overview | Google Assistant SDK | Google Developers. https:// devel opers. google. com/ assis tant/ sdk/ overv iew. Accessed 2 Feb 2020 18. Julius. https:// julius. osdn. jp/. Accessed 14 Sept 2021 19. Janome v0.3 documentation ( ja). https:// mocob eta. github. io/ janome/. Accessed 2 Feb 2020 20. https:// github. com/ irvs/ ros2_ tms. Accessed 16 Sept 2021 21. Zafari F, Gkelias A, Leung KK (2019) A survey of indoor localization sys- tems and technologies. IEEE Communications Surveys and Tutorials 21(3), 2568–2599. https:// doi. org/ 10. 1109/ COMST. 2019. 29115 58. arxiv: 1709. 22. Martin F, Gines J, Vargas D, Rodraguez-Lera FJ, Matellan V (2018) Planning topological navigation for complex indoor environments. In: 2018 IEEE/ RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1–9. https:// doi. org/ 10. 1109/ IROS. 2018. 85940 38https:// ieeex plore. ieee. org/ docum ent/ 85940 38/ Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ROBOMECH Journal Springer Journals

Development of ROS2-TMS: new software platform for informationally structured environment

Loading next page...
 
/lp/springer-journals/development-of-ros2-tms-new-software-platform-for-informationally-g5TbjaJgEV

References (5)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
eISSN
2197-4225
DOI
10.1186/s40648-021-00216-2
Publisher site
See Article on Publisher Site

Abstract

This study proposes a new software platform, called ROS2-TMS, for an informationally structured environment. An informationally structured environment is vital for developing intelligent service robots by embedding various sensors in the environment to enhance the sensing capability and intelligence of robots. Thus far, we have been developing a software platform, named ROS-TMS, for an informationally structured environment, which connects various sensors and robots using ROS architecture. In recent years, ROS2, a next-generation version of ROS, has been released. ROS2 has many advantages, such as enhanced security, QoS control, and support for various platforms. ROS2-TMS, a new version of ROS-TMS, is developed not only by porting existing modules in ROS-TMS, such as the control system for a communication robot, but also by adding useful functions utilizing new features in ROS2. For instance, we added a voice user interface to control robots and various devices in the environment, such as lights or a bed. In addition, we implemented a new task scheduler that provides a cancelation function to stop running tasks and improve the security of the platform. Keywords: Service robot, Informationally structured environment, Internet of things, Cyber physical system, Ambient sensing Introduction the surrounding environment to collect, analyze, and In recent decades, with the decline in birthrate and aging retain environmental information. As a software platform of population, labor shortage has become a crucial issue for an ISE, we have been developing ROS-TMS [1], which in various fields such as medical and nursing care. As a realizes service robots that coexist with humans. ROS- solution to this problem, the realization of life support TMS connects various sensors embedded in the environ- services using artificial intelligence and service robots has ment and service robots using a robot operating system attracted considerable attention. However, unlike robots (ROS) [2], which is a general-purpose robot middleware. operated in factories, the environment in which service In recent years, ROS2  [3], a next-generation version of robots work for daily life support is diverse and dynami- ROS, has been released. ROS2 has many advantages, cally changing. Therefore, it is difficult to fully under - such as enhanced security, QoS control, support for vari- stand the surrounding situation based on the embedded ous platforms, and advanced navigation algorithms [4]. sensors of the robot alone. One of the key solutions to In this study, we propose a novel software platform for this problem is an informationally structured environ- an ISE, named ROS2-TMS. The characteristics of ROS2- ment (ISE), in which sensors are distributed throughout TMS are as follows. (1) Upgradation of the middleware from ROS to ROS2: Enhanced security, QoS control, and latest navigation algorithms are available. (2) Uni- *Correspondence: itsuka@irvs.ait.kyushu-u.ac.jp fied task management mechanism: Regardless of the Kyushu University, 744 Motooka, Nishi-ku, Fukuoka-shi, Fukuoka 819-0395, type of device (for example, robots, room lighting, and Japan © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 2 of 19 intelligent home appliances), service tasks can be man- Various research topics and references on IoRT are intro- aged uniformly as subtasks in the task manager. This duced in [13]. makes it easier to perform advanced coordination of vari- The robot operating system (ROS)  [2], an open source ous devices. (3) Cancelation function: By managing sub- platform, has contributed significantly to the develop - tasks in all devices using the ROS2 Action protocol, all ment and research of mobile and service robots. The service tasks can be terminated during execution regard- new generation of ROS, ROS2 [3], has enhanced security less of the complexity of the task, thus, increasing safety. and QoS control and supports the latest development environments, such as C++14 and Python3. The latest robot autonomous mobility package Navigation2 is also Related works available in ROS2. Navigation2 [4] consists of a behavior ISE enables the flexible operation of service robots by tree, which represents a management mechanism that is collecting, analyzing, and managing data for a complex higher than the autonomous movement of the robot, and environment by deploying distributed sensors not only it is easy to describe complex control such as temporary on the robot but also in the environment. Several systems avoidance behavior for environments wherein autono- have been developed on ISE thus far. In the “Robotic mous movement is difficult. Consequently, it has the Room”  [5] at The University of Tokyo, an approach to potential to cope with complex everyday environments. deploy various sensors in the environment was pro- One of the important updates in the internal design is the posed to observe the status of patients. In the “Intelligent action protocol  [14], which is useful for time-consum- Room”  [6] at MIT, AILab deployed multiple cameras in ing tasks, such as the moving task of a robot or a pick- a room to track the location of humans and determine ing up task of a manipulator. With the evolution of the their direction to provide services. “Intelligent Sweet design, an identifier is issued for each action protocol to Home”  [7] proposed a robotic platform that includes an distinguish each action. This action protocol is incorpo - intelligent bed with pressure sensors and an autonomous rated into the task scheduler proposed in this research mobile wheelchair in the context of ambient assisted liv- and plays a major role in connecting individual tasks and ing (AAL), which aims to promote independent living subtasks. for the elderly and disabled. These studies on ISEs have focused on two aspects: (1) deploying sensors in the From ROST ‑ MS to ROS2‑TMS environment to collect advanced information regarding The authors began to develop an ISE in the “Robot Town humans, robots, and objects, and (2) providing services Project” in 2005 and have been developing a software to assist humans through robots and intelligent home platform named Town Management System (TMS). appliances. In [1], we proposed a software platform, ROS-TMS 5.0 In recent years, various systems related to ISE have (Fig. 1), and a hardware platform, Big Sensor Box (Fig. 2), been proposed and commercialized, such as AAL, smart for an ISE. ROS-TMS 5.0 is developed as a core software houses, and the Internet of Robotic Things (IoRT). In platform for IoRT and has some novel functions, such as the field of smart homes, smart speakers, such as Ama - a care receiver-watching service and a voice control for zon’s “Alexa” [8] and Google’s “Google Assistant” [9], are service robots. widely used in households, and the technology to control However, there were four concerning points about home appliances, lighting, and other Internet of Things ROS-TMS: (IoT) devices through voice interfaces is widely adopted in households. In [10], it was stated that the presence 1. Using ROS as middleware: ROS initially supported of a central interface, such as a smart speaker, increases Python2; thus, the ROS-TMS executable used the quality of the robot service. IoRT is a field that incor - Python2. However, support for Python2 was dis- porates IoT technology, in which various devices in the home environment are connected to the Internet and robotics technology. As a method of component coor- dination using IoT technology, there is coordination between sensors to collect information using multiple sensors, as well as coordination on actuators to perform various services; for example, [11] identifies user activi - ties and habits by collecting information from a group of sensors installed in a smart house, such as a pressure sensor installed in a bed. In [12], a service robot was con- nected to an elevator control unit to move the service Fig. 1 ROS-TMS robot across multiple floors of an apartment building. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 3 of 19 which can be used in the C++14 / Python3 environ- ment. Also, advanced ROS2 technologies such as QoS control and Navigation2 are applicable in ROS2- TMS. 2. Redesign of task execution: In ROS-TMS, the micro- phone module was responsible not only for under- standing the user’s speech but also for task search and task execution. In addition, ROS2 does not have SMACH, the task execution machine used in ROS- TMS; therefore, the task execution had to be rebuilt. Therefore, we separated task search and task execu - tion from the roles of the microphone module and implemented task search in the task search node and Fig. 2 Big sensor box task execution and management in the task scheduler module. Furthermore, task execution was performed using ROS2 actions, and we were able to add func- continued in 2020. We believe that the platform tions for failure behavior and stopping during execu- should support Python3 and the latest development tion. resources. 3. Redesign of tasks and subtasks: In ROS-TMS, there 2. Difficulty in task implementation: The ROS-TMS were only three types of subtasks: moving the robot microphone module, which is responsible for voice and grasping and releasing by the robot hand. In input, transcribes the user’s speech, searches for ser- ROS2-TMS, the scope of subtasks was expanded to vice tasks, and executes tasks specific to those ser - include robots as well as robotic beds, room light- vices (sending commands to devices such as robots, ing, and speakers. As a result, all service tasks can lightings, and robotic beds using ROS topics, ser- be managed using a task scheduler. In addition, tasks vices, socket communication, etc.). Therefore, the linked to multiple devices can be added simply by dependency between task implementation and the entering the configuration information of these sub - microphone module is high, and it is necessary to tasks in the database. develop the microphone control node again when 4. Addition of cancel function: In ROS-TMS, a user tasks are added. cannot stop a task in the middle of the execution. 3. Limited subtasks: In ROS-TMS, the objects to be ROS2-TMS has a newly developed task scheduler managed by the task scheduler were limited to with ROS2 action, which can stop a task in the mid- robots only, and there were only three types of sub- dle of an execution according to the user’s request. tasks: robot movement, robot arm grasp, and release. Each subtask defines its behavior when canceled Together with the problems discussed in 2) above, such that adding a task does not need to define a new it was difficult to implement services that scheduled behavior. robot movements and other devices. 4. Cannot stop a task while it is running: For example, The rest of the paper is organized as follows: The ROS2- when a robot is moving and bumps into something TMS modules section describes the structure of the inadvertently, it is necessary to perform an emer- modules in ROS2-TMS. The Robots, Sensors, and User gency stop of the service task. However, ROS-TMS Request Devices section describes the modules related does not allow the user to perform an emergency to the devices in ROS2-TMS, such as robots and robotic stop while the task is in progress. beds. The Task execution flow section describes an overview of task execution. The Voice interfaces (TMS_ Therefore, we developed a new software platform for an UR) section describes the design of the voice user inter- ISE, named ROS2-TMS. This platform has the following face. The Database (TMS_DB) section describes the four features to solve these four problems: design of the database. The Task scheduler (TMS_TS) section describes the design of the task scheduler. The 1. Using ROS2 as middleware: While ROS uses C++03 Robot service experiment section describes the service / Python2 as its development environment, ROS2 tasks that can be provided by ROS2-TMS and provides can now use C++14 / Python3. Therefore, ROS2- examples of their execution. Finally, the section  Con- TMS ports the core modules of ROS-TMS to ROS2, clusions provides the conclusion. TMS_DB: Database Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 4 of 19 ROS2‑TMS modules Sensor system module (TMS_SS) Sensor  data are ROS2-TMS has a wide variety of functions as an ROS2 interpreted and converted into higher-level environ- node. ROS2-TMS is composed of modules connected mental information and stored in TMS_DB. in a hierarchical manner as shown in Fig.  3. In a ROS2- State analyzer module (TMS_SA) This module TMS service, various modules play roles by intercon- receives information from TMS_SS and estimates necting with each other, such as interpreting the user’s the state of the environment. For example, we plan speech, selecting a task to be executed, planning, and to estimate the user’s health status, but we are still in even the robot’s behavior. The contents of each module the concept stage. Details are given in the “Conclu- are described below. sions” section. Architecture Robots, sensors, and user request devices Database module (TMS_DB) Stores environmental In this study, we upgraded the middleware for each information managed by ROS2-TMS in a database. device from ROS to ROS2, improved the communica- The database is developed with mongoDB. tion quality of the Data Distributed Service (DDS), and User request module (TMS_UR) Receives task expanded the functions of ROS2, such as the cancelation requests from users and sends task execution function. Some sensors and robots used in Big Sensor requests to TMS_TS. Box and ROS2-TMS are shown in Fig. 4. Task scheduler module (TMS_TS) The requested ser - vice is executed by combining subtasks. In the pre- Robot controller module (TMS_RC) vious study (ROS_TMS), only robots were subject to subtask management, whereas in this study, we Robotic bed An electric bed (Rakusho Z KQ-7302, expanded the scope of task management to intelli- Paramount Bed) is controlled by RaspberryPi Zero, gent home appliances such as beds and speakers. which can raise the upper body and the height of the Robot planning module (TMS_RP) From some sub- bed. tasks of the robot commanded by TMS_TS, motion Communication robot (Double 2, Double 3) Double planning was performed to correctly execute the 2  [15] and Double 3  [16] are communication robots subtasks. manufactured by Double Robotics. Double 2 and Robot controller module (TMS_RC) This is a module Double 3 are controlled by attached iPad and Linux that executes the planned subtasks using the robot. PC, respectively. We implemented a moving task for A dedicated module is implemented for each robot. the robots to a specified position and angle using a Sensor driver module (TMS_SD) The system acti - motion capture system and the ROS2 Navigation2 vates various sensors embedded in the environment package. and publishes the acquired sensor data. Sensor system module (TMS_SS) User Wearable Heart rate sensor (WHS-1) Whs-1 is a wearable heart rate sensor developed by Union Tool Corporation that updates the heart rate information TMS_UR: User Request in the database at each timestep at which the user’s heart rate is measured. TMS_TS: Task Scheduler Motion capture system (VICON) VICON is a motion capture system that uses multiple motion-tracking TMS_SA TMS_SS TMS_RP cameras to recognize the position of an object. We State Analyzer Sensor System Robot Planning used it to estimate the position of the robot. TMS_SD: TMS_RC: Sensor Driver Robot Controller User request module (TMS_UR) Sensor Robot Microphone device A USB microphone is attached to Fig. 3 ROS2-TMS architecture Intel’s NUC to receive voice requests from users. User I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 5 of 19 Sensor System Robot Controller User Request Robot Intelligent appliances Collect sensor infomation Interface with users 0DUNHU Motion capture Robotic bed Mic Communication Speaker Heart rate sensor Robot Fig. 4 Sensors and robots in TMS_SS, TMS_RC, TMS_UR modules Speaker device A USB speaker is attached to the TMS_TS, TMS_DB, and the task search node in TMS_ Raspberry Pi 3 to play a conversation with the user UR were executed on a dedicated ROS2-TMS server in using the Google Assistant API, as well as announce- our experiments. ments and sound effects when the task starts using An overview of task execution from a user’s voice com- TMS_TS. mand to the robot and lighting control is presented in Fig.  5. The following bullet points correspond to the fol - lowing figure: Task execution flow To run the ROS2-TMS service, the modules shown in the 1. The user requests a command from the microphone. previous sections need to control corresponding devices 2. The microphone control node transcribes the voice and tasks coordinately. In TMS_RC, the robots and the and sends a string to the task search node. robotic bed are controlled. In TMS_RP, the robots are 3. The task search node searches for related task IDs connected to TMS_TS through Navigation2, and the and objects in the database. robotic bed is connected directly to TMS_TS. The micro - 4. The task search node passes the task IDs and objects phones and the speakers are managed in TMS_UR. to TMS_TS. TMS_DB: Database TMS_UR TMS_TS 3. 5. 9. TMS_RP Subtask (Navigation2) Task control node Subtask node TMS_RC 1. 2. 4. 6. Find a task Mic Manage tasks Thread Microphone Task search node Task management Robot control node node Thread Subtask Bed Task control node Subtask Lighting Fig. 5 Flowchart of processes from user request to task execution in ROS2-TMS Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 6 of 19 5. The task management node integrates task and service that is also installed in Google’s smart speakers, object information. to transcribe text and provide simple responses. 6. The task management node creates a task control Note that the Google Assistant API is only used for node to execute the task. speech recognition and transcription, weather forecast- 7. If necessary, more task control nodes are generated ing, and other general-purpose responses, and not for and coordinated by an additional number of tokens operating devices such as robotic beds. All device opera- for parallel execution. tions on ROS2-TMS were performed via ROS2. 8. Each task control node eventually requests one sub- The ROS-TMS also had a voice interface. However, a task node. microphone control node implemented the task search 9. Each subtask node executes its process. and execution functions, which had to be developed when a task was added (Fig.  6a). In ROS2-TMS, the task The subsequent sections on “Voice interfaces (TMS_ search and execution functions are outsourced to the UR),” “Task scheduler (TMS_TS),” and “Database (TMS_ other nodes. In particular, the task scheduler can manage DB)” describe the function of the individual modules in the execution of all tasks. As a result, the development of detail. a microphone module is no longer necessary when tasks are added or changed. In addition, tasks that combine multiple devices can be realized using only the task infor- Voice interfaces (TMS_UR) mation in the database (Fig. 6b). Currently, smart speakers are being released by various First, the wake word “ROS-TMS” is detected by companies, and home automation, in which home appli- Julius  [18]. Once the wake word is detected, Google ances are controlled by smart speakers, is also gaining Assistant transcribes the request. The transcribed text is popularity. In ROS2-TMS, we implemented a voice inter- separated into words by Janome  [19] and then matched face using the Google Assistant API [17], a voice assistant to tags in the database for task execution by ROS2-TMS. Fig. 6 Task execution flow of User Interface ( TMS_UR) and Task Scheduler ( TMS_TS) I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 7 of 19 If an appropriate task is found, TMS_TS schedules the ID: A unique ID in the database. execution of the task. If a proper task cannot be found, Name: A name that represents this object. the speaker outputs the response using Google Assistant. Type: The name of the type that the object repre - “Cancel” is also a wake word recognized by Julius. sents, such as “room_place.” The same type must When Julius recognizes “cancel,” TMS_TS is requested to have the same properties. terminate the current task immediately. Tags for searching: Store several related words for searching the object. In addition to this, there shold Database (TMS_DB) be properties for each type. The database stores tasks and environmental informa - tion, such as a map of the environment, task information, The information in the database can be referenced and robot, human, and object positions. from other information. For example, the robot move- The task information included an ID of the task, a ment task (ID: 9001) refers to the patrol points speci- sequence of subtasks that complied with the task, tags fied by the user and stored in the database; thus, the used for voice search, and the text that was announced robot moves autonomously along these points (Fig.  7). when the task was started. It is described in detail in the “Integrate tasks and data- In addition to these tasks, other objects can be added bases” subsection under the “Task Scheduler (TMS_ to the database. The following four pieces of information TS)” section. are required to work with the voice interface: “ROS-TMS, Double, go to the bed.” Search in the database Task Information Object (location) information “Robot movement task” “Nearly bed” (a) Searching for tasks andobjects in thedatabase Subtask composition of “Move” task “Nearly bed” data 9001${“position”: (room_place.position), type: “room_place” “orientation”: (room_place.orientation)} position: [8.65, 1.62, 0.0] orientation: [0.0, 0.0, 0.28, 0.96] Composition of subtasks to be executed 9001${“position”: [8.65, 1.62, 0.0], “orientation”:[0.0, 0.0, 0.28, 0.96]} (b) Linking movement tasks withlocation information Fig. 7 Task definition in database Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 8 of 19 Task scheduler (TMS_TS) consists of the following contents. All tasks are stored The task scheduler module (TMS_TS) receives the task as data structures in the database and thus have no sub- requested from the user request (TMS_UR), analyzes stance as programs. the request, and executes the task while interpolating the necessary information in cooperation with the database. • Tags for searching by a requested text In our previous study  [1], we adopted the approach • Set of subtasks named “task information structuring.” In this approach, • Text announced when starting a task a task is defined as a combination of subtasks, which are basic actions shared among different tasks. We imple - If a task needs to refer to environmental information in mented moving, grasping, and handling subtasks, all of the database, such as the destination of a robot in a mov- which are fundamental functions for service robots. In ing task, the type of necessary information is also speci- addition to these subtasks, we implemented new sub- fied in the data structure above. tasks in ROS2-TMS, which control not only robots but A subtask is implemented as an ROS2 node that can be also devices such as speakers, beds, and room lighting. In executed by robots or devices. A subtask is composed of addition, all tasks can be canceled using the ROS2 Action the following contents. protocol. Consequently, ROS2-TMS has the following advan- • Unique ID for each subtask tages compared to ROS-TMS. • Processing implementation • Implementation of the cancelation process • Controllable devices are increased. Various devices, such as speakers, room lighting, and beds, can be Because a task is defined as a set of subtasks, a new task controlled simultaneously to provide service tasks can be represented as a set of existing subtasks; thus, it is with robots (Fig. 8). not necessary to implement the execution code of robots • Since all operations are centrally controlled by the or devices for each task individually. task manager using the ROS2 Action protocol, any task can be terminated or canceled immediately. Method of adding a subtask It is implemented by inheriting the SubtaskNodeBase Two‑layer service structure: tasks and subtasks class, which inherits from the ROS2 Node class and then We defined and provided services using a two-layer overrides the following items: structure in ROS2-TMS, that is, the task and subtask layers. Name: Name of the ROS2 node. A task is defined by a one-to-one correspondence with The ID of the subtask: Assign a number between a service that can be requested in ROS2-TMS. The task 9000 and 9999. Robot subtask Bed subtask 9001 : The robot moves to "Nearly Bed" 9102 : Raise the head side of the bed 9900 : Wait 3 seconds 9300: Announce "Raised the bed" 9200 : Lighting on Connect 9300 : Announce "Turned on the lights " 9300 : Announce "Double has arrived" Lights subtask Speaker subtask 9300: Announce "Good morning" Subtasks “Good morning” task Fig. 8 “Good morning” task consisting of subtasks for various devices I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 9 of 19 Function at runtime: Define the process at runtime. • Raise (9100) or lower (9101) the head and leg sides of Because arguments assigned by the task scheduler the bed. are allocated, it is possible to change the behavior • Raise (9102) or lower (9103) the head side of the bed. according to the arguments. • Raise (9104) or lower (9105) the leg side of the bed. Function when canceled: The action to be taken can • Raise (9106) or lower (9107) the height of the bed. be defined when cancelation is requested. Other subtasks Implementation of subtasks • Turn on (9200) or off (9201) the lights in the room. Subtasks are implemented as the minimum processes • Play a text from the speaker (9300). executed by the robots and various devices. All subtasks • Wait for specified seconds (9900). were implemented as nodes in the ROS2. IDs from 9000 to 9999 were assigned to the subtasks. Currently, 13 sub- tasks are defined in the database. The detailed behaviors Connecting subtasks within a task when executing the subtasks are shown in Table 1. A task is defined as a set of subtasks and stored with string information in the database. The subtask execu - Subtask for communication robot (tms_rc_double) tion token is represented by “subtask_id” or “subtask_id $ ID 9001 is a subtask for a robot, such as the communica- json_format_argument,” and the task is specified by con - tion robot Double 2, to move to a destination. The desti - necting them with the sequential execution token “+” or nation is defined with a position and an orientation and the parallel execution token “|.” Examples of connecting specified when the subtask is called. This movement sub - multiple subtasks using this notation are shown in Fig. 9. task uses the Navigation2 package and a behavior tree to More complex tasks can be represented with the move. Backus-Naur Form (BNF) notation as follows: �task − structure� ::= �task� • Double 2 moves to the destination (9001). �task� ::= �task��task��operator� | �subtask� ′′ ′′ �operator� ::= “ + | “| Subtasks for robotic bed (tms_rc_bed) ′′ IDs in the 9100s are subtasks for a robotic bed. For each �subtask� ::= “subtask − id subtask, the execution time in seconds is defined as an ′′ | “subtask − id$json − arguments argument. Table 1 Definition of the implemented subtasks Name IDs Arguments Behavior Behavior on cancelation Robot movement subtask 9001 Goal position (x, y, z) Requests the Navigation2 package to autonomously move the Cancels the currently Goal angle (w, x, y, z) robot to the target location using ROS2 action communication. If executed request to Navi- the request fails, it informs the upper-level task control node that gation2 by ROS2 action. it has failed. Subtasks for robotic bed Execution seconds Requests the bed control node (websocket operation) in TMS_RC Sends a stop command to perform the respective operation via a websocket, such as rais- to the bed control node ing the bed. If another task has already made the bed request, this via a websocket. subtask informs the upper task control node of the failure. Subtasks for the lights 9200 None An HTTP request is sent to the lighting management server None 9201 located in the big sensor box to turn the lights on and off. Failure is reported to the upper task control node if no response is received from the server within the 5-second timeout period. A subtask for the speaker 9300 Text to be spoken Request the speaker control node in TMS_UR to speak using the None ROS2 service. A subtask for the speaker 9900 Execution seconds Wait for the specified time. None Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 10 of 19 Fig. 9 Task notations �task − structure� represents the connection between the and �property − name� is the property’s name of the data- ′′ subtasks. “subtask − id is the ID of the subtask to be base object. ′′ executed, and “subtask − id$json − arguments is the ID The integration of object information and the task and some arguments. For example, the string command is performed by the task search node in TMS_UR and for raising the height of the bed for 17 s is represented as the task management node in TMS_TS. For example, details of the operation from 3 to 5 shown in the “Task ′′ ′′ 9106${ sec : 17.0} execution flow” section are as follows: This command implies that the subtask to raise the height 1. The task search node in TMS_UR checks the string of the bed (ID: 9106) is executed for 17 s. against the task tag in the database and obtains the task ID and object. Integrate tasks and databases 2. The task search node sends the task ID and objects to The task description in the database can be linked with the task management node in TMS_TS. other objects. 3. The task management node retrieves the task from (�object − type�.�property − name�) the task ID. After that, it replaces the above syntaxes of the startup readout text and subtask structure with The syntax above can be replaced by writing it in the the information of the object. subtask structure or the startup readout property. 4 A task control node is generated, and the task is exe- �object − type� is the type of information in the database, cuted. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 11 of 19 Here is an example of a robot moving task. When a user • Raise or lower the leg side of the bed for a certain says “ROS-TMS, Double, go to the bed,” a task search period. node in TMS_UR retrieves the “Robot movement task” • Raise or lower the height of the bed for a certain and the “Nearly bed” location information from the tags period. in the database (Fig. 7a). Subsequentry, the task manage- ment node in TMS_TS replaces the information in the “Robot movement task” with the contents of the “Nearly Tasks related to the lights (2 types) bed” (Fig. 7b). It then executes the task. • Turn on or off the lights in the room. Method of adding a task Adding a task is performed by storing the following infor- Tasks that combine multiple devices (2 types) mation in the database. • “Good morning” task ID: Set a unique number that does not overlap with • “Good night” task any other object in the database. Name: Give a name to the task content. Type: Set the string “task” to distinguish it from other objects in the database. Task execution Subtask composition: Set the composition of the TMS_TS dynamically allocates a thread to execute the subtasks using the syntax described in the “Con- requested task when the execution of a task is requested necting subtasks within a task” subsection. by TMS_UR (Fig.  10a). The number of threads is set to Required tags and tags: If a word in a required tag n + 1 , where n is the number of tokens “| ” that represent is included in a sentence, this task is searched. If a the parallel execution. word in the tag is included in a sentence, the priority A control node is an ROS2 node that corresponds to of this task is increased in the searched tasks. each thread. Several control nodes are created when tasks Announcement text: An announcement at startup start, and these nodes are connected to each other by and when object integration fails (optional). Set the the ROS2 action protocol in the tree structure. The task text to be spoken by the speaker at startup. The text scheduler represents the root, the control nodes repre- can also be set when the integration with the data- sent the branches, and the subtask execution nodes rep- base fails; for example, when the target point of the resent the leaves. Action communication in ROS2  [14] robot’s movement task does not exist in the database. handles accept or reject requests, and if accepted, the status of the action is returned when the action is con- cluded, such as succeeded, failed, or canceled. In addi- tion, a cancelation request can be accepted while the Implementation of tasks action is being executed. Currently, there are 13 types of tasks. Tasks exist in a database, and their behavior can be based on a single Cancel function subtask or a combination of subtasks. In ROS-TMS, a user cannot stop a task in progress. To implement this cancelation function, an update to the A task related to the communication robot (1 type) action function in ROS2 is required. ROS action commu- nication consists of two types of entities: clients and serv- • Double 2 moves to the destination. ers. However, the server could not recognize which client requested it. In ROS2, when a client requests the server, Currently, there are 3 target locations in the database: the client generates a UUID [14]. This UUID is used as an near the entrance, near the kitchen, and the bed, and identifier of the request between the server and the cli - Double2 can move to them. ent to manage each request individually. This new feature was used to map tasks to subtasks in the task scheduler. Tasks related to the robotic bed (8 types) In ROS2-TMS, even when a task is running, the task can be stopped immediately by receiving a “cancel” voice • Raise or lower the head and leg sides of the bed for a request issued by the user or other cancelation com- certain period. mands. The cancel operation is realized by sending a • Raise or lower the head side of the bed for a certain cancel command to the subtasks currently running. It is period. an emergency stop function, and there is currently no Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 12 of 19 further serial operations of subtasks are terminated Run in thread-1 + (Fig. 10b, c). Error termination If the subtask is not completed or canceled, its subtask node informs the task search node in the TMS_UR of Run in thread-2 | + the failure through the requested task control node or its upper-level task management node. When the task Run in thread-3 search node is informed of the failure, it sends a mes- 9001 9300 sage to the speaker that the task has failed or has been canceled. 9102 9300 9300 + : sequential execution Tasks that combine subtasks 9900 9200 | : parallel execution The task scheduler in ROS2-TMS can manage not only (a) Threading robots but also IoT devices such as beds, lights, and microphones, and ROS2-TMS can provide various ser- vices by combining robots and intelligent home appli- Run in thread-1 ances. Examples of services that are newly added to the ROS2-TMS, such as the “Good morning” and “Good night” tasks, are shown in Fig.  11. Both tasks combine the subtasks for the communication robot Double 2, a robotic bed, lighting, and speaker. For example, the Run in thread-2 | + “Good morning” task is defined as shown in Fig. 12. As shown above, these tasks consist of sequential and Successful Run in thread-3 9001 9300 termination parallel executions of several subtasks and are highly In operation complicated. We confirmed that the cancel function is 9102 9300 9300 appropriately executed even for these complex tasks. In operation + : sequential execution 9900 9200 | : parallel execution Service experiments using ROS2‑TMS and big sensor box (b)Inoperation We conducted an experiment to verify the operation of services with a voice interface using ROS2-TMS in Big Run in thread-1 Sensor Box (Fig. 2). The big sensor box is a hardware platform for an infor - mationally structured environment. As described in [1], it is located on the second floor of the COI building at the Ito campus of Kyushu University and is a habitable Run in thread-2 | + space with a living room, bedroom, and kitchen. In the room, there is a service robot, such as Double 2 and Dou- Successful Run in thread-3 9001 9300 + termination ble 3, and a motion capture system that can acquire the Canceled robot’s position, a robotic bed, and a server that manages 9102 9300 9300 the room lighting. In this study, we interconnected them using ROS2 communication and experimented with pro- + : sequential execution 9900 9200 | : parallel execution viding services using ROS2-TMS. We confirmed that ROS2-TMS can launch proper tasks (c) Canceling according to a user’s voice request through the micro- Fig. 10 Thread allocation for “Good morning” task phone device and perform tasks such as user interaction (daily conversation), bed operations, lighting control, and robot motion by combining the proper information in function to resume a task that has been stopped in the the database. middle. The following subsections show how each task was per - Even if the execution time is specified when start - formed. Detailed timecharts of each task are shown in ing subtasks, the execution is stopped immediately, and Figs. 13 and 14. I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 13 of 19 high is Mount Fuji?,” proper responses were provided by Google Assistant (Fig. 15). Lighting control 9001:The robot movesto"Nearly Bed" When a user requests tasks related to the lighting such as “ROS-TMS, turn on the lights” or “ROS-TMS, turn off the 9102:Raisethe head side of the bed 9900:Wait3 seconds lights,” ROS2-TMS chooses and executes the proper tasks by sending commands to the lighting management server in Big Sensor Box (Fig. 16). 9300:Announce "Raised thebed" 9200: Lighting on Robotic bed control 9300:Announce"Turned on thelights" 9300:Announce"Double hasarrived" When a user requests tasks related to a robotic bed, such as “ROS-TMS, raise the height of the bed,” ROS2-TMS exe- cutes the proper tasks to control the robotic bed (Fig. 17a). 9300:Announce "Goodmorning" Robot control When the user requests “ROS-TMS, Double, go to the kitchen,” TMS_TS in ROS2-TMS generates a motion plan for the communication robot Double 2 to move to the (a) “Good morning” task kitchen. The coordinates of the kitchen are stored in the database and interpolated to generate motion commands in TMS_TS. In addition, if the user requests “ROS-TMS, Double, go to the bed,” Double 2 moves to the vicinity of the bed by linking with the bed coordinates stored in the 9001:The robotmoves to "Kitchen" database. (Fig. 18a, b). 9300: Announce "Lay down thebed"9103: Lower theheadsideofthe bed Execution of complex tasks We implemented the “Good morning” and “Good night” 9300:Announce "Double hasreturned to thekitchen" tasks to confirm the sequential and parallel executions of subtasks. Both tasks consisted of several subtasks related 9300: "Turnoff the lights"9201:Lightingoff to robot motion, bed operation, and room lighting control connected sequentially or in parallel. To confirm the flex - ibility of the task representation, we implemented these tasks with several different connections of subtasks and 9300:Announce"Good night" confirmed that these tasks were executed appropriately even with complex task structures. “Good morning” task (b) “Good night” task The “Good morning” task performs the following three Fig. 11 Example of tasks combining subtasks subtasks in parallel. • Raise the upper body of the bed and announce “Raised Daily conversation the bed” from the speaker. When a user asks ROS2-TMS questions such as “ROS- • Wait for 3 s, turn on the lights, and announce “Turned TMS, What’s the weather today?” or “ROS-TMS, How on the lights.” 9102${“sec”:20.0} 9300${“announce”: “Raised the bed”} +9900${“wait sec”: 3.0} 9200 +9300${“announce”: “Turned on the lights”} + | 9001${“position”: [8.65,1.62, 0.0],“orientation”: [0.0, 0.0, 0.280, 0.960]} 9300${“announce”: “Doublehas arrived”} + | 9300${“announce”: “Goodmorning”} + Fig. 12 Task notation of “Good morning” task Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 14 of 19 (a) “Weather forecast” time chart. User 0: Askthe weatherto microphone.Speaker 1: Answer theweather. (b) “Switching off the light” time chart. User 0: “ROS-TMS,turnoffthe lights”. Speaker 1: Announcementat task startup. Light 2: Turned off. (c) “Lower the head side of thebed”timechart. User 0: “ROS-TMS,lower thehead sideofthe bed”. Speaker 1: Announcementat task startup. Bed 2: Lowered thehead side. (d) “Canceling bed control task”timechart. User 0: “ROS-TMS,lower thehead side of thebed”. Speaker 1: Announcementat task startup. Bed 2: Loweredthe head side. User 3: “Cancel”. Speaker 4: Announcement at task cancelation. (e) “Double goes to the kitchen”timechart. User 0: “ROS-TMS, Double, go to thekitchen”. Speaker 1: Announcementat task startup.Robot 2: Went to thekitchen. (f) “Double goes to thebed”time chart. User 0: “ROS-TMS, Double, go to thebed”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. (g) “Canceling move task” time chart. User 0: “ROS-TMS, Double,gotothe bed”.Speaker 1: Announcement at task startup. Robot 2: Went to thenearsideofthe bed. User 3: “Cancel”. Speaker 4: Announcementattask cancelation. Fig. 13 Time chart of task execution (1 of 2) I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 15 of 19 (a) Execute “Good morning” task time chart. User 0: “ROS-TMS,goodmorning”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. Speaker Robot 3: Announced“Doublehas arrived”.Bed 4: Raisethe head sideofthe bed. Speaker Bed 5: Announced “Raised thebed”. Light 6: Turned on.Speaker Light 7: Announced “Turnedonthe lights”. Speaker GoodMorning 8: Announced “Good morning”. (b) Canceling “Good morning” task time chart. User 0: “ROS-TMS,goodmorning”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe bed. Bed 4: Raisethe head side of thebed.Light 6: Turned on.Speaker Light 7: Announced“Turned on the lights”. User Cancel 9: “Cancel”.Speaker Cancel 10: An- nouncementat task cancelation. (c) Execute “Good night” task time chart. User 0: “ROS-TMS,goodnight”. Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe kitchen.Speaker Robot 3: Announced“Doublehas returned to the kitchen”. Speaker Bed 4: Announced “Laydownthe bed”. Bed 5: Loweredthe head sideofthe bed. Speaker Light 6: Announced “Turnoffthe lights”. Light 7: Turned off. Speaker GoodNight 8: Announced “Goodnight”. (d) Canceling “Good night” task time chart. User 0: “ROS-TMS, good night”.Speaker 1: Announcement at task startup. Robot 2: Went to thenear sideofthe kitchen. Speaker Bed 4: Announced “Lay down thebed”. Bed 5: Lowered thehead sideofthe bed. User Cancel 9: “Cancel”. Speaker Cancel 10: Announcement at task cancelation. Fig. 14 Time chart of task execution (2 of 2) Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 16 of 19 User System User System Today in Fukuoka, it’ll be sunny, ROS-TMS, What’s the weather today? ROS-TMS, Double, go to the kitchen. Double goes to the kitchen. with a forecast high of 13 and a low of 2. 1 2 1 2 3 (a) Doublegoestothe kitchen Fig. 15 Weather forecast User System ROS-TMS, Double, go to the bed. Double goes to the bed. 1 2 3 User System ROS-TMS, turn off the lights. Turn off the lights. 1 2 (b) Doublegoestothe bed User System User System ROS-TMS, Double, Double goes to the Canceled the Cancel. go to the bed. bed. task. Fig. 16 Switching off the light 13 2 Stop moving midway. User System (c) Canceling move task ROS-TMS, lower the head side Lower the head side of the bed. of the bed. Fig. 18 Move task for communication robot 1 2 “Good night” task The “Good night” task executes two subtasks in parallel. In addition, there is a further parallel operation within the first operation. The following two operations are exe - (a) Lowerthe head sideofthe bed cuted in parallel. User System User System • A task lays the head side of the bed down and ROS-TMS, lower the Lower the head Canceled the Cancel. head side of the bed. side of the bed. task. announces “Lay down the bed” from the speaker and then turns off the lights in the room and announces 13 2 “Turn off the lights.” • After Double 2 returns to the kitchen, the speaker announces, “Double has returned to the kitchen.” Stop lowering the bed. After completing all the subtasks, “Good night” is (b) Cancelingbed control task announced from the speaker (Fig. 20a). Fig. 17 Bed control task Cancel tasks If the word “cancel” is included in the user’s command, TMS_TS sends a cancel command to the control nodes • Move the communication robot Double 2 close to in all subtasks executing currently. When the control the bed and announce “Double has arrived.” node in the subtask receives the cancel command, the control node sends the cancel command to the subtasks After completing all the subtasks, “Good morning” is and the control nodes of the child subtasks executing announced from the speaker (Fig. 19a). I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 17 of 19 User System System User System System ROS-TMS, good Start the “Good morning” Turned on the lights. ROS-TMS, good night. Start the “Good night” task. Lay down the bed. morning. task. 1 2 1 2 System System System System System System Double has returned Turn off the lights. Raised the bed. Double has arrived. Good morning! Good night! to the kitchen. 3 4 3 4 (a) Execute “Goodmorning” task (a) Execute “Goodnight”task User System System User System System ROS-TMS, good Start the “Good morning” Turned on the lights. ROS-TMS, good night. Start the “Good night” task. Lay down the bed. morning. task. 1 2 1 2 User System User System Cancel. Canceled the task. Cancel. Canceled the task. 3 4 3 4 (b) Canceling “Good morning” task (b)Canceling“Good night” task Fig. 19 “Good morning” task Fig. 20 “Good night” task in parallel. When all the control nodes in the sub- • The middleware was changed from ROS to ROS2, tasks receive the cancel command, the task is stopped and various modules were ported and newly devel- (Figs. 17b, c, 19b, and 20b). oped. Currently, ROS2-TMS does not actively use secu- Conclusions rity and QoS settings. However, as we continue our In this study, we presented a new software platform for research, QoS will used to configure sensor streams, an ISE, named ROS2-TMS, and demonstrated a few such as cameras, which have a large amount of data. robot services using ROS2-TMS. ROS2-TMS is publicly For the robot movement subtask, we used Naviga- available and can be downloaded from [20]. The differ - tion2 and the behavior tree. The application of ROS2- ences between ROS2-TMS and the previous ROS-TMS TMS with this behavior tree is more flexible than system are summarized as follows. ROS-TMS for moving in complex environments. Itsuka et al. ROBOMECH Journal (2022) 9:1 Page 18 of 19 Acknowledgements • The interaction with a user was realized using the This work was partially supported by the Cabinet Office (CAO) Cross-Ministe - Google Assistant API. rial Strategic Innovation Promotion Program (SIP), “An intelligent knowledge • The task scheduler was extended to manage not only processing infrastructure, integrating physical and virtual domains” (funding agency: NEDO). robots but also various IoT devices such as intelligent beds, lighting, and speakers. This makes it possible to Authors’ contributions provide various services that combine robots and IoT TI drafted the manuscript. RK constructed the study concept. AK and RK man- aged the study. TI and MS developed the system and carried out the experi- devices. ments. All members verified the content of their contributions. All authors • To improve safety, we added a new function that read and approved the final manuscript. allows the user to stop running tasks by issuing a Funding cancelation request to the ROS nodes using the This work was partially supported by the Cabinet Office (CAO), Cross-Ministe - ROS2 Action protocol. rial Strategic Innovation Promotion Program (SIP), “An intelligent knowledge processing infrastructure, integrating physical and virtual domains” (funding agency: NEDO). Future works includes the following improvements. Availability of data and materials Not applicable. Low‑cost localization system In this study, an optical motion capture system was used Declarations to estimate the position of the robot. Although this sys- Competing interests tem can identify the robot position with high accuracy, The authors declare that they have no competing interests. it is considerably expensive. Currently, we are testing less expensive range-based localization systems such as Received: 25 September 2021 Accepted: 28 December 2021 Wi-Fi, RFID, UWB, and visible light  [21], and we believe that low-cost autonomous mobile services can be pro- vided by applying these technologies. References 1. Sakamoto J, Kiyoyama K, Matsumoto K, Pyo Y, Kawamura A, Kurazume R (2018) Development of ros-tms 5.0 for informationally structured environ- Services by health status ment. ROBOMECH J 5:24. https:// doi. org/ 10. 1186/ s40648- 018- 0123-9 In this study, we developed a module for a wearable heart 2. ROS.org | Powering the world’s robots. https:// www. ros. org/. Accessed 3 rate sensor. We will develop a service robot that recog- Aug 2021 3. ROS 2 Documentation: Foxy. https:// docs. ros. org/ en/ foxy/ index. html. nizes not only the human’s location but also the health Accessed 30 July 2021 condition using environmental sensors as a state analyzer 4. Macenski S, Martin F, White R, Clavero JG (2020) The Marathon 2: a naviga- module (TMS_SA, in Fig. 3) and provides proper services tion system. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 2718–2725. https:// doi. org/ 10. 1109/ IROS4 according to the health status of the individual. 5743. 2020. 93412 07. https:// github. com/ ros- plann ing/ navig ation2https:// ieeex plore. ieee. org/ docum ent/ 93412 07/ 5. Sato T, Nishida Y, Mizoguchi H (1996) Robotic room: symbiosis with Dynamic planning of tasks, such as autonomous robot human through behavior media. Robot Auton Syst 18(1–2):185–194. movement https:// doi. org/ 10. 1016/ 0921- 8890(96) 00004-8 6. Brooks RA (1997) The intelligent room project. In: Proceedings of the 2nd The autonomous movement of robots in large areas, International Conference on Cognitive Technology (CT ’97). CT ’97, p 271. including doors or elevators, requires coordination with IEEE Computer Society, USA surrounding actuators  [12, 22]. Therefore, even for the 7. Park KH, Bien Z, Lee JJ, Kim BK, Lim JT, Kim JO, Lee H, Stefanov DH, Kim DJ, Jung JW, Do JH, Seo KH, Kim CH, Song WG, Lee WJ (2007) Robotic same autonomous movement, the composition of the smart house to assist people with movement disabilities. Auton Robots subtasks differ significantly between movement in a 22(2):183–198. https:// doi. org/ 10. 1007/ s10514- 006- 9012-9 room and movement in a large area across rooms. For 8. Amazon Alexa Voice AI | Alexa Developer Official Site. https:// devel oper. amazon. com/ en- US/ alexa. Accessed 23 Aug 2021 such environment-dependent tasks, we will extend the 9. Google Assistant. https:// assis tant. google. com/. Accessed 23 Aug 2021 task scheduler so that the necessary functions can be 10. Kwak SS, San Kim J, Moon BJ, Kang D, Choi J (2020) Robots versus dynamically linked. speakers: what type of central smart home interface consumers prefer? In: 2020 IEEE/RSJ International Conference on Intelligent Robots and In addition, when several tasks are executed simultane- Systems (IROS), pp 11397–11404. https:// doi. org/ 10. 1109/ IROS4 5743. ously, it is necessary to monitor and appropriately control 2020. 93417 48. https:// ieeex plore. ieee. org/ docum ent/ 93417 48/ the progress of all the tasks currently running. Thus, we 11. Melo N, Lee J, Suzuki R (2018) Identification of the User’s Habits based on Activity Information. In: 2018 IEEE/RSJ International Conference on will design an efficient interface to present detailed and Intelligent Robots and Systems (IROS), pp 2014–2019. https:// doi. org/ 10. comprehensive information, including the task status, 1109/ IROS. 2018. 85938 73. https:// ieeex plore. ieee. org/ docum ent/ 85938 location of humans and robots, and human conditions. 73/ I tsuka et al. ROBOMECH Journal (2022) 9:1 Page 19 of 19 12. Cavallo F, Limosani R, Manzi A, Bonaccorsi M, Esposito R, Di Rocco M, Pecora F, Teti G, Saffiotti A, Dario P (2014) Development of a socially believable multi-robot solution from town to home. Cognit Comput 6(4):954–967. https:// doi. org/ 10. 1007/ s12559- 014- 9290-z 13. Simoens P, Dragone M, Saffiotti A (2018) The internet of robotic things: a review of concept, added value and applications. Int J Adv Robot Syst 15(1):1–11. https:// doi. org/ 10. 1177/ 17298 81418 759424 14. ROS2 Design | Actions. http:// design. ros2. org/ artic les/ actio ns. html. Accessed 4 Feb 2020 15. Double Robotics - Telepresence Robot for Telecommuters. https:// www. doubl erobo tics. com/ doubl e2. html. Accessed 2 Feb 2020 16. Double Robotics—Telepresence Robot for the Hybrid Office. https:// www. doubl erobo tics. com/ doubl e3. html. Accessed 15 Sept 2021 17. Overview | Google Assistant SDK | Google Developers. https:// devel opers. google. com/ assis tant/ sdk/ overv iew. Accessed 2 Feb 2020 18. Julius. https:// julius. osdn. jp/. Accessed 14 Sept 2021 19. Janome v0.3 documentation ( ja). https:// mocob eta. github. io/ janome/. Accessed 2 Feb 2020 20. https:// github. com/ irvs/ ros2_ tms. Accessed 16 Sept 2021 21. Zafari F, Gkelias A, Leung KK (2019) A survey of indoor localization sys- tems and technologies. IEEE Communications Surveys and Tutorials 21(3), 2568–2599. https:// doi. org/ 10. 1109/ COMST. 2019. 29115 58. arxiv: 1709. 22. Martin F, Gines J, Vargas D, Rodraguez-Lera FJ, Matellan V (2018) Planning topological navigation for complex indoor environments. In: 2018 IEEE/ RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1–9. https:// doi. org/ 10. 1109/ IROS. 2018. 85940 38https:// ieeex plore. ieee. org/ docum ent/ 85940 38/ Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations.

Journal

ROBOMECH JournalSpringer Journals

Published: Jan 31, 2022

Keywords: Service robot; Informationally structured environment; Internet of things; Cyber physical system; Ambient sensing

There are no references for this article.