Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Rapid prototyping for series of tasks in atypical environment: robotic system with reliable program-based and flexible learning-based approaches

Rapid prototyping for series of tasks in atypical environment: robotic system with reliable... We propose a novel robotic system that combines both a reliable programming-based approach and a highly generalizable learning-based approach. How to design and implement a series of tasks in an atypical environment is a challenging issue. If all tasks are implemented using a programming-based approach, the development costs will be huge. However, if a learning-based approach is used, reliability is an issue. In this paper, we propose novel design guidelines that focus on the respective advantages of programming-based and learning-based approaches and select them so that they complement each other. We use a program-based approach for motions that is rough behavior and a learning-based approach for motion that is required complex interaction between robot and object of robot tasks and are difficult to achieve with a program. Our learning approach can easily and rapidly accomplish a series of tasks consisting of various motions because it does not require a computational model of an object to be designed in advance. We demonstrate a series of tasks in which randomly arranged parts are assembled using an actual robot. Keywords: Autonomous robot, Assembly, Motion generation, Deep predictive learning Introduction in unexpected situations. In factory automation, it is also Robotic automation has traditionally been used mainly desirable for robots to work autonomously in a new work for repetitive tasks in known environments, such as environment without humans having to teach them oper- automotive assembly. However, in recent years, there ations in detail since a quick response to frequent process has been a growing need to automate atypical tasks in changes is required for small-volume production. How- unknown or undeveloped environments that are difficult ever, there are several issues that need to be addressed in to perform with conventional robotic technology. For order to design a robot system that can perform atypical example, shortages of workers are becoming more seri- tasks. ous at construction and maintenance sites where danger- Big issues for design a robot system for perform series ous and heavy work is required. For robots to perform of task in atypical tasks is the implementation cost. At on-site work, the emphasis is not on the ability to repeat atypical work sites, robots need to be adaptive to the simple tasks at high speed and with high precision as in conditions of work objects and the positioning of parts. the past but on the ability to perform tasks reliably even To execute a series of tasks autonomously, three actions must be done: (1) recognizing the work situation and planning a procedure, (2) recognizing the position, *Correspondence: hiroshi.ito.ws@hitachi.com shape, and posture of parts, tools, etc., and (3) controlling Center for Technology Innovation - Controls and Robotics, Research & the motion of the robot. However, if we were to imple- Development Group, Hitachi, Ltd., Ibaraki 312-0025, Japan ment these functions for each task, the development Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 2 of 14 costs would be huge. In addition, a high level of exper- and actions that can be realized with relatively sim- tise is required to developtasks that involve contact with ple definitions were defined as program-based ones, an object (screw fastening and fitting) or are difficult to while complex actions were defined as learning- describe in a program (handling flexible objects). There - based ones. For example, a program-based approach fore, learning-based approaches have been proposed to is used for operations such as reaching, where the accomplish complex tasks that are difficult to accomplish robot moves dynamically, and safety and reliabil- with conventional programming-based approaches [1–3]. ity are required, and a learning approach is used for However, learning-based approaches have safety and operations such as screw fastening and fitting, where reliability issues. Unlike image recognition and natural the robot’s work area is limited, and the operations in language processing, robots perform tasks that involve this area are difficult to accomplish with a program. physical contact in the real world. This can lead to colli - The system designer can easily and rapidly realize a sions and other dangers. Therefore, it is necessary to have robotic system in which programming and learning both “safety” to prevent failure and damage as a result of approaches coexist by implementing various func- physical contact and collision and “reliability” to ensure tions on the basis of the design guidelines presented that behavioral rules are improved through learning. For in this paper. the former, safety has been ensured by detecting abnor- 2. Realizing a series of tasks in an atypical environment: malities and loads of robots by analyzing signals such as The robot can flexibly generate motions even in an current values, acoustic sensors, and acceleration sen- atypical environment where the placement of parts, sors [4–7]. However, the learning approach is a black box work areas, and tools change, and a series of assem- because it acquires behavioral rules on the basis of data, bly tasks can be accomplished with an 80% success and there are still problems with reliability. Some meth- rate. In the experiment, the alignment of each part is ods have been proposed to use a model-based controller important because the assembly work is performed as prior knowledge for deep reinforcement learning [8, using parts placed in random positions. Our learn- 9] or to monitor the state of a system in real time dur- ing-based approach allows the robot to acquire the ing operation generation and execute a predetermined desired operation by being taught the motion using recovery action when an anomaly is detected [10–12]. teleoperation and learning models. Therefore, there However, none of these methods focus on the reliability is no need to design a computational model of an of behavior. object in advance. During motion execution, the In this paper, we aim to accomplish complex tasks in robot can perform screw fastening and object align- an atypical environment in a simple and rapid way that is ment on the basis of its sensor information in real both reliable and flexible. How to design and implement time. a set of tasks in an atypical environment is a challenging issue. If we try to implement all tasks using a program- ming-based approach, the implementation costs will be Design concept huge. However, if a learning-based approach is used, reli- How to design and implement a series of tasks in an ability becomes an issue. In this paper, we propose a novel atypical environment is a challenging issue. If all the robotic system that focuses on the respective advantages tasks are implemented using a programming approach, of reliable program-based and flexible learning-based the costs of development will be huge, and if a learning- motion generation, which complement each other. To based approach is used, the data collection cost, learning verify the effectiveness of the proposed method, as a cost, and reliability will become issues. In this section, first step for an atypical environment task, an assembly we focus on the advantages of both reliable program- task with randomly placed tools and parts is performed ming-based and flexible learning-based methods, and we with a real robot. This task is very challenging compared describe how to design robotic systems that complement with general assembly tasks because it requires trajectory each other’s shortcomings. planning based on the positions of randomly placed parts and tools, as well as their positioning and screw fasten- Task categorization ing. The contributions of this paper are as follows. The issue is whether a programming-based or learning- based approach is more suitable for implementing each 1. Categorization of robotic tasks based on the therblig: function of a robot and how best to combine them. As a A method is proposed for designing a robotic sys- solution to this problem, in this paper, we define a clas - tem using therbligs that visualizes and improves the sification of robot functions and the execution order of efficiency of human tasks. The robot’s various tasks each function with reference to the therblig. Therbligs (e.g., grasping, carrying, fitting) were categorized, are 18 kinds of elemental motions that a worker needs I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 3 of 14 in order to perform a task [13, 14]. In the field of indus - • Disassemble: Separating multiple components that trial engineering, therbligs are used to visualize the are joined. Loosen the screws and disassemble the content and series of human work and to propose and parts. more efficient work methods and improve them. Here, • Plan: Planning to do this next or later. Think about we describe how to design a robotic system on the basis the procedure of the task. of the therblig. First, we extracted 12 kinds of elemental motions Implementing all these 12 kinds of elemental motions for robotic system design while excluding six elemen- with a programming-based approach would lead to tal motions that should be unnecessary for design the huge development costs. Therefore, we propose a design robot tasks, these are stopped motion elements that method (guidelines) for a robot system with reliability is no longer needed for work (i.e. “hold”, “unavoidable and versatility by classifying the 12 motions into three delay”, “avoidable delay”, “rest”), “pre-position” element categories on the basis of the characteristics of each func- that become unnecessary by optimizing in advance and tion and then implementing them as modules. Table 1(a) “find” element that is almost same as “search” and sel - shows three categorized elemental motions, and (b) dom use. Each elemental motion is explained using the shows the corresponding robot functions. (1)–(3) Show screw handling in assembly tasks as an example. the three categories, and their details are as follows. (1) “Perception” is a motion to judge or measure without • Search: Searching the location of an object with the actual robot action based on information such as vision, eyes or hands. The five senses other than the eyes, and corresponds to functions such as image recogni- such as when searching for holes with the fingertips, tion, object detection, and position estimation based on are also equivalent. Find out where the screws are. the robot’s visual information. It is also used for check- • Inspect: Determining an object by a defined standard ing the task status (progress) and motion planning based (quality or characteristics) using visual, auditory, tac- on visual information. Since deep learning has made tile, etc. Check how tightly screws have been tight- recognition technologies more diverse and accurate, ened between parts. it is possible to choose a method of implementation in • Select: Choosing one out of several. Select an appro- accordance with the task. Template matching [15], which priate screw from several screws. is conventionally used, is effective when the number of • Grasp: Grasping an object with hands or fingers. target objects is limited or patterned, while learning- Grab the screw. based object recognition algorithms [16–18] are effec - • Release load: Releasing control of an object. Release tive for various types and complex shapes of objects. the screw. (2) “Motion” is relatively simple motion that have not • Transport empty: Reaching an object with an empty complex interactions between robot and object of robot hand. Reach out to where the screw is. tasks, and it also requires a high level of reliability and • Transport load: Moving an object in hand from one safety because the robot moves widely in the real world. place to another. Carry the screw. Many of the motions are somewhat patterned, such as • Position: Adjusting the position and orientation of “take object A and place it at position B.” Teaching play- objects. Adjust the position to insert the screw into back, point-to-point (PtP), and trajectory planning algo- the hole. rithms such as rapidly-exploring random tree (RRT) [19, • Use: Manipulating tools, instruments, and equip- 20] are examples of implementation methods. (3) “Sen- ment for a purpose. Tighten the screws. sorimotor” is a motion that is necessary to have complex • Assemble: Joining two parts together. Use the screws interactions between robot and object of robot task, and to fix the two parts together. relies on the five senses and adjusts the position and force Table 1 Categorization of robot functions based on therbligs (1) Perception (2) Motion (3) Sensorimotor (a) Basic motion Search Grasp Position Inspect Release load Use Select Transport empty Assemble Plan Transport load Disassemble (b) Key function of robot control Image recognition Point-to-point motion Learning-based Object detection Interference avoidance motion generation Position estimation Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 4 of 14 in accordance with the situation. For tasks that are dif- to detect the position and state labels of objects. (2) The ficult to describe in a program, such as screw tightening process manager switches tasks using real-time process and fitting, a learning-based motion generation method planning methods [25, 26] or pre-designed if-then rules. would be effective. The diversification and develop - Next, we describe the series in which tasks are executed ment of learning-based approaches makes it possible to as shown in Fig. 1c. In this paper, a task consists of up to perform complex tasks [21–24]. Deep reinforcement three kinds of modules. A typical robotic system consists learning, imitation learning, and other implementation of “perception” and “motion planning and control.” It rec- methods are examples of implementation methods. ognizes object names and locations on the basis of vision We have described the categorization of robot func- and executes trajectories that satisfy geometric, mecha- tions and examples of their implementation, referring to nistic, and dynamic constraints on the basis of the rec- the therblig. In the next section, we describe the execu- ognition results. If we apply this to the categorization in tion procedure and how to combine the three functions. Table 1, the task will transition in the order of “perception module” and “motion module.” However, not all tasks can System architecture be realized with a program-based motion module. Some When a robot is performing a task, the timing at which tasks define the “motion module” to be followed by the tasks are switched and the order in which the motion and “sensorimotor module.” Basically, a programming-based sensorimotor are executed are issues. Figure 1 shows the approach is used for rough motions that are defined as execution procedure (system configuration) for the series “Motion” in Table  1, and a learning-based approach is of tasks proposed in this paper. The squares in the figure used for complex motions that are defined as “Sensori - represent each of the functions defined above (hereafter motor” in Table  1. u Th s, the number of modules (sub - referred to as modules). The modules consist of a “per - tasks) varies depending on the task. For example, Task 1 ception module” for recognizing situations, a “process in Fig.  1 consists of subtask 1 to pick up the object and manager” for switching tasks, and a “motion module” Subtask 2 to place it at the target position. Subtask 1 gen- and a “sensorimotor module” for executing subtasks. erates a grasping motion with the motion module on the On a simple production line, each task is executed in a basis of the position information of the target object rec- feed-forward sequence. However, with a pre-planned ognized by the perception module. Similarly, subtask 2 work schedule, it is not possible to respond to unex- generates a motion to place the object at the goal position pected events. In particular, since this paper assumes with the motion module on the basis of the goal posi- an atypical environment where the positions of objects tion information recognized by the perception module. change, process control in a feed-forward manner is dif- However, if positioning accuracy is required, such as in ficult. To manage work processes in an atypical environ - assembly work, it is impossible to complete a series of ment, it is necessary to recognize where objects are and tasks by simply placing the object at the goal position what state they are in on the basis of vision information. and it is required to adjust position according to state of In this paper, we use the perception module of Fig.  1 (1) each parts that would be assembled, such as adjustment Fig. 1 Software architecture for performing series of tasks I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 5 of 14 motion with vision sensing or searching motion with robot arms can be remotely controlled using a joystick force feedback. In addition, implementing a high-preci- to teach the desired motion. Impedance control can also sion positioning algorithm using a programming-based be performed by using a torque sensor at each joint. For approach requires complex programming and real-world example, even if the axis of the driver and the screw hole tuning. Therefore, we use the sensorimotor module to is misaligned when tightening a screw and the hand is adjust the position after the motion module. This makes overloaded, it can prevent the screw and screw hole it possible to perform complex tasks robustly with lower from being overloaded by passively moving the hand. development costs. However, since there is an inverse relationship between As described above, by defining (implementing) vari - arm strength and positioning accuracy with impedance ous functions and execution procedures for the robot on control, and the arm does not move accurately in rela- the basis of the task categories and design policies pro- tion to the hand position command, the control mode is posed in this paper, a robot system can be constructed in switched depending on the task. In this paper, we used which a highly reliable program base and a highly robust position control mode when executing the motion mod- learning base coexist. ule and impedance control mode when executing the sensorimotor module, which involves contact with the Set up for verification object. To verify the effectiveness of the proposed method, we performed an assembly task in an atypical environment. Assembly task for verification Here, we describe the experimental setup and the experi- Recently, robots are required to have the ability to handle mental task. high-mix low-volume production, rather than the con- ventional ability to repeat simple tasks at high speed and Robotic hardware with high precision. These robots must have the ability to Figure  2 shows the experimental setup. As shown in (a), work autonomously in atypical environments. As men- two robot arms (KUKA LBR iiwa 14 R820) are used, and tioned above, in this paper, we focus on assembly tasks a robotic hand (Robotiq 2F-85 Adaptive Gripper) and in the atypical environment shown in Fig.  3. To evaluate an RGB camera (Buffalo BSW500M Series) are attached such tasks in this environment, the parts and workbench to the end of the arms. The other equipment includes a are not fixed but are randomly placed within the robot’s bird’s-eye-view camera (Intel RealSenes D435i) for tak- operating range before work is started. The assembly ing full view of work place, and an electric screwdriver task consists of five steps: (1) pick and place a randomly connected to a tool balancer. The workbench, cover, placed base, (2) pick and place a randomly placed cover, base, and screws are the parts needed for assembly. (b) (3) grasp the electric screwdriver, (4) pick up a screw Shows an enlarged view near the robotic hand when (attach it to the screw bit with the magnetic bit on the tip hand grasping the screw driver. As shown in (b), hand of the electric driver), and (5) screw the four corners of camera installed near base of robotic hand and it can the cover to the base. To avoid contact between the robot take an image of near the finger of robotic hand. The and floor, the assembly is performed on a workbench. We Fig. 2 Experimental environment: sensors and equipment Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 6 of 14 Process manager We used the Single Shot Multibox Detector (SSD) [17] as an object recognition algorithm based on deep learning to recognize the work process. To reduce the implemen- tation costs, we used the TensorFlow Object Detection API [27]. The robot performed rule-based task switch - ing on the basis of object labels recognized by SSD. For example, when two parts are to be assembled, the name of each part is output as an object label before assem- bly, and the states are combined after assembly. If a part name is output, the corresponding action (motion or/ and sensorimotor module) will be performed; if a state is output, the next task will be performed. u Th s, a series of tasks can be accomplished by combining object recog- Fig. 3 Assembly tasks in atypical environment. Each part and tool are nition and rule-based process planning. General object randomly placed recognition algorithms, such as template matching, have low robustness to illumination changes. In particular, it is difficult to accurately recognize metals such as assembly evaluated the success rate of the proposed method by parts because the way an object looks depends on how performing a series of assembly tasks 10 times. In these the light hits it (reflection angle). In comparison, SSD tasks, some complex behavior that needs interaction can perform object detection in real time and robustly between robot and object for assembly task is required, against illumination and background changes. Further- for example, align two parts while considering position more, it can simultaneously predict location information and orientation of each parts so that the two parts over- in addition to object label information. We can share the lap exactly, or adjust position and force of the hand to fit modules and reduce development costs by passing the the screw to the tip of the screwdriver. label information predicted by SSD to the process man- ager and the location information to the motion module. System configuration In this section, we describe the specific method to con - figure the robot system with classification of the experi - Perception module mental tasks and the specific implementation method. Since SSD detects the positional information of objects Table  2 shows the tasks and their execution procedures. in the two-dimensional plane of a camera image, it can- The assembly task consisted of five tasks, which were not be used directly for robot control. For the robot to executed in order from 1 to 5. The numbers in each task perform tasks on the basis of location information from correspond to the numbers in Fig.  3. The subtasks were SSD, the information is converted into 3D coordinates executed in order from A to C. The implementation of the real environment. The camera used for perception method for each function is described below. in this paper is bird’s-eye-view camera that is shown in Table 2 Task categorization for assembly work Task Sub-task (A) Perception (B) Motion (C) Sensorimotor (1) Base operation Picking Base Pick up base – Placing Workbench Place base on workbench – (2) Cover operation Picking Cover Pick up cover – Moving Workbench Layer cover over base Alignment adjustment Placing Workbench Place cover on base – (3) Take screwdriver Grasping Screwdriver Reach for screwdriver Position adjustment Moving Screw holder Move to screw holder – (4) Pick up screw Moving Screw – Approach screw Grasping Screw – Pick up screw (5) Screw Fasten Moving Screw hole Move to screw hole – Fasten Screw hole Fasten screw – I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 7 of 14 Fig.  2, and it has a one-to-one correspondence between yˆ t + 1 information at the next time , and update the t+1 each pixel of an RGB image and depth image. Therefore, weights of the model to minimize the error between the the center coordinates of the object recognized by SSD yˆ x predicted value and the true value , and (3) at t+1 t+1 ( f , f ) are converted to 3D positional information (x, y, z) x y execution time, the robot is made to generate sequen- using a depth image. In addition, to enable the process tial motions by inputting the robot’s sensor information manager to check the progress of the work, we trained x and inputting the predicted value (motion command the module to change the recognition result in accord- value) to the robot for the next time. This method can ance with the state of the object. be used to perform various tasks, such as flexible object handling, which is difficult to do with the conventional Motion module method [31, 32]. In the motion module, the robot generates a trajectory on Figure  4 shows the details of the sensorimotor mod the basis of object position information (x, y, z). Point-to- ule. Using a raw visual image of a robot increases the Point (PtP) control is used to generate the robot’s motion. calculation cost, making it difficult to generate motion PtP control is implemented in common industrial robots in real time. When a visual image is simply resized, the and can be implemented at a small cost. The robot auto - important areas for the task are also compressed into a matically generates motions toward the goal position on small size, making it difficult to recognize detailed tasks the basis of the position information of the object recog- and states. In a typical environment where the position nized by the perception module. For example, in Task (2) of an object does not change, it is sufficient to crop a of Table 2, the robot grasps the cover and places it at the specific area of the camera image. However, in an atypi goal position. However, since both the cover and the base cal environment where the position of an object ran- stand are placed in random positions and postures, it is domly changes, simple cropping cannot be used. In this difficult to accurately align both positions. Therefore, in paper, we extract and resize images of the surround- the subtask “Moving,” the sensorimotor module is used ings on the basis of the location information of the to adjust the position of the cover to overlap the base object recognized by the perception module, as shown platform perfectly. in Fig.  4a. The object position is extracted only at the initial time t = 0 , and an image of the same region is Sensorimotor module used continuously after that to ensure real-time perfor- For the sensorimotor module, we use “Deep Predictive mance. Figure  4b shows the motion generation model Learning (DPL),” which is able to acquire desired behav- (DPL module) used in this paper. The model consists of iors with low data-collection and learning costs [28– a convolutional layer [33] that extracts image features 30]. This method learns a time series of sensory-motor from the robot’s visual information, a long short-term information when the robot operates in the real world, memory (LSTM) [34] that learns image features and the enabling it to perform complex tasks that are difficult robot’s body information in a time series, and a trans to realize with programs. Specifically, it consists of posed convolution layer that reconstructs images from three steps: (1) collect sensory-motor information (e.g., image features. The LSTM predicts the next-time image camera image, joint angle, and torque) with the robot (situation) and motor command from the current sen- as learning data when a human teleoperates the robot sor information. By learning visual and physical infor- or performs direct teaching, (2) input the sensor infor- mation simultaneously, the convolutional layer extracts mation x at time t into the model, output the sensor t the appropriate image features for motion generation Fig. 4 Network architecture of sensorimotor module Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 8 of 14 [35]. The robot generates real-time motions after the the position and orientation of a cover change randomly motor commands predicted by the LSTM are inputted. each time. It is not easy to grasp a randomly placed cover As shown in Table  2, the sensorimotor module is used and align its position and orientation so that it overlaps for three tasks: (1) adjusting the position and orienta- the base perfectly. Even if a joystick is used to remotely tion of the cover and base, (2) grasping the electric driver, control a robot to teach movements, there is the problem and (3) adjusting the position of the screwdriver bit and of learning not proceeding well because human move- the top of the screw (hexagonal hole) and inserting it. ments are inconsistent. In particular, it is difficult to We designed the these tasks with sensorimotor modules achieve high-precision operation due to minute misalign- since these have characteristics that is required interac- ment of the end state. Here, we used the motion teaching tion with robot and object of tasks, and also significant method shown in Fig.  5 to ensure that the training data impact on the later stages of the process. If the positions contains consistency. (a) Shows the sample of collected of the cover and the base are misaligned, the cover may image data by bird’s-eye-view camera, and it is found that fall off the base during the task, or the screw-hole posi - the center coordinates of the robot hand (yellow dotted tions may not match, and the task may fail. In addition, line) and the cover (red dotted line) were not always on a the grasping position and direction of the electric driver straight line. Desired operation is that align the orienta- will change the way in which screws are picked up and tion of cover and then align the position of covers, like tightened. To accomplish a series of tasks with a high flow from (d) to (b). For achieving this motion, we collect success rate, the adjustment operation of the sensorimo- the data that move position randomly from the aligned tor module is indispensable. Note that we used program state, then rotate orientation randomly. When the model based approach for sub-task of fastening the screw since training, the time series of the collected data is inverted it was easy to achieve by using impedance control so that (played backwards). It is based on the ease of generat- following hand position to screw hole when pre-exami- ing motions that shift from an aligned state to a random nation, though it should be classified as sensorimotor direction and position, and this made it possible to col- motion based on the proposal. lect data that were consistent with the end state. There - fore, it is expected that the robot can move an object in a random initial position to the same end state each Datasets for deep‑learning method time like as shown in from (d) to (b) during motion gen- Training data of perception module eration. We acquired the image data of bird’s-eye-view In this paper, we used a pre-trained SSD model to train camera as input data and command data of hand posi- the recognition and grasping position of assembly parts. tioning and rotating as output data of DPL. 504 training A total of 693 images of nine objects were taken using data where the orientation and position of the cover were the bird’s-eye-view camera at the top of the experimental changed, furthermore changing the grasping finger posi - apparatus: the workbench, base, grasping position of the tion and orientation of cover were acquired. Each piece base, cover, grasping position of the cover, the base on the of data was acquired for 10 s at a sampling rate of 10 Hz workbench, the cover on the workbench, the screw stor- per piece of data. age area, and the electric screwdriver. The images were taken when the position and orientation of the object were randomly changed. We prepared a set of images of assembly parts and their labels (object name and location information) as training data. To increase the amount of training data, we performed image positioning, rotation, and flipping for data augmentation. In addition, by ran - domly varying the brightness and contrast of the images, we obtained an object recognition model that was robust to changes in illumination. Training data for sensorimotor module In deep predictive learning, the model learns sensor information as a robot operates in the real world. In this paper, we taught the robot the three motions shown in Table 2(C). Since the method for teaching an operation is different for each task, the details are given below. Fig. 5 Overview of motion teaching: position adjustment of cover First, we describe the teaching method for adjusting and base the position of the cover. In an atypical environment, I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 9 of 14 Next, we describe the teaching method for screw han- the pickup work was performed by moving (searching) dling. Here, screw handling consists of the two opera- back and forth and left and right while pressing the bit tions shown in Fig.  6, the screw approach motion and against the screw. Impedance control was used to pre- the screw pickup motion. (a) Is a motion for bringing the vent overloading between the robot and the screw. The electric screwdriver close to the screw (screw approach robot was taught two motions as screw pickup motions: motion). Two motions are taught to the robot for the (1) perform adjustment in the left-right (y-axis) direc- screw approach action: (1) leftward (positive y) move- tion and (2) in the front-back (x-axis) direction. The six ment and (2) forward (positive x) movement. As a recov- horizontal arrows shown in (b) are teaching positions, ery operation in the case of operation failure, we taught and 378 training data that is same contents as (a) were several operations to move the screwdriver directly above acquired. The training data were collected by a person the screw from different directions as shown by the thin using a joystick to remotely control the robot. arrows in Fig.  6a. This allowed the robot to go back and Finally, we describe the teaching method for electric continue the positioning operation even if the amount screwdriver grasping. The robot grasped the electric of movement of the hand position was too large and screwdriver, which was placed on a holder. The height of thus passed the target position. We acquired the image the electric screwdriver was the same each time, but the data of hand camera as input data and command data screwdriver was placed at random positions. Therefore, of hand positioning as output data of DPL. 441 training the robot was taught the motion in the same way as the data where the approach motion when the position of screw approach motion described above. We acquired 86 the screw storage area was changed were acquired. From training data that is same contents as motion of picking the bird’s-eye-view camera, it was possible to recognize up screw by randomizing the initial position of the driver the screw storage area and generate motion toward its each time. The robot was taught to move from the screw - vicinity, but the resolution was not high enough to rec- driver grasping position to a random position. The time ognize the exact location of the screw. Here, the robot’s series was reversed (played backwards) and used as train- hand camera was used to generate an accurate approach ing data. This made it possible to execute position adjust - motion to the screw. (b) Is the action of fitting the bit of ment to align the robot hand with the screwdriver in a the electric screwdriver into the hexagonal hole at the random position. top of the screw (screw pickup motion). In the screw approach operation, it is difficult for the bit of the electric Results and discussion screwdriver to stop exactly above the screw. Therefore, In this section, we verify whether the perception module and sensorimotor module implemented using the learn- ing approach worked properly. In addition, we describe the results of 10 assembly operations performed in an atypical environment as a validation of the effectiveness of the proposed method. Object recognition Figure  7 shows the results of object recognition using SSD. (a) Is the initial state of assembly, (b) is after placing the base, and (c) is the end-of-task state. The parts and tools required for assembly were recognized. The model Fig. 6 Overview of motion teaching: screw approach and pickup Fig. 7 Task execution result: object labels and states recognized by SSD Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 10 of 14 recognized the object name as a label when each part was behavior only with robot arm 1, shown in Fig. 2, and ver- on the floor and the state name as a label when they were ify whether the same behavior can be executed with robot combined. For example, as shown in Fig.  7c, the model arm 2, which has not yet been trained. Each robot was output “Cover on Base” with the base and cover placed installed symmetrically around the bird’s-eye-view cam- on top of the bench. By switching the recognition label in era. Therefore, the image from the camera appeared dif - accordance with the status of the part, the process man- ferently to each robot working on the task. In Fig. 8a, the ager could check the progress status and switch tasks. robot arm appears to be on the left side, while in (b), it is on the right side. Therefore, to transfer the model learned Cover adjustment motion with robot arm 1 to robot arm 2, the input images and Figure 8 shows the results of adjusting the position of the command values to the DLP were inverted. Specifically, cover. The positions of the workbench and the base were the input images were flipped left and right, and the signs set up randomly. The center position of the image (inter - of the command values in the left and right (x) directions section of the white dotted lines) is indicated to make it were reversed for position and orientation adjustment. easier to understand the position change. (a) Shows that (b) Is the execution result for robot arm 2. The direction the robot learned to adjust the position of the cover. The and position of the cover could be adjusted even though robot adjusted the orientation to make a parallel line the position of the robot arm was reversed from that between the silver cover and the black base stand (step of the learning process. By simply learning the motion 2). Then, it adjusted the position of the cover and the of one robot arm, it was possible for the other robot to base stand (step 3). Since the size of the cover and the perform the same motion. This allows us to reduce the base stand were the same, the robot adjusted the posi- cost of motion learning. However, this is limited to cases tion to make the black base invisible. Steps 2 and 3 corre- where there is symmetry in the sensor information of the spond to Fig. 5d, c, respectively, indicating that the robot robot and the work object. generated a sequence of the motions. Thus, even if the base and cover were placed randomly and furthermore Screw pickup motion the grasping position were different each time, the robot Figure 9 shows a hand camera image of the robot during could adjust its position accurately. the execution of the screw handling operation. (a–d) Are Next, we discuss the reusability of the cover adjust- the operation of approaching the screw, and (e–h) are the ment module. We verified whether the robot, which had screw pickup operation. The robot was holding the elec - not learned the cover positioning operation, could adjust tric screwdriver and stopped in front of the screw stor- the position of the cover. In an atypical environment, the age area. There were several screws, and the robot went robot arm is expected to move appropriately in accord- to grab the leftmost one. At point (c), it recognized the ance with the position of the manipulated object. How- end position of the screw and generated a return motion. ever, reusability (commonality and diversion) of tasks is When the tip of the screwdriver bit and the screw were expected because implementing each behavior would aligned, it moved in the forward direction. Then, it incur development costs. In this section, we will learn a switched to the screw pickup operation and executed the Fig. 8 Task execution result: positioning cover on randomly placed base I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 11 of 14 Fig. 9 Task execution result: approaching top of screw and picking it up search operation to fit the tip of the screwdriver into the perception module was used to check the current work hexagonal hole of the screw. The robot could perform status and detect the position of the object. Next, the the tasks shown in Fig.  6 by generating (adjusting) the process manager selected a task on the basis of the rec- motions in real time. ognition results. The process manager was implemented with if-then rules. Task 1 was executed since the base was Verification of series of assembly tasks placed on the green desk. Task 1 consisted of two sub- To verify the effectiveness of the proposed method, tasks: pick up motion toward the recognized base and assembly work was performed with random positions of place motion toward the workbench. After Task 1 was parts, work tables, etc. Figure  10 shows the initial status completed, the perception module was used to check the of the parts and workbench before the start of assem- work status. The base was now placed on top of the work - bly for each session. The white dashed line is shown in bench, which changed the state name from workbench to the center of the image to make it easier to understand “Bench and base” as in Fig.  7b. This change in the state the situation of misalignment. In (a–f), the work was name caused the process manager to select Task 2. Task started with the base and cover placed on both sides of 2 involved the motion module and sensorimotor mod- the working table respectively, and in (g–j), the base and ule. Task 2-1 shows the grasped cover stacked on top of cover were placed on only one side of the table. Further- the base table, and it can be seen that a small misalign- more, in addition to the positions and orientations of the ment occurred. If the cover were placed on the base in base and cover, the work was started with the positions this state, it would fall off. Therefore, in Task 2-2, the of the workbench, screw storage area, and electric driver cover adjustment module was used to align the position changed. and orientation of both covers. Program-based execution Figure  11 shows the assembly process with different of simple operations was followed by fine-tuning using initial positions for the objects. The numbers in the fig - a learning-based approach to generate reliable and gen- ure correspond to the task numbers in Table  2. In this eralizable motions. At the end of Task 2, the state name section, we explain the software architecture of Fig.  1 changed from “Bench and base” to “Cover on base” as in in comparison with the actual operation. First, the Fig.  7c. A series of tasks can be performed by executing Fig. 10 Atypical environment used to verify sequence of tasks Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 12 of 14 Fig. 12 Task execution result: task aborted due to failed screw pickup popped out of the screw holder, making it impossible to continue working. We consider there to be two reasons for this failure. The first is that the accuracy of stopping the screw approach motion was low. The screwdriver bit may have come off the top of the screw during the next process because the position was slightly off when it should have stopped directly above the screw. The second is that the direc - tion of the screw pickup motion was not appropriate. The screw approach/pickup motion uses a learning approach, which enables the operation to be performed with gen- eralized performance. However, it is difficult to elucidate the causes of failures and take countermeasures because this approach is data-driven. One possible solution is to improve the accuracy of the operation by increasing the number of its learning patterns. Future tasks include real-time failure determination and estimation of recov- ery operations [11, 12, 25] and improvements to stability through hybridization with conventional control [36]. Fig. 11 Task execution result: parts and tools are in different positions Conclusion In this paper, we developed a novel robotic system that uses both a reliable programming-based approach and Task 3 through Task 5 in the same manner as described a robust learning-based approach. The program-based above. approach is used for four elemental motion that have We confirmed that the robot could flexibly generate not complex interactions between robot and object of motions and realize a series of assembly operations even robot tasks and is rough behavior, and the learning- in an atypical environment where the positions of parts, based approach is used for four elemental motion that is workplaces, and tools changed. Ten assembly opera- required complex interaction between robot and object tions were performed, and the work was completed eight of robot tasks and that are difficult to describe in a pro - times. By implementing the various functions of a robot gram. Our learning approach does not require the prior on the basis of the therblig, a robot system can be real- design of a computational model of an object. The robot’s ized in which a reliable program-based approach and a visual-image and joint-angle information can be used flexible learning-based approach coexist. to fasten screws and adjust the positions of objects. To verify the effectiveness of the proposed method, we cre - Failure case ated an assembly task with randomly placed tools and Figure  12 shows the case of a task failure. (a) Shows the parts using a real robot. This task was challenging com - screw pickup operation being executed. Normally, the pared with a typical assembly task because it required screwdriver bit would be inserted into the screw hole, but trajectory planning based on the positions of the ran- the screwdriver bit came off the top of the screw. In this domly placed parts and tools, and it also required that experiment, the screwdriver bit used had a built-in mag- their positions be adjusted and that screws be fastened. net for picking up screws. Therefore, in (b), the driver bit As a future works, upgrade of process manager will be stuck to the side of the screw when it fell. In (c), the screw required. Current process manager is configured by I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 13 of 14 analysis. In: 2019 IEEE international conference on prognostics and health simple if-then rules, however autonomous function to management (ICPHM), IEEE, pp 1–8 configur the motion modules from the stored modules 8. Johannink T, Bahl S, Nair A, Luo J, Kumar A, Loskyll M, Ojea JA, Solowjow E, will be required for adapting more atypical task with- Levine S (2019) Residual reinforcement learning for robot control. In: 2019 international conference on robotics and automation (ICRA), IEEE, pp out high implementation cost. And considering the best 6023–6029 perception method will also be issues so that process 9. Okawa Y, Sasaki T, Iwane H (2019) Control approach combining manager determines the suitable task and motion by reinforcement learning and model-based control. In: 2019 12th Asian control conference (ASCC), IEEE, pp 1419–1424 recognizing the situation of task and environment and 10. Suzuki K, Mori H, Ogata T (2018) Motion switching with sensory and automatically configure motions. Furthermore, we will instruction signals by designing dynamical systems using deep neural update the system to be able to perform recovery opera- network. IEEE Robot Autom Lett 3(4):3481–3488 11. Chen T, Liu X, Xia B, Wang W, Lai Y (2020) Unsupervised anomaly detec- tions based on work failure decisions in order to improve tion of industrial robots using sliding-window convolutional variational reliability. autoencoder. IEEE Access 8:47072–47081 12. Hung C-M, Sun L, Wu Y, Havoutis I, Posner I (2021) Introspective visuo- Acknowledgements motor control: exploiting uncertainty in deep visuomotor control for Not applicable. failure recovery. arXiv preprint arXiv: 2103. 11881 13. Price B (1989) Frank and lillian gilbreth and the manufacture and mar- Authors’ contributions keting of motion study, 1908-1924. Business and economic history, pp All authors contributed to the concept and overall development of this study. 88–98 HI developed the software for the learning-based approach. NA implemented 14. Gilbreth FB (1909) Bricklaying system. MC Clark Publishing Company, the rest of the software and performed the experiments. All authors worked New York together to wrote the manuscript. All authors read and approved the final 15. Brunelli R (2009) Template matching techniques in computer vision: manuscript. theory and practice. Wiley, Hoboken 16. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Funding unified, real-time object detection. In: Proceedings of the IEEE confer - Not applicable. ence on computer vision and pattern recognition, pp 779–788 17. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Availability of data and materials Ssd: Single shot multibox detector. In: European conference on com- Not applicable. puter vision, Springer, pp 21–37 18. Mahler J, Liang J, Niyaz S, Laskey M, Doan R, Liu X, Ojea JA, Goldberg K (2017) Dex-net 2.0: deep learning to plan robust grasps with synthetic Declarations point clouds and analytic grasp metrics. arXiv preprint arXiv: 1703. Competing interests 19. LaValle SM et al (1998) Rapidly-exploring random trees: a new tool for The authors declare that they have no competing interests. path planning 20. LaValle SM, Kuffner JJ Jr (2001) Randomized kinodynamic planning. Int Author details J Robot Res 20(5):378–400 Center for Technology Innovation - Controls and Robotics, Research & Devel- 21. Duan Y, Andrychowicz M, Stadie BC, Ho J, Schneider J, Sutskever I, opment Group, Hitachi, Ltd., Ibaraki 312-0025, Japan. Present Address: Center Abbeel P, Zaremba W (2017) One-shot imitation learning. arXiv preprint for Technology Innovation - Controls and Robotics, Research & Development arXiv: 1703. 07326 Group, Hitachi, Ltd., Ibaraki 312-0025, Japan. 22. Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P (2017) Domain randomization for transferring deep neural networks from Received: 5 November 2021 Accepted: 17 February 2022 simulation to the real world. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 23–30 23. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37(4–5):421–436 References 24. Yu T, Finn C, Xie A, Dasari S, Zhang T, Abbeel P, Levine S (2018) One- 1. Arnold S, Yamazaki K (2019) Fast and flexible multi-step cloth manipula- shot imitation from observing humans via domain-adaptive meta- tion planning using an encode-manipulate-decode network (em* d net). learning. arXiv preprint arXiv: 1802. 01557 Front Neurorobot 13:22 25. Tokuda S, Katayama M, Yamakita M, Oyama H (2020) Generating new 2. Sasagawa A, Fujimoto K, Sakaino S, Tsuji T (2020) Imitation learning based lower abstract task operator using grid-tli. In: 2020 IEEE/RSJ interna- on bilateral control for human-robot cooperation. IEEE Robot Autom Lett tional conference on intelligent robots and systems (IROS), IEEE, pp 5(4):6169–6176 6578–6584 3. Zeng A, Song S, Lee J, Rodriguez A, Funkhouser T (2020) Tossingbot: 26. Matsuoka S, Sawaragi T, Horiguchi Y, Nakanishi H (2016) Hierarchical plan- learning to throw arbitrary objects with residual physics. IEEE Trans Robot ning for error recovery in automated industrial robotic systems. In: 2016 36(4):1307–1319 IEEE international conference on systems, man, and cybernetics (SMC), 4. Olsson E, Funk P, Bengtsson M (2004) Fault diagnosis of industrial robots IEEE, pp 001406–001410 using acoustic signals and case-based reasoning. In: European confer- 27. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, ence on case-based reasoning, Springer, pp 686–701 Song Y, Guadarrama S et al. (2017) Speed/accuracy trade-offs for modern 5. Hornung R, Urbanek H, Klodmann J, Osendorfer C, Van Der Smagt P convolutional object detectors. In: Proceedings of the IEEE conference on (2014) Model-free robot anomaly detection. In: 2014 IEEE/RSJ interna- computer vision and pattern recognition, pp 7310–7311 tional conference on intelligent robots and systems, IEEE, pp 3676–3683. 28. Noda K, Arie H, Suga Y, Ogata T (2014) Multimodal integration learn- 6. Jaber AA, Bicker R (2014) The optimum selection of wavelet transform ing of robot behavior using deep neural networks. Robot Auton Syst parameters for the purpose of fault detection in an industrial robot. In: 62(6):721–736 2014 IEEE international conference on control system, computing and 29. Kase K, Suzuki K, Yang P-C, Mori H, Ogata T (2018) Put-in-box task gener- engineering (ICCSCE 2014), IEEE, pp 304–309 ated from multiple discrete tasks by ahumanoid robot using deep learn- 7. Cheng F, Raghavan A, Jung D, Sasaki Y, Tajika Y (2019) High-accuracy ing. In: 2018 IEEE international conference on robotics and automation unsupervised fault detection of industrial robots using current signal (ICRA), IEEE, pp 6447–6452 Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 14 of 14 30. Ichiwara H, Ito H, Yamamoto K, Mori H, Ogata T (2021) Spatial atten- tion point network for deep-learning-based robust autonomous robot motion generation. arXiv preprint arXiv: 2103. 01598 31. Yang P-C, Sasaki K, Suzuki K, Kase K, Sugano S, Ogata T (2016) Repeatable folding task by humanoid robot worker using deep learning. IEEE Robot Autom Lett 2(2):397–403 32. Suzuki K, Kanamura M, Suga Y, Mori H, Ogata T (2021) In-air knotting of rope using dual-arm robot based on deep learning. arXiv preprint arXiv: 2103. 09402 33. Fukushima K, Miyake S, Ito T (1983) Neocognitron: a neural network model for a mechanism of visual pattern recognition. IEEE Trans Syst Man Cybern 5:826–834 34. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780 35. Ito H, Yamamoto K, Mori H, Ogata T (2020) Evaluation of generalization performance of visuo-motor learning by analyzing internal state struc- tured from robot motion. New Gener Comput 38:7–22 36. Suzuki K, Mori H, Ogata T (2021) Compensation for undefined behaviors during robot task execution by switching controllers depending on embedded dynamics in rnn. IEEE Robot Autom Lett 6(2):3475–3482 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ROBOMECH Journal Springer Journals

Rapid prototyping for series of tasks in atypical environment: robotic system with reliable program-based and flexible learning-based approaches

ROBOMECH Journal , Volume 9 (1) – Mar 4, 2022

Loading next page...
 
/lp/springer-journals/rapid-prototyping-for-series-of-tasks-in-atypical-environment-robotic-APtxNCMkUu

References (14)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
eISSN
2197-4225
DOI
10.1186/s40648-022-00222-y
Publisher site
See Article on Publisher Site

Abstract

We propose a novel robotic system that combines both a reliable programming-based approach and a highly generalizable learning-based approach. How to design and implement a series of tasks in an atypical environment is a challenging issue. If all tasks are implemented using a programming-based approach, the development costs will be huge. However, if a learning-based approach is used, reliability is an issue. In this paper, we propose novel design guidelines that focus on the respective advantages of programming-based and learning-based approaches and select them so that they complement each other. We use a program-based approach for motions that is rough behavior and a learning-based approach for motion that is required complex interaction between robot and object of robot tasks and are difficult to achieve with a program. Our learning approach can easily and rapidly accomplish a series of tasks consisting of various motions because it does not require a computational model of an object to be designed in advance. We demonstrate a series of tasks in which randomly arranged parts are assembled using an actual robot. Keywords: Autonomous robot, Assembly, Motion generation, Deep predictive learning Introduction in unexpected situations. In factory automation, it is also Robotic automation has traditionally been used mainly desirable for robots to work autonomously in a new work for repetitive tasks in known environments, such as environment without humans having to teach them oper- automotive assembly. However, in recent years, there ations in detail since a quick response to frequent process has been a growing need to automate atypical tasks in changes is required for small-volume production. How- unknown or undeveloped environments that are difficult ever, there are several issues that need to be addressed in to perform with conventional robotic technology. For order to design a robot system that can perform atypical example, shortages of workers are becoming more seri- tasks. ous at construction and maintenance sites where danger- Big issues for design a robot system for perform series ous and heavy work is required. For robots to perform of task in atypical tasks is the implementation cost. At on-site work, the emphasis is not on the ability to repeat atypical work sites, robots need to be adaptive to the simple tasks at high speed and with high precision as in conditions of work objects and the positioning of parts. the past but on the ability to perform tasks reliably even To execute a series of tasks autonomously, three actions must be done: (1) recognizing the work situation and planning a procedure, (2) recognizing the position, *Correspondence: hiroshi.ito.ws@hitachi.com shape, and posture of parts, tools, etc., and (3) controlling Center for Technology Innovation - Controls and Robotics, Research & the motion of the robot. However, if we were to imple- Development Group, Hitachi, Ltd., Ibaraki 312-0025, Japan ment these functions for each task, the development Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 2 of 14 costs would be huge. In addition, a high level of exper- and actions that can be realized with relatively sim- tise is required to developtasks that involve contact with ple definitions were defined as program-based ones, an object (screw fastening and fitting) or are difficult to while complex actions were defined as learning- describe in a program (handling flexible objects). There - based ones. For example, a program-based approach fore, learning-based approaches have been proposed to is used for operations such as reaching, where the accomplish complex tasks that are difficult to accomplish robot moves dynamically, and safety and reliabil- with conventional programming-based approaches [1–3]. ity are required, and a learning approach is used for However, learning-based approaches have safety and operations such as screw fastening and fitting, where reliability issues. Unlike image recognition and natural the robot’s work area is limited, and the operations in language processing, robots perform tasks that involve this area are difficult to accomplish with a program. physical contact in the real world. This can lead to colli - The system designer can easily and rapidly realize a sions and other dangers. Therefore, it is necessary to have robotic system in which programming and learning both “safety” to prevent failure and damage as a result of approaches coexist by implementing various func- physical contact and collision and “reliability” to ensure tions on the basis of the design guidelines presented that behavioral rules are improved through learning. For in this paper. the former, safety has been ensured by detecting abnor- 2. Realizing a series of tasks in an atypical environment: malities and loads of robots by analyzing signals such as The robot can flexibly generate motions even in an current values, acoustic sensors, and acceleration sen- atypical environment where the placement of parts, sors [4–7]. However, the learning approach is a black box work areas, and tools change, and a series of assem- because it acquires behavioral rules on the basis of data, bly tasks can be accomplished with an 80% success and there are still problems with reliability. Some meth- rate. In the experiment, the alignment of each part is ods have been proposed to use a model-based controller important because the assembly work is performed as prior knowledge for deep reinforcement learning [8, using parts placed in random positions. Our learn- 9] or to monitor the state of a system in real time dur- ing-based approach allows the robot to acquire the ing operation generation and execute a predetermined desired operation by being taught the motion using recovery action when an anomaly is detected [10–12]. teleoperation and learning models. Therefore, there However, none of these methods focus on the reliability is no need to design a computational model of an of behavior. object in advance. During motion execution, the In this paper, we aim to accomplish complex tasks in robot can perform screw fastening and object align- an atypical environment in a simple and rapid way that is ment on the basis of its sensor information in real both reliable and flexible. How to design and implement time. a set of tasks in an atypical environment is a challenging issue. If we try to implement all tasks using a program- ming-based approach, the implementation costs will be Design concept huge. However, if a learning-based approach is used, reli- How to design and implement a series of tasks in an ability becomes an issue. In this paper, we propose a novel atypical environment is a challenging issue. If all the robotic system that focuses on the respective advantages tasks are implemented using a programming approach, of reliable program-based and flexible learning-based the costs of development will be huge, and if a learning- motion generation, which complement each other. To based approach is used, the data collection cost, learning verify the effectiveness of the proposed method, as a cost, and reliability will become issues. In this section, first step for an atypical environment task, an assembly we focus on the advantages of both reliable program- task with randomly placed tools and parts is performed ming-based and flexible learning-based methods, and we with a real robot. This task is very challenging compared describe how to design robotic systems that complement with general assembly tasks because it requires trajectory each other’s shortcomings. planning based on the positions of randomly placed parts and tools, as well as their positioning and screw fasten- Task categorization ing. The contributions of this paper are as follows. The issue is whether a programming-based or learning- based approach is more suitable for implementing each 1. Categorization of robotic tasks based on the therblig: function of a robot and how best to combine them. As a A method is proposed for designing a robotic sys- solution to this problem, in this paper, we define a clas - tem using therbligs that visualizes and improves the sification of robot functions and the execution order of efficiency of human tasks. The robot’s various tasks each function with reference to the therblig. Therbligs (e.g., grasping, carrying, fitting) were categorized, are 18 kinds of elemental motions that a worker needs I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 3 of 14 in order to perform a task [13, 14]. In the field of indus - • Disassemble: Separating multiple components that trial engineering, therbligs are used to visualize the are joined. Loosen the screws and disassemble the content and series of human work and to propose and parts. more efficient work methods and improve them. Here, • Plan: Planning to do this next or later. Think about we describe how to design a robotic system on the basis the procedure of the task. of the therblig. First, we extracted 12 kinds of elemental motions Implementing all these 12 kinds of elemental motions for robotic system design while excluding six elemen- with a programming-based approach would lead to tal motions that should be unnecessary for design the huge development costs. Therefore, we propose a design robot tasks, these are stopped motion elements that method (guidelines) for a robot system with reliability is no longer needed for work (i.e. “hold”, “unavoidable and versatility by classifying the 12 motions into three delay”, “avoidable delay”, “rest”), “pre-position” element categories on the basis of the characteristics of each func- that become unnecessary by optimizing in advance and tion and then implementing them as modules. Table 1(a) “find” element that is almost same as “search” and sel - shows three categorized elemental motions, and (b) dom use. Each elemental motion is explained using the shows the corresponding robot functions. (1)–(3) Show screw handling in assembly tasks as an example. the three categories, and their details are as follows. (1) “Perception” is a motion to judge or measure without • Search: Searching the location of an object with the actual robot action based on information such as vision, eyes or hands. The five senses other than the eyes, and corresponds to functions such as image recogni- such as when searching for holes with the fingertips, tion, object detection, and position estimation based on are also equivalent. Find out where the screws are. the robot’s visual information. It is also used for check- • Inspect: Determining an object by a defined standard ing the task status (progress) and motion planning based (quality or characteristics) using visual, auditory, tac- on visual information. Since deep learning has made tile, etc. Check how tightly screws have been tight- recognition technologies more diverse and accurate, ened between parts. it is possible to choose a method of implementation in • Select: Choosing one out of several. Select an appro- accordance with the task. Template matching [15], which priate screw from several screws. is conventionally used, is effective when the number of • Grasp: Grasping an object with hands or fingers. target objects is limited or patterned, while learning- Grab the screw. based object recognition algorithms [16–18] are effec - • Release load: Releasing control of an object. Release tive for various types and complex shapes of objects. the screw. (2) “Motion” is relatively simple motion that have not • Transport empty: Reaching an object with an empty complex interactions between robot and object of robot hand. Reach out to where the screw is. tasks, and it also requires a high level of reliability and • Transport load: Moving an object in hand from one safety because the robot moves widely in the real world. place to another. Carry the screw. Many of the motions are somewhat patterned, such as • Position: Adjusting the position and orientation of “take object A and place it at position B.” Teaching play- objects. Adjust the position to insert the screw into back, point-to-point (PtP), and trajectory planning algo- the hole. rithms such as rapidly-exploring random tree (RRT) [19, • Use: Manipulating tools, instruments, and equip- 20] are examples of implementation methods. (3) “Sen- ment for a purpose. Tighten the screws. sorimotor” is a motion that is necessary to have complex • Assemble: Joining two parts together. Use the screws interactions between robot and object of robot task, and to fix the two parts together. relies on the five senses and adjusts the position and force Table 1 Categorization of robot functions based on therbligs (1) Perception (2) Motion (3) Sensorimotor (a) Basic motion Search Grasp Position Inspect Release load Use Select Transport empty Assemble Plan Transport load Disassemble (b) Key function of robot control Image recognition Point-to-point motion Learning-based Object detection Interference avoidance motion generation Position estimation Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 4 of 14 in accordance with the situation. For tasks that are dif- to detect the position and state labels of objects. (2) The ficult to describe in a program, such as screw tightening process manager switches tasks using real-time process and fitting, a learning-based motion generation method planning methods [25, 26] or pre-designed if-then rules. would be effective. The diversification and develop - Next, we describe the series in which tasks are executed ment of learning-based approaches makes it possible to as shown in Fig. 1c. In this paper, a task consists of up to perform complex tasks [21–24]. Deep reinforcement three kinds of modules. A typical robotic system consists learning, imitation learning, and other implementation of “perception” and “motion planning and control.” It rec- methods are examples of implementation methods. ognizes object names and locations on the basis of vision We have described the categorization of robot func- and executes trajectories that satisfy geometric, mecha- tions and examples of their implementation, referring to nistic, and dynamic constraints on the basis of the rec- the therblig. In the next section, we describe the execu- ognition results. If we apply this to the categorization in tion procedure and how to combine the three functions. Table 1, the task will transition in the order of “perception module” and “motion module.” However, not all tasks can System architecture be realized with a program-based motion module. Some When a robot is performing a task, the timing at which tasks define the “motion module” to be followed by the tasks are switched and the order in which the motion and “sensorimotor module.” Basically, a programming-based sensorimotor are executed are issues. Figure 1 shows the approach is used for rough motions that are defined as execution procedure (system configuration) for the series “Motion” in Table  1, and a learning-based approach is of tasks proposed in this paper. The squares in the figure used for complex motions that are defined as “Sensori - represent each of the functions defined above (hereafter motor” in Table  1. u Th s, the number of modules (sub - referred to as modules). The modules consist of a “per - tasks) varies depending on the task. For example, Task 1 ception module” for recognizing situations, a “process in Fig.  1 consists of subtask 1 to pick up the object and manager” for switching tasks, and a “motion module” Subtask 2 to place it at the target position. Subtask 1 gen- and a “sensorimotor module” for executing subtasks. erates a grasping motion with the motion module on the On a simple production line, each task is executed in a basis of the position information of the target object rec- feed-forward sequence. However, with a pre-planned ognized by the perception module. Similarly, subtask 2 work schedule, it is not possible to respond to unex- generates a motion to place the object at the goal position pected events. In particular, since this paper assumes with the motion module on the basis of the goal posi- an atypical environment where the positions of objects tion information recognized by the perception module. change, process control in a feed-forward manner is dif- However, if positioning accuracy is required, such as in ficult. To manage work processes in an atypical environ - assembly work, it is impossible to complete a series of ment, it is necessary to recognize where objects are and tasks by simply placing the object at the goal position what state they are in on the basis of vision information. and it is required to adjust position according to state of In this paper, we use the perception module of Fig.  1 (1) each parts that would be assembled, such as adjustment Fig. 1 Software architecture for performing series of tasks I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 5 of 14 motion with vision sensing or searching motion with robot arms can be remotely controlled using a joystick force feedback. In addition, implementing a high-preci- to teach the desired motion. Impedance control can also sion positioning algorithm using a programming-based be performed by using a torque sensor at each joint. For approach requires complex programming and real-world example, even if the axis of the driver and the screw hole tuning. Therefore, we use the sensorimotor module to is misaligned when tightening a screw and the hand is adjust the position after the motion module. This makes overloaded, it can prevent the screw and screw hole it possible to perform complex tasks robustly with lower from being overloaded by passively moving the hand. development costs. However, since there is an inverse relationship between As described above, by defining (implementing) vari - arm strength and positioning accuracy with impedance ous functions and execution procedures for the robot on control, and the arm does not move accurately in rela- the basis of the task categories and design policies pro- tion to the hand position command, the control mode is posed in this paper, a robot system can be constructed in switched depending on the task. In this paper, we used which a highly reliable program base and a highly robust position control mode when executing the motion mod- learning base coexist. ule and impedance control mode when executing the sensorimotor module, which involves contact with the Set up for verification object. To verify the effectiveness of the proposed method, we performed an assembly task in an atypical environment. Assembly task for verification Here, we describe the experimental setup and the experi- Recently, robots are required to have the ability to handle mental task. high-mix low-volume production, rather than the con- ventional ability to repeat simple tasks at high speed and Robotic hardware with high precision. These robots must have the ability to Figure  2 shows the experimental setup. As shown in (a), work autonomously in atypical environments. As men- two robot arms (KUKA LBR iiwa 14 R820) are used, and tioned above, in this paper, we focus on assembly tasks a robotic hand (Robotiq 2F-85 Adaptive Gripper) and in the atypical environment shown in Fig.  3. To evaluate an RGB camera (Buffalo BSW500M Series) are attached such tasks in this environment, the parts and workbench to the end of the arms. The other equipment includes a are not fixed but are randomly placed within the robot’s bird’s-eye-view camera (Intel RealSenes D435i) for tak- operating range before work is started. The assembly ing full view of work place, and an electric screwdriver task consists of five steps: (1) pick and place a randomly connected to a tool balancer. The workbench, cover, placed base, (2) pick and place a randomly placed cover, base, and screws are the parts needed for assembly. (b) (3) grasp the electric screwdriver, (4) pick up a screw Shows an enlarged view near the robotic hand when (attach it to the screw bit with the magnetic bit on the tip hand grasping the screw driver. As shown in (b), hand of the electric driver), and (5) screw the four corners of camera installed near base of robotic hand and it can the cover to the base. To avoid contact between the robot take an image of near the finger of robotic hand. The and floor, the assembly is performed on a workbench. We Fig. 2 Experimental environment: sensors and equipment Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 6 of 14 Process manager We used the Single Shot Multibox Detector (SSD) [17] as an object recognition algorithm based on deep learning to recognize the work process. To reduce the implemen- tation costs, we used the TensorFlow Object Detection API [27]. The robot performed rule-based task switch - ing on the basis of object labels recognized by SSD. For example, when two parts are to be assembled, the name of each part is output as an object label before assem- bly, and the states are combined after assembly. If a part name is output, the corresponding action (motion or/ and sensorimotor module) will be performed; if a state is output, the next task will be performed. u Th s, a series of tasks can be accomplished by combining object recog- Fig. 3 Assembly tasks in atypical environment. Each part and tool are nition and rule-based process planning. General object randomly placed recognition algorithms, such as template matching, have low robustness to illumination changes. In particular, it is difficult to accurately recognize metals such as assembly evaluated the success rate of the proposed method by parts because the way an object looks depends on how performing a series of assembly tasks 10 times. In these the light hits it (reflection angle). In comparison, SSD tasks, some complex behavior that needs interaction can perform object detection in real time and robustly between robot and object for assembly task is required, against illumination and background changes. Further- for example, align two parts while considering position more, it can simultaneously predict location information and orientation of each parts so that the two parts over- in addition to object label information. We can share the lap exactly, or adjust position and force of the hand to fit modules and reduce development costs by passing the the screw to the tip of the screwdriver. label information predicted by SSD to the process man- ager and the location information to the motion module. System configuration In this section, we describe the specific method to con - figure the robot system with classification of the experi - Perception module mental tasks and the specific implementation method. Since SSD detects the positional information of objects Table  2 shows the tasks and their execution procedures. in the two-dimensional plane of a camera image, it can- The assembly task consisted of five tasks, which were not be used directly for robot control. For the robot to executed in order from 1 to 5. The numbers in each task perform tasks on the basis of location information from correspond to the numbers in Fig.  3. The subtasks were SSD, the information is converted into 3D coordinates executed in order from A to C. The implementation of the real environment. The camera used for perception method for each function is described below. in this paper is bird’s-eye-view camera that is shown in Table 2 Task categorization for assembly work Task Sub-task (A) Perception (B) Motion (C) Sensorimotor (1) Base operation Picking Base Pick up base – Placing Workbench Place base on workbench – (2) Cover operation Picking Cover Pick up cover – Moving Workbench Layer cover over base Alignment adjustment Placing Workbench Place cover on base – (3) Take screwdriver Grasping Screwdriver Reach for screwdriver Position adjustment Moving Screw holder Move to screw holder – (4) Pick up screw Moving Screw – Approach screw Grasping Screw – Pick up screw (5) Screw Fasten Moving Screw hole Move to screw hole – Fasten Screw hole Fasten screw – I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 7 of 14 Fig.  2, and it has a one-to-one correspondence between yˆ t + 1 information at the next time , and update the t+1 each pixel of an RGB image and depth image. Therefore, weights of the model to minimize the error between the the center coordinates of the object recognized by SSD yˆ x predicted value and the true value , and (3) at t+1 t+1 ( f , f ) are converted to 3D positional information (x, y, z) x y execution time, the robot is made to generate sequen- using a depth image. In addition, to enable the process tial motions by inputting the robot’s sensor information manager to check the progress of the work, we trained x and inputting the predicted value (motion command the module to change the recognition result in accord- value) to the robot for the next time. This method can ance with the state of the object. be used to perform various tasks, such as flexible object handling, which is difficult to do with the conventional Motion module method [31, 32]. In the motion module, the robot generates a trajectory on Figure  4 shows the details of the sensorimotor mod the basis of object position information (x, y, z). Point-to- ule. Using a raw visual image of a robot increases the Point (PtP) control is used to generate the robot’s motion. calculation cost, making it difficult to generate motion PtP control is implemented in common industrial robots in real time. When a visual image is simply resized, the and can be implemented at a small cost. The robot auto - important areas for the task are also compressed into a matically generates motions toward the goal position on small size, making it difficult to recognize detailed tasks the basis of the position information of the object recog- and states. In a typical environment where the position nized by the perception module. For example, in Task (2) of an object does not change, it is sufficient to crop a of Table 2, the robot grasps the cover and places it at the specific area of the camera image. However, in an atypi goal position. However, since both the cover and the base cal environment where the position of an object ran- stand are placed in random positions and postures, it is domly changes, simple cropping cannot be used. In this difficult to accurately align both positions. Therefore, in paper, we extract and resize images of the surround- the subtask “Moving,” the sensorimotor module is used ings on the basis of the location information of the to adjust the position of the cover to overlap the base object recognized by the perception module, as shown platform perfectly. in Fig.  4a. The object position is extracted only at the initial time t = 0 , and an image of the same region is Sensorimotor module used continuously after that to ensure real-time perfor- For the sensorimotor module, we use “Deep Predictive mance. Figure  4b shows the motion generation model Learning (DPL),” which is able to acquire desired behav- (DPL module) used in this paper. The model consists of iors with low data-collection and learning costs [28– a convolutional layer [33] that extracts image features 30]. This method learns a time series of sensory-motor from the robot’s visual information, a long short-term information when the robot operates in the real world, memory (LSTM) [34] that learns image features and the enabling it to perform complex tasks that are difficult robot’s body information in a time series, and a trans to realize with programs. Specifically, it consists of posed convolution layer that reconstructs images from three steps: (1) collect sensory-motor information (e.g., image features. The LSTM predicts the next-time image camera image, joint angle, and torque) with the robot (situation) and motor command from the current sen- as learning data when a human teleoperates the robot sor information. By learning visual and physical infor- or performs direct teaching, (2) input the sensor infor- mation simultaneously, the convolutional layer extracts mation x at time t into the model, output the sensor t the appropriate image features for motion generation Fig. 4 Network architecture of sensorimotor module Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 8 of 14 [35]. The robot generates real-time motions after the the position and orientation of a cover change randomly motor commands predicted by the LSTM are inputted. each time. It is not easy to grasp a randomly placed cover As shown in Table  2, the sensorimotor module is used and align its position and orientation so that it overlaps for three tasks: (1) adjusting the position and orienta- the base perfectly. Even if a joystick is used to remotely tion of the cover and base, (2) grasping the electric driver, control a robot to teach movements, there is the problem and (3) adjusting the position of the screwdriver bit and of learning not proceeding well because human move- the top of the screw (hexagonal hole) and inserting it. ments are inconsistent. In particular, it is difficult to We designed the these tasks with sensorimotor modules achieve high-precision operation due to minute misalign- since these have characteristics that is required interac- ment of the end state. Here, we used the motion teaching tion with robot and object of tasks, and also significant method shown in Fig.  5 to ensure that the training data impact on the later stages of the process. If the positions contains consistency. (a) Shows the sample of collected of the cover and the base are misaligned, the cover may image data by bird’s-eye-view camera, and it is found that fall off the base during the task, or the screw-hole posi - the center coordinates of the robot hand (yellow dotted tions may not match, and the task may fail. In addition, line) and the cover (red dotted line) were not always on a the grasping position and direction of the electric driver straight line. Desired operation is that align the orienta- will change the way in which screws are picked up and tion of cover and then align the position of covers, like tightened. To accomplish a series of tasks with a high flow from (d) to (b). For achieving this motion, we collect success rate, the adjustment operation of the sensorimo- the data that move position randomly from the aligned tor module is indispensable. Note that we used program state, then rotate orientation randomly. When the model based approach for sub-task of fastening the screw since training, the time series of the collected data is inverted it was easy to achieve by using impedance control so that (played backwards). It is based on the ease of generat- following hand position to screw hole when pre-exami- ing motions that shift from an aligned state to a random nation, though it should be classified as sensorimotor direction and position, and this made it possible to col- motion based on the proposal. lect data that were consistent with the end state. There - fore, it is expected that the robot can move an object in a random initial position to the same end state each Datasets for deep‑learning method time like as shown in from (d) to (b) during motion gen- Training data of perception module eration. We acquired the image data of bird’s-eye-view In this paper, we used a pre-trained SSD model to train camera as input data and command data of hand posi- the recognition and grasping position of assembly parts. tioning and rotating as output data of DPL. 504 training A total of 693 images of nine objects were taken using data where the orientation and position of the cover were the bird’s-eye-view camera at the top of the experimental changed, furthermore changing the grasping finger posi - apparatus: the workbench, base, grasping position of the tion and orientation of cover were acquired. Each piece base, cover, grasping position of the cover, the base on the of data was acquired for 10 s at a sampling rate of 10 Hz workbench, the cover on the workbench, the screw stor- per piece of data. age area, and the electric screwdriver. The images were taken when the position and orientation of the object were randomly changed. We prepared a set of images of assembly parts and their labels (object name and location information) as training data. To increase the amount of training data, we performed image positioning, rotation, and flipping for data augmentation. In addition, by ran - domly varying the brightness and contrast of the images, we obtained an object recognition model that was robust to changes in illumination. Training data for sensorimotor module In deep predictive learning, the model learns sensor information as a robot operates in the real world. In this paper, we taught the robot the three motions shown in Table 2(C). Since the method for teaching an operation is different for each task, the details are given below. Fig. 5 Overview of motion teaching: position adjustment of cover First, we describe the teaching method for adjusting and base the position of the cover. In an atypical environment, I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 9 of 14 Next, we describe the teaching method for screw han- the pickup work was performed by moving (searching) dling. Here, screw handling consists of the two opera- back and forth and left and right while pressing the bit tions shown in Fig.  6, the screw approach motion and against the screw. Impedance control was used to pre- the screw pickup motion. (a) Is a motion for bringing the vent overloading between the robot and the screw. The electric screwdriver close to the screw (screw approach robot was taught two motions as screw pickup motions: motion). Two motions are taught to the robot for the (1) perform adjustment in the left-right (y-axis) direc- screw approach action: (1) leftward (positive y) move- tion and (2) in the front-back (x-axis) direction. The six ment and (2) forward (positive x) movement. As a recov- horizontal arrows shown in (b) are teaching positions, ery operation in the case of operation failure, we taught and 378 training data that is same contents as (a) were several operations to move the screwdriver directly above acquired. The training data were collected by a person the screw from different directions as shown by the thin using a joystick to remotely control the robot. arrows in Fig.  6a. This allowed the robot to go back and Finally, we describe the teaching method for electric continue the positioning operation even if the amount screwdriver grasping. The robot grasped the electric of movement of the hand position was too large and screwdriver, which was placed on a holder. The height of thus passed the target position. We acquired the image the electric screwdriver was the same each time, but the data of hand camera as input data and command data screwdriver was placed at random positions. Therefore, of hand positioning as output data of DPL. 441 training the robot was taught the motion in the same way as the data where the approach motion when the position of screw approach motion described above. We acquired 86 the screw storage area was changed were acquired. From training data that is same contents as motion of picking the bird’s-eye-view camera, it was possible to recognize up screw by randomizing the initial position of the driver the screw storage area and generate motion toward its each time. The robot was taught to move from the screw - vicinity, but the resolution was not high enough to rec- driver grasping position to a random position. The time ognize the exact location of the screw. Here, the robot’s series was reversed (played backwards) and used as train- hand camera was used to generate an accurate approach ing data. This made it possible to execute position adjust - motion to the screw. (b) Is the action of fitting the bit of ment to align the robot hand with the screwdriver in a the electric screwdriver into the hexagonal hole at the random position. top of the screw (screw pickup motion). In the screw approach operation, it is difficult for the bit of the electric Results and discussion screwdriver to stop exactly above the screw. Therefore, In this section, we verify whether the perception module and sensorimotor module implemented using the learn- ing approach worked properly. In addition, we describe the results of 10 assembly operations performed in an atypical environment as a validation of the effectiveness of the proposed method. Object recognition Figure  7 shows the results of object recognition using SSD. (a) Is the initial state of assembly, (b) is after placing the base, and (c) is the end-of-task state. The parts and tools required for assembly were recognized. The model Fig. 6 Overview of motion teaching: screw approach and pickup Fig. 7 Task execution result: object labels and states recognized by SSD Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 10 of 14 recognized the object name as a label when each part was behavior only with robot arm 1, shown in Fig. 2, and ver- on the floor and the state name as a label when they were ify whether the same behavior can be executed with robot combined. For example, as shown in Fig.  7c, the model arm 2, which has not yet been trained. Each robot was output “Cover on Base” with the base and cover placed installed symmetrically around the bird’s-eye-view cam- on top of the bench. By switching the recognition label in era. Therefore, the image from the camera appeared dif - accordance with the status of the part, the process man- ferently to each robot working on the task. In Fig. 8a, the ager could check the progress status and switch tasks. robot arm appears to be on the left side, while in (b), it is on the right side. Therefore, to transfer the model learned Cover adjustment motion with robot arm 1 to robot arm 2, the input images and Figure 8 shows the results of adjusting the position of the command values to the DLP were inverted. Specifically, cover. The positions of the workbench and the base were the input images were flipped left and right, and the signs set up randomly. The center position of the image (inter - of the command values in the left and right (x) directions section of the white dotted lines) is indicated to make it were reversed for position and orientation adjustment. easier to understand the position change. (a) Shows that (b) Is the execution result for robot arm 2. The direction the robot learned to adjust the position of the cover. The and position of the cover could be adjusted even though robot adjusted the orientation to make a parallel line the position of the robot arm was reversed from that between the silver cover and the black base stand (step of the learning process. By simply learning the motion 2). Then, it adjusted the position of the cover and the of one robot arm, it was possible for the other robot to base stand (step 3). Since the size of the cover and the perform the same motion. This allows us to reduce the base stand were the same, the robot adjusted the posi- cost of motion learning. However, this is limited to cases tion to make the black base invisible. Steps 2 and 3 corre- where there is symmetry in the sensor information of the spond to Fig. 5d, c, respectively, indicating that the robot robot and the work object. generated a sequence of the motions. Thus, even if the base and cover were placed randomly and furthermore Screw pickup motion the grasping position were different each time, the robot Figure 9 shows a hand camera image of the robot during could adjust its position accurately. the execution of the screw handling operation. (a–d) Are Next, we discuss the reusability of the cover adjust- the operation of approaching the screw, and (e–h) are the ment module. We verified whether the robot, which had screw pickup operation. The robot was holding the elec - not learned the cover positioning operation, could adjust tric screwdriver and stopped in front of the screw stor- the position of the cover. In an atypical environment, the age area. There were several screws, and the robot went robot arm is expected to move appropriately in accord- to grab the leftmost one. At point (c), it recognized the ance with the position of the manipulated object. How- end position of the screw and generated a return motion. ever, reusability (commonality and diversion) of tasks is When the tip of the screwdriver bit and the screw were expected because implementing each behavior would aligned, it moved in the forward direction. Then, it incur development costs. In this section, we will learn a switched to the screw pickup operation and executed the Fig. 8 Task execution result: positioning cover on randomly placed base I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 11 of 14 Fig. 9 Task execution result: approaching top of screw and picking it up search operation to fit the tip of the screwdriver into the perception module was used to check the current work hexagonal hole of the screw. The robot could perform status and detect the position of the object. Next, the the tasks shown in Fig.  6 by generating (adjusting) the process manager selected a task on the basis of the rec- motions in real time. ognition results. The process manager was implemented with if-then rules. Task 1 was executed since the base was Verification of series of assembly tasks placed on the green desk. Task 1 consisted of two sub- To verify the effectiveness of the proposed method, tasks: pick up motion toward the recognized base and assembly work was performed with random positions of place motion toward the workbench. After Task 1 was parts, work tables, etc. Figure  10 shows the initial status completed, the perception module was used to check the of the parts and workbench before the start of assem- work status. The base was now placed on top of the work - bly for each session. The white dashed line is shown in bench, which changed the state name from workbench to the center of the image to make it easier to understand “Bench and base” as in Fig.  7b. This change in the state the situation of misalignment. In (a–f), the work was name caused the process manager to select Task 2. Task started with the base and cover placed on both sides of 2 involved the motion module and sensorimotor mod- the working table respectively, and in (g–j), the base and ule. Task 2-1 shows the grasped cover stacked on top of cover were placed on only one side of the table. Further- the base table, and it can be seen that a small misalign- more, in addition to the positions and orientations of the ment occurred. If the cover were placed on the base in base and cover, the work was started with the positions this state, it would fall off. Therefore, in Task 2-2, the of the workbench, screw storage area, and electric driver cover adjustment module was used to align the position changed. and orientation of both covers. Program-based execution Figure  11 shows the assembly process with different of simple operations was followed by fine-tuning using initial positions for the objects. The numbers in the fig - a learning-based approach to generate reliable and gen- ure correspond to the task numbers in Table  2. In this eralizable motions. At the end of Task 2, the state name section, we explain the software architecture of Fig.  1 changed from “Bench and base” to “Cover on base” as in in comparison with the actual operation. First, the Fig.  7c. A series of tasks can be performed by executing Fig. 10 Atypical environment used to verify sequence of tasks Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 12 of 14 Fig. 12 Task execution result: task aborted due to failed screw pickup popped out of the screw holder, making it impossible to continue working. We consider there to be two reasons for this failure. The first is that the accuracy of stopping the screw approach motion was low. The screwdriver bit may have come off the top of the screw during the next process because the position was slightly off when it should have stopped directly above the screw. The second is that the direc - tion of the screw pickup motion was not appropriate. The screw approach/pickup motion uses a learning approach, which enables the operation to be performed with gen- eralized performance. However, it is difficult to elucidate the causes of failures and take countermeasures because this approach is data-driven. One possible solution is to improve the accuracy of the operation by increasing the number of its learning patterns. Future tasks include real-time failure determination and estimation of recov- ery operations [11, 12, 25] and improvements to stability through hybridization with conventional control [36]. Fig. 11 Task execution result: parts and tools are in different positions Conclusion In this paper, we developed a novel robotic system that uses both a reliable programming-based approach and Task 3 through Task 5 in the same manner as described a robust learning-based approach. The program-based above. approach is used for four elemental motion that have We confirmed that the robot could flexibly generate not complex interactions between robot and object of motions and realize a series of assembly operations even robot tasks and is rough behavior, and the learning- in an atypical environment where the positions of parts, based approach is used for four elemental motion that is workplaces, and tools changed. Ten assembly opera- required complex interaction between robot and object tions were performed, and the work was completed eight of robot tasks and that are difficult to describe in a pro - times. By implementing the various functions of a robot gram. Our learning approach does not require the prior on the basis of the therblig, a robot system can be real- design of a computational model of an object. The robot’s ized in which a reliable program-based approach and a visual-image and joint-angle information can be used flexible learning-based approach coexist. to fasten screws and adjust the positions of objects. To verify the effectiveness of the proposed method, we cre - Failure case ated an assembly task with randomly placed tools and Figure  12 shows the case of a task failure. (a) Shows the parts using a real robot. This task was challenging com - screw pickup operation being executed. Normally, the pared with a typical assembly task because it required screwdriver bit would be inserted into the screw hole, but trajectory planning based on the positions of the ran- the screwdriver bit came off the top of the screw. In this domly placed parts and tools, and it also required that experiment, the screwdriver bit used had a built-in mag- their positions be adjusted and that screws be fastened. net for picking up screws. Therefore, in (b), the driver bit As a future works, upgrade of process manager will be stuck to the side of the screw when it fell. In (c), the screw required. Current process manager is configured by I to and Nakamura ROBOMECH Journal (2022) 9:7 Page 13 of 14 analysis. In: 2019 IEEE international conference on prognostics and health simple if-then rules, however autonomous function to management (ICPHM), IEEE, pp 1–8 configur the motion modules from the stored modules 8. Johannink T, Bahl S, Nair A, Luo J, Kumar A, Loskyll M, Ojea JA, Solowjow E, will be required for adapting more atypical task with- Levine S (2019) Residual reinforcement learning for robot control. In: 2019 international conference on robotics and automation (ICRA), IEEE, pp out high implementation cost. And considering the best 6023–6029 perception method will also be issues so that process 9. Okawa Y, Sasaki T, Iwane H (2019) Control approach combining manager determines the suitable task and motion by reinforcement learning and model-based control. In: 2019 12th Asian control conference (ASCC), IEEE, pp 1419–1424 recognizing the situation of task and environment and 10. Suzuki K, Mori H, Ogata T (2018) Motion switching with sensory and automatically configure motions. Furthermore, we will instruction signals by designing dynamical systems using deep neural update the system to be able to perform recovery opera- network. IEEE Robot Autom Lett 3(4):3481–3488 11. Chen T, Liu X, Xia B, Wang W, Lai Y (2020) Unsupervised anomaly detec- tions based on work failure decisions in order to improve tion of industrial robots using sliding-window convolutional variational reliability. autoencoder. IEEE Access 8:47072–47081 12. Hung C-M, Sun L, Wu Y, Havoutis I, Posner I (2021) Introspective visuo- Acknowledgements motor control: exploiting uncertainty in deep visuomotor control for Not applicable. failure recovery. arXiv preprint arXiv: 2103. 11881 13. Price B (1989) Frank and lillian gilbreth and the manufacture and mar- Authors’ contributions keting of motion study, 1908-1924. Business and economic history, pp All authors contributed to the concept and overall development of this study. 88–98 HI developed the software for the learning-based approach. NA implemented 14. Gilbreth FB (1909) Bricklaying system. MC Clark Publishing Company, the rest of the software and performed the experiments. All authors worked New York together to wrote the manuscript. All authors read and approved the final 15. Brunelli R (2009) Template matching techniques in computer vision: manuscript. theory and practice. Wiley, Hoboken 16. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Funding unified, real-time object detection. In: Proceedings of the IEEE confer - Not applicable. ence on computer vision and pattern recognition, pp 779–788 17. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Availability of data and materials Ssd: Single shot multibox detector. In: European conference on com- Not applicable. puter vision, Springer, pp 21–37 18. Mahler J, Liang J, Niyaz S, Laskey M, Doan R, Liu X, Ojea JA, Goldberg K (2017) Dex-net 2.0: deep learning to plan robust grasps with synthetic Declarations point clouds and analytic grasp metrics. arXiv preprint arXiv: 1703. Competing interests 19. LaValle SM et al (1998) Rapidly-exploring random trees: a new tool for The authors declare that they have no competing interests. path planning 20. LaValle SM, Kuffner JJ Jr (2001) Randomized kinodynamic planning. Int Author details J Robot Res 20(5):378–400 Center for Technology Innovation - Controls and Robotics, Research & Devel- 21. Duan Y, Andrychowicz M, Stadie BC, Ho J, Schneider J, Sutskever I, opment Group, Hitachi, Ltd., Ibaraki 312-0025, Japan. Present Address: Center Abbeel P, Zaremba W (2017) One-shot imitation learning. arXiv preprint for Technology Innovation - Controls and Robotics, Research & Development arXiv: 1703. 07326 Group, Hitachi, Ltd., Ibaraki 312-0025, Japan. 22. Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P (2017) Domain randomization for transferring deep neural networks from Received: 5 November 2021 Accepted: 17 February 2022 simulation to the real world. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 23–30 23. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37(4–5):421–436 References 24. Yu T, Finn C, Xie A, Dasari S, Zhang T, Abbeel P, Levine S (2018) One- 1. Arnold S, Yamazaki K (2019) Fast and flexible multi-step cloth manipula- shot imitation from observing humans via domain-adaptive meta- tion planning using an encode-manipulate-decode network (em* d net). learning. arXiv preprint arXiv: 1802. 01557 Front Neurorobot 13:22 25. Tokuda S, Katayama M, Yamakita M, Oyama H (2020) Generating new 2. Sasagawa A, Fujimoto K, Sakaino S, Tsuji T (2020) Imitation learning based lower abstract task operator using grid-tli. In: 2020 IEEE/RSJ interna- on bilateral control for human-robot cooperation. IEEE Robot Autom Lett tional conference on intelligent robots and systems (IROS), IEEE, pp 5(4):6169–6176 6578–6584 3. Zeng A, Song S, Lee J, Rodriguez A, Funkhouser T (2020) Tossingbot: 26. Matsuoka S, Sawaragi T, Horiguchi Y, Nakanishi H (2016) Hierarchical plan- learning to throw arbitrary objects with residual physics. IEEE Trans Robot ning for error recovery in automated industrial robotic systems. In: 2016 36(4):1307–1319 IEEE international conference on systems, man, and cybernetics (SMC), 4. Olsson E, Funk P, Bengtsson M (2004) Fault diagnosis of industrial robots IEEE, pp 001406–001410 using acoustic signals and case-based reasoning. In: European confer- 27. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, ence on case-based reasoning, Springer, pp 686–701 Song Y, Guadarrama S et al. (2017) Speed/accuracy trade-offs for modern 5. Hornung R, Urbanek H, Klodmann J, Osendorfer C, Van Der Smagt P convolutional object detectors. In: Proceedings of the IEEE conference on (2014) Model-free robot anomaly detection. In: 2014 IEEE/RSJ interna- computer vision and pattern recognition, pp 7310–7311 tional conference on intelligent robots and systems, IEEE, pp 3676–3683. 28. Noda K, Arie H, Suga Y, Ogata T (2014) Multimodal integration learn- 6. Jaber AA, Bicker R (2014) The optimum selection of wavelet transform ing of robot behavior using deep neural networks. Robot Auton Syst parameters for the purpose of fault detection in an industrial robot. In: 62(6):721–736 2014 IEEE international conference on control system, computing and 29. Kase K, Suzuki K, Yang P-C, Mori H, Ogata T (2018) Put-in-box task gener- engineering (ICCSCE 2014), IEEE, pp 304–309 ated from multiple discrete tasks by ahumanoid robot using deep learn- 7. Cheng F, Raghavan A, Jung D, Sasaki Y, Tajika Y (2019) High-accuracy ing. In: 2018 IEEE international conference on robotics and automation unsupervised fault detection of industrial robots using current signal (ICRA), IEEE, pp 6447–6452 Ito and Nakamura ROBOMECH Journal (2022) 9:7 Page 14 of 14 30. Ichiwara H, Ito H, Yamamoto K, Mori H, Ogata T (2021) Spatial atten- tion point network for deep-learning-based robust autonomous robot motion generation. arXiv preprint arXiv: 2103. 01598 31. Yang P-C, Sasaki K, Suzuki K, Kase K, Sugano S, Ogata T (2016) Repeatable folding task by humanoid robot worker using deep learning. IEEE Robot Autom Lett 2(2):397–403 32. Suzuki K, Kanamura M, Suga Y, Mori H, Ogata T (2021) In-air knotting of rope using dual-arm robot based on deep learning. arXiv preprint arXiv: 2103. 09402 33. Fukushima K, Miyake S, Ito T (1983) Neocognitron: a neural network model for a mechanism of visual pattern recognition. IEEE Trans Syst Man Cybern 5:826–834 34. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780 35. Ito H, Yamamoto K, Mori H, Ogata T (2020) Evaluation of generalization performance of visuo-motor learning by analyzing internal state struc- tured from robot motion. New Gener Comput 38:7–22 36. Suzuki K, Mori H, Ogata T (2021) Compensation for undefined behaviors during robot task execution by switching controllers depending on embedded dynamics in rnn. IEEE Robot Autom Lett 6(2):3475–3482 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations.

Journal

ROBOMECH JournalSpringer Journals

Published: Mar 4, 2022

Keywords: Autonomous robot; Assembly; Motion generation; Deep predictive learning

There are no references for this article.