Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

Antun Ivanovic; Lovro Markovic; Marko Car; Ivan Duvnjak; Matko Orsag

doi:10.3390/app11188279

Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

Ivanovic, Antun;Markovic, Lovro;Car, Marko;Duvnjak, Ivan;Orsag, Matko 2021-09-07 00:00:00 applied sciences Article Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators 1 , 1 1 2 1 Antun Ivanovic * , Lovro Markovic , Marko Car , Ivan Duvnjak and Matko Orsag Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, 10000 Zagreb, Croatia; lovro.markovic@fer.hr (L.M.); marko.car@fer.hr (M.C.); matko.orsag@fer.hr (M.O.) Faculty of Civil Engineering, University of Zagreb, Fra Andrije Kacica-Miosica 26, 10000 Zagreb, Croatia; ivan.duvnjak@grad.unizg.hr * Correspondence: antun.ivanovic@fer.hr Featured Application: The main idea of this paper was to deploy a team of unmanned aerial vehi- cles (UAVs) to attach a sensor to a bridge using a two-component adhesive in order to perform an inspection. Constant pressure must be applied for several minutes to form a bond between two adhesives. Therefore, one UAV sprays the colored component of an adhesive while the aerial ma- nipulator transports the sensor, detects the contact point and attaches the sensor to it. A trajectory planning algorithm was developed around the dynamic model of the UAV and the manipulator attached to it, ensuring that the end-effector is parallel to the wall normal. Finally, the aerial ma- nipulator achieves and maintains contact with a predeﬁned force through an adaptive impedance control approach. Abstract: Periodic bridge inspections are required every several years to determine the state of a bridge. Most commonly, the inspection is performed using specialized trucks allowing human Citation: Ivanovic, A.; Markovic, L.; inspectors to review the conditions underneath the bridge, which requires a road closure. The aim of Car, M.; Duvnjak, I.; Orsag, M. this paper was to use aerial manipulators to mount sensors on the bridge to collect the necessary Towards Autonomous Bridge data, thus eliminating the need for the road closure. To do so, a two-step approach is proposed: an Inspection: Sensor Mounting Using unmanned aerial vehicle (UAV) equipped with a pressurized canister sprays the ﬁrst glue component Aerial Manipulators. Appl. Sci. 2021, onto the target area; afterward, the aerial manipulator detects the precise location of the sprayed area, 11, 8279. https://doi.org/ and mounts the required sensor coated with the second glue component. The visual detection is 10.3390/app11188279 based on an Red Green Blue - Depth (RGB-D) sensor and provides the target position and orientation. Academic Editor: Alessandro A trajectory is then planned based on the detected contact point, and it is executed through the Gasparetto adaptive impedance control capable of achieving and maintaining a desired force reference. Such an approach allows for the two glue components to form a solid bond. The described pipeline Received: 23 July 2021 is validated in a simulation environment while the visual detection is tested in an experimental Accepted: 1 September 2021 environment. Published: 7 September 2021 Keywords: aerial robotics; inspection and maintenance; aerial manipulation; multirotor control Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional afﬁl- iations. 1. Introduction The world of unmanned aerial vehicles (UAVs) has been rapidly growing in recent years. As their design and control are perfected, these aerial vehicles have become more and more available. Nowadays, off-the-shelf ready-to-ﬂy UAVs can be found and bought Copyright: © 2021 by the authors. in shops, which makes them available to virtually anybody. This, in turn, has sparked a Licensee MDPI, Basel, Switzerland. great deal of public interest in UAVs since their potential can be found in applications such This article is an open access article as agriculture, various inspections (bridges, buildings, wind turbines), geodetic terrain distributed under the terms and mapping, the ﬁlm industry, and even for hobby enthusiasts to ﬂy and record videos from a conditions of the Creative Commons ﬁrst-person perspective. The vast majority of commercially available UAVs are equipped Attribution (CC BY) license (https:// with a camera, while more specialized vehicles for terrain mapping or crop spraying offer creativecommons.org/licenses/by/ a more diverse sensor suite. 4.0/). Appl. Sci. 2021, 11, 8279. https://doi.org/10.3390/app11188279 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 8279 2 of 21 All of the aforementioned systems primarily observe and gather data about the environment, while having little to no ability to interact with and change the environment. One way to augment these vehicles for physical interaction is to attach a lightweight manipulator to their body, which is the main interest of the aerial manipulation ﬁeld. Although such vehicles are more complex for both modeling and control, their beneﬁt lies in performing versatile tasks that require interaction with the environment. In general, there are three types of bridge inspections: periodic, special and damage inspections. Periodic bridge inspections differ from country to country according to national standards, and are usually performed at least once every two to three years. Special inspections are typically used to monitor the condition of deﬁcient elements at speciﬁc locations based on predeﬁned requirements. Damage inspections are usually performed after events that have occurred due to environmental impacts or human actions. The aim of a bridge inspection is to evaluate and assess structural safety and reliability. Current techniques are based on traditional visual inspection with a combination of non- destructive methods (NDTs). Traditional visual inspection is performed by experienced (trained) engineers and using specialized trucks equipped with the cranes and basket, that allow inspectors to review the conditions underneath the bridge. During the inspection, the engineers are equipped with various NDT [1] tools to detect construction faults and defects such as corrosion, cracks, voids, weakening connections, and concrete delamination. Some of these NDTs require mounting small sensors to collect data, such as accelerometers, strain gauges, tilt meters and various transducers for acoustic or pressure measurements. Afterwards, the bridge is excited with vibrations, sound waves, tapping, etc., and mounted sensors record responses to these speciﬁc excitations. Furthermore, there are usually requirements for performing measurements during the bridge inspection, such as the short- and long-term monitoring of vibrations, strains, displacements, etc. Mainly, these inspections offer valuable information about the current bridge conditions, but there are a number of disadvantages. The use of trucks during inspections requires total or temporary road closures, which at the same time require safety measures to keep trafﬁc ﬂowing as freely as possible. In addition, inspectors often encounter challenges in reaching all portions or elements in narrow areas, such as tight spaces between girders, beams and vaults. The aforementioned signiﬁcantly increases the time and overall cost of the inspection. An aerial robot, with the potential to reach these challenging locations on the bridge, could signiﬁcantly reduce the time and cost of these inspections and improve worker safety. Moreover, we note that the aforementioned sensors are relatively lightweight, which makes them suitable for transportation and mounting with an aerial robot. 1.1. Concept We envision a team of robots working together to attach sensors to bridges and similar grade separation infrastructure. In theory, such a task could be accomplished with a single aerial robot, at the cost of a complex mechanical design. The proposed team shown in Figure 1 consists of two drones. One drone applies the adhesive material, and the other attaches sensors. We envision a two-stage process using two-component adhesives which form a solid bond from two separate reactive components: the “resin” and the “hardener”. The ﬁrst UAV applies the resin by spraying it onto the surface, while the second one attaches the sensor with the hardener already applied before the ﬂight. It is important to follow the prescribed ratio of the resin and the hardener to achieve the desired physical properties of the adhesive. Only when mixed together do the two components form the adhesive. The reaction typically begins immediately after the two components are mixed and the bond strength depends both on maintaining the contact and the viscosity of the mixed adhesive during the process. Manufacturers can control the cure rate to achieve various working times (worklife) until ﬁnal bond strength is achieved, ranging from minutes to weeks. Resin bases are usually more viscous than their respective hardener and are generally applied by brush, roller, applicator or spray. In this work, we propose attaching a canister of pressurized resin to the UAV, and spray it through a Appl. Sci. 2021, 11, 8279 3 of 21 nozzle onto the infrastructure surface. In this scenario, the spray needs to be softer and less turbulent to reduce the amount of material lost due to bouncing and it must be colored for the detection in the second stage. Spraying with drones is not a novel concept [2,3], so without loss of generality, we will omit the details of this design and instead focus on detecting, navigating to, and sustaining contact with the sprayed surface. In typical applications, the assemblies are usually kept in contact until the sufﬁcient strength of the bond is achieved. When fully cured, two-component adhesives are typically tough and rigid with good temperature and chemical resistance. We rely on the robotic arm attached to the second aerial vehicle to apply a controlled contact force between the sensor and the surface. Maintaining this ﬁxed assembly contact through the impedance control system enables us to achieve a successful curing process and create a permanent bond between the sensor and the infrastructure. After the ﬁrst UAV sprays the resin onto the surface, the second aerial robot ﬁnds the sprayed part and applies the contact with the sensor ’s surface. Before takeoff, the surface of the sensor is brushed with a hardener. Once contact is made, it is maintained for the prescribed curing time, after which the aerial robot disembarks and leaves the sensor attached to the surface. Figure 1. Two aerial robots working together to attach sensors to different parts of a bridge and similar grade separation infrastructure. The one on the left is used to spray the resin onto the surface, while the aerial robot on the right maintains contact to the surface with the sensor attached to its end-effector. 1.2. Contributions This paper focuses on developing a method for mounting sensors on a bridge wall using an aerial manipulator. The ﬁrst contribution is augmenting the model-based motion planning with the adaptive impedance controller. The motion planning method accounts for the underactuated nature of the multirotor UAV and corrects the end-effector conﬁgu- ration for an appropriate approach. This method also relies on the dexterity analysis which keeps the manipulator conﬁguration within its optimal region, ensuring that the manipula- tor is never fully extended or contracted while mounting a sensor. The second contribution is the visual blob detection which locates and tracks the appropriate sensor mounting point. The blob detection has been experimentally veriﬁed in an indoor environment, yielding the reliable and robust tracking of the mount location, as well as the blob plane orientation. Finally, the third contribution is the simulation analysis of the system’s performance, con- ducted on a straight and inclined wall approach. The simulation concentrates on testing the motion planning together with the impedance controller, performing a repeatability analysis and ensuring that the desired contact force is achieved. Appl. Sci. 2021, 11, 8279 4 of 21 2. Related Work In the world of aerial inspections, a number of UAV-based solutions are being pro- posed by researchers. In [4], a technical survey for bridge inspections is given. Researchers in [5] present the project AERIAL COgnitive Integrated Multi-task Robotic System with Extended Operation Range and Safety (AERIAL-CORE) which focuses on power lines inspection, maintenance and installing bird diverters and line spacers. Most of these approaches are based in conjunction with new technologies to ensure faster and cheaper in- spections. Nowadays, UAVs use high-resolution cameras for visual inspections and employ point cloud methods based on digital photogrammetry [6], Light Detection And Rang- ing (LiDAR)-based methods [7], digital image correlation [8], etc. There are also reports for visual compensation during aerial grasping [9], aerial grasping in strong winds [10], and the development of a fully actuated aerial manipulator for performing inspections underneath a bridge [11]. According to the experimental testing of contact-based bridge inspections, there is a need to develop a solution for mounting application sensors (such as accelerometers, strain gauges and tilt meters) on a bridge using a UAV. It is expected that a sophisticated system with the possibility of automatic sensor mounting will increase the frequency of measurements without interrupting trafﬁc, ensure the safety of inspectors as well as reduce inspection time and overall costs. As mentioned earlier, the second UAV needs to be aware of the position of the sprayed adhesive which is applied in a blob-like pattern. For this purpose, an Red Green Blue - Depth (RGB-D) camera is used due to its favorable dimensions and weight. It provides image and depth information about the environment which proves useful for object local- ization and UAV navigation. Such cameras were commonly found on UAVs and unmanned ground vehicles (UGVs) present at the recent MBZIRC 2020 competition. In [12,13], RGB-D information is used for color-based brick detection and localization for the wall-building challenge using UAVs and UGVs, respectively, while in [14] the authors use a Convolu- tional Neural Network (CNN)-based UAV detection and tracking method for the intruder UAV interception challenge. Furthermore, visual sensors proved useful in [15], where the authors performed a contact-based inspection of a ﬂat surface with an aerial manipulator. The surface position and orientation was obtained by applying random sample consensus (RANSAC) on the RGB-D information. A thorough survey of 2D object detection methods from UAVs was given in [16]. In this paper, the authors present a modular framework for object detection in which a simple contour-based blob detector is implemented. The goal is to use RGB-D information to enable an autonomous inspection workﬂow. The blob position is obtained by segmenting the depth data at the points where the object is detected in the image, while RANSAC [17] is used to determine its orientation. After the successful detection of a blob-like pattern, it is necessary to attach the inspection sensor. The ﬁrst phase of the sensor attachment is achieving contact, and the second is maintaining that contact to allow for the two adhesive components to form a bond. Generally, the contact can be achieved with or without force measurements. In [18], contact with the wall is performed and maintained. Researchers in [19] performed wall contact and aerial writing experiments. The work presented in [20] modeled and exploited the effects of the ceiling effect to perform an inspection underneath a bridge. The common denominator in the former approaches is maintaining the contact without any force feedback. Although mounting a force sensor on a UAV increases both mechanical and control complexity, an immediate beneﬁt is the ability to maintain precise contact force regardless of the environment. In [21], the researchers used a force/torque sensor to achieve compliant control while pulling a rope and a semi-ﬂexible bar. A fully actuated UAV with a manipulator has been employed in [22] to compare force feedback control with and without the force/torque sensor. Researchers in [23] used a single degree of freedom manipulator with a force sensor mounted at the end-effector to press an emergency switch. Relying on the blob-like pattern detection and the impedance control, a trajectory for achieving contact is required to steer the aerial manipulator towards the contact point. While mounting the sensor, it is essential that the approach and contact are perpendicular Appl. Sci. 2021, 11, 8279 5 of 21 to the wall plane. This can be considered as a task constraint imposed on the planner which the aerial manipulator has to satisfy. Researchers in [24] propose a task constrained planner for a redundant robotic manipulator that enables them to do everyday tasks such as opening drawers or picking up objects. In [25], a task-constrained planner was developed for underactuated manipulators. Since multirotor UAVs are typically underactuated systems, it is necessary to address dynamics and kinematics while planning the end- effector trajectory. Aerial manipulator 6D end-effector trajectory tracking based on the differential ﬂatness principle was presented in [26]. The underactuated nature of multirotor UAVs can cause unexpected deviations in end-effector conﬁguration. Researchers in [27] address this particular problem by including the dynamic model of the system into the planning procedure. In our previous work [28], a trajectory planning method based on the full dynamic model of an aerial manipulator was developed. In this paper, we further augmented this method to plan for the desired force required by the impedance controller. 3. Mathematical Model In this section, the mathematical model of the aerial manipulator is presented. The coordinate systems convention is depicted in Figure 2. Furthermore, an analysis for the manipulator dexterity and reach was performed. 3.1. Kinematics The inertial frame is deﬁned as L . The body-ﬁxed frame L is attached to the W B center of gravity of the UAV. The position of the UAV in the world frame is given h i h i T T with p = x y z 2, and the attitude vector is Q = f q y . Combining the position and attitude vectors deﬁned the generalized coordinates of the UAV as h i 6 B B T T q = 2 R . Written in a matrix form T , the transformation contains (p ) Q W W both the position and orientation of the UAV obtained through an on-board sensor fusion b 44 or through an external positioning system (i.e., GPS). The notation T 2 R was used to denote a homogeneous transformation matrix between frames a and b. A rigid attachment between the body of the UAV and the base of the manipulator L was considered, denoted with the transformation matrix T . The manipulator used in this work was a M = 3 degree-of-freedom (DoF) serial chain manipulator with the end-effector attached to the last joint. The DH parameters of the arm are given in Table 1. Using this ee notation, one can write a transformation matrix T between the manipulator base and its end-effector as a function of joint variables q , q and q . For brevity, the expression for 1 2 3 ee the entire matrix T is left out and only the end-effector position and its approach vector equations are written using the well-known abbreviation cos(q + q ) := C : 1 2 12 2 3 2 3 a (C + C ) + d C C 1 1 12 3 123 123 6 7 6 7 ee ee p = a (S + S ) + d S , z = S (1) 4 5 4 5 1 1 12 3 123 123 0 0 0 0 Table 1. DH parameters of the 3-DoF manipulator attached to the UAV. A virtual joint q is added to fully comply with the DH convention. Link sizes a and d are omitted for clarity. 1 3 Joint q d a a q 0 0 0 a 1 1 q 0 0 0 a 2 1 p p q 0 0 2 2 q 0 d 0 0 4 Appl. Sci. 2021, 11, 8279 6 of 21 L 0 2 z ee Figure 2. Coordinate systems of the world, UAV and the 3-DoF manipulator. Putting it all together, the full kinematic chain of the aerial manipulator can be con- structed as ee B 0 ee T = T T T , (2) W W B 0 0 B ee combining the ﬁxed transformation T with T and T depend on UAV and manipu- B W 0 lator motion. Since there is obvious coupling between the motion of the body and the manipulator arm, a b 2 [0, 1] parameter is introduced to distribute the end-effector motion commands either to the UAV global position control or manipulator joint position control. To this end, the following distribution relationship is used: DP = b DP UAV DP = (1 b) DP, (3) arm where DP is used to denote the desired aerial manipulator displacement expressed as the following combination body and arm motion: DP = DP + DP UAV arm = b DP + (1 b) DP. (4) The manipulator displacement is denoted by DP and the UAV displacement by arm DP , where both DP and DP are expressed in the coordinate system L . With UAV arm UAV 0 b = 1, the UAV motion is used to control the position of the end-effector. When b = 0, the situation is reversed and the manipulator motion is used to move the end-effector. For every other b, the end-effector motion is obtained in part by the UAV body and the manipulator arm motion. There are obvious advantages in combining the motion of the UAV and the manip- ulator arm. The UAV can move in 3D space beyond the reach of the arm; however, the motion of the UAV is not as precise and dynamically decoupled. The kinematics of the ee T arm enable the end-effector to obtain the desired approach angle z = [cos(d), sin(d), 0] , ee which under the hovering assumption, becomes equal to the global approach vector z pointing towards the contact point on the infrastructure. The straightforward mathematical manipulation of Equation (1) allows for writing the constraint equation: q = d q q , (5) 3 1 2 which ensures that the manipulator points in the right direction, where d is the desired manipulator inclination in the body x–z plane. Appl. Sci. 2021, 11, 8279 7 of 21 To ﬁnd the optimal manipulator pose during contact, the dexterity D and the reach R of the pose were taken into account, while considering that the joints are as far as possible from their physical limits L. Since the motion of the arm is constrained with its approach h i ee ee dp dp 0 0 axis condition, a reduced form of a Jacobian matrix was used J = , to derive the dq dq 1 2 pose dexterity index D = J J and determine how far the current pose is from the null ee T ee space of the manipulator [29]. The reach of the pose R = (p ) p was also taken into 0 0 account, since the goal was to keep the end-effector and the contact point as far away from the UAV body. Finally, the following equation is deﬁned: 2 2 2 2 (q Q )(q Q ) 1max 2max 1 2 L = , (6) 2 2 Q Q 1max 2max to measure how far away the given conﬁguration is (i.e., q , q ) from the joint limits 1 2 Q , Q . Normalizing D,R and L enables combining the three conditions into a 2max 1max h i single manifold M = D R L and ﬁnd the optimal conﬁguration q = for q q q M 1 2 3 the desired approach angle d. The described method is depicted in Figure 3 for the speciﬁc case of the approach angle d = 0 , but can be extended to any value of the approach angle through Equation (5). As a side note, the manipulator attachment on the top of the UAV body was chosen to be able to reach surfaces underneath the bridge. Although this shifts the center of gravity upwards, the stability of the system is not compromised since the manipulator is constructed of lightweight materials. Figure 3. The visual decomposition of dexterity D, reach R, limit L and the overall combined surface. This analysis is performed for d = 0 : (a) the dexterity D surface shows the measure of how far the manipulator is from the null space. Values around zero are closer to the null space; (b) the reach R surface shows how far the end-effector can move in a certain conﬁguration. This value tends towards zero as the arm approaches a folded conﬁguration; (c) the limit L depicts how far a certain conﬁguration is from the physical limits of the manipulator joints; and (d) the combined manifold M of the formerly described surfaces. Higher values offer better trade-off between dexterity, reach and limit, deﬁning the optimal manipulator conﬁguration q . 3.2. Dynamics The most complicated task of the aerial manipulator is attaching the sensor to a wall and maintaining the required force reference while the two-component adhesive hardens. Appl. Sci. 2021, 11, 8279 8 of 21 To successfully perform such a task, the coupled UAV-manipulator system dynamics have to be addressed for the precise end-effector conﬁguration planning. Considering the UAV dynamics only, the derivative of the generalized coordinates h i 6 B T B T B T can be deﬁned as q ˙ = (p ˙ ) (w ) 2 R . Here, the (p ˙ ) is the linear velocity of W W B T the body of the UAV in the world frame and the (w ) represents the angular velocity of the UAV in the world frame. The UAV’s propulsion system consists of n propellers rigidly attached to the body. Each propeller produces force and torque along the z axis. The vector of the propeller rotational velocities is simply deﬁned as h i W = W . . . W 2 R . (7) UAV 1 n Force and torque produced by each propeller are non-linear functions depending on the rotational velocity W . Rather than using the rotational velocities as control inputs, UAV they can be mapped to a more convenient space. Namely, the mapped control input space can be written as u = K diag(W ) W , (8) UAV UAV UAV h i 4n where K 2 R is the mapping matrix and u = u u u u , where u UAV 1 2 3 4 4 represents the net thrust and u , u and u are moments around the body frame axes. 1 2 3 As stated earlier, the manipulator consists of three rotational DoFs. Therefore, the joint h i positions of the manipulator are deﬁned as q = q q q . The rotational velocity of 1 2 3 each joint is a time derivative of joint positions q ˙ = dq /dt. The torque of each joint is M M h i considered the control input of the manipulator u = . M t t t 1 2 3 The resulting generalized coordinates of the aerial manipulator can be written as h i q = 2 R , and the velocities can be obtained in the same manner as q q UAV M h i q ˙ = 2 R . The resulting control inputs of the system can be expressed as q ˙ q ˙ UAV M h i u = u u 2 R . Finally, the full system dynamics can be written as UAV M M(q)q ¨ + c(q, q ˙ ) + g(q) = u, (9) 97 7 where M(q) 2 R is the inertia matrix, c(q, q ˙ ) 2 R is the vector of centrifugal and Coriolis forces, g(q) 2 R is the gravitational term. 4. Control System The overall control of the aerial manipulator consists of several nested control loops. The complete controller overview, with motion planning and blob detection blocks, is depicted in Figure 4. 4.1. Aerial Manipulator Control At the inner most level, the UAV is controlled through cascade attitude and rate controllers. The input to these controllers is the desired orientation and based on the state, the output is the vector of the rotors’ angular velocities. The second level of control, which uses the inner attitude control loop, consists of two additional cascades, the position and the velocity control. These controllers receive a referent position and velocity feed-forward value to generate the desired vehicle orientation and thrust. The manipulator joints are controlled through standard Proportional, Integral, Derivative (PID) controllers; however, in a real-world setting, servo motors with integrated control are typically used. As mentioned earlier, it is important to track the desired force after contact with a wall is achieved. To accomplish this, an adaptive impedance controller is employed to generate an appropriate setpoint for the position controller. This controller receives a trajectory Appl. Sci. 2021, 11, 8279 9 of 21 supplied by the mission planner, which steers the aerial manipulator towards the sensor mounting target on the bridge. Aerial manipulator control Position and Attitude and velocity control rate control Manipulator control Adaptive Motion Blob detection impedance planning and estimation control Figure 4. The overall functional schematic of the system. The aerial manipulator control subsystem is necessary for the controlling position and attitude of the UAV, and manipulator joints. On top of this controller, the adaptive impedance control is employed in order to track the desired force. Motion planning generates an appropriate trajectory based on the target point supplied by the blob detection algorithm. 4.2. Adaptive Impedance Control The objective of the adaptive impedance controller is to ensure a stable physical interaction between the aerial manipulator and the environment [30]. As mentioned earlier, the standard UAV control scheme is based on position and attitude controllers. When interacting with the environment, the desired contact force must be considered. The position controlled system can be extended to follow the desired force by introducing an impedance ﬁlter. The design of such a ﬁlter is explained here for a single DoF. The behavior of the system is deﬁned by the target impedance as e(t) = m(x¨ (t) x ¨ (t)) + b(x ˙ (t) x ˙ (t)) + k(x (t) x (t)), (10) c r c r c r where m, b and k are constants, x (t) is the referent position, provided to the impedance ﬁlter as an input, and x (t) is the output of the impedance ﬁlter representing the position command. The ﬁlter is designed as a linear second-order system with a dynamic rela- tionship between the position and the contact force tracking error e(t) so that it mimics a mass-spring–damper system. The contact force tracking error is deﬁned as follows: e(t) = f (t) f (t), (11) where f (t) is the other ﬁlter input deﬁning the referent force, and f (t) is the measured (ex- erted) contact force. If the environment is modeled as a ﬁrst-order elastic system (equivalent spring system) with unknown stiffness k , the measured force can be approximated as f (t) = k (x(t) x (t)), (12) e e where x(t) is the position of the manipulator and x (t) is the position of the environment in an unexcited state. By substituting Equation (12) in Equation (11), the position of the aerial manipulator can be expressed as follows: f (t) e(t) x(t) = + x (t). (13) e Appl. Sci. 2021, 11, 8279 10 of 21 Assuming that the commanded position value can be achieved by the aerial manipula- tor, i.e., x = x , the substitution of Equation (13) in Equation (10), the system in the steady state can be described as follows: k k f (t) e r e(t) = + x (t) x (t) . (14) e r k + k k e e For a contact force error of zero in the steady state, the following must hold: f (t) x (t) = + x (t). (15) r e In other words, the position setpoint has to be designed in such a way that it compen- sates for the displacement of the environment due to the exerted contact force. To ensure this, a value of the unknown environment stiffness k is needed. Furthermore, k plays a e e fundamental role in the stability of the impedance ﬁlter Equation (10), which ultimately affects the stability of the aerial manipulator while in contact with the environment. A sta- ble contact between the aerial manipulator and the environment can be ensured using the Hurwitz stability criterion, by designing the system with b/m > 0 and (k + k )/m > 0. However, since k is unknown, an adaptation law for the position setpoint that guarantees the contact stability while compensating for this hidden, unknown parameter is proposed. The adaptation law is derived starting from Equation (15). An adaptive parameter k(t) is introduced so that: x (t) = k(t) f (t) + x (t). (16) r r e It can be shown using the Lyapunov stability analysis that the following adaptation dynamics equation for k(t) will yield a stable system response: ... kk ˙(t) + bk ¨(t) + mk (t) = gs(t) + g s ˙ (t). (17) We refer the interested reader to the proof which can be found in the Appendix A. 5. Motion Planning As discussed in Section 1.1, the main concept of this paper was to use a team of two UAVs, each applying one component of the adhesive. To apply the “resin” component, the UAV has to plan a collision-free trajectory and position itself in front of the target area to start spraying. This is fundamentally different from mounting a sensor coated with “hardener ”. In the latter case, apart from planning a collision-free trajectory, the manipulator-endowed UAV has to apply pressure for a certain amount of time for the two components to mix. From the perspective of motion planning, the planner needs to be augmented to include a manipulator with three degrees of freedom, contact force and the weighing parameter b. To successfully maintain the pressure, the planner relies on the impedance controller described in Section 4.2. Furthermore, one of the requirements when mounting the sensor on the wall is for the sensor to be perpendicular to the wall. Therefore, it is necessary to take the underactuated nature of the multirotor UAVs into account during the motion planning. Namely, the errors in the planned end-effector conﬁguration were mainly induced due to the roll and pitch angles while executing the planned motion. In our previous work [28], we developed a model-based motion planner for aerial manipulators that is capable of correcting the aforementioned end-effector deviations. In this paper, the idea from [28] was extended to consider the impedance control when obtaining the full state of the aerial manipulator. Appl. Sci. 2021, 11, 8279 11 of 21 5.1. Waypoint Conﬁguration When dealing with an aerial manipulator, exerting some contact force inevitably yields a high dimensional waypoint conﬁguration. We deﬁne a single waypoint as a set of UAV and joint poses, together with the force reference and motion distribution factor b: h i T T T w = q q f b 2 R , (18) B M r 6 3 where q 2 R and q 2 R are generalized coordinates of the UAV and the manipulator B M h i deﬁned in Section 3.1. The force reference vector f = 2 R and weighing f f f r x y z scalar parameter b are required by the impedance controller. Furthermore, the impedance controller assumes a step change of these values. Ideally, the change should occur at the moment of contact since no force can be exerted without contact. Therefore, these values are only changed at the ﬁnal waypoint. Apart from the desired force and the parameter b, the ﬁnal waypoint must contain the UAV position and orientation, as well as the manipulator joint conﬁguration. Specifying these values relies on the blob detection algorithm presented in Section 6. Namely, the algorithm outputs the position and orientation of the detected blob in the world frame. Following the manipulator dexterity and reach analysis described in Section 3.1, the optimal manipulator conﬁguration q is obtained based on the provided plane normal. The optimal manipulator conﬁguration is then used as the desired conﬁguration for the ﬁnal waypoint. This way, during operation, the manipulator never reaches a fully extended or contracted pose, which allows the impedance controller to command both the arm and the UAV to achieve and maintain the desired force. 5.2. Trajectory Planning There are three phases in the trajectory planning procedure. First, an initial trajectory is planned based on the provided waypoints. Second, the initial trajectory is sent to a simulated model in order to obtain the full state of the aerial manipulator during the trajectory execution. Third, the end-effector conﬁguration is corrected based on the full state of the vehicle, and the ﬁnal trajectory is sent to the target aerial manipulator. Initial Trajectory To execute a smooth motion towards the desired waypoint, we use a suitable time- optimal path planning by reachability analysis (TOPP-RA) trajectory planner [31]. The TOPP-RA algorithm searches for the time-optimal trajectory and is based on a “bang-bang” principle on the generalized torque of each DoF. The planner is capable of receiving the input waypoints of an arbitrary dimension and output a smooth trajectory. Each DoF has to be provided with dynamical constraints in terms of velocity and acceleration, which are respected during the trajectory generation process. As mentioned, the input to the TOPP-RA trajectory is the path of a set of n 2 waypoints: n o P = w j w 2 R , i 2 (0, 1, . . . , n) . (19) i i Based on the dynamical constraints, the output of the TOPP-RA planner is a sampled trajectory: n o 313 T = t(kT ) j t(kT ) 2 R , k 2 (0, . . . , n ) , (20) s s s t h i 313 T T T where t = (w) (w ˙ ) (w ¨ ) 2 R is a single sampled trajectory point consisting of position, velocity and acceleration; T is the sampling time; and n is the number of s t points in the sampled trajectory. Note that each trajectory point contains both roll and pitch angles. Although these angles can be planned through the TOPP-RA algorithm, Appl. Sci. 2021, 11, 8279 12 of 21 they are omitted at this point because of the underactuated nature of the multirotor UAV. Nevertheless, they are used later in the paper when the model corrections are applied. The impedance controller expects a step change in the force and weighing parameter b referent values. To satisfy this requirement, large constraints for velocity and acceleration are imposed for these DoFs. However, because other DoFs have constraints below their physical limit, the overall force and b trajectory has a slower, dynamically smooth proﬁle. These proﬁles also have overshoots and undershoots which are not acceptable because they are not within the hard constraints required for b. To tackle this problem, a simple piecewise constant velocity interpolation was applied to the force and b. This way, a large velocity constraint produces a step change which is a suitable input to the impedance controller. A visual example of the difference between the TOPP-RA and piecewise constant velocity interpolation is depicted in Figure 5. 0.8 TOPP-RA 0.6 Constant velocity 0.4 0.2 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 t[s] Figure 5. Visual comparison between TOPP-RA and the piecewise constant velocity interpolation. Waypoints for both trajectories are kept the same, and around t = 2.5 s, the middle waypoint(yellow cross) is reached. Although the dynamical constraints are the same, TOPP-RA takes other degrees of freedom into account and produces a trajectory with overshoot which is not suitable for parameter b. 5.3. Model-Based Corrections The initial trajectory from Equation (20) is planned without any consideration about the underactuated nature of the multirotor UAV. To obtain the unknowns, namely roll and pitch angles, the initial trajectory can be executed in a simulation environment. The chosen simulation environment is, in our case, Gazebo, because it is realistic and supports the robotics operating system (ROS), which is the backbone of our implementation. The simulated aerial manipulator is based on the mathematical model described in Section 3. The standard cascade PID controllers are employed for low-level attitude and high-level position control. The impedance controller is built on top of the position controller and pro- vides a position reference based on the input trajectory. More details about the simulation environment are provided in Section 7. The ﬁrst step is executing the initial trajectory in the aforementioned simulation environment. While executing, the roll and pitch angles are recorded as they are needed for obtaining the full state of the UAV. Rearranging Equation (2) and plugging the unknown roll and pitch angles in the full state of the UAV, the transform of the end-effector in the manipulator base frame can be obtained: ee 1 B 1 ee T = (T ) (T ) T . (21) L B W W Usingthe inverse kinematics of the manipulator, joint values q for the desired end- effector conﬁguration are obtained. This way, the null space of the aerial manipulator is used for the end-effector correction. Note that due to the conﬁguration of the manipulator, an exact solution of the inverse kinematics will not always exist. In such a case, an approximate closest solution is used instead. The ﬁnal trajectory is constructed by replacing the initial q with the corrected values. This trajectory is afterwards sent to the target aerial manipulator. The careful reader should note that the developed three DoF manipulator operates on the x and z position in the body frame, as well as the pitch angle. This allows the impedance controller to maintain the orientation perpendicular to the wall, while compensating for the - Appl. Sci. 2021, 11, 8279 13 of 21 UAV body motion in the x and z axes. However, the system will experience disturbances and control errors which will act on the roll and pitch angle, and the lateral movement along the body y axis. We can address these issues either with mechanical dampers or by adding additional degrees of freedom to the manipulator, which will be explored in future work. 6. Blob Detection This section presents the methods we propose to detect the hardener blob position and orientation. A modular object detection framework, as shown in Figure 6, is designed to ensure a reliable blob pose detection. Since the detection is to be done on board the UAVs, RGB-D cameras are selected. Therefore, the inputs to the framework are images and organized point clouds obtained from the visual sensor. The remainder of this section introduces the individual components of the framework and adds implementation details where necessary. Figure 6. The pipeline for the modular object detection framework. Inputs to the system are an arbitrary number of sensor messages (image, depth, point cloud, etc.) along with sensor–world frame transformations. Output is the detected blob pose in the world frame. The synchronizer and detector are modular components, while the linear Kalman ﬁlter, world transformation and pose tracker stay invariant. The sensor message synchronizer is responsible for the time-based synchronization of the given sensor message streams. In the case of blob detection, a module that synchronizes images and organized pointclouds from an RGB-D camera is derived. This is necessary since the algorithm detects the blob in both 2D image space and 3D point clouds, which are not necessarily sampled simultaneously. The underlying implementation uses ROS libraries to synchronize messages with an approximate time policy. An object detector attempts to ﬁnd a set of object poses using synchronized sensor data. The module used in this paper detects blob poses and is implemented in the following way. First, all the blob positions and radii are found in the image frame using the standard blob detection functionality found in the OpenCV libraries. Second, the depth information corresponding to the detected blobs is isolated from the organized point cloud. Finally, blob positions are calculated as centroids of the corresponding depth positions, while the orientation is obtained through the random sample consensus (RANSAC) algorithm from the Point Cloud Library (PCL). The remaining framework components are independent from synchronizer and detec- tor modules. The pose tracker is used to track the obtained object’s poses through multiple frames based on the closest Euclidean distance criterion. This component solves the issue of multiple objects being visible, as it always outputs the pose of the currently tracked object. Moreover, it increases the robustness of the system since it remembers the object poses for a certain number of frames, which allows some leniency with the detector. The goal of the world transformation component is to transform the tracked pose from the sensor to the world frame using the estimated odometry from an external source Appl. Sci. 2021, 11, 8279 14 of 21 that any UAV should have access to. Additionally, since the blob poses are to be sent as references to the trajectory planner, it is important to correctly compute the blob orientation. Since the blob is a ﬂat surface, there are two equally correct possible orientations that can be detected. Therefore, the blob orientation is chosen as follows: R if r r 0 1B blob 1blob R = , (22) blob R R otherwise blob 180 where r is the heading component of the UAV rotation matrix expressed in world 1B h i coordinates R = and r is the heading component of the blob rota- r r r B 1blob 1B 2B 3B h i tion matrix expressed in world coordinates R = and R = r r r blob 180 1blob 2blob 3blob diag(1,1, 1). Finally, a linear Kalman ﬁlter with a constant velocity model is used to further increase the robustness of the system and provide smoother blob position estimates. The constant velocity model for each axis is given as follows: " # 1 T x = F x + w , F = , (23) k+1 k k k k 0 1 where T is the discretization step, x 2 R is the state vector containing the position and velocity along the corresponding axis and w 2 R is the process noise. The observation model along a single axis is given as follows: h i z = H x + v , H = , (24) 1 0 k x k k k where z 2 R is the position observation along the corresponding axis and v 2 R is the k k measurement noise. If the detector is unable to provide measurements and the pose tracker removes the pose from the tracking set, the linear Kalman ﬁlter is still able to provide blob position estimates. Experimental validation of the described methods is performed in an indoor Optitrack environment with an Intel Realsense D435 RGB-D camera. To ensure ground truth is available for detection validation, reﬂective markers are attached to both the camera and the blob. In order to determine the transformation between the camera optical frame and the reﬂective markers attached to the camera, an optimization-based calibration approach is used as described in [32]. Results are shown in Figures 7 and 8. The experiments are performed with the UAV in constant motion while looking at the general direction of the painted blob. Figure 7 shows a relative difference between the ground truth UAV motion in the world frame and the UAV motion as observed from the detected blob frame. Figure 8 presents the comparison of ground truth and detected blob positions expressed in the world frame. It is important to note that camera calibration errors can manifest themselves as static offsets between the detected and ground truth blob positions in Figure 8. However, in this case, the visual detection provided a reliable blob tracking results which is a direct consequence of careful camera calibration. Appl. Sci. 2021, 11, 8279 15 of 21 0.4 0.2 -0.2 -0.4 0.6 0.4 0.2 -0.2 W W p [k] - p [0] B B 0.2 blob blob p [k] - p [0] B B -0.2 0 5 10 15 20 25 30 35 40 45 Time / [s] Figure 7. This ﬁgure shows the comparison of normalized motion of the UAV body frame as observed W W from the Optitrack world frame and as observed from the detected blob frame labeled p [k] p [0] B B blob blob and p [k] p [0], respectively. B B 0.5 0.4 0.3 0.2 0.1 -0.1 -0.2 0.2 p - ground truth blob 0.1 p - estimate blob -0.1 5 10 15 20 25 30 35 40 45 Time / [s] Figure 8. This ﬁgure shows the comparison of the ground truth blob position and the blob position estimate expressed in the Optitrack world frame. The root mean squared error across all three axes is 4.6 cm. Z / [m] Y / [m] Z / [m] Y / [m] X / [m] X / [m] Appl. Sci. 2021, 11, 8279 16 of 21 7. Simulation The environment used for simulating the UAV and manipulator dynamics, as well as the contact with the environment, is the widely accepted Gazebo simulator. It is real- istic and highly modular, with a large community and a support for the robot operating system (ROS), which is also the primary implementation environment for impedance control, motion planning and blob detection. Through ROS, Gazebo has a large variety of developed plugins realistically simulating various sensors and actuators. All simu- lations were conducted with Linux Ubuntu 18.04 operating system and ROS Melodic middleware installed. The UAV is modeled as a single rigid body with n propellers mounted at the end of each arm. As propulsion units, these propellers generate thrust along the z axis of the UAV body. To simulate the propeller dynamics, the rotors_simulator package is used. It contains a plugin that models thrust based on the user-provided propeller parameters [33]. Furthermore, to obtain the UAV attitude and position, IMU and odometry plugins are mounted on the vehicle. The manipulator was mounted on the body of the UAV and consists of three joints connected with links. A rod type tool is mounted as the end-effector, with a force-torque sensor required by the impedance controller. Furthermore, a monocular camera with an infrared projector is also mounted for the blob detection. 7.1. End-Effector Motion Distribution Analysis Given some end-effector conﬁguration, the inverse kinematics is responsible for ﬁnding the UAV position and yaw angle, as well as the manipulator joint values that satisfy the desired conﬁguration. The parameter b from Equation (3) deﬁnes a ratio of how much the manipulator joints and UAV position and orientation contribute to achieving the desired end-effector conﬁguration, as described in Section 3.1. Recalling the values, b = 1 only moves the UAV in the direction of the desired end-effector conﬁguration; and b = 0 uses the inverse kinematics of the manipulator to achieve the desired conﬁguration. To determine the inﬂuence of b on the overall system, an analysis was conducted with different b values. The desired end-effector conﬁguration was chosen to be in contact with a plane perpendicular to the bridge wall which required the force reference along the x axis. The waypoints for the trajectory planner were kept the same across all trials, and only b was changed. The results of this analysis are depicted in Figure 9. As can be observed, all trials produced very similar results with oscillating force upon contact and eventually reaching the desired reference, providing us with no obvious conclusion regarding how to select the optimal b. However, following the dexterity analysis from Section 3.1, and only relying on the manipulator motion might drive the system close to its limits due to the UAV body movement. On the other hand, the motion of the UAV induces disturbances in the end-effector pose control. The manipulator is therefore responsible for compensating errors introduced by the motion of the UAV body. Taking all of the aforementioned into account, the value is chosen as b = 0.5 so that both the manipulator and the UAV are simultaneously used to maintain a steady contact force. 0.2 -=0 -=0.2 -=0.4 -=0.5 -=0.6 -0.2 -=0.8 -=1 Reference -0.4 -0.6 -0.8 -1 0 5 10 15 20 25 30 t[s] Figure 9. Force response comparison for different values of the parameter b. The analysis was conducted on a plane perpendicular to the ground where the force reference along the x axis is required. f [N] x Appl. Sci. 2021, 11, 8279 17 of 21 7.2. Bridge Sensor Mounting Since the concept of this paper was to mount inspection sensors on a bridge, the simulation trials were tailored in the same direction. After spraying the ﬁrst component, it is necessary to achieve and maintain a stable contact while the second adhesive component on the sensor dries. Since the manipulator is attached above the propellers, the workspace of the manipulator is limited to contact above the UAV or on the plane perpendicular to the ground. Naturally, the ﬁrst set of simulation trials were conducted by holding the desired force on a plane perpendicular to the ground. In this case, the contact force only acts along the x axis and the response is depicted in Figure 10. The time delay between the planned and executed contact is present due to the impedance ﬁlter which slows down the dynamics of the referent trajectory. After the initial contact, there are some oscillations and an overshoot which diminish over time and the desired force reference is achieved. Measured Reference -0.2 -0.4 -0.6 -0.8 0 5 10 15 20 25 30 35 40 45 50 t[s] Figure 10. Force response in case of a contact plane perpendicular to the ground, d = 0 . The second set of simulation trials included an inclined contact plane. This requires the UAV approach from below the plane and achieving contact perpendicular to the plane. Since the plane is inclined for d = 68 , the planned force referent values have components in both the x and z axes, as shown in Figure 11. Similarly to the previous example, the force response has some oscillations around the instance of contact, but it eventually settles and reaches the desired force reference. Measured -0.05 Reference -0.1 -0.15 -0.2 -0.25 -0.3 0 5 10 15 20 25 30 35 40 45 50 t[s] 0.5 0.4 0.3 0.2 0.1 -0.1 0 5 10 15 20 25 30 35 40 45 50 t[s] Figure 11. Force response in the case of the contact plane inclined for d = 68 . The simulation tests for d = 0 and d = 68 were performed n = 10 times for each case, as depicted in Figure 12. The left portion of the ﬁgure is a dot product between the normal of the blob r and the end-effector orientation vector r . If the value of the dot t ee product r r = 1, the two vectors are parallel which results in a successful approach. For t ee f [N] f [N] f [N] x x z Appl. Sci. 2021, 11, 8279 18 of 21 both angles, the dot product is very close to 1 and the orientation error is negligible. On the right, the distance between the center of the target and contact point is shown. The error distance is in both cases less than 0.1 m, which ensures the relatively high precision of sensor mounting, well within margins for the bridge inspection. The accompanying video of simulation tests can be found on our YouTube channel [34]. Figure 12. (Left): box and whiskers plot of the dot product between the blob plane normal vector r and the end-effector orientation vector r ; and (Right): box and whiskers plot of the distance from ee the target point after contact. 8. Conclusions This paper presents a step towards autonomous bridge inspection by investigating the possibility of mounting various inspection sensors using an aerial manipulator. Cur- rently, inspectors use specialized trucks with cranes and baskets in order to access the area underneath the bridge. This inevitably leads to road closure which poses an inconvenience for both inspectors and trafﬁc. To alleviate this problem, the aforementioned aerial manip- ulators can be used to access difﬁcult-to-reach areas of the bridge. As mounting sensors require forming a bond between the wall and sensor, we envision using a two-component adhesive with a short cure time. Since the aerial manipulator has to achieve and maintain contact with the sensor mount point, short cure times are desirable because of te limited ﬂight time of these platforms. Nevertheless, current ﬂight times of outdoor multirotors reach up to 30 min, which ensures enough time for the two adhesive components to form the bond. Although preliminary, the results of this paper seem promising. The visual detection was extensively tested and reliably tracks the blob position. The adaptive impedance controller is capable of maintaining the required force. Even though there are some oscillations and settling times in the force response, in practical use, it does not make much difference since the curing time of the adhesive is at least several minutes. The trajectory planner was augmented to plan in the force space which allows for setting the force reference step change before the contact. The simulation results show the high repeatability of the overall system which gives us the conﬁdence to perform experiments in a real-world environment. Our ﬁrst step in future work was to perform experiments in a controlled laboratory environment. The outdoor environment poses a different set of challenges including the lower accuracy positioning system and unpredictable disturbances, i.e., wind gusts. Since these factors will inevitably reﬂect on the overall end-effector accuracy, we are looking into augmenting the manipulator to be able to compensate for lateral movements, as well as roll and yaw angles. To further increase the system’s accuracy, the developed visual Appl. Sci. 2021, 11, 8279 19 of 21 tracker will be used to improve feedback around the tracked blob on the bridge wall in real-world experiments. Author Contributions: All authors contributed equally to this work. Conceptualization, M.O. and I.D.; methodology, A.I., L.M. and M.C.; software, A.I., L.M. and M.C.; investigation, A.I.; writing— original draft preparation, A.I.; writing—review and editing, L.M., I.D. and M.O.; supervision, I.D. and M.O.; project administration, I.D. and M.O.; funding acquisition I.D. and M.O. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the European Commission Horizon 2020 Programme through the project under G.A. number 820434, named Energy-Aware BIM Cloud Platform in a Cost-Effective Building Renovation Context—ENCORE. Furthermore, this research was part of the scientiﬁc project Autonomous System for Assessment and Prediction of Infrastructure Integrity (ASAP), ﬁnanced by the European Union through the European Regional Development Fund—The Competitiveness and Cohesion Operational Programme (KK.01.1.1.04.0041). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Conﬂicts of Interest: The authors declare no conﬂict of interest. Appendix A Here, we show detailed proof that the adaptation law (17) is stable. Assuming that THE referent force f (t) is constant, f (t) = F , the time derivatives of Equation (16) are: r r r x˙ (t) = k ˙(t)F + x˙ (t), r r e (A1) x¨ (t) = k ¨(t)F + x¨ (t), r r e while the derivatives of Equation (13) yield: e(t) ˙ ˙ x(t) = + x (t), (A2) e ¨(t) x ¨(t) = + x¨ (t). By susbstituting Equations (A1) and (A2) into Equation (10), the dynamics of the contact force error can be obtained as me ¨(t) + be˙(t) + (k + k )e(t) = g(t), (A3) where: ˙ ¨ g(t) = F [k(1 k k(t)) k (bk(t) + mk(t))], (A4) r e e and x = x . The adaptation law thus determines the dynamics of the adaptation parameter k(t) and deﬁnes the dynamics of the contact force error. Formally, the adaptation law should enforce g(t) ! g such that e(t) ! 0 and k(t) ! 1/k . For the Lyapunov candidate: h i 1 1 2 2 V(t) = p e(t) + p e˙ (t) + [g(t) g ] , (A5) 1 2 2 2g with p , p and g as positive parameters, the condition V(t) 0 yields: 1 2 2 2 g(t)g ˙(t) + 2g(t)[ p e(t) + p e˙(t)] g ˙(t)g 0. (A6) 1 2 g g Appl. Sci. 2021, 11, 8279 20 of 21 After reordering, we obtain: g ˙(t)[g(t) g ] gs(t)g(t), (A7) where s(t) = [ p e(t) + p e˙(t)]. By choosing: 1 2 g ˙(t) = gs(t) + g s ˙ (t), (A8) where g is a positive constant, Lyapunov condition Equation (A7) becomes: g(t) g , (A9) i.e., for the adaptation law to be stable, g(t) should be bounded. Since x , x ˙ and x ¨ r r r are bounded, so are e, e˙ and e ¨. Therefore, g(t) is also bounded, i.e., the condition in Equation (A9) is satisﬁed. The adaptation law is ﬁnally obtained by taking the derivative of Equation (A3), and substituting g ˙(t) with Equation (A8), yields the (17). Parameters g and g dictate the adaptation dynamics. Based on the measured contact force, the error adaptation law Equation (17) estimates the adaptation parameter k (reciprocal value of the environment stiffness), which is then used in Equation (16) for calculating the referent position x . References 1. Kashif Ur Rehman, S.; Ibrahim, Z.; Memon, S.A.; Jameel, M. Nondestructive test methods for concrete bridges: A review. Constr. Build. Mater. 2016, 107, 58–86. [CrossRef] 2. Hunt, G.; Mitzalis, F.; Alhinai, T.; Hooper, P.A.; Kovac, M. 3D printing with ﬂying robots. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 4493–4499. [CrossRef] 3. Vempati, A.S.; Kamel, M.; Stilinovic, N.; Zhang, Q.; Reusser, D.; Sa, I.; Nieto, J.; Siegwart, R.; Beardsley, P. PaintCopter: An Autonomous UAV for Spray Painting on Three-Dimensional Surfaces. IEEE Robot. Autom. Lett. 2018, 3, 2862–2869. [CrossRef] 4. Jeong, E.; Seo, J.; Wacker, J. Literature Review and Technical Survey on Bridge Inspection Using Unmanned Aerial Vehicles. J. Perform. Constr. Facil. 2020, 34, 04020113. [CrossRef] 5. Cacace, J.; Orozco-Soto, S.M.; Suarez, A.; Caballero, A.; Orsag, M.; Bogdan, S.; Vasiljevic, G.; Ebeid, E.; Rodriguez, J.A.A.; Ollero, A. Safe Local Aerial Manipulation for the Installation of Devices on Power Lines: AERIAL-CORE First Year Results and Designs. Appl. Sci. 2021, 11, 6220. [CrossRef] 6. Zollini, S.; Alicandro, M.; Dominici, D.; Quaresima, R.; Giallonardo, M. UAV Photogrammetry for Concrete Bridge Inspection Using Object-Based Image Analysis (OBIA). Remote Sens. 2020, 12, 3180. [CrossRef] 7. Bolourian, N.; Hammad, A. LiDAR-equipped UAV path planning considering potential locations of defects for bridge inspection. Autom. Constr. 2020, 117, 103250. [CrossRef] 8. Reagan, D.; Sabato, A.; Niezrecki, C. Unmanned aerial vehicle acquisition of three-dimensional digital image correlation measurements for structural health monitoring of bridges. In Proceedings of the Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infrastructure, Portland, OR, USA, 25–29 March 2017; Wu, H.F., Gyekenyesi, A.L., Shull, P.J., Yu, T.Y., Eds.; International Society for Optics and Photonics, SPIE: Bellingham, DC, USA, 2017; Volume 10169, pp. 68–77. [CrossRef] 9. Chen, H.; Quan, F.; Fang, L.; Zhang, S. Aerial Grasping with a Lightweight Manipulator Based on Multi-Objective Optimization and Visual Compensation. Sensors 2019, 19, 4253. [CrossRef] [PubMed] 10. Zhang, G.; He, Y.; Dai, B.; Gu, F.; Yang, L.; Han, J.; Liu, G. Aerial Grasping of an Object in the Strong Wind: Robust Control of an Aerial Manipulator. Appl. Sci. 2019, 9, 2230. [CrossRef] 11. Sanchez-Cuevas, P.J.; Gonzalez-Morgado, A.; Cortes, N.; Gayango, D.B.; Jimenez-Cano, A.E.; Ollero, A.; Heredia, G. Fully- Actuated Aerial Manipulator for Infrastructure Contact Inspection: Design, Modeling, Localization, and Control. Sensors 2020, 20, 4708. [CrossRef] [PubMed] 12. Baca, T.; Penicka, R.; Stepan, P.; Petrlik, M.; Spurny, V.; Hert, D.; Saska, M. Autonomous Cooperative Wall Building by a Team of Unmanned Aerial Vehicles in the MBZIRC 2020 Competition. arXiv 2020, arXiv:2012.05946. 13. Vatavuk, I.; Polic, M.; Hrabar, I.; Petric, F.; Orsag, M.; Bogdan, S. Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at MBZIRC 2020. J. Field Robot. 2021, submitted for publication. 14. Barišic, ´ A.; Petric, F.; Bogdan, S. Brain over Brawn—Using a Stereo Camera to Detect, Track and Intercept a Faster UAV by Reconstructing Its Trajectory. arXiv 2021, arXiv:2107.00962. 15. Markovic, L.; Car, M.; Orsag, M.; Bogdan, S. Adaptive stiffness estimation impedance control for achieving sustained contact in aerial manipulation. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation, Xi’an, China, 30 May–5 June 2021. Appl. Sci. 2021, 11, 8279 21 of 21 16. Cazzato, D.; Cimarelli, C.; Sanchez-Lopez, J.L.; Voos, H.; Leo, M. A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles. J. Imaging 2020, 6, 78. [CrossRef] 17. Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM 1981, 24, 381–395. [CrossRef] 18. Fumagalli, M.; Naldi, R.; Macchelli, A.; Carloni, R.; Stramigioli, S.; Marconi, L. Modeling and control of a ﬂying robot for contact inspection. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal, 7–12 October 2012; pp. 3532–3537. [CrossRef] 19. Alexis, K.; Darivianakis, G.; Burri, M.; Siegwart, R. Aerial Robotic Contact-based Inspection: Planning and Control. Auton. Robot. 2016, 40, 631–655. [CrossRef] 20. Jimenez-Cano, A.E.; Sanchez-Cuevas, P.J.; Grau, P.; Ollero, A.; Heredia, G. Contact-Based Bridge Inspection Multirotors: Design, Modeling, and Control Considering the Ceiling Effect. IEEE Robot. Autom. Lett. 2019, 4, 3561–3568. [CrossRef] 21. Cataldi, E.; Muscio, G.; Trujillo, M.A.; Rodriguez, Y.; Pierri, F.; Antonelli, G.; Caccavale, F.; Viguria, A.; Chiaverini, S.; Ollero, A. Impedance Control of an aerial-manipulator: Preliminary results. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 3848–3853. [CrossRef] 22. Nava, G.; Sablé, Q.; Tognon, M.; Pucci, D.; Franchi, A. Direct Force Feedback Control and Online Multi-Task Optimization for Aerial Manipulators. IEEE Robot. Autom. Lett. 2020, 5, 331–338. [CrossRef] 23. Meng, X.; He, Y.; Li, Q.; Gu, F.; Yang, L.; Yan, T.; Han, J. Contact Force Control of an Aerial Manipulator in Pressing an Emergency Switch Process. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 2107–2113. [CrossRef] 24. Stilman, M. Global Manipulation Planning in Robot Joint Space With Task Constraints. IEEE Trans. Robot. 2010, 26, 576–584. [CrossRef] 25. Cefalo, M.; Oriolo, G. Task-constrained motion planning for underactuated robots. In Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, 26–30 May 2015; pp. 2965–2970. [CrossRef] 26. Yu, Y.; Lippiello, V. 6D Pose Task Trajectory Tracking for a Class of 3D Aerial Manipulator From Differential Flatness. IEEE Access 2019, 7, 52257–52265. [CrossRef] 27. Tognon, M.; Cataldi, E.; Chavez, H.A.T.; Antonelli, G.; Cortés, J.; Franchi, A. Control-Aware Motion Planning for Task-Constrained Aerial Manipulation. IEEE Robot. Autom. Lett. 2018, 3, 2478–2484. [CrossRef] 28. Ivanovic, A.; Car, M.; Orsag, M.; Bogdan, S. Exploiting Null Space in Aerial Manipulation through Model-In-The-Loop Motion Planning. In Proceedings of the 2020 International Conference on Unmanned Aircraft Systems (ICUAS), Athens, Greece, 1–4 September 2020; pp. 686–693. [CrossRef] 29. Zargarbashi, S.; Khan, W.; Angeles, J. The Jacobian condition number as a dexterity index in 6R machining robots. Robot. Comput.-Integr. Manuf. 2012, 28, 694–699. [CrossRef] 30. Car, M.; Ivanovic, A.; Orsag, M.; Bogdan, S. Position-based adaptive impedance control for a UAV. In Proceedings of the 2018 International Conference on Unmanned Aircraft Systems (ICUAS), Dallas, TX, USA 12–15 June 2018; pp. 957–963. [CrossRef] 31. Pham, H.; Pham, Q. A New Approach to Time-Optimal Path Parameterization Based on Reachability Analysis. IEEE Trans. Robot. 2018, 34, 645–659. [CrossRef] 32. Maric, B.; Polic, M.; Tabak, T.; Orsag, M. Unsupervised optimization approach to in situ calibration of collaborative human–robot interaction tools. In Proceedings of the 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Karlsruhe, Germany, 14–16 September 2020; pp. 255–262. [CrossRef] 33. Furrer, F.; Burri, M.; Achtelik, M.; Siegwart, R. Robot Operating System (ROS): The Complete Reference (Volume 1). In Studies in Computational Intelligence; Chapter RotorS—A Modular Gazebo MAV Simulator Framework; Springer International Publishing: Cham, Switzerland, 2016; pp. 595–625. [CrossRef] 34. LARICS. Aerial Manipulation Approach for Autonomous Bridge Inspection. Available online: https://youtu.be/bBaxPMZEV04 (accessed on 23 July 2021). http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Sciences Multidisciplinary Digital Publishing Institute http://www.deepdyve.com/lp/multidisciplinary-digital-publishing-institute/towards-autonomous-bridge-inspection-sensor-mounting-using-aerial-nvO8jfmiqw

Loading next page...

References (36)

T. Báča, Robert Pěnička, P. Štěpán, Matěj Petrlík, Vojtech Spurný, Daniel Heřt, M. Saska (2020)
Autonomous Cooperative Wall Building by a Team of Unmanned Aerial Vehicles in the MBZIRC 2020 Competition
ArXiv, abs/2012.05946
Lovro Markovic, Marko Car, M. Orsag, S. Bogdan (2021)
Adaptive stiffness estimation impedance control for achieving sustained contact in aerial manipulation
2021 IEEE International Conference on Robotics and Automation (ICRA)
D. Reagan, A. Sabato, C. Niezrecki (2017)
Unmanned aerial vehicle acquisition of three-dimensional digital image correlation measurements for structural health monitoring of bridges
, 10169
Dario Cazzato, Claudio Cimarelli, Jose Sanchez-Lopez, H. Voos, Marco Leo (2020)
A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles
Journal of Imaging, 6
Anurag Vempati, Mina Kamel, N. Stilinovic, Qixuan Zhang, D. Reusser, Inkyu Sa, Juan Nieto, R. Siegwart, P. Beardsley (2018)
PaintCopter: An Autonomous UAV for Spray Painting on Three-Dimensional Surfaces
IEEE Robotics and Automation Letters, 3
Euiseok Jeong, Junwon Seo, J. Wacker (2020)
Literature Review and Technical Survey on Bridge Inspection Using Unmanned Aerial Vehicles
Journal of Performance of Constructed Facilities, 34
Yushu Yu, V. Lippiello (2019)
6D Pose Task Trajectory Tracking for a Class of 3D Aerial Manipulator From Differential Flatness
IEEE Access, 7
Marko Car, Antun Ivanovic, M. Orsag, S. Bogdan (2018)
Position-based adaptive impedance control for a UAV
2018 International Conference on Unmanned Aircraft Systems (ICUAS)
E. Cataldi, G. Muscio, M. Trujillo, Yamnia Rodríguez, F. Pierri, G. Antonelli, F. Caccavale, A. Viguria, S. Chiaverini, A. Ollero (2016)
Impedance Control of an aerial-manipulator: Preliminary results
2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Jonathan Cacace, S. Orozco-Soto, A. Suárez, A. Caballero, M. Orsag, S. Bogdan, Goran Vasiljević, E. Ebeid, Jose Rodriguez, A. Ollero (2021)
Safe Local Aerial Manipulation for the Installation of Devices on Power Lines: AERIAL-CORE First Year Results and Designs
Applied Sciences
Sardar Rehman, Z. Ibrahim, S. Memon, M. Jameel (2016)
Nondestructive test methods for concrete bridges: A review
Construction and Building Materials, 107
S. Zargarbashi, W. Khan, J. Angeles (2012)
The Jacobian condition number as a dexterity index in 6R machining robots
Robotics and Computer-integrated Manufacturing, 28
A. Barišić, Frano Petric, S. Bogdan (2021)
Brain over Brawn - Using a Stereo Camera to Detect, Track and Intercept a Faster UAV by Reconstructing Its Trajectory
ArXiv, abs/2107.00962
Massimo Cefalo, G. Oriolo (2015)
Task-constrained motion planning for underactuated robots
2015 IEEE International Conference on Robotics and Automation (ICRA)
A. Jimenez-Cano, P. Sánchez-Cuevas, P. Grau, A. Ollero, G. Heredia (2019)
Contact-Based Bridge Inspection Multirotors: Design, Modeling, and Control Considering the Ceiling Effect
IEEE Robotics and Automation Letters, 4
M. Fumagalli, R. Naldi, A. Macchelli, R. Carloni, S. Stramigioli, L. Marconi (2012)
Modeling and control of a flying robot for contact inspection
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems
Neshat Bolourian, A. Hammad (2020)
LiDAR-equipped UAV path planning considering potential locations of defects for bridge inspection
Automation in Construction, 117
Xiangdong Meng, Yuqing He, Qi Li, F. Gu, Liying Yang, T. Yan, Jianda Han (2018)
Contact Force Control of an Aerial Manipulator in Pressing an Emergency Switch Process
2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Antun Ivanovic, Marko Car, M. Orsag, S. Bogdan (2020)
Exploiting Null Space in Aerial Manipulation through Model-In-The-Loop Motion Planning
2020 International Conference on Unmanned Aircraft Systems (ICUAS)
Graham Hunt, Faidon Mitzalis, T. Alhinai, P. Hooper, M. Kovač (2014)
3D printing with flying robots
2014 IEEE International Conference on Robotics and Automation (ICRA)
M. Fischler, R. Bolles (1981)
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
Commun. ACM, 24
Mike Stilman (2010)
Global Manipulation Planning in Robot Joint Space With Task Constraints
IEEE Transactions on Robotics, 26
P. Sánchez-Cuevas, A. González-Morgado, Nicolas Cortes, Diego Gayango, A. Jimenez-Cano, A. Ollero, G. Heredia (2020)
Fully-Actuated Aerial Manipulator for Infrastructure Contact Inspection: Design, Modeling, Localization, and Control
Sensors (Basel, Switzerland), 20
K. Alexis, Georgios Darivianakis, M. Burri, R. Siegwart (2016)
Aerial robotic contact-based inspection: planning and control
Autonomous Robots, 40
H. Pham, Quang-Cuong Pham (2017)
A New Approach to Time-Optimal Path Parameterization Based on Reachability Analysis
IEEE Transactions on Robotics, 34
Fadri Furrer, M. Burri, Markus Achtelik, R. Siegwart (2016)
RotorS—A Modular Gazebo MAV Simulator Framework
, 625
A. Koubâa (2016)
Robot Operating System (ROS): The Complete Reference (Volume 7)
Robot Operating System (ROS)
S. Zollini, M. Alicandro, D. Dominici, R. Quaresima, M. Giallonardo (2020)
UAV Photogrammetry for Concrete Bridge Inspection Using Object-Based Image Analysis (OBIA)
Remote. Sens., 12
Gabriele Nava, Quentin Sablé, M. Tognon, D. Pucci, A. Franchi (2020)
Direct Force Feedback Control and Online Multi-Task Optimization for Aerial Manipulators
IEEE Robotics and Automation Letters, 5
M. Tognon, E. Cataldi, H. Chavez, G. Antonelli, Juan Cortés, A. Franchi (2018)
Control-Aware Motion Planning for Task-Constrained Aerial Manipulation
IEEE Robotics and Automation Letters, 3
A. Barišić, Frano Petric, S. Bogdan (2021)
Brain over Brawn: Using a Stereo Camera to Detect, Track, and Intercept a Faster UAV by Reconstructing the Intruder's Trajectory
Field Robotics, 2
Bruno Maric, Marsela Polic, Tomislav Tabak, M. Orsag (2020)
Unsupervised optimization approach to in situ calibration of collaborative human-robot interaction tools
2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)
Guangyu Zhang, Yuqing He, Bo Dai, F. Gu, Liying Yang, Jianda Han, Guangjun Liu (2019)
Aerial Grasping of an Object in the Strong Wind: Robust Control of an Aerial Manipulator
Applied Sciences
Aerial Manipulation Approach for Autonomous Bridge Inspection
Ivo Vatavuk, Marsela Poli'c, Ivan Hrabar, Frano Petri'c, M. Orsag, S. Bogdan (2022)
Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at MBZIRC 2020
Field Robotics, 2
Haoyao Chen, Fengyu Quan, L. Fang, Shiwu Zhang (2019)
Aerial Grasping with a Lightweight Manipulator Based on Multi-Objective Optimization and Visual Compensation
Sensors (Basel, Switzerland), 19

Publisher: Multidisciplinary Digital Publishing Institute
Copyright: © 1996-2021 MDPI (Basel, Switzerland) unless otherwise stated Disclaimer The statements, opinions and data contained in the journals are solely those of the individual authors and contributors and not of the publisher and the editor(s). MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Terms and Conditions Privacy Policy
ISSN: 2076-3417
DOI: 10.3390/app11188279
Publisher site: See Article on Publisher Site

Abstract

applied sciences Article Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators 1 , 1 1 2 1 Antun Ivanovic * , Lovro Markovic , Marko Car , Ivan Duvnjak and Matko Orsag Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, 10000 Zagreb, Croatia; lovro.markovic@fer.hr (L.M.); marko.car@fer.hr (M.C.); matko.orsag@fer.hr (M.O.) Faculty of Civil Engineering, University of Zagreb, Fra Andrije Kacica-Miosica 26, 10000 Zagreb, Croatia; ivan.duvnjak@grad.unizg.hr * Correspondence: antun.ivanovic@fer.hr Featured Application: The main idea of this paper was to deploy a team of unmanned aerial vehi- cles (UAVs) to attach a sensor to a bridge using a two-component adhesive in order to perform an inspection. Constant pressure must be applied for several minutes to form a bond between two adhesives. Therefore, one UAV sprays the colored component of an adhesive while the aerial ma- nipulator transports the sensor, detects the contact point and attaches the sensor to it. A trajectory planning algorithm was developed around the dynamic model of the UAV and the manipulator attached to it, ensuring that the end-effector is parallel to the wall normal. Finally, the aerial ma- nipulator achieves and maintains contact with a predeﬁned force through an adaptive impedance control approach. Abstract: Periodic bridge inspections are required every several years to determine the state of a bridge. Most commonly, the inspection is performed using specialized trucks allowing human Citation: Ivanovic, A.; Markovic, L.; inspectors to review the conditions underneath the bridge, which requires a road closure. The aim of Car, M.; Duvnjak, I.; Orsag, M. this paper was to use aerial manipulators to mount sensors on the bridge to collect the necessary Towards Autonomous Bridge data, thus eliminating the need for the road closure. To do so, a two-step approach is proposed: an Inspection: Sensor Mounting Using unmanned aerial vehicle (UAV) equipped with a pressurized canister sprays the ﬁrst glue component Aerial Manipulators. Appl. Sci. 2021, onto the target area; afterward, the aerial manipulator detects the precise location of the sprayed area, 11, 8279. https://doi.org/ and mounts the required sensor coated with the second glue component. The visual detection is 10.3390/app11188279 based on an Red Green Blue - Depth (RGB-D) sensor and provides the target position and orientation. Academic Editor: Alessandro A trajectory is then planned based on the detected contact point, and it is executed through the Gasparetto adaptive impedance control capable of achieving and maintaining a desired force reference. Such an approach allows for the two glue components to form a solid bond. The described pipeline Received: 23 July 2021 is validated in a simulation environment while the visual detection is tested in an experimental Accepted: 1 September 2021 environment. Published: 7 September 2021 Keywords: aerial robotics; inspection and maintenance; aerial manipulation; multirotor control Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional afﬁl- iations. 1. Introduction The world of unmanned aerial vehicles (UAVs) has been rapidly growing in recent years. As their design and control are perfected, these aerial vehicles have become more and more available. Nowadays, off-the-shelf ready-to-ﬂy UAVs can be found and bought Copyright: © 2021 by the authors. in shops, which makes them available to virtually anybody. This, in turn, has sparked a Licensee MDPI, Basel, Switzerland. great deal of public interest in UAVs since their potential can be found in applications such This article is an open access article as agriculture, various inspections (bridges, buildings, wind turbines), geodetic terrain distributed under the terms and mapping, the ﬁlm industry, and even for hobby enthusiasts to ﬂy and record videos from a conditions of the Creative Commons ﬁrst-person perspective. The vast majority of commercially available UAVs are equipped Attribution (CC BY) license (https:// with a camera, while more specialized vehicles for terrain mapping or crop spraying offer creativecommons.org/licenses/by/ a more diverse sensor suite. 4.0/). Appl. Sci. 2021, 11, 8279. https://doi.org/10.3390/app11188279 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 8279 2 of 21 All of the aforementioned systems primarily observe and gather data about the environment, while having little to no ability to interact with and change the environment. One way to augment these vehicles for physical interaction is to attach a lightweight manipulator to their body, which is the main interest of the aerial manipulation ﬁeld. Although such vehicles are more complex for both modeling and control, their beneﬁt lies in performing versatile tasks that require interaction with the environment. In general, there are three types of bridge inspections: periodic, special and damage inspections. Periodic bridge inspections differ from country to country according to national standards, and are usually performed at least once every two to three years. Special inspections are typically used to monitor the condition of deﬁcient elements at speciﬁc locations based on predeﬁned requirements. Damage inspections are usually performed after events that have occurred due to environmental impacts or human actions. The aim of a bridge inspection is to evaluate and assess structural safety and reliability. Current techniques are based on traditional visual inspection with a combination of non- destructive methods (NDTs). Traditional visual inspection is performed by experienced (trained) engineers and using specialized trucks equipped with the cranes and basket, that allow inspectors to review the conditions underneath the bridge. During the inspection, the engineers are equipped with various NDT [1] tools to detect construction faults and defects such as corrosion, cracks, voids, weakening connections, and concrete delamination. Some of these NDTs require mounting small sensors to collect data, such as accelerometers, strain gauges, tilt meters and various transducers for acoustic or pressure measurements. Afterwards, the bridge is excited with vibrations, sound waves, tapping, etc., and mounted sensors record responses to these speciﬁc excitations. Furthermore, there are usually requirements for performing measurements during the bridge inspection, such as the short- and long-term monitoring of vibrations, strains, displacements, etc. Mainly, these inspections offer valuable information about the current bridge conditions, but there are a number of disadvantages. The use of trucks during inspections requires total or temporary road closures, which at the same time require safety measures to keep trafﬁc ﬂowing as freely as possible. In addition, inspectors often encounter challenges in reaching all portions or elements in narrow areas, such as tight spaces between girders, beams and vaults. The aforementioned signiﬁcantly increases the time and overall cost of the inspection. An aerial robot, with the potential to reach these challenging locations on the bridge, could signiﬁcantly reduce the time and cost of these inspections and improve worker safety. Moreover, we note that the aforementioned sensors are relatively lightweight, which makes them suitable for transportation and mounting with an aerial robot. 1.1. Concept We envision a team of robots working together to attach sensors to bridges and similar grade separation infrastructure. In theory, such a task could be accomplished with a single aerial robot, at the cost of a complex mechanical design. The proposed team shown in Figure 1 consists of two drones. One drone applies the adhesive material, and the other attaches sensors. We envision a two-stage process using two-component adhesives which form a solid bond from two separate reactive components: the “resin” and the “hardener”. The ﬁrst UAV applies the resin by spraying it onto the surface, while the second one attaches the sensor with the hardener already applied before the ﬂight. It is important to follow the prescribed ratio of the resin and the hardener to achieve the desired physical properties of the adhesive. Only when mixed together do the two components form the adhesive. The reaction typically begins immediately after the two components are mixed and the bond strength depends both on maintaining the contact and the viscosity of the mixed adhesive during the process. Manufacturers can control the cure rate to achieve various working times (worklife) until ﬁnal bond strength is achieved, ranging from minutes to weeks. Resin bases are usually more viscous than their respective hardener and are generally applied by brush, roller, applicator or spray. In this work, we propose attaching a canister of pressurized resin to the UAV, and spray it through a Appl. Sci. 2021, 11, 8279 3 of 21 nozzle onto the infrastructure surface. In this scenario, the spray needs to be softer and less turbulent to reduce the amount of material lost due to bouncing and it must be colored for the detection in the second stage. Spraying with drones is not a novel concept [2,3], so without loss of generality, we will omit the details of this design and instead focus on detecting, navigating to, and sustaining contact with the sprayed surface. In typical applications, the assemblies are usually kept in contact until the sufﬁcient strength of the bond is achieved. When fully cured, two-component adhesives are typically tough and rigid with good temperature and chemical resistance. We rely on the robotic arm attached to the second aerial vehicle to apply a controlled contact force between the sensor and the surface. Maintaining this ﬁxed assembly contact through the impedance control system enables us to achieve a successful curing process and create a permanent bond between the sensor and the infrastructure. After the ﬁrst UAV sprays the resin onto the surface, the second aerial robot ﬁnds the sprayed part and applies the contact with the sensor ’s surface. Before takeoff, the surface of the sensor is brushed with a hardener. Once contact is made, it is maintained for the prescribed curing time, after which the aerial robot disembarks and leaves the sensor attached to the surface. Figure 1. Two aerial robots working together to attach sensors to different parts of a bridge and similar grade separation infrastructure. The one on the left is used to spray the resin onto the surface, while the aerial robot on the right maintains contact to the surface with the sensor attached to its end-effector. 1.2. Contributions This paper focuses on developing a method for mounting sensors on a bridge wall using an aerial manipulator. The ﬁrst contribution is augmenting the model-based motion planning with the adaptive impedance controller. The motion planning method accounts for the underactuated nature of the multirotor UAV and corrects the end-effector conﬁgu- ration for an appropriate approach. This method also relies on the dexterity analysis which keeps the manipulator conﬁguration within its optimal region, ensuring that the manipula- tor is never fully extended or contracted while mounting a sensor. The second contribution is the visual blob detection which locates and tracks the appropriate sensor mounting point. The blob detection has been experimentally veriﬁed in an indoor environment, yielding the reliable and robust tracking of the mount location, as well as the blob plane orientation. Finally, the third contribution is the simulation analysis of the system’s performance, con- ducted on a straight and inclined wall approach. The simulation concentrates on testing the motion planning together with the impedance controller, performing a repeatability analysis and ensuring that the desired contact force is achieved. Appl. Sci. 2021, 11, 8279 4 of 21 2. Related Work In the world of aerial inspections, a number of UAV-based solutions are being pro- posed by researchers. In [4], a technical survey for bridge inspections is given. Researchers in [5] present the project AERIAL COgnitive Integrated Multi-task Robotic System with Extended Operation Range and Safety (AERIAL-CORE) which focuses on power lines inspection, maintenance and installing bird diverters and line spacers. Most of these approaches are based in conjunction with new technologies to ensure faster and cheaper in- spections. Nowadays, UAVs use high-resolution cameras for visual inspections and employ point cloud methods based on digital photogrammetry [6], Light Detection And Rang- ing (LiDAR)-based methods [7], digital image correlation [8], etc. There are also reports for visual compensation during aerial grasping [9], aerial grasping in strong winds [10], and the development of a fully actuated aerial manipulator for performing inspections underneath a bridge [11]. According to the experimental testing of contact-based bridge inspections, there is a need to develop a solution for mounting application sensors (such as accelerometers, strain gauges and tilt meters) on a bridge using a UAV. It is expected that a sophisticated system with the possibility of automatic sensor mounting will increase the frequency of measurements without interrupting trafﬁc, ensure the safety of inspectors as well as reduce inspection time and overall costs. As mentioned earlier, the second UAV needs to be aware of the position of the sprayed adhesive which is applied in a blob-like pattern. For this purpose, an Red Green Blue - Depth (RGB-D) camera is used due to its favorable dimensions and weight. It provides image and depth information about the environment which proves useful for object local- ization and UAV navigation. Such cameras were commonly found on UAVs and unmanned ground vehicles (UGVs) present at the recent MBZIRC 2020 competition. In [12,13], RGB-D information is used for color-based brick detection and localization for the wall-building challenge using UAVs and UGVs, respectively, while in [14] the authors use a Convolu- tional Neural Network (CNN)-based UAV detection and tracking method for the intruder UAV interception challenge. Furthermore, visual sensors proved useful in [15], where the authors performed a contact-based inspection of a ﬂat surface with an aerial manipulator. The surface position and orientation was obtained by applying random sample consensus (RANSAC) on the RGB-D information. A thorough survey of 2D object detection methods from UAVs was given in [16]. In this paper, the authors present a modular framework for object detection in which a simple contour-based blob detector is implemented. The goal is to use RGB-D information to enable an autonomous inspection workﬂow. The blob position is obtained by segmenting the depth data at the points where the object is detected in the image, while RANSAC [17] is used to determine its orientation. After the successful detection of a blob-like pattern, it is necessary to attach the inspection sensor. The ﬁrst phase of the sensor attachment is achieving contact, and the second is maintaining that contact to allow for the two adhesive components to form a bond. Generally, the contact can be achieved with or without force measurements. In [18], contact with the wall is performed and maintained. Researchers in [19] performed wall contact and aerial writing experiments. The work presented in [20] modeled and exploited the effects of the ceiling effect to perform an inspection underneath a bridge. The common denominator in the former approaches is maintaining the contact without any force feedback. Although mounting a force sensor on a UAV increases both mechanical and control complexity, an immediate beneﬁt is the ability to maintain precise contact force regardless of the environment. In [21], the researchers used a force/torque sensor to achieve compliant control while pulling a rope and a semi-ﬂexible bar. A fully actuated UAV with a manipulator has been employed in [22] to compare force feedback control with and without the force/torque sensor. Researchers in [23] used a single degree of freedom manipulator with a force sensor mounted at the end-effector to press an emergency switch. Relying on the blob-like pattern detection and the impedance control, a trajectory for achieving contact is required to steer the aerial manipulator towards the contact point. While mounting the sensor, it is essential that the approach and contact are perpendicular Appl. Sci. 2021, 11, 8279 5 of 21 to the wall plane. This can be considered as a task constraint imposed on the planner which the aerial manipulator has to satisfy. Researchers in [24] propose a task constrained planner for a redundant robotic manipulator that enables them to do everyday tasks such as opening drawers or picking up objects. In [25], a task-constrained planner was developed for underactuated manipulators. Since multirotor UAVs are typically underactuated systems, it is necessary to address dynamics and kinematics while planning the end- effector trajectory. Aerial manipulator 6D end-effector trajectory tracking based on the differential ﬂatness principle was presented in [26]. The underactuated nature of multirotor UAVs can cause unexpected deviations in end-effector conﬁguration. Researchers in [27] address this particular problem by including the dynamic model of the system into the planning procedure. In our previous work [28], a trajectory planning method based on the full dynamic model of an aerial manipulator was developed. In this paper, we further augmented this method to plan for the desired force required by the impedance controller. 3. Mathematical Model In this section, the mathematical model of the aerial manipulator is presented. The coordinate systems convention is depicted in Figure 2. Furthermore, an analysis for the manipulator dexterity and reach was performed. 3.1. Kinematics The inertial frame is deﬁned as L . The body-ﬁxed frame L is attached to the W B center of gravity of the UAV. The position of the UAV in the world frame is given h i h i T T with p = x y z 2, and the attitude vector is Q = f q y . Combining the position and attitude vectors deﬁned the generalized coordinates of the UAV as h i 6 B B T T q = 2 R . Written in a matrix form T , the transformation contains (p ) Q W W both the position and orientation of the UAV obtained through an on-board sensor fusion b 44 or through an external positioning system (i.e., GPS). The notation T 2 R was used to denote a homogeneous transformation matrix between frames a and b. A rigid attachment between the body of the UAV and the base of the manipulator L was considered, denoted with the transformation matrix T . The manipulator used in this work was a M = 3 degree-of-freedom (DoF) serial chain manipulator with the end-effector attached to the last joint. The DH parameters of the arm are given in Table 1. Using this ee notation, one can write a transformation matrix T between the manipulator base and its end-effector as a function of joint variables q , q and q . For brevity, the expression for 1 2 3 ee the entire matrix T is left out and only the end-effector position and its approach vector equations are written using the well-known abbreviation cos(q + q ) := C : 1 2 12 2 3 2 3 a (C + C ) + d C C 1 1 12 3 123 123 6 7 6 7 ee ee p = a (S + S ) + d S , z = S (1) 4 5 4 5 1 1 12 3 123 123 0 0 0 0 Table 1. DH parameters of the 3-DoF manipulator attached to the UAV. A virtual joint q is added to fully comply with the DH convention. Link sizes a and d are omitted for clarity. 1 3 Joint q d a a q 0 0 0 a 1 1 q 0 0 0 a 2 1 p p q 0 0 2 2 q 0 d 0 0 4 Appl. Sci. 2021, 11, 8279 6 of 21 L 0 2 z ee Figure 2. Coordinate systems of the world, UAV and the 3-DoF manipulator. Putting it all together, the full kinematic chain of the aerial manipulator can be con- structed as ee B 0 ee T = T T T , (2) W W B 0 0 B ee combining the ﬁxed transformation T with T and T depend on UAV and manipu- B W 0 lator motion. Since there is obvious coupling between the motion of the body and the manipulator arm, a b 2 [0, 1] parameter is introduced to distribute the end-effector motion commands either to the UAV global position control or manipulator joint position control. To this end, the following distribution relationship is used: DP = b DP UAV DP = (1 b) DP, (3) arm where DP is used to denote the desired aerial manipulator displacement expressed as the following combination body and arm motion: DP = DP + DP UAV arm = b DP + (1 b) DP. (4) The manipulator displacement is denoted by DP and the UAV displacement by arm DP , where both DP and DP are expressed in the coordinate system L . With UAV arm UAV 0 b = 1, the UAV motion is used to control the position of the end-effector. When b = 0, the situation is reversed and the manipulator motion is used to move the end-effector. For every other b, the end-effector motion is obtained in part by the UAV body and the manipulator arm motion. There are obvious advantages in combining the motion of the UAV and the manip- ulator arm. The UAV can move in 3D space beyond the reach of the arm; however, the motion of the UAV is not as precise and dynamically decoupled. The kinematics of the ee T arm enable the end-effector to obtain the desired approach angle z = [cos(d), sin(d), 0] , ee which under the hovering assumption, becomes equal to the global approach vector z pointing towards the contact point on the infrastructure. The straightforward mathematical manipulation of Equation (1) allows for writing the constraint equation: q = d q q , (5) 3 1 2 which ensures that the manipulator points in the right direction, where d is the desired manipulator inclination in the body x–z plane. Appl. Sci. 2021, 11, 8279 7 of 21 To ﬁnd the optimal manipulator pose during contact, the dexterity D and the reach R of the pose were taken into account, while considering that the joints are as far as possible from their physical limits L. Since the motion of the arm is constrained with its approach h i ee ee dp dp 0 0 axis condition, a reduced form of a Jacobian matrix was used J = , to derive the dq dq 1 2 pose dexterity index D = J J and determine how far the current pose is from the null ee T ee space of the manipulator [29]. The reach of the pose R = (p ) p was also taken into 0 0 account, since the goal was to keep the end-effector and the contact point as far away from the UAV body. Finally, the following equation is deﬁned: 2 2 2 2 (q Q )(q Q ) 1max 2max 1 2 L = , (6) 2 2 Q Q 1max 2max to measure how far away the given conﬁguration is (i.e., q , q ) from the joint limits 1 2 Q , Q . Normalizing D,R and L enables combining the three conditions into a 2max 1max h i single manifold M = D R L and ﬁnd the optimal conﬁguration q = for q q q M 1 2 3 the desired approach angle d. The described method is depicted in Figure 3 for the speciﬁc case of the approach angle d = 0 , but can be extended to any value of the approach angle through Equation (5). As a side note, the manipulator attachment on the top of the UAV body was chosen to be able to reach surfaces underneath the bridge. Although this shifts the center of gravity upwards, the stability of the system is not compromised since the manipulator is constructed of lightweight materials. Figure 3. The visual decomposition of dexterity D, reach R, limit L and the overall combined surface. This analysis is performed for d = 0 : (a) the dexterity D surface shows the measure of how far the manipulator is from the null space. Values around zero are closer to the null space; (b) the reach R surface shows how far the end-effector can move in a certain conﬁguration. This value tends towards zero as the arm approaches a folded conﬁguration; (c) the limit L depicts how far a certain conﬁguration is from the physical limits of the manipulator joints; and (d) the combined manifold M of the formerly described surfaces. Higher values offer better trade-off between dexterity, reach and limit, deﬁning the optimal manipulator conﬁguration q . 3.2. Dynamics The most complicated task of the aerial manipulator is attaching the sensor to a wall and maintaining the required force reference while the two-component adhesive hardens. Appl. Sci. 2021, 11, 8279 8 of 21 To successfully perform such a task, the coupled UAV-manipulator system dynamics have to be addressed for the precise end-effector conﬁguration planning. Considering the UAV dynamics only, the derivative of the generalized coordinates h i 6 B T B T B T can be deﬁned as q ˙ = (p ˙ ) (w ) 2 R . Here, the (p ˙ ) is the linear velocity of W W B T the body of the UAV in the world frame and the (w ) represents the angular velocity of the UAV in the world frame. The UAV’s propulsion system consists of n propellers rigidly attached to the body. Each propeller produces force and torque along the z axis. The vector of the propeller rotational velocities is simply deﬁned as h i W = W . . . W 2 R . (7) UAV 1 n Force and torque produced by each propeller are non-linear functions depending on the rotational velocity W . Rather than using the rotational velocities as control inputs, UAV they can be mapped to a more convenient space. Namely, the mapped control input space can be written as u = K diag(W ) W , (8) UAV UAV UAV h i 4n where K 2 R is the mapping matrix and u = u u u u , where u UAV 1 2 3 4 4 represents the net thrust and u , u and u are moments around the body frame axes. 1 2 3 As stated earlier, the manipulator consists of three rotational DoFs. Therefore, the joint h i positions of the manipulator are deﬁned as q = q q q . The rotational velocity of 1 2 3 each joint is a time derivative of joint positions q ˙ = dq /dt. The torque of each joint is M M h i considered the control input of the manipulator u = . M t t t 1 2 3 The resulting generalized coordinates of the aerial manipulator can be written as h i q = 2 R , and the velocities can be obtained in the same manner as q q UAV M h i q ˙ = 2 R . The resulting control inputs of the system can be expressed as q ˙ q ˙ UAV M h i u = u u 2 R . Finally, the full system dynamics can be written as UAV M M(q)q ¨ + c(q, q ˙ ) + g(q) = u, (9) 97 7 where M(q) 2 R is the inertia matrix, c(q, q ˙ ) 2 R is the vector of centrifugal and Coriolis forces, g(q) 2 R is the gravitational term. 4. Control System The overall control of the aerial manipulator consists of several nested control loops. The complete controller overview, with motion planning and blob detection blocks, is depicted in Figure 4. 4.1. Aerial Manipulator Control At the inner most level, the UAV is controlled through cascade attitude and rate controllers. The input to these controllers is the desired orientation and based on the state, the output is the vector of the rotors’ angular velocities. The second level of control, which uses the inner attitude control loop, consists of two additional cascades, the position and the velocity control. These controllers receive a referent position and velocity feed-forward value to generate the desired vehicle orientation and thrust. The manipulator joints are controlled through standard Proportional, Integral, Derivative (PID) controllers; however, in a real-world setting, servo motors with integrated control are typically used. As mentioned earlier, it is important to track the desired force after contact with a wall is achieved. To accomplish this, an adaptive impedance controller is employed to generate an appropriate setpoint for the position controller. This controller receives a trajectory Appl. Sci. 2021, 11, 8279 9 of 21 supplied by the mission planner, which steers the aerial manipulator towards the sensor mounting target on the bridge. Aerial manipulator control Position and Attitude and velocity control rate control Manipulator control Adaptive Motion Blob detection impedance planning and estimation control Figure 4. The overall functional schematic of the system. The aerial manipulator control subsystem is necessary for the controlling position and attitude of the UAV, and manipulator joints. On top of this controller, the adaptive impedance control is employed in order to track the desired force. Motion planning generates an appropriate trajectory based on the target point supplied by the blob detection algorithm. 4.2. Adaptive Impedance Control The objective of the adaptive impedance controller is to ensure a stable physical interaction between the aerial manipulator and the environment [30]. As mentioned earlier, the standard UAV control scheme is based on position and attitude controllers. When interacting with the environment, the desired contact force must be considered. The position controlled system can be extended to follow the desired force by introducing an impedance ﬁlter. The design of such a ﬁlter is explained here for a single DoF. The behavior of the system is deﬁned by the target impedance as e(t) = m(x¨ (t) x ¨ (t)) + b(x ˙ (t) x ˙ (t)) + k(x (t) x (t)), (10) c r c r c r where m, b and k are constants, x (t) is the referent position, provided to the impedance ﬁlter as an input, and x (t) is the output of the impedance ﬁlter representing the position command. The ﬁlter is designed as a linear second-order system with a dynamic rela- tionship between the position and the contact force tracking error e(t) so that it mimics a mass-spring–damper system. The contact force tracking error is deﬁned as follows: e(t) = f (t) f (t), (11) where f (t) is the other ﬁlter input deﬁning the referent force, and f (t) is the measured (ex- erted) contact force. If the environment is modeled as a ﬁrst-order elastic system (equivalent spring system) with unknown stiffness k , the measured force can be approximated as f (t) = k (x(t) x (t)), (12) e e where x(t) is the position of the manipulator and x (t) is the position of the environment in an unexcited state. By substituting Equation (12) in Equation (11), the position of the aerial manipulator can be expressed as follows: f (t) e(t) x(t) = + x (t). (13) e Appl. Sci. 2021, 11, 8279 10 of 21 Assuming that the commanded position value can be achieved by the aerial manipula- tor, i.e., x = x , the substitution of Equation (13) in Equation (10), the system in the steady state can be described as follows: k k f (t) e r e(t) = + x (t) x (t) . (14) e r k + k k e e For a contact force error of zero in the steady state, the following must hold: f (t) x (t) = + x (t). (15) r e In other words, the position setpoint has to be designed in such a way that it compen- sates for the displacement of the environment due to the exerted contact force. To ensure this, a value of the unknown environment stiffness k is needed. Furthermore, k plays a e e fundamental role in the stability of the impedance ﬁlter Equation (10), which ultimately affects the stability of the aerial manipulator while in contact with the environment. A sta- ble contact between the aerial manipulator and the environment can be ensured using the Hurwitz stability criterion, by designing the system with b/m > 0 and (k + k )/m > 0. However, since k is unknown, an adaptation law for the position setpoint that guarantees the contact stability while compensating for this hidden, unknown parameter is proposed. The adaptation law is derived starting from Equation (15). An adaptive parameter k(t) is introduced so that: x (t) = k(t) f (t) + x (t). (16) r r e It can be shown using the Lyapunov stability analysis that the following adaptation dynamics equation for k(t) will yield a stable system response: ... kk ˙(t) + bk ¨(t) + mk (t) = gs(t) + g s ˙ (t). (17) We refer the interested reader to the proof which can be found in the Appendix A. 5. Motion Planning As discussed in Section 1.1, the main concept of this paper was to use a team of two UAVs, each applying one component of the adhesive. To apply the “resin” component, the UAV has to plan a collision-free trajectory and position itself in front of the target area to start spraying. This is fundamentally different from mounting a sensor coated with “hardener ”. In the latter case, apart from planning a collision-free trajectory, the manipulator-endowed UAV has to apply pressure for a certain amount of time for the two components to mix. From the perspective of motion planning, the planner needs to be augmented to include a manipulator with three degrees of freedom, contact force and the weighing parameter b. To successfully maintain the pressure, the planner relies on the impedance controller described in Section 4.2. Furthermore, one of the requirements when mounting the sensor on the wall is for the sensor to be perpendicular to the wall. Therefore, it is necessary to take the underactuated nature of the multirotor UAVs into account during the motion planning. Namely, the errors in the planned end-effector conﬁguration were mainly induced due to the roll and pitch angles while executing the planned motion. In our previous work [28], we developed a model-based motion planner for aerial manipulators that is capable of correcting the aforementioned end-effector deviations. In this paper, the idea from [28] was extended to consider the impedance control when obtaining the full state of the aerial manipulator. Appl. Sci. 2021, 11, 8279 11 of 21 5.1. Waypoint Conﬁguration When dealing with an aerial manipulator, exerting some contact force inevitably yields a high dimensional waypoint conﬁguration. We deﬁne a single waypoint as a set of UAV and joint poses, together with the force reference and motion distribution factor b: h i T T T w = q q f b 2 R , (18) B M r 6 3 where q 2 R and q 2 R are generalized coordinates of the UAV and the manipulator B M h i deﬁned in Section 3.1. The force reference vector f = 2 R and weighing f f f r x y z scalar parameter b are required by the impedance controller. Furthermore, the impedance controller assumes a step change of these values. Ideally, the change should occur at the moment of contact since no force can be exerted without contact. Therefore, these values are only changed at the ﬁnal waypoint. Apart from the desired force and the parameter b, the ﬁnal waypoint must contain the UAV position and orientation, as well as the manipulator joint conﬁguration. Specifying these values relies on the blob detection algorithm presented in Section 6. Namely, the algorithm outputs the position and orientation of the detected blob in the world frame. Following the manipulator dexterity and reach analysis described in Section 3.1, the optimal manipulator conﬁguration q is obtained based on the provided plane normal. The optimal manipulator conﬁguration is then used as the desired conﬁguration for the ﬁnal waypoint. This way, during operation, the manipulator never reaches a fully extended or contracted pose, which allows the impedance controller to command both the arm and the UAV to achieve and maintain the desired force. 5.2. Trajectory Planning There are three phases in the trajectory planning procedure. First, an initial trajectory is planned based on the provided waypoints. Second, the initial trajectory is sent to a simulated model in order to obtain the full state of the aerial manipulator during the trajectory execution. Third, the end-effector conﬁguration is corrected based on the full state of the vehicle, and the ﬁnal trajectory is sent to the target aerial manipulator. Initial Trajectory To execute a smooth motion towards the desired waypoint, we use a suitable time- optimal path planning by reachability analysis (TOPP-RA) trajectory planner [31]. The TOPP-RA algorithm searches for the time-optimal trajectory and is based on a “bang-bang” principle on the generalized torque of each DoF. The planner is capable of receiving the input waypoints of an arbitrary dimension and output a smooth trajectory. Each DoF has to be provided with dynamical constraints in terms of velocity and acceleration, which are respected during the trajectory generation process. As mentioned, the input to the TOPP-RA trajectory is the path of a set of n 2 waypoints: n o P = w j w 2 R , i 2 (0, 1, . . . , n) . (19) i i Based on the dynamical constraints, the output of the TOPP-RA planner is a sampled trajectory: n o 313 T = t(kT ) j t(kT ) 2 R , k 2 (0, . . . , n ) , (20) s s s t h i 313 T T T where t = (w) (w ˙ ) (w ¨ ) 2 R is a single sampled trajectory point consisting of position, velocity and acceleration; T is the sampling time; and n is the number of s t points in the sampled trajectory. Note that each trajectory point contains both roll and pitch angles. Although these angles can be planned through the TOPP-RA algorithm, Appl. Sci. 2021, 11, 8279 12 of 21 they are omitted at this point because of the underactuated nature of the multirotor UAV. Nevertheless, they are used later in the paper when the model corrections are applied. The impedance controller expects a step change in the force and weighing parameter b referent values. To satisfy this requirement, large constraints for velocity and acceleration are imposed for these DoFs. However, because other DoFs have constraints below their physical limit, the overall force and b trajectory has a slower, dynamically smooth proﬁle. These proﬁles also have overshoots and undershoots which are not acceptable because they are not within the hard constraints required for b. To tackle this problem, a simple piecewise constant velocity interpolation was applied to the force and b. This way, a large velocity constraint produces a step change which is a suitable input to the impedance controller. A visual example of the difference between the TOPP-RA and piecewise constant velocity interpolation is depicted in Figure 5. 0.8 TOPP-RA 0.6 Constant velocity 0.4 0.2 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 t[s] Figure 5. Visual comparison between TOPP-RA and the piecewise constant velocity interpolation. Waypoints for both trajectories are kept the same, and around t = 2.5 s, the middle waypoint(yellow cross) is reached. Although the dynamical constraints are the same, TOPP-RA takes other degrees of freedom into account and produces a trajectory with overshoot which is not suitable for parameter b. 5.3. Model-Based Corrections The initial trajectory from Equation (20) is planned without any consideration about the underactuated nature of the multirotor UAV. To obtain the unknowns, namely roll and pitch angles, the initial trajectory can be executed in a simulation environment. The chosen simulation environment is, in our case, Gazebo, because it is realistic and supports the robotics operating system (ROS), which is the backbone of our implementation. The simulated aerial manipulator is based on the mathematical model described in Section 3. The standard cascade PID controllers are employed for low-level attitude and high-level position control. The impedance controller is built on top of the position controller and pro- vides a position reference based on the input trajectory. More details about the simulation environment are provided in Section 7. The ﬁrst step is executing the initial trajectory in the aforementioned simulation environment. While executing, the roll and pitch angles are recorded as they are needed for obtaining the full state of the UAV. Rearranging Equation (2) and plugging the unknown roll and pitch angles in the full state of the UAV, the transform of the end-effector in the manipulator base frame can be obtained: ee 1 B 1 ee T = (T ) (T ) T . (21) L B W W Usingthe inverse kinematics of the manipulator, joint values q for the desired end- effector conﬁguration are obtained. This way, the null space of the aerial manipulator is used for the end-effector correction. Note that due to the conﬁguration of the manipulator, an exact solution of the inverse kinematics will not always exist. In such a case, an approximate closest solution is used instead. The ﬁnal trajectory is constructed by replacing the initial q with the corrected values. This trajectory is afterwards sent to the target aerial manipulator. The careful reader should note that the developed three DoF manipulator operates on the x and z position in the body frame, as well as the pitch angle. This allows the impedance controller to maintain the orientation perpendicular to the wall, while compensating for the - Appl. Sci. 2021, 11, 8279 13 of 21 UAV body motion in the x and z axes. However, the system will experience disturbances and control errors which will act on the roll and pitch angle, and the lateral movement along the body y axis. We can address these issues either with mechanical dampers or by adding additional degrees of freedom to the manipulator, which will be explored in future work. 6. Blob Detection This section presents the methods we propose to detect the hardener blob position and orientation. A modular object detection framework, as shown in Figure 6, is designed to ensure a reliable blob pose detection. Since the detection is to be done on board the UAVs, RGB-D cameras are selected. Therefore, the inputs to the framework are images and organized point clouds obtained from the visual sensor. The remainder of this section introduces the individual components of the framework and adds implementation details where necessary. Figure 6. The pipeline for the modular object detection framework. Inputs to the system are an arbitrary number of sensor messages (image, depth, point cloud, etc.) along with sensor–world frame transformations. Output is the detected blob pose in the world frame. The synchronizer and detector are modular components, while the linear Kalman ﬁlter, world transformation and pose tracker stay invariant. The sensor message synchronizer is responsible for the time-based synchronization of the given sensor message streams. In the case of blob detection, a module that synchronizes images and organized pointclouds from an RGB-D camera is derived. This is necessary since the algorithm detects the blob in both 2D image space and 3D point clouds, which are not necessarily sampled simultaneously. The underlying implementation uses ROS libraries to synchronize messages with an approximate time policy. An object detector attempts to ﬁnd a set of object poses using synchronized sensor data. The module used in this paper detects blob poses and is implemented in the following way. First, all the blob positions and radii are found in the image frame using the standard blob detection functionality found in the OpenCV libraries. Second, the depth information corresponding to the detected blobs is isolated from the organized point cloud. Finally, blob positions are calculated as centroids of the corresponding depth positions, while the orientation is obtained through the random sample consensus (RANSAC) algorithm from the Point Cloud Library (PCL). The remaining framework components are independent from synchronizer and detec- tor modules. The pose tracker is used to track the obtained object’s poses through multiple frames based on the closest Euclidean distance criterion. This component solves the issue of multiple objects being visible, as it always outputs the pose of the currently tracked object. Moreover, it increases the robustness of the system since it remembers the object poses for a certain number of frames, which allows some leniency with the detector. The goal of the world transformation component is to transform the tracked pose from the sensor to the world frame using the estimated odometry from an external source Appl. Sci. 2021, 11, 8279 14 of 21 that any UAV should have access to. Additionally, since the blob poses are to be sent as references to the trajectory planner, it is important to correctly compute the blob orientation. Since the blob is a ﬂat surface, there are two equally correct possible orientations that can be detected. Therefore, the blob orientation is chosen as follows: R if r r 0 1B blob 1blob R = , (22) blob R R otherwise blob 180 where r is the heading component of the UAV rotation matrix expressed in world 1B h i coordinates R = and r is the heading component of the blob rota- r r r B 1blob 1B 2B 3B h i tion matrix expressed in world coordinates R = and R = r r r blob 180 1blob 2blob 3blob diag(1,1, 1). Finally, a linear Kalman ﬁlter with a constant velocity model is used to further increase the robustness of the system and provide smoother blob position estimates. The constant velocity model for each axis is given as follows: " # 1 T x = F x + w , F = , (23) k+1 k k k k 0 1 where T is the discretization step, x 2 R is the state vector containing the position and velocity along the corresponding axis and w 2 R is the process noise. The observation model along a single axis is given as follows: h i z = H x + v , H = , (24) 1 0 k x k k k where z 2 R is the position observation along the corresponding axis and v 2 R is the k k measurement noise. If the detector is unable to provide measurements and the pose tracker removes the pose from the tracking set, the linear Kalman ﬁlter is still able to provide blob position estimates. Experimental validation of the described methods is performed in an indoor Optitrack environment with an Intel Realsense D435 RGB-D camera. To ensure ground truth is available for detection validation, reﬂective markers are attached to both the camera and the blob. In order to determine the transformation between the camera optical frame and the reﬂective markers attached to the camera, an optimization-based calibration approach is used as described in [32]. Results are shown in Figures 7 and 8. The experiments are performed with the UAV in constant motion while looking at the general direction of the painted blob. Figure 7 shows a relative difference between the ground truth UAV motion in the world frame and the UAV motion as observed from the detected blob frame. Figure 8 presents the comparison of ground truth and detected blob positions expressed in the world frame. It is important to note that camera calibration errors can manifest themselves as static offsets between the detected and ground truth blob positions in Figure 8. However, in this case, the visual detection provided a reliable blob tracking results which is a direct consequence of careful camera calibration. Appl. Sci. 2021, 11, 8279 15 of 21 0.4 0.2 -0.2 -0.4 0.6 0.4 0.2 -0.2 W W p [k] - p [0] B B 0.2 blob blob p [k] - p [0] B B -0.2 0 5 10 15 20 25 30 35 40 45 Time / [s] Figure 7. This ﬁgure shows the comparison of normalized motion of the UAV body frame as observed W W from the Optitrack world frame and as observed from the detected blob frame labeled p [k] p [0] B B blob blob and p [k] p [0], respectively. B B 0.5 0.4 0.3 0.2 0.1 -0.1 -0.2 0.2 p - ground truth blob 0.1 p - estimate blob -0.1 5 10 15 20 25 30 35 40 45 Time / [s] Figure 8. This ﬁgure shows the comparison of the ground truth blob position and the blob position estimate expressed in the Optitrack world frame. The root mean squared error across all three axes is 4.6 cm. Z / [m] Y / [m] Z / [m] Y / [m] X / [m] X / [m] Appl. Sci. 2021, 11, 8279 16 of 21 7. Simulation The environment used for simulating the UAV and manipulator dynamics, as well as the contact with the environment, is the widely accepted Gazebo simulator. It is real- istic and highly modular, with a large community and a support for the robot operating system (ROS), which is also the primary implementation environment for impedance control, motion planning and blob detection. Through ROS, Gazebo has a large variety of developed plugins realistically simulating various sensors and actuators. All simu- lations were conducted with Linux Ubuntu 18.04 operating system and ROS Melodic middleware installed. The UAV is modeled as a single rigid body with n propellers mounted at the end of each arm. As propulsion units, these propellers generate thrust along the z axis of the UAV body. To simulate the propeller dynamics, the rotors_simulator package is used. It contains a plugin that models thrust based on the user-provided propeller parameters [33]. Furthermore, to obtain the UAV attitude and position, IMU and odometry plugins are mounted on the vehicle. The manipulator was mounted on the body of the UAV and consists of three joints connected with links. A rod type tool is mounted as the end-effector, with a force-torque sensor required by the impedance controller. Furthermore, a monocular camera with an infrared projector is also mounted for the blob detection. 7.1. End-Effector Motion Distribution Analysis Given some end-effector conﬁguration, the inverse kinematics is responsible for ﬁnding the UAV position and yaw angle, as well as the manipulator joint values that satisfy the desired conﬁguration. The parameter b from Equation (3) deﬁnes a ratio of how much the manipulator joints and UAV position and orientation contribute to achieving the desired end-effector conﬁguration, as described in Section 3.1. Recalling the values, b = 1 only moves the UAV in the direction of the desired end-effector conﬁguration; and b = 0 uses the inverse kinematics of the manipulator to achieve the desired conﬁguration. To determine the inﬂuence of b on the overall system, an analysis was conducted with different b values. The desired end-effector conﬁguration was chosen to be in contact with a plane perpendicular to the bridge wall which required the force reference along the x axis. The waypoints for the trajectory planner were kept the same across all trials, and only b was changed. The results of this analysis are depicted in Figure 9. As can be observed, all trials produced very similar results with oscillating force upon contact and eventually reaching the desired reference, providing us with no obvious conclusion regarding how to select the optimal b. However, following the dexterity analysis from Section 3.1, and only relying on the manipulator motion might drive the system close to its limits due to the UAV body movement. On the other hand, the motion of the UAV induces disturbances in the end-effector pose control. The manipulator is therefore responsible for compensating errors introduced by the motion of the UAV body. Taking all of the aforementioned into account, the value is chosen as b = 0.5 so that both the manipulator and the UAV are simultaneously used to maintain a steady contact force. 0.2 -=0 -=0.2 -=0.4 -=0.5 -=0.6 -0.2 -=0.8 -=1 Reference -0.4 -0.6 -0.8 -1 0 5 10 15 20 25 30 t[s] Figure 9. Force response comparison for different values of the parameter b. The analysis was conducted on a plane perpendicular to the ground where the force reference along the x axis is required. f [N] x Appl. Sci. 2021, 11, 8279 17 of 21 7.2. Bridge Sensor Mounting Since the concept of this paper was to mount inspection sensors on a bridge, the simulation trials were tailored in the same direction. After spraying the ﬁrst component, it is necessary to achieve and maintain a stable contact while the second adhesive component on the sensor dries. Since the manipulator is attached above the propellers, the workspace of the manipulator is limited to contact above the UAV or on the plane perpendicular to the ground. Naturally, the ﬁrst set of simulation trials were conducted by holding the desired force on a plane perpendicular to the ground. In this case, the contact force only acts along the x axis and the response is depicted in Figure 10. The time delay between the planned and executed contact is present due to the impedance ﬁlter which slows down the dynamics of the referent trajectory. After the initial contact, there are some oscillations and an overshoot which diminish over time and the desired force reference is achieved. Measured Reference -0.2 -0.4 -0.6 -0.8 0 5 10 15 20 25 30 35 40 45 50 t[s] Figure 10. Force response in case of a contact plane perpendicular to the ground, d = 0 . The second set of simulation trials included an inclined contact plane. This requires the UAV approach from below the plane and achieving contact perpendicular to the plane. Since the plane is inclined for d = 68 , the planned force referent values have components in both the x and z axes, as shown in Figure 11. Similarly to the previous example, the force response has some oscillations around the instance of contact, but it eventually settles and reaches the desired force reference. Measured -0.05 Reference -0.1 -0.15 -0.2 -0.25 -0.3 0 5 10 15 20 25 30 35 40 45 50 t[s] 0.5 0.4 0.3 0.2 0.1 -0.1 0 5 10 15 20 25 30 35 40 45 50 t[s] Figure 11. Force response in the case of the contact plane inclined for d = 68 . The simulation tests for d = 0 and d = 68 were performed n = 10 times for each case, as depicted in Figure 12. The left portion of the ﬁgure is a dot product between the normal of the blob r and the end-effector orientation vector r . If the value of the dot t ee product r r = 1, the two vectors are parallel which results in a successful approach. For t ee f [N] f [N] f [N] x x z Appl. Sci. 2021, 11, 8279 18 of 21 both angles, the dot product is very close to 1 and the orientation error is negligible. On the right, the distance between the center of the target and contact point is shown. The error distance is in both cases less than 0.1 m, which ensures the relatively high precision of sensor mounting, well within margins for the bridge inspection. The accompanying video of simulation tests can be found on our YouTube channel [34]. Figure 12. (Left): box and whiskers plot of the dot product between the blob plane normal vector r and the end-effector orientation vector r ; and (Right): box and whiskers plot of the distance from ee the target point after contact. 8. Conclusions This paper presents a step towards autonomous bridge inspection by investigating the possibility of mounting various inspection sensors using an aerial manipulator. Cur- rently, inspectors use specialized trucks with cranes and baskets in order to access the area underneath the bridge. This inevitably leads to road closure which poses an inconvenience for both inspectors and trafﬁc. To alleviate this problem, the aforementioned aerial manip- ulators can be used to access difﬁcult-to-reach areas of the bridge. As mounting sensors require forming a bond between the wall and sensor, we envision using a two-component adhesive with a short cure time. Since the aerial manipulator has to achieve and maintain contact with the sensor mount point, short cure times are desirable because of te limited ﬂight time of these platforms. Nevertheless, current ﬂight times of outdoor multirotors reach up to 30 min, which ensures enough time for the two adhesive components to form the bond. Although preliminary, the results of this paper seem promising. The visual detection was extensively tested and reliably tracks the blob position. The adaptive impedance controller is capable of maintaining the required force. Even though there are some oscillations and settling times in the force response, in practical use, it does not make much difference since the curing time of the adhesive is at least several minutes. The trajectory planner was augmented to plan in the force space which allows for setting the force reference step change before the contact. The simulation results show the high repeatability of the overall system which gives us the conﬁdence to perform experiments in a real-world environment. Our ﬁrst step in future work was to perform experiments in a controlled laboratory environment. The outdoor environment poses a different set of challenges including the lower accuracy positioning system and unpredictable disturbances, i.e., wind gusts. Since these factors will inevitably reﬂect on the overall end-effector accuracy, we are looking into augmenting the manipulator to be able to compensate for lateral movements, as well as roll and yaw angles. To further increase the system’s accuracy, the developed visual Appl. Sci. 2021, 11, 8279 19 of 21 tracker will be used to improve feedback around the tracked blob on the bridge wall in real-world experiments. Author Contributions: All authors contributed equally to this work. Conceptualization, M.O. and I.D.; methodology, A.I., L.M. and M.C.; software, A.I., L.M. and M.C.; investigation, A.I.; writing— original draft preparation, A.I.; writing—review and editing, L.M., I.D. and M.O.; supervision, I.D. and M.O.; project administration, I.D. and M.O.; funding acquisition I.D. and M.O. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the European Commission Horizon 2020 Programme through the project under G.A. number 820434, named Energy-Aware BIM Cloud Platform in a Cost-Effective Building Renovation Context—ENCORE. Furthermore, this research was part of the scientiﬁc project Autonomous System for Assessment and Prediction of Infrastructure Integrity (ASAP), ﬁnanced by the European Union through the European Regional Development Fund—The Competitiveness and Cohesion Operational Programme (KK.01.1.1.04.0041). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Conﬂicts of Interest: The authors declare no conﬂict of interest. Appendix A Here, we show detailed proof that the adaptation law (17) is stable. Assuming that THE referent force f (t) is constant, f (t) = F , the time derivatives of Equation (16) are: r r r x˙ (t) = k ˙(t)F + x˙ (t), r r e (A1) x¨ (t) = k ¨(t)F + x¨ (t), r r e while the derivatives of Equation (13) yield: e(t) ˙ ˙ x(t) = + x (t), (A2) e ¨(t) x ¨(t) = + x¨ (t). By susbstituting Equations (A1) and (A2) into Equation (10), the dynamics of the contact force error can be obtained as me ¨(t) + be˙(t) + (k + k )e(t) = g(t), (A3) where: ˙ ¨ g(t) = F [k(1 k k(t)) k (bk(t) + mk(t))], (A4) r e e and x = x . The adaptation law thus determines the dynamics of the adaptation parameter k(t) and deﬁnes the dynamics of the contact force error. Formally, the adaptation law should enforce g(t) ! g such that e(t) ! 0 and k(t) ! 1/k . For the Lyapunov candidate: h i 1 1 2 2 V(t) = p e(t) + p e˙ (t) + [g(t) g ] , (A5) 1 2 2 2g with p , p and g as positive parameters, the condition V(t) 0 yields: 1 2 2 2 g(t)g ˙(t) + 2g(t)[ p e(t) + p e˙(t)] g ˙(t)g 0. (A6) 1 2 g g Appl. Sci. 2021, 11, 8279 20 of 21 After reordering, we obtain: g ˙(t)[g(t) g ] gs(t)g(t), (A7) where s(t) = [ p e(t) + p e˙(t)]. By choosing: 1 2 g ˙(t) = gs(t) + g s ˙ (t), (A8) where g is a positive constant, Lyapunov condition Equation (A7) becomes: g(t) g , (A9) i.e., for the adaptation law to be stable, g(t) should be bounded. Since x , x ˙ and x ¨ r r r are bounded, so are e, e˙ and e ¨. Therefore, g(t) is also bounded, i.e., the condition in Equation (A9) is satisﬁed. The adaptation law is ﬁnally obtained by taking the derivative of Equation (A3), and substituting g ˙(t) with Equation (A8), yields the (17). Parameters g and g dictate the adaptation dynamics. Based on the measured contact force, the error adaptation law Equation (17) estimates the adaptation parameter k (reciprocal value of the environment stiffness), which is then used in Equation (16) for calculating the referent position x . References 1. Kashif Ur Rehman, S.; Ibrahim, Z.; Memon, S.A.; Jameel, M. Nondestructive test methods for concrete bridges: A review. Constr. Build. Mater. 2016, 107, 58–86. [CrossRef] 2. Hunt, G.; Mitzalis, F.; Alhinai, T.; Hooper, P.A.; Kovac, M. 3D printing with ﬂying robots. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 4493–4499. [CrossRef] 3. Vempati, A.S.; Kamel, M.; Stilinovic, N.; Zhang, Q.; Reusser, D.; Sa, I.; Nieto, J.; Siegwart, R.; Beardsley, P. PaintCopter: An Autonomous UAV for Spray Painting on Three-Dimensional Surfaces. IEEE Robot. Autom. Lett. 2018, 3, 2862–2869. [CrossRef] 4. Jeong, E.; Seo, J.; Wacker, J. Literature Review and Technical Survey on Bridge Inspection Using Unmanned Aerial Vehicles. J. Perform. Constr. Facil. 2020, 34, 04020113. [CrossRef] 5. Cacace, J.; Orozco-Soto, S.M.; Suarez, A.; Caballero, A.; Orsag, M.; Bogdan, S.; Vasiljevic, G.; Ebeid, E.; Rodriguez, J.A.A.; Ollero, A. Safe Local Aerial Manipulation for the Installation of Devices on Power Lines: AERIAL-CORE First Year Results and Designs. Appl. Sci. 2021, 11, 6220. [CrossRef] 6. Zollini, S.; Alicandro, M.; Dominici, D.; Quaresima, R.; Giallonardo, M. UAV Photogrammetry for Concrete Bridge Inspection Using Object-Based Image Analysis (OBIA). Remote Sens. 2020, 12, 3180. [CrossRef] 7. Bolourian, N.; Hammad, A. LiDAR-equipped UAV path planning considering potential locations of defects for bridge inspection. Autom. Constr. 2020, 117, 103250. [CrossRef] 8. Reagan, D.; Sabato, A.; Niezrecki, C. Unmanned aerial vehicle acquisition of three-dimensional digital image correlation measurements for structural health monitoring of bridges. In Proceedings of the Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infrastructure, Portland, OR, USA, 25–29 March 2017; Wu, H.F., Gyekenyesi, A.L., Shull, P.J., Yu, T.Y., Eds.; International Society for Optics and Photonics, SPIE: Bellingham, DC, USA, 2017; Volume 10169, pp. 68–77. [CrossRef] 9. Chen, H.; Quan, F.; Fang, L.; Zhang, S. Aerial Grasping with a Lightweight Manipulator Based on Multi-Objective Optimization and Visual Compensation. Sensors 2019, 19, 4253. [CrossRef] [PubMed] 10. Zhang, G.; He, Y.; Dai, B.; Gu, F.; Yang, L.; Han, J.; Liu, G. Aerial Grasping of an Object in the Strong Wind: Robust Control of an Aerial Manipulator. Appl. Sci. 2019, 9, 2230. [CrossRef] 11. Sanchez-Cuevas, P.J.; Gonzalez-Morgado, A.; Cortes, N.; Gayango, D.B.; Jimenez-Cano, A.E.; Ollero, A.; Heredia, G. Fully- Actuated Aerial Manipulator for Infrastructure Contact Inspection: Design, Modeling, Localization, and Control. Sensors 2020, 20, 4708. [CrossRef] [PubMed] 12. Baca, T.; Penicka, R.; Stepan, P.; Petrlik, M.; Spurny, V.; Hert, D.; Saska, M. Autonomous Cooperative Wall Building by a Team of Unmanned Aerial Vehicles in the MBZIRC 2020 Competition. arXiv 2020, arXiv:2012.05946. 13. Vatavuk, I.; Polic, M.; Hrabar, I.; Petric, F.; Orsag, M.; Bogdan, S. Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at MBZIRC 2020. J. Field Robot. 2021, submitted for publication. 14. Barišic, ´ A.; Petric, F.; Bogdan, S. Brain over Brawn—Using a Stereo Camera to Detect, Track and Intercept a Faster UAV by Reconstructing Its Trajectory. arXiv 2021, arXiv:2107.00962. 15. Markovic, L.; Car, M.; Orsag, M.; Bogdan, S. Adaptive stiffness estimation impedance control for achieving sustained contact in aerial manipulation. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation, Xi’an, China, 30 May–5 June 2021. Appl. Sci. 2021, 11, 8279 21 of 21 16. Cazzato, D.; Cimarelli, C.; Sanchez-Lopez, J.L.; Voos, H.; Leo, M. A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles. J. Imaging 2020, 6, 78. [CrossRef] 17. Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM 1981, 24, 381–395. [CrossRef] 18. Fumagalli, M.; Naldi, R.; Macchelli, A.; Carloni, R.; Stramigioli, S.; Marconi, L. Modeling and control of a ﬂying robot for contact inspection. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal, 7–12 October 2012; pp. 3532–3537. [CrossRef] 19. Alexis, K.; Darivianakis, G.; Burri, M.; Siegwart, R. Aerial Robotic Contact-based Inspection: Planning and Control. Auton. Robot. 2016, 40, 631–655. [CrossRef] 20. Jimenez-Cano, A.E.; Sanchez-Cuevas, P.J.; Grau, P.; Ollero, A.; Heredia, G. Contact-Based Bridge Inspection Multirotors: Design, Modeling, and Control Considering the Ceiling Effect. IEEE Robot. Autom. Lett. 2019, 4, 3561–3568. [CrossRef] 21. Cataldi, E.; Muscio, G.; Trujillo, M.A.; Rodriguez, Y.; Pierri, F.; Antonelli, G.; Caccavale, F.; Viguria, A.; Chiaverini, S.; Ollero, A. Impedance Control of an aerial-manipulator: Preliminary results. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 3848–3853. [CrossRef] 22. Nava, G.; Sablé, Q.; Tognon, M.; Pucci, D.; Franchi, A. Direct Force Feedback Control and Online Multi-Task Optimization for Aerial Manipulators. IEEE Robot. Autom. Lett. 2020, 5, 331–338. [CrossRef] 23. Meng, X.; He, Y.; Li, Q.; Gu, F.; Yang, L.; Yan, T.; Han, J. Contact Force Control of an Aerial Manipulator in Pressing an Emergency Switch Process. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 2107–2113. [CrossRef] 24. Stilman, M. Global Manipulation Planning in Robot Joint Space With Task Constraints. IEEE Trans. Robot. 2010, 26, 576–584. [CrossRef] 25. Cefalo, M.; Oriolo, G. Task-constrained motion planning for underactuated robots. In Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, 26–30 May 2015; pp. 2965–2970. [CrossRef] 26. Yu, Y.; Lippiello, V. 6D Pose Task Trajectory Tracking for a Class of 3D Aerial Manipulator From Differential Flatness. IEEE Access 2019, 7, 52257–52265. [CrossRef] 27. Tognon, M.; Cataldi, E.; Chavez, H.A.T.; Antonelli, G.; Cortés, J.; Franchi, A. Control-Aware Motion Planning for Task-Constrained Aerial Manipulation. IEEE Robot. Autom. Lett. 2018, 3, 2478–2484. [CrossRef] 28. Ivanovic, A.; Car, M.; Orsag, M.; Bogdan, S. Exploiting Null Space in Aerial Manipulation through Model-In-The-Loop Motion Planning. In Proceedings of the 2020 International Conference on Unmanned Aircraft Systems (ICUAS), Athens, Greece, 1–4 September 2020; pp. 686–693. [CrossRef] 29. Zargarbashi, S.; Khan, W.; Angeles, J. The Jacobian condition number as a dexterity index in 6R machining robots. Robot. Comput.-Integr. Manuf. 2012, 28, 694–699. [CrossRef] 30. Car, M.; Ivanovic, A.; Orsag, M.; Bogdan, S. Position-based adaptive impedance control for a UAV. In Proceedings of the 2018 International Conference on Unmanned Aircraft Systems (ICUAS), Dallas, TX, USA 12–15 June 2018; pp. 957–963. [CrossRef] 31. Pham, H.; Pham, Q. A New Approach to Time-Optimal Path Parameterization Based on Reachability Analysis. IEEE Trans. Robot. 2018, 34, 645–659. [CrossRef] 32. Maric, B.; Polic, M.; Tabak, T.; Orsag, M. Unsupervised optimization approach to in situ calibration of collaborative human–robot interaction tools. In Proceedings of the 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Karlsruhe, Germany, 14–16 September 2020; pp. 255–262. [CrossRef] 33. Furrer, F.; Burri, M.; Achtelik, M.; Siegwart, R. Robot Operating System (ROS): The Complete Reference (Volume 1). In Studies in Computational Intelligence; Chapter RotorS—A Modular Gazebo MAV Simulator Framework; Springer International Publishing: Cham, Switzerland, 2016; pp. 595–625. [CrossRef] 34. LARICS. Aerial Manipulation Approach for Autonomous Bridge Inspection. Available online: https://youtu.be/bBaxPMZEV04 (accessed on 23 July 2021).

Journal

Applied Sciences – Multidisciplinary Digital Publishing Institute

Published: Sep 7, 2021

Keywords: aerial robotics; inspection and maintenance; aerial manipulation; multirotor control

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

Towards Autonomous Bridge Inspection: Sensor Mounting Using Aerial Manipulators

References (36)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies