Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

Teng Sun; Hongbin Liu

doi:10.1007/s10514-019-09896-7

Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

Sun, Teng; Liu, Hongbin 2020-05-21 00:00:00 Haptic exploration is a process of using haptic feedback to interact and perceive an unknown object. It is an essential approach to understand the physical and geometrical properties of the object. While numerous research has been carried out for haptic exploration on static objects, haptic exploration on objects with dynamic movements has not been reported. It is due to the signiﬁcant challenges to achieve robust force and velocity control when the object is nonstationary. In this work, a novel adaptive force and velocity control algorithm based on intrinsic contact sensing (ICS) for haptic surface exploration of dynamic objects is presented. A fuzzy-logic control framework making use of the information obtained from ICS has been developed. To validate the proposed control algorithm, extensive surface exploration experiments have been carried out on objects with different surface properties, geometries, stiffness, and concave or convex patterns. The validation results demonstrate the high accuracy and robustness of the proposed algorithm using different experimental platforms. Keywords Haptic · Autonomous control · Fuzzy logic 1 Introduction contact sensing hardware and intelligent control algorithms (Tegin and Wikander 2005). Haptic exploration of the surface is a fundamental tool for Due to the direct interaction with the object, signiﬁcant humans to understand the surface properties of unknown hurdles in control and sensing are presented for haptic explo- objects with only tactile sensing (Lederman and Klatzky ration (Okamura and Cutkosky 2001). With the emergence of 1993, 2009), especially in reduced visibility scenarios like new force/tactile sensing hardware and data processing meth- underwater, smoky disaster environment and inside tissues ods, several new haptic exploration algorithms on unknown during surgery. It has many useful applications in areas like objects and unstructured environment have been developed. robot surgery (Amirabdollahian et al. 2017), shape recov- Lepora et al. have been working on the development of ery (Rosales et al. 2018), heritage recording (Jamil et al. the optical tactile sensors with bio-mimetic morphologies 2018), dexterous robot hand (Kappassov et al. 2015). Unlike (Ward-Cherrier et al. 2018), and they have used thus sen- computer vision which has been progressing tremendously in sors for robust contour following with deep learning method recent years, haptic sensing and interaction remain as techni- (Lepora et al. 2019). Yin et al. (2018) developed a bio- cally challenging due to the relatively limited development of inspired tactile sensor skin for measuring dynamic shear force and vibration. Fishel and Loeb (2012) and Xu et al. (2013) proposed a Bayesian exploration method for iden- tiﬁcation of textures where they choose the exploratory Electronic supplementary material The online version of this article (https://doi.org/10.1007/s10514-019-09896-7) contains movements that would yield the most useful information. supplementary material, which is available to authorized users. Hellman et al. (2018) presented a reinforcement learning method for contour following with haptic feedback. Yang and B Hongbin Liu Lepora (2017) tried to combine vision for object exploration hongbin.liu@kcl.ac.uk for edge estimation. Sommer and Billard (2016) presented a Teng Sun Multi-contact haptic exploration method manly for grasping teng.sun@kcl.ac.uk tasks. Su et al. (2012) controlled the movement of the robot Haptic Mechatronics and Medical Robotics (HaMMeR) Lab, arm in the contact normal direction with haptic feedback to King’s College London, London, UK 123 774 Autonomous Robots (2020) 44:773–790 characterize the compliance of the pressed object. Similarly, tangential force (Liu et al. 2012, 2015). With the precise con- Sornkarn and Nanayakkara (2017) used a soft robotic probe tact information, an efﬁcient algorithm has been designed to measure object stiffness with haptic control. based on normal force conditions to control the robot ﬁnger Konstantinova et al. (2014) presented that humans use to explore the surface while maintaining the contact normal different force or velocity control strategies to detect abnor- force (Back et al. 2015). However, there are three major lim- mality inside soft objects, and the performance of the itations in the previous algorithms to prevent it from being perception can be enhanced by controlling the exploration applicable to dynamic objects. First, previous control method speed. And Okamura (2000) showed that it is important to is prone to induce large force control error when the surface maintain a proper normal force level during sliding explo- has large curvature changes or uneven. Second, the controller ration procedures to guarantee the force sensing accuracy, has a ﬁxed set of parameters and cannot be adapted to differ- considering the practical limitation of the sensor hardware. ent objects to achieve similar good performance. Third, the Furthermore, if the explore velocity can be controlled, more velocity of the ﬁnger sliding is not controllable. information of the explored object such as surface viscosity In this study, the following contributions have been (Liu et al. 2012), break-away ratio (Song et al. 2014) can be made: (1) A robust and adaptive normal force and explore revealed. These all showed the paramount importance of the (scanning) velocity control algorithm was developed based good control of contact forces and scanning velocity during on haptic information during surface exploration. It was haptic exploration. evaluated using a self-fabricated contact sensing ﬁnger on However, to date, to achieve the force and explore velocity non-stationary objects. Objects with variant surface physi- control during haptic exploration is still an open challenge, cal and geometry properties were tested and results show and an effective method for doing so has not been found that the normal force control method can achieve an aver- in literature. In most of the previous works, the exploration age mean square error (MSE) less than 0.006 and standard procedure has been carried out by scanning over a relatively deviation (SD) less than 0.07 N. Also, the explore velocity ﬂat surface of the object or by pressing. Some researchers can be controlled with an average Δv of 0.48 mm/s and SD considered force control during 1 DoF compressing motion of 2.07 mm/s. (2) A fuzzy-logic control framework was cre- (Su et al. 2012; Sornkarn and Nanayakkara 2017). A few ated that can increase the ﬂexibility of the control algorithm, studies have been done on the sliding exploration but only especially when the object is non-stationary and its proper- limited to static objects with simple geometry (like ﬂat sur- ties are unstructured. The control rules of the fuzzy method face), avoiding the problem of feedback force and velocity can be changed adaptively corresponding to different objects control (Fishel and Loeb 2012; Hellman et al. 2018). Similar being explored by shifting the membership boundaries. (3) to the work proposed in this study, Lepora et al. (2017) has The control algorithm can be applied to devices with different proposed an exploratory tactile servoing method where the kinematic conﬁgurations that have different workspace and next exploration position was decided with a Bayesian per- force ranges. The algorithm was validated successfully with ceptual framework, but the force was not actively controlled a 6 DoFs robot arm (with a contact sensing tip mounted) in their study and the objects were static. Li et al. (2013) using objects with different stiffness and geometries. (4) introduced a tactile servoing control framework adapting a Even though the type of contact was point contact in the PID controller, which can be used to realize a speciﬁc tactile tests, it also works for surface contact (using tactile array interaction pattern. Yang et al. (2013) presented a learning sensor) if contact locations can be obtained in real-time. algorithm for simultaneous force control and haptic identiﬁ- cation, where they used complicated models instead of real force sensor, and the algorithm was only tested in simulation. 2 Experimental platform Okamura and Cutkosky (2001) used a 2 DoFs robot ﬁnger to detect features with haptic exploration, where they presented Three platforms have been created to validate the proposed that the force was kept low; however, no detailed results were algorithm, which are denoted as platform-a, b and c. The shown. Therefore, to the best of the surveyed knowledge, the designed experimental platform-a consists of two main parts: robust and accurate normal force and explore velocity con- one is the self-fabricated contact sensing ﬁnger, and the other trol when the object is dynamically moving during surface is the Universal Robot (UR)-3 robot arm. The contact sens- exploration has not been reported. ing ﬁnger consists of two joints, one ﬁngertip and the base To overcome the challenges, continuous study has been connecting the UR-3 robot arm, as shown in Fig. 1. Each carried out on enhancing haptic sensing capability to realise joint consists of a DC motor (Maxon DC motor RE10 with adaptive contact force and velocity control. Adapting ICS encoder) and a bevel-gear box (reduction: 1:1). The Maxon with 6-axis force-torque measurement (Bicchi et al. 1993), control panel (POS2 24/2) is used to control the position or a real-time contact sensing algorithm has been developed velocity of the motor. The maximal frequency of the position to provide instantaneous contact location, normal force and or velocity control is 1 kHz, and such high frequency enables 123 Autonomous Robots (2020) 44:773–790 775 Fig. 2 The haptic exploration platform-c involving a UR-3 robot arm and a contact sensing tip. The tip has a Nano17 force-torque sensor and a self-fabricated cap made of silicon. The tip is mounted to the end-effector of the arm using a 3D printed connection materials that are softer and more ﬂexible will be future work. Inside the ﬁngertip, an ATI Nano17 force-torque sen- Fig. 1 The contacting sensing ﬁnger (platform-a), which consists of 2 sor (resolution: 1/160 N) was installed. It is a six-axis sensor joints and a ﬁngertip, is shown in the top pictures. The joints consist providing force and torque data along its x, y, z-axis, and the of 3D printed links and metal connection parts, and each has a Maxon motor and bevel gearbox inside. An ATI Nano17 6-axis force-torque data can be updated at a frequency of 7200 Hz with a DAQ sensor is embedded inside the ﬁngertip and covered with a polyurethane data acquisition box. rubber layer. It has a 3D printed base that can be mounted on the UR-3 The software was developed with ROS (robot operating robot arm (bottom picture) as platform-b. The robot arm can rotate to system, www.ros.org) using an Ubuntu PC (CPU: i7-6700 any desired angle to allow the ﬁnger to explore more areas @3.4GHz and 16G RAM). The PCIe-6320 data acquisition card (up to 16 kHz) is used to obtain sensor data, and then the high-level (trajectory generation and decision making) ROS programs are used to communicate with the card to get control to be accurate and rapid. For motor control, a multi- and process the data. In practical, the tactile sensing toolbox thread method is used to eliminate the asynchronization when (TST) proposed in Ciotti et al. (2019) was used for sensor controlling multiple motors, which ensures the accuracy of data processing and contact location estimation. Including the position control and is beneﬁcial for the control of more the other modules like the control and calculation, the loop joints or ﬁngers. The high precision of the motor control frequency of the ROS is 100 Hz, meaning it updates the sensor panel and the self-locking mechanism on the joint (the baf- data and ﬁnish all the necessary computations every 1/100 s. ﬂes at 90 degrees) ensure the repeatability of the exploration, The ﬁnger is installed on a 3D printed base, which can be making the online multi-time haptic exploration possible. mounted on the UR-3 robot arm. This ﬁnger and robot arm Experimental tests were also carried out to record the position combined platform is denoted as the platform-b. The base can deviation of the joints after several explorations and input the be rotated to any angle through the robot arm to achieve the error into the control system as the compensation to reduce exploration on more surfaces, as shown in Fig. 1 bottom. For the error caused by mechanical fault and friction between the the next steps, more ﬁngers will be mounted on the base to components. realize the larger area’s exploration and achieve more actions For fabricating the ﬁngertip, the inner core was ﬁrstly 3D- other than surface exploration. These together will help to lay printed with plastic and then embedded into a layer of 2 mm a solid hardware foundation for the active exploration and TM rubber. The Poly 74-30 liquid rubber from Polytek was dexterous manipulation system. used to construct the polyurethane rubber layer, its hardness Moreover, to evaluate the proposed algorithm with larger is 30A Shore, and young ’s Modulus is 100psi. The whole working space and force ranges, the UR-3 robot arm was used ﬁngertip (3D printed core and rubber layer) has very high for haptic surface exploration as platform-c. The Nano17 sen- hardness and is close to rigid body; therefore there was no sor was covered with a silicone cover and ﬁxed onto the robot deformation within the range of the forces used in exper- arm end-effector to provide real-time force-torque data for iments, and no effect of hysteresis. The study of ﬁngertip the control algorithm, shown in Fig. 2. Since the workspace 123 776 Autonomous Robots (2020) 44:773–790 of the ﬁnger is limited and the force generated by it is rel- Table 1 Parameter notation for control system atively small, the robot arm with more ﬂexibilities helps to Name Deﬁnition validate the proposed algorithm in larger workspace and force ε Force error between measured and target, N control range. On the contrary, the contacting sensing ﬁnger ε ˙ Change rate of error, N/s can realize ﬁner force and velocity control due to the high F Desired normal force, abs value, N frequency and accuracy of the motors, which is a good tool N d to verify the robustness of the control algorithm in the low F Measured normal force, abs value, N N m target range. More will be detailed in the experiment section. K Fuzzy control output, a coefﬁcient P Coordinates of current position in base frame now P Coordinates of contact point in ﬁngertip frame 3 Control algorithm P Coordinates of next position in base frame next With the experimental platform, an adaptive real-time nor- mal force and explore velocity control algorithm has been Table 2 Parameter calculations developed. A fuzzy-logic framework using the haptic infor- Notation Calculation mation was designed for haptic exploration on objects with dynamic movements. In this section, the overall control pro- Frames Base frame {O}(X , Y ), O O cess will be shown ﬁrst, and then the detailed introduction of ﬁngertip frame {E}(Y , Z ) E E the main components constituting the control algorithm will Sum of joint angles θ = θ + θ 12 1 2 be presented. Angle of contact point θ = arctan2(z , y ) c e e in ﬁngertip frame 3.1 Overall control process Angle of the normal θ = θ − θ N 12 c force in base frame Correcting force F = abs(K (F − F )) N d The overall control process of the proposed algorithm is N Virtual tangential force F = K F ,(K =−1) P T P shown in Fig. 3 and Tables 1, 2 show the deﬁnition of the T Angle between β = variables. The control algorithm is a feedback control sys- correcting and guiding arctan2(−F , K F /|K |) T N tem, and the input of it has two parts: force and torque values force from the Nano17 force-torque sensor, each with three com- Angle of guiding force α = θ + β T T ponents ([ f , f , f )] and [m , m , m ] ), and the position x y z x y z in base frame information from the photoelectric encoders inside motors Next position to reach P = P + S ·[cosα, sinα] next now v (E1, E2). The control process starts when the ﬁngertip is contacting the object, and to determine that state, a threshold function is used, i.e. it is determined that the ﬁnger is contact- to humans’ attempt to feel the object with only touch. With ing the object when the force obtained from the force sensor the force and torque values obtained from the force-torque is larger than a pre-decided value ϑ . With this method, at sensor, the contact point position in ﬁngertip frame P can be the beginning, the ﬁnger is controlled to move close to the c obtained through methods presented in Liu et al. (2015) and surface of the object, and the contact state is conﬁrmed when 2 2 2 Ciotti et al. (2019). With (E1, E2) from motor encoders, the sqrt ( f + f + f )>ϑ . In practice, this action is similar x y z contact location in the base frame P is obtained through now forward kinematics. With P obtained, the next is to decide the new loca- now tion on the surface for the ﬁngertip to reach while keeping the normal force as desired. The resultant normal force and tangential force are obtained simultaneously with the con- tact position through the TST. The control target F is then subtracted from the measured normal force to get the error ε and ε is divided by the cycling time to get the changing t t rate of the error: ε ˙ = ε /dt.And (ε , ε ˙ ) are the input of t t t t the fuzzy control method. The output of the fuzzy control method is a coefﬁcient K , which is used to generate a cor- Fig. 3 The overall ﬂow of the control algorithm, which is a closed-loop recting force, helping the ﬁnger maintain the normal force. control system. The output K is obtained after the inputs (ε , ε ˙ ) have t t In detail, each K is used to provide a correcting force vec- been processed, and the inputs are updated with the motor movement. The overall loop rate is 100 Hz using ROS tor that leads the ﬁnger to move either far away or closer to 123 Autonomous Robots (2020) 44:773–790 777 ⎡ ⎤ ∂ S k − f z + f y − m the object surface along the normal direction of the object. y e z e x ∂x ⎢ ⎥ Then a virtual tangential force F is obtained by taking the ∂ S ⎢ ⎥ k − f x + f z − m z e x e y ⎢ ⎥ ∂ y opposite direction of the measure friction force F ,tomake T ⎢ ⎥ g(x) = =0(1) ⎢ ∂ S ⎥ k − f z + f x − m the ﬁnger to move forward along the surface of the object. y e y e z ⎢ ⎥ ∂z ⎣ ⎦ 2 2 2 These two forces form a total force vector that can be used x y z e e e + + − 1 2 2 2 to decide the next explore location. As shown in Table 2, K a b c is ﬁrst used to decide the magnitude of the correction force E T The contact information x =[P , k] can be obtained by (F ), and the angle α of the total force in frame {O} is calcu- identifying the roots of g(x) = 0. The Levenberg–Marquardt lated after getting the required angles (θ and β ). After that, (LM) method, which is a gradient-based iterative method for a S parameter is used to determine the size of the increment non-linear parameter estimation, was used to get the solution for the next location (P ), whose unit is mm. It can be next iteratively. After P has been acquired, with joint angles obtained through experiments or controlled with closed-loop measured by motor encoders, P can be obtained through control. now forward kinematics: With P been determined, the control signals for motors next are generated through inverse kinematics of the platform. The motors are then controlled by motor controllers to rotate to P = P ·[T T T ]· R (2) now 1 2 3 1 desired positions. The force changes induced by the move- ment of the object or local geometric changes will increase ⎡ ⎤ cosθ −sinθ 0 cosθ l 1 1 1 1 the force error, and the control algorithm will make adjust- ⎢ ⎥ sinθ cosθ 0 sinθ l + l 1 1 1 1 0 ments accordingly as quickly as possible within its capability. ⎢ ⎥ T = (3) ⎣ ⎦ 00 1 0 This control loop keeps cycling until the ﬁnger is not contact- 00 0 1 ing the object or any one of the ﬁnger joints has reached the predeﬁned limit. After that, the haptic surface exploration on ⎡ ⎤ cosθ −sinθ 0 cosθ l 2 2 2 2 an area of the contacting object is completed subsequently. ⎢ ⎥ sinθ cosθ 0 sinθ l 2 2 2 2 ⎢ ⎥ T = (4) ⎣ ⎦ 00 1 0 00 0 1 3.2 Contact location estimation ⎡ ⎤ 100 y ⎢ ⎥ 01 0 −(l + z ) 3 e ⎢ ⎥ In the control algorithm, there are two main coordinate T = (5) ⎣ ⎦ 001 0 frames, the base {O} and ellipsoid ﬁngertip frame {E}, and 000 1 the control method is based in the base frame. The control goal is to determine the next exploration location and main- R =[0, 0, 0, 1] (6) tain the magnitude of the normal force simultaneously. To realize this goal, the current contact point in the base frame P needs to be calculated ﬁrstly. Let the coordinates of 3.3 Next position determination now the contact point to be P =[x , y , z ] in frame {E}, and e e e P =[x , y , z ] in frame {O}. With the obtained current contact position in the base frame, now o o o It should be noted that the ﬁngertip material used in the the next position was derived considering the contact normal experimental platform is very stiff and the effect of hysteresis force. The parameters required for the next position deci- can be ignored. And to verify this, experiments were carried sion were calculated geometrically, and for easier view, a to evaluate the force-deformation relationship when loading schematic diagram shown in Fig. 4 was drew. Table 2 shows and releasing forces on the ﬁngertip. As a result, the force the calculation of the relevant variables. Two key points have reached 5 N when the deformation was less than 0.5 mm; thus, been considered when deciding the next position to explore: it was reasonable to ignore the deformation of the ﬁngertip the ﬁrst one is to maintain the magnitude of the normal force in this study. Therefore, the equilibrium equation in Liu et al. within the desired range, the other one is to control the ﬁn- (2015) has been simpliﬁed to (1), where a = b = 15, c = 9.5 ger to move forward following the surface of the object. For are the dimensions of the ellipsoid ﬁngertip, S is the func- the ﬁrst point, a virtual normal force was generated based on tion for the ellipsoid, and k is a scaling factor. As future (ε , ε ˙ ), which can guide the ﬁnger to move closer or away t t work, softer ﬁngertips will be fabricated to estimate the sur- from the normal direction of the object, and the normal force face haptic exploration on objects with different stiffness, will increase if closer to, and vice versa. This virtual normal which will be of great help in the research of ﬂexible and force is called the “correcting force”, and denoted as F . K soft manipulators. was obtained through the fuzzy control method, which will be 123 778 Autonomous Robots (2020) 44:773–790 which means the response time is affected by S . For exam- ple, if the object moves away from the ﬁnger, the error will become larger and P will be closer to the object next to increase the force. After P has been obtained, the motor control signals next were obtained through inverse kinematics. By rotating the motors to the required positions, the ﬁnger will move to explore the next location. Repeating this action until the ﬁnger ﬁnished contacting the object surface, the surface exploration was completed while maintaining the local nor- mal force in the whole exploration process. As shown above, providing the geometric conﬁguration of the tip, forward and inverse kinematics of the platform, the control variables can be obtained, regardless of the number of DoF of the platform. Which indicates that the control algorithm is not limited to this two joints ﬁnger, it can also be applied to more com- Fig. 4 A schematic diagram showing the calculation of related angles plicated device like robotic arm or soft robots. However, the when deciding the next explore location. The deﬁnition of the notations can be found in Table 2. The black curve contacting the contact sensing parameters for the fuzzy control method and rotation matrix ﬁnger is a sketch of the object outline. The right-hand side is a magniﬁed for different devices are slightly different due to their speciﬁc view of the contact area details. This ﬁgure only shows the situation moving resolution, exploration direction and force ranges. where the correcting force F has the same direction with the real Therefore, for different devices, re-calibration is necessary normal force, which means the real force is smaller than the target force, and the ﬁnger needs the push towards the object to increase the to achieve the best performance, which in theory needs to force be done only once. Practically, the calibration of the fuzzy control method can be done automatically with optimized methods by miniaturizing the error between the requested detailed in the next section. In practice, this decision-making and real haptic exploration data, which could be explored more in future work. strategy is similar to the rule human beings use when they explore the surface of the object. Humans normally push fur- ther when they want to increase the pressure and leave away 3.4 Fuzzy control method to reduce it. For the second point, a virtual tangential force F was Fuzzy control method has been studied extensively in robot introduced proportional to the tangential force on the ﬁnger- systems recently (Aviles et al. 2016). da Fonseca et al. tip, but in the opposite direction. This force can ensure that the (2017a) and da Fonseca et al. (2017b) used the fuzzy control ﬁnger moves forward along the surface of the object. Com- method to realize grasp and exploration tasks with a three- bined with F , a total force F was obtained. The force F ﬁngered robotic hand and have achieved good results. It is a G G can guide the ﬁnger to move along the surface of the object method that mimics the intuitive decision-making process of and maintain the normal force, which is called the “guid- humans. Humans can solve complex problems using impre- ing force”. And based on this guiding force, the direction of cise information such as common sense, expert knowledge, the next position P in base frame was calculated. Table or prior experience which does not require precise models next 2 shows the detailed calculation of related angles. First, the (Takagi and Sugeno 1993). Such information can be repre- angle θ of the normal force in frame {O} was calculated sented by linguistic rules, like in If-Then format (Tanaka and based on the angle of contact point in frame {E}, then the Wang 2004). And fuzzy logic is the theory of fuzzy sets that angle β between F and F was obtained after obtaining the can represent mathematically the human spirit for approx- forces, ﬁnally the angle α of F in frame {O} was calculated. imation reasoning based on imprecise information, which Next, S was introduced to determine the incremental is also a major advantage over other classic control meth- size of the next exploration position compared to the cur- ods that usually require very precise modelling or parameter rent position. As mentioned above, it can determine not only ﬁne-tuning. When the external environment is complex, or the velocity of exploration but also the control accuracy the current state is unknown, it will be difﬁcult to obtain an and sensitivity to external disturbances induced by geome- accurate model due to the unclear state and inﬂuence fac- try changes, surface roughness and the movement of objects. tors. In such scenarios, the fuzzy control method can better Practically, when the object moves, the force error with the satisfy the control requirements (de Silva 1995). For exam- target force changes accordingly, the control method makes ple, when a person touches a soft or brittle object, they will adjustments to eliminate the error as quickly as possible, choose to use less pressure to avoid breaking the object, and 123 Autonomous Robots (2020) 44:773–790 779 when they encounter a bump or obstacle, they will leave the while for the PID method, they are 0.11 N and 0.013 N. The surface of the object for further exploration correspondingly. results indicate the better ﬂexibility of the fuzzy method when These decisions are not precisely made but intuited by learnt dealing with dynamic and unknown object. Also, by chang- knowledge or former experience. ing the boundary of the memberships, the control method can Conventional control techniques such as PID control, be adapted to different objects. Therefore, in this study, the nonlinear feedback control, etc. can provide good stability, fuzzy control method was selected. robustness to uncertainties and disturbances when the under- A fuzzy control system mainly consists of 4 compo- lying assumptions are satisﬁed and values of the controller nents: Fuzziﬁers: It maps the real-valued input into fuzzy parameters are known. However, these control algorithms sets characterized by membership functions. Rule Base:It are inﬂexible and cannot generally handle vague or chang- is a knowledge base consisting of linguistic rules in If-Then ing situations. Fuzzy control method is a particular type of format. It can be obtained through experiments or experi- intelligent control, which has a knowledge-base within the ences. Fuzzy Inference Engine: Using the If-Then rules in controller, and the control actions are generated through an the knowledge base, it performs reasoning by producing a inference mechanism. It can handle vague or incomplete fuzzy output according to the fuzzy input given by the fuzzi- information and the knowledge itself can evolve through ﬁer. De-fuzziﬁers: It converts the fuzzy output given by the learning (de Silva 1995). Experiments using classic control fuzzy inference engine to produce a real-valued output. To method—PID control have been carried out, and the results create the fuzzy control framework of the control algorithm, show that for a ﬁxed object and a certain area on the object, these four components were designed. the normal force can be controlled at a relatively ﬁne level The inputs of the system are the force error between the during exploration with ﬁxed P, I and D values. However, for measured and target value and its changing rate: (ε , ε ˙ ).The t t another object or a different area on the surface, the P, I and input will be converted into fuzzy sets represented by mem- D values needed to be ﬁne-tuned before they can be used to bership functions, and the continuous fuzzy set was used in get similar performance. As in this study, the ﬁnger was con- this work, which is denoted as: trolled to explore objects with dynamic movement at random areas, the fuzzy control method which has more ﬂexibilities is μ (x ) F = (7) more suitable. Figure 5 shows the normal force result of using the fuzzy method and PID method for the rough plate (no. 25, shown in Sect. 4). The parameters were obtained for both where x is indicating the input data. The boundaries are methods using the light bulb as the learning object, and the deﬁned as the region of the universe of which 0 <μ (x)< rough plate was used for the testing object. As a result, the SD 1. There are common membership functions such as Triangu- and MSE of the resultant normal force (target normal force is lar function, Gaussian function, Bell function, etc. Different 0.6 N) when using the fuzzy method are 0.08 N and 0.0067 N, types of membership functions were tested, and the Gaussian function has been selected, which showed the best perfor- mance. The fuzzy output is also an inferred membership function, and by moving the boundaries of output member- Normal force result - Fuzzy vs PID 1.2 ship functions, the coverage of different memberships can PID Fuzzy be changed, and the ﬁnal control inclination following pre- deﬁned logic rules is modiﬁed adaptively. For example, after 0.8 one exploration, if the average normal force error is too large, 0.6 the boundaries can be shifted rightward to decrease the cover- age of the closing action. In this way, if the controller attempts 0.4 to increase the normal force, the magnitude will be smaller. 0.2 Figures 6, 7 and 8 show the fuzzy control frame of using the Gaussian function for the input and output memberships. The 0 600 1200 1800 2400 3000 3600 4200 4800 5400 Time (ms) number of output memberships is decided by the knowledge base, and they are shown in different line types. Fig. 5 The ﬁgure shows the normal force result during the exploration For creating the knowledge base, a 9-rule IF-THEN (IF of the rough plate (no. 25) using the fuzzy and PID method. The param- premise THEN conclusion) rule base was formed shown in eters were obtained for both methods using the light bulb as the training object, and the rough plate was used as the testing object. It can be seen Table 3. In which “Leave” means controlling the ﬁnger to that the vibration of the normal force when using the PID method is leave away from the surface of the object, “Close” means much larger than that when using the fuzzy method. The SD and MSE moving closer to the object and “Stay” means maintaining the of the normal force are 0.08 N and 0.0067 N when using the fuzzy con- distance from the object in the contact normal direction. And trol method, and 0.11 N and 0.013 N when using the PID method. The results indicate the better ﬂexibility of the fuzzy method “small”, “medium” and “large” mean the level of the actions. Force (N) 780 Autonomous Robots (2020) 44:773–790 Fuzzy input distribution - e when ε is positive and ε ˙ is zero, the conclusion can either 1 t t Negative 0.9 Zero be “leave medium” if ε has larger effect or “leave small” if Positive 0.8 the effect of ε is not signiﬁcant. As mentioned before, fuzzy 0.7 logic represents mathematically the experienced knowledge 0.6 for approximation reasoning; it requires the learnt knowl- 0.5 edge or experience to decide the rules. The bulb object was 0.4 0.3 chosen to get the fuzzy rules and decide the range of output 0.2 membership functions. For the fuzzy inference engine, the 0.1 Mamdani (max-min) inference method was selected, which -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 can present a fuzzy output in membership function according e (N) to the fuzzy input. The centroid method was selected (also known as the centre of gravity (CoG)), which provided us Fig. 6 The ﬁgure is showing the Gaussian distributions of the fuzzy input (ε ). The line types present the different memberships. ε has the t t with satisﬁed output values. The CoG for continuous form range of (− 0.5, 0.5)N, and it has three components: the negative, zero can be calculated using: and positive with mean value of − 0.5, 0, and 0.5 respectively μ (z)zd F z Fuzzy input distribution - de K = z = (8) μ (z)d F z Negative 0.9 Zero Positive 0.8 where μ (z) is the membership function for the output and 0.7 K = z is the output value. 0.6 0.5 With the input being mapped into membership functions, 0.4 the output K used for deciding the next explore location 0.3 was obtained with the de-fuzziﬁer, following the rules in the 0.2 knowledge base. Thus, the fuzzy control method has been 0.1 ﬁnalized. For the software implementation of the fuzzy con- -10 -8 -6 -4 -2 0 2468 10 trol method, the fuzzylite library (https://fuzzylite.com/) has de (N/s) been used. Fig. 7 The ﬁgure is showing the Gaussian distributions of the fuzzy input (ε ˙ ). The line types present the different memberships. ε ˙ has the t t 3.5 Explore velocity control range of (− 10, 10)N, and it has three components: the negative, zero and positive with mean value of − 10, 0, and 10 respectively It has been found out that the S is the parameter that affects the explore velocity and study on how S can be used to Both (ε , ε ˙ ) have values of positive, zero and negative, the control the explore velocity has been carried out. For each t t combination of these situations forms 9 (3x3) rules. The rules exploration, S can either be ﬁxed or controlled with closed- can be adjusted based on the experiment results, for example, loop control. In the test, when S was ﬁxed, the average Fig. 8 The ﬁgure is showing the Fuzzy ouput distribution - K Gaussian distributions of the Leave lerge Leave medium fuzzy output K . The line types Leave small 0.9 Zero present the different Close small Close medium memberships. As shown in the 0.8 Close large ﬁgure, K has the range of 0.7 (− 1,1), and it is following 9 rules involving 7 actions: leave 0.6 large, leave medium, leave small, stay, close small, close 0.5 medium and close large, with 0.4 μ of − 1, − 0.7, − 0.3, 0, 0.25, 0.6 and 1 respectively 0.3 0.2 0.1 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Probabilities Probabilities Probabilities Autonomous Robots (2020) 44:773–790 781 Table 3 Fuzzy knowledge base IF-THEN Linguistic rules ε Positive (0 to 0.5) Zero (−0.2to0.2) Negative(−0.5to0) ε ˙ Positive (0 to 10) Leave large Leave medium Close small Zero (− 10 to 10) Leave medium Stay Close medium Negative (− 10 to 0) Leave small Close medium Close large explore velocity turned out to be similar for different objects, but the standard deviation was relatively high. In this work, a PID controller for the velocity control was created. And for creating the controller, the real-time velocity was calculated with v = l /dt, where l is the length the ﬁnger moved in m m m the base frame in one loop and dt is the time used for one loop. After ﬁltering the v with a moving average ﬁlter (the ﬁlter takes 10 points as the input), the error with the desired v was fed into the PID controller to generate the S value for d v the next position decision. A new S will change the explore velocity in the next movement. Algorithm 1 shows the detail of the control process. Algorithm 1 Velocity control algorithm 1: Initialise variables, l = 0, count = 1, I = 0, S = 0.45mm, m v decide k , k , k , input target velocity v ,ε = v p i d d v d 2: while joint angles θ ,θ <θ do 1 2 limit 3: if count == 1 then 4: Get current time t and contact position P 5: Action: Haptic exploration with initial S Fig. 9 The objects used in the haptic exploration experiments. From top 6: count ← count + 1 to bottom, left to right is: bulb, mouse, 3D printed statue, cup, human 7: Get new contact position P and time t ﬁnger, soft box, sponge (softer), black sponge, metal spring, glass bottle; 8: Compute the distance the ﬁnger travelled: l ← dis(P , P ) m t and plates made with different materials and patterns: mirror stainless, 9: Compute the velocity v ← l /(t − t ) m m satin stainless, BA stainless (canvas), BA stainless (squares), BA stain- 10: Smooth v with moving average ﬁlter m less (13SD), BA stainless (cambridge), BA stainless (11PS), bronze 11: New error ε ← v − v , I ← I + ε ∗ dt d m stainless (pippin), green stainless (poppy), BA stainless (5WL), BA v v 12: Compute the new S with PID controller: stainless (2WL), BA stainless (7GM), BA stainless (6WL), BA stain- 13: S ← k ∗ ε + k ∗ I + k ∗ (ε − ε )/dt p i d v less (9EH), red stainless (pearl) v v v 14: Store current error: ε ← ε ,time t ← t 15: Store current position P ← P 16: Action: Haptic exploration with S active exploration algorithm where velocity is important for 17: return 0 Return 0 if ﬁnished successfully deciding the time cost. When S was changed with the PID controller, the stan- dard deviation of the velocity became smaller, thus, the 4 Haptic surface exploration experiments velocity control for surface exploration has been achieved. The velocity control can help to get velocity related properties With different platforms, various haptic surface exploration such as viscous friction coefﬁcient f which can be obtained experiments were carried out to evaluate the performance by measuring the viscous friction F and the coefﬁcient is of the developed control algorithm. In this section, the one- calculated using f = F /v. With the explore velocity been time exploration with the contact sensing ﬁnger (platform-a) v v controlled, the instantaneous power generated by the friction will be introduced ﬁrst, followed by the adaptive parameter during the contact and sliding is also controllable, which tuning for the multi-time explorations using platform-b, then is helpful in manufacturing procedures like grinding. More- the haptic exploration with the robot arm (platform-c) will over, it can be beneﬁcial to the control of explore efﬁciency be presented, and the velocity control experiments using the during haptic perception, facilitating the development of an platform-a will be shown at last. 123 782 Autonomous Robots (2020) 44:773–790 Table 4 Force control results of haptic exploration Resultant normal force standard deviation (N) and MSE No. F -0.2 N F -0.4 N F -0.5 N F -0.6 N F -0.7 N d d d d d 1 0.047 0.0038 0.045 0.035 0.075 0.0077 0.09 0.0083 0.091 0.0085 2 0.054 0.0036 0.056 0.037 0.059 0.006 0.076 0.0067 0.083 0.008 3 0.052 0.0033 0.07 0.006 0.076 0.0074 0.083 0.0084 0.12 0.013 4 0.048 0.0026 0.05 0.0033 0.065 0.0067 0.075 0.0063 0.09 0.0083 5 0.063 0.0048 0.07 0.0055 0.086 0.008 0.1 0.01 0.13 0.012 6 0.051 0.003 0.062 0.006 0.072 0.0065 0.076 0.0066 0.09 0.01 7 0.04 0.0033 0.052 0.005 0.064 0.0059 0.085 0.0082 0.09 0.0085 8 0.04 0.003 0.045 0.004 0.075 0.0065 0.078 0.0075 0.093 0.0094 9 0.047 0.0032 0.053 0.003 0.077 0.007 0.086 0.0088 0.1 0.01 10 0.045 0.0031 0.053 0.003 0.064 0.0079 0.089 0.0085 0.095 0.009 11 0.039 0.0027 0.05 0.0039 0.068 0.0052 0.071 0.006 0.076 0.0076 12 0.04 0.0027 0.05 0.004 0.066 0.0052 0.069 0.0077 0.079 0.0073 13 0.04 0.003 0.049 0.0032 0.066 0.0062 0.069 0.0066 0.09 0.0082 14 0.053 0.0037 0.057 0.0036 0.066 0.0047 0.071 0.0048 0.08 0.0078 15 0.043 0.003 0.056 0.0039 0.07 0.0059 0.072 0.007 0.093 0.0083 16 0.038 0.0031 0.056 0.0039 0.073 0.006 0.079 0.0067 0.095 0.0085 17 0.042 0.0038 0.052 0.0035 0.079 0.0066 0.068 0.0071 0.09 0.0082 18 0.029 0.0032 0.052 0.0033 0.067 0.006 0.073 0.0063 0.086 0.0086 19 0.038 0.0033 0.048 0.0038 0.067 0.0052 0.076 0.007 0.09 0.0085 20 0.035 0.003 0.055 0.0046 0.079 0.0076 0.081 0.0075 0.09 0.009 21 0.037 0.0027 0.054 0.0035 0.071 0.0057 0.075 0.007 0.09 0.0084 22 0.054 0.0034 0.057 0.0042 0.068 0.008 0.078 0.0088 0.1 0.0093 23 0.056 0.0037 0.059 0.003 0.07 0.006 0.073 0.0085 0.093 0.01 24 0.049 0.0038 0.058 0.0046 0.082 0.008 0.089 0.0096 0.1 0.0096 25 0.055 0.0036 0.062 0.0045 0.085 0.008 0.095 0.0085 0.12 0.0099 Avg. 0.045 0.0036 0.055 0.0039 0.072 0.0064 0.076 0.0072 0.093 0.0089 4.1 Single time haptic surface exploration target force. The μ and σ of the velocity vector were also v v calculated. Algorithm 2 summarizes the experiment protocol. For carrying out the experiments with the contact sensing ﬁn- To provide the object with dynamic movement, two ways ger (platform-a), 10 objects with different surface properties have been used, the ﬁrst was to use hand to hold the object (friction, roughness), geometry and stiffness, and 15 plates which would induce slight movement during the haptic made of different materials with different surface patterns exploration, and the other was to move the object with ran- were selected, shown in Fig. 9. It should be noted that, due dom displacement through a created device. For the second to the limited power of the motor, the maximum force the approach, the object was mounted on the linear guide shown ﬁnger can generate is less than 1.1 N, thus, the controllable in Fig. 10, which can generate a random displacement of force was constrained to 0.2–0.7 N in the tests. The target 2–4 mm during the exploration. Please refer to the video forces used in the experiments were shown in Table 4.For of “dynamic_object_exploration (MOESM1).mp4” for more each object, the haptic exploration started after the contact details. was conﬁrmed and ended until the ﬁnger ﬁnished contacting The normal force control results of one-time exploration the object. The S was chosen from 0.35, 0.45, and 0.55 mm on the light bulb, black sponge, rough plate (no. 25), smooth randomly for each exploration. During each exploration, the plate (no. 12) and 3D printed statue hold by hand are shown normal force and velocity at each contact point were stored in Fig. 11. It can be seen that, ignoring the rising period in vectors. And after the exploration, the mean value μ and of the normal force, the performance of the control is sat- SD σ of the force vector were calculated, and the MSE was isﬁed for the bulb, black sponge and smooth plate, which obtained by comparing the resultant normal force with the in total achieved an average standard deviation of 0.044 N. 123 Autonomous Robots (2020) 44:773–790 783 Algorithm 2 Object haptic surface exploration 1: Initialise vector V , V , V to store the resultant normal force, veloc- f v d ity and target force 2: Initialise variables, pointer = 1, sum = 0, input target force F and put it into the vector V ← F d d 3: Initialise the parameters for fuzzy control method (FCM) 4: if contact normal force F >ϑ then N f 5: while joint angles θ ,θ <θ do 1 2 limit 6: Get current contact normal force and store it in vector V (pointer ) ← F f n 7: Compute next moving position P with FCM next 8: Get the new angles to move next: θ ,θ ← IK (P ) 1 2 next 9: Action: The Motors rotate to the new positions 10: Record thetimeused dt and compute the distance L travelled 11: Compute the explore velocity v ← L/dt 12: Store the velocity in vector V (pointer ) ← v 13: Update pointer ← pointer + 1 14: Smooth the data with ﬁlter, (V , V ) ← filter (V , V ),toget f v f v Fig. 10 The device used to generate a random displacement for the rid of the erroneous data objects. It consists of a linear guide and a 3D printed gripper. The linear 15: Compute μ ← Mean(V ), σ ← SD(V ) f f f f guide is controlled with a gear that can be rotated manually. The linear 16: Compute μ ← Mean(V ), σ ← SD(V ) v v v v guide moves 1 mm after rotating the gear by one circle. The 3D printed 17: for each element i in V and V do f d gripper can be adjusted on the metal rails for different objects 18: sum ← sum + (V (i ) − V (i )) f d 19: MSE ← sum/pointer Normal Force during Haptic Exploration 20: return 0 Return 0 if ﬁnished successfully 1.2 Bulb Black Sponge RoughPlate Smooth Plate Statue However, for the statue and rough plate, the vibrations of 0.8 the control are larger (SD = 0.075 N) due to the complex 0.6 geometry and large concave areas on the surface. In this 0.4 work, for getting the ﬁnal results of the haptic exploration experiments, only the data when the force became stable 0.2 were used, which means that the force data in the rising 0 1500 3000 4500 6000 7500 period were omitted. In practical, the object was held by Time (ms) hand and not ﬁxed during the haptic exploration, which added dynamic movement to the object. There are two Fig. 11 Normal force results of the one-time haptic exploration on the notable facts during the experiments. First, the initial contact surface of the light bulb, black sponge, rough plate, smooth plate and 3D-printed statue. The force control target was 0.6 N and the S value position was random and different for each test, and sec- was kept at 0.45 mm. As can be seen, the rough plate and 3D-printed ond, as the hand was shaking when holding the object, the statue have the largest vibrations. This is because the 3D-printed statue object was moving slightly (1–3 mm) during the exploration. has the most complex geometries and the rough plate has large concave These all increased the difﬁculties for the control algorithm patterns, both of them make it difﬁcult to adjust the force during the exploration. And for the light bulb, black sponge and smooth plate, and in return can show the robustness of the algorithm. regardless of their complex geometry, texture, and stiffness, the force Please refer to videos of “Force_control_01 (MOESM4), control results are satisﬁed, showing the robustness of the algorithm. Force_control_02 (MOESM5), surface_exploration_statue However, as can be seen, the rising time of the algorithm is sluggish, (MOESM6).mp4” for more details. which could be improved with a faster control loop 4.2 Adaptive parameter tuning The adaptive parameter tuning method was developed for After the one-time exploration experiments were carried out, multi-time explorations. After each exploration, if the aver- the ﬁnger (platform-a) was mounted onto the end-effector of age force value μ was too much smaller than the target the robot arm, forming the platform-b. The robot arm end- and the MSE was large, the control algorithm will shift the effector can rotate to any degree after each exploration to help boundaries of the output fuzzy memberships to the right to the ﬁnger explore more areas on the surface of the object. In make the “close” action to have more coverage of the whole the experiments, the arm was controlled to rotate every 30 output distribution. This can be done by increasing the mean degrees after one exploration, and the exploration was carried values (μ ) of the Gaussian distribution of the “close” close out again, in total, the arm rotated 60 degrees and returned to membership functions shown in Fig. 8. Through this way, the starting position. The resultant haptic information of all the ﬁnger will push more during the next exploration for the the exploration procedures were recorded. error correction. Meanwhile, it is required to guarantee that Force (N) 784 Autonomous Robots (2020) 44:773–790 the changing of the control parameters will not induce too obtained. The detailed results can be found in Table 4, and much noise to the haptic exploration by monitoring the stan- from the results, the following points were concluded. dard deviation σ . Similarly, if μ was too much larger than f f F and MSE was large, the boundaries will be shifted left- 1. When the target force is increasing, the MSE and σ d f wards to make the “leave” action to play a more important become larger. It could be due to the reason that larger role during the exploration. Also, it has been learnt that the K errors require more correcting force to adjust, and there value is a notable parameter for the adjustment of the normal are chances that the ﬁnger is unable to correct such errors force during the exploration, because a larger K will increase quickly enough with constrained power. the movement of the ﬁnger, which permits the force control 2. Objects with more complex geometries tend to have to become more dramatic. Thus, a variable for the K has larger MSE and σ , such as the statue and the bulb. been introduced: K = α ∗ K , to change its scale. For differ- Objects with changing stiffness also have relatively ent devices, the α needs to be calibrated to achieve the best higher MSE and σ , like the human ﬁnger. The rea- k f performance. For instance, for objects with larger curvature son could be that the complex geometries and changing changes, a larger α can help to adjust the normal force more stiffness of the object surface will induce dramatic force quickly. In this study, for the validation experiments, α was changes during the haptic exploration, and the force set to 1 for all the objects, the adaptive selection of α will errors can not be eliminated completely and quickly with be developed in future work. Algorithm 3 summarizes the the control algorithm, as a consequence, the error accu- adaptive control algorithm. mulates and the MSE and SD become larger than the situation when exploring the smoother objects 3. When the target force is getting larger, the hand holding Algorithm 3 Adaptive control algorithm the objects is more unstable. The stronger handshaking 1: Deﬁne Δf as the magnitude of changing of the membership func- induces more instabilities to the system, especially when tions, Δf = 0.1 the force is 0.6 N or larger, making the noise (σ ) larger. 2: After one exploration, get μ ← mean(V ), σ ← SD(V ) and f f f f Also, for objects with complex geometries, this effect is MSE 3: while abs(F − μ )>ϑ and MSE >ϑ and σ <ϑ do d f 1 e f σ greater, while for ﬂat plates, this effect is smaller. 4: Increase the μ of “Close” membership functions, which close 4. In Fig. 9, the order of the plates was made by looking means the output distributions of “Close” move rightwards: at the surface texture and patterns rather than the real μ ← μ + Δf close close 5: Action: Haptic exploration again with new FCM roughness and surface pattern complexity level. For plate 6: Get new μ ,σ and MSE f f 1, even though it has the smoothest surface by vision, due 7: while abs(F − μ )<ϑ and MSE >ϑ and σ <ϑ do d f 2 e f σ to the material property of the ﬁngertip, during the explo- 8: Decrease the μ of “Close” membership functions, which close ration, the contact of the ﬁngertip and surface of the plate means the output distributions of “Close” move leftwards: μ ← close can generate viscous friction that makes the control of μ − Δf close the force harder, especially when the force is greater than 9: Action: Haptic exploration again with new FCM 10: Get new μ ,σ and MSE f f 0.4 N. The 15th plate (no. 25) has large and deep concave 11: return 0 Return 0 if ﬁnished successfully areas, and plate13 (no. 23) has large and high convex areas, these patterns will induce sudden force changes during the exploration, making the MSE and σ larger. For each experimental setup, 9 times of exploration were 5. For soft objects that can deform with force applied, when carried out on random areas (3 times with the arm at 0, 30, the force is large, the deformation of the object makes the and 60 degrees) of the objects with random S values cho- contact location estimation algorithm become inaccurate, sen from 0.35 mm, 0.45 mm, and 0.55 mm. 5 target forces thus, forces larger than 0.7 N were not used in the exper- have been set, and the total number of surface explorations iments, especially for the yellow soft sponge and soft was 1125 in the force control experiments. After each explo- black box. The metal spring has gaps between threads ration, the mean value μ and σ of the force vector were f f that can generate frequent drops of force values during calculated, and the MSE was obtained. Also, the algorithm the exploration, making the control harder. adjusted its fuzzy control method parameters to achieve an 6. When the target force is small (like 0.2 N and 0.4 N), the overall optimised performance. After 9 times of exploration, MSE and σ of different objects are similar for the same the average of the σ and MSE were obtained. target force, however, when the force is larger, the differ- ences of the MSE and σ of different objects get larger. 4.3 Contact finger exploration results and discussion Results of the black rubber, plate1 (no. 11), statue, plate15 (no. 25), human ﬁnger and yellow soft sponge represent- After all the haptic surface exploration experiments have ing different types of objects were used to plot ﬁgures been carried out, the overall results of the force control were (shown in Figs. 12, 13). 123 Autonomous Robots (2020) 44:773–790 785 Target Force - SD 0.13 Sponge 0.12 Statue 0.11 Plate1 0.1 Plate15 Finger 0.09 Soft Sponge 0.08 0.07 0.06 0.05 0.04 0.03 0.2 0.3 0.4 0.5 0.6 0.7 Target Force (N) Fig. 12 The SD results for different objects with different target forces. Fig. 14 The points in the ﬁgure are generated with the SD value, MSE The black sponge and plate1 are objects with ﬂat and ﬁne surface that value and the object number as the x, y and z coordinate. It can be are easier for the force control, and their SDs are smaller than other seen that the resultant control performance of the haptic exploration on objects. The 3D printed statue is the object with the most complex different objects are similar within a small range shown by a transparent geometry, and plate15 has large concave areas, these features all hinder cylinder, which reveals the robustness of the proposed algorithm the control performance, and the SDs for these two objects are larger. The human ﬁnger has different stiffness and changing geometry in dif- ferent locations, thereby the SD of which is the largest. The SD of the simplifying the software implementation. Despite these lim- yellow soft sponge is small at the beginning but gets larger when the itations, it can be seen that the average MSE is 0.006 and σ force is increasing due to the fact that when the force increases, the is 0.07 N. Also, from Fig. 14, it can be seen that the MSE and ﬁnger penetrates the soft layer of the sponge and touches the plastic surface underneath (Color ﬁgure online) σ for all objects are constrained within a controllable range, which can show the robustness of the control algorithm for -3 Target Force - MSE different objects with various properties and dynamic move- Sponge ment. Moreover, videos of the real-time experiments show Statue Plate1 that the platform-a works well with subtle motions following Plate15 the surface of the dynamically moving object with complex Finger Soft Sponge outline, when the target force is small. 4.4 Haptic exploration with robot arm UR-3 Since the workspace of the ﬁnger is limited with an average 0.2 0.3 0.4 0.5 0.6 0.7 travelling length of 60 mm, the robot arm UR-3 was used to Target Force (N) verify the control method with larger workspace and force ranges. A contacting sensing tip was mounted onto the robot Fig. 13 The MSE results for different objects with different target arm end-effector, forming the platform-c. The study of con- forces. The black sponge and plate1 have smaller MSE. The 3D printed statue and plate15 with more complex geometries and patterns have trolling the robot arm and ﬁnger simultaneously to realize larger MSE. The human ﬁnger has the largest MSE because of its chang- a synergic control is also envisioned, which can be useful ing stiffness and geometry. MSE of the yellow soft sponge is changing for more complex haptic exploration. The robot arm can due to the same reason explained in Fig. 12 (Color ﬁgure online) reach the object at almost any point on the surface, and the exploration is not constrained in any speciﬁc plane, but for From the results, the limitations of the platform-a and con- one-time exploration, the controlled movement is within two trol system have been learnt. First, the force it can generate axes. The robot arm can also realize a sequence of move- is limited, making it difﬁcult to correct the error quickly ments, covering more surfaces of the object. Figure 15 (left) enough, especially when the target force is more massive. shows the designed experimental setup. The Nano 17 sensor This can be improved by using more powerful motors, but was mounted on the robot arm with a 3D printed connection at the same time, the size of the platform will vary since part and covered with a hemisphere cap made of silicone powerful motors usually require more space. Second, the rubber (Ecoﬂex-50, Shore A 50). The stiffness of the rubber high-level control is working at 100 Hz currently, which may cap is large enough that there is no hysteresis effect during not be fast enough to adapt to quick force changes induced the experiments. A device that can generate a random dis- by complex geometry or stiffness changes. The relative low placement within 5–12 mm in the vertical direction was used working frequency is due to the time consumption of the soft- to add movement to the holding object. The corresponding ware modules like the angle calculation and the fuzzy control kinematic and fuzzy control parameters of the control algo- method. This can be improved by using higher spec PCs or rithm were modiﬁed for controlling the robot arm platform MSE SD (N) 786 Autonomous Robots (2020) 44:773–790 Fig. 15 Left: The haptic exploration platform-c using the UR3 robot arm and a contact sensing tip. The object is mounted on a moveable device that can move 5–12 mm vertically. Right: The schematic draw- ing of how the necessary angles have been calculated for deciding the next moving location of the tip. Same with Fig. 4, this only shows the situation where the correcting force F has the same direction of the real normal force Fig. 16 The objects used in haptic exploration experiments with the robot arm platform-c. From left to right: (1) metal bowl, (2) sphere like (platform-c). Similarly, the required angles can be calculated hat made by silicon, (3) plastic bottle ﬁlled with water (4) ﬂat sponge, with the equations below: (5) sponge with curved shape, the explore trajectory is drew with green colour (Color ﬁgure online) θ = 2 ∗ π − θ − θ + 1.5π − θ + 2π − θ (9) E j2 j3 j4 j5 θ = atan2(y , z ) (10) achieve accurately is no smaller than 0.8 mm, and the force it C e e generates is much larger than the ﬁnger per movement. These θ = θ + θ (11) N E C all present that the robot arm is suitable for larger force con- β = atan2(F , F ) (12) T N trol scenarios. α = β + θ (13) 5 objects shown in Fig. 16 were used to evaluate the perfor- mance of the control algorithm. There are four soft objects, And the next position to reach is: all can deform when force applied, and 3 of them have com- plex outlines. There is one metal object with a curved shape. P = P + S ·[cosα, sinα] (14) next now v For different objects, different target forces were set, this is because with too much force applied, the deformation of the where θ ,θ ,θ ,θ are the current joint angles of the soft object will cause inaccurate sensor readings, while for j2 j3 j4 j5 robot, which can be obtained through the driver in ROS. the solid object, the force changed by the smallest displace- With the proposed control algorithm, the robot arm was ment is more than 1 N. Thus, the target forces were set to 1 N, controlled to explore the surface of the object while keeping 1.25 N and 1.5 N for the soft objects, and 1.5 N, 2 N, 2.5 N the normal force as desired. Since the robot arm has more for the metal object. The S size was chosen from 0.8 mm, DoFs and can work in larger target force range (more power- 1 mm and 1.5 mm. For each object, 10 times of exploration ful), it can explore the object in a much wider workspace with with the adaptive control method were carried out. The video feedbacks from the contact sensing tip, which helps to realize of “surface_exploration_ur3-1 (MOESM7).mp4” shows the larger area’s haptic exploration and make more exploratory haptic surface exploration on the sponge with curved shape. procedures (EPs) available (Lederman and Klatzky 1993). The objects were mounted on the moveable platform with More EPs will allow the haptic perception system to obtain 5–12 mm random vertical displacement. more information about the object, which will be beneﬁcial Table 5 shows the results of the haptic exploration exper- in future research on the active haptic control and perception. iments using the platform-c. From the results, it can be seen Here, the haptic surface exploration is the main focus, and that the average MSE and σ are larger than those obtained more EPs will be implemented in the next study. with the contact sensing ﬁnger (platform-a) due to larger However, the control of the robot arm has limitations due target forces. As mentioned before, when the target force to the slow frequency of the software. The control loop is gets larger, the MSE and σ also increase. It has been learnt limited to 10 Hz in this study, which may hinder the response that, in some situations, even though the algorithm output speed when adjusting the force or velocity. Moreover, com- the largest correcting movement, it was still not enough to pared to the contact sensing ﬁnger, the smallest S it can correct the force in time. Thus, it becomes necessary for the 123 Autonomous Robots (2020) 44:773–790 787 Table 5 Force control results Resultant normal force standard deviation (N) and MSE with robot arm Object no. Target normal Target normal Target normal force-1 N/1.5 N force-1.25 N/2 N force-1.5 N/2.5 N 1 0.21 0.03 0.31 0.035 0.38 0.042 2 0.15 0.013 0.17 0.019 0.19 0.024 3 0.14 0.016 0.18 0.02 0.21 0.027 4 0.13 0.02 0.15 0.02 0.16 0.03 5 0.12 0.013 0.1 0.015 0.18 0.025 Avg. 0.15 0.0184 0.182 0.0218 0.224 0.0296 Table 6 Velocity control results Open loop resultant explore velocity mean and SD PID control resultant explore velocity mean and SD S value (mm) F -0.2 N F -0.4 N F -0.6 N v (mm/s) F -0.2 N F -0.4 N F -0.6 N Δv v d d d d d d d 0.15 1.92 2.11 1.86 2.31 1.78 2.18 2 1.78 1.41 1.75 1.54 1.81 1.45 0.22 0.25 3.16 2.59 3.16 2.68 2.95 2.5 3 2.69 1.73 2.65 1.78 2.72 1.67 0.31 0.35 3.72 2.68 3.8 2.8 3.68 2.64 4 3.62 1.78 3.58 1.86 3.65 1.76 0.38 0.45 4.69 2.86 4.55 2.85 4.67 2.84 5 4.56 1.91 4.51 1.9 4.59 1.89 0.45 0.55 5.52 3.02 5.55 3.12 5.3 2.9 6 5.51 2.01 5.46 2.08 5.54 1.93 0.50 0.65 6.99 3.15 6.89 3.2 6.43 2.96 7 6.48 2.1 6.42 2.13 6.4 2.1 0.57 0.75 7.53 3.24 7.22 3.34 7.19 2.9 8 7.43 2.22 7.37 2.43 7.46 2.23 0.58 0.85 8.38 3.32 8.02 3.54 8.23 3.09 9 8.35 2.41 8.41 2.56 8.41 2.56 0.61 0.95 9.03 3.5 9.04 3.66 8.95 3.41 10 9.26 2.8 9.27 2.84 9.34 2.75 0.71 algorithm to know the relationship of the indentation and the PID control method was applied to adjust the S value force changes for different objects, through which way, the during the exploration to check the change of σ . The target more accurate S can be decided for adjusting the force error. velocity for the PID control was set from 2 to 10 mm/s. The As the next step, the robot arm can be controlled to push the mean velocity was calculated by v = L /T , where L is m e e e object ﬁrst to characterize the indentation and force changing the total explored distance and T is the overall time used relation of the object, and then the best S can be derived. for the exploration process. For each object, 3 times explo- However, the overall result of this experimental validation is ration were carried out for each ﬁxed S value and 3 times stratiﬁed, conﬁrming the robustness of the control algorithm. exploration for each target velocity using the PID control. Moreover, as can be seen from videos of the real experiment Table 6 shows the results, and the values are averages of (refer to videos of ”dynamic_object_exploration_ur3 the three objects. It can be seen from the results, when S is (MOESM2), dynamic_object_exploration_ur3-with_back ﬁxed, the mean explore velocities are similar regardless of ground (MOESM3).mp4”), the arm can adjust the position the target force. In addition, for different objects, the veloc- quickly following the movement of the object. ity results are also similar. For example, when S is 0.45 mm and F is 0.4 N, the mean velocities for the three objects are 4.63 mm/s, 4.59 mm/s and 4.43 mm/s respectively. There- 4.5 Velocity control results fore, with different S values, the overall exploration velocity can be controlled during each haptic exploration on various To verify the performance of the velocity control method, objects, as shown in Fig. 17. This is because when S is used haptic exploration experiments with changing S values for deciding the next moving location, the increment along using the contact sensing ﬁnger (platform-a) were carried the tangential direction of the contact surface on the object out. The target force was set to 0.2 N, 0.4 N and 0.6 N in decides the moving velocity when the used time for every experiments, and the resultant mean value μ and σ were v v loop is the same (almost). The videos of “velocity_control-01 obtained with different S values. Three objects: the black (MOESM8), velocity_control-02 (MOESM9).mp4” show sponge, the light bulb, and plate-15 (no. 25) were selected. the examples of the open-loop velocity control. However, First, a ﬁxed S value was used during each haptic explo- the σ is relatively high with an average value of 2.94 mm/s ration to check velocity changes and compute the σ , then 123 788 Autonomous Robots (2020) 44:773–790 Sv - Mean velocity in two ways: (1) Increase the frequency of the overall con- Target force-0.2N trol loop, which can help to update the error more rapidly Target force-0.4N 8 Target force-0.6N and the platform can be controlled to take correction actions 7 more quickly to eliminate the error. In this way, the overall control quality could be improved, especially when there are sudden surface curvature changes on the object. (2) Modify the mechanical design of the ﬁnger joints to allow smaller S value for deciding the next location. As shown in Table 6, a smaller S enables the system to respond to larger and quicker changes more smoothly. This works when the ﬁn- 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sv (mm) ger is controlled with very slow velocities and small target force. However, in this study, the position control of the joints Fig. 17 The ﬁgure is showing the changing of the mean velocity of one becomes inaccurate when S is less than 0.1 mm due to the exploration with relation to the S changes using the open-loop control. v The velocity is obtained by getting the average of all sample data. It can mechanical limitation of the motors and joints. be seen that as S increases, the velocity increases and the relation of these two values is close to a linear correlation regardless of the control target force. These curves show that S can be used to control the overall 5 Conclusion velocity of the exploration procedure As the conclusion, with the proposed control method, the contact normal force and explore velocity during haptic sur- face exploration on objects with dynamic movement can be controlled adaptively, regardless of their surface and geom- etry properties. There are three points worthy of further research to improve the performance of the control method. First, during the exploration, if the force change generated by the movement of the device in the normal force direc- tion is not enough to correct the error, the adjustment will be delayed; and if the force change is too large, it will generate overshoots, inducing more ﬂuctuations to the system. Thus, understanding the indentation and force changes relation is Fig. 18 The points in this ﬁgure are generated with the σ , Δv and essential to make the control more precise. For unknown v as the x, y and z coordinate. Δv is obtained by abs(v − v ).The d d m objects, this relation can be obtained ﬁrst as a pre-information curves on the left side are representing the relation between SD and v and fed back to the system to adjust the related parameters. at different F , and curves on the right side are representing the relation between Δv and v at different F . It can be seen from the ﬁgure, the d d Also, this can help us understand the stiffness of the object, points are constrained in a cylinder with a larger radius compared to presenting more information about the contacting object. the force control results, showing the relative larger SD of the velocity Second, it is important to understand the best control force control result. However, it also shows that the control method can be range of each device, since the force ranges generated by dif- used to control the explore velocity with low absolute errors ferent devices are varied. By selecting or designing different devices or platforms, different forces can be controlled pre- when S is ﬁxed. When using the PID controller, the σ of the cisely. Third, because the S ranges, what each device can v v v velocity is around 2/3 of that when S is ﬁxed. And an average achieve are different, the control resolution (force change per absolute error Δv of 0.48 mm/s has been achieved, showing movement) is also varied for each device. To understand the the good performance of the feedback velocity control. resolution of the force control for each speciﬁc device is also With the feedback control, the performance of explore important to achieve more robust and precise control. velocity control is not excellent as the σ is still not low As the extensions of this work, we will focus on the active enough with an average value of 2.07 mm/s. As shown in haptic exploration, perception and soft contact. This pro- Fig. 18,the σ and Δv for each v are constrained in a trans- posed control method can be applied to any situations where v d parent cylinder that is larger than the one in Fig. 14. It could contact happens, and the contact force is critical to the con- be caused by two main reasons. The ﬁrst is the unstructured tacting object. Also, if the normal force and velocity can surface of the object, which adds many difﬁculties to the be controlled, it is helpful to implement different manipu- control process, and the other is the limited frequency and lation tasks that require different contact forces and explore relative long response time of the control software. Consid- velocities. This will lead to an active exploration method that ering the limitations, the velocity control can be improved can change exploration strategies (different force or velocity) Velocity (mm/s) Autonomous Robots (2020) 44:773–790 789 actively depending on the utility function for more efﬁcient ﬁngered robotic hand. In 2017 IEEE international symposium on robotics and intelligent sensors (IRIS) (pp. 311–317). object recognition or manipulation tasks. In addition, we will da Fonseca, V. P., Kucherhan, D. J., de Oliveira, T. E. A., Zhi, D., & test ﬁngertips made of softer material that can deform during Petriu, E. M. (2017a). Fuzzy controlled object manipulation using contact, which will be helpful when contacting soft objects a three-ﬁngered robotic hand. In 2017 Annual IEEE international like tissues inside the organs or fragile daily objects. More- systems conference (SysCon) (pp. 1–6). de Silva, C. W. (1995). Intelligent control: fuzzy logic applications. over, more ﬁngers will be created to form a manipulator, https://academic.microsoft.com/paper/598128971 combing the robot arm, the whole platform can be used to Fishel, J. A., & Loeb, G. E. (2012). Bayesian exploration for intelligent carry out dexterous manipulations with controllable force identiﬁcation of textures. Frontiers in Neurorobotics, 6,4. and velocity. As reported in Okamura and Cutkosky (2001), Hellman, R. B., Tekin, C., van der Schaar, M., & Santos, V. J. (2018). Functional contour-following via haptic perception and reinforce- with controllable haptic explorations, dexterous manipula- ment learning. IEEE Transactions on Haptics, 11, 61–72. tion can be realised with grasping, holding and other hand Jamil, M. H., Annor, P. S., Sharfman, J., Parthesius, R., Garachon, I., & controls. Finally, by combing the haptic feedback with the Eid, M. (2018). The role of haptics in digital archaeology and her- surgery devices, it will be beneﬁcial in robotic surgery to itage recording processes—IEEE conference publication. In IEEE international symposium on haptic, audio and visual environments help the surgeon to monitor and control the amount of force and games (HAVE). applied to minimise the risk of tissue damage (Amirabdol- Kappassov, Z., Corrales, J. A., & Perdereau, V. (2015). Tactile sens- lahian et al. 2017). ing in dexterous robot hands—Review. Robotics and Autonomous Systems, 74, 195–220. Acknowledgements This work is supported by the EPSRC (No. Konstantinova, J., Li, M., Mehra, G., Dasgupta, P., Althoefer, K., & EP/N020421/1) and NSFC (No. 51520105006). And we would like Nanayakkara, T. (2014). Behavioral characteristics of manual pal- to thank Dr. Hak Keung Lam for his kind suggestions on fuzzy control pation to localize hard nodules in soft tissues. IEEE Transactions method. on Biomedical Engineering, 61, 1651–1659. Lederman, S. J., & Klatzky, R. L. (1993). Extracting object properties Open Access This article is licensed under a Creative Commons through haptic exploration. Acta Psychologica, 84, 29–40. Attribution 4.0 International License, which permits use, sharing, adap- Lederman, S. J., & Klatzky, R. L. (2009). Haptic perception: A tutorial. tation, distribution and reproduction in any medium or format, as Attention, Perception and Psychophysics, 71, 1439–1459. long as you give appropriate credit to the original author(s) and the Lepora, N. F., Aquilina, K., & Cramphorn, L. (2017). Exploratory tactile source, provide a link to the Creative Commons licence, and indi- servoing with active touch. IEEE Robotics and Automation Letters, cate if changes were made. The images or other third party material 2, 1156–1163. in this article are included in the article’s Creative Commons licence, Lepora, N. F., Church, A., de Kerckhove, C., Hadsell, R., & Lloyd, J. unless indicated otherwise in a credit line to the material. If material (2019). From pixels to percepts: Highly robust edge perception and is not included in the article’s Creative Commons licence and your contour following using deep learning and an optical biomimetic intended use is not permitted by statutory regulation or exceeds the tactile sensor. IEEE Robotics and Automation Letters, 4, 2101– permitted use, you will need to obtain permission directly from the copy- 2107. right holder. To view a copy of this licence, visit http://creativecomm Li, Q., Schürmann, C., Haschke, R., & Ritter, H. J. (2013). A control ons.org/licenses/by/4.0/. framework for tactile servoing. In Robotics: Science and systems 2013,Vol.9, https://academic.microsoft.com/paper/102778925. Liu, H., Song, X., Bimbo, J., Seneviratne, L., & Althoefer, K. (2012). Surface material recognition through haptic exploration using an intelligent contact sensing ﬁnger. In 2012 IEEE/RSJ international References conference on intelligent robots and systems. Liu, H., Nguyen, K. C., Perdereau, V., Bimbo, J., Back, J., Godden, Amirabdollahian, F., Livatino, S., Vahedi, B., Gudipati, R., Sheen, P., M., et al. (2015). Finger contact sensing and the application in Gawrie-Mohan, S., et al. (2017). Prevalence of haptic feedback in dexterous hand manipulation. Autonomous Robots, 39, 25–41. robot-mediated surgery: A systematic review of literature. Journal Okamura, A. M. (2000). Haptic exploration of objects with rolling and of Robotic Surgery, 12, 11–25. sliding. Ph.D. thesis, Stanford University, Department of Mechan- Aviles, A. I., Alsaleh, S. M., Montseny, E., Sobrevilla, P., & Casals, A. ical Engineering. (2016). A deep-neuro-fuzzy approach for estimating the interac- Okamura, A. M., & Cutkosky, M. R. (2001). Feature detection for hap- tion forces in robotic surgery. In IEEE international conference on tic exploration with robotic ﬁngers. The International Journal of fuzzy systems (FUZZ-IEEE) (pp. 1113–1119). Robotics Research, 20, 925–938. Back, J., Bimbo, J., Noh, Y., Seneviratne, L., Althoefer, K., & Liu, Okamura, A. M., & Cutkosky, M. R. (2001). Feature detection for hap- H. (2015). Control a contact sensing ﬁnger for surface haptic tic exploration with robotic ﬁngers. The International Journal of exploration. In IEEE international conference on robotics and Robotics Research, 20, 925–938. automation (ICRA) (pp. 2736–2741). Rosales, C., Spinelli, F., Gabiccini, M., Zito, C., & Wyatt, J. L. Bicchi, A., Salisbury, J. K., & Brock, D. L. (1993). Contact sensing (2018). Gpatlasrrt: A local tactile exploration planner for recover- from force measurements. The International Journal of Robotics ing the shape of novel objects. International Journal of Humanoid Robotics, 15, 1850014. Research, 12, 249–262. Sommer, N., & Billard, A. (2016). Multi-contact haptic exploration and Ciotti, S., Sun, T., Battaglia, E., Bicchi, A., Liu, H., & Bianchi, M. grasping with tactile sensors. Robotics and Autonomous Systems, (2019). Soft tactile sensing: Retrieving force, torque and contact 85, 48–61. point information from deformable surfaces. In: 2019 IEEE inter- Song, X., Liu, H., Althoefer, K., Nanayakkara, T., & Seneviratne, L. D. national conference on robotics and automation, ICRA. (2014). Efﬁcient break-away friction ratio and slip prediction based da Fonseca, V. P., de Oliveira, T. E. A., Eyre, K., & Petriu, E. M. (2017b). Stable grasping and object reorientation with a three- 123 790 Autonomous Robots (2020) 44:773–790 Hongbin Liu is an Associate on haptic surface exploration. IEEE Transactions on Robotics, 30, Professor (Senior Lecturer) in the 203–219. school of biomedical engineering Sornkarn, N., & Nanayakkara, T. (2017). Can a soft robotic probe use and imaging sciences, King’s Col- stiffness control like a human ﬁnger to improve efﬁcacy of haptic lege London (KCL) where he is perception? IEEE Transactions on Haptics, 10, 183–195. directing the Haptic Mechatron- Su, Z., Fishel, J. A., Yamamoto, T., & Loeb, G. E. (2012). Use of tactile feedback to control exploratory movements to characterize object ics and Medical Robotics Labora- compliance. Frontiers in Neurorobotics, 6,7. tory (HaMMeR). His lab designs Takagi, T., & Sugeno, M. (1993). Fuzzy identiﬁcation of systems and robots with advanced haptic per- its applications to modeling and control. In Readings in fuzzy sets ception and interaction capabili- ties to address unmet needs in for intelligent systems (pp. 387–403). medicine, enabling safer and more Tanaka, K., & Wang, H. O. (2004). Fuzzy control systems design and effective diagnosis and treatment. analysis: A linear matrix inequality approach. New York: Wiley. Dr. Liu is a member of the IEEE, Tegin, J., & Wikander, J. (2005). Tactile sensing in intelligent robotic and a Technical Committee Mem- manipulation—A review. Industrial Robot: An International Jour- ber of IEEE EMBS BioRobotics. He has published over 100 peer- nal, 32, 64–70. reviewed publications at top international robotic journals and confer- Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Gian- ences and is inventor for 4 patents. His research has been funded by naccini, M. E., Rossiter, J., et al. (2018). The tactip family: Soft EPSRC, Innovate UK, NHS Trust and EU Commissions, with strong optical tactile sensors with 3d-printed biomimetic morphologies. industrial collaborations such as IBM, Ericsson. More information can Soft Robotics, 5, 216–227. be found on our online website: http://kclhammerlab.com/. Xu, D., Loeb, G. E., & Fishel, J. A. (2013). Tactile identiﬁcation of objects using Bayesian exploration. In IEEE international confer- ence on robotics and automation (ICRA). Yang, C., & Lepora, N. F. (2017) Object exploration using vision and active touch. In IEEE/RSJ international conference on intelligent robots and systems (IROS). Yang, C., Li, Z., & Burdet, E. (2013). Human like learning algorithm for simultaneous force control and haptic identiﬁcation. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp 710–715). Yin, J., Aspinall, P., Santos, V. J., & Posner, J. D. (2018). Measuring dynamic shear force and vibration with a bioinspired tactile sensor skin. IEEE Sensors Journal, 18, 3544–3553. Publisher’s Note Springer Nature remains neutral with regard to juris- dictional claims in published maps and institutional afﬁliations. Teng Sun received the B.Sc. degree in mechanical engineering and automation from Beijing Institute of Technology, Beijing, China, in 2011, and worked as an autonomous driving engineer in Nissan Advanced Technology Centre, Atsugi, Japan from 2011 to 2014. He is currently work- ing toward the Ph.D. degree in robotics in the Haptic Mechatron- ics and Medical Robotics (HaM- MeR) Laboratory, Centre for Robotics Research, Department of Informatics, King’s College Lon- don, London, UK. His current research interests include contact loca- tion estimation based on intrinsic contact sensing, object recognition based on haptic information through Bayesian classiﬁer, active percep- tion and decision making with information gain and utility method. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Autonomous Robots Springer Journals http://www.deepdyve.com/lp/springer-journals/adaptive-force-and-velocity-control-based-on-intrinsic-contact-sensing-U7xJWuJqVp

Loading next page...

References (35)

Hongbin Liu, Xiaojing Song, João Bimbo, L. Seneviratne, K. Althoefer (2012)
Surface material recognition through haptic exploration using an intelligent contact sensing finger
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems
Zhe Su, Jeremy Fishel, Tomonori Yamamoto, G. Loeb (2012)
Use of tactile feedback to control exploratory movements to characterize object compliance
Frontiers in Neurorobotics, 6
R. Hellman, Cem Tekin, M. Schaar, V. Santos (2018)
Functional Contour-following via Haptic Perception and Reinforcement Learning
IEEE Transactions on Haptics, 11
Chuanyu Yang, N. Lepora (2017)
Object exploration using vision and active touch
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Jeremy Fishel, G. Loeb (2012)
Bayesian Exploration for Intelligent Identification of Textures
Frontiers in Neurorobotics, 6
Kazuo Tanaka, Hong Wang (2008)
Fuzzy Control Systems Design and Analysis: A Linear Matrix Inequality Approach
Xiaojing Song, Hongbin Liu, K. Althoefer, T. Nanayakkara, L. Seneviratne (2014)
Efficient Break-Away Friction Ratio and Slip Prediction Based on Haptic Surface Exploration
IEEE Transactions on Robotics, 30
S. Lederman, R. Klatzky (2009)
Haptic perception: A tutorial
Attention, Perception, & Psychophysics, 71
Junghwan Back, João Bimbo, Y. Noh, L. Seneviratne, K. Althoefer, Hongbin Liu (2014)
Control a contact sensing finger for surface haptic exploration
2014 IEEE International Conference on Robotics and Automation (ICRA)
Angelica Avilés, Samar Alsaleh, E. Montseny, P. Sobrevilla, A. Casals (2016)
A Deep-Neuro-Fuzzy approach for estimating the interaction forces in Robotic surgery
2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)
N. Lepora, Alex Church, Conrad Kerckhove, R. Hadsell, John Lloyd (2018)
From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and an Optical Biomimetic Tactile Sensor
IEEE Robotics and Automation Letters, 4
T. Takagi, M. Sugeno (1985)
Fuzzy identification of systems and its applications to modeling and control
IEEE Transactions on Systems, Man, and Cybernetics, SMC-15
A. Bicchi, J. Salisbury, D. Brock (1990)
Contact Sensing from Force Measurements
The International Journal of Robotics Research, 12
(2017)
Exploratory tactile servoingwith active touch
Hongbin Liu, K. Nguyen, V. Perdereau, João Bimbo, Junghwan Back, Matthew Godden, L. Seneviratne, K. Althoefer (2015)
Finger contact sensing and the application in dexterous hand manipulation
Autonomous Robots, 39
S. Lederman, R. Klatzky (1993)
Extracting object properties through haptic exploration.
Acta psychologica, 84 1
A. Okamura, M. Cutkosky (2001)
Feature Detection for Haptic Exploration with Robotic Fingers
The International Journal of Robotics Research, 20
Vinicius Fonseca, T. Oliveira, Katerina Eyre, E. Petriu (2017)
Stable grasping and object reorientation with a three-fingered robotic hand
2017 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS)
Nantachai Sornkarn, T. Nanayakkara (2017)
Can a Soft Robotic Probe Use Stiffness Control Like a Human Finger to Improve Efficacy of Haptic Perception?
IEEE Transactions on Haptics, 10
Jianzhu Yin, Peter Aspinall, V. Santos, J. Posner (2018)
Measuring Dynamic Shear Force and Vibration With a Bioinspired Tactile Sensor Skin
IEEE Sensors Journal, 18
Benjamin Ward-Cherrier, Nicholas Pestell, Luke Cramphorn, B. Winstone, M. Giannaccini, J. Rossiter, N. Lepora (2018)
The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies
Soft Robotics, 5
F. Amirabdollahian, S. Livatino, B. Vahedi, Radhika Gudipati, Patrick Sheen, Shan Gawrie-Mohan, N. Vasdev (2018)
Prevalence of haptic feedback in robot-mediated surgery: a systematic review of literature
Journal of Robotic Surgery, 12
J. Konstantinova, Min Li, Gautam Mehra, P. Dasgupta, K. Althoefer, T. Nanayakkara (2014)
Behavioral Characteristics of Manual Palpation to Localize Hard Nodules in Soft Tissues
IEEE Transactions on Biomedical Engineering, 61
Johan Tegin, J. Wikander (2005)
Tactile sensing in intelligent robotic manipulation - a review
Ind. Robot, 32
S. Ciotti, Teng Sun, Edoardo Battaglia, A. Bicchi, Hongbin Liu, M. Bianchi (2019)
Soft tactile sensing: retrieving force, torque and contact point information from deformable surfaces
2019 International Conference on Robotics and Automation (ICRA)
Danfei Xu, G. Loeb, Jeremy Fishel (2013)
Tactile identification of objects using Bayesian exploration
2013 IEEE International Conference on Robotics and Automation
Qiang Li, C. Schürmann, R. Haschke, H. Ritter (2013)
A Control Framework for Tactile Servoing
, 09
Muhammad Jamil, Prince Annor, J. Sharfman, R. Parthesius, I. Garachon, M. Eid (2018)
The Role of Haptics in Digital Archaeology and Heritage Recording Processes
2018 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE)
Nicolas Sommer, A. Billard (2016)
Multi-contact haptic exploration and grasping with tactile sensors
Robotics Auton. Syst., 85
Vinicius Fonseca, D. Kucherhan, T. Oliveira, Da Zhi, E. Petriu (2017)
Fuzzy controlled object manipulation using a three-fingered robotic hand
2017 Annual IEEE International Systems Conference (SysCon)
C. Rosales, F. Spinelli, M. Gabiccini, Claudio Zito, J. Wyatt (2018)
GPAtlasRRT: A Local Tactile Exploration Planner for Recovering the Shape of Novel Objects
Int. J. Humanoid Robotics, 15
F. Karray (1997)
Intelligent control: Fuzzy logic applications
Engineering Applications of Artificial Intelligence, 10
Zhanat Kappasov, J. Corrales, V. Perdereau (2015)
Tactile sensing in dexterous robot hands - Review
Robotics Auton. Syst., 74
A. Okamura, Michael Turner, M. Cutkosky (1997)
Haptic exploration of objects with rolling and sliding
Proceedings of International Conference on Robotics and Automation, 3
Chenguang Yang, Zhijun Li, E. Burdet (2013)
Human like learning algorithm for simultaneous force control and haptic identification
2013 IEEE/RSJ International Conference on Intelligent Robots and Systems

Publisher: Springer Journals
Copyright: Copyright © The Author(s) 2020
ISSN: 0929-5593
eISSN: 1573-7527
DOI: 10.1007/s10514-019-09896-7
Publisher site: See Article on Publisher Site

Abstract

Haptic exploration is a process of using haptic feedback to interact and perceive an unknown object. It is an essential approach to understand the physical and geometrical properties of the object. While numerous research has been carried out for haptic exploration on static objects, haptic exploration on objects with dynamic movements has not been reported. It is due to the signiﬁcant challenges to achieve robust force and velocity control when the object is nonstationary. In this work, a novel adaptive force and velocity control algorithm based on intrinsic contact sensing (ICS) for haptic surface exploration of dynamic objects is presented. A fuzzy-logic control framework making use of the information obtained from ICS has been developed. To validate the proposed control algorithm, extensive surface exploration experiments have been carried out on objects with different surface properties, geometries, stiffness, and concave or convex patterns. The validation results demonstrate the high accuracy and robustness of the proposed algorithm using different experimental platforms. Keywords Haptic · Autonomous control · Fuzzy logic 1 Introduction contact sensing hardware and intelligent control algorithms (Tegin and Wikander 2005). Haptic exploration of the surface is a fundamental tool for Due to the direct interaction with the object, signiﬁcant humans to understand the surface properties of unknown hurdles in control and sensing are presented for haptic explo- objects with only tactile sensing (Lederman and Klatzky ration (Okamura and Cutkosky 2001). With the emergence of 1993, 2009), especially in reduced visibility scenarios like new force/tactile sensing hardware and data processing meth- underwater, smoky disaster environment and inside tissues ods, several new haptic exploration algorithms on unknown during surgery. It has many useful applications in areas like objects and unstructured environment have been developed. robot surgery (Amirabdollahian et al. 2017), shape recov- Lepora et al. have been working on the development of ery (Rosales et al. 2018), heritage recording (Jamil et al. the optical tactile sensors with bio-mimetic morphologies 2018), dexterous robot hand (Kappassov et al. 2015). Unlike (Ward-Cherrier et al. 2018), and they have used thus sen- computer vision which has been progressing tremendously in sors for robust contour following with deep learning method recent years, haptic sensing and interaction remain as techni- (Lepora et al. 2019). Yin et al. (2018) developed a bio- cally challenging due to the relatively limited development of inspired tactile sensor skin for measuring dynamic shear force and vibration. Fishel and Loeb (2012) and Xu et al. (2013) proposed a Bayesian exploration method for iden- tiﬁcation of textures where they choose the exploratory Electronic supplementary material The online version of this article (https://doi.org/10.1007/s10514-019-09896-7) contains movements that would yield the most useful information. supplementary material, which is available to authorized users. Hellman et al. (2018) presented a reinforcement learning method for contour following with haptic feedback. Yang and B Hongbin Liu Lepora (2017) tried to combine vision for object exploration hongbin.liu@kcl.ac.uk for edge estimation. Sommer and Billard (2016) presented a Teng Sun Multi-contact haptic exploration method manly for grasping teng.sun@kcl.ac.uk tasks. Su et al. (2012) controlled the movement of the robot Haptic Mechatronics and Medical Robotics (HaMMeR) Lab, arm in the contact normal direction with haptic feedback to King’s College London, London, UK 123 774 Autonomous Robots (2020) 44:773–790 characterize the compliance of the pressed object. Similarly, tangential force (Liu et al. 2012, 2015). With the precise con- Sornkarn and Nanayakkara (2017) used a soft robotic probe tact information, an efﬁcient algorithm has been designed to measure object stiffness with haptic control. based on normal force conditions to control the robot ﬁnger Konstantinova et al. (2014) presented that humans use to explore the surface while maintaining the contact normal different force or velocity control strategies to detect abnor- force (Back et al. 2015). However, there are three major lim- mality inside soft objects, and the performance of the itations in the previous algorithms to prevent it from being perception can be enhanced by controlling the exploration applicable to dynamic objects. First, previous control method speed. And Okamura (2000) showed that it is important to is prone to induce large force control error when the surface maintain a proper normal force level during sliding explo- has large curvature changes or uneven. Second, the controller ration procedures to guarantee the force sensing accuracy, has a ﬁxed set of parameters and cannot be adapted to differ- considering the practical limitation of the sensor hardware. ent objects to achieve similar good performance. Third, the Furthermore, if the explore velocity can be controlled, more velocity of the ﬁnger sliding is not controllable. information of the explored object such as surface viscosity In this study, the following contributions have been (Liu et al. 2012), break-away ratio (Song et al. 2014) can be made: (1) A robust and adaptive normal force and explore revealed. These all showed the paramount importance of the (scanning) velocity control algorithm was developed based good control of contact forces and scanning velocity during on haptic information during surface exploration. It was haptic exploration. evaluated using a self-fabricated contact sensing ﬁnger on However, to date, to achieve the force and explore velocity non-stationary objects. Objects with variant surface physi- control during haptic exploration is still an open challenge, cal and geometry properties were tested and results show and an effective method for doing so has not been found that the normal force control method can achieve an aver- in literature. In most of the previous works, the exploration age mean square error (MSE) less than 0.006 and standard procedure has been carried out by scanning over a relatively deviation (SD) less than 0.07 N. Also, the explore velocity ﬂat surface of the object or by pressing. Some researchers can be controlled with an average Δv of 0.48 mm/s and SD considered force control during 1 DoF compressing motion of 2.07 mm/s. (2) A fuzzy-logic control framework was cre- (Su et al. 2012; Sornkarn and Nanayakkara 2017). A few ated that can increase the ﬂexibility of the control algorithm, studies have been done on the sliding exploration but only especially when the object is non-stationary and its proper- limited to static objects with simple geometry (like ﬂat sur- ties are unstructured. The control rules of the fuzzy method face), avoiding the problem of feedback force and velocity can be changed adaptively corresponding to different objects control (Fishel and Loeb 2012; Hellman et al. 2018). Similar being explored by shifting the membership boundaries. (3) to the work proposed in this study, Lepora et al. (2017) has The control algorithm can be applied to devices with different proposed an exploratory tactile servoing method where the kinematic conﬁgurations that have different workspace and next exploration position was decided with a Bayesian per- force ranges. The algorithm was validated successfully with ceptual framework, but the force was not actively controlled a 6 DoFs robot arm (with a contact sensing tip mounted) in their study and the objects were static. Li et al. (2013) using objects with different stiffness and geometries. (4) introduced a tactile servoing control framework adapting a Even though the type of contact was point contact in the PID controller, which can be used to realize a speciﬁc tactile tests, it also works for surface contact (using tactile array interaction pattern. Yang et al. (2013) presented a learning sensor) if contact locations can be obtained in real-time. algorithm for simultaneous force control and haptic identiﬁ- cation, where they used complicated models instead of real force sensor, and the algorithm was only tested in simulation. 2 Experimental platform Okamura and Cutkosky (2001) used a 2 DoFs robot ﬁnger to detect features with haptic exploration, where they presented Three platforms have been created to validate the proposed that the force was kept low; however, no detailed results were algorithm, which are denoted as platform-a, b and c. The shown. Therefore, to the best of the surveyed knowledge, the designed experimental platform-a consists of two main parts: robust and accurate normal force and explore velocity con- one is the self-fabricated contact sensing ﬁnger, and the other trol when the object is dynamically moving during surface is the Universal Robot (UR)-3 robot arm. The contact sens- exploration has not been reported. ing ﬁnger consists of two joints, one ﬁngertip and the base To overcome the challenges, continuous study has been connecting the UR-3 robot arm, as shown in Fig. 1. Each carried out on enhancing haptic sensing capability to realise joint consists of a DC motor (Maxon DC motor RE10 with adaptive contact force and velocity control. Adapting ICS encoder) and a bevel-gear box (reduction: 1:1). The Maxon with 6-axis force-torque measurement (Bicchi et al. 1993), control panel (POS2 24/2) is used to control the position or a real-time contact sensing algorithm has been developed velocity of the motor. The maximal frequency of the position to provide instantaneous contact location, normal force and or velocity control is 1 kHz, and such high frequency enables 123 Autonomous Robots (2020) 44:773–790 775 Fig. 2 The haptic exploration platform-c involving a UR-3 robot arm and a contact sensing tip. The tip has a Nano17 force-torque sensor and a self-fabricated cap made of silicon. The tip is mounted to the end-effector of the arm using a 3D printed connection materials that are softer and more ﬂexible will be future work. Inside the ﬁngertip, an ATI Nano17 force-torque sen- Fig. 1 The contacting sensing ﬁnger (platform-a), which consists of 2 sor (resolution: 1/160 N) was installed. It is a six-axis sensor joints and a ﬁngertip, is shown in the top pictures. The joints consist providing force and torque data along its x, y, z-axis, and the of 3D printed links and metal connection parts, and each has a Maxon motor and bevel gearbox inside. An ATI Nano17 6-axis force-torque data can be updated at a frequency of 7200 Hz with a DAQ sensor is embedded inside the ﬁngertip and covered with a polyurethane data acquisition box. rubber layer. It has a 3D printed base that can be mounted on the UR-3 The software was developed with ROS (robot operating robot arm (bottom picture) as platform-b. The robot arm can rotate to system, www.ros.org) using an Ubuntu PC (CPU: i7-6700 any desired angle to allow the ﬁnger to explore more areas @3.4GHz and 16G RAM). The PCIe-6320 data acquisition card (up to 16 kHz) is used to obtain sensor data, and then the high-level (trajectory generation and decision making) ROS programs are used to communicate with the card to get control to be accurate and rapid. For motor control, a multi- and process the data. In practical, the tactile sensing toolbox thread method is used to eliminate the asynchronization when (TST) proposed in Ciotti et al. (2019) was used for sensor controlling multiple motors, which ensures the accuracy of data processing and contact location estimation. Including the position control and is beneﬁcial for the control of more the other modules like the control and calculation, the loop joints or ﬁngers. The high precision of the motor control frequency of the ROS is 100 Hz, meaning it updates the sensor panel and the self-locking mechanism on the joint (the baf- data and ﬁnish all the necessary computations every 1/100 s. ﬂes at 90 degrees) ensure the repeatability of the exploration, The ﬁnger is installed on a 3D printed base, which can be making the online multi-time haptic exploration possible. mounted on the UR-3 robot arm. This ﬁnger and robot arm Experimental tests were also carried out to record the position combined platform is denoted as the platform-b. The base can deviation of the joints after several explorations and input the be rotated to any angle through the robot arm to achieve the error into the control system as the compensation to reduce exploration on more surfaces, as shown in Fig. 1 bottom. For the error caused by mechanical fault and friction between the the next steps, more ﬁngers will be mounted on the base to components. realize the larger area’s exploration and achieve more actions For fabricating the ﬁngertip, the inner core was ﬁrstly 3D- other than surface exploration. These together will help to lay printed with plastic and then embedded into a layer of 2 mm a solid hardware foundation for the active exploration and TM rubber. The Poly 74-30 liquid rubber from Polytek was dexterous manipulation system. used to construct the polyurethane rubber layer, its hardness Moreover, to evaluate the proposed algorithm with larger is 30A Shore, and young ’s Modulus is 100psi. The whole working space and force ranges, the UR-3 robot arm was used ﬁngertip (3D printed core and rubber layer) has very high for haptic surface exploration as platform-c. The Nano17 sen- hardness and is close to rigid body; therefore there was no sor was covered with a silicone cover and ﬁxed onto the robot deformation within the range of the forces used in exper- arm end-effector to provide real-time force-torque data for iments, and no effect of hysteresis. The study of ﬁngertip the control algorithm, shown in Fig. 2. Since the workspace 123 776 Autonomous Robots (2020) 44:773–790 of the ﬁnger is limited and the force generated by it is rel- Table 1 Parameter notation for control system atively small, the robot arm with more ﬂexibilities helps to Name Deﬁnition validate the proposed algorithm in larger workspace and force ε Force error between measured and target, N control range. On the contrary, the contacting sensing ﬁnger ε ˙ Change rate of error, N/s can realize ﬁner force and velocity control due to the high F Desired normal force, abs value, N frequency and accuracy of the motors, which is a good tool N d to verify the robustness of the control algorithm in the low F Measured normal force, abs value, N N m target range. More will be detailed in the experiment section. K Fuzzy control output, a coefﬁcient P Coordinates of current position in base frame now P Coordinates of contact point in ﬁngertip frame 3 Control algorithm P Coordinates of next position in base frame next With the experimental platform, an adaptive real-time nor- mal force and explore velocity control algorithm has been Table 2 Parameter calculations developed. A fuzzy-logic framework using the haptic infor- Notation Calculation mation was designed for haptic exploration on objects with dynamic movements. In this section, the overall control pro- Frames Base frame {O}(X , Y ), O O cess will be shown ﬁrst, and then the detailed introduction of ﬁngertip frame {E}(Y , Z ) E E the main components constituting the control algorithm will Sum of joint angles θ = θ + θ 12 1 2 be presented. Angle of contact point θ = arctan2(z , y ) c e e in ﬁngertip frame 3.1 Overall control process Angle of the normal θ = θ − θ N 12 c force in base frame Correcting force F = abs(K (F − F )) N d The overall control process of the proposed algorithm is N Virtual tangential force F = K F ,(K =−1) P T P shown in Fig. 3 and Tables 1, 2 show the deﬁnition of the T Angle between β = variables. The control algorithm is a feedback control sys- correcting and guiding arctan2(−F , K F /|K |) T N tem, and the input of it has two parts: force and torque values force from the Nano17 force-torque sensor, each with three com- Angle of guiding force α = θ + β T T ponents ([ f , f , f )] and [m , m , m ] ), and the position x y z x y z in base frame information from the photoelectric encoders inside motors Next position to reach P = P + S ·[cosα, sinα] next now v (E1, E2). The control process starts when the ﬁngertip is contacting the object, and to determine that state, a threshold function is used, i.e. it is determined that the ﬁnger is contact- to humans’ attempt to feel the object with only touch. With ing the object when the force obtained from the force sensor the force and torque values obtained from the force-torque is larger than a pre-decided value ϑ . With this method, at sensor, the contact point position in ﬁngertip frame P can be the beginning, the ﬁnger is controlled to move close to the c obtained through methods presented in Liu et al. (2015) and surface of the object, and the contact state is conﬁrmed when 2 2 2 Ciotti et al. (2019). With (E1, E2) from motor encoders, the sqrt ( f + f + f )>ϑ . In practice, this action is similar x y z contact location in the base frame P is obtained through now forward kinematics. With P obtained, the next is to decide the new loca- now tion on the surface for the ﬁngertip to reach while keeping the normal force as desired. The resultant normal force and tangential force are obtained simultaneously with the con- tact position through the TST. The control target F is then subtracted from the measured normal force to get the error ε and ε is divided by the cycling time to get the changing t t rate of the error: ε ˙ = ε /dt.And (ε , ε ˙ ) are the input of t t t t the fuzzy control method. The output of the fuzzy control method is a coefﬁcient K , which is used to generate a cor- Fig. 3 The overall ﬂow of the control algorithm, which is a closed-loop recting force, helping the ﬁnger maintain the normal force. control system. The output K is obtained after the inputs (ε , ε ˙ ) have t t In detail, each K is used to provide a correcting force vec- been processed, and the inputs are updated with the motor movement. The overall loop rate is 100 Hz using ROS tor that leads the ﬁnger to move either far away or closer to 123 Autonomous Robots (2020) 44:773–790 777 ⎡ ⎤ ∂ S k − f z + f y − m the object surface along the normal direction of the object. y e z e x ∂x ⎢ ⎥ Then a virtual tangential force F is obtained by taking the ∂ S ⎢ ⎥ k − f x + f z − m z e x e y ⎢ ⎥ ∂ y opposite direction of the measure friction force F ,tomake T ⎢ ⎥ g(x) = =0(1) ⎢ ∂ S ⎥ k − f z + f x − m the ﬁnger to move forward along the surface of the object. y e y e z ⎢ ⎥ ∂z ⎣ ⎦ 2 2 2 These two forces form a total force vector that can be used x y z e e e + + − 1 2 2 2 to decide the next explore location. As shown in Table 2, K a b c is ﬁrst used to decide the magnitude of the correction force E T The contact information x =[P , k] can be obtained by (F ), and the angle α of the total force in frame {O} is calcu- identifying the roots of g(x) = 0. The Levenberg–Marquardt lated after getting the required angles (θ and β ). After that, (LM) method, which is a gradient-based iterative method for a S parameter is used to determine the size of the increment non-linear parameter estimation, was used to get the solution for the next location (P ), whose unit is mm. It can be next iteratively. After P has been acquired, with joint angles obtained through experiments or controlled with closed-loop measured by motor encoders, P can be obtained through control. now forward kinematics: With P been determined, the control signals for motors next are generated through inverse kinematics of the platform. The motors are then controlled by motor controllers to rotate to P = P ·[T T T ]· R (2) now 1 2 3 1 desired positions. The force changes induced by the move- ment of the object or local geometric changes will increase ⎡ ⎤ cosθ −sinθ 0 cosθ l 1 1 1 1 the force error, and the control algorithm will make adjust- ⎢ ⎥ sinθ cosθ 0 sinθ l + l 1 1 1 1 0 ments accordingly as quickly as possible within its capability. ⎢ ⎥ T = (3) ⎣ ⎦ 00 1 0 This control loop keeps cycling until the ﬁnger is not contact- 00 0 1 ing the object or any one of the ﬁnger joints has reached the predeﬁned limit. After that, the haptic surface exploration on ⎡ ⎤ cosθ −sinθ 0 cosθ l 2 2 2 2 an area of the contacting object is completed subsequently. ⎢ ⎥ sinθ cosθ 0 sinθ l 2 2 2 2 ⎢ ⎥ T = (4) ⎣ ⎦ 00 1 0 00 0 1 3.2 Contact location estimation ⎡ ⎤ 100 y ⎢ ⎥ 01 0 −(l + z ) 3 e ⎢ ⎥ In the control algorithm, there are two main coordinate T = (5) ⎣ ⎦ 001 0 frames, the base {O} and ellipsoid ﬁngertip frame {E}, and 000 1 the control method is based in the base frame. The control goal is to determine the next exploration location and main- R =[0, 0, 0, 1] (6) tain the magnitude of the normal force simultaneously. To realize this goal, the current contact point in the base frame P needs to be calculated ﬁrstly. Let the coordinates of 3.3 Next position determination now the contact point to be P =[x , y , z ] in frame {E}, and e e e P =[x , y , z ] in frame {O}. With the obtained current contact position in the base frame, now o o o It should be noted that the ﬁngertip material used in the the next position was derived considering the contact normal experimental platform is very stiff and the effect of hysteresis force. The parameters required for the next position deci- can be ignored. And to verify this, experiments were carried sion were calculated geometrically, and for easier view, a to evaluate the force-deformation relationship when loading schematic diagram shown in Fig. 4 was drew. Table 2 shows and releasing forces on the ﬁngertip. As a result, the force the calculation of the relevant variables. Two key points have reached 5 N when the deformation was less than 0.5 mm; thus, been considered when deciding the next position to explore: it was reasonable to ignore the deformation of the ﬁngertip the ﬁrst one is to maintain the magnitude of the normal force in this study. Therefore, the equilibrium equation in Liu et al. within the desired range, the other one is to control the ﬁn- (2015) has been simpliﬁed to (1), where a = b = 15, c = 9.5 ger to move forward following the surface of the object. For are the dimensions of the ellipsoid ﬁngertip, S is the func- the ﬁrst point, a virtual normal force was generated based on tion for the ellipsoid, and k is a scaling factor. As future (ε , ε ˙ ), which can guide the ﬁnger to move closer or away t t work, softer ﬁngertips will be fabricated to estimate the sur- from the normal direction of the object, and the normal force face haptic exploration on objects with different stiffness, will increase if closer to, and vice versa. This virtual normal which will be of great help in the research of ﬂexible and force is called the “correcting force”, and denoted as F . K soft manipulators. was obtained through the fuzzy control method, which will be 123 778 Autonomous Robots (2020) 44:773–790 which means the response time is affected by S . For exam- ple, if the object moves away from the ﬁnger, the error will become larger and P will be closer to the object next to increase the force. After P has been obtained, the motor control signals next were obtained through inverse kinematics. By rotating the motors to the required positions, the ﬁnger will move to explore the next location. Repeating this action until the ﬁnger ﬁnished contacting the object surface, the surface exploration was completed while maintaining the local nor- mal force in the whole exploration process. As shown above, providing the geometric conﬁguration of the tip, forward and inverse kinematics of the platform, the control variables can be obtained, regardless of the number of DoF of the platform. Which indicates that the control algorithm is not limited to this two joints ﬁnger, it can also be applied to more com- Fig. 4 A schematic diagram showing the calculation of related angles plicated device like robotic arm or soft robots. However, the when deciding the next explore location. The deﬁnition of the notations can be found in Table 2. The black curve contacting the contact sensing parameters for the fuzzy control method and rotation matrix ﬁnger is a sketch of the object outline. The right-hand side is a magniﬁed for different devices are slightly different due to their speciﬁc view of the contact area details. This ﬁgure only shows the situation moving resolution, exploration direction and force ranges. where the correcting force F has the same direction with the real Therefore, for different devices, re-calibration is necessary normal force, which means the real force is smaller than the target force, and the ﬁnger needs the push towards the object to increase the to achieve the best performance, which in theory needs to force be done only once. Practically, the calibration of the fuzzy control method can be done automatically with optimized methods by miniaturizing the error between the requested detailed in the next section. In practice, this decision-making and real haptic exploration data, which could be explored more in future work. strategy is similar to the rule human beings use when they explore the surface of the object. Humans normally push fur- ther when they want to increase the pressure and leave away 3.4 Fuzzy control method to reduce it. For the second point, a virtual tangential force F was Fuzzy control method has been studied extensively in robot introduced proportional to the tangential force on the ﬁnger- systems recently (Aviles et al. 2016). da Fonseca et al. tip, but in the opposite direction. This force can ensure that the (2017a) and da Fonseca et al. (2017b) used the fuzzy control ﬁnger moves forward along the surface of the object. Com- method to realize grasp and exploration tasks with a three- bined with F , a total force F was obtained. The force F ﬁngered robotic hand and have achieved good results. It is a G G can guide the ﬁnger to move along the surface of the object method that mimics the intuitive decision-making process of and maintain the normal force, which is called the “guid- humans. Humans can solve complex problems using impre- ing force”. And based on this guiding force, the direction of cise information such as common sense, expert knowledge, the next position P in base frame was calculated. Table or prior experience which does not require precise models next 2 shows the detailed calculation of related angles. First, the (Takagi and Sugeno 1993). Such information can be repre- angle θ of the normal force in frame {O} was calculated sented by linguistic rules, like in If-Then format (Tanaka and based on the angle of contact point in frame {E}, then the Wang 2004). And fuzzy logic is the theory of fuzzy sets that angle β between F and F was obtained after obtaining the can represent mathematically the human spirit for approx- forces, ﬁnally the angle α of F in frame {O} was calculated. imation reasoning based on imprecise information, which Next, S was introduced to determine the incremental is also a major advantage over other classic control meth- size of the next exploration position compared to the cur- ods that usually require very precise modelling or parameter rent position. As mentioned above, it can determine not only ﬁne-tuning. When the external environment is complex, or the velocity of exploration but also the control accuracy the current state is unknown, it will be difﬁcult to obtain an and sensitivity to external disturbances induced by geome- accurate model due to the unclear state and inﬂuence fac- try changes, surface roughness and the movement of objects. tors. In such scenarios, the fuzzy control method can better Practically, when the object moves, the force error with the satisfy the control requirements (de Silva 1995). For exam- target force changes accordingly, the control method makes ple, when a person touches a soft or brittle object, they will adjustments to eliminate the error as quickly as possible, choose to use less pressure to avoid breaking the object, and 123 Autonomous Robots (2020) 44:773–790 779 when they encounter a bump or obstacle, they will leave the while for the PID method, they are 0.11 N and 0.013 N. The surface of the object for further exploration correspondingly. results indicate the better ﬂexibility of the fuzzy method when These decisions are not precisely made but intuited by learnt dealing with dynamic and unknown object. Also, by chang- knowledge or former experience. ing the boundary of the memberships, the control method can Conventional control techniques such as PID control, be adapted to different objects. Therefore, in this study, the nonlinear feedback control, etc. can provide good stability, fuzzy control method was selected. robustness to uncertainties and disturbances when the under- A fuzzy control system mainly consists of 4 compo- lying assumptions are satisﬁed and values of the controller nents: Fuzziﬁers: It maps the real-valued input into fuzzy parameters are known. However, these control algorithms sets characterized by membership functions. Rule Base:It are inﬂexible and cannot generally handle vague or chang- is a knowledge base consisting of linguistic rules in If-Then ing situations. Fuzzy control method is a particular type of format. It can be obtained through experiments or experi- intelligent control, which has a knowledge-base within the ences. Fuzzy Inference Engine: Using the If-Then rules in controller, and the control actions are generated through an the knowledge base, it performs reasoning by producing a inference mechanism. It can handle vague or incomplete fuzzy output according to the fuzzy input given by the fuzzi- information and the knowledge itself can evolve through ﬁer. De-fuzziﬁers: It converts the fuzzy output given by the learning (de Silva 1995). Experiments using classic control fuzzy inference engine to produce a real-valued output. To method—PID control have been carried out, and the results create the fuzzy control framework of the control algorithm, show that for a ﬁxed object and a certain area on the object, these four components were designed. the normal force can be controlled at a relatively ﬁne level The inputs of the system are the force error between the during exploration with ﬁxed P, I and D values. However, for measured and target value and its changing rate: (ε , ε ˙ ).The t t another object or a different area on the surface, the P, I and input will be converted into fuzzy sets represented by mem- D values needed to be ﬁne-tuned before they can be used to bership functions, and the continuous fuzzy set was used in get similar performance. As in this study, the ﬁnger was con- this work, which is denoted as: trolled to explore objects with dynamic movement at random areas, the fuzzy control method which has more ﬂexibilities is μ (x ) F = (7) more suitable. Figure 5 shows the normal force result of using the fuzzy method and PID method for the rough plate (no. 25, shown in Sect. 4). The parameters were obtained for both where x is indicating the input data. The boundaries are methods using the light bulb as the learning object, and the deﬁned as the region of the universe of which 0 <μ (x)< rough plate was used for the testing object. As a result, the SD 1. There are common membership functions such as Triangu- and MSE of the resultant normal force (target normal force is lar function, Gaussian function, Bell function, etc. Different 0.6 N) when using the fuzzy method are 0.08 N and 0.0067 N, types of membership functions were tested, and the Gaussian function has been selected, which showed the best perfor- mance. The fuzzy output is also an inferred membership function, and by moving the boundaries of output member- Normal force result - Fuzzy vs PID 1.2 ship functions, the coverage of different memberships can PID Fuzzy be changed, and the ﬁnal control inclination following pre- deﬁned logic rules is modiﬁed adaptively. For example, after 0.8 one exploration, if the average normal force error is too large, 0.6 the boundaries can be shifted rightward to decrease the cover- age of the closing action. In this way, if the controller attempts 0.4 to increase the normal force, the magnitude will be smaller. 0.2 Figures 6, 7 and 8 show the fuzzy control frame of using the Gaussian function for the input and output memberships. The 0 600 1200 1800 2400 3000 3600 4200 4800 5400 Time (ms) number of output memberships is decided by the knowledge base, and they are shown in different line types. Fig. 5 The ﬁgure shows the normal force result during the exploration For creating the knowledge base, a 9-rule IF-THEN (IF of the rough plate (no. 25) using the fuzzy and PID method. The param- premise THEN conclusion) rule base was formed shown in eters were obtained for both methods using the light bulb as the training object, and the rough plate was used as the testing object. It can be seen Table 3. In which “Leave” means controlling the ﬁnger to that the vibration of the normal force when using the PID method is leave away from the surface of the object, “Close” means much larger than that when using the fuzzy method. The SD and MSE moving closer to the object and “Stay” means maintaining the of the normal force are 0.08 N and 0.0067 N when using the fuzzy con- distance from the object in the contact normal direction. And trol method, and 0.11 N and 0.013 N when using the PID method. The results indicate the better ﬂexibility of the fuzzy method “small”, “medium” and “large” mean the level of the actions. Force (N) 780 Autonomous Robots (2020) 44:773–790 Fuzzy input distribution - e when ε is positive and ε ˙ is zero, the conclusion can either 1 t t Negative 0.9 Zero be “leave medium” if ε has larger effect or “leave small” if Positive 0.8 the effect of ε is not signiﬁcant. As mentioned before, fuzzy 0.7 logic represents mathematically the experienced knowledge 0.6 for approximation reasoning; it requires the learnt knowl- 0.5 edge or experience to decide the rules. The bulb object was 0.4 0.3 chosen to get the fuzzy rules and decide the range of output 0.2 membership functions. For the fuzzy inference engine, the 0.1 Mamdani (max-min) inference method was selected, which -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 can present a fuzzy output in membership function according e (N) to the fuzzy input. The centroid method was selected (also known as the centre of gravity (CoG)), which provided us Fig. 6 The ﬁgure is showing the Gaussian distributions of the fuzzy input (ε ). The line types present the different memberships. ε has the t t with satisﬁed output values. The CoG for continuous form range of (− 0.5, 0.5)N, and it has three components: the negative, zero can be calculated using: and positive with mean value of − 0.5, 0, and 0.5 respectively μ (z)zd F z Fuzzy input distribution - de K = z = (8) μ (z)d F z Negative 0.9 Zero Positive 0.8 where μ (z) is the membership function for the output and 0.7 K = z is the output value. 0.6 0.5 With the input being mapped into membership functions, 0.4 the output K used for deciding the next explore location 0.3 was obtained with the de-fuzziﬁer, following the rules in the 0.2 knowledge base. Thus, the fuzzy control method has been 0.1 ﬁnalized. For the software implementation of the fuzzy con- -10 -8 -6 -4 -2 0 2468 10 trol method, the fuzzylite library (https://fuzzylite.com/) has de (N/s) been used. Fig. 7 The ﬁgure is showing the Gaussian distributions of the fuzzy input (ε ˙ ). The line types present the different memberships. ε ˙ has the t t 3.5 Explore velocity control range of (− 10, 10)N, and it has three components: the negative, zero and positive with mean value of − 10, 0, and 10 respectively It has been found out that the S is the parameter that affects the explore velocity and study on how S can be used to Both (ε , ε ˙ ) have values of positive, zero and negative, the control the explore velocity has been carried out. For each t t combination of these situations forms 9 (3x3) rules. The rules exploration, S can either be ﬁxed or controlled with closed- can be adjusted based on the experiment results, for example, loop control. In the test, when S was ﬁxed, the average Fig. 8 The ﬁgure is showing the Fuzzy ouput distribution - K Gaussian distributions of the Leave lerge Leave medium fuzzy output K . The line types Leave small 0.9 Zero present the different Close small Close medium memberships. As shown in the 0.8 Close large ﬁgure, K has the range of 0.7 (− 1,1), and it is following 9 rules involving 7 actions: leave 0.6 large, leave medium, leave small, stay, close small, close 0.5 medium and close large, with 0.4 μ of − 1, − 0.7, − 0.3, 0, 0.25, 0.6 and 1 respectively 0.3 0.2 0.1 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Probabilities Probabilities Probabilities Autonomous Robots (2020) 44:773–790 781 Table 3 Fuzzy knowledge base IF-THEN Linguistic rules ε Positive (0 to 0.5) Zero (−0.2to0.2) Negative(−0.5to0) ε ˙ Positive (0 to 10) Leave large Leave medium Close small Zero (− 10 to 10) Leave medium Stay Close medium Negative (− 10 to 0) Leave small Close medium Close large explore velocity turned out to be similar for different objects, but the standard deviation was relatively high. In this work, a PID controller for the velocity control was created. And for creating the controller, the real-time velocity was calculated with v = l /dt, where l is the length the ﬁnger moved in m m m the base frame in one loop and dt is the time used for one loop. After ﬁltering the v with a moving average ﬁlter (the ﬁlter takes 10 points as the input), the error with the desired v was fed into the PID controller to generate the S value for d v the next position decision. A new S will change the explore velocity in the next movement. Algorithm 1 shows the detail of the control process. Algorithm 1 Velocity control algorithm 1: Initialise variables, l = 0, count = 1, I = 0, S = 0.45mm, m v decide k , k , k , input target velocity v ,ε = v p i d d v d 2: while joint angles θ ,θ <θ do 1 2 limit 3: if count == 1 then 4: Get current time t and contact position P 5: Action: Haptic exploration with initial S Fig. 9 The objects used in the haptic exploration experiments. From top 6: count ← count + 1 to bottom, left to right is: bulb, mouse, 3D printed statue, cup, human 7: Get new contact position P and time t ﬁnger, soft box, sponge (softer), black sponge, metal spring, glass bottle; 8: Compute the distance the ﬁnger travelled: l ← dis(P , P ) m t and plates made with different materials and patterns: mirror stainless, 9: Compute the velocity v ← l /(t − t ) m m satin stainless, BA stainless (canvas), BA stainless (squares), BA stain- 10: Smooth v with moving average ﬁlter m less (13SD), BA stainless (cambridge), BA stainless (11PS), bronze 11: New error ε ← v − v , I ← I + ε ∗ dt d m stainless (pippin), green stainless (poppy), BA stainless (5WL), BA v v 12: Compute the new S with PID controller: stainless (2WL), BA stainless (7GM), BA stainless (6WL), BA stain- 13: S ← k ∗ ε + k ∗ I + k ∗ (ε − ε )/dt p i d v less (9EH), red stainless (pearl) v v v 14: Store current error: ε ← ε ,time t ← t 15: Store current position P ← P 16: Action: Haptic exploration with S active exploration algorithm where velocity is important for 17: return 0 Return 0 if ﬁnished successfully deciding the time cost. When S was changed with the PID controller, the stan- dard deviation of the velocity became smaller, thus, the 4 Haptic surface exploration experiments velocity control for surface exploration has been achieved. The velocity control can help to get velocity related properties With different platforms, various haptic surface exploration such as viscous friction coefﬁcient f which can be obtained experiments were carried out to evaluate the performance by measuring the viscous friction F and the coefﬁcient is of the developed control algorithm. In this section, the one- calculated using f = F /v. With the explore velocity been time exploration with the contact sensing ﬁnger (platform-a) v v controlled, the instantaneous power generated by the friction will be introduced ﬁrst, followed by the adaptive parameter during the contact and sliding is also controllable, which tuning for the multi-time explorations using platform-b, then is helpful in manufacturing procedures like grinding. More- the haptic exploration with the robot arm (platform-c) will over, it can be beneﬁcial to the control of explore efﬁciency be presented, and the velocity control experiments using the during haptic perception, facilitating the development of an platform-a will be shown at last. 123 782 Autonomous Robots (2020) 44:773–790 Table 4 Force control results of haptic exploration Resultant normal force standard deviation (N) and MSE No. F -0.2 N F -0.4 N F -0.5 N F -0.6 N F -0.7 N d d d d d 1 0.047 0.0038 0.045 0.035 0.075 0.0077 0.09 0.0083 0.091 0.0085 2 0.054 0.0036 0.056 0.037 0.059 0.006 0.076 0.0067 0.083 0.008 3 0.052 0.0033 0.07 0.006 0.076 0.0074 0.083 0.0084 0.12 0.013 4 0.048 0.0026 0.05 0.0033 0.065 0.0067 0.075 0.0063 0.09 0.0083 5 0.063 0.0048 0.07 0.0055 0.086 0.008 0.1 0.01 0.13 0.012 6 0.051 0.003 0.062 0.006 0.072 0.0065 0.076 0.0066 0.09 0.01 7 0.04 0.0033 0.052 0.005 0.064 0.0059 0.085 0.0082 0.09 0.0085 8 0.04 0.003 0.045 0.004 0.075 0.0065 0.078 0.0075 0.093 0.0094 9 0.047 0.0032 0.053 0.003 0.077 0.007 0.086 0.0088 0.1 0.01 10 0.045 0.0031 0.053 0.003 0.064 0.0079 0.089 0.0085 0.095 0.009 11 0.039 0.0027 0.05 0.0039 0.068 0.0052 0.071 0.006 0.076 0.0076 12 0.04 0.0027 0.05 0.004 0.066 0.0052 0.069 0.0077 0.079 0.0073 13 0.04 0.003 0.049 0.0032 0.066 0.0062 0.069 0.0066 0.09 0.0082 14 0.053 0.0037 0.057 0.0036 0.066 0.0047 0.071 0.0048 0.08 0.0078 15 0.043 0.003 0.056 0.0039 0.07 0.0059 0.072 0.007 0.093 0.0083 16 0.038 0.0031 0.056 0.0039 0.073 0.006 0.079 0.0067 0.095 0.0085 17 0.042 0.0038 0.052 0.0035 0.079 0.0066 0.068 0.0071 0.09 0.0082 18 0.029 0.0032 0.052 0.0033 0.067 0.006 0.073 0.0063 0.086 0.0086 19 0.038 0.0033 0.048 0.0038 0.067 0.0052 0.076 0.007 0.09 0.0085 20 0.035 0.003 0.055 0.0046 0.079 0.0076 0.081 0.0075 0.09 0.009 21 0.037 0.0027 0.054 0.0035 0.071 0.0057 0.075 0.007 0.09 0.0084 22 0.054 0.0034 0.057 0.0042 0.068 0.008 0.078 0.0088 0.1 0.0093 23 0.056 0.0037 0.059 0.003 0.07 0.006 0.073 0.0085 0.093 0.01 24 0.049 0.0038 0.058 0.0046 0.082 0.008 0.089 0.0096 0.1 0.0096 25 0.055 0.0036 0.062 0.0045 0.085 0.008 0.095 0.0085 0.12 0.0099 Avg. 0.045 0.0036 0.055 0.0039 0.072 0.0064 0.076 0.0072 0.093 0.0089 4.1 Single time haptic surface exploration target force. The μ and σ of the velocity vector were also v v calculated. Algorithm 2 summarizes the experiment protocol. For carrying out the experiments with the contact sensing ﬁn- To provide the object with dynamic movement, two ways ger (platform-a), 10 objects with different surface properties have been used, the ﬁrst was to use hand to hold the object (friction, roughness), geometry and stiffness, and 15 plates which would induce slight movement during the haptic made of different materials with different surface patterns exploration, and the other was to move the object with ran- were selected, shown in Fig. 9. It should be noted that, due dom displacement through a created device. For the second to the limited power of the motor, the maximum force the approach, the object was mounted on the linear guide shown ﬁnger can generate is less than 1.1 N, thus, the controllable in Fig. 10, which can generate a random displacement of force was constrained to 0.2–0.7 N in the tests. The target 2–4 mm during the exploration. Please refer to the video forces used in the experiments were shown in Table 4.For of “dynamic_object_exploration (MOESM1).mp4” for more each object, the haptic exploration started after the contact details. was conﬁrmed and ended until the ﬁnger ﬁnished contacting The normal force control results of one-time exploration the object. The S was chosen from 0.35, 0.45, and 0.55 mm on the light bulb, black sponge, rough plate (no. 25), smooth randomly for each exploration. During each exploration, the plate (no. 12) and 3D printed statue hold by hand are shown normal force and velocity at each contact point were stored in Fig. 11. It can be seen that, ignoring the rising period in vectors. And after the exploration, the mean value μ and of the normal force, the performance of the control is sat- SD σ of the force vector were calculated, and the MSE was isﬁed for the bulb, black sponge and smooth plate, which obtained by comparing the resultant normal force with the in total achieved an average standard deviation of 0.044 N. 123 Autonomous Robots (2020) 44:773–790 783 Algorithm 2 Object haptic surface exploration 1: Initialise vector V , V , V to store the resultant normal force, veloc- f v d ity and target force 2: Initialise variables, pointer = 1, sum = 0, input target force F and put it into the vector V ← F d d 3: Initialise the parameters for fuzzy control method (FCM) 4: if contact normal force F >ϑ then N f 5: while joint angles θ ,θ <θ do 1 2 limit 6: Get current contact normal force and store it in vector V (pointer ) ← F f n 7: Compute next moving position P with FCM next 8: Get the new angles to move next: θ ,θ ← IK (P ) 1 2 next 9: Action: The Motors rotate to the new positions 10: Record thetimeused dt and compute the distance L travelled 11: Compute the explore velocity v ← L/dt 12: Store the velocity in vector V (pointer ) ← v 13: Update pointer ← pointer + 1 14: Smooth the data with ﬁlter, (V , V ) ← filter (V , V ),toget f v f v Fig. 10 The device used to generate a random displacement for the rid of the erroneous data objects. It consists of a linear guide and a 3D printed gripper. The linear 15: Compute μ ← Mean(V ), σ ← SD(V ) f f f f guide is controlled with a gear that can be rotated manually. The linear 16: Compute μ ← Mean(V ), σ ← SD(V ) v v v v guide moves 1 mm after rotating the gear by one circle. The 3D printed 17: for each element i in V and V do f d gripper can be adjusted on the metal rails for different objects 18: sum ← sum + (V (i ) − V (i )) f d 19: MSE ← sum/pointer Normal Force during Haptic Exploration 20: return 0 Return 0 if ﬁnished successfully 1.2 Bulb Black Sponge RoughPlate Smooth Plate Statue However, for the statue and rough plate, the vibrations of 0.8 the control are larger (SD = 0.075 N) due to the complex 0.6 geometry and large concave areas on the surface. In this 0.4 work, for getting the ﬁnal results of the haptic exploration experiments, only the data when the force became stable 0.2 were used, which means that the force data in the rising 0 1500 3000 4500 6000 7500 period were omitted. In practical, the object was held by Time (ms) hand and not ﬁxed during the haptic exploration, which added dynamic movement to the object. There are two Fig. 11 Normal force results of the one-time haptic exploration on the notable facts during the experiments. First, the initial contact surface of the light bulb, black sponge, rough plate, smooth plate and 3D-printed statue. The force control target was 0.6 N and the S value position was random and different for each test, and sec- was kept at 0.45 mm. As can be seen, the rough plate and 3D-printed ond, as the hand was shaking when holding the object, the statue have the largest vibrations. This is because the 3D-printed statue object was moving slightly (1–3 mm) during the exploration. has the most complex geometries and the rough plate has large concave These all increased the difﬁculties for the control algorithm patterns, both of them make it difﬁcult to adjust the force during the exploration. And for the light bulb, black sponge and smooth plate, and in return can show the robustness of the algorithm. regardless of their complex geometry, texture, and stiffness, the force Please refer to videos of “Force_control_01 (MOESM4), control results are satisﬁed, showing the robustness of the algorithm. Force_control_02 (MOESM5), surface_exploration_statue However, as can be seen, the rising time of the algorithm is sluggish, (MOESM6).mp4” for more details. which could be improved with a faster control loop 4.2 Adaptive parameter tuning The adaptive parameter tuning method was developed for After the one-time exploration experiments were carried out, multi-time explorations. After each exploration, if the aver- the ﬁnger (platform-a) was mounted onto the end-effector of age force value μ was too much smaller than the target the robot arm, forming the platform-b. The robot arm end- and the MSE was large, the control algorithm will shift the effector can rotate to any degree after each exploration to help boundaries of the output fuzzy memberships to the right to the ﬁnger explore more areas on the surface of the object. In make the “close” action to have more coverage of the whole the experiments, the arm was controlled to rotate every 30 output distribution. This can be done by increasing the mean degrees after one exploration, and the exploration was carried values (μ ) of the Gaussian distribution of the “close” close out again, in total, the arm rotated 60 degrees and returned to membership functions shown in Fig. 8. Through this way, the starting position. The resultant haptic information of all the ﬁnger will push more during the next exploration for the the exploration procedures were recorded. error correction. Meanwhile, it is required to guarantee that Force (N) 784 Autonomous Robots (2020) 44:773–790 the changing of the control parameters will not induce too obtained. The detailed results can be found in Table 4, and much noise to the haptic exploration by monitoring the stan- from the results, the following points were concluded. dard deviation σ . Similarly, if μ was too much larger than f f F and MSE was large, the boundaries will be shifted left- 1. When the target force is increasing, the MSE and σ d f wards to make the “leave” action to play a more important become larger. It could be due to the reason that larger role during the exploration. Also, it has been learnt that the K errors require more correcting force to adjust, and there value is a notable parameter for the adjustment of the normal are chances that the ﬁnger is unable to correct such errors force during the exploration, because a larger K will increase quickly enough with constrained power. the movement of the ﬁnger, which permits the force control 2. Objects with more complex geometries tend to have to become more dramatic. Thus, a variable for the K has larger MSE and σ , such as the statue and the bulb. been introduced: K = α ∗ K , to change its scale. For differ- Objects with changing stiffness also have relatively ent devices, the α needs to be calibrated to achieve the best higher MSE and σ , like the human ﬁnger. The rea- k f performance. For instance, for objects with larger curvature son could be that the complex geometries and changing changes, a larger α can help to adjust the normal force more stiffness of the object surface will induce dramatic force quickly. In this study, for the validation experiments, α was changes during the haptic exploration, and the force set to 1 for all the objects, the adaptive selection of α will errors can not be eliminated completely and quickly with be developed in future work. Algorithm 3 summarizes the the control algorithm, as a consequence, the error accu- adaptive control algorithm. mulates and the MSE and SD become larger than the situation when exploring the smoother objects 3. When the target force is getting larger, the hand holding Algorithm 3 Adaptive control algorithm the objects is more unstable. The stronger handshaking 1: Deﬁne Δf as the magnitude of changing of the membership func- induces more instabilities to the system, especially when tions, Δf = 0.1 the force is 0.6 N or larger, making the noise (σ ) larger. 2: After one exploration, get μ ← mean(V ), σ ← SD(V ) and f f f f Also, for objects with complex geometries, this effect is MSE 3: while abs(F − μ )>ϑ and MSE >ϑ and σ <ϑ do d f 1 e f σ greater, while for ﬂat plates, this effect is smaller. 4: Increase the μ of “Close” membership functions, which close 4. In Fig. 9, the order of the plates was made by looking means the output distributions of “Close” move rightwards: at the surface texture and patterns rather than the real μ ← μ + Δf close close 5: Action: Haptic exploration again with new FCM roughness and surface pattern complexity level. For plate 6: Get new μ ,σ and MSE f f 1, even though it has the smoothest surface by vision, due 7: while abs(F − μ )<ϑ and MSE >ϑ and σ <ϑ do d f 2 e f σ to the material property of the ﬁngertip, during the explo- 8: Decrease the μ of “Close” membership functions, which close ration, the contact of the ﬁngertip and surface of the plate means the output distributions of “Close” move leftwards: μ ← close can generate viscous friction that makes the control of μ − Δf close the force harder, especially when the force is greater than 9: Action: Haptic exploration again with new FCM 10: Get new μ ,σ and MSE f f 0.4 N. The 15th plate (no. 25) has large and deep concave 11: return 0 Return 0 if ﬁnished successfully areas, and plate13 (no. 23) has large and high convex areas, these patterns will induce sudden force changes during the exploration, making the MSE and σ larger. For each experimental setup, 9 times of exploration were 5. For soft objects that can deform with force applied, when carried out on random areas (3 times with the arm at 0, 30, the force is large, the deformation of the object makes the and 60 degrees) of the objects with random S values cho- contact location estimation algorithm become inaccurate, sen from 0.35 mm, 0.45 mm, and 0.55 mm. 5 target forces thus, forces larger than 0.7 N were not used in the exper- have been set, and the total number of surface explorations iments, especially for the yellow soft sponge and soft was 1125 in the force control experiments. After each explo- black box. The metal spring has gaps between threads ration, the mean value μ and σ of the force vector were f f that can generate frequent drops of force values during calculated, and the MSE was obtained. Also, the algorithm the exploration, making the control harder. adjusted its fuzzy control method parameters to achieve an 6. When the target force is small (like 0.2 N and 0.4 N), the overall optimised performance. After 9 times of exploration, MSE and σ of different objects are similar for the same the average of the σ and MSE were obtained. target force, however, when the force is larger, the differ- ences of the MSE and σ of different objects get larger. 4.3 Contact finger exploration results and discussion Results of the black rubber, plate1 (no. 11), statue, plate15 (no. 25), human ﬁnger and yellow soft sponge represent- After all the haptic surface exploration experiments have ing different types of objects were used to plot ﬁgures been carried out, the overall results of the force control were (shown in Figs. 12, 13). 123 Autonomous Robots (2020) 44:773–790 785 Target Force - SD 0.13 Sponge 0.12 Statue 0.11 Plate1 0.1 Plate15 Finger 0.09 Soft Sponge 0.08 0.07 0.06 0.05 0.04 0.03 0.2 0.3 0.4 0.5 0.6 0.7 Target Force (N) Fig. 12 The SD results for different objects with different target forces. Fig. 14 The points in the ﬁgure are generated with the SD value, MSE The black sponge and plate1 are objects with ﬂat and ﬁne surface that value and the object number as the x, y and z coordinate. It can be are easier for the force control, and their SDs are smaller than other seen that the resultant control performance of the haptic exploration on objects. The 3D printed statue is the object with the most complex different objects are similar within a small range shown by a transparent geometry, and plate15 has large concave areas, these features all hinder cylinder, which reveals the robustness of the proposed algorithm the control performance, and the SDs for these two objects are larger. The human ﬁnger has different stiffness and changing geometry in dif- ferent locations, thereby the SD of which is the largest. The SD of the simplifying the software implementation. Despite these lim- yellow soft sponge is small at the beginning but gets larger when the itations, it can be seen that the average MSE is 0.006 and σ force is increasing due to the fact that when the force increases, the is 0.07 N. Also, from Fig. 14, it can be seen that the MSE and ﬁnger penetrates the soft layer of the sponge and touches the plastic surface underneath (Color ﬁgure online) σ for all objects are constrained within a controllable range, which can show the robustness of the control algorithm for -3 Target Force - MSE different objects with various properties and dynamic move- Sponge ment. Moreover, videos of the real-time experiments show Statue Plate1 that the platform-a works well with subtle motions following Plate15 the surface of the dynamically moving object with complex Finger Soft Sponge outline, when the target force is small. 4.4 Haptic exploration with robot arm UR-3 Since the workspace of the ﬁnger is limited with an average 0.2 0.3 0.4 0.5 0.6 0.7 travelling length of 60 mm, the robot arm UR-3 was used to Target Force (N) verify the control method with larger workspace and force ranges. A contacting sensing tip was mounted onto the robot Fig. 13 The MSE results for different objects with different target arm end-effector, forming the platform-c. The study of con- forces. The black sponge and plate1 have smaller MSE. The 3D printed statue and plate15 with more complex geometries and patterns have trolling the robot arm and ﬁnger simultaneously to realize larger MSE. The human ﬁnger has the largest MSE because of its chang- a synergic control is also envisioned, which can be useful ing stiffness and geometry. MSE of the yellow soft sponge is changing for more complex haptic exploration. The robot arm can due to the same reason explained in Fig. 12 (Color ﬁgure online) reach the object at almost any point on the surface, and the exploration is not constrained in any speciﬁc plane, but for From the results, the limitations of the platform-a and con- one-time exploration, the controlled movement is within two trol system have been learnt. First, the force it can generate axes. The robot arm can also realize a sequence of move- is limited, making it difﬁcult to correct the error quickly ments, covering more surfaces of the object. Figure 15 (left) enough, especially when the target force is more massive. shows the designed experimental setup. The Nano 17 sensor This can be improved by using more powerful motors, but was mounted on the robot arm with a 3D printed connection at the same time, the size of the platform will vary since part and covered with a hemisphere cap made of silicone powerful motors usually require more space. Second, the rubber (Ecoﬂex-50, Shore A 50). The stiffness of the rubber high-level control is working at 100 Hz currently, which may cap is large enough that there is no hysteresis effect during not be fast enough to adapt to quick force changes induced the experiments. A device that can generate a random dis- by complex geometry or stiffness changes. The relative low placement within 5–12 mm in the vertical direction was used working frequency is due to the time consumption of the soft- to add movement to the holding object. The corresponding ware modules like the angle calculation and the fuzzy control kinematic and fuzzy control parameters of the control algo- method. This can be improved by using higher spec PCs or rithm were modiﬁed for controlling the robot arm platform MSE SD (N) 786 Autonomous Robots (2020) 44:773–790 Fig. 15 Left: The haptic exploration platform-c using the UR3 robot arm and a contact sensing tip. The object is mounted on a moveable device that can move 5–12 mm vertically. Right: The schematic draw- ing of how the necessary angles have been calculated for deciding the next moving location of the tip. Same with Fig. 4, this only shows the situation where the correcting force F has the same direction of the real normal force Fig. 16 The objects used in haptic exploration experiments with the robot arm platform-c. From left to right: (1) metal bowl, (2) sphere like (platform-c). Similarly, the required angles can be calculated hat made by silicon, (3) plastic bottle ﬁlled with water (4) ﬂat sponge, with the equations below: (5) sponge with curved shape, the explore trajectory is drew with green colour (Color ﬁgure online) θ = 2 ∗ π − θ − θ + 1.5π − θ + 2π − θ (9) E j2 j3 j4 j5 θ = atan2(y , z ) (10) achieve accurately is no smaller than 0.8 mm, and the force it C e e generates is much larger than the ﬁnger per movement. These θ = θ + θ (11) N E C all present that the robot arm is suitable for larger force con- β = atan2(F , F ) (12) T N trol scenarios. α = β + θ (13) 5 objects shown in Fig. 16 were used to evaluate the perfor- mance of the control algorithm. There are four soft objects, And the next position to reach is: all can deform when force applied, and 3 of them have com- plex outlines. There is one metal object with a curved shape. P = P + S ·[cosα, sinα] (14) next now v For different objects, different target forces were set, this is because with too much force applied, the deformation of the where θ ,θ ,θ ,θ are the current joint angles of the soft object will cause inaccurate sensor readings, while for j2 j3 j4 j5 robot, which can be obtained through the driver in ROS. the solid object, the force changed by the smallest displace- With the proposed control algorithm, the robot arm was ment is more than 1 N. Thus, the target forces were set to 1 N, controlled to explore the surface of the object while keeping 1.25 N and 1.5 N for the soft objects, and 1.5 N, 2 N, 2.5 N the normal force as desired. Since the robot arm has more for the metal object. The S size was chosen from 0.8 mm, DoFs and can work in larger target force range (more power- 1 mm and 1.5 mm. For each object, 10 times of exploration ful), it can explore the object in a much wider workspace with with the adaptive control method were carried out. The video feedbacks from the contact sensing tip, which helps to realize of “surface_exploration_ur3-1 (MOESM7).mp4” shows the larger area’s haptic exploration and make more exploratory haptic surface exploration on the sponge with curved shape. procedures (EPs) available (Lederman and Klatzky 1993). The objects were mounted on the moveable platform with More EPs will allow the haptic perception system to obtain 5–12 mm random vertical displacement. more information about the object, which will be beneﬁcial Table 5 shows the results of the haptic exploration exper- in future research on the active haptic control and perception. iments using the platform-c. From the results, it can be seen Here, the haptic surface exploration is the main focus, and that the average MSE and σ are larger than those obtained more EPs will be implemented in the next study. with the contact sensing ﬁnger (platform-a) due to larger However, the control of the robot arm has limitations due target forces. As mentioned before, when the target force to the slow frequency of the software. The control loop is gets larger, the MSE and σ also increase. It has been learnt limited to 10 Hz in this study, which may hinder the response that, in some situations, even though the algorithm output speed when adjusting the force or velocity. Moreover, com- the largest correcting movement, it was still not enough to pared to the contact sensing ﬁnger, the smallest S it can correct the force in time. Thus, it becomes necessary for the 123 Autonomous Robots (2020) 44:773–790 787 Table 5 Force control results Resultant normal force standard deviation (N) and MSE with robot arm Object no. Target normal Target normal Target normal force-1 N/1.5 N force-1.25 N/2 N force-1.5 N/2.5 N 1 0.21 0.03 0.31 0.035 0.38 0.042 2 0.15 0.013 0.17 0.019 0.19 0.024 3 0.14 0.016 0.18 0.02 0.21 0.027 4 0.13 0.02 0.15 0.02 0.16 0.03 5 0.12 0.013 0.1 0.015 0.18 0.025 Avg. 0.15 0.0184 0.182 0.0218 0.224 0.0296 Table 6 Velocity control results Open loop resultant explore velocity mean and SD PID control resultant explore velocity mean and SD S value (mm) F -0.2 N F -0.4 N F -0.6 N v (mm/s) F -0.2 N F -0.4 N F -0.6 N Δv v d d d d d d d 0.15 1.92 2.11 1.86 2.31 1.78 2.18 2 1.78 1.41 1.75 1.54 1.81 1.45 0.22 0.25 3.16 2.59 3.16 2.68 2.95 2.5 3 2.69 1.73 2.65 1.78 2.72 1.67 0.31 0.35 3.72 2.68 3.8 2.8 3.68 2.64 4 3.62 1.78 3.58 1.86 3.65 1.76 0.38 0.45 4.69 2.86 4.55 2.85 4.67 2.84 5 4.56 1.91 4.51 1.9 4.59 1.89 0.45 0.55 5.52 3.02 5.55 3.12 5.3 2.9 6 5.51 2.01 5.46 2.08 5.54 1.93 0.50 0.65 6.99 3.15 6.89 3.2 6.43 2.96 7 6.48 2.1 6.42 2.13 6.4 2.1 0.57 0.75 7.53 3.24 7.22 3.34 7.19 2.9 8 7.43 2.22 7.37 2.43 7.46 2.23 0.58 0.85 8.38 3.32 8.02 3.54 8.23 3.09 9 8.35 2.41 8.41 2.56 8.41 2.56 0.61 0.95 9.03 3.5 9.04 3.66 8.95 3.41 10 9.26 2.8 9.27 2.84 9.34 2.75 0.71 algorithm to know the relationship of the indentation and the PID control method was applied to adjust the S value force changes for different objects, through which way, the during the exploration to check the change of σ . The target more accurate S can be decided for adjusting the force error. velocity for the PID control was set from 2 to 10 mm/s. The As the next step, the robot arm can be controlled to push the mean velocity was calculated by v = L /T , where L is m e e e object ﬁrst to characterize the indentation and force changing the total explored distance and T is the overall time used relation of the object, and then the best S can be derived. for the exploration process. For each object, 3 times explo- However, the overall result of this experimental validation is ration were carried out for each ﬁxed S value and 3 times stratiﬁed, conﬁrming the robustness of the control algorithm. exploration for each target velocity using the PID control. Moreover, as can be seen from videos of the real experiment Table 6 shows the results, and the values are averages of (refer to videos of ”dynamic_object_exploration_ur3 the three objects. It can be seen from the results, when S is (MOESM2), dynamic_object_exploration_ur3-with_back ﬁxed, the mean explore velocities are similar regardless of ground (MOESM3).mp4”), the arm can adjust the position the target force. In addition, for different objects, the veloc- quickly following the movement of the object. ity results are also similar. For example, when S is 0.45 mm and F is 0.4 N, the mean velocities for the three objects are 4.63 mm/s, 4.59 mm/s and 4.43 mm/s respectively. There- 4.5 Velocity control results fore, with different S values, the overall exploration velocity can be controlled during each haptic exploration on various To verify the performance of the velocity control method, objects, as shown in Fig. 17. This is because when S is used haptic exploration experiments with changing S values for deciding the next moving location, the increment along using the contact sensing ﬁnger (platform-a) were carried the tangential direction of the contact surface on the object out. The target force was set to 0.2 N, 0.4 N and 0.6 N in decides the moving velocity when the used time for every experiments, and the resultant mean value μ and σ were v v loop is the same (almost). The videos of “velocity_control-01 obtained with different S values. Three objects: the black (MOESM8), velocity_control-02 (MOESM9).mp4” show sponge, the light bulb, and plate-15 (no. 25) were selected. the examples of the open-loop velocity control. However, First, a ﬁxed S value was used during each haptic explo- the σ is relatively high with an average value of 2.94 mm/s ration to check velocity changes and compute the σ , then 123 788 Autonomous Robots (2020) 44:773–790 Sv - Mean velocity in two ways: (1) Increase the frequency of the overall con- Target force-0.2N trol loop, which can help to update the error more rapidly Target force-0.4N 8 Target force-0.6N and the platform can be controlled to take correction actions 7 more quickly to eliminate the error. In this way, the overall control quality could be improved, especially when there are sudden surface curvature changes on the object. (2) Modify the mechanical design of the ﬁnger joints to allow smaller S value for deciding the next location. As shown in Table 6, a smaller S enables the system to respond to larger and quicker changes more smoothly. This works when the ﬁn- 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sv (mm) ger is controlled with very slow velocities and small target force. However, in this study, the position control of the joints Fig. 17 The ﬁgure is showing the changing of the mean velocity of one becomes inaccurate when S is less than 0.1 mm due to the exploration with relation to the S changes using the open-loop control. v The velocity is obtained by getting the average of all sample data. It can mechanical limitation of the motors and joints. be seen that as S increases, the velocity increases and the relation of these two values is close to a linear correlation regardless of the control target force. These curves show that S can be used to control the overall 5 Conclusion velocity of the exploration procedure As the conclusion, with the proposed control method, the contact normal force and explore velocity during haptic sur- face exploration on objects with dynamic movement can be controlled adaptively, regardless of their surface and geom- etry properties. There are three points worthy of further research to improve the performance of the control method. First, during the exploration, if the force change generated by the movement of the device in the normal force direc- tion is not enough to correct the error, the adjustment will be delayed; and if the force change is too large, it will generate overshoots, inducing more ﬂuctuations to the system. Thus, understanding the indentation and force changes relation is Fig. 18 The points in this ﬁgure are generated with the σ , Δv and essential to make the control more precise. For unknown v as the x, y and z coordinate. Δv is obtained by abs(v − v ).The d d m objects, this relation can be obtained ﬁrst as a pre-information curves on the left side are representing the relation between SD and v and fed back to the system to adjust the related parameters. at different F , and curves on the right side are representing the relation between Δv and v at different F . It can be seen from the ﬁgure, the d d Also, this can help us understand the stiffness of the object, points are constrained in a cylinder with a larger radius compared to presenting more information about the contacting object. the force control results, showing the relative larger SD of the velocity Second, it is important to understand the best control force control result. However, it also shows that the control method can be range of each device, since the force ranges generated by dif- used to control the explore velocity with low absolute errors ferent devices are varied. By selecting or designing different devices or platforms, different forces can be controlled pre- when S is ﬁxed. When using the PID controller, the σ of the cisely. Third, because the S ranges, what each device can v v v velocity is around 2/3 of that when S is ﬁxed. And an average achieve are different, the control resolution (force change per absolute error Δv of 0.48 mm/s has been achieved, showing movement) is also varied for each device. To understand the the good performance of the feedback velocity control. resolution of the force control for each speciﬁc device is also With the feedback control, the performance of explore important to achieve more robust and precise control. velocity control is not excellent as the σ is still not low As the extensions of this work, we will focus on the active enough with an average value of 2.07 mm/s. As shown in haptic exploration, perception and soft contact. This pro- Fig. 18,the σ and Δv for each v are constrained in a trans- posed control method can be applied to any situations where v d parent cylinder that is larger than the one in Fig. 14. It could contact happens, and the contact force is critical to the con- be caused by two main reasons. The ﬁrst is the unstructured tacting object. Also, if the normal force and velocity can surface of the object, which adds many difﬁculties to the be controlled, it is helpful to implement different manipu- control process, and the other is the limited frequency and lation tasks that require different contact forces and explore relative long response time of the control software. Consid- velocities. This will lead to an active exploration method that ering the limitations, the velocity control can be improved can change exploration strategies (different force or velocity) Velocity (mm/s) Autonomous Robots (2020) 44:773–790 789 actively depending on the utility function for more efﬁcient ﬁngered robotic hand. In 2017 IEEE international symposium on robotics and intelligent sensors (IRIS) (pp. 311–317). object recognition or manipulation tasks. In addition, we will da Fonseca, V. P., Kucherhan, D. J., de Oliveira, T. E. A., Zhi, D., & test ﬁngertips made of softer material that can deform during Petriu, E. M. (2017a). Fuzzy controlled object manipulation using contact, which will be helpful when contacting soft objects a three-ﬁngered robotic hand. In 2017 Annual IEEE international like tissues inside the organs or fragile daily objects. More- systems conference (SysCon) (pp. 1–6). de Silva, C. W. (1995). Intelligent control: fuzzy logic applications. over, more ﬁngers will be created to form a manipulator, https://academic.microsoft.com/paper/598128971 combing the robot arm, the whole platform can be used to Fishel, J. A., & Loeb, G. E. (2012). Bayesian exploration for intelligent carry out dexterous manipulations with controllable force identiﬁcation of textures. Frontiers in Neurorobotics, 6,4. and velocity. As reported in Okamura and Cutkosky (2001), Hellman, R. B., Tekin, C., van der Schaar, M., & Santos, V. J. (2018). Functional contour-following via haptic perception and reinforce- with controllable haptic explorations, dexterous manipula- ment learning. IEEE Transactions on Haptics, 11, 61–72. tion can be realised with grasping, holding and other hand Jamil, M. H., Annor, P. S., Sharfman, J., Parthesius, R., Garachon, I., & controls. Finally, by combing the haptic feedback with the Eid, M. (2018). The role of haptics in digital archaeology and her- surgery devices, it will be beneﬁcial in robotic surgery to itage recording processes—IEEE conference publication. In IEEE international symposium on haptic, audio and visual environments help the surgeon to monitor and control the amount of force and games (HAVE). applied to minimise the risk of tissue damage (Amirabdol- Kappassov, Z., Corrales, J. A., & Perdereau, V. (2015). Tactile sens- lahian et al. 2017). ing in dexterous robot hands—Review. Robotics and Autonomous Systems, 74, 195–220. Acknowledgements This work is supported by the EPSRC (No. Konstantinova, J., Li, M., Mehra, G., Dasgupta, P., Althoefer, K., & EP/N020421/1) and NSFC (No. 51520105006). And we would like Nanayakkara, T. (2014). Behavioral characteristics of manual pal- to thank Dr. Hak Keung Lam for his kind suggestions on fuzzy control pation to localize hard nodules in soft tissues. IEEE Transactions method. on Biomedical Engineering, 61, 1651–1659. Lederman, S. J., & Klatzky, R. L. (1993). Extracting object properties Open Access This article is licensed under a Creative Commons through haptic exploration. Acta Psychologica, 84, 29–40. Attribution 4.0 International License, which permits use, sharing, adap- Lederman, S. J., & Klatzky, R. L. (2009). Haptic perception: A tutorial. tation, distribution and reproduction in any medium or format, as Attention, Perception and Psychophysics, 71, 1439–1459. long as you give appropriate credit to the original author(s) and the Lepora, N. F., Aquilina, K., & Cramphorn, L. (2017). Exploratory tactile source, provide a link to the Creative Commons licence, and indi- servoing with active touch. IEEE Robotics and Automation Letters, cate if changes were made. The images or other third party material 2, 1156–1163. in this article are included in the article’s Creative Commons licence, Lepora, N. F., Church, A., de Kerckhove, C., Hadsell, R., & Lloyd, J. unless indicated otherwise in a credit line to the material. If material (2019). From pixels to percepts: Highly robust edge perception and is not included in the article’s Creative Commons licence and your contour following using deep learning and an optical biomimetic intended use is not permitted by statutory regulation or exceeds the tactile sensor. IEEE Robotics and Automation Letters, 4, 2101– permitted use, you will need to obtain permission directly from the copy- 2107. right holder. To view a copy of this licence, visit http://creativecomm Li, Q., Schürmann, C., Haschke, R., & Ritter, H. J. (2013). A control ons.org/licenses/by/4.0/. framework for tactile servoing. In Robotics: Science and systems 2013,Vol.9, https://academic.microsoft.com/paper/102778925. Liu, H., Song, X., Bimbo, J., Seneviratne, L., & Althoefer, K. (2012). Surface material recognition through haptic exploration using an intelligent contact sensing ﬁnger. In 2012 IEEE/RSJ international References conference on intelligent robots and systems. Liu, H., Nguyen, K. C., Perdereau, V., Bimbo, J., Back, J., Godden, Amirabdollahian, F., Livatino, S., Vahedi, B., Gudipati, R., Sheen, P., M., et al. (2015). Finger contact sensing and the application in Gawrie-Mohan, S., et al. (2017). Prevalence of haptic feedback in dexterous hand manipulation. Autonomous Robots, 39, 25–41. robot-mediated surgery: A systematic review of literature. Journal Okamura, A. M. (2000). Haptic exploration of objects with rolling and of Robotic Surgery, 12, 11–25. sliding. Ph.D. thesis, Stanford University, Department of Mechan- Aviles, A. I., Alsaleh, S. M., Montseny, E., Sobrevilla, P., & Casals, A. ical Engineering. (2016). A deep-neuro-fuzzy approach for estimating the interac- Okamura, A. M., & Cutkosky, M. R. (2001). Feature detection for hap- tion forces in robotic surgery. In IEEE international conference on tic exploration with robotic ﬁngers. The International Journal of fuzzy systems (FUZZ-IEEE) (pp. 1113–1119). Robotics Research, 20, 925–938. Back, J., Bimbo, J., Noh, Y., Seneviratne, L., Althoefer, K., & Liu, Okamura, A. M., & Cutkosky, M. R. (2001). Feature detection for hap- H. (2015). Control a contact sensing ﬁnger for surface haptic tic exploration with robotic ﬁngers. The International Journal of exploration. In IEEE international conference on robotics and Robotics Research, 20, 925–938. automation (ICRA) (pp. 2736–2741). Rosales, C., Spinelli, F., Gabiccini, M., Zito, C., & Wyatt, J. L. Bicchi, A., Salisbury, J. K., & Brock, D. L. (1993). Contact sensing (2018). Gpatlasrrt: A local tactile exploration planner for recover- from force measurements. The International Journal of Robotics ing the shape of novel objects. International Journal of Humanoid Robotics, 15, 1850014. Research, 12, 249–262. Sommer, N., & Billard, A. (2016). Multi-contact haptic exploration and Ciotti, S., Sun, T., Battaglia, E., Bicchi, A., Liu, H., & Bianchi, M. grasping with tactile sensors. Robotics and Autonomous Systems, (2019). Soft tactile sensing: Retrieving force, torque and contact 85, 48–61. point information from deformable surfaces. In: 2019 IEEE inter- Song, X., Liu, H., Althoefer, K., Nanayakkara, T., & Seneviratne, L. D. national conference on robotics and automation, ICRA. (2014). Efﬁcient break-away friction ratio and slip prediction based da Fonseca, V. P., de Oliveira, T. E. A., Eyre, K., & Petriu, E. M. (2017b). Stable grasping and object reorientation with a three- 123 790 Autonomous Robots (2020) 44:773–790 Hongbin Liu is an Associate on haptic surface exploration. IEEE Transactions on Robotics, 30, Professor (Senior Lecturer) in the 203–219. school of biomedical engineering Sornkarn, N., & Nanayakkara, T. (2017). Can a soft robotic probe use and imaging sciences, King’s Col- stiffness control like a human ﬁnger to improve efﬁcacy of haptic lege London (KCL) where he is perception? IEEE Transactions on Haptics, 10, 183–195. directing the Haptic Mechatron- Su, Z., Fishel, J. A., Yamamoto, T., & Loeb, G. E. (2012). Use of tactile feedback to control exploratory movements to characterize object ics and Medical Robotics Labora- compliance. Frontiers in Neurorobotics, 6,7. tory (HaMMeR). His lab designs Takagi, T., & Sugeno, M. (1993). Fuzzy identiﬁcation of systems and robots with advanced haptic per- its applications to modeling and control. In Readings in fuzzy sets ception and interaction capabili- ties to address unmet needs in for intelligent systems (pp. 387–403). medicine, enabling safer and more Tanaka, K., & Wang, H. O. (2004). Fuzzy control systems design and effective diagnosis and treatment. analysis: A linear matrix inequality approach. New York: Wiley. Dr. Liu is a member of the IEEE, Tegin, J., & Wikander, J. (2005). Tactile sensing in intelligent robotic and a Technical Committee Mem- manipulation—A review. Industrial Robot: An International Jour- ber of IEEE EMBS BioRobotics. He has published over 100 peer- nal, 32, 64–70. reviewed publications at top international robotic journals and confer- Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Gian- ences and is inventor for 4 patents. His research has been funded by naccini, M. E., Rossiter, J., et al. (2018). The tactip family: Soft EPSRC, Innovate UK, NHS Trust and EU Commissions, with strong optical tactile sensors with 3d-printed biomimetic morphologies. industrial collaborations such as IBM, Ericsson. More information can Soft Robotics, 5, 216–227. be found on our online website: http://kclhammerlab.com/. Xu, D., Loeb, G. E., & Fishel, J. A. (2013). Tactile identiﬁcation of objects using Bayesian exploration. In IEEE international confer- ence on robotics and automation (ICRA). Yang, C., & Lepora, N. F. (2017) Object exploration using vision and active touch. In IEEE/RSJ international conference on intelligent robots and systems (IROS). Yang, C., Li, Z., & Burdet, E. (2013). Human like learning algorithm for simultaneous force control and haptic identiﬁcation. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp 710–715). Yin, J., Aspinall, P., Santos, V. J., & Posner, J. D. (2018). Measuring dynamic shear force and vibration with a bioinspired tactile sensor skin. IEEE Sensors Journal, 18, 3544–3553. Publisher’s Note Springer Nature remains neutral with regard to juris- dictional claims in published maps and institutional afﬁliations. Teng Sun received the B.Sc. degree in mechanical engineering and automation from Beijing Institute of Technology, Beijing, China, in 2011, and worked as an autonomous driving engineer in Nissan Advanced Technology Centre, Atsugi, Japan from 2011 to 2014. He is currently work- ing toward the Ph.D. degree in robotics in the Haptic Mechatron- ics and Medical Robotics (HaM- MeR) Laboratory, Centre for Robotics Research, Department of Informatics, King’s College Lon- don, London, UK. His current research interests include contact loca- tion estimation based on intrinsic contact sensing, object recognition based on haptic information through Bayesian classiﬁer, active percep- tion and decision making with information gain and utility method.

Journal

Autonomous Robots – Springer Journals

Published: May 21, 2020

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

Adaptive force and velocity control based on intrinsic contact sensing during surface exploration of dynamic objects

References (35)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies