Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Surface and Event Characterization - Proximal Audio Sensing to improve Manual and Robotic Device Interventions

Surface and Event Characterization - Proximal Audio Sensing to improve Manual and Robotic Device... DE GRUYTER Current Directions in Biomedical Engineering 2022;8(1): 1-4 Michael Friebe*, Axel Boese, Katarzyna Heryan, Moritz Spiller, Thomas Sühn, Nazila Esmaeili, Alfredo Illanes Surface and Event Characterization - Proximal Audio Sensing to improve Manual and Robotic Device Interventions Fig. 1 An audio sensor is attached to the proximal end of an interventional device (e.g., guidewire / catheter / robotic instruments / aspiration needle / biopsy device) outside the body and without the need for a dedicated device, wires, and with greatly reduced sterilization efforts. The acquisition system preprocesses and Ailters the signal that is then sent to the processing unit. The user interface depends on the application and can range from a visual „trafAic light“ feedback to an ampliAication and magniAication of speci Aic sounds. https://doi.org/10.1515/cdmbe-2022-0001 on initial attempts to use this technology with robotic arms for surface characterization and interventional vascular Abstract: Minimal-invasive procedures come with procedures that gain increased attention in combination with significant advantages for the patient. They also come with robotic devices. In summary, Proximal Audio Sensing could problems as the navigation/guidance of the devices to a target be a versatile, cost-effective and powerful tool to guide location is either based on pre-operatively acquired images minimally invasive needle interventions and enable (semi-) and then performed free-hand or is accompanied by intra- autonomous robot-assisted surgery. operative imaging such as MRI or CT that is expensive, complicated and produces artifacts. Using robotic systems for Keywords: audio sensing, proximal sensor, audio moving and guiding these interventional and therapeutic feature extraction, signal processing, device guidance devices adds additional issues like lack of palpation sensation and missing tissue feedback. While it is possible to add sensors to the distal tip, this creates other obstacles concerning reduced functionality, cables, sterility issues and 1. Introduction added complexity and cost. We propose to use a proximally attached audio sensor to record the tissue tool interaction and The device tip is often in an echogenic shadow when provide real-time feedback to the clinician. This paper reports using ultrasound as an external guidance device for interventional procedures. Computed Tomography produces reconstruction - and MRI has extensive susceptibility - ______ *Corresponding author: Prof. Michael Friebe, PhD: AGH artifacts. Integrating robotic systems into the clinical process University of Science and Technology, Department of - current examples use telemanipulated robots to move Measurement and Electronics, Krakow, Poland and Otto-von- diagnostic and therapeutic devices - adds issues like lack of Guericke University, Medical Faculty, INKA Innolab, Magdeburg, palpation sensation and missing tissue feedback. Moving Germany, friebe@agh.edu.pl towards semi-autonomous or even fully autonomous robot- Axel Boese, PhD: Otto-von-Guericke University, Medical Faculty , INKA Innolab, Magdeburg, Germany , assisted surgery will require additional sensory input that axel.boese@med.ovgu.de works in combination with predictive machine-learning- Katerzyan Heryan: AGH University of Science and based event segmentation and characterization tools. Technology, Department of Measurement and Electronics, Several sensor-based approaches have been proposed for Krakow, Poland tissue interaction assessment for haptic feedback in needle Moritz Spiller, Thomas Sühn, Nazila Esmaeili, Alfredo and guide wires [1,2]. The main drawback of these Illanes, PhD: SURAG Medical GmbH, Magdeburg, Germany Open Access. © 2022 The Author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 1 approaches is that the sensors are usually located at the distal integrates pre-processing, signal filtering and amplification, end of the instrument, i.e., invasively in the part of the device and the conversion to digital data. The information is then that is inserted inside the body. Moreover, due to this distal sent to a processing system performing advanced signal placement characteristic, they come with a degradation of the processing and feature extraction. device’s clinical efficiency due to the size of the sensors and The results can subsequently be displayed and/or the placement of required cables or wires. converted into other signals that are meaningful and valuable An alternative would be a clip-on device attached to the to the surgeon. This could be a visual signal indicating proximal end of an existing and technically not altered device certain audible events, like crossing a tissue layer, or it could that requires no or very little change to the current clinical be used to select and amplify the audio information in human workflow and setup. audible signals. The actual user interface and the data We have proposed in the past to use audio signals translation into human-perceptible information are still a created by the tissue tool interactions on needles and other work in progress. The setup on needles was tested in-vitro in devices and believe that this could be a viable approach to a laboratory environment with different layers of animal solve some of the mentioned drawbacks [3]. tissue and comparison of audio signals with Ultrasound video In this paper, we want to report on the initial attempts to information as well as correlation with a force measurement a d d t h i s t e c h n o l o g y t o r o b o t i c a r m s f o r s u r f a c e system. characterization [4] and for interventional vascular A d d i t i o n a l l y, w e p e r f o r m e d s e v e r a l h u n d r e d procedures that gain increased attention in combination with interventions using pig hearts and vessels with a guide wire robotic devices [5,6]. simulating a cardiac intervention to verify whether we could We will also discuss additional application opportunities detect relevant events (vessel touching, bumping, and of this technology and the following research steps that penetration) [5,6]. The proximally obtained audio w o u l d r e q u i r e i n t e r- d i s c i p l i n a r y a n d m u l t i - c e n t e r i n f o r m a t i o n w a s p r o c e s s e d u s i n g T i m e - Va r i a n t collaborative research approaches. Autoregressive Modelling (TV AR), and the results in 2D and 3D together with a calculation of the maximum pole energy (see Fig. 3). 2. Materials and Methods The technical principle is based on picking up a vibroacoustic signal generated through the movement of devices in human tissue (tissue-tool interaction) or of devices that are in contact with human tissue. The signal propagates over the device shaft and can be picked up on the proximal end using a dedicated clip-on microphone (see Fig. 1 and Fig. 4). Fig. 3 Audio signals for different events during a guide wire / catheter procedure with an audio sensor attached to the proximal end of the guide wire. All examined events produced distinctively different results that allow event clustering with high accuracy [6] Subsequently, we added the setup to a robotic arm of a Fig. 2 Initial audio receive setup using a 3D printed adapter for Da Vinci system (see Figure 4) to measure the audio needles (left) and for connection to the proximal end of a guide wire characteristics when moving the forceps over different (right). materials. During the experiments, the robotic instrument was moved with equal speed and pressure. Please note that the Initially, we used a 3D printed adaptor and a connection is indirect via the housing. conventional stethoscope attached to a needle or a guide wire [4] (see Fig. 2). In the meantime, this was replaced with a dedicated and lightweight MEMS clip-on microphone that 2 Fig. 4 Audio device attached to the control unit of the Forceps. The forceps wheels were Aixed to limit the tip movement (b), sensor attached using double sided tape (c). Fig. 5 The left graph shows the excellent correlation between extracted low-frequency-based vibrations in a needle insertion 3. Results experiment from using the audio signal and an ultrasound video image and on the right the ones for the high-frequency components Guidewire / Vascular Procedure also compared the low- and high-frequency components of The experiments for detecting events (vessel touching, the audio signal with a corresponding video analysis of an bumping, and penetration) during a guide wire/catheter ultrasound (see Fig. 6). procedure with an audio sensor attached to the proximal end We found excellent event correlation in both frequency of the guide wire have shown that all events produced bands (see Fig. 5). distinctively different signals that allow event clustering with In the meantime, we have created a device that can be high accuracy. The different events could be clustered with added to needle and endoscopic instruments integrating the high accuracy (98%) using various machine learning sensing and processing part (see Fig. 6 left). This setup was methods. used for kidney punctures and to classify relevant events using a Veress needle for creating the endoscopic procedure Robotic Grasper on Surfaces access. All with very encouraging results (publications are With a novel and advanced device setup, we analyzed work in progress). the characteristics of moving a grasper (robotic forceps) over different non-animal and animal tissues to see whether these 3. Discussion and Conclusion produce distinctive audio responses. We could show that each of the four different surfaces/tissues (Fig. 5) that we We also have conducted and completed several examined produces a distinct time/frequency response and, experiments to identify the effects of certain external therefore, could enable tissue differentiation. procedural variations that could occur. An example is shown in Figure 7, where we examined the impact of different Other Potential Applications insertion velocities of a needle and different users into the To verify the validity of the process and method for tissue. needle-based applications (e.g., biopsies, aspirations), we Fig. 6 Experimental setup including a synchronous US and audio data acquisition for manual needle insertions in porcine. Kidney placed in gelatine phantom — AGH, 2020. 3 Otto-von-Guericke-University, Magdeburg, Germany, trying to commercialize the technology presented. Ethical approval: The research did not involve any humans. Animal trials followed all the relevant national regulations, and institutional policies and were performed in accordance with the tenets of the Helsinki Declaration and were approved by the authors’ institutional review board or equivalent committee. Fig. 7 Study of needle insertions at different velocities. References [1] Trejos, A., Patel, R. & Naish, M. Force sensing and its Vibroacoustic events are only observable when the application in minimally invasive surgery and therapy: a device is in motion concerning the tissue. Audio provides survey. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering real-time information, but certain significant events (e.g., Science 224, 1435–1454 (2010). crossing a tissue layer) can only be presented after they [2] Ferrari, M., Werner, G. S., Bahrmann, P., Richartz, B. M. & happen. Figulla, H. R. Turbulent flow as a cause for underestimating This can, however, also be used to predict upcoming coronary flow reserve measured by doppler guide wire. adverse events. An example is the Veress needle procedure Cardiovascular ultrasound 4, 14 (2006). that must cross a known number of layers before it enters the [3] Illanes A, Boese A, Maldonado I, Pashazadeh A, Schaufler A, abdominal cavity. By adding Proximal Audio Sensing, the Navab N, and Friebe M (2018). Novel Clinical Device Tracking clinician can be alerted about the progress. Abdominal organ and Tissue Characterization Using Proximally Placed Audio Signal Acquisition and Processing. Scientific Reports, 2018. injuries could be prevented. DOI: 10.1038/ s41598-018-30641-0 Another application would be to guide interventional [4] Chen C, Sühn T, Kalmar M, Maldonado I, Wex C, Croner R, needles during vascular access or to provide information on Boese A, Friebe M and A. Illanes (2019). Texture harmful events during Cardiac Catheterization. If the differentiation using audio signal analysis with robotic clinician is informed about penetrating an artery, it is easily interventional instruments. Computers in biology and m e d i c possible to retract the device before hurting internal i n e ( 2 0 1 9 ) : 1 0 3 3 7 0 . d o i : 1 0 . 1 0 1 6 / j.compbiomed.2019.103370 structures. Our past research has demonstrated that it is possible to [5] Illanes A, Schauffler A, Maldonado I, Boese A, and Friebe M (2017). Time-varying Acoustic Emission Characterization for identify events during minimally invasive procedures and Guidewire Coronary Artery Perforation Identification. even for robot-assisted interventions. We also envision more Computing in Cardiology 44 (2017): 1. doi: 10.22489/ predictive and event characterization applications such as CinC.2017.135-113 automatic tissue identification after generating more data and [6] Mahmoodian N, Schaufler A, Pashazadeh A, Boese A, Friebe applying more machine-learning-based clustering and M, and Illanes A (2019). Proximal detection of guide wire characterization. perforation using feature extraction from bispectral audio signal analysis combined with machine learning. Computers in We further believe that Proximal Audio Sensing and biology and medicine 107 (2019): 10-17. doi: 10.1016/ subsequent signal processing have many more clinical j.compbiomed.2019.02.001 applications and believe that it should be added as an additional sense to minimally invasive procedures. Author Statement Research funding: The author state that there is no funding involved. Conflict of interest: NE, MS, and TS are Ph.D. students and employees at SURAG Medical with AI as the CEO. SURAG is a spin-off company of the INKA Innolab at the http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Current Directions in Biomedical Engineering de Gruyter

Surface and Event Characterization - Proximal Audio Sensing to improve Manual and Robotic Device Interventions

Loading next page...
 
/lp/de-gruyter/surface-and-event-characterization-proximal-audio-sensing-to-improve-liQ03fT96X

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
de Gruyter
Copyright
© 2022 by Walter de Gruyter Berlin/Boston
eISSN
2364-5504
DOI
10.1515/cdmbe-2022-0001
Publisher site
See Article on Publisher Site

Abstract

DE GRUYTER Current Directions in Biomedical Engineering 2022;8(1): 1-4 Michael Friebe*, Axel Boese, Katarzyna Heryan, Moritz Spiller, Thomas Sühn, Nazila Esmaeili, Alfredo Illanes Surface and Event Characterization - Proximal Audio Sensing to improve Manual and Robotic Device Interventions Fig. 1 An audio sensor is attached to the proximal end of an interventional device (e.g., guidewire / catheter / robotic instruments / aspiration needle / biopsy device) outside the body and without the need for a dedicated device, wires, and with greatly reduced sterilization efforts. The acquisition system preprocesses and Ailters the signal that is then sent to the processing unit. The user interface depends on the application and can range from a visual „trafAic light“ feedback to an ampliAication and magniAication of speci Aic sounds. https://doi.org/10.1515/cdmbe-2022-0001 on initial attempts to use this technology with robotic arms for surface characterization and interventional vascular Abstract: Minimal-invasive procedures come with procedures that gain increased attention in combination with significant advantages for the patient. They also come with robotic devices. In summary, Proximal Audio Sensing could problems as the navigation/guidance of the devices to a target be a versatile, cost-effective and powerful tool to guide location is either based on pre-operatively acquired images minimally invasive needle interventions and enable (semi-) and then performed free-hand or is accompanied by intra- autonomous robot-assisted surgery. operative imaging such as MRI or CT that is expensive, complicated and produces artifacts. Using robotic systems for Keywords: audio sensing, proximal sensor, audio moving and guiding these interventional and therapeutic feature extraction, signal processing, device guidance devices adds additional issues like lack of palpation sensation and missing tissue feedback. While it is possible to add sensors to the distal tip, this creates other obstacles concerning reduced functionality, cables, sterility issues and 1. Introduction added complexity and cost. We propose to use a proximally attached audio sensor to record the tissue tool interaction and The device tip is often in an echogenic shadow when provide real-time feedback to the clinician. This paper reports using ultrasound as an external guidance device for interventional procedures. Computed Tomography produces reconstruction - and MRI has extensive susceptibility - ______ *Corresponding author: Prof. Michael Friebe, PhD: AGH artifacts. Integrating robotic systems into the clinical process University of Science and Technology, Department of - current examples use telemanipulated robots to move Measurement and Electronics, Krakow, Poland and Otto-von- diagnostic and therapeutic devices - adds issues like lack of Guericke University, Medical Faculty, INKA Innolab, Magdeburg, palpation sensation and missing tissue feedback. Moving Germany, friebe@agh.edu.pl towards semi-autonomous or even fully autonomous robot- Axel Boese, PhD: Otto-von-Guericke University, Medical Faculty , INKA Innolab, Magdeburg, Germany , assisted surgery will require additional sensory input that axel.boese@med.ovgu.de works in combination with predictive machine-learning- Katerzyan Heryan: AGH University of Science and based event segmentation and characterization tools. Technology, Department of Measurement and Electronics, Several sensor-based approaches have been proposed for Krakow, Poland tissue interaction assessment for haptic feedback in needle Moritz Spiller, Thomas Sühn, Nazila Esmaeili, Alfredo and guide wires [1,2]. The main drawback of these Illanes, PhD: SURAG Medical GmbH, Magdeburg, Germany Open Access. © 2022 The Author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 1 approaches is that the sensors are usually located at the distal integrates pre-processing, signal filtering and amplification, end of the instrument, i.e., invasively in the part of the device and the conversion to digital data. The information is then that is inserted inside the body. Moreover, due to this distal sent to a processing system performing advanced signal placement characteristic, they come with a degradation of the processing and feature extraction. device’s clinical efficiency due to the size of the sensors and The results can subsequently be displayed and/or the placement of required cables or wires. converted into other signals that are meaningful and valuable An alternative would be a clip-on device attached to the to the surgeon. This could be a visual signal indicating proximal end of an existing and technically not altered device certain audible events, like crossing a tissue layer, or it could that requires no or very little change to the current clinical be used to select and amplify the audio information in human workflow and setup. audible signals. The actual user interface and the data We have proposed in the past to use audio signals translation into human-perceptible information are still a created by the tissue tool interactions on needles and other work in progress. The setup on needles was tested in-vitro in devices and believe that this could be a viable approach to a laboratory environment with different layers of animal solve some of the mentioned drawbacks [3]. tissue and comparison of audio signals with Ultrasound video In this paper, we want to report on the initial attempts to information as well as correlation with a force measurement a d d t h i s t e c h n o l o g y t o r o b o t i c a r m s f o r s u r f a c e system. characterization [4] and for interventional vascular A d d i t i o n a l l y, w e p e r f o r m e d s e v e r a l h u n d r e d procedures that gain increased attention in combination with interventions using pig hearts and vessels with a guide wire robotic devices [5,6]. simulating a cardiac intervention to verify whether we could We will also discuss additional application opportunities detect relevant events (vessel touching, bumping, and of this technology and the following research steps that penetration) [5,6]. The proximally obtained audio w o u l d r e q u i r e i n t e r- d i s c i p l i n a r y a n d m u l t i - c e n t e r i n f o r m a t i o n w a s p r o c e s s e d u s i n g T i m e - Va r i a n t collaborative research approaches. Autoregressive Modelling (TV AR), and the results in 2D and 3D together with a calculation of the maximum pole energy (see Fig. 3). 2. Materials and Methods The technical principle is based on picking up a vibroacoustic signal generated through the movement of devices in human tissue (tissue-tool interaction) or of devices that are in contact with human tissue. The signal propagates over the device shaft and can be picked up on the proximal end using a dedicated clip-on microphone (see Fig. 1 and Fig. 4). Fig. 3 Audio signals for different events during a guide wire / catheter procedure with an audio sensor attached to the proximal end of the guide wire. All examined events produced distinctively different results that allow event clustering with high accuracy [6] Subsequently, we added the setup to a robotic arm of a Fig. 2 Initial audio receive setup using a 3D printed adapter for Da Vinci system (see Figure 4) to measure the audio needles (left) and for connection to the proximal end of a guide wire characteristics when moving the forceps over different (right). materials. During the experiments, the robotic instrument was moved with equal speed and pressure. Please note that the Initially, we used a 3D printed adaptor and a connection is indirect via the housing. conventional stethoscope attached to a needle or a guide wire [4] (see Fig. 2). In the meantime, this was replaced with a dedicated and lightweight MEMS clip-on microphone that 2 Fig. 4 Audio device attached to the control unit of the Forceps. The forceps wheels were Aixed to limit the tip movement (b), sensor attached using double sided tape (c). Fig. 5 The left graph shows the excellent correlation between extracted low-frequency-based vibrations in a needle insertion 3. Results experiment from using the audio signal and an ultrasound video image and on the right the ones for the high-frequency components Guidewire / Vascular Procedure also compared the low- and high-frequency components of The experiments for detecting events (vessel touching, the audio signal with a corresponding video analysis of an bumping, and penetration) during a guide wire/catheter ultrasound (see Fig. 6). procedure with an audio sensor attached to the proximal end We found excellent event correlation in both frequency of the guide wire have shown that all events produced bands (see Fig. 5). distinctively different signals that allow event clustering with In the meantime, we have created a device that can be high accuracy. The different events could be clustered with added to needle and endoscopic instruments integrating the high accuracy (98%) using various machine learning sensing and processing part (see Fig. 6 left). This setup was methods. used for kidney punctures and to classify relevant events using a Veress needle for creating the endoscopic procedure Robotic Grasper on Surfaces access. All with very encouraging results (publications are With a novel and advanced device setup, we analyzed work in progress). the characteristics of moving a grasper (robotic forceps) over different non-animal and animal tissues to see whether these 3. Discussion and Conclusion produce distinctive audio responses. We could show that each of the four different surfaces/tissues (Fig. 5) that we We also have conducted and completed several examined produces a distinct time/frequency response and, experiments to identify the effects of certain external therefore, could enable tissue differentiation. procedural variations that could occur. An example is shown in Figure 7, where we examined the impact of different Other Potential Applications insertion velocities of a needle and different users into the To verify the validity of the process and method for tissue. needle-based applications (e.g., biopsies, aspirations), we Fig. 6 Experimental setup including a synchronous US and audio data acquisition for manual needle insertions in porcine. Kidney placed in gelatine phantom — AGH, 2020. 3 Otto-von-Guericke-University, Magdeburg, Germany, trying to commercialize the technology presented. Ethical approval: The research did not involve any humans. Animal trials followed all the relevant national regulations, and institutional policies and were performed in accordance with the tenets of the Helsinki Declaration and were approved by the authors’ institutional review board or equivalent committee. Fig. 7 Study of needle insertions at different velocities. References [1] Trejos, A., Patel, R. & Naish, M. Force sensing and its Vibroacoustic events are only observable when the application in minimally invasive surgery and therapy: a device is in motion concerning the tissue. Audio provides survey. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering real-time information, but certain significant events (e.g., Science 224, 1435–1454 (2010). crossing a tissue layer) can only be presented after they [2] Ferrari, M., Werner, G. S., Bahrmann, P., Richartz, B. M. & happen. Figulla, H. R. Turbulent flow as a cause for underestimating This can, however, also be used to predict upcoming coronary flow reserve measured by doppler guide wire. adverse events. An example is the Veress needle procedure Cardiovascular ultrasound 4, 14 (2006). that must cross a known number of layers before it enters the [3] Illanes A, Boese A, Maldonado I, Pashazadeh A, Schaufler A, abdominal cavity. By adding Proximal Audio Sensing, the Navab N, and Friebe M (2018). Novel Clinical Device Tracking clinician can be alerted about the progress. Abdominal organ and Tissue Characterization Using Proximally Placed Audio Signal Acquisition and Processing. Scientific Reports, 2018. injuries could be prevented. DOI: 10.1038/ s41598-018-30641-0 Another application would be to guide interventional [4] Chen C, Sühn T, Kalmar M, Maldonado I, Wex C, Croner R, needles during vascular access or to provide information on Boese A, Friebe M and A. Illanes (2019). Texture harmful events during Cardiac Catheterization. If the differentiation using audio signal analysis with robotic clinician is informed about penetrating an artery, it is easily interventional instruments. Computers in biology and m e d i c possible to retract the device before hurting internal i n e ( 2 0 1 9 ) : 1 0 3 3 7 0 . d o i : 1 0 . 1 0 1 6 / j.compbiomed.2019.103370 structures. Our past research has demonstrated that it is possible to [5] Illanes A, Schauffler A, Maldonado I, Boese A, and Friebe M (2017). Time-varying Acoustic Emission Characterization for identify events during minimally invasive procedures and Guidewire Coronary Artery Perforation Identification. even for robot-assisted interventions. We also envision more Computing in Cardiology 44 (2017): 1. doi: 10.22489/ predictive and event characterization applications such as CinC.2017.135-113 automatic tissue identification after generating more data and [6] Mahmoodian N, Schaufler A, Pashazadeh A, Boese A, Friebe applying more machine-learning-based clustering and M, and Illanes A (2019). Proximal detection of guide wire characterization. perforation using feature extraction from bispectral audio signal analysis combined with machine learning. Computers in We further believe that Proximal Audio Sensing and biology and medicine 107 (2019): 10-17. doi: 10.1016/ subsequent signal processing have many more clinical j.compbiomed.2019.02.001 applications and believe that it should be added as an additional sense to minimally invasive procedures. Author Statement Research funding: The author state that there is no funding involved. Conflict of interest: NE, MS, and TS are Ph.D. students and employees at SURAG Medical with AI as the CEO. SURAG is a spin-off company of the INKA Innolab at the

Journal

Current Directions in Biomedical Engineeringde Gruyter

Published: Jul 1, 2022

Keywords: audio sensing; proximal sensor; audio feature extraction; signal processing; device guidance

There are no references for this article.