A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

Tan-Nhu Nguyen; Marie-Christine Ho Ba Tho; Tien-Tuan Dao

doi:10.1155/2020/5039329

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

Nguyen, Tan-Nhu;Ho Ba Tho, Marie-Christine;Dao, Tien-Tuan 2020-02-20 00:00:00 Hindawi Applied Bionics and Biomechanics Volume 2020, Article ID 5039329, 30 pages https://doi.org/10.1155/2020/5039329 Review Article A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations Tan-Nhu Nguyen , Marie-Christine Ho Ba Tho , and Tien-Tuan Dao Sorbonne University, Université de Technologie de Compiègne, CNRS, UMR 7338 Biomechanics and Bioengineering, Centre de Recherche Royallieu, CS 60 319 Compiègne, France Correspondence should be addressed to Tien-Tuan Dao; tien-tuan.dao@utc.fr Received 12 September 2019; Revised 22 January 2020; Accepted 5 February 2020; Published 20 February 2020 Academic Editor: Jose Merodio Copyright © 2020 Tan-Nhu Nguyen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Simulating deformations of soft tissues is a complex engineering task, and it is even more diﬃcult when facing the constraint between computation speed and system accuracy. However, literature lacks of a holistic review of all necessary aspects (computational approaches, interaction devices, system architectures, and clinical validations) for developing an eﬀective system of soft-tissue simulations. This paper summarizes and analyses recent achievements of resolving these issues to estimate general trends and weakness for future developments. A systematic review process was conducted using the PRISMA protocol with three reliable scientiﬁc search engines (ScienceDirect, PubMed, and IEEE). Fifty-ﬁve relevant papers were ﬁnally selected and included into the review process, and a quality assessment procedure was also performed on them. The computational approaches were categorized into mesh, meshfree, and hybrid approaches. The interaction devices concerned about combination between virtual surgical instruments and force-feedback devices, 3D scanners, biomechanical sensors, human interface devices, 3D viewers, and 2D/3D optical cameras. System architectures were analysed based on the concepts of system execution schemes and system frameworks. In particular, system execution schemes included distribution-based, multithread-based, and multimodel-based executions. System frameworks are grouped into the input and output interaction frameworks, the graphic interaction frameworks, the modelling frameworks, and the hybrid frameworks. Clinical validation procedures are ordered as three levels: geometrical validation, model behavior validation, and user acceptability/safety validation. The present review paper provides useful information to characterize how real-time medical simulation systems with soft-tissue deformations have been developed. By clearly analysing advantages and drawbacks in each system development aspect, this review can be used as a reference guideline for developing systems of soft-tissue simulations. 1. Introduction ing to computation speed (or computation time) and system accuracy. Computation speed is the number of computing iterations that a soft-tissue simulation system can be exe- In a human body, tissues are commonly classiﬁed into hard and soft tissues. While hard tissues do not deform during cuted in one second on a speciﬁc hardware conﬁguration. It the motions of human bodies, soft tissues always deform is usually measured in frames per second (FPS) or Hertz when interacting with themselves, other tissues, and surgical (Hz). Computation time is a time duration needed to run tools. Modeling soft-tissue deformations in an entire organ data acquisition, data pre-/postprocessing, physical behavior simulation, and data visualization in a soft-tissue simulation or only in parts of an organ is still one of the most challenging issues in the biomedical engineering ﬁeld. In particular, eﬀec- system. Moreover, two types of accuracies were considered. tive integration of soft-tissue deformation behaviors into The ﬁrst one relates to model accuracy that quantiﬁes the medical simulation systems has faced two constraints relat- closeness of agreement between the simulated and the real 2 Applied Bionics and Biomechanics consumes large computation cost from a system. System behaviors of soft tissues. The second one deals with the sys- tem accuracy that was aﬀected by interaction device accu- architectures must also be developed to compromisingly racy, algorithm accuracy, and model accuracy. Interaction cooperate all system components such as soft-tissue models and input/output interaction devices. On this aspect, system device accuracy is the degree of closeness of the measured values of a physical quantity to its true values. Algorithm execution schemes and system frameworks should be care- accuracy quantiﬁes the correctness of an implemented com- fully selected to optimize system performance. Finally, once putational process in relation to the true process. Note that fully developed, the system must be validated through diﬀer- these accuracies should be within the clinically acceptable ent validation levels so that it can be used in a target clinical application. Those validation levels include geometrical accuracy bounds for each medical application. In fact, to real- istically simulate both geometric deformations and mechan- validation, model validation, system validation, and user ical behaviors of soft tissues within a medical simulation acceptability/safety validation. Generally speaking, to simu- system, computation speed must be in real time [1], and late soft-tissue deformations in real time while keeping an the system accuracy must be within a desired tolerance level acceptable realistic level of soft-tissue behaviors, all of the above aspects must be individually and systematically ana- according to each medical application. Note that real time is commonly deﬁned as a rate compatible with the graphic ani- lyzed and developed. mation rate of 30 frames per second (FPS) [2]. Moreover, real Although the issues of real-time soft tissue simulations time also includes the responding rate of force feedbacks were also reviewed, previous review studies rarely analyzed when soft tissues collide with other objects. This rate must how real-time challenges were solved eﬀectively in a whole system. In particular, all system development aspects should be between 100 Hz and 1000 Hz so that human tactile per- ceptions can feel collisions without interruptions [3]. It is be thoroughly reviewed to describe how both computation important to note that although real time is one of the most speed and system accuracy requirements were achieved. important requirements for clinical applications, most soft- However, the studies just focused on simulating speciﬁc types tissue simulation systems hardly satisﬁed both acceptable of soft tissues in medical applications, and they did not con- cern how eﬀectively the real-time constraint was solved. For model accuracy and real-time computation speed [4]. For instance, Murai et al. stated that the acquisition of internal example, in an interesting review paper proposed by Delin- somatosensory data in real time was crucial because the real gette a full description of realistic soft-tissue modeling in medical simulations was described [9]. However, it just time could be used in online diagnosis and assessment pro- cesses in surgical applications [5]. Ho et al. showed that the showed out three main problems when realistically simulat- ing soft tissue in medical simulation systems, but the visualization and computing of deformations in real time are essential in surgical simulation of soft tissues [6]. In the methods for solving those problems were not been analyzed. ﬁeld of image-guided surgeries, the estimation of soft-tissue Other than that, this review was conducted in the year 1998 when technologies were in an initial development stage, so deformations in real time is also one of the most important challenges [7]. Note that in image-guided surgery systems, numerous studies that eﬀectively solved the soft-tissue defor- mation issues have not been analyzed in this review study. computation time is commonly expensive due to online data acquisition from medical imaging and additional data pro- Sun et al. [1] also examined a relative diversity of tissue sim- cessing. In fact, most simulation systems with soft-tissue ulation procedures with the help of computer technologies. Although this study covered aspects in the tissue simulation deformations hardly satisfy real-time requirements [8], and they cannot both correctly compute soft-tissue deformations procedure (3D reconstructions, tissue classiﬁcations, and clinical applications), it did not focus on soft-tissue modeling and eﬀectively achieve real-time computation speeds [7]. However, despite this hard constraint, numerous strategies and just ﬁnished at describing general ideas of each aspect have been developed for improving both computation speed rather than analyzing advantages and disadvantages of methods/algorithms employed in each aspect. Moreover, and accuracy of soft-tissue simulation systems. Developing soft-tissue simulation systems is a complex this study was not answered how the challenge of achiev- engineering task composing of multiple aspects. Each of ing both real-time computation speeds and acceptable sys- them has its own contribution to the accuracy and speed of tem accuracy was solved. Mainly analyzing advantages and the target system. From system engineering point of view, disadvantages of modeling physical deformations, Nealen et al. [10] presented a full description about mathematical four important aspects of a real-time medical simulation sys- tem include computational approaches, interaction devices, functions, explanations of the physical meaning, and anal- system architectures, and clinical validations. Computational yses of computation results, but they mainly concerned approaches for modeling soft-tissue deformations are ﬁrst accuracies of each modeling method rather than the com- developed according to current requirements about compu- putation speed when employed in a speciﬁc simulation system. Up to now, with the abundant developments of tation speed and system accuracy. It is important to note that the computation speed and system accuracy are mainly software/hardware technologies and soft-tissue modeling aﬀected by the choice of appropriate computational methods, numerous studies have reasonably proposed approaches for estimating deformations of soft tissues inter- eﬀective solutions for both achieving real-time computa- acted with external input factors. Interaction devices are then tion speeds and acceptable system accuracy in simulation systems. However, they have not been summarized in a selected to interface between soft-tissue models and real physical environments. This interface needs to be both in real systematic way and analyzed completely to estimate general time and in an acceptable accuracy. This requirement often trends and weakness for future developments. Consequently, Applied Bionics and Biomechanics 3 this review study. They also participated into the quality to complement those gaps, this review paper is proposed to answer the following questions: assessment. Consensus discussion was done when necessary for solving disagreements. The number of included/excluded (1) How have computational approaches been developed articles is summarized in Table 2. Firstly, the duplicates were for both achieving real-time computation speed and checked with the duplication tool in the Mendeley software. keeping acceptable system accuracy? The number of duplicated papers at this stage was 1,610 for all search terms. Then, the general and speciﬁc eligibility cri- (2) Which interaction devices have been interfaced eﬀec- teria were applied to all unduplicated articles. The title inclu- tively in real-time soft-tissue simulation systems? sion criteria were ﬁrst used for ﬁltering out the irrelevant articles. The included articles at this phase were 973, which (3) How have system architectures been developed for were then enrolled to the abstract ﬁltering criteria for select- cooperating with computational approaches and ing the most pertinent articles. After reading all the abstracts, interaction devices in real time? 92 included articles were then read in full-text to select the (4) How have been the real-time soft-tissue simulation best qualitative and quantitative articles for systematic systems validated in clinical applications? review. Finally, the number of included articles was 55. Spe- ciﬁcally, the ﬂow chart of the selection procedure illustrating Moreover, real-time soft-tissue simulation systems pro- the number of included/excluded articles after each selection posed in literature were analysed sequentially and summarized stage is shown in Figure 1. To answer the identiﬁed research according to four system development aspects: computational questions, the selected 55 papers were categorized into four approaches, interaction devices, system architectures, and classes. The ﬁrst category concerns the computational clinical validations. Moreover, trends and gaps of each devel- approaches for modeling deformations of human soft tissues opment aspect were also presented. Recommendations for in real time. The second category relates to the disadvantages future researches were ﬁnally proposed. and advantages of interaction devices for getting the external data from soft tissues and visualizing the processed data. The 2. Materials and Methods third category deals with the characteristics of medical hard- ware/software systems consisting of graphic user interfaces A systematic review method was conducted using the (GUIs), programming languages, programming frameworks, PRISMA protocol [11] (Figure 1). Three scientiﬁc databases and other techniques for developing soft-tissue simulation were chosen: ScienceDirect, PubMed, and IEEE. In more systems. The ﬁnal category composes of system validations details, a focus on human soft tissues like upper/lower limb in clinical contexts and the analyses of user acceptability muscles, facial muscles, livers, and skins was done. A special and safety requirements of developed systems. Additionally, attention was also given on the contributions related to the each selected paper could also be grouped on multiple cate- improvement of computational methods and/or employing gories if their contents related to more than one category. eﬀective hardware/software system architectures for real- time medical simulation systems. Finally, other articles 2.2. Eligibility Criteria. The inclusion/exclusion criteria were focused on analyzing applications of real-time soft-tissue clearly deﬁned based on the meaning of each search termi- models for system validation, user acceptability and safety nology. The list of inclusion criteria for each search terminol- requirements were included. Note that in this present review, ogy is shown in Table 3. In addition, to keep the literature at a the method refers to the development strategy of mathemat- high academic level, only journal articles were considered for ical constitutive formulations of soft-tissue deformations the present review. Moreover, the articles in conferences with based on a speciﬁc computational approach. Reviewed a couple of pages are initially eliminated. Other kinds of low- studies relate to mesh-based and meshfree methods. An algo- quality written forms such as letters, judgements, and book rithm concerns the procedure to compute soft-tissue defor- chapters were also not selected. Other than that, the articles mations using speciﬁc modeling methods. A model refers that were not written in English were excluded from the to the mathematical representation of soft-tissue deforma- literature review. tions using mesh-based and meshfree-based methods. A set of search terminologies were deﬁned for the literature investi- 2.3. Quality Assessment. The quality assessment procedure gation, and then, each terminology was presented in a search was established to rate the quality of each analyzed paper. term by using AND/OR operators. The used search terminol- Eighteen yes-no assessment items were deﬁned and used. ogies and their appropriate search terms are listed in Table 1. Papers related to computational approaches bias were evalu- For the systematic information retrieval process, journal arti- ated using the following four items: (1) Was the method ade- cles published up to December 2017 were assessed. quately used/developed and described for the involved tissue 2.1. Selection Methodology. Selection was the most signiﬁcant behavior? (2) Was the veriﬁcation well-performed for the used/developed method? (3) Was the validation systemati- procedure for choosing both qualitatively and quantitatively appropriate articles for the systematic review. After identiﬁ- cally performed for the used/developed method? (4) Did cation from the search engines, retrieved articles were auto- the method really satisfy the real-time constraint? Papers matically saved to their suitable folders using the Mendeley related to interaction devices bias were evaluated using the paper management system. Two independent reviewers following four items: (5) Was the devices well selected for the system? (6) Was the device accuracy adequate for the (TNN and TTD) screened and selected relevant papers for 4 Applied Bionics and Biomechanics Number of searched records based on all search terms n = 361,756 ScienceDirect: n = 319,210 PubMed: n = 44,609 IEEE: n = 1,157 Duplicates n = 1,610 Number of records aer removing duplications n = 360,146 Records excluded from title conditions: IC #1.1: n = 5,465 IC #2.1: n = 13,542 IC #3.1: n = 335 IC #4.1: n = 1,543 IC #5.1: n = 42,209 IC #6.1: n = 4,431 IC #7.1: n = 135,265 IC #8.1: n = 147,136 IC #9.1: n = 9,247 Total excluded: n = 359,173 Records included based on title conditions n = 973 Records excluded from abstract conditions: IC #1.2: n = 128 IC #2.2: n = 37 IC #3.2: n = 169 IC #4.2: n = 171 IC #5.2: n = 83 IC #6.2: n = 125 IC #7.2: n = 76 IC #8.2: n = 45 IC #9.2: n = 47 Total excluded: n = 881 Records included based on abstract conditions n = 92 Records excluded from content conditions n = 37 Studies included for the systematic review n = 55 Figure 1: Workﬂow of the selection process using PRISMA protocol for the performed systematic review. real-time constraints? (7) Was the device easy enough to use system scalable? (12) Were the system frameworks ade- for a clinical routine practice? (8) Is the device price suitable quately selected for implementing the system of interest? for a clinical setting? Papers related to system architecture Papers related to clinical validation bias were evaluated using bias were evaluated using the following four items: (9) Was the following six items: (13) Was the study adequately vali- the system adequately described? (10) Was the system devel- dated with in vitro data? (14) Was the study adequately vali- oped with the participation of the end users? (11) Was the dated with in vivo data? (15) Was the study adequately Included Eligibility Screening Identiﬁcation Applied Bionics and Biomechanics 5 Table 1: The search terms used for the systematic review process. # Search terminologies (terms) Search terms (STs) ST #1: Real-time AND computer-aided AND medical AND 1 Term #1: Computer-aided medical simulations/systems (simulations OR systems) 2 Term #2: Real-time biomedical simulations/systems ST #2: Real-time AND biomedical AND (simulations OR systems) 3 Term #3: Real-time facial simulations ST #3: Real-time AND facial AND simulations 4 Term #4: Real-time liver deformation models ST #4: Real-time AND liver AND deformation AND models 5 Term #5: Real-time medical simulations/systems ST #5: Real-time AND medical AND (simulations OR systems) 6 Term #6: Real-time muscle deformation models ST #6: Real-time AND muscle AND deformation AND models 7 Term #7: Real-time surgery ST #7: Real-time AND surgery 8 Term #8: Real-time ﬁnite element methods ST #8: Real-time AND ﬁnite AND element AND methods 9 Term #9: Real-time soft-tissue deformations ST #9: Real-time AND soft AND tissue AND deformations computation frame rates were nearly 30 FPS. In addition, validated with patient data? (16) Was the level of validation suitable for translating the outcomes into clinical routine all interaction devices were all accurate enough for use in practices? (17) Was the user acceptability performed for clinical routines with acceptable prices, and they were also patients? (18) Was the user acceptability performed for clin- well selected for appropriate computational approaches and ical experts? system architectures. Moreover, the data transmission band- Note that the user acceptability validation is commonly widths of these selected devices were relatively much faster conducted after developing a full-simulation system. This than the computational and graphical rendering speeds, so validation targets at validating the acceptability level related they were all suitable for real-time applications. Over 50% to graphic system’s user interfaces, system’s ease-of-use, sys- of articles have implemented their developed computational tem’s functions, system’s robustness, etc., during short-term approaches into a simulation system. They also well and/or long-term evaluation campaigns for clinicians. described the architectures and frameworks of the imple- Regarding the veriﬁcation of the developed method, an error mented systems for future developments. However, these check list related to the input data, algorithm execution, and systems were rarely developed with the participation of end output visualization is deﬁned. The “well-performed” cate- users. They were mainly tested with the developers and did gory is assigned to a paper if all these three elements are not have many feedbacks from users. Most of implemented satisﬁed. simulation systems could not be directly transferred into the clinical routine practices due to lack of validations with in vitro, in vivo, and real patient data. The computed results 3. Results of simulation systems were often validated with in vitro data 3.1. Overall Quality Assessment Analysis. Statistical results of acquired from phantom tissues with physical testing machines. Due to diﬃculties of acquiring data from living the quality assessment procedure are presented in Table 4. Overall, most selected articles well described, veriﬁed, and organs, only 13% of studies conducted clinical validations validated the computational approaches. Tissue behaviors using in vivo data. Moreover, only external data such as deformations were available. Finally, the user and expert were well described in selected studies. Over 80% of articles modelled the tissue physical characteristics in the methods acceptability aspects were occasionally (i.e., only 4% and while the others just focused on soft-tissue deformations. 7% of studies) investigated. Note that most developed sys- Most authors all well conducted veriﬁcations (76%) and val- tems were initially designed for testing and verifying the idations steps (89%). For examples, in the study of Cotin et al. computational approaches rather than for developing real clinical applications. [12], after the developed methods are clearly described, the authors designed an example system using the method and analyzed the computed results. Their outputs were compared 3.2. Computational Approaches. To achieve real-time com- with other methods and showed a faster computation time putation speed when rendering and computing soft-tissue and higher accuracy level. Visualizations were also clearly deformations, two modeling approaches have been com- monly adopted. The ﬁrst approach that we called model presented to show computed deformations and collisions with a virtual surgical tool. System performance and accuracy development (MD) mainly focuses on geometry discretiza- were also measured and veriﬁed. Thus, the veriﬁcation was tion strategy and mathematical constitutive formulations of well-performed in this study. The veriﬁcation procedure soft-tissue stress-strain relationships. Soft-tissue models was not well-performed in Allard et al. [13] because they developed using this approach are commonly executed with a single-thread platform in a faster and/or more accurate mainly introduced the SOFA framework, and the authors just veriﬁed their results by visual assessments. Although manner. The second approach that we named as constitutive the real-time constraint was strongly required in the study model implementations (MI) relate to the algorithmic imple- objectives, only 65% of the developed computational mentations of the existing constitutive models using devel- approaches really satisﬁed this constraint. The others just oped methods for soft-tissue deformations onto a more powerful hardware conﬁguration such as Graphic Processing nearly reached the real-time conditions. For example, the 6 Applied Bionics and Biomechanics Table 2: The number of included/excluded articles according to the selection procedure. Duplication Title Title Abstract Abstract Content Content Search terms ScienceDirect PubMed IEEE All Duplicates included excluded included excluded included excluded included ST #1 5,537 67 13 5,617 21 5,596 5,465 131 128 3 2 1 ST #2 10,873 2,873 284 14,030 447 13,583 13,542 41 37 4 3 1 ST #3 3,638 87 14 533 19 514 335 179 169 10 0 10 ST #4 1,689 39 10 1,738 19 1,719 1,543 176 171 5 4 1 ST #5 32,857 9,762 290 42,909 605 42,304 42,209 95 83 12 7 5 ST #6 4,560 21 3 4,583 19 4,564 4,431 133 125 8 5 3 ST #7 104,034 31,339 367 135,727 371 135,356 135,265 91 76 15 3 12 ST #8 146,837 261 153 147,251 60 147,191 147,136 55 45 10 7 3 ST #9 9,185 160 23 9,368 49 9,319 9,247 72 47 25 6 19 Total 319,210 44,609 1,157 361,756 1,610 360,146 359,173 973 881 92 37 55 Applied Bionics and Biomechanics 7 Table 3: The inclusion criteria for each search terminology. # Search terms (STs) Inclusion conditions (ICs) IC #1.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “medical”, “simulations”, and “computer-aided” keywords and (2) the title concerns the supports of computers in soft-tissue simulations executing in real time 1ST#1 IC #1.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the support of computer in medical systems, medical simulations, and medical applications so that they can be executed in real time; (2) the abstract describes the medical system architectures and the interactions of computer’s input/output devices in clinical environments; and (3) the system developed in the paper focuses on simulating human soft tissues IC #2.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “biomedical”, and “simulations” keywords and (2) the title concerns the issues of real-time simulation in biomedical applications 2ST#2 IC #2.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the analyses of real time in biomedical applications/systems and (2) the abstract focuses on analysing the computational approaches, the system architectures, or the characteristics of real time in biomedical applications IC #3.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “facial” keywords, and (2) the title concerns the computational approaches to simulate the human faces 3ST#3 IC #3.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the development of computational techniques or system designs for modelling the facial mimics/expressions/muscles and (2) the developed techniques must be able to execute in real time IC #4.1: the title must satisfy all following conditions: (1) the title contains “real-time”, “liver”, and “models” keywords and (2) the title concerns the modelling methods of the human liver in real time 4ST#4 IC #4.2: the abstract must satisfy all following conditions: (1) the abstract concerns the issues of computational approaches for modelling the human liver and (2) the computational approaches must be executed in real time IC #5.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “medical”, and “simulations”/”systems” keywords and (2) the title is aimed at developing the computational methods for modelling the soft tissue in medical environments 5ST#5 IC #5.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns computational approaches or system architectures for modelling soft tissues in medical environments and (2) the system must be run in real time IC #6.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “muscle”, and “models” keywords and (2) the title considers the computational methods for modelling the human muscles in real time 6ST#6 IC #6.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the developments of computational techniques for modelling and simulating human muscles so that they can run in real time and (2) the abstract shows the implementations of muscle deformable models in clinical environments IC #7.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “surgery” keywords and (2) the title illustrates the surgical simulations/systems applied in human soft tissues executed in real time 7ST#7 IC #7.2: the abstract must satisfy all of the following conditions: (1) the abstract describes the surgical simulations/systems for human soft tissues and (2) the abstract concerns system architectures of surgical simulations or systems so that they can execute in real time IC#8.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “ﬁnite element” keywords and (2) the title concerns the ﬁnite element modelling methods for human soft tissues in real time IC #8.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the human soft-tissue 8ST#8 modelling method in real time based on the ﬁnite element modelling methods and (2) the abstract is aimed at developing, generating, and analysing the variations of ﬁnite element modelling methods to get the real-time requirements IC #9.1: the title must satisfy all following conditions: (1) the title contains “real-time”, “soft tissue”, and “deformations”/”models” keywords and (2) the title considers the modelling methods of the human soft-tissue deformations executing in real time 9ST#9 IC #9.2: the abstract must satisfy all of the following conditions: (1) the abstract illustrates the computational approaches for development the models of human soft-tissue deformations and (2) the abstract is aimed at developing, analysing, and generating the modelling methods Unit (GPU) system. Thus, systems can compute soft-tissue programming algorithms to parallelize the execution tasks models faster and more robustly than the traditional ones. of a developed modeling method, which was traditionally Particularly, this concept refers to a family of more suitable running on a single-thread platform. For example, Berkley 8 Applied Bionics and Biomechanics Table 4: Summary of the statistical results of the quality assessment procedure. Quality assessment criteria %of “yes” scores (%) Computational approaches’ bias 1. Was the method adequately used/developed and described for the involved tissue behavior? 82 2. Was the veriﬁcation well performed for the used/developed method? 76 3. Was the validation systematically performed for the used/developed method? 89 4. Did the method really satisfy the real-time constraints? 65 Interaction devices’ bias 5. Was the devices well selected for the system? 49 6. Was the device accuracy adequate for the real-time constraint? 47 7. Was the device easy enough to use for a clinical routine practice? 47 8. Is the device price suitable for a clinical setting? 47 System architectures’ bias 9. Was the system adequately described? 65 10. Was the system developed with the participation of the end users? 15 11. Was the system scalable? 53 12. Were the system frameworks adequately selected for implementing the system of interest? 45 Clinical applications bias 13. Was the study adequately validated with in vitro data? 33 14. Was the study adequately validated with in vivo data? 13 15. Was the study adequately validated with patient data? 18 16. Was the level of validation suitable for translating the outcomes into clinical routine practices? 29 17. Was the user acceptability performed for patients? 4 18. Was the user acceptability performed for clinical experts? 7 et al. addressed a MD study related to the development of a methods refer to the development of the ﬁnite element Linearized FEM (L-FEM) method built from the reduced method (FEM) and its variations to simulate the soft-tissue object kinematics [14]. The L-FEM method is suitable for deformations in real time (Figure 3). The meshfree-based modeling linear elasticity of soft tissues. This method is faster modeling techniques refer to the decomposition of soft- than the FEM. Moreover, in the study of Joldes et al., the total tissue model into simpler physical submodels or representa- tions without meshing the domains of interest (Figure 4). Lagrangian (TL) formulation was applied to improve the computation speed of the traditional FEM [15]. Additionally, The hybrid modeling methods take advantage of cooperating the total Lagrangian explicit dynamic FEM (TLED-FEM) multiple modeling methods to increase both computation formulation was developed by Miller et al. and it could run speeds and model accuracy. The distribution of selected stud- faster than the FEM when executing on the same CPU- ies according to each modeling method is shown in Figure 2. based platform [16]. Regarding the model implementation The result shows that up to 51% of the studies related to the (MI) approach, only the implicit time integration of FEM mesh-based methods. The use of the meshfree-based method has been proven to be the most suitable for parallel methods reaches over 42%. Finally, the percentage of hybrid methods is around 7%. implementation. This method was implemented in a GPU platform by Taylor et al. [17]. It is interesting to note that most studies focused at devel- 3.3. Model Development Approaches oping new mathematical methods for modeling the soft- tissue deformations rather than implementing the developed 3.3.1. Mesh-Based Modelling Methods. Mesh-based modeling modeling methods into a speciﬁc hardware conﬁguration to methods are grouped into four common computation strate- accelerate the computation speed. The distribution of the gies: the ﬁnite element modeling method (FEM), the two approaches throughout the selected literature is illus- precomputation-based FEM, the formulation-adapted FEM, trated in Figure 2. Obviously, among the total of 55 and the boundary element methods (Figure 5). Note that in studies, over 80% of the studies proposed the model devel- this present review, the term “deformation models” relates to soft-tissue models developed using a speciﬁc modeling opment of soft-tissue deformations while only 18% of studies took advantages of speciﬁc hardware to accelerate method while the term “simulation models” refers to numer- available modeling methods. Regarding the MD approach, ical models in general meaning. we grouped all developed computational methods into three The ﬁnite element method (FEM) has been popularly categories: mesh, meshfree, and hybrid modeling methods employed in the literature despite of its very high computa- tional cost. Deformable objects are geometrically meshed by (Tables 5–7). In more details, the mesh-based modeling Applied Bionics and Biomechanics 9 Model implementation (MI) 18% Computational approaches Model development (MD) 82% Mesh-based 51% Meshfree-based 42% Combination-based 7% Figure 2: Distribution of computational approaches (MD and MI) and associated techniques for MD approach in the literature. a set of elementary components called ﬁnite elements. These solution for dealing with the topological changes in cutting elements are connected by nodes whose quantity deﬁnes the simulations [20]. Peterlik et al. simulated the human liver with realistic haptic feedback and deformations embedded size of the FE model. Material properties are commonly assigned into each ﬁnite element. Then, the physical behavior with both nonlinear geometric and material parameters [3]. of solid object deformations is described by a set of constitu- Morooka et al. designed a navigation system for the mini- tive equations. Finally, the resolution of these equations on mally invasive surgeries using a neural network model [31]. the nodes with prescribed boundary and loading conditions Martínez-Martínez et al. used the decision tree and two leads to the stress-strain relationships of the deformable tree-based ensemble methods for simulating the breast com- objects. FEM provides a very high level of accuracy and real- pression [36]. Lorente et al. applied decision trees, random istic deformations in both linear and nonlinear cases. For forests, and extremely randomized trees models to simulate example, Wu et al. [30] modeled the facial muscles by FEM biomechanical behaviors of a human liver during the breath- to animate the facial expressions. Each single muscle was ing action [8]. Tonutti et al. also applied artiﬁcial neural net- considered an incompressible and hyperelastic material. works (ANNs) and support vector regression (SVR) Each muscle model includes 1,180 nodes and 28,320 DOF. algorithms for learning the precomputed data from the Note that the computing time could not be achieved in real FEM model of a human tumor [7]. Luboz et al. used a set of pressure frames compressed into a small number of modes time. Karami et al. employed also the FEM for modeling the extraocular muscles (EOMs) in an eye to estimate the by proper orthogonal decomposition [37]. This method muscular activations and directions [35]. The eyeball model allows the summarized modes to be described by a linear includes 1,970 nodes and 8,638 elements. Each muscle model set of scalar coeﬃcients, and this reduced set of pressure includes 1,100 nodes and 2,673 elements. The computation map modes was then inputted to the FE to compute the strain ﬁeld modes. time needed to solve the model was 20 ms. The precomputation-based FEM is the most popular var- The formulation-adapted FEM has been developed by iation of FEM. This method uses the relationship between the mathematically alternating the FEM formulations with the mechanical forces and the deformations precomputed from other modeling methods. One of them is called linearized the accurate FEM with full physical and biomechanical char- FEM (L-FEM) in which the kinematic behavior of the simu- lated object is linearized to the ﬁrst order of approximations acteristics to train an approximate model. To achieve this goal, a database of the accurate FE simulation outcomes during a speciﬁc timing period. Thus, the FEM model built needs to be constructed a priori. The computational accuracy from the reduced object kinematic is also simpliﬁed and exe- and speed of the simulated model depend on the types of cuted much faster than the original one. Due to the simpliﬁ- employed approximate techniques such as linear/nonlinear cation, the L-FEM is only suitable for modeling the soft tissues with linear elastic materials. For instance, Berkley regression functions and machine learning (ML). By using this strategy, Cotin et al. developed a liver surgical simulation et al. applied the L-FEM to the virtual suturing application system [12]. Sedef et al. provided a solution for real time and [14]. Moreover, Audette et al. divided a FEM model into realistic FEM for simulating viscoelastic tissue behavior in multiple submeshes [18]. All submeshes were computed medical training based on the experimental data collected independently in parallel threads of a real-time operating system to output the local deformations. Garcia et al. from a robotic tester [19]. Sela et al. proposed an eﬀective 10 Applied Bionics and Biomechanics Table 5: Classiﬁcation of developed modelling methods for soft-tissue deformations in real time: mesh-based techniques. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations Precomputation-based 1400 N Linear elasticity 7 ms (force feedback) FEM (pre-comp FEM) Dec AlphaStation Cotin et al. [12] MD The human liver 6,500 tetrahedral approximated by linear Nonlinear elasticity 8 ms (force feedback) 400 MHz elements functions 863 N 1 kHz (force feedback) Berkley et al. [14] MD Linearized FEM (L-FEM) The human skin Linear elasticity Surface triangle 1 GHz Athelon CPU 30 Hz (model rendering) elements ∗∗ Audette et al. [18] MI Multirate FEM (MR-FEM) The human brain Linear elasticity 10 kHz (force feedback) NI Dual Pentium PC Precomputation-based 51 N ∗∗∗ FEM (pre-comp FEM) Linear 1 kHz (force feedback) 153 DOF Pentium IV 2.4 GHz Sedef et al. [19] MD The soft-tissue cube using linear viscoelastic viscoelasticity 100 Hz (model rendering) 136 tetrahedral dual CPU formulations elements Precomputation-based FEM (pre-comp FEM) 1 kHz (force feedback and P4-2.8 GHz CPU, Sela et al. [20] MD The human skin Linear elasticity 12,108 polygons using discontinuous free model cutting) 1 GB RAM form deformations ∗∗∗∗ Total Lagrangian explicit 6000 E , 6741 N Karol Miller et al. [16] MD NI Nonlinear elasticity 16 ms (model deformation) 3.2 GHz Pentium IV dynamic (TLED) FEM Hexahedral elements Matrix system reduction 3.8 ms–35.7 ms From 266 N–1,579 E 2.4 GHz Pentium IV García et al. [21] MD NI Linear elasticity FEM (MSR-FEM) (solving the system) to 110 N–587 E CPU, 1 GB 2.1 ms (one system time 2,200 E-2535 N Joldes et al. [22] MD Total Lagrangian (TL) FEM NI Nonlinear elasticity CPU step) Hexahedral elements 3.2 GHz P4 CPU, From 11,168 E to Total Lagrangian explicit From 14.0 to 10.7 times 2 GB RAM Taylor et al. [17] MI The human brain Nonlinear elasticity 46,655 E NVIDIA GeForce dynamic (TLED) FEM faster than CPU Tetrahedral elements 7900 GT GPU Total Lagrangian explicit Hyperelasticity 12 ms (model deformation) 15,050 E, 16,710 N 3 GHz Intel Core Duo Joldes et al. [15] MD dynamic FEM The human brain (neo-Hookean) 1 kHz (haptic feedback) 7,000 DOF CPU (TLED-FEM) 3.54 s (3000 system time GPU NVIDIA CUDA step running) 16,825 E-12,693 N Tesla C1060 (240 FEM (NL-FEM) Joldes et al. [23] MI The human brain Nonlinear elasticity implemented on GPU 19.95 s (3000 system 125,292 E-95,669 N 1.296 GHz cores, 4 GB time-step running) high-speed memory) Total Lagrangian explicit GPU NVIDIA CUDA dynamic FEM <4 s (deformation 18,000 N–30,000 E tesla C870 (128 Wittek et al. [24] MI The human brain Nonlinear elasticity (TLED-FEM) implemented prediction) ~50,000 DOF 600 MHz cores, on GPU 1.5 GB memory) Applied Bionics and Biomechanics 11 Table 5: Continued. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations 0.54 s Precomputation-based 1,777 E–501 N 9.89 s (stiﬀness and tangent FEM (pre-comp FEM) 10,270 E–2,011 N AMD Opteron 2 GHz Peterlík et al. [3] MD The human liver Nonlinear elasticity stiﬀness matrix computing) using radial basic functions Surface triangle CPU, 8 GB RAM 1 kHz (haptic feedback) (RBF) elements 30 Hz (model rendering) Hyperelasticity (general polynomial, Total Lagrangian FEM Lapeer et al. [25] MI The human skin reduced polynomial, >1 kHz (haptic feedback) 100 E–50,000 E GPU (TL-FEM) and ogden formulation) Multiplicative Jacobian Porohyperelasticity, 13 FPS (model 20,700 E–4,300 N Marchesseau et al. [26] MD energy decomposition FEM The human liver CPU Viscohyperelasticity deformation) Tetrahedral elements (MJED-FEM) 1.4 FPS (model computing model on CPU) 41,000 N The human cataract Linear elasticity 46.15 FPS (model Tetrahedral elements Courtecuisse et al. [27] MI Linearized FEM (L-FEM) The human liver combined with a GPU computing on GPU) 3,874 N The brain tumor corotational method 64 ms (model computing on Tetrahedral elements GPU) 31,008 N Discontinuous basic 13.9 ms (model computing Turkiyyah et al. [28] MD The human skin Linear elasticity Surface triangle CPU function FEM (DBF-FEM) and mesh updating) elements 7,182 E–8,514 N Order reduction method The human cornea 20 Hz (model and graphic Hexahedral elements 2 GHz CPU, 2 GB Niroomandi et al. [29] MD Nonlinear elasticity (ORM) FEM The human liver updating) 10,519 E-2853 N RAM Tetrahedral elements Finite element method The superﬁcial 560 E–1180 N Wu et al. [30] MD Nonlinear elasticity NI CPU (FEM) fascia in a face 28,320 DOF Precomputation-based Morooka et al. [31] MD FEM (pre-comp FEM) The phantom liver NI NI 15,616 E-4,804 N CPU using neuro networks Element-by-element Maﬁ and Sirouspour precondition conjugate The human 10 times faster than CPU 6361 E–13,3784 E MI Linear elasticity NDIVIDA GTX 470 [32] gradient FEM (EbE stomach for model computing 1295 N–25462 E PCG-FEM) 12 Applied Bionics and Biomechanics Table 5: Continued. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations 70 FPS (system iteration) 1,300 tetrahedral Linear elasticity Precondition FEM The heterogeneous 1 kHz (haptic feedback) elements Courtecuisse et al. [33] MI combined with a 256 core GPU (pre-cond FEM) soft tissues 22 ms (node adding or 150 contact points corotational method removing) 3,874 N Total Lagrangian explicit A general cube Hyperelasticity 0.309 s–163.402 s (one NVIDIA GTX460 Strbac et al. [34] MI 125 E–91,125 E dynamic (TLED) FEM mesh (neo-Hookean) solution time step) GPU Eyeball: The extraocular 8638 E–1970 N Finite element modelling Karami et al. [35] MD muscles (EOMs) in Linear elasticity 20 ms (model deformation) Muscle: CPU method (FEM) an eye 2673 E-864 N Tetrahedral elements Precomputation-based FEM (pre-comp FEM) Hyperelastic 313,000 E-62,000 N 2.6 GHz Intel (R) Martínez et al. [36] MD The human breast <0.2 s (model compression) using artiﬁcial neuro (Mooney-Rivlin) Tetrahedral elements Xeon (R) CPU networks Precomputation-based 2.89 s (model computing 3.4 GHz Intel Core i7, FEM (pre-comp FEM) using machine learning) From 379,800 N to Lorente et al. [8] MD The human liver Nonlinear elasticity 8 GB RAM, OS X El using artiﬁcial neuro 51.63 s (model computing 420,690 N Capitan networks using FEM) Precomputation-based FEM (pre-comp FEM) <10 ms (model prediction 6,442 N-1,087 E Tonutti et al. [7] MD using artiﬁcial neuro The brain tumor Nonlinear elasticity Core i7 2.9 GHz CPU using neural network) Tetrahedral elements networks and support vector regression Precomputation-based 27,649 E FEM (pre-comp FEM) <1 s (strain ﬁeld Luboz et al. [37] MD The butt area Nonlinear elasticity Hex-dominant CPU using the reduced order computing) elements modelling method ∗ ∗∗ ∗∗∗ ∗∗∗∗ N: nodes; NI: no information; DOF: degree-of-freedom; E: elements. Applied Bionics and Biomechanics 13 Table 6: Classiﬁcation of developed modelling methods for soft tissue deformations in real time: meshfree-based techniques. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 16 FPS (model 82 mass points SGI Impact workstation, Mass-spring system deformation) Nedel and Thalmann [38] MD The muscle Linear elasticity method (MSM) 84 FPS (model 17 mass points MIPS R10000 CPU deformation) <150 N Boundary element The general cube 15 Hz (model R-4400 CPU, 64 MB Monserrat et al. [4] MD Linear elasticity Surface triangle method (BEM) mesh deformation) RAM elements Statistical analysis 1 minute (facial Pentium II, 333 MHz Goto and Lee [39] MD method (SAM)- The human face NI NI feature detection) CPU muscle Mesh geometry 475 ms (facial VRML-like representation deformation) 1,253 V-2,444 F Pentium II 450 MHz Bonamico et al. [40] MD The human face Linear elasticity (VRML) & radial basis 1,430 ms (facial 4,152 V-8,126 F CPU, 128 MB RAM function (RBF)- deformation) muscle 216 N-1,440 E 24 FPS (system Surface triangle Sun Ultra 60 Workstation Mass-spring system Nonlinear iteration) elements Brown et al. [2] MD The blood vessel 450 MHz CPU, 1 GB method (MSM) viscoelasticity 6 FPS (system 8,000 N-66,120 E RAM iteration) Surface triangle elements ~10,000 V Laplacian surface Sorkine et al. [41] MD The face model Linear elasticity 0.07 s (model solving) Surface triangle 2.0 GHz Pentium IV CPU deformation (LSD) elements Personal facial expression space 12 FPS (facial Chandrasiri et al. [42] MD The human face Linear elasticity NI 1 GHz Athlon CPU method (PEES)- animation) muscle From 53,3380 N to Mass tensor method The cube Mollemans et al. [43] MD Linear elasticity From 24.57 s to 2.3 s 10,368 N CPU (MTM) The human face Tetrahedral mesh Mass-spring system 8,000 N method (MSM) 48 Hz–3,000 Hz SGI Prism Server 4 GPU, Chen et al. [44] MD The human brain Linear elasticity Surface triangle combined with (haptic feedback) 8 CPU, 32 GB RAM elements quasistatic algorithm 14 Applied Bionics and Biomechanics Table 6: Continued. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 4,891 V GPU NVIDIA 6,800, Mass-spring system The human inguinal 73 FPS (system Pentium IV 3.0 GHz López-Cano et al. [45] MI Linear elasticity Surface triangle method (MSM) region iteration) elements CPU, 1 GB RAM Point collocation- 1 ms (model 1,186 polygons Pentium IV 2 GHz CPU, Lim and De [46] MD based ﬁnite spheres The human liver Nonlinear elasticity deformation) Polygon elements NVIDIA Quadro4 XGL (PCMFS) 16 ms (muscle tension Intel Xeon 3.33 GHz Inverse dynamic estimation) CPU, 3.25 GB RAM, Murai et al. [5] MD The human muscles Linear elasticity 274 muscles computation (IDC) 15 FPS (model NVIDIA Quadro FX3700 rendering) GPU 5 ms (model deformation) 96 N-270 E Basafa and Farahmand Mass-spring-damper Nonlinear 150 Hz (haptic Tetrahedral mesh 3.2 GHz Core Duo CPU, MD The cube model [47] method (MSD) viscoelasticity feedback) 500 N 1 GB RAM 30 Hz (model Tetrahedral mesh rendering) Windows 2000 or Laplacian surface Wang et al. [48] MD The human nose Linear elasticity NI NI Windows XP, 512 MB deformation (LSD) RAM or 250 MB 1 kHz (haptic 917 E Mass-spring system feedback) Intel Core2 Q6600 CPU, Ho et al. [6] MD The human eardrum Linear elasticity Surface triangle method (MSM) 30 Hz (model NVIDIA GeForce 9,600 elements rendering) Radial basic function 5,272 V-10,330 F Intel Core 2 Duo E7200 0.0316 s (one system Wan et al. [49] MD (RBF) & geodesic The human face Linear elasticity Surface triangle 2.53 GHz CPU, 2 GB frame computing) distance-muscle elements RAM 73.8 FPM (facial Intel Xeon 2.4 GHz Thin-shell animation) 40 markers 16-Core CPU, NVIDIA Le et al. [50] MD deformation method The human face Linear elasticity 164.3 FPM (facial 100 markers Tesla C1060 240-Core (TSD)-muscle animation) GPU Elastic-plus-muscle- Zhang et al. [51] MD distribution-based The facial muscles Linear elasticity NI NI NI (E+MD) >200 FPS (graphic Facial motion rendering on PC) Core i7 3.5 CPU Weng et al. [52] MD regression algorithm The human face NI 30 FPS (graphic 75 facial markers Intel Atom 2.0 GHz CPU (FMR)-muscle rendering on mobile devices) Applied Bionics and Biomechanics 15 Table 6: Continued. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 4.02 ms (one model computation 4,430 E–1,128 N Hyperelastic mass link iteration) Tetrahedral mesh Core 2 Duo 2.40 GHz Goulette and Chen [53] MD method for FEM The cube model Viscohyperelasticity 21.24 ms (one model 21,436 E–5,591 N CPU, 3.45 GB RAM (HEML-FEM) computation Tetrahedral mesh iteration) The time-saving volume-energy 30 Hz (model Core i7-4700 3.4 GHz Zhang et al. [54] MD conserved ChainMail The cube model Nonlinear elasticity NI rendering) CPU method (TSVE- Chainmail) 2 minutes (system Radial basis function initializing) Woodward et al. [55] MD mapping approach The human face Linear elasticity NI NI Up to 30 Hz (facial (RBF)-muscle feature detection) Core i7-4790 3.60 GHz Marquardt radial The general cube 0.1509 s (model 121 nodes CPU, 8 GB RAM, Intel Zhou et al. [56] MD basis meshless Nonlinear elasticity model deformation) Tetrahedral mesh HD Graphics 4600 method (MRM) (64 MB) 16 Applied Bionics and Biomechanics Table 7: Classiﬁcation of developed modelling methods for soft-tissue deformations in real time: combination-based techniques. Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed Geometry discretization conﬁgurations Precomputation-based FEM (pre-comp FEM) & 40 Hz (model deformation) 760 vertices–4,000 edges 233 MHz Dec Alpha Cotin et al. [57] MD mass tensor method The blood vessel Linear elasticity 500 Hz (haptic feedback) 8,000 tetrahedral elements Workstation (MTM) & hybrid modelling method (HMM) 1.6 GHz Dothan <25 ms (model Yarnitzky et al. [58] MD Dynamics-based & FEM The foot soft-tissue Linear elasticity 100 nodes Pentium IV CPU, deformation) 1 GB RAM Multi-cooperative methods Allard et al. [13] MD NI NI NI NI NI (multi-Corp) Boundary element method Linear elasticity 2.26 GHz Pentium (BEM) & mass-spring with an extra From 0.99 ms to 4.17 ms M CPU, GeForce Zhu and Gu [59] MD The human liver From 200 to 1,200 nodes system (MSM) & particle mass-spring (model deformation) 9650 M GPU, 2 GB surface interpolation (PSI) model RAM Applied Bionics and Biomechanics 17 Modeled so tissues Described behaviors Generic so tissue (i) Linear and nonlinear elasticity Liver and liver Model development (MD) (ii) Linear viscoelasticity phantom Cornea (ii) Hyperelasticity (Neo-Hookeanand Mooney-Rivlin) Skin Face (iv) Poro-hyperelasticity Brain Extraocular muscles Breast Mesh-based techniques Described behaviors Modeled so tissues Generic so tissue (i) Linear and nonlinear elasticity Liver Linear elasticity combined with a co-rotational method Model implementation (MI) (ii) Skin (ii) Hyperelasticity (general polynomial, reduced Cataract Brain polynomial and Ogden formulation) Tumour (iv) Hyperelasticity (Neo-Hookean) Stomach Figure 3: Overview of all modeled soft tissues and diﬀerent described behaviors for mesh-based studies. Modeled so tissues Described behaviors Generic so tissue Skeletal muscles (i) Linear and nonlinear elasticity Model development (MD) Face Facial muscles (ii) Nonlinear viscoelasticity Nose Liver (iii) Visco-hyperelasticity Eardrum Brain Blood vessel Meshfree-based techniques Described behaviors Modeled so tissues (i) Linear elasticity Inguinal region Model implementation (MI) Figure 4: Overview of all modeled soft tissues and diﬀerent described behaviors for meshfree-based studies. presented another reduction method called matrix system each iteration. Turkiyyah et al. aimed at physically simulating reduction FEM (MSR-FEM) [21]. The method focused the mesh cutting in real time thanks to the controlled discon- rather on computing the regions of interest than the whole tinuities in the basic functions and the fast incremental model. The order reduction method (ORM) was developed methods for updating the global deformations [28]. Finally, by Niroomandi et al. to reduce the complex computation of element-by-element precondition conjugate gradient FEM nonlinear FEM for real-time simulations [29]. The total (EbE PCG-FEM) was developed by Maﬁ and Sirouspour [32]. This method combined the FEM with a conjugate Lagrangian (TL) formulation was also applied in a FE model to improve the computation speed. Joldes et al. used this gradient method by alternating the mesh topological com- approach to develop a FE model for an eﬃcient hourglass putation at run time by iterations. Thus, the developed control application [15]. A variation of this method, called model would be faster than the original one using FEM and total Lagrangian explicit dynamic FEM (TLED-FEM), was required less system memories during execution. A new pre- also developed by Miller et al. for an image-guided surgery conditioning technique (pre-cond FEM) was also proposed applications [16]. This method was also employed by Joldes by Courtecuisse et al. for improving the computational time et al. to simulate the deformations of a human brain [15]. of soft-tissue deformations [33]. This technique could simu- They all successfully improved both the sizes and computa- late topologically changes and haptic feedbacks of homoge- tion speeds of the developed models. Another version of neous and heterogeneous materials in acceptable accuracy. TL-FEM proposed by Marchesseau et al. was called multipli- The boundary element methods are based on surface cative Jacobian energy decomposition (MJED) FEM [26]. deformations to deduce the internal deformations in real This approach optimizes the generation of stiﬀness matrix time. Monserrat et al. developed a surgery simulation system in TL-FEM to solve the linear system of equations during using this method [4]. Compared with the FEM, the BEM 18 Applied Bionics and Biomechanics Mesh-based techniques Finite element Pre-computation- method based FEM (FEM) Boundary Formulation- element adapted FEM methods Figure 5: Overview of common computation strategies for mesh-based studies. only required the discretization of the object’s surface so that form the model’s stiﬀness matrix and to describe biomechan- ical characteristics of the soft-tissue object [56]. In particular, it could provide an optimized, fast, and easy implementation. Another surface-based method for developing the soft-tissue this approach does not need to preprocess all cell elements to models was called Laplacian surface deformation (LSD) was estimate the global deformations like mesh-based modeling ﬁrst proposed in Sorkine et al. [41]. The method represented methods do. Consequently, the meshfree-based modeling the object surface based on the Laplacian of the mesh. Wang techniques are much faster than the mesh-based modeling et al. also employed the LSD method for nose surgery in a strategies, and they can simulate large deformations in real complete surgical system for automatic individual prosthesis time. Because of these advantages, the meshfree-based design [48]. Goto et al. used the statistical analysis method modeling methods have received much attentions from (SAM) for detecting features on the facial surface through research community in the recent years. One of the most 2D images, and then, the detected features were mapped to popular methods using the meshfree-based strategy is the a generic 3D facial model for generating the expressions mass-spring system modeling (MSM) method. Nedel et al. using the surface deformation method [39]. Moreover, the applied the MSM method to model the muscle deformations computation speed of the facial expression estimators was in real time [38]. Brown et al. applied the MSM method for a surgical training system [2]. Chen et al. also used the MSM enhanced by using a scaling polygon mesh method based on iterative edge contractions by Bonamico et al. [40]. for developing a deformable model for haptic surgery simula- Chandrasiri et al. proposed a strategy for converting the tion [44]. The MSM was also applied to simulate the 3D acquired facial expressions to the MPEG-4 FAP [60], stream model of the human inguinal region by López-Cano et al. to deform the 3D surface facial models robustly and in real [45]. Ho et al. developed a deformable tympanic membrane using the MSM method for simulating the real-time defor- time [42]. Wan et al. [49] and Woodward et al. [55] used the landmark-based and muscle-based facial expression esti- mation and cutting in a virtual reality myringotomy simula- mation to animate the 3D surface facial model. The used tor [6]. Another well-known method of meshfree-based methods were radial basic function (RBF) and geodesic dis- method is called the mass tensor method (MTM) in which tance. Le et al. took advantages of the thin-shell linear defor- the modeled object is approximated into a tetrahedron mesh. Inside each tetrahedron in the MTM, the displacement vec- mation model to reconstruct the facial pose via the facial marker displacements [50]. tors of four vertices are linearly interpolated into the dis- placement ﬁeld of this tetrahedron [57]. The MTM was 3.3.2. Meshfree-Based Modelling Methods. Compared to the used by Mollemans et al. to simulate the soft-tissue deforma- mesh-based modeling methods, meshfree-based modeling tions after bone displacement [43]. An improvement of MSM called mass-spring-damper (MSD) modeling methods was methods use discrete points for representing continuum, and it takes advantages of interpolation methods to solve proposed by Basafa and Farahmand [47]. The results illus- the partial diﬀerential equations (PDEs) [59]. Thus, a simu- trated that with a simple cube model including 96 nodes lated soft-tissue object is commonly modeled as a distribu- and 270 tetrahedrons, the computation time was just about tion of discrete nodes inside to form a complete volumetric 0.005 s for each step. Another improvement of MSM was developed by Goulette et al. called hyperelastic mass links model. These nodes are embedded with a shape function to Applied Bionics and Biomechanics 19 requirement level of real-time constraints. Zhu and Gu (HEML) in which the forces at a speciﬁc node are considered a sum of force functions from the neighboring nodes con- also applied multiple modeling methods to develop a nected with it [53]. Experiments showed that with the hybrid deformable model for real-time surgical simulation [59]. Diﬀerent cooperative components exist in the system 21,436-tetrahedron HEML model, the computation time was at 21.24 ms corresponding with 47 FPS. A diﬀerent such as boundary the element method (BEM), the mass- aspect for meshfree-based methods is proposed by Lim and spring method (MSM), and a particle surface interpolation De known as the point collocation-based method of ﬁnite algorithm. sphere (PCMFS) [46]. The technique was based on the com- 3.4. Model Implementation Approaches. The model imple- bination between the multiresolution approaches and the fast analysis strategies for nonlinear deformations for the active mentation (MI) approach mainly focusses on algorithmic regions where being contacted by the surgical tool tip. A dis- implementation of soft-tissue models based on developed tinctive modeling method for meshfree-based method was modeling methods onto a more powerful hardware conﬁgu- inverse dynamic computation (IDC) proposed by Murai ration. This approach can improve the computational perfor- mance of the developed soft-tissue deformation models, even et al. for the musculoskeletal system [5]. Zhang et al. devel- oped an elastic-plus-muscle-distribution-based (E+MD) to faster and more robust than the MD approach. In particular, model the facial muscle distribution for generating the facial the MI approach mostly aims at ﬁnding more suitable pro- expressions in real time [51]. Another method called the gramming algorithms to parallelize the execution functions time-saving volume-energy conserved ChainMail (TSVE- of the soft-tissue deformation models onto a graphic process- ing unit (GPU) platform rather than onto a central process- ChainMail) was proposed by Zhang et al. [54]. The method was developed from the traditional ChainMail method in ing unit (CPU) platform. Basically, GPUs are comprised of which the model is represented as a spring system. Zhou highly parallel architectures. Each separate GPU contains et al. have also proposed a Marquardt radial basis meshless numerous processors and memory segmentations, and each method (MRM) for the soft-tissue cutting [56]. In addition processor works independently on its own data distribution. Consequently, although the clock frequencies of GPUs are to these studies, it is important to note that a large range of soft-tissue models (brain, ligament, and atrioventricular often smaller than CPUs, the overall computation speed of valves) were also developed using the element-free Galerkin GPUs are much faster than CPUs, even when CPUs can be method and isogeometric method [61–65]. Due to the used composed of multiple processing cores up to now. Further- keywords, this present review does not include these works. more, various programming frameworks supported for model implementations have been improved in an easier and ﬂexible Thus, interested readers could use more speciﬁc keywords to get information about these methods. ways. Two classical interfaces have been employed for pro- gramming on GPUs have been OpenGL, application pro- gramming interfaces (APIs), DirectX, CUDA from NVIDA, 3.3.3. Hybrid Modelling Methods. Hybrid methods have been intensively investigated in the literature due to its cooperative and CTM from ATI. These frameworks have been written in high-level C-programming language which bring many functions which take advantages from multiple methods. For instance, although the mass-tensor method (MTM) is fast beneﬁts for modelers to implement their developed methods and suitable for simulation of the soft-tissue deformation in executing on GPU eﬀectively [17]. An analysis of GPU imple- mentations for surgical simulations was reviewed by Sørensen real time, it still lacked the realistic biomechanical character- istics, especially when simulating the nonlinear materials. On and Mosegaard [66]. They concluded that GPUs would become much powerful and cost-eﬀective platforms for the other hand, the FEM has realistic simulation of biome- chanical behaviors of soft tissues, but it has high computation implementing the soft-tissue deformation models in real- cost. Additionally, the precomputation-based methods time medical environments. However, to be able to achieve beneﬁts from this implementation approach, the developed (pre-comp FEM) have very high performances for simulating the soft-tissue deformations in real time based on the pre- modeling methods must be compatible and be able to recon- computed data from the FEM, but they cannot handle the ﬁgure with parallel computations [66]. The ﬁrst model imple- topological changes. Consequently, the combination between mentation strategy was proposed by Taylor et al. [17]. The MTM, FEM, and pre-comp FEM can not only simulate the authors implemented a model using the total Lagrangian explicit dynamic (TLED) FEM onto a NVIDIA GeForce deformation in real time but also handle cutting and tearing realistically with nonlinear materials. This approach was ﬁrst 7900 GT GPU platform, and the results showed that the com- developed by Cotin et al. [57]. The result of this study showed putation speed of the implemented model was much faster that the update frequency was able to reach at 40 Hz with an than the CPU-implemented model. A human brain model MTM having 760 vertices and 4000 edges. Yarnitzky et al. using the TLED-FEM was also implemented on the NVIDIA Tesla C870 GPU platform by Wittek et al., and the computa- combined the physically kinematic model with the local FE model to estimate the stresses and deformations inside a tional performance was also accelerated signiﬁcantly [24]. For plantar foot’s soft tissues during gait [58]. Allard et al. instance, with the brain model of 18,000 nodes and 30,000 ele- introduced a well-known SOFA framework supporting bio- ments (approximately 50,000 degrees of freedom), the average medical researchers modularly and ﬂexibly to develop new time for estimating the brain deformations was less than 4 s when implemented on GPU, and the time implemented on soft-tissue deformation models [13]. The framework was comprised of multiple modeling methods combined eﬀec- the CPU platform was up to 40 s. A model using the explicit tively to simulate the soft tissues according to their FEM in a real-time skin simulator was also implemented on 20 Applied Bionics and Biomechanics Not using interaction devices e 3-D viewers e 2-D optical cameras e biomechanical sensors e PC's human interface devices e 3-D optical cameras e 3D scanners e force feedback devices e virtual surgical instruments 0 5 10 15 20 25 # of studies Figure 6: The distribution of using interaction devices in the chosen literature. a GPU platform by Lapper et al. and the simulation results human interface devices, the 3D viewers, and the 2D and 3D could be accelerated to reach real-time goal [25]. Joldes et al. optical cameras. The statistic distribution of used interaction devices is shown in Figure 6. It is clearly showed that the vir- employed a GPU platform using a programming guide NVI- DIA Compute Uniﬁed Device Architecture (CUDA) to speed tual surgical instruments have been popularly used with 19 up the TLED-FEM human brain model [23]. Strbac et al. also studies, and the least used device was the 3D viewers and employed the model using TLED-FEM onto multiple the 2D optical cameras with only 3 studies. The second most General-Purpose Graphics Processing Units (GPGPUs) to popular interaction devices are the force-feedback devices which were found on 15 studies. Other remaining interaction evaluate the eﬃciency of this implementation with the current commercial solutions [34]. The experimental evaluations devices have been utilized by only from 4 to 6 studies. In fact, showed that when the size of the model increased from 125 most of the studies have taken advantage of the force- to 91,125 elements, the computational time was from 1 s to feedback devices always combined with the virtual surgical 1 h 39 min 37 s running on Abaqus commercial software, from instruments for interacting with the simulated model. More- over, other interaction devices certainly used in computer 0.149 s to 34.143 s running on the most powerful GPU of GTX980. Courtecuisse et al. proposed an implementation, systems, such as computer screens and computer keyboards, called linearized FEM (L-FEM), which was the combination are not deeply analyzed in this review paper due to their obvi- between the linear elastic material with a FEM [27]. Maﬁ ous contributions to the simulation system. et al. deployed a model using element-by-element precondi- 3.6. Virtual Surgical Instruments and Force-Feedback Devices. tioned conjugate gradient (EbE PCG) FEM method in the GTX470 GPU platform for speeding up the deformation com- The virtual surgical tools have been widely combined with putation in real time [32]. The implicit time integration of a force-feedback devices to communicate between a user and nonlinear FEM on the GPU platform was also performed by a simulated model so that the simulation system could Courtecuisse et al. [33]. become more ﬂexible and realistic. The functions of virtual surgical instruments are to transfer the controlled signals 3.5. Interaction Devices. In addition to the model develop- from external real devices to the simulated model and to feed- ment methods and the implementation approaches, the back the calculated biomechanical parameters from the simu- interaction devices contribute signiﬁcantly to the whole sys- lated model to the external haptic devices [2, 14, 44, 47]. The tem accuracy and computational time. After user commands speed of transmitting data from/to simulated models must be relatively high so that the visualization and haptic feedback are transferred to the computer system through input devices, the computer system must execute the simulated can be simulated realistically [2, 14]. Force-feedback devices model according to the commanded strategies. Once each are the input/output devices having a function of interfacing simulation iteration is completed, the estimated feedbacks between a user and a virtual surgical tool. When the interac- from the simulated model are transmitted to the user tions are received from the virtual surgical tool, the simulated model will react and calculate haptic forces during each sim- through the output devices. Consequently, the total accuracy of both input/output interaction devices and soft-tissue ulation iteration. These computed haptic forces are ﬁnally models must be at least equal to the desired accuracy toler- feedbacked to the device through the virtual surgical tool ances in each medical application. Diﬀerent interaction [12, 18–20, 27, 44, 53]. As a result, the user will feel like they devices have been used in the reviewed studies. However, are interacting with a real soft tissue when the reaction forces are received from the force-feedback device [3, 12]. For exam- due to the focused objectives on developing the modeling methods, up to 42% of reviewed studies did not use interac- ple, Cotin et al. used this combination in surgical simulation tion devices in their simulation systems. The interaction to provide haptic sensations for the surgeon [12]. Audette devices in the remaining studies could be divided into diﬀer- et al. used a 7-degree-of-freedom (DOF) haptic device com- ent types: the virtual surgical instruments and force-feedback bined with a surgical tool whose tip is ﬁxed at the end of the haptic device to make the simulation system more realistic devices, the 3D scanners, the biomechanical sensors, the PC’s Interaction devices Applied Bionics and Biomechanics 21 monitoring foot’s internal deformations under outside inter- [18]. In the study of Chen et al., a commercial PHANTOM haptic device with 3 DOF force feedback and 6 DOF position actions [58]. Electromyography (EMG) was also used in the and orientation was used for haptic surgery simulation [44]. study of Murai et al. for acquiring muscle tensions [5]. Sela et al. developed a new force feedback system called the SensAble™ PHANTOM Desktop™ haptic device [20]. The 3.6.3. The PC’s Human Interface Devices. Some PC’s human device was combined with a pen-sized handle, and they are interface devices have also been used widely in the real-time all connected to a robot arm with ﬂexible engine-forced joints soft-tissue simulation systems. They are all ﬂexible and easy to simulate a virtual surgical scalpel. Courtecuisse et al. used a for user manipulation. For example, Lopez-Cano et al. used a PC mouse as a surgical tool interacted with a virtual pointer virtual laparoscopic grasper, which was managed by a Xitact IHP haptic device [27]. Peterlik et al. used a virtual haptic to deform the 3D model of the human inguinal region [45]. interface point (HIP) controlled by a PHANTOM haptic This conﬁguration could simulate vertical and horizontal device to calculate haptic forces reacted from a liver model stretching deformations of the simulated model. In some based on displacements between the HIP’s position and the simulated systems, the PC mouse could only be controlled to rotate the view angle of the simulated model [21]. More- 3D surface model of the liver [3]. over, it could be used for drawing the cut shapes onto the 3.6.1. 3D Scanners. The 3D scanners can be divided into two surface model [48]. categories: structural and surface scanners. The most widely used structural scanners in the literature have been CT and 3.6.4. 3D Viewers. The 3D viewers are the output interaction devices. They are composed of two separate high-resolution MRI. Cotin et al. created an anatomical model of a human liver from MRI images [12]. Marchesseau et al. used CT screens or two glasses attached together horizontally for dis- images for creating the geometrical model of a liver [26]. playing two diﬀerent image frames to human eyes. Based on a Besides, 3D surface scanners were also used in the studies stereo geometrical model from human vision, the 3D viewers relating to surface-based soft-tissue modeling methods. The can create depth perceptions for human brains, so when using the 3D viewer for visualizing the simulated model, most popular surface-based scanners employed were laser scanners and ultrasonic scanners. They took advantage of the graphic rendering can become more realistic. A classic measuring the time-of-ﬂight of laser/ultrasound beams for 3D viewer was presented by Brown et al. using the stereo estimating the distance between the laser/ultrasound sources glasses for enhancing the illusion of depth in the video frames and the object’s surface. These scanners are fast and able to [2]. In the visualization system, there were two image frames displayed: one image frame was colored in red, and the others acquire the object surfaces in real time. The laser scanners are much more accurate than ultrasound scanners, but laser was colored in cyan. The stereo glasses included two diﬀerent beams can be very harmful to the living soft tissues during color ﬁlter glasses for the left and right lens, so at the same time, each human eye would see a diﬀerent image frame. long acquisition period. For example, Monserrat et al. employed the 3D ultrasonic scanner (SAC GP10, Smart Each pair of image frame was shifted horizontally for creating the depth information. A diﬀerent 3D viewing device called a EDDY System, USA) for capturing the 3D outside surface of the simulated object based on the boundary element 3D stereo visor was used to visualize a simulated model in the modeling method [4]. The combination between surface virtual reality myringotomy simulation in the study of Ho et al. [6]. This device included two diﬀerent high-resolution scanners and structural scanners was also proven to be eﬀec- tive for accurate reconstructing both surface and structural screens for displaying two diﬀerent image frames at the same time. details. Wang et al. combined a 3D laser scanner with the lat- eral X-ray scanners in their methods [48]. In fact, the 3D laser images containing both 3D geometrical point cloud 3.6.5. 2D and 3D Optical Cameras. The 2D optical cameras are the input interaction devices having the functions of and colors of the human face were transformed to the lateral X-ray image for comparing and cutting the nose part on the acquiring 2D image frames of object surfaces. In the applica- face. This combination provided a high-quality and patient- tion of facial expression recognition, Chandrasiri et al. speciﬁc model of the human face appearance. mounted a complementary metal oxide semiconductor (CMOS) camera to a headphone to capture 2D color video frames of a user face [42]. An ordinary web camera available 3.6.2. Biomechanical Sensors. Most biomechanical sensors used in soft-tissue modeling systems are electromagnetic sen- on a mobile device was also used by Weng et al. in the appli- cation of real-time facial animations [52]. In particular, the sors, force sensors, and electromyography sensors. Brown et al. used an electromagnetic tracker (miniBRID of Ascen- oﬄine 2D images acquired from a camera were also analyzed sion Technology Cooperation) to track behaviors of a real for detecting the facial expressions and cloning them to other 2D facial images in the study of Zhang et al. [51]. One of the surgical forceps [2]. Sedef et al. attached a force sensor (ATI Industrial Automation’s Nano 17) at the end of a real most drawbacks of 2D optical cameras is the inability of surgical probe for measuring forces inside a surgical trocar reconstructing depth information from a single view of so that the user could feel like being in a real minimally inva- vision, so multiple optical cameras have been cooperated to sive surgery [19]. Yarnitzky et al. employed ultra-thin force form a 3D optical camera system for detecting the 3D data. A motion capture device was utilized to capture 3D motions sensors arranged under the bony prominences of each foot for measuring the forces under the calcaneus, metatarsal of a human during dynamic movements in the study of heads, and phalanges in real time in an application of Murai et al. [5]. A stereo optical motion capture was also 22 Applied Bionics and Biomechanics liver simulation system [3]. The main thread called haptic combined with facial markers in the study of Wan et al. for detecting facial animations [49]. Over 36 facial markers were thread is acquiring positions of a haptic device, detecting col- detected and followed by mocap, and their motions were lisions, calculating haptic forces, and computing model’s deformations. The simulation system designed in the study then converted to MPEG-4 standard’sdeﬁnition of facial ani- mations. Woodward et al. used an oﬀ-the-shelf stereo web- of Goulette et al. also included multiple computation mod- cam for marker-based facial animation application [55]. ules for accelerating the system execution [53]. Two modules Applied to minimally invasive surgeries in the study of were threaded to execute in parallel on an Intel Core 2 Duo at Moroka et al., the 3D optical camera was integrated into a 2.40 GHz, 3.45 GB of RAM. As a result, the visual rate could be reached up to 47 FPS. Audette et al. designed a surgical stereo endoscopy whose size was small enough to be used in restricted navigation spaces [31]. simulation system which included their own developed hap- tic device with 7 DOFs [18]. To control this haptic device, an 3.7. System Architectures. Computational approaches and intelligent I/O board, called the DAP5216a/626, operating interaction devices have been developed throughout the individually with the computer system was proposed. Another scheme for system execution called a multimodel literature to improve computational accuracy and speed of soft-tissue deformation models, but they will not operate representation, was proposed by Allard et al. within the eﬀectively and robustly in real time if soft-tissue models SOFA framework [13]. In this scheme, each soft-tissue simu- and interaction devices are not well-cooperated in a system lation components could be represented by multiple model- architecture. This section will synthesize system execution ing methods related to real-time deformation simulation, accurate collision detection, or realistic interaction computa- schemes and programming frameworks of the system archi- tectures developed in the literature. tion. Finally, the task of programmers was to design a switcher to eﬀectively alternate modeling methods according 3.7.1. System Execution Schemes. Cotin et al. designed the to each appropriate simulation issue. ﬁrst execution scheme called the distributed execution 3.7.2. System Frameworks. System development frameworks scheme in which a computer system and a Dec AlphaStation were closely cooperated [12]. The computer system is aimed must be selected carefully so that the system could be devel- at computing the haptic forces and exchanging data with the oped both in high productivity and short time-to-market. haptic device while the Dec AlphaStation visualized the Generally speaking, a software framework is a generalization deformations of this soft-tissue model in real time. The com- software structure in which programmers can contribute their written codes to modify this structure to a speciﬁc appli- munication environment between two computer systems was the ethernet connection. Chen et al. distributed a developed cation. Taking advantages of available conﬁgurations and haptic surgery simulation onto two computing systems prebuilt libraries, simulation systems could be developed much more ﬂexibly and faster than in traditional develop- [44]. While an SGI Prism Visualization Server with 4 ATI FireGL GPUs covered graphical simulations, a Windows ment procedures. The system frameworks can be divided into four groups: the input/output interaction frameworks, computer system controlled the haptic devices and simulated the haptic feedbacks. The system could manage more than the graphic interaction frameworks, the modelling frame- one simulated model by using a new peripheral protocol works, and the hybrid frameworks. Regarding the input/out- put interaction frameworks, haptic devices have been called virtual reality peripheral network (VRPN) developed by the University of North Carolina. Two workstation sys- commonly used in the literature, and they are often inter- faced with computer systems by GHOST [3, 19, 44] and tems were also cooperated on a simulation system for the minimally invasive surgery proposed by Morooka et al. PHANTOM [44] input interaction framework. These input [31]. All model computations were performed by the ﬁrst interaction frameworks are all free and open source. More- over, other standard input interaction devices, such as key- workstation while the second workstation only performs visualization of deformations and virtual tools in the form boards, PC mouse, web cameras, and microphones, can of a 3D stereo vision for improving depth sensations. Note also interface with computer systems through application that the limitation of transmission bandwidth leaded to the programming interfaces (APIs) supported by Microsoft latency between the visualization force-feedback. To solve Windows systems [42, 45, 48]. Regarding the graphic inter- action frameworks, the most employed graphic framework this issue, the multithread execution scheme was proposed, Brown et al., which distributed two tasks of deformation was OpenGL in which 2D and 3D vector graphics can be rapidly rendered by GPU-platform boards. The rendering visualization and collision detection on two diﬀerent execu- tion threads on a single dual-processor machine (Sun Ultra tasks can be executed on a separate computer system or on 60 with two 450 MHz processors) [2]. The system included a local thread [3, 6, 18, 19, 38, 44, 45, 59]. In particular, the OpenGL framework can be embedded in multiple types three intercooperative simulators: a deformable object simu- lator, a tool simulator, and a collision detection module. The of operating systems such as Android, iOS, Linux, Windows, idea of multiple-thread executing on a single computer sys- and various embedded operating systems. Moreover, it can tem was also applied by Sedef et al. in a soft-tissue simulation also support for writing in multiple programming languages system including a phantom haptic device, a computer (e.g., C++, Python, C#, and Cg). In addition, the CUDA™ graphic framework was developed by Nvidia Corporation. screen, and a simulated model [19]. Peterlik et al. also imple- mented two asynchronous computation threads executed on There have been numerous studies using CUDA framework an AMD Opteron Processor 250 (2 GHz) PC to operate a for implementing their simulation system on of-the-shelf Applied Bionics and Biomechanics 23 tom could be used to give the validating data for the geo- graphic GPU boards and achieving great beneﬁts from parallel execution structure in real-time computations metrical validation [7]. Regarding the model behavior [17, 23, 24, 27, 32, 34]. Another general graphic framework validations, the physical characteristics of the simulated models must be assessed with the real physical data at diﬀer- called OpenCL™ was also developed for ﬂexible parallel implementation. An image processing framework called ent deforming states. One of the most popular schemes is to Virtual Place (AZE Co.) was also used for converting 3D use the calculated data from a standard commercial simula- deformations to stereo video frames for creating depth feel- tion software as baseline data. For example, a model using ing on human visions [31]. Regarding the modelling frame- linear viscoelastic FEM was validated through a compression test solved by both the proposed computation approach and works, GHS3D [26], TetGen [47], and CDAJ-Modeler [31] were employed for generating mesh models from CT/MRI the ANSYS ﬁnite element software package. Obtained results images. Additionally, Maxilim software could also support showed that the maximum error of displacement was less for boundary condition simulations [43]. Moreover, the than 1% [19]. The ANASYS software package was also used CHOLMOD open source library could also be used for solv- for validating a model based on a machine learning-based FEM method [36]. Recently, the Marquardt-based model ing the linear systems in real time [50]. Finally, the combina- tional frameworks have been developed to provide a much has also been validated by ANSYS in a liver simulation sys- more ﬂexible and multifunctional environment for develop- tem [56]. The Abaqus software was also used for validation ing a whole system. MATLAB is a powerful combinational purpose [17, 24, 32]. Other used FE packages relate to MSC framework including a facial analysis toolbox used for facial NASTRAN 2003 which was used by Yarnitzky et al. [58] and LS-DYNA™ which was used by Joldes et al. [15]. Note expression analysis [42]; a toolbox called iso2mesh was used to generate a tetrahedral mesh of a brain and its tumor [7]; an that open source packages were also employed. The SOFA artiﬁcial neuro network toolbox was employed to train the framework was the execution environment for performance force-deformation data [7]; an optimization toolbox was evaluation between the developed method and the previous used to obtain optimal parameters of simulation models that modelling methods [26]. In addition to geometrical and model validations, user acceptability/safety validation needs represent for simulated physical quantities; the OpenGL graphic library could be supported in the MATLAB environ- to be performed to evaluate the quality of interfaces between ment for simulating interaction between soft-tissue model the system and its users in real clinical applications. One of and surgical tools [47]. An Android programming platform the most popular schemes of this validation level is to collect was also used [52]. Additionally, supporting for FEM physi- feedbacks from experts and patients who have been experi- enced with the developed system. Ho et al. validated their vir- cal modelling, the Fast FE Modelling Software Platform [14] and GetFEM++ [3] were also employed. For parallel tual reality myringotomy simulation system by a face-validity threading, the RTAI-patched Linux was used for satisfying study in which a validated questionnaire was delivered to eight otolaryngologists and four senior otolaryngology res- the hard real-time requirements [18]. Other powerful and more multifunctional system frameworks are the SOFA idents for evaluating the system after a long period inter- action with the simulator [6]. Tonutti et al. conducted [13, 26, 29, 33] and CHAI3D [6] frameworks. In fact, they support various libraries and modules for implementing a their validation procedure on surgeons with and without complete simulation system including input/output inter- implementing the developed system, and the diﬀerence results were evaluated for proving the eﬀectiveness of the action device drivers, geometrical model libraries, model- ling algorithms libraries, and graphic rendering modules. simulation system [7]. 3.8. Clinical Validations. To translate the outcomes of the 4. Discussion developed simulation systems into clinical routine practices, a systematic validation must be required. All validation 4.1. Computational Approaches. The FE modelling methods eﬀorts done in the literature were ordered as three validation and its variations have been commonly used for developing levels: geometrical validations, model behavior validations, soft-tissue deformation models. However, the trade-oﬀ and user acceptability/safety validations. between system accuracy and computation speed remains a The most important components in a simulation system challenging issue. The FEM can simulate deformations for are the geometrical and physical models. To accurately simu- complex soft tissues [67, 68]. In particular, a commercial late the target soft tissues, the geometrical appearances of FE solver was commonly used to evaluate the accuracy of a both the simulated models and real objects must be well- new FE algorithm [21, 22, 24, 32]. However, using the FE ﬁtted. Geometries at a speciﬁc state are commonly compared method, real-time requirements are only satisﬁed if being with the real in vivo/in vitro data acquired from a relatively modelled with a smaller number of elements [30, 35] or being accurate measurement method. CT/MRI were usually used accelerated by GPU implementation [23, 24]. In general, the to reconstruct the real 3D geometrical models of the tissues computational cost of the FEM increases exponentially with of interest. Due to expensive processing time and resources, the expansion of the number of nodes especially in case of this scheme is just suitable for oﬄine geometrical validation. simulation of the nonlinear materials. Consequently, various For example, real data from patients under maxillofacial development techniques have been developed to improve surgeries were stored and compared with the predicting their computation speeds. The most used approach was the appearances for improving the reproducibility capacity of precomputation-based technique. This approach has been the simulation system [43]. Moreover, the soft-tissue phan- proven as a fast and robust technique for simulating 24 Applied Bionics and Biomechanics This method could be considered for the compromising solu- deformations in real time, but large deformations with com- plex material properties and constitutive laws and topological tion between the biomechanical accuracy and computation changes on the ﬂy could not be handled during the system eﬃciency in real time. A diﬀerent technique was point collocation-based method of ﬁnite sphere (PCMFS) [46]. iterations because the trained model cannot be updated online [3, 7, 8, 12, 19, 20, 31, 36, 37]. Other FEM variations By just focusing on the local region of interests, the simula- could signiﬁcantly improve the performance of the FEM. tion time could be decreased signiﬁcantly, but the method One of them was the case of linearizing the kinematic of for detecting the region of interest was still not deﬁned eﬀec- the simulated object [14] in which the model could be cut fas- tively. More globally, the method called inverse dynamic computation (IDC) [5] could compute the muscle tensions ter than the original model using FEM, but the speed was not fast enough for realistic visualization in medical applications. based on the external data from sensors. Despite of the The idea of dividing a FEM mesh into multiple submeshes to acceptable accuracy, the method could not analyze a single be executed in parallel [18] could initially increase the com- muscle. This idea could be found in the method called elas- putation speeds, but this method leaded to the limited num- tic-plus-muscle-distribution-based (E+MD) [51] in which the facial expression could be used for investigating the inter- ber of threads being able to handle on a real-time operation system. Other development methods such as matrix system nal muscle tensions. A recent method called time-saving reduction (MSR-FEM) [21] and the order reduction method volume-energy-conserved ChainMail (TSVE-ChainMail) (ORM-FEM) [29] based on the reduction of the FEM’s stiﬀ- [54] was powerful in handling both topological changes and ness matrix could improve the processing time, but they just simulating the isotropic, anisotropic, and heterogeneous materials at the real-time rate. At this stage, the real-time simulated small deformations. The total Lagrangian algo- rithm in conjunction with FEM [22] and its modiﬁcation deformations could be achieved but the interactions among known as total Lagrangian explicit dynamic (TLED-FEM) the simulated objects remains a diﬃcult task. To overcome [16] allowed element precomputations, so a less computation this drawback surface-based methods (e.g., boundary ele- cost would be required for each time step. It is important to ment method (BEM) [4], Laplacian surface deformation (LSD) [41], and Marquardt radial basis meshless method note that hyperelastic and viscoeleastic constitutive models were implemented with explicit time integration schemes in (MRM) [56]) have been proposed. These approaches could a straightforward manner using homemade or commercial estimate the internal deformations based on the surface changes, and it could handle the interaction between mod- FE solvers. Moreover, multiplicative Jacobian Energy Decom- position (MJED-FEM) [26] with implicit time integration elled soft tissues through surface interactions, but they were not able to simulate inhomogeneous materials, nonlinear schemes could be used to model hyperelastic, viscoelastic, and poroelastic behaviors of the soft tissues. However, these elastics, and topological changes on the ﬂy. In addition, methods could not handle interactions with other simulated model cutting and needle penetration issues were also stud- ied using the extended ﬁnite element method (XFEM) and objects and topological changes. The topological changes could be handled in the method proposed by Turkiyyard meshfree-based approaches for soft tissues. In particular, the extended ﬁnite element method (XFEM) has been used et al. [28], but they could not solve eﬀectively the cured cuts, partial cuts, and multiple cuts inside elements. More eﬀec- to study complex hard tissue (tooth [69], maxillary molar, tively for simulating the topological changes was element- and endodontic cavities [70]) models with fracture and crack propagation behaviors and soft tissue (cornea [71]) models by-element precondition conjugate gradient FEM (EbE PCG-FEM) [32] method, but it was not suitable for simulating with cutting simulation. This open new avenue to model bio- logical tissues with more complex interaction behaviors. the heterogeneous materials. Another potential method called preconditioning FEM (pre-cond FEM) [33] could solve this In addition, many studies have been conducted for the problem dramatically. It could both simulate the topological implemented model based on developed soft-tissue deforma- tion method on to the GPU-parallel computing platform, changes and the haptic feedback of homogeneous and hetero- geneous materials with acceptable accuracy and real-time and they can all achieve much better accelerations when frame rates. compared with the conventional developing approach. Not On the other hand, the meshfree-based techniques have all developed modeling methods are suitable for this been achieved great attention in the recent years. All the approach, so the implicit time integration of nonlinear FEM method has been proven to be the most suitable for par- meshfree-based methods have been very fast and highly adaptive to topological changes, but they are less realistic allel implementation. Furthermore, the additional reconﬁ- than the mesh-based methods from biomechanical point of gurations must be approved to the current methods to view. The most popular meshfree-based method was mass- adapt with the implemented hardware platforms. When the spring system (MSM) [2, 6, 38, 44, 45]. This method could model developing approach reaches its limitation, new implementation strategies will be necessary for accelerating handle the deformations and topological changes, but it could not simulate accurately with nonlinear material char- their current computation performances. acteristics. The improvements of MSM method were the It is important to note that the computation speed and mass-tensor method (MTM) [43] and mass-spring-damper resources depend on each particular application (e.g., surgi- (MSD) method [47]. They could handle the nonlinear mate- cal planning or surgical simulation) of soft-tissue deforma- tion systems. For example, real-time soft-tissue deformation rial more eﬀectively than the MSM due to the use of nonlin- ear mass springs in the method. Another improvement of behavior, high-speed device interaction, and skill-based MSM was the hyperelastic mass link (HEML) method [53]. training ability could be more important criteria to be Applied Bionics and Biomechanics 25 and right scenes [2, 6]. Even more realistically, the haptic achieved for a computer-aided surgical simulation system. Besides, surgical planning system focuses on the whole feedback devices receive calculated haptic forces from sim- workﬂow from data acquisition and previsualization of a ulated models to create collision feeling for human tactile [3, 12, 18, 20, 27, 44, 53, 57]. Consequently, the cooperation speciﬁc surgical intervention and then predeﬁne the optimal surgical steps. between the 3D viewers and the haptic feedback devices will Generally speaking, a large range of methods were devel- become much more powerful in generating realistic sensa- oped to simulate the soft-tissue deformations. Each method tions for humans [2, 6]. In particular, force-feedback devices showed its robustness and accuracy for a speciﬁc case study. have been commonly used for many medical applications (e.g., surgical simulation, surgical trainings, or minimally There exists no universal methods, and the selection of the methods depends directly on the application. It is important invasive surgeries [18, 44, 53]). Force-feedback devices have to note that real-time deformations with topological changes been ﬂexibly cooperated with various types of virtual surgical on the ﬂy, and accurate object interactions remain challeng- tools (e.g., virtual haptic interface point (HIP), the virtual ing issues. One of the potential solutions relates to the use blade, or the virtual scalpel). The most widely haptic device used in the literature is the SensAble™ PHANTOM Desk- of multiple modeling methods in a whole simulation system. However, an eﬀective cooperation strategy should be estab- top™ haptic device, and its ﬂexibilities are dependent on the lished, and the requirement of advanced computational number of DOFs. It is important to note that to simulate resources needs to be satisﬁed. force feedbacks realistically, the haptic forces must be esti- Finally, the computation speed of a soft-tissue simulation mated and transferred to the force-feedback device at speeds from 500 Hz to 1000 Hz, so this means that the computation system depends strongly on the use of constitutive behavior laws for modeling soft-tissue physiology. Elastic, hyperelas- speeds of simulated models must be faster than those speeds tic, and viscoelastic laws were commonly used in the devel- [18, 57]. Furthermore, a separate controller must be installed oped real-time simulation systems for the upper/lower limb and executed one or multiple computer system to keep the muscle, facial muscle, liver, and skin tissues. It is important real-time computation speed [3, 19, 27, 44, 53]. On the other hand, several biomechanical quantities have to note that more complex constitutive laws such as electro- mechanical models could be used in general for modeling not been measured directly from the biomechanical sensors, the skeletal muscle [68] or myocardium [72, 73]. However, so simulated models are often used to infer internal physical these complex models deal with additional computational characteristics based on external knowledge. For example, in need and requirements to reach a real-time ability for medi- the case of musculoskeletal tracking, EMG sensors are often fused with musculoskeletal models for inferring individual cal simulation systems. Linear and nonlinear stress-strain relationships were described in the elastic material. Hypere- muscle tensions according to the markers’ motions, which lastic material was described using Neo-Hookean and are tracked by the 3D optical camera system [5, 58]. The soft-tissue physical parameters can also be inferred from Mooney-Rivlin formulations. It is important to note that some additional components were integrated into linear elas- soft-tissue deformation models by the movements of surface markers instead of direct measurements from the sensors. tic law to improve the computation speed and model accu- racy. For example, the combination of a linear elastic law The surface makers are proven to be very robust and ﬂexible with a corotational method was performed (Courtecuisse for estimating outside deformations, but the limited number of markers being able to put on a soft-tissue surface leads to et al. [33]) or an extra mass-spring model was integrated into a linear elastic law (Zhu and Gu [59]). Regarding all analyzed decrease the estimated deformation resolutions and so are the resultant calculations [31, 49, 55]. The 3D scanners such simulation systems for soft tissues, the most used law is the linear elastic one. The use of more complex laws (hyperelastic as MRI/CT scanners [8, 12, 26] and 3D ultrasonic scanners and viscoelastic) leads to a larger number of model parame- [4] have been employed for detail surface reconstruction in 3D spaces, but their slow acquisition times (in case of ters and of course computation speed. MRI/CT scanners), lacking of surface characteristics (in case 4.2. Interaction Devices in Real-Time Simulation Systems. of 3D laser scanners and ultrasonic scanners), and harmful There have been various kinds of interaction devices contrib- infections to human health (in case of CT and laser scanners) uting diﬀerently to the system’s reality and computation per- make them not suitable for tracking external deformations of soft tissue in real time and in long-term use. This issue was formance. While the output interaction devices mainly provide realistic visualizations and reactions to human initially resolved by the combination between the 2D optical senses, the input interactions have the fundamental involve- cameras and the X-ray images for adding more surface char- ment to the computation performance, especially in both acteristics [48], but the appearances were static and could not model accuracy and computation speed. Regarding the out- estimate deformations on the ﬂy. Consequently, other devices having the ability of acquiring both detail surface put interaction devices, the computer screens display appear- ances of simulated models and their deformations when deformations in 3D spaces and surface characteristics online interacted with virtual surgical instruments and/or other sur- are substantially required for improving computation speeds rounding structures [6]. However, their lacking of depth and model accuracy of soft-tissue simulation systems. information makes visualizations possible only in 2D space. 4.3. Suitable Execution Scheme in Simulation Systems. To The 3D viewers can complement this drawback. Like human visions, this interaction devices can create 3D virtual sensa- manage the data transmission from/to I/O interaction tion for human vision based on the diﬀerences between left devices and to compute the simulated model in an optimal 26 Applied Bionics and Biomechanics Finally, the user acceptability/safety validation was per- way, a suitable system execution scheme must be developed. There are two main system execution schemes. In the formed with the end users including patients, trainees, and distribution-based scheme [12, 31, 44], system tasks are experts through questionnaires. Note that this approach is relatively subjective and qualitative. In addition to these val- highly parallelized in multiple computing machines, which are interfaced through a limited bandwidth and slow- idation levels, system validation should be performed in transmission environments. This scheme allows system tasks which the whole system was evaluated rather than each sys- to execute independently and take advantages of multiple tem’s components. This stage targets at analyzing system computing hardware, but the problems appear when having functions, system robustness, and system computation per- formances during short-term and/or long-term working delays in communication between multiple machines. Thus, data transmissions are still not fast enough for eﬀectively durations. While the system functions are relatively easy to communicating among multiple computer systems. This verify by comparing with the proposed development func- issue has been initially solved by numerous attempts such tions at the designing stage, the system robustness and com- as high-speed ethernets and high-speed data transmission putation performances must be tested after short-term and long-term working durations. Although this validation pro- protocols, but they are not eﬃcient enough for transmitting large information in real-time. In the multithread scheme cess is necessary for a stable and robust system, rarely, studies [2, 3, 19, 53], system tasks are executed on multiple comput- in the literature conduct this validation. ing threads. Because all threads are connected through a very high-speed internal bus, there is nearly a delay in data trans- 5. Current Trends, Limitations and mission among threads. However, because of the limitation Future Recommendations of computation strength and memory capacity of a single thread inside a computer system, the simulation task(s) must The trends of the current computational approaches relate to be simpliﬁed and optimized to be able to execute on a single (1) mathematical formulation of physical laws applicable on thread. This can be a challenging task for model develop- image-based soft tissue geometries, (2) real-time simulation ments and implementations. Fortunately, this challenge can achievement of soft-tissue deformation with simple constitu- be easily resolved by the development of hardware technol- tive laws, and (3) model implementation on speciﬁc hard- ogy with more threads integrated on a single CPU or even ware conﬁguration to speed up the computational time. more CPU facilitated on a single computer system. In addi- However, soft-tissue behavior is commonly anisotropic, vis- tion, cooperation of the two execution schemes was also coelastic, inhomogeneous, and nearly incompressible with found in the literature [13, 18]. In this case, various types of large deformation. In fact, the consideration of all physiolog- data acquisition boards have been developed to fast manage ical aspects is practically diﬃcult, particularly for a real-time the input/output data streams. These boards are designed in simulation system. Thus, modeling assumptions related to a mobile hardware and can be easily plugged in to a comput- constitutive laws, geometrical discretization, and boundary ing machine through a speciﬁc high-speed transmission and loading conditions were commonly performed for a spe- channel and a software driver. Consequently, this system ciﬁc application. Further studies need to be investigated to conﬁguration can take advantages of both distribution- develop more accurate computational approaches for simu- based and multithread-based execution schemes. lating complex soft-tissue behaviors in real-time conditions. The hybrid modelling approach in combining several 4.4. Clinical Validations. Generally speaking, the clinical val- methods is a potential solution leading to maximize the idation is the ﬁnal system development stage to determine advantage of each method and overcome the limitations of whether the simulation system is acceptably suitable for clin- the other. ical routine practices. Current clinical validation procedures Concerning the interaction devices, the ability of acquir- were grouped into three levels: geometrical validations, ing multiple types of data both in real-time and accurate model behavior validations, and user acceptability/safety val- manner and the portability of sensors are the current trends. idations. Regarding geometrical and model behavior valida- Multiple sensors could be embedded into a single well- tions, the validation data have been commonly acquired calibrated structure and worked as an independent conﬁgu- from standard simulation software, phantom soft-tissue ration. These types of sensors, therefore, are more accurate organs, or postoperation data. There is a lack of in vivo data and faster than manually calibrated sensor systems. In fact, for accurately validating the simulation system in real medi- some multiple function sensors such as the KINECT™ devel- cal environments. The use of accurate CT/MRI data is prom- oped by Microsoft®, XTION™ developed by Asus, and other ising, but this approach is not suitable for online validation. well calibrated stereo cameras are good recommendations for The use of standard simulation software to validate the phys- this requirement. However, current sensors are diﬃcult to ical behaviors of simulated models also faces some problems. acquire deep information on the soft tissues, which are cru- It is important to note that most of soft-tissue materials (e.g., cial for in vivo modeling and simulation. In particular, there muscle, fat, and skin) are unavailable in these types of soft- have been no sensors having the ability of acquiring these ware. Thus, only simpliﬁed behaviors were validated with data in real time, so there is a need for a new type of sensor classical mechanical laws (e.g., linear elastic or hyperelastic that can get the internal structures and/or textures in real laws). Consequently, more experimental protocols should time. In fact, complex data processing schemes need to be be investigated to characterize the soft-tissue behaviors investigated in the future to study the external-internal rela- and use them for enhancing model behavior validations. tionship of the soft tissues leading to a predictive solution of Applied Bionics and Biomechanics 27 tions. This review provides useful information to describe internal structures from external information. Statistical shape modeling (SSM) or artiﬁcial intelligence- (AI-) based how each aspect has been developed and how they have been approaches are potential methods for such complex objective. cooperated for both executing in real time and keeping real- Regarding the system architecture and execution scheme, istic behaviors of soft tissues. By clearly analysing advan- the availability of powerful and open frameworks for medical tages and drawbacks in each system development aspect, imaging processing (e.g., 3D Slicer), data visualization (e.g., this review paper can be used as a reference guideline for OpenGL, VTK), and simulation (e.g. SOFA) is the current system developers to choose their suitable system’s compo- trend, which can speed up the development of new systems. nents while developing soft-tissue simulation systems. However, the compatibility between these frameworks Finally, this review paper identiﬁed some recommendations becomes a potential drawback. To deal with this obstacle, for future researches. the community should work together to deﬁne a common computational protocol and promote its use within any sys- Conflicts of Interest tem development for future applications. In addition, all developed system execution schemes are very hard to pro- The authors declare that there is no conﬂict of interest gram without the help of system frameworks. Most used regarding the publication of this paper. system frameworks mainly supported for programming mul- tithread schemes rather than distributed schemes and combi- Acknowledgments nation schemes. Moreover, they did not well manage the memories between internal threads. For further recommen- This work was carried out and funded in the framework of dations, more system frameworks should be developed for the Labex MS2T. It was supported by the French Govern- supporting the communication between threads. Further- ment, through the program “Investments for the future” more, frameworks for programming, the distributed schemes managed by the National Agency for Research (Reference also need to be investigated for supporting connection and ANR-11-IDEX-0004-02). We acknowledged also the Région data transmission between multiple computing machines. Hauts-de-France for funding. More software development kits (SDKs) should be developed for general sensors such as single cameras, stereo cameras, References and laser scanners so that deeper information could be extracted and estimated. In addition, to translate developed [1] W. Sun and P. Lal, “Recent development on computer aided systems into clinical routine practices, software development tissue engineering — a review,” Computer Methods and Pro- workﬂow dedicated for medical application should be grams in Biomedicine, vol. 67, no. 2, pp. 85–103, 2002. followed to ensure a high-quality and reliable medical soft- [2] J. Brown, S. Sorkin, J.-C. Latombe, K. Montgomery, and ware for the beneﬁt of involved patients and clinicians. M. Stephanides, “Algorithmic tools for real-time microsurgery Finally, multilevel validation becomes an avoidable task simulation,” Medical Image Analysis, vol. 6, no. 3, pp. 289–300, when developing a real-time medical simulation with soft- tissue deformation. Quantitative assessment of accuracy is [3] I. Peterlík, M. Sedef, C. Basdogan, and L. Matyska, “Real-time also focused. However, when developing a soft-tissue sim- visio-haptic interaction with static soft tissue models having ulation system, a systematic validation procedure must be geometric and material nonlinearity,” Computers & Graphics, simultaneously proposed. Speciﬁcally, a system must be vol. 34, no. 1, pp. 43–54, 2010. sequentially validated through all validation levels. The [4] C. Monserrat, U. Meier, M. Alcaniz, F. Chinesta, and M. C. type of validating data for each validation process should be Juan, “A new approach for the real-time simulation of tissue clearly deﬁned, and acquisition processes should also be deformations in surgery simulation,” Computer Methods and Programs in Biomedicine, vol. 64, no. 2, pp. 77–85, 2001. planned. Moreover, an assessment program could be included in the system to evaluate the development progress of the [5] A. Murai, K. Kurosaki, K. Yamane, and Y. Nakamura, “Muscu- loskeletal-see-through mirror: computational modeling and users during each level of trainings. In user acceptability/saf- algorithm for whole-body muscle activity visualization in real ety validations, a supervising task programmed to run simul- time,” Progress in Biophysics and Molecular Biology, vol. 103, taneously with the simulation system could be used to track no. 2-3, pp. 310–317, 2010. users’ behaviors. Additionally, a function that include ques- [6] A. K. Ho, H. Alsaﬀar, P. C. Doyle, H. M. Ladak, and S. K. tionnaires for evaluating user acceptability could be included Agrawal, “Virtual reality myringotomy simulation with real- in the system’s functions, so the results are automatically time deformation: development and validity testing,” The processed and sent to developers for further development. Laryngoscope, vol. 122, no. 8, pp. 1844–1851, 2012. [7] M. Tonutti, G. Gras, and G. Z. Yang, “A machine learning 6. Conclusion approach for real-time modelling of tissue deformation in image-guided neurosurgery,” Artiﬁcial Intelligence in Medi- The present review paper was conducted to summarize the cine, vol. 80, pp. 39–47, 2017. literature works about real-time soft-tissue deformation sys- [8] D. Lorente, F. Martínez-Martínez, M. J. Rupérez et al., “A tems. Throughout this review, studies relating to real-time framework for modelling the biomechanical behaviour of the soft-tissue simulations have been analysed according to four human liver during breathing in real time using machine system engineering aspects: computational approaches, learning,” Expert Systems with Applications, vol. 71, pp. 342– interaction devices, system architectures, and clinical valida- 357, 2017. 28 Applied Bionics and Biomechanics [24] A. Wittek, G. Joldes, M. Couton, S. K. Warﬁeld, and K. Miller, [9] H. Delingette, “Toward realistic soft-tissue modeling in medical simulation,” Proceedings of the IEEE, vol. 86, no. 3, “Patient-speciﬁc non-linear ﬁnite element modelling for pre- pp. 512–523, 1998. dicting soft organ deformation in real-time; application to non-rigid neuroimage registration,” Progress in Biophysics [10] A. Nealen, M. Müller, R. Keiser, E. Boxerman, and M. Carlson, and Molecular Biology, vol. 103, no. 2-3, pp. 292–303, 2010. “Physically based deformable models in computer graphics,” Computer Graphics Forum, vol. 25, no. 4, pp. 809–836, [25] R. J. Lapeer, P. D. Gasson, and V. Karri, “Simulating plastic 2006. surgery: from human skin tensile tests, through hyperelastic ﬁnite element models to real-time haptics,” Progress in Bio- [11] D. Moher, A. Liberati, J. Tetzlaﬀ, and D. G. Altman, “Preferred physics and Molecular Biology, vol. 103, no. 2-3, pp. 208–216, reporting items for systematic reviews and meta-analyses: the PRISMA statement,” International Journal of Surgery, vol. 8, no. 5, pp. 336–341, 2010. [26] S. Marchesseau, T. Heimann, S. Chatelin, R. Willinger, and H. Delingette, “Fast porous visco-hyperelastic soft tissue [12] S. Cotin, H. Delingette, and N. Ayache, “Real-time elastic model for surgery simulation: application to liver surgery,” deformations of soft tissues for surgery simulation,” IEEE Progress in Biophysics and Molecular Biology, vol. 103, Transactions on Visualization and Computer Graphics, vol. 5, no. 2-3, pp. 185–196, 2010. no. 1, pp. 62–73, 1999. [27] H. Courtecuisse, H. Jung, J. Allard, C. Duriez, D. Y. Lee, and [13] J. Allard, S. Cotin, F. Faure et al., “SOFA - an open source S. Cotin, “GPU-based real-time soft tissue deformation with framework for medical simulation,” Stud Health Technol cutting and haptic feedback,” Progress in Biophysics and Inform, vol. 125, pp. 13–18, 2007. Molecular Biology, vol. 103, no. 2-3, pp. 159–168, 2010. [14] J. Berkley, G. Turkiyyah, D. Berg, M. Ganter, and S. Weghorst, [28] G. M. Turkiyyah, W. B. Karam, Z. Ajami, and A. Nasri, “Mesh “Real-time ﬁnite element modeling for surgery simulation: an cutting during real-time physical simulation,” Computer- application to virtual suturing,” IEEE Transactions on Visual- Aided Design, vol. 43, no. 7, pp. 809–819, 2011. ization and Computer Graphics, vol. 10, no. 3, pp. 314–325, 2004. [29] S. Niroomandi, I. Alfaro, E. Cueto, and F. Chinesta, “Account- ing for large deformations in real-time simulations of soft tis- [15] G. R. Joldes, A. Wittek, and K. Miller, “Suite of ﬁnite element sues based on reduced-order models,” Computer Methods algorithms for accurate computation of soft tissue deformation and Programs in Biomedicine, vol. 105, no. 1, pp. 1–12, for surgical simulation,” Medical Image Analysis, vol. 13, no. 6, pp. 912–919, 2009. [30] T. Wu, A. P. L. Hung, P. Hunter, and K. Mithraratne, “Model- [16] K. Miller, G. Joldes, D. Lance, and A. Wittek, “Total Lagrang- ling facial expressions: a framework for simulating nonlinear ian explicit dynamics ﬁnite element algorithm for computing soft tissue deformations using embedded 3D muscles,” Finite soft tissue deformation,” Communications in Numerical Elements in Analysis and Design, vol. 76, pp. 63–70, 2013. Methods in Engineering, vol. 23, no. 2, pp. 121–134, 2007. [31] K. Morooka, Y. Nakasuka, R. Kurazume, X. Chen, [17] Z. A. Taylor, M. Cheng, and S. Ourselin, “High-speed nonlin- T. Hasegawa, and M. Hashizume, “Navigation system with ear ﬁnite element analysis for surgical simulation using real-time ﬁnite element analysis for minimally invasive sur- graphics processing units,” IEEE Transactions on Medical gery,” in 2013 35th Annual International Conference of the Imaging, vol. 27, no. 5, pp. 650–663, 2008. IEEE Engineering in Medicine and Biology Society (EMBC), [18] M. A. Audette, V. Hayward, O. Astley, M. Doyon, G. A. pp. 2996–2999, Osaka, Japan, 2013. McCallister, and K. Chinzei, “A PC-based system architecture [32] R. Maﬁ and S. Sirouspour, “GPU-based acceleration of com- for real-time ﬁnite element-based tool-speciﬁc surgical simula- putations in nonlinear ﬁnite element deformation analysis,” tion,” International Congress Series, vol. 1268, pp. 378–383, International Journal for Numerical Methods in Biomedical Engineering, vol. 30, no. 3, pp. 365–381, 2014. [19] M. Sedef, E. Samur, and C. Basdogan, “Real-time ﬁnite- element simulation of linear viscoelastic tissue behavior based [33] H. Courtecuisse, J. Allard, P. Kerfriden, S. P. A. Bordas, S. Cotin, and C. Duriez, “Real-time simulation of contact and on experimental data,” IEEE Computer Graphics and Applica- tions, vol. 26, no. 6, pp. 58–68, 2006. cutting of heterogeneous soft-tissues,” Medical Image Analysis, vol. 18, no. 2, pp. 394–410, 2014. [20] G. Sela, J. Subag, A. Lindblad, D. Albocher, S. Schein, and [34] V. Strbac, J. Vander Sloten, and N. Famaey, “Analyzing the G. Elber, “Real-time haptic incision simulation using FEM- based discontinuous free-form deformation,” Computer-Aided potential of GPGPUs for real-time explicit ﬁnite element anal- ysis of soft tissue deformation using CUDA,” Finite Elements Design, vol. 39, no. 8, pp. 685–693, 2007. in Analysis and Design, vol. 105, pp. 79–89, 2015. [21] M. García, O. D. Robles, L. Pastor, and A. Rodríguez, “MSRS: a [35] A. Karami, M. Eghtesad, and S. A. Haghpanah, “Prediction of fast linear solver for the real-time simulation of deformable objects,” Computers & Graphics, vol. 32, no. 3, pp. 293–306, muscle activation for an eye movement with ﬁnite element modeling,” Computers in Biology and Medicine, vol. 89, pp. 368–378, 2017. [22] G. R. Joldes, A. Wittek, and K. Miller, “An eﬃcient hourglass control implementation for the uniform strain hexahedron [36] F. Martínez-Martínez, M. J. Rupérez-Moreno, M. Martínez- using the total Lagrangian formulation,” Communications in Sober et al., “A ﬁnite element-based machine learning Numerical Methods in Engineering, vol. 24, no. 11, pp. 1315– approach for modeling the mechanical behavior of the breast 1323, 2008. tissues under compression in real-time,” Computers in Biology and Medicine, vol. 90, pp. 116–124, 2017. [23] G. R. Joldes, A. Wittek, and K. Miller, “Real-time nonlinear ﬁnite element computations on GPU – application to neuro- [37] V. Luboz, M. Bailet, C. Boichon Grivot et al., “Personalized surgical simulation,” Computer Methods in Applied Mechanics modeling for real-time pressure ulcer prevention in sitting pos- and Engineering, vol. 199, no. 49-52, pp. 3305–3314, 2010. ture,” Journal of Tissue Viability, vol. 27, no. 1, pp. 54–58, 2018. Applied Bionics and Biomechanics 29 [53] F. Goulette and Z.-W. Chen, “Fast computation of soft tissue [38] L. P. Nedel and D. Thalmann, “Real time muscle deforma- tions using mass-spring systems,” in Proceedings. Computer deformations in real-time simulation with hyper-elastic mass Graphics International (Cat. No.98EX149), pp. 156–165, links,” Computer Methods in Applied Mechanics and Engineer- Hannover, Germany, 1998. ing, vol. 295, pp. 18–38, 2015. [39] T. Goto, W. S. Lee, and N. Magnenat-thalmann, “Facial feature [54] J. Zhang, Y. Zhong, J. Smith, and C. Gu, “A new ChainMail extraction for quick 3D face modeling,” Signal Processing: approach for real-time soft tissue simulation,” Bioengineered, Image Communication, vol. 17, no. 3, pp. 243–259, 2002. vol. 7, no. 4, pp. 246–252, 2016. [40] C. Bonamico, M. Costa, F. Lavagetto, and R. Pockaj, “Real- [55] A. Woodward, Y. H. Chan, R. Gong et al., “A low cost frame- time MPEG-4 facial animation with 3D scalable meshes,” work for real-time marker based 3-D human expression Signal Processing: Image Communication, vol. 17, no. 9, modeling,” Journal of Applied Research and Technology, pp. 743–757, 2002. vol. 15, no. 1, pp. 61–77, 2017. [41] O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rössl, and [56] J. Zhou, Z. Luo, C. Li, and M. Deng, “Real-time deformation of H.-P. Seidel, “Laplacian surface editing,” in Proceedings of the human soft tissues: A radial basis meshless 3D model based on 2004 Eurographics/ACM SIGGRAPH symposium on Geometry Marquardt's algorithm,” Computer Methods and Programs in processing - SGP '04, pp. 175–184, New York, NY, USA, 2004. Biomedicine, vol. 153, pp. 237–252, 2018. [42] N. P. Chandrasiri, T. Naemura, M. Ishizuka, H. Harashima, [57] S. Cotin, H. Delingette, and N. Ayache, “A hybrid elastic model and I. Barakonyi, “Internet communication using real-time for real-time cutting, deformations, and force feedback for sur- facial expression analysis and synthesis,” IEEE Multimedia, gery training and simulation,” The Visual Computer, vol. 16, vol. 11, no. 3, pp. 20–29, 2004. no. 8, pp. 437–452, 2000. [43] W. Mollemans, F. Schutyser, N. Nadjmi, and P. Suetens, “Very [58] G. Yarnitzky, Z. Yizhar, and A. Gefen, “Real-time subject- fast soft tissue predictions with mass tensor model for maxillo- speciﬁc monitoring of internal deformations and stresses in facial surgery planning systems,” International Congress Series, the soft tissues of the foot: a new approach in gait analysis,” vol. 1281, pp. 491–496, 2005. Journal of Biomechanics, vol. 39, no. 14, pp. 2673–2689, 2006. [44] P. Chen, K. E. Barner, and K. V. Steiner, “A displacement [59] B. Zhu and L. Gu, “A hybrid deformable model for real-time driven real-time deformable model for haptic surgery simu- surgical simulation,” Computerized Medical Imaging and lation,” in 2006 14th Symposium on Haptic Interfaces for Graphics, vol. 36, no. 5, pp. 356–365, 2012. Virtual Environment and Teleoperator Systems, pp. 499–505, [60] I. S. Pandzic and R. Forchheimer, MPEG-4 Facial Animation: Alexandria, VA, USA, 2006. The Standard, Implementation and Applications, John Wiley [45] M. López-Cano, J. Rodríguez-Navarro, A. Rodríguez-Baeza, & Sons, 2003. M. Armengol-Carrasco, and A. Susín, “A real-time dynamic [61] M. Doblaré, E. Cueto, B. Calvo, M. A. Martínez, J. M. Garcia, 3D model of the human inguinal region for surgical educa- and J. Cegoñino, “On the employ of meshless methods in bio- tion,” Computers in Biology and Medicine, vol. 37, no. 9, mechanics,” Computer Methods in Applied Mechanics and pp. 1321–1326, 2007. Engineering, vol. 194, no. 6-8, pp. 801–821, 2005. [46] Y. J. Lim and S. De, “Real time simulation of nonlinear tissue [62] G. Y. Zhang, A. Wittek, G. R. Joldes, X. Jin, and K. Miller, “A response in virtual surgery using the point collocation-based three-dimensional nonlinear meshfree algorithm for simulat- method of ﬁnite spheres,” Computer Methods in Applied ing mechanical responses of soft tissue,” Engineering Analysis Mechanics and Engineering, vol. 196, no. 31-32, pp. 3011– with Boundary Elements, vol. 42, pp. 60–66, 2014. 3024, 2007. [63] M. H. Doweidar, B. Calvo, I. Alfaro, P. Groenenboom, and [47] E. Basafa and F. Farahmand, “Real-time simulation of the non- M. Doblaré, “A comparison of implicit and explicit natural ele- linear visco-elastic deformations of soft tissues,” International ment methods in large strains problems: application to soft Journal of Computer Assisted Radiology and Surgery, vol. 6, biological tissues modeling,” Computer Methods in Applied no. 3, pp. 297–307, 2011. Mechanics and Engineering, vol. 199, no. 25-28, pp. 1691– [48] J. Wang, S. Liao, X. Zhu et al., “Real time 3D simulation for 1700, 2010. nose surgery and automatic individual prosthesis design,” [64] F. Roohbakhshan and R. A. Sauer, “Eﬃcient isogeometric thin Computer Methods and Programs in Biomedicine, vol. 104, shell formulations for soft biological materials,” Biomechanics no. 3, pp. 472–479, 2011. and Modeling in Mechanobiology, vol. 16, no. 5, pp. 1569– [49] X. Wan, S. Liu, J. X. Chen, and X. Jin, “Geodesic distance- 1597, 2017. based realistic facial animation using RBF interpolation,” [65] D. Kamensky, F. Xu, C. H. Lee, J. Yan, Y. Bazilevs, and M. C. Computing in Science & Engineering, vol. 14, no. 5, Hsu, “A contact formulation based on a volumetric potential: pp. 49–55, 2012. application to isogeometric simulations of atrioventricular [50] B. H. Le, M. Zhu, and Z. Deng, “Marker optimization for facial valves,” Computer Methods in Applied Mechanics and Engi- motion acquisition and deformation,” IEEE Transactions on neering, vol. 330, pp. 522–546, 2018. Visualization and Computer Graphics, vol. 19, no. 11, [66] T. S. Sørensen and J. Mosegaard, “An introduction to GPU pp. 1859–1871, 2013. accelerated surgical simulation,” in Biomedical Simulation, [51] Y. Zhang, W. Lin, B. Zhou, Z. Chen, B. Sheng, and J. Wu, M. Harders and G. Székely, Eds., pp. 93–104, Springer, Berlin, “Facial expression cloning with elastic and muscle models,” Heidelberg, 2006. Journal of Visual Communication and Image Representation, [67] T. T. Dao, A.-X. Fan, S. Dakpé, P. Pouletaut, M. Rachik, and vol. 25, no. 5, pp. 916–927, 2014. M. C. H. B. Tho, “Image-based skeletal muscle coordination: [52] Y. Weng, C. Cao, Q. Hou, and K. Zhou, “Real-time facial ani- case study on a subject speciﬁc facial mimic simulation,” Jour- mation on mobile devices,” Graphical Models, vol. 76, no. 3, nal of Mechanics in Medicine and Biology, vol. 18, no. 2, article pp. 172–179, 2014. 1850020, 2018. 30 Applied Bionics and Biomechanics [68] T. T. Dao and M.-C. H. B. Tho, “A systematic review of contin- uum modeling of skeletal muscles: current trends, limitations, and recommendations,” Applied Bionics and Biomechanics, vol. 2018, Article ID 7631818, 17 pages, 2018. [69] B. Wan, M. Shahmoradi, Z. Zhang et al., “Modelling of stress distribution and fracture in dental occlusal ﬁssures,” Scientiﬁc Reports, vol. 9, no. 1, p. 4682, 2019. [70] Y. Zhang, Y. Liu, Y. She, Y. Liang, F. Xu, and C. Fang, “The eﬀect of endodontic access cavities on fracture resistance of ﬁrst maxillary molar using the extended ﬁnite element method,” Journal of Endodontia, vol. 45, no. 3, pp. 316–321, [71] S. Niroomandi, I. Alfaro, D. González, E. Cueto, and F. Chinesta, “Real-time simulation of surgery by reduced- order modeling and X-FEM techniques,” International Journal for Numerical Methods in Biomedical Engineering, vol. 28, no. 5, pp. 574–588, 2012. [72] R. Avazmohammadi, D. S. Li, T. Leahy et al., “An integrated inverse model-experimental approach to determine soft tissue three-dimensional constitutive parameters: application to post-infarcted myocardium,” Biomechanics and Modeling in Mechanobiology, vol. 17, no. 1, pp. 31–53, 2018. [73] R. Avazmohammadi, J. S. Soares, D. S. Li, S. S. Raut, R. C. Gorman, and M. S. Sacks, “A contemporary look at biome- chanical models of myocardium,” Annual Review of Biomedi- cal Engineering, vol. 21, pp. 417–442, 2019. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Bionics and Biomechanics Hindawi Publishing Corporation http://www.deepdyve.com/lp/hindawi-publishing-corporation/a-systematic-review-of-real-time-medical-simulations-with-soft-tissue-Y1BVGBOUm0

Loading next page...

References (76)

R. Avazmohammadi, J. Soares, David Li, Samarth Raut, R. Gorman, M. Sacks (2019)
A Contemporary Look at Biomechanical Models of Myocardium.
Annual review of biomedical engineering, 21
W. Mollemans, F. Schutyser, N. Nadjmi, P. Suetens (2005)
Very fast soft tissue predictions with mass tensor model for maxillofacial surgery planning systems
, 1281
(2003)
Forchheimer, MPEG-4 Facial Animation: The Standard, Implementation and Applications
S. Cotin, H. Delingette, N. Ayache (1999)
Real-Time Elastic Deformations of Soft Tissues for Surgery Simulation
IEEE Trans. Vis. Comput. Graph., 5
T. Goto, Won-Sook Lee, N. Magnenat-Thalmann (2002)
Facial feature extraction for quick 3D face modeling
Signal Process. Image Commun., 17
Y. Weng, Chen Cao, Qiming Hou, Kun Zhou (2014)
Real-time facial animation on mobile devices
Graph. Model., 76
Tim Wu, A. Hung, P. Hunter, K. Mithraratne (2013)
Modelling facial expressions: A framework for simulating nonlinear soft tissue deformations using embedded 3D muscles
Finite Elements in Analysis and Design, 76
V. Strbac, J. Sloten, N. Famaey (2015)
Analyzing the potential of GPGPUs for real-time explicit finite element analysis of soft tissue deformation using CUDA
Finite Elements in Analysis and Design, 105
N. Chandrasiri, T. Naemura, M. Ishizuka, H. Harashima, I. Barakonyi (2004)
Internet communication using real-time facial expression analysis and synthesis
IEEE MultiMedia, 11
G. Joldes, A. Wittek, K. Miller (2009)
Suite of finite element algorithms for accurate computation of soft tissue deformation for surgical simulation
Medical image analysis, 13 6
D. Moher, A. Liberati, J. Tetzlaff, D. Altman (2009)
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement
Open Medicine, 3
A. Wittek, G. Joldes, Mathieu Couton, S. Warfield, K. Miller (2010)
Patient-specific non-linear finite element modelling for predicting soft organ deformation in real-time: application to non-rigid neuroimage registration.
Progress in biophysics and molecular biology, 103 2-3
R. Avazmohammadi, David Li, Thomas Leahy, Elizabeth Shih, J. Soares, J. Gorman, R. Gorman, M. Sacks (2017)
An integrated inverse model-experimental approach to determine soft tissue three-dimensional constitutive parameters: application to post-infarcted myocardium
Biomechanics and Modeling in Mechanobiology, 17
D. Lorente, F. Martínez-Martínez, M. Rupérez, M. Lago, M. Martínez-Sober, Pablo Escandell-Montero, J. Martínez-Martínez, S. Martínez-Sanchís, Antonio Serrano, Carlos Aranda, J. Martín-Guerrero (2017)
A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning
Expert Syst. Appl., 71
Pei Chen, K. Barner, K. Steiner (2006)
A Displacement Driven Real-Time Deformable Model For Haptic Surgery Simulation
2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems
A. Liberati D. Moher (2010)
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement,
International Journal of Surgery, 8
T. Dao, A. Fan, S. Dakpé, P. Pouletaut, M. Rachik, M. Tho (2018)
IMAGE-BASED SKELETAL MUSCLE COORDINATION: CASE STUDY ON A SUBJECT SPECIFIC FACIAL MIMIC SIMULATION
Journal of Mechanics in Medicine and Biology, 18
H. Delingette (1998)
Toward realistic soft-tissue modeling in medical simulation
Proc. IEEE, 86
K. Morooka, Yousuke Nakasuka, R. Kurazume, Xian Chen, T. Hasegawa, M. Hashizume (2013)
Navigation system with real-time finite element analysis for minimally invasive surgery
2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
Yihao Zhang, Weiyao Lin, Bing Zhou, Zhenzhong Chen, Bin Sheng, Jianxin Wu (2014)
Facial expression cloning with elastic and muscle models
ArXiv, abs/1503.00088
G. Joldes, A. Wittek, K. Miller (2010)
Real-Time Nonlinear Finite Element Computations on GPU - Application to Neurosurgical Simulation.
Computer methods in applied mechanics and engineering, 199 49-52
G. Yarnitzky, Z. Yizhar, Amit Gefen (2006)
Real-time subject-specific monitoring of internal deformations and stresses in the soft tissues of the foot: a new approach in gait analysis.
Journal of biomechanics, 39 14
M. Doweidar, B. Calvo, I. Alfaro, P. Groenenboom, M. Doblaré (2010)
A comparison of implicit and explicit natural element methods in large strains problems: Application to soft biological tissues modeling
Computer Methods in Applied Mechanics and Engineering, 199
Guiyong Zhang, A. Wittek, G. Joldes, X. Jin, K. Miller (2014)
A three-dimensional nonlinear meshfree algorithm for simulating mechanical responses of soft tissue
Engineering Analysis With Boundary Elements, 42
Mert Sedef, Evren Samur, C. Basdogan (2006)
Real-Time Finite-Element Simulation of Linear Viscoelastic Tissue Behavior Based on Experimental Data
IEEE Computer Graphics and Applications, 26
C. Bonamico, M. Costa, F. Lavagetto, R. Pockaj (2002)
Real-time MPEG-4 facial animation with 3D scalable meshes
Signal Process. Image Commun., 17
B. Le, Mingyang Zhu, Z. Deng (2013)
Marker Optimization for Facial Motion Acquisition and Deformation
IEEE Transactions on Visualization and Computer Graphics, 19
B. Zhu and L. Gu (2012)
A hybrid deformable model for real-time surgical simulation,
Computerized Medical Imaging and Graphics, 36
H. Courtecuisse, Hoeryong Jung, J. Allard, Christian Duriez, D. Lee, S. Cotin (2010)
GPU-based real-time soft tissue deformation with cutting and haptic feedback.
Progress in biophysics and molecular biology, 103 2-3
G. Joldes, A. Wittek, K. Miller (2007)
An efficient hourglass control implementation for the uniform strain hexahedron using the Total Lagrangian formulation
Communications in Numerical Methods in Engineering, 24
Ramin Mafi, S. Sirouspour (2014)
GPU‐based acceleration of computations in nonlinear finite element deformation analysis
International Journal for Numerical Methods in Biomedical Engineering, 30
F. Martínez-Martínez, María Rupérez-Moreno, M. Martínez-Sober, J. Llorens, D. Lorente, Antonio Serrano, S. Martínez-Sanchís, Carlos Aranda, J. Martín-Guerrero (2017)
A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time
Computers in biology and medicine, 90
V. Luboz, M. Bailet, Christelle Grivot, M. Rochette, B. Diot, M. Bucki, Y. Payan (2017)
Personalized modeling for real-time pressure ulcer prevention in sitting posture.
Journal of tissue viability, 27 1
J. Brown, S. Sorkin, J. Latombe, K. Montgomery, Michael Stephanides (2002)
Algorithmic tools for real-time microsurgery simulation
Medical image analysis, 6 3
Jing-xiao Wang, Shenghui Liao, Xing-Hao Zhu, Y. Wang, Chong-xiang Ling, Xi Ding, Yi-ming Fang, Xiu-hua Zhang (2011)
Real time 3D simulation for nose surgery and automatic individual prosthesis design
Computer methods and programs in biomedicine, 104 3
Andrew Ho, H. Alsaffar, P. Doyle, H. Ladak, S. Agrawal (2012)
Virtual reality myringotomy simulation with real‐time deformation: Development and validity testing
The Laryngoscope, 122
I. Peterlík, Mert Sedef, C. Basdogan, Ludk Matyska (2010)
Technical Section: Real-time visio-haptic interaction with static soft tissue models having geometric and material nonlinearity
Computers & Graphics, 34
Jinao Zhang, Y. Zhong, Julian Smith, Chengfan Gu (2016)
A new ChainMail approach for real-time soft tissue simulation
Bioengineered, 7
Wei Sun, Pallavi Lal (2002)
Recent development on computer aided tissue engineering - a review
Computer methods and programs in biomedicine, 67 2
K. Miller, G. Joldes, Dane Lance, A. Wittek (2006)
Total Lagrangian explicit dynamics finite element algorithm for computing soft tissue deformation
Communications in Numerical Methods in Engineering, 23
F. Goulette, Zhuo Chen (2015)
Fast computation of soft tissue deformations in real-time simulation with Hyper-Elastic Mass Links
Computer Methods in Applied Mechanics and Engineering, 295
Farshad Roohbakhshan, R. Sauer (2016)
Efficient isogeometric thin shell formulations for soft biological materials
Biomechanics and Modeling in Mechanobiology, 16
J. Allard, S. Cotin, F. Faure, Pierre-Jean Bensoussan, F. Poyer, Christian Duriez, H. Delingette, L. Grisoni (2007)
EP4A: Software and Computer Based Simulator Research: Development and Outlook SOFA—An Open Source Framework for Medical Simulation
Simulation In Healthcare: The Journal of the Society for Simulation in Healthcare, 2
Carlos Aranda, U. Meier, M. Raya, F. Chinesta, M. Juan (2001)
A new approach for the real-time simulation of tissue deformations in surgery simulation
Computer methods and programs in biomedicine, 64 2
Siamak Niroomandi, I. Alfaro, E. Cueto, F. Chinesta (2012)
Accounting for large deformations in real-time simulations of soft tissues based on reduced-order models
Computer methods and programs in biomedicine, 105 1
T. Dao, M. Tho (2018)
A Systematic Review of Continuum Modeling of Skeletal Muscles: Current Trends, Limitations, and Recommendations
Applied Bionics and Biomechanics, 2018
R. Lapeer, P. Gasson, V. Karri (2010)
Simulating plastic surgery: from human skin tensile tests, through hyperelastic finite element models to real-time haptics.
Progress in biophysics and molecular biology, 103 2-3
Marcos García, O. Robles, L. Pastor, Angel Rodríguez (2008)
MSRS: A fast linear solver for the real-time simulation of deformable objects
Comput. Graph., 32
D. Kamensky, Fei Xu, Chung-Hao Lee, Jinhui Yan, Y. Bazilevs, M. Hsu (2018)
A contact formulation based on a volumetric potential: Application to isogeometric simulations of atrioventricular valves.
Computer methods in applied mechanics and engineering, 330
Y. Lim, S. De (2007)
Real time simulation of nonlinear tissue response in virtual surgery using the point collocation-based method of finite spheres
Computer Methods in Applied Mechanics and Engineering, 196
S. Cotin, H. Delingette, N. Ayache (1999)
A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation
The Visual Computer, 16
Michele Tonutti, Gauthier Gras, Guang-Zhong Yang (2017)
A machine learning approach for real-time modelling of tissue deformation in image-guided neurosurgery
Artificial intelligence in medicine, 80
A. Woodward, Y. Chan, Rui Gong, M. Nguyen, Trevor Gee, P. Delmas, G. Gimel'farb, Jorge Flores (2017)
A low cost framework for real-time marker based 3-D human expression modeling
Journal of Applied Research and Technology, 15
M. López-Cano, J. Rodríguez-Navarro, A. Rodríguez-Baeza, M. Armengol-Carrasco, A. Susín (2007)
A real-time dynamic 3D model of the human inguinal region for surgical education
Computers in biology and medicine, 37 9
Xianmei Wan, S. Liu, Jim Chen, Xiaogang Jin (2012)
Geodesic Distance-Based Realistic Facial Animation Using RBF Interpolation
Computing in Science & Engineering, 14
M. Audette, V. Hayward, Oliver Astley, M. Doyon, G. McCallister, K. Chinzei (2004)
A PC-based system architecture for real-time finite element-based tool-specific surgical simulation
Yiyi Zhang, Yuxuan Liu, Yahu She, Ye Liang, Fei Xu, C. Fang (2019)
The Effect of Endodontic Access Cavities on Fracture Resistance of First Maxillary Molar Using the Extended Finite Element Method
Journal of Endodontics, 45
G. Turkiyyah, Wajih Karam, Zeina Ajami, A. Nasri (2009)
Mesh cutting during real-time physical simulation
Comput. Aided Des., 43
Boyang Wan, M. Shahmoradi, Zhongpu Zhang, Y. Shibata, Babak Sarrafpour, M. Swain, Qing Li (2019)
Modelling of stress distribution and fracture in dental occlusal fissures
Scientific Reports, 9
M. Doblaré, E. Cueto, B. Calvo, M. Martínez, Jorge Garcia, J. Cegoñino (2005)
On the employ of meshless methods in biomechanics
Computer Methods in Applied Mechanics and Engineering, 194
S. Marchesseau, T. Heimann, S. Chatelin, R. Willinger, H. Delingette (2010)
Fast porous visco-hyperelastic soft tissue model for surgery simulation: application to liver surgery.
Progress in biophysics and molecular biology, 103 2-3
A. Murai, Kosuke Kurosaki, K. Yamane, Yoshihiko Nakamura (2010)
Musculoskeletal-see-through mirror: computational modeling and algorithm for whole-body muscle activity visualization in real time.
Progress in biophysics and molecular biology, 103 2-3
A. Karami, M. Eghtesad, S. Haghpanah (2017)
Prediction of muscle activation for an eye movement with finite element modeling
Computers in biology and medicine, 89
E. Basafa, F. Farahmand (2011)
Real-time simulation of the nonlinear visco-elastic deformations of soft tissues
International Journal of Computer Assisted Radiology and Surgery, 6
O. Sorkine-Hornung, D. Cohen-Or, Y. Lipman, M. Alexa, Christian Rössl, H. Seidel (2004)
Laplacian surface editing
W. B. Karam G. M. Turkiyyah (2011)
Mesh cutting during real-time physical simulation,
Computer-Aided Design, 43
H. Courtecuisse, J. Allard, P. Kerfriden, S. Bordas, S. Cotin, Christian Duriez (2014)
Real-time simulation of contact and cutting of heterogeneous soft-tissues
Medical image analysis, 18 2
Siamak Niroomandi, I. Alfaro, D. González, Elías Cueto, Francisco Chinesta (2012)
Real‐time simulation of surgery by reduced‐order modeling and X‐FEM techniques
International Journal for Numerical Methods in Biomedical Engineering, 28
J. Berkley, G. Turkiyyah, D. Berg, M. Ganter, S. Weghorst (2004)
Real-time finite element modeling for surgery simulation: an application to virtual suturing
IEEE Transactions on Visualization and Computer Graphics, 10
Z. Taylor, Mario Cheng, S. Ourselin (2008)
High-Speed Nonlinear Finite Element Analysis for Surgical Simulation Using Graphics Processing Units
IEEE Transactions on Medical Imaging, 27
Jianyong Zhou, Zu Luo, Chunquan Li, M. Deng (2018)
Real-time deformation of human soft tissues: A radial basis meshless 3D model based on Marquardt's algorithm
Computer methods and programs in biomedicine, 153
S. Cotin J. Allard (2007)
SOFA - an open source framework for medical simulation,
Stud Health Technol Inform, 125
L. Nedel, D. Thalmann (1998)
Real time muscle deformations using mass-spring systems
Proceedings. Computer Graphics International (Cat. No.98EX149)
Andrew Nealen, Matthias Müller, Richard Keiser, Eddy Boxerman, Mark Carlson (2006)
Physically Based Deformable Models in Computer Graphics
Computer Graphics Forum, 25
T. Sørensen, Jesper Mosegaard (2006)
An Introduction to GPU Accelerated Surgical Simulation
G. Sela, Jacob Subag, Alex Lindblad, Dan Albocher, Sagi Schein, G. Elber (2007)
Real-time haptic incision simulation using FEM-based discontinuous free-form deformation
Comput. Aided Des., 39

Publisher: Hindawi Publishing Corporation
Copyright: Copyright © 2020 Tan-Nhu Nguyen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
ISSN: 1176-2322
eISSN: 1754-2103
DOI: 10.1155/2020/5039329
Publisher site: See Article on Publisher Site

Abstract

Hindawi Applied Bionics and Biomechanics Volume 2020, Article ID 5039329, 30 pages https://doi.org/10.1155/2020/5039329 Review Article A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations Tan-Nhu Nguyen , Marie-Christine Ho Ba Tho , and Tien-Tuan Dao Sorbonne University, Université de Technologie de Compiègne, CNRS, UMR 7338 Biomechanics and Bioengineering, Centre de Recherche Royallieu, CS 60 319 Compiègne, France Correspondence should be addressed to Tien-Tuan Dao; tien-tuan.dao@utc.fr Received 12 September 2019; Revised 22 January 2020; Accepted 5 February 2020; Published 20 February 2020 Academic Editor: Jose Merodio Copyright © 2020 Tan-Nhu Nguyen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Simulating deformations of soft tissues is a complex engineering task, and it is even more diﬃcult when facing the constraint between computation speed and system accuracy. However, literature lacks of a holistic review of all necessary aspects (computational approaches, interaction devices, system architectures, and clinical validations) for developing an eﬀective system of soft-tissue simulations. This paper summarizes and analyses recent achievements of resolving these issues to estimate general trends and weakness for future developments. A systematic review process was conducted using the PRISMA protocol with three reliable scientiﬁc search engines (ScienceDirect, PubMed, and IEEE). Fifty-ﬁve relevant papers were ﬁnally selected and included into the review process, and a quality assessment procedure was also performed on them. The computational approaches were categorized into mesh, meshfree, and hybrid approaches. The interaction devices concerned about combination between virtual surgical instruments and force-feedback devices, 3D scanners, biomechanical sensors, human interface devices, 3D viewers, and 2D/3D optical cameras. System architectures were analysed based on the concepts of system execution schemes and system frameworks. In particular, system execution schemes included distribution-based, multithread-based, and multimodel-based executions. System frameworks are grouped into the input and output interaction frameworks, the graphic interaction frameworks, the modelling frameworks, and the hybrid frameworks. Clinical validation procedures are ordered as three levels: geometrical validation, model behavior validation, and user acceptability/safety validation. The present review paper provides useful information to characterize how real-time medical simulation systems with soft-tissue deformations have been developed. By clearly analysing advantages and drawbacks in each system development aspect, this review can be used as a reference guideline for developing systems of soft-tissue simulations. 1. Introduction ing to computation speed (or computation time) and system accuracy. Computation speed is the number of computing iterations that a soft-tissue simulation system can be exe- In a human body, tissues are commonly classiﬁed into hard and soft tissues. While hard tissues do not deform during cuted in one second on a speciﬁc hardware conﬁguration. It the motions of human bodies, soft tissues always deform is usually measured in frames per second (FPS) or Hertz when interacting with themselves, other tissues, and surgical (Hz). Computation time is a time duration needed to run tools. Modeling soft-tissue deformations in an entire organ data acquisition, data pre-/postprocessing, physical behavior simulation, and data visualization in a soft-tissue simulation or only in parts of an organ is still one of the most challenging issues in the biomedical engineering ﬁeld. In particular, eﬀec- system. Moreover, two types of accuracies were considered. tive integration of soft-tissue deformation behaviors into The ﬁrst one relates to model accuracy that quantiﬁes the medical simulation systems has faced two constraints relat- closeness of agreement between the simulated and the real 2 Applied Bionics and Biomechanics consumes large computation cost from a system. System behaviors of soft tissues. The second one deals with the sys- tem accuracy that was aﬀected by interaction device accu- architectures must also be developed to compromisingly racy, algorithm accuracy, and model accuracy. Interaction cooperate all system components such as soft-tissue models and input/output interaction devices. On this aspect, system device accuracy is the degree of closeness of the measured values of a physical quantity to its true values. Algorithm execution schemes and system frameworks should be care- accuracy quantiﬁes the correctness of an implemented com- fully selected to optimize system performance. Finally, once putational process in relation to the true process. Note that fully developed, the system must be validated through diﬀer- these accuracies should be within the clinically acceptable ent validation levels so that it can be used in a target clinical application. Those validation levels include geometrical accuracy bounds for each medical application. In fact, to real- istically simulate both geometric deformations and mechan- validation, model validation, system validation, and user ical behaviors of soft tissues within a medical simulation acceptability/safety validation. Generally speaking, to simu- system, computation speed must be in real time [1], and late soft-tissue deformations in real time while keeping an the system accuracy must be within a desired tolerance level acceptable realistic level of soft-tissue behaviors, all of the above aspects must be individually and systematically ana- according to each medical application. Note that real time is commonly deﬁned as a rate compatible with the graphic ani- lyzed and developed. mation rate of 30 frames per second (FPS) [2]. Moreover, real Although the issues of real-time soft tissue simulations time also includes the responding rate of force feedbacks were also reviewed, previous review studies rarely analyzed when soft tissues collide with other objects. This rate must how real-time challenges were solved eﬀectively in a whole system. In particular, all system development aspects should be between 100 Hz and 1000 Hz so that human tactile per- ceptions can feel collisions without interruptions [3]. It is be thoroughly reviewed to describe how both computation important to note that although real time is one of the most speed and system accuracy requirements were achieved. important requirements for clinical applications, most soft- However, the studies just focused on simulating speciﬁc types tissue simulation systems hardly satisﬁed both acceptable of soft tissues in medical applications, and they did not con- cern how eﬀectively the real-time constraint was solved. For model accuracy and real-time computation speed [4]. For instance, Murai et al. stated that the acquisition of internal example, in an interesting review paper proposed by Delin- somatosensory data in real time was crucial because the real gette a full description of realistic soft-tissue modeling in medical simulations was described [9]. However, it just time could be used in online diagnosis and assessment pro- cesses in surgical applications [5]. Ho et al. showed that the showed out three main problems when realistically simulat- ing soft tissue in medical simulation systems, but the visualization and computing of deformations in real time are essential in surgical simulation of soft tissues [6]. In the methods for solving those problems were not been analyzed. ﬁeld of image-guided surgeries, the estimation of soft-tissue Other than that, this review was conducted in the year 1998 when technologies were in an initial development stage, so deformations in real time is also one of the most important challenges [7]. Note that in image-guided surgery systems, numerous studies that eﬀectively solved the soft-tissue defor- mation issues have not been analyzed in this review study. computation time is commonly expensive due to online data acquisition from medical imaging and additional data pro- Sun et al. [1] also examined a relative diversity of tissue sim- cessing. In fact, most simulation systems with soft-tissue ulation procedures with the help of computer technologies. Although this study covered aspects in the tissue simulation deformations hardly satisfy real-time requirements [8], and they cannot both correctly compute soft-tissue deformations procedure (3D reconstructions, tissue classiﬁcations, and clinical applications), it did not focus on soft-tissue modeling and eﬀectively achieve real-time computation speeds [7]. However, despite this hard constraint, numerous strategies and just ﬁnished at describing general ideas of each aspect have been developed for improving both computation speed rather than analyzing advantages and disadvantages of methods/algorithms employed in each aspect. Moreover, and accuracy of soft-tissue simulation systems. Developing soft-tissue simulation systems is a complex this study was not answered how the challenge of achiev- engineering task composing of multiple aspects. Each of ing both real-time computation speeds and acceptable sys- them has its own contribution to the accuracy and speed of tem accuracy was solved. Mainly analyzing advantages and the target system. From system engineering point of view, disadvantages of modeling physical deformations, Nealen et al. [10] presented a full description about mathematical four important aspects of a real-time medical simulation sys- tem include computational approaches, interaction devices, functions, explanations of the physical meaning, and anal- system architectures, and clinical validations. Computational yses of computation results, but they mainly concerned approaches for modeling soft-tissue deformations are ﬁrst accuracies of each modeling method rather than the com- developed according to current requirements about compu- putation speed when employed in a speciﬁc simulation system. Up to now, with the abundant developments of tation speed and system accuracy. It is important to note that the computation speed and system accuracy are mainly software/hardware technologies and soft-tissue modeling aﬀected by the choice of appropriate computational methods, numerous studies have reasonably proposed approaches for estimating deformations of soft tissues inter- eﬀective solutions for both achieving real-time computa- acted with external input factors. Interaction devices are then tion speeds and acceptable system accuracy in simulation systems. However, they have not been summarized in a selected to interface between soft-tissue models and real physical environments. This interface needs to be both in real systematic way and analyzed completely to estimate general time and in an acceptable accuracy. This requirement often trends and weakness for future developments. Consequently, Applied Bionics and Biomechanics 3 this review study. They also participated into the quality to complement those gaps, this review paper is proposed to answer the following questions: assessment. Consensus discussion was done when necessary for solving disagreements. The number of included/excluded (1) How have computational approaches been developed articles is summarized in Table 2. Firstly, the duplicates were for both achieving real-time computation speed and checked with the duplication tool in the Mendeley software. keeping acceptable system accuracy? The number of duplicated papers at this stage was 1,610 for all search terms. Then, the general and speciﬁc eligibility cri- (2) Which interaction devices have been interfaced eﬀec- teria were applied to all unduplicated articles. The title inclu- tively in real-time soft-tissue simulation systems? sion criteria were ﬁrst used for ﬁltering out the irrelevant articles. The included articles at this phase were 973, which (3) How have system architectures been developed for were then enrolled to the abstract ﬁltering criteria for select- cooperating with computational approaches and ing the most pertinent articles. After reading all the abstracts, interaction devices in real time? 92 included articles were then read in full-text to select the (4) How have been the real-time soft-tissue simulation best qualitative and quantitative articles for systematic systems validated in clinical applications? review. Finally, the number of included articles was 55. Spe- ciﬁcally, the ﬂow chart of the selection procedure illustrating Moreover, real-time soft-tissue simulation systems pro- the number of included/excluded articles after each selection posed in literature were analysed sequentially and summarized stage is shown in Figure 1. To answer the identiﬁed research according to four system development aspects: computational questions, the selected 55 papers were categorized into four approaches, interaction devices, system architectures, and classes. The ﬁrst category concerns the computational clinical validations. Moreover, trends and gaps of each devel- approaches for modeling deformations of human soft tissues opment aspect were also presented. Recommendations for in real time. The second category relates to the disadvantages future researches were ﬁnally proposed. and advantages of interaction devices for getting the external data from soft tissues and visualizing the processed data. The 2. Materials and Methods third category deals with the characteristics of medical hard- ware/software systems consisting of graphic user interfaces A systematic review method was conducted using the (GUIs), programming languages, programming frameworks, PRISMA protocol [11] (Figure 1). Three scientiﬁc databases and other techniques for developing soft-tissue simulation were chosen: ScienceDirect, PubMed, and IEEE. In more systems. The ﬁnal category composes of system validations details, a focus on human soft tissues like upper/lower limb in clinical contexts and the analyses of user acceptability muscles, facial muscles, livers, and skins was done. A special and safety requirements of developed systems. Additionally, attention was also given on the contributions related to the each selected paper could also be grouped on multiple cate- improvement of computational methods and/or employing gories if their contents related to more than one category. eﬀective hardware/software system architectures for real- time medical simulation systems. Finally, other articles 2.2. Eligibility Criteria. The inclusion/exclusion criteria were focused on analyzing applications of real-time soft-tissue clearly deﬁned based on the meaning of each search termi- models for system validation, user acceptability and safety nology. The list of inclusion criteria for each search terminol- requirements were included. Note that in this present review, ogy is shown in Table 3. In addition, to keep the literature at a the method refers to the development strategy of mathemat- high academic level, only journal articles were considered for ical constitutive formulations of soft-tissue deformations the present review. Moreover, the articles in conferences with based on a speciﬁc computational approach. Reviewed a couple of pages are initially eliminated. Other kinds of low- studies relate to mesh-based and meshfree methods. An algo- quality written forms such as letters, judgements, and book rithm concerns the procedure to compute soft-tissue defor- chapters were also not selected. Other than that, the articles mations using speciﬁc modeling methods. A model refers that were not written in English were excluded from the to the mathematical representation of soft-tissue deforma- literature review. tions using mesh-based and meshfree-based methods. A set of search terminologies were deﬁned for the literature investi- 2.3. Quality Assessment. The quality assessment procedure gation, and then, each terminology was presented in a search was established to rate the quality of each analyzed paper. term by using AND/OR operators. The used search terminol- Eighteen yes-no assessment items were deﬁned and used. ogies and their appropriate search terms are listed in Table 1. Papers related to computational approaches bias were evalu- For the systematic information retrieval process, journal arti- ated using the following four items: (1) Was the method ade- cles published up to December 2017 were assessed. quately used/developed and described for the involved tissue 2.1. Selection Methodology. Selection was the most signiﬁcant behavior? (2) Was the veriﬁcation well-performed for the used/developed method? (3) Was the validation systemati- procedure for choosing both qualitatively and quantitatively appropriate articles for the systematic review. After identiﬁ- cally performed for the used/developed method? (4) Did cation from the search engines, retrieved articles were auto- the method really satisfy the real-time constraint? Papers matically saved to their suitable folders using the Mendeley related to interaction devices bias were evaluated using the paper management system. Two independent reviewers following four items: (5) Was the devices well selected for the system? (6) Was the device accuracy adequate for the (TNN and TTD) screened and selected relevant papers for 4 Applied Bionics and Biomechanics Number of searched records based on all search terms n = 361,756 ScienceDirect: n = 319,210 PubMed: n = 44,609 IEEE: n = 1,157 Duplicates n = 1,610 Number of records aer removing duplications n = 360,146 Records excluded from title conditions: IC #1.1: n = 5,465 IC #2.1: n = 13,542 IC #3.1: n = 335 IC #4.1: n = 1,543 IC #5.1: n = 42,209 IC #6.1: n = 4,431 IC #7.1: n = 135,265 IC #8.1: n = 147,136 IC #9.1: n = 9,247 Total excluded: n = 359,173 Records included based on title conditions n = 973 Records excluded from abstract conditions: IC #1.2: n = 128 IC #2.2: n = 37 IC #3.2: n = 169 IC #4.2: n = 171 IC #5.2: n = 83 IC #6.2: n = 125 IC #7.2: n = 76 IC #8.2: n = 45 IC #9.2: n = 47 Total excluded: n = 881 Records included based on abstract conditions n = 92 Records excluded from content conditions n = 37 Studies included for the systematic review n = 55 Figure 1: Workﬂow of the selection process using PRISMA protocol for the performed systematic review. real-time constraints? (7) Was the device easy enough to use system scalable? (12) Were the system frameworks ade- for a clinical routine practice? (8) Is the device price suitable quately selected for implementing the system of interest? for a clinical setting? Papers related to system architecture Papers related to clinical validation bias were evaluated using bias were evaluated using the following four items: (9) Was the following six items: (13) Was the study adequately vali- the system adequately described? (10) Was the system devel- dated with in vitro data? (14) Was the study adequately vali- oped with the participation of the end users? (11) Was the dated with in vivo data? (15) Was the study adequately Included Eligibility Screening Identiﬁcation Applied Bionics and Biomechanics 5 Table 1: The search terms used for the systematic review process. # Search terminologies (terms) Search terms (STs) ST #1: Real-time AND computer-aided AND medical AND 1 Term #1: Computer-aided medical simulations/systems (simulations OR systems) 2 Term #2: Real-time biomedical simulations/systems ST #2: Real-time AND biomedical AND (simulations OR systems) 3 Term #3: Real-time facial simulations ST #3: Real-time AND facial AND simulations 4 Term #4: Real-time liver deformation models ST #4: Real-time AND liver AND deformation AND models 5 Term #5: Real-time medical simulations/systems ST #5: Real-time AND medical AND (simulations OR systems) 6 Term #6: Real-time muscle deformation models ST #6: Real-time AND muscle AND deformation AND models 7 Term #7: Real-time surgery ST #7: Real-time AND surgery 8 Term #8: Real-time ﬁnite element methods ST #8: Real-time AND ﬁnite AND element AND methods 9 Term #9: Real-time soft-tissue deformations ST #9: Real-time AND soft AND tissue AND deformations computation frame rates were nearly 30 FPS. In addition, validated with patient data? (16) Was the level of validation suitable for translating the outcomes into clinical routine all interaction devices were all accurate enough for use in practices? (17) Was the user acceptability performed for clinical routines with acceptable prices, and they were also patients? (18) Was the user acceptability performed for clin- well selected for appropriate computational approaches and ical experts? system architectures. Moreover, the data transmission band- Note that the user acceptability validation is commonly widths of these selected devices were relatively much faster conducted after developing a full-simulation system. This than the computational and graphical rendering speeds, so validation targets at validating the acceptability level related they were all suitable for real-time applications. Over 50% to graphic system’s user interfaces, system’s ease-of-use, sys- of articles have implemented their developed computational tem’s functions, system’s robustness, etc., during short-term approaches into a simulation system. They also well and/or long-term evaluation campaigns for clinicians. described the architectures and frameworks of the imple- Regarding the veriﬁcation of the developed method, an error mented systems for future developments. However, these check list related to the input data, algorithm execution, and systems were rarely developed with the participation of end output visualization is deﬁned. The “well-performed” cate- users. They were mainly tested with the developers and did gory is assigned to a paper if all these three elements are not have many feedbacks from users. Most of implemented satisﬁed. simulation systems could not be directly transferred into the clinical routine practices due to lack of validations with in vitro, in vivo, and real patient data. The computed results 3. Results of simulation systems were often validated with in vitro data 3.1. Overall Quality Assessment Analysis. Statistical results of acquired from phantom tissues with physical testing machines. Due to diﬃculties of acquiring data from living the quality assessment procedure are presented in Table 4. Overall, most selected articles well described, veriﬁed, and organs, only 13% of studies conducted clinical validations validated the computational approaches. Tissue behaviors using in vivo data. Moreover, only external data such as deformations were available. Finally, the user and expert were well described in selected studies. Over 80% of articles modelled the tissue physical characteristics in the methods acceptability aspects were occasionally (i.e., only 4% and while the others just focused on soft-tissue deformations. 7% of studies) investigated. Note that most developed sys- Most authors all well conducted veriﬁcations (76%) and val- tems were initially designed for testing and verifying the idations steps (89%). For examples, in the study of Cotin et al. computational approaches rather than for developing real clinical applications. [12], after the developed methods are clearly described, the authors designed an example system using the method and analyzed the computed results. Their outputs were compared 3.2. Computational Approaches. To achieve real-time com- with other methods and showed a faster computation time putation speed when rendering and computing soft-tissue and higher accuracy level. Visualizations were also clearly deformations, two modeling approaches have been com- monly adopted. The ﬁrst approach that we called model presented to show computed deformations and collisions with a virtual surgical tool. System performance and accuracy development (MD) mainly focuses on geometry discretiza- were also measured and veriﬁed. Thus, the veriﬁcation was tion strategy and mathematical constitutive formulations of well-performed in this study. The veriﬁcation procedure soft-tissue stress-strain relationships. Soft-tissue models was not well-performed in Allard et al. [13] because they developed using this approach are commonly executed with a single-thread platform in a faster and/or more accurate mainly introduced the SOFA framework, and the authors just veriﬁed their results by visual assessments. Although manner. The second approach that we named as constitutive the real-time constraint was strongly required in the study model implementations (MI) relate to the algorithmic imple- objectives, only 65% of the developed computational mentations of the existing constitutive models using devel- approaches really satisﬁed this constraint. The others just oped methods for soft-tissue deformations onto a more powerful hardware conﬁguration such as Graphic Processing nearly reached the real-time conditions. For example, the 6 Applied Bionics and Biomechanics Table 2: The number of included/excluded articles according to the selection procedure. Duplication Title Title Abstract Abstract Content Content Search terms ScienceDirect PubMed IEEE All Duplicates included excluded included excluded included excluded included ST #1 5,537 67 13 5,617 21 5,596 5,465 131 128 3 2 1 ST #2 10,873 2,873 284 14,030 447 13,583 13,542 41 37 4 3 1 ST #3 3,638 87 14 533 19 514 335 179 169 10 0 10 ST #4 1,689 39 10 1,738 19 1,719 1,543 176 171 5 4 1 ST #5 32,857 9,762 290 42,909 605 42,304 42,209 95 83 12 7 5 ST #6 4,560 21 3 4,583 19 4,564 4,431 133 125 8 5 3 ST #7 104,034 31,339 367 135,727 371 135,356 135,265 91 76 15 3 12 ST #8 146,837 261 153 147,251 60 147,191 147,136 55 45 10 7 3 ST #9 9,185 160 23 9,368 49 9,319 9,247 72 47 25 6 19 Total 319,210 44,609 1,157 361,756 1,610 360,146 359,173 973 881 92 37 55 Applied Bionics and Biomechanics 7 Table 3: The inclusion criteria for each search terminology. # Search terms (STs) Inclusion conditions (ICs) IC #1.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “medical”, “simulations”, and “computer-aided” keywords and (2) the title concerns the supports of computers in soft-tissue simulations executing in real time 1ST#1 IC #1.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the support of computer in medical systems, medical simulations, and medical applications so that they can be executed in real time; (2) the abstract describes the medical system architectures and the interactions of computer’s input/output devices in clinical environments; and (3) the system developed in the paper focuses on simulating human soft tissues IC #2.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “biomedical”, and “simulations” keywords and (2) the title concerns the issues of real-time simulation in biomedical applications 2ST#2 IC #2.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the analyses of real time in biomedical applications/systems and (2) the abstract focuses on analysing the computational approaches, the system architectures, or the characteristics of real time in biomedical applications IC #3.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “facial” keywords, and (2) the title concerns the computational approaches to simulate the human faces 3ST#3 IC #3.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the development of computational techniques or system designs for modelling the facial mimics/expressions/muscles and (2) the developed techniques must be able to execute in real time IC #4.1: the title must satisfy all following conditions: (1) the title contains “real-time”, “liver”, and “models” keywords and (2) the title concerns the modelling methods of the human liver in real time 4ST#4 IC #4.2: the abstract must satisfy all following conditions: (1) the abstract concerns the issues of computational approaches for modelling the human liver and (2) the computational approaches must be executed in real time IC #5.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “medical”, and “simulations”/”systems” keywords and (2) the title is aimed at developing the computational methods for modelling the soft tissue in medical environments 5ST#5 IC #5.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns computational approaches or system architectures for modelling soft tissues in medical environments and (2) the system must be run in real time IC #6.1: the title must satisfy all of the following conditions: (1) the title contains “real-time”, “muscle”, and “models” keywords and (2) the title considers the computational methods for modelling the human muscles in real time 6ST#6 IC #6.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the developments of computational techniques for modelling and simulating human muscles so that they can run in real time and (2) the abstract shows the implementations of muscle deformable models in clinical environments IC #7.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “surgery” keywords and (2) the title illustrates the surgical simulations/systems applied in human soft tissues executed in real time 7ST#7 IC #7.2: the abstract must satisfy all of the following conditions: (1) the abstract describes the surgical simulations/systems for human soft tissues and (2) the abstract concerns system architectures of surgical simulations or systems so that they can execute in real time IC#8.1: the title must satisfy all of the following conditions: (1) the title contains “real-time” and “ﬁnite element” keywords and (2) the title concerns the ﬁnite element modelling methods for human soft tissues in real time IC #8.2: the abstract must satisfy all of the following conditions: (1) the abstract concerns the human soft-tissue 8ST#8 modelling method in real time based on the ﬁnite element modelling methods and (2) the abstract is aimed at developing, generating, and analysing the variations of ﬁnite element modelling methods to get the real-time requirements IC #9.1: the title must satisfy all following conditions: (1) the title contains “real-time”, “soft tissue”, and “deformations”/”models” keywords and (2) the title considers the modelling methods of the human soft-tissue deformations executing in real time 9ST#9 IC #9.2: the abstract must satisfy all of the following conditions: (1) the abstract illustrates the computational approaches for development the models of human soft-tissue deformations and (2) the abstract is aimed at developing, analysing, and generating the modelling methods Unit (GPU) system. Thus, systems can compute soft-tissue programming algorithms to parallelize the execution tasks models faster and more robustly than the traditional ones. of a developed modeling method, which was traditionally Particularly, this concept refers to a family of more suitable running on a single-thread platform. For example, Berkley 8 Applied Bionics and Biomechanics Table 4: Summary of the statistical results of the quality assessment procedure. Quality assessment criteria %of “yes” scores (%) Computational approaches’ bias 1. Was the method adequately used/developed and described for the involved tissue behavior? 82 2. Was the veriﬁcation well performed for the used/developed method? 76 3. Was the validation systematically performed for the used/developed method? 89 4. Did the method really satisfy the real-time constraints? 65 Interaction devices’ bias 5. Was the devices well selected for the system? 49 6. Was the device accuracy adequate for the real-time constraint? 47 7. Was the device easy enough to use for a clinical routine practice? 47 8. Is the device price suitable for a clinical setting? 47 System architectures’ bias 9. Was the system adequately described? 65 10. Was the system developed with the participation of the end users? 15 11. Was the system scalable? 53 12. Were the system frameworks adequately selected for implementing the system of interest? 45 Clinical applications bias 13. Was the study adequately validated with in vitro data? 33 14. Was the study adequately validated with in vivo data? 13 15. Was the study adequately validated with patient data? 18 16. Was the level of validation suitable for translating the outcomes into clinical routine practices? 29 17. Was the user acceptability performed for patients? 4 18. Was the user acceptability performed for clinical experts? 7 et al. addressed a MD study related to the development of a methods refer to the development of the ﬁnite element Linearized FEM (L-FEM) method built from the reduced method (FEM) and its variations to simulate the soft-tissue object kinematics [14]. The L-FEM method is suitable for deformations in real time (Figure 3). The meshfree-based modeling linear elasticity of soft tissues. This method is faster modeling techniques refer to the decomposition of soft- than the FEM. Moreover, in the study of Joldes et al., the total tissue model into simpler physical submodels or representa- tions without meshing the domains of interest (Figure 4). Lagrangian (TL) formulation was applied to improve the computation speed of the traditional FEM [15]. Additionally, The hybrid modeling methods take advantage of cooperating the total Lagrangian explicit dynamic FEM (TLED-FEM) multiple modeling methods to increase both computation formulation was developed by Miller et al. and it could run speeds and model accuracy. The distribution of selected stud- faster than the FEM when executing on the same CPU- ies according to each modeling method is shown in Figure 2. based platform [16]. Regarding the model implementation The result shows that up to 51% of the studies related to the (MI) approach, only the implicit time integration of FEM mesh-based methods. The use of the meshfree-based method has been proven to be the most suitable for parallel methods reaches over 42%. Finally, the percentage of hybrid methods is around 7%. implementation. This method was implemented in a GPU platform by Taylor et al. [17]. It is interesting to note that most studies focused at devel- 3.3. Model Development Approaches oping new mathematical methods for modeling the soft- tissue deformations rather than implementing the developed 3.3.1. Mesh-Based Modelling Methods. Mesh-based modeling modeling methods into a speciﬁc hardware conﬁguration to methods are grouped into four common computation strate- accelerate the computation speed. The distribution of the gies: the ﬁnite element modeling method (FEM), the two approaches throughout the selected literature is illus- precomputation-based FEM, the formulation-adapted FEM, trated in Figure 2. Obviously, among the total of 55 and the boundary element methods (Figure 5). Note that in studies, over 80% of the studies proposed the model devel- this present review, the term “deformation models” relates to soft-tissue models developed using a speciﬁc modeling opment of soft-tissue deformations while only 18% of studies took advantages of speciﬁc hardware to accelerate method while the term “simulation models” refers to numer- available modeling methods. Regarding the MD approach, ical models in general meaning. we grouped all developed computational methods into three The ﬁnite element method (FEM) has been popularly categories: mesh, meshfree, and hybrid modeling methods employed in the literature despite of its very high computa- tional cost. Deformable objects are geometrically meshed by (Tables 5–7). In more details, the mesh-based modeling Applied Bionics and Biomechanics 9 Model implementation (MI) 18% Computational approaches Model development (MD) 82% Mesh-based 51% Meshfree-based 42% Combination-based 7% Figure 2: Distribution of computational approaches (MD and MI) and associated techniques for MD approach in the literature. a set of elementary components called ﬁnite elements. These solution for dealing with the topological changes in cutting elements are connected by nodes whose quantity deﬁnes the simulations [20]. Peterlik et al. simulated the human liver with realistic haptic feedback and deformations embedded size of the FE model. Material properties are commonly assigned into each ﬁnite element. Then, the physical behavior with both nonlinear geometric and material parameters [3]. of solid object deformations is described by a set of constitu- Morooka et al. designed a navigation system for the mini- tive equations. Finally, the resolution of these equations on mally invasive surgeries using a neural network model [31]. the nodes with prescribed boundary and loading conditions Martínez-Martínez et al. used the decision tree and two leads to the stress-strain relationships of the deformable tree-based ensemble methods for simulating the breast com- objects. FEM provides a very high level of accuracy and real- pression [36]. Lorente et al. applied decision trees, random istic deformations in both linear and nonlinear cases. For forests, and extremely randomized trees models to simulate example, Wu et al. [30] modeled the facial muscles by FEM biomechanical behaviors of a human liver during the breath- to animate the facial expressions. Each single muscle was ing action [8]. Tonutti et al. also applied artiﬁcial neural net- considered an incompressible and hyperelastic material. works (ANNs) and support vector regression (SVR) Each muscle model includes 1,180 nodes and 28,320 DOF. algorithms for learning the precomputed data from the Note that the computing time could not be achieved in real FEM model of a human tumor [7]. Luboz et al. used a set of pressure frames compressed into a small number of modes time. Karami et al. employed also the FEM for modeling the extraocular muscles (EOMs) in an eye to estimate the by proper orthogonal decomposition [37]. This method muscular activations and directions [35]. The eyeball model allows the summarized modes to be described by a linear includes 1,970 nodes and 8,638 elements. Each muscle model set of scalar coeﬃcients, and this reduced set of pressure includes 1,100 nodes and 2,673 elements. The computation map modes was then inputted to the FE to compute the strain ﬁeld modes. time needed to solve the model was 20 ms. The precomputation-based FEM is the most popular var- The formulation-adapted FEM has been developed by iation of FEM. This method uses the relationship between the mathematically alternating the FEM formulations with the mechanical forces and the deformations precomputed from other modeling methods. One of them is called linearized the accurate FEM with full physical and biomechanical char- FEM (L-FEM) in which the kinematic behavior of the simu- lated object is linearized to the ﬁrst order of approximations acteristics to train an approximate model. To achieve this goal, a database of the accurate FE simulation outcomes during a speciﬁc timing period. Thus, the FEM model built needs to be constructed a priori. The computational accuracy from the reduced object kinematic is also simpliﬁed and exe- and speed of the simulated model depend on the types of cuted much faster than the original one. Due to the simpliﬁ- employed approximate techniques such as linear/nonlinear cation, the L-FEM is only suitable for modeling the soft tissues with linear elastic materials. For instance, Berkley regression functions and machine learning (ML). By using this strategy, Cotin et al. developed a liver surgical simulation et al. applied the L-FEM to the virtual suturing application system [12]. Sedef et al. provided a solution for real time and [14]. Moreover, Audette et al. divided a FEM model into realistic FEM for simulating viscoelastic tissue behavior in multiple submeshes [18]. All submeshes were computed medical training based on the experimental data collected independently in parallel threads of a real-time operating system to output the local deformations. Garcia et al. from a robotic tester [19]. Sela et al. proposed an eﬀective 10 Applied Bionics and Biomechanics Table 5: Classiﬁcation of developed modelling methods for soft-tissue deformations in real time: mesh-based techniques. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations Precomputation-based 1400 N Linear elasticity 7 ms (force feedback) FEM (pre-comp FEM) Dec AlphaStation Cotin et al. [12] MD The human liver 6,500 tetrahedral approximated by linear Nonlinear elasticity 8 ms (force feedback) 400 MHz elements functions 863 N 1 kHz (force feedback) Berkley et al. [14] MD Linearized FEM (L-FEM) The human skin Linear elasticity Surface triangle 1 GHz Athelon CPU 30 Hz (model rendering) elements ∗∗ Audette et al. [18] MI Multirate FEM (MR-FEM) The human brain Linear elasticity 10 kHz (force feedback) NI Dual Pentium PC Precomputation-based 51 N ∗∗∗ FEM (pre-comp FEM) Linear 1 kHz (force feedback) 153 DOF Pentium IV 2.4 GHz Sedef et al. [19] MD The soft-tissue cube using linear viscoelastic viscoelasticity 100 Hz (model rendering) 136 tetrahedral dual CPU formulations elements Precomputation-based FEM (pre-comp FEM) 1 kHz (force feedback and P4-2.8 GHz CPU, Sela et al. [20] MD The human skin Linear elasticity 12,108 polygons using discontinuous free model cutting) 1 GB RAM form deformations ∗∗∗∗ Total Lagrangian explicit 6000 E , 6741 N Karol Miller et al. [16] MD NI Nonlinear elasticity 16 ms (model deformation) 3.2 GHz Pentium IV dynamic (TLED) FEM Hexahedral elements Matrix system reduction 3.8 ms–35.7 ms From 266 N–1,579 E 2.4 GHz Pentium IV García et al. [21] MD NI Linear elasticity FEM (MSR-FEM) (solving the system) to 110 N–587 E CPU, 1 GB 2.1 ms (one system time 2,200 E-2535 N Joldes et al. [22] MD Total Lagrangian (TL) FEM NI Nonlinear elasticity CPU step) Hexahedral elements 3.2 GHz P4 CPU, From 11,168 E to Total Lagrangian explicit From 14.0 to 10.7 times 2 GB RAM Taylor et al. [17] MI The human brain Nonlinear elasticity 46,655 E NVIDIA GeForce dynamic (TLED) FEM faster than CPU Tetrahedral elements 7900 GT GPU Total Lagrangian explicit Hyperelasticity 12 ms (model deformation) 15,050 E, 16,710 N 3 GHz Intel Core Duo Joldes et al. [15] MD dynamic FEM The human brain (neo-Hookean) 1 kHz (haptic feedback) 7,000 DOF CPU (TLED-FEM) 3.54 s (3000 system time GPU NVIDIA CUDA step running) 16,825 E-12,693 N Tesla C1060 (240 FEM (NL-FEM) Joldes et al. [23] MI The human brain Nonlinear elasticity implemented on GPU 19.95 s (3000 system 125,292 E-95,669 N 1.296 GHz cores, 4 GB time-step running) high-speed memory) Total Lagrangian explicit GPU NVIDIA CUDA dynamic FEM <4 s (deformation 18,000 N–30,000 E tesla C870 (128 Wittek et al. [24] MI The human brain Nonlinear elasticity (TLED-FEM) implemented prediction) ~50,000 DOF 600 MHz cores, on GPU 1.5 GB memory) Applied Bionics and Biomechanics 11 Table 5: Continued. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations 0.54 s Precomputation-based 1,777 E–501 N 9.89 s (stiﬀness and tangent FEM (pre-comp FEM) 10,270 E–2,011 N AMD Opteron 2 GHz Peterlík et al. [3] MD The human liver Nonlinear elasticity stiﬀness matrix computing) using radial basic functions Surface triangle CPU, 8 GB RAM 1 kHz (haptic feedback) (RBF) elements 30 Hz (model rendering) Hyperelasticity (general polynomial, Total Lagrangian FEM Lapeer et al. [25] MI The human skin reduced polynomial, >1 kHz (haptic feedback) 100 E–50,000 E GPU (TL-FEM) and ogden formulation) Multiplicative Jacobian Porohyperelasticity, 13 FPS (model 20,700 E–4,300 N Marchesseau et al. [26] MD energy decomposition FEM The human liver CPU Viscohyperelasticity deformation) Tetrahedral elements (MJED-FEM) 1.4 FPS (model computing model on CPU) 41,000 N The human cataract Linear elasticity 46.15 FPS (model Tetrahedral elements Courtecuisse et al. [27] MI Linearized FEM (L-FEM) The human liver combined with a GPU computing on GPU) 3,874 N The brain tumor corotational method 64 ms (model computing on Tetrahedral elements GPU) 31,008 N Discontinuous basic 13.9 ms (model computing Turkiyyah et al. [28] MD The human skin Linear elasticity Surface triangle CPU function FEM (DBF-FEM) and mesh updating) elements 7,182 E–8,514 N Order reduction method The human cornea 20 Hz (model and graphic Hexahedral elements 2 GHz CPU, 2 GB Niroomandi et al. [29] MD Nonlinear elasticity (ORM) FEM The human liver updating) 10,519 E-2853 N RAM Tetrahedral elements Finite element method The superﬁcial 560 E–1180 N Wu et al. [30] MD Nonlinear elasticity NI CPU (FEM) fascia in a face 28,320 DOF Precomputation-based Morooka et al. [31] MD FEM (pre-comp FEM) The phantom liver NI NI 15,616 E-4,804 N CPU using neuro networks Element-by-element Maﬁ and Sirouspour precondition conjugate The human 10 times faster than CPU 6361 E–13,3784 E MI Linear elasticity NDIVIDA GTX 470 [32] gradient FEM (EbE stomach for model computing 1295 N–25462 E PCG-FEM) 12 Applied Bionics and Biomechanics Table 5: Continued. Geometry Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed discretization conﬁgurations 70 FPS (system iteration) 1,300 tetrahedral Linear elasticity Precondition FEM The heterogeneous 1 kHz (haptic feedback) elements Courtecuisse et al. [33] MI combined with a 256 core GPU (pre-cond FEM) soft tissues 22 ms (node adding or 150 contact points corotational method removing) 3,874 N Total Lagrangian explicit A general cube Hyperelasticity 0.309 s–163.402 s (one NVIDIA GTX460 Strbac et al. [34] MI 125 E–91,125 E dynamic (TLED) FEM mesh (neo-Hookean) solution time step) GPU Eyeball: The extraocular 8638 E–1970 N Finite element modelling Karami et al. [35] MD muscles (EOMs) in Linear elasticity 20 ms (model deformation) Muscle: CPU method (FEM) an eye 2673 E-864 N Tetrahedral elements Precomputation-based FEM (pre-comp FEM) Hyperelastic 313,000 E-62,000 N 2.6 GHz Intel (R) Martínez et al. [36] MD The human breast <0.2 s (model compression) using artiﬁcial neuro (Mooney-Rivlin) Tetrahedral elements Xeon (R) CPU networks Precomputation-based 2.89 s (model computing 3.4 GHz Intel Core i7, FEM (pre-comp FEM) using machine learning) From 379,800 N to Lorente et al. [8] MD The human liver Nonlinear elasticity 8 GB RAM, OS X El using artiﬁcial neuro 51.63 s (model computing 420,690 N Capitan networks using FEM) Precomputation-based FEM (pre-comp FEM) <10 ms (model prediction 6,442 N-1,087 E Tonutti et al. [7] MD using artiﬁcial neuro The brain tumor Nonlinear elasticity Core i7 2.9 GHz CPU using neural network) Tetrahedral elements networks and support vector regression Precomputation-based 27,649 E FEM (pre-comp FEM) <1 s (strain ﬁeld Luboz et al. [37] MD The butt area Nonlinear elasticity Hex-dominant CPU using the reduced order computing) elements modelling method ∗ ∗∗ ∗∗∗ ∗∗∗∗ N: nodes; NI: no information; DOF: degree-of-freedom; E: elements. Applied Bionics and Biomechanics 13 Table 6: Classiﬁcation of developed modelling methods for soft tissue deformations in real time: meshfree-based techniques. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 16 FPS (model 82 mass points SGI Impact workstation, Mass-spring system deformation) Nedel and Thalmann [38] MD The muscle Linear elasticity method (MSM) 84 FPS (model 17 mass points MIPS R10000 CPU deformation) <150 N Boundary element The general cube 15 Hz (model R-4400 CPU, 64 MB Monserrat et al. [4] MD Linear elasticity Surface triangle method (BEM) mesh deformation) RAM elements Statistical analysis 1 minute (facial Pentium II, 333 MHz Goto and Lee [39] MD method (SAM)- The human face NI NI feature detection) CPU muscle Mesh geometry 475 ms (facial VRML-like representation deformation) 1,253 V-2,444 F Pentium II 450 MHz Bonamico et al. [40] MD The human face Linear elasticity (VRML) & radial basis 1,430 ms (facial 4,152 V-8,126 F CPU, 128 MB RAM function (RBF)- deformation) muscle 216 N-1,440 E 24 FPS (system Surface triangle Sun Ultra 60 Workstation Mass-spring system Nonlinear iteration) elements Brown et al. [2] MD The blood vessel 450 MHz CPU, 1 GB method (MSM) viscoelasticity 6 FPS (system 8,000 N-66,120 E RAM iteration) Surface triangle elements ~10,000 V Laplacian surface Sorkine et al. [41] MD The face model Linear elasticity 0.07 s (model solving) Surface triangle 2.0 GHz Pentium IV CPU deformation (LSD) elements Personal facial expression space 12 FPS (facial Chandrasiri et al. [42] MD The human face Linear elasticity NI 1 GHz Athlon CPU method (PEES)- animation) muscle From 53,3380 N to Mass tensor method The cube Mollemans et al. [43] MD Linear elasticity From 24.57 s to 2.3 s 10,368 N CPU (MTM) The human face Tetrahedral mesh Mass-spring system 8,000 N method (MSM) 48 Hz–3,000 Hz SGI Prism Server 4 GPU, Chen et al. [44] MD The human brain Linear elasticity Surface triangle combined with (haptic feedback) 8 CPU, 32 GB RAM elements quasistatic algorithm 14 Applied Bionics and Biomechanics Table 6: Continued. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 4,891 V GPU NVIDIA 6,800, Mass-spring system The human inguinal 73 FPS (system Pentium IV 3.0 GHz López-Cano et al. [45] MI Linear elasticity Surface triangle method (MSM) region iteration) elements CPU, 1 GB RAM Point collocation- 1 ms (model 1,186 polygons Pentium IV 2 GHz CPU, Lim and De [46] MD based ﬁnite spheres The human liver Nonlinear elasticity deformation) Polygon elements NVIDIA Quadro4 XGL (PCMFS) 16 ms (muscle tension Intel Xeon 3.33 GHz Inverse dynamic estimation) CPU, 3.25 GB RAM, Murai et al. [5] MD The human muscles Linear elasticity 274 muscles computation (IDC) 15 FPS (model NVIDIA Quadro FX3700 rendering) GPU 5 ms (model deformation) 96 N-270 E Basafa and Farahmand Mass-spring-damper Nonlinear 150 Hz (haptic Tetrahedral mesh 3.2 GHz Core Duo CPU, MD The cube model [47] method (MSD) viscoelasticity feedback) 500 N 1 GB RAM 30 Hz (model Tetrahedral mesh rendering) Windows 2000 or Laplacian surface Wang et al. [48] MD The human nose Linear elasticity NI NI Windows XP, 512 MB deformation (LSD) RAM or 250 MB 1 kHz (haptic 917 E Mass-spring system feedback) Intel Core2 Q6600 CPU, Ho et al. [6] MD The human eardrum Linear elasticity Surface triangle method (MSM) 30 Hz (model NVIDIA GeForce 9,600 elements rendering) Radial basic function 5,272 V-10,330 F Intel Core 2 Duo E7200 0.0316 s (one system Wan et al. [49] MD (RBF) & geodesic The human face Linear elasticity Surface triangle 2.53 GHz CPU, 2 GB frame computing) distance-muscle elements RAM 73.8 FPM (facial Intel Xeon 2.4 GHz Thin-shell animation) 40 markers 16-Core CPU, NVIDIA Le et al. [50] MD deformation method The human face Linear elasticity 164.3 FPM (facial 100 markers Tesla C1060 240-Core (TSD)-muscle animation) GPU Elastic-plus-muscle- Zhang et al. [51] MD distribution-based The facial muscles Linear elasticity NI NI NI (E+MD) >200 FPS (graphic Facial motion rendering on PC) Core i7 3.5 CPU Weng et al. [52] MD regression algorithm The human face NI 30 FPS (graphic 75 facial markers Intel Atom 2.0 GHz CPU (FMR)-muscle rendering on mobile devices) Applied Bionics and Biomechanics 15 Table 6: Continued. Computation Geometry Reference Approach Modelling methods Soft-tissue types Tissue Behaviors Hardware conﬁgurations time/speed discretization 4.02 ms (one model computation 4,430 E–1,128 N Hyperelastic mass link iteration) Tetrahedral mesh Core 2 Duo 2.40 GHz Goulette and Chen [53] MD method for FEM The cube model Viscohyperelasticity 21.24 ms (one model 21,436 E–5,591 N CPU, 3.45 GB RAM (HEML-FEM) computation Tetrahedral mesh iteration) The time-saving volume-energy 30 Hz (model Core i7-4700 3.4 GHz Zhang et al. [54] MD conserved ChainMail The cube model Nonlinear elasticity NI rendering) CPU method (TSVE- Chainmail) 2 minutes (system Radial basis function initializing) Woodward et al. [55] MD mapping approach The human face Linear elasticity NI NI Up to 30 Hz (facial (RBF)-muscle feature detection) Core i7-4790 3.60 GHz Marquardt radial The general cube 0.1509 s (model 121 nodes CPU, 8 GB RAM, Intel Zhou et al. [56] MD basis meshless Nonlinear elasticity model deformation) Tetrahedral mesh HD Graphics 4600 method (MRM) (64 MB) 16 Applied Bionics and Biomechanics Table 7: Classiﬁcation of developed modelling methods for soft-tissue deformations in real time: combination-based techniques. Hardware Reference Approach Modelling methods Soft-tissue types Tissue behaviors Computation time/speed Geometry discretization conﬁgurations Precomputation-based FEM (pre-comp FEM) & 40 Hz (model deformation) 760 vertices–4,000 edges 233 MHz Dec Alpha Cotin et al. [57] MD mass tensor method The blood vessel Linear elasticity 500 Hz (haptic feedback) 8,000 tetrahedral elements Workstation (MTM) & hybrid modelling method (HMM) 1.6 GHz Dothan <25 ms (model Yarnitzky et al. [58] MD Dynamics-based & FEM The foot soft-tissue Linear elasticity 100 nodes Pentium IV CPU, deformation) 1 GB RAM Multi-cooperative methods Allard et al. [13] MD NI NI NI NI NI (multi-Corp) Boundary element method Linear elasticity 2.26 GHz Pentium (BEM) & mass-spring with an extra From 0.99 ms to 4.17 ms M CPU, GeForce Zhu and Gu [59] MD The human liver From 200 to 1,200 nodes system (MSM) & particle mass-spring (model deformation) 9650 M GPU, 2 GB surface interpolation (PSI) model RAM Applied Bionics and Biomechanics 17 Modeled so tissues Described behaviors Generic so tissue (i) Linear and nonlinear elasticity Liver and liver Model development (MD) (ii) Linear viscoelasticity phantom Cornea (ii) Hyperelasticity (Neo-Hookeanand Mooney-Rivlin) Skin Face (iv) Poro-hyperelasticity Brain Extraocular muscles Breast Mesh-based techniques Described behaviors Modeled so tissues Generic so tissue (i) Linear and nonlinear elasticity Liver Linear elasticity combined with a co-rotational method Model implementation (MI) (ii) Skin (ii) Hyperelasticity (general polynomial, reduced Cataract Brain polynomial and Ogden formulation) Tumour (iv) Hyperelasticity (Neo-Hookean) Stomach Figure 3: Overview of all modeled soft tissues and diﬀerent described behaviors for mesh-based studies. Modeled so tissues Described behaviors Generic so tissue Skeletal muscles (i) Linear and nonlinear elasticity Model development (MD) Face Facial muscles (ii) Nonlinear viscoelasticity Nose Liver (iii) Visco-hyperelasticity Eardrum Brain Blood vessel Meshfree-based techniques Described behaviors Modeled so tissues (i) Linear elasticity Inguinal region Model implementation (MI) Figure 4: Overview of all modeled soft tissues and diﬀerent described behaviors for meshfree-based studies. presented another reduction method called matrix system each iteration. Turkiyyah et al. aimed at physically simulating reduction FEM (MSR-FEM) [21]. The method focused the mesh cutting in real time thanks to the controlled discon- rather on computing the regions of interest than the whole tinuities in the basic functions and the fast incremental model. The order reduction method (ORM) was developed methods for updating the global deformations [28]. Finally, by Niroomandi et al. to reduce the complex computation of element-by-element precondition conjugate gradient FEM nonlinear FEM for real-time simulations [29]. The total (EbE PCG-FEM) was developed by Maﬁ and Sirouspour [32]. This method combined the FEM with a conjugate Lagrangian (TL) formulation was also applied in a FE model to improve the computation speed. Joldes et al. used this gradient method by alternating the mesh topological com- approach to develop a FE model for an eﬃcient hourglass putation at run time by iterations. Thus, the developed control application [15]. A variation of this method, called model would be faster than the original one using FEM and total Lagrangian explicit dynamic FEM (TLED-FEM), was required less system memories during execution. A new pre- also developed by Miller et al. for an image-guided surgery conditioning technique (pre-cond FEM) was also proposed applications [16]. This method was also employed by Joldes by Courtecuisse et al. for improving the computational time et al. to simulate the deformations of a human brain [15]. of soft-tissue deformations [33]. This technique could simu- They all successfully improved both the sizes and computa- late topologically changes and haptic feedbacks of homoge- tion speeds of the developed models. Another version of neous and heterogeneous materials in acceptable accuracy. TL-FEM proposed by Marchesseau et al. was called multipli- The boundary element methods are based on surface cative Jacobian energy decomposition (MJED) FEM [26]. deformations to deduce the internal deformations in real This approach optimizes the generation of stiﬀness matrix time. Monserrat et al. developed a surgery simulation system in TL-FEM to solve the linear system of equations during using this method [4]. Compared with the FEM, the BEM 18 Applied Bionics and Biomechanics Mesh-based techniques Finite element Pre-computation- method based FEM (FEM) Boundary Formulation- element adapted FEM methods Figure 5: Overview of common computation strategies for mesh-based studies. only required the discretization of the object’s surface so that form the model’s stiﬀness matrix and to describe biomechan- ical characteristics of the soft-tissue object [56]. In particular, it could provide an optimized, fast, and easy implementation. Another surface-based method for developing the soft-tissue this approach does not need to preprocess all cell elements to models was called Laplacian surface deformation (LSD) was estimate the global deformations like mesh-based modeling ﬁrst proposed in Sorkine et al. [41]. The method represented methods do. Consequently, the meshfree-based modeling the object surface based on the Laplacian of the mesh. Wang techniques are much faster than the mesh-based modeling et al. also employed the LSD method for nose surgery in a strategies, and they can simulate large deformations in real complete surgical system for automatic individual prosthesis time. Because of these advantages, the meshfree-based design [48]. Goto et al. used the statistical analysis method modeling methods have received much attentions from (SAM) for detecting features on the facial surface through research community in the recent years. One of the most 2D images, and then, the detected features were mapped to popular methods using the meshfree-based strategy is the a generic 3D facial model for generating the expressions mass-spring system modeling (MSM) method. Nedel et al. using the surface deformation method [39]. Moreover, the applied the MSM method to model the muscle deformations computation speed of the facial expression estimators was in real time [38]. Brown et al. applied the MSM method for a surgical training system [2]. Chen et al. also used the MSM enhanced by using a scaling polygon mesh method based on iterative edge contractions by Bonamico et al. [40]. for developing a deformable model for haptic surgery simula- Chandrasiri et al. proposed a strategy for converting the tion [44]. The MSM was also applied to simulate the 3D acquired facial expressions to the MPEG-4 FAP [60], stream model of the human inguinal region by López-Cano et al. to deform the 3D surface facial models robustly and in real [45]. Ho et al. developed a deformable tympanic membrane using the MSM method for simulating the real-time defor- time [42]. Wan et al. [49] and Woodward et al. [55] used the landmark-based and muscle-based facial expression esti- mation and cutting in a virtual reality myringotomy simula- mation to animate the 3D surface facial model. The used tor [6]. Another well-known method of meshfree-based methods were radial basic function (RBF) and geodesic dis- method is called the mass tensor method (MTM) in which tance. Le et al. took advantages of the thin-shell linear defor- the modeled object is approximated into a tetrahedron mesh. Inside each tetrahedron in the MTM, the displacement vec- mation model to reconstruct the facial pose via the facial marker displacements [50]. tors of four vertices are linearly interpolated into the dis- placement ﬁeld of this tetrahedron [57]. The MTM was 3.3.2. Meshfree-Based Modelling Methods. Compared to the used by Mollemans et al. to simulate the soft-tissue deforma- mesh-based modeling methods, meshfree-based modeling tions after bone displacement [43]. An improvement of MSM called mass-spring-damper (MSD) modeling methods was methods use discrete points for representing continuum, and it takes advantages of interpolation methods to solve proposed by Basafa and Farahmand [47]. The results illus- the partial diﬀerential equations (PDEs) [59]. Thus, a simu- trated that with a simple cube model including 96 nodes lated soft-tissue object is commonly modeled as a distribu- and 270 tetrahedrons, the computation time was just about tion of discrete nodes inside to form a complete volumetric 0.005 s for each step. Another improvement of MSM was developed by Goulette et al. called hyperelastic mass links model. These nodes are embedded with a shape function to Applied Bionics and Biomechanics 19 requirement level of real-time constraints. Zhu and Gu (HEML) in which the forces at a speciﬁc node are considered a sum of force functions from the neighboring nodes con- also applied multiple modeling methods to develop a nected with it [53]. Experiments showed that with the hybrid deformable model for real-time surgical simulation [59]. Diﬀerent cooperative components exist in the system 21,436-tetrahedron HEML model, the computation time was at 21.24 ms corresponding with 47 FPS. A diﬀerent such as boundary the element method (BEM), the mass- aspect for meshfree-based methods is proposed by Lim and spring method (MSM), and a particle surface interpolation De known as the point collocation-based method of ﬁnite algorithm. sphere (PCMFS) [46]. The technique was based on the com- 3.4. Model Implementation Approaches. The model imple- bination between the multiresolution approaches and the fast analysis strategies for nonlinear deformations for the active mentation (MI) approach mainly focusses on algorithmic regions where being contacted by the surgical tool tip. A dis- implementation of soft-tissue models based on developed tinctive modeling method for meshfree-based method was modeling methods onto a more powerful hardware conﬁgu- inverse dynamic computation (IDC) proposed by Murai ration. This approach can improve the computational perfor- mance of the developed soft-tissue deformation models, even et al. for the musculoskeletal system [5]. Zhang et al. devel- oped an elastic-plus-muscle-distribution-based (E+MD) to faster and more robust than the MD approach. In particular, model the facial muscle distribution for generating the facial the MI approach mostly aims at ﬁnding more suitable pro- expressions in real time [51]. Another method called the gramming algorithms to parallelize the execution functions time-saving volume-energy conserved ChainMail (TSVE- of the soft-tissue deformation models onto a graphic process- ing unit (GPU) platform rather than onto a central process- ChainMail) was proposed by Zhang et al. [54]. The method was developed from the traditional ChainMail method in ing unit (CPU) platform. Basically, GPUs are comprised of which the model is represented as a spring system. Zhou highly parallel architectures. Each separate GPU contains et al. have also proposed a Marquardt radial basis meshless numerous processors and memory segmentations, and each method (MRM) for the soft-tissue cutting [56]. In addition processor works independently on its own data distribution. Consequently, although the clock frequencies of GPUs are to these studies, it is important to note that a large range of soft-tissue models (brain, ligament, and atrioventricular often smaller than CPUs, the overall computation speed of valves) were also developed using the element-free Galerkin GPUs are much faster than CPUs, even when CPUs can be method and isogeometric method [61–65]. Due to the used composed of multiple processing cores up to now. Further- keywords, this present review does not include these works. more, various programming frameworks supported for model implementations have been improved in an easier and ﬂexible Thus, interested readers could use more speciﬁc keywords to get information about these methods. ways. Two classical interfaces have been employed for pro- gramming on GPUs have been OpenGL, application pro- gramming interfaces (APIs), DirectX, CUDA from NVIDA, 3.3.3. Hybrid Modelling Methods. Hybrid methods have been intensively investigated in the literature due to its cooperative and CTM from ATI. These frameworks have been written in high-level C-programming language which bring many functions which take advantages from multiple methods. For instance, although the mass-tensor method (MTM) is fast beneﬁts for modelers to implement their developed methods and suitable for simulation of the soft-tissue deformation in executing on GPU eﬀectively [17]. An analysis of GPU imple- mentations for surgical simulations was reviewed by Sørensen real time, it still lacked the realistic biomechanical character- istics, especially when simulating the nonlinear materials. On and Mosegaard [66]. They concluded that GPUs would become much powerful and cost-eﬀective platforms for the other hand, the FEM has realistic simulation of biome- chanical behaviors of soft tissues, but it has high computation implementing the soft-tissue deformation models in real- cost. Additionally, the precomputation-based methods time medical environments. However, to be able to achieve beneﬁts from this implementation approach, the developed (pre-comp FEM) have very high performances for simulating the soft-tissue deformations in real time based on the pre- modeling methods must be compatible and be able to recon- computed data from the FEM, but they cannot handle the ﬁgure with parallel computations [66]. The ﬁrst model imple- topological changes. Consequently, the combination between mentation strategy was proposed by Taylor et al. [17]. The MTM, FEM, and pre-comp FEM can not only simulate the authors implemented a model using the total Lagrangian explicit dynamic (TLED) FEM onto a NVIDIA GeForce deformation in real time but also handle cutting and tearing realistically with nonlinear materials. This approach was ﬁrst 7900 GT GPU platform, and the results showed that the com- developed by Cotin et al. [57]. The result of this study showed putation speed of the implemented model was much faster that the update frequency was able to reach at 40 Hz with an than the CPU-implemented model. A human brain model MTM having 760 vertices and 4000 edges. Yarnitzky et al. using the TLED-FEM was also implemented on the NVIDIA Tesla C870 GPU platform by Wittek et al., and the computa- combined the physically kinematic model with the local FE model to estimate the stresses and deformations inside a tional performance was also accelerated signiﬁcantly [24]. For plantar foot’s soft tissues during gait [58]. Allard et al. instance, with the brain model of 18,000 nodes and 30,000 ele- introduced a well-known SOFA framework supporting bio- ments (approximately 50,000 degrees of freedom), the average medical researchers modularly and ﬂexibly to develop new time for estimating the brain deformations was less than 4 s when implemented on GPU, and the time implemented on soft-tissue deformation models [13]. The framework was comprised of multiple modeling methods combined eﬀec- the CPU platform was up to 40 s. A model using the explicit tively to simulate the soft tissues according to their FEM in a real-time skin simulator was also implemented on 20 Applied Bionics and Biomechanics Not using interaction devices e 3-D viewers e 2-D optical cameras e biomechanical sensors e PC's human interface devices e 3-D optical cameras e 3D scanners e force feedback devices e virtual surgical instruments 0 5 10 15 20 25 # of studies Figure 6: The distribution of using interaction devices in the chosen literature. a GPU platform by Lapper et al. and the simulation results human interface devices, the 3D viewers, and the 2D and 3D could be accelerated to reach real-time goal [25]. Joldes et al. optical cameras. The statistic distribution of used interaction devices is shown in Figure 6. It is clearly showed that the vir- employed a GPU platform using a programming guide NVI- DIA Compute Uniﬁed Device Architecture (CUDA) to speed tual surgical instruments have been popularly used with 19 up the TLED-FEM human brain model [23]. Strbac et al. also studies, and the least used device was the 3D viewers and employed the model using TLED-FEM onto multiple the 2D optical cameras with only 3 studies. The second most General-Purpose Graphics Processing Units (GPGPUs) to popular interaction devices are the force-feedback devices which were found on 15 studies. Other remaining interaction evaluate the eﬃciency of this implementation with the current commercial solutions [34]. The experimental evaluations devices have been utilized by only from 4 to 6 studies. In fact, showed that when the size of the model increased from 125 most of the studies have taken advantage of the force- to 91,125 elements, the computational time was from 1 s to feedback devices always combined with the virtual surgical 1 h 39 min 37 s running on Abaqus commercial software, from instruments for interacting with the simulated model. More- over, other interaction devices certainly used in computer 0.149 s to 34.143 s running on the most powerful GPU of GTX980. Courtecuisse et al. proposed an implementation, systems, such as computer screens and computer keyboards, called linearized FEM (L-FEM), which was the combination are not deeply analyzed in this review paper due to their obvi- between the linear elastic material with a FEM [27]. Maﬁ ous contributions to the simulation system. et al. deployed a model using element-by-element precondi- 3.6. Virtual Surgical Instruments and Force-Feedback Devices. tioned conjugate gradient (EbE PCG) FEM method in the GTX470 GPU platform for speeding up the deformation com- The virtual surgical tools have been widely combined with putation in real time [32]. The implicit time integration of a force-feedback devices to communicate between a user and nonlinear FEM on the GPU platform was also performed by a simulated model so that the simulation system could Courtecuisse et al. [33]. become more ﬂexible and realistic. The functions of virtual surgical instruments are to transfer the controlled signals 3.5. Interaction Devices. In addition to the model develop- from external real devices to the simulated model and to feed- ment methods and the implementation approaches, the back the calculated biomechanical parameters from the simu- interaction devices contribute signiﬁcantly to the whole sys- lated model to the external haptic devices [2, 14, 44, 47]. The tem accuracy and computational time. After user commands speed of transmitting data from/to simulated models must be relatively high so that the visualization and haptic feedback are transferred to the computer system through input devices, the computer system must execute the simulated can be simulated realistically [2, 14]. Force-feedback devices model according to the commanded strategies. Once each are the input/output devices having a function of interfacing simulation iteration is completed, the estimated feedbacks between a user and a virtual surgical tool. When the interac- from the simulated model are transmitted to the user tions are received from the virtual surgical tool, the simulated model will react and calculate haptic forces during each sim- through the output devices. Consequently, the total accuracy of both input/output interaction devices and soft-tissue ulation iteration. These computed haptic forces are ﬁnally models must be at least equal to the desired accuracy toler- feedbacked to the device through the virtual surgical tool ances in each medical application. Diﬀerent interaction [12, 18–20, 27, 44, 53]. As a result, the user will feel like they devices have been used in the reviewed studies. However, are interacting with a real soft tissue when the reaction forces are received from the force-feedback device [3, 12]. For exam- due to the focused objectives on developing the modeling methods, up to 42% of reviewed studies did not use interac- ple, Cotin et al. used this combination in surgical simulation tion devices in their simulation systems. The interaction to provide haptic sensations for the surgeon [12]. Audette devices in the remaining studies could be divided into diﬀer- et al. used a 7-degree-of-freedom (DOF) haptic device com- ent types: the virtual surgical instruments and force-feedback bined with a surgical tool whose tip is ﬁxed at the end of the haptic device to make the simulation system more realistic devices, the 3D scanners, the biomechanical sensors, the PC’s Interaction devices Applied Bionics and Biomechanics 21 monitoring foot’s internal deformations under outside inter- [18]. In the study of Chen et al., a commercial PHANTOM haptic device with 3 DOF force feedback and 6 DOF position actions [58]. Electromyography (EMG) was also used in the and orientation was used for haptic surgery simulation [44]. study of Murai et al. for acquiring muscle tensions [5]. Sela et al. developed a new force feedback system called the SensAble™ PHANTOM Desktop™ haptic device [20]. The 3.6.3. The PC’s Human Interface Devices. Some PC’s human device was combined with a pen-sized handle, and they are interface devices have also been used widely in the real-time all connected to a robot arm with ﬂexible engine-forced joints soft-tissue simulation systems. They are all ﬂexible and easy to simulate a virtual surgical scalpel. Courtecuisse et al. used a for user manipulation. For example, Lopez-Cano et al. used a PC mouse as a surgical tool interacted with a virtual pointer virtual laparoscopic grasper, which was managed by a Xitact IHP haptic device [27]. Peterlik et al. used a virtual haptic to deform the 3D model of the human inguinal region [45]. interface point (HIP) controlled by a PHANTOM haptic This conﬁguration could simulate vertical and horizontal device to calculate haptic forces reacted from a liver model stretching deformations of the simulated model. In some based on displacements between the HIP’s position and the simulated systems, the PC mouse could only be controlled to rotate the view angle of the simulated model [21]. More- 3D surface model of the liver [3]. over, it could be used for drawing the cut shapes onto the 3.6.1. 3D Scanners. The 3D scanners can be divided into two surface model [48]. categories: structural and surface scanners. The most widely used structural scanners in the literature have been CT and 3.6.4. 3D Viewers. The 3D viewers are the output interaction devices. They are composed of two separate high-resolution MRI. Cotin et al. created an anatomical model of a human liver from MRI images [12]. Marchesseau et al. used CT screens or two glasses attached together horizontally for dis- images for creating the geometrical model of a liver [26]. playing two diﬀerent image frames to human eyes. Based on a Besides, 3D surface scanners were also used in the studies stereo geometrical model from human vision, the 3D viewers relating to surface-based soft-tissue modeling methods. The can create depth perceptions for human brains, so when using the 3D viewer for visualizing the simulated model, most popular surface-based scanners employed were laser scanners and ultrasonic scanners. They took advantage of the graphic rendering can become more realistic. A classic measuring the time-of-ﬂight of laser/ultrasound beams for 3D viewer was presented by Brown et al. using the stereo estimating the distance between the laser/ultrasound sources glasses for enhancing the illusion of depth in the video frames and the object’s surface. These scanners are fast and able to [2]. In the visualization system, there were two image frames displayed: one image frame was colored in red, and the others acquire the object surfaces in real time. The laser scanners are much more accurate than ultrasound scanners, but laser was colored in cyan. The stereo glasses included two diﬀerent beams can be very harmful to the living soft tissues during color ﬁlter glasses for the left and right lens, so at the same time, each human eye would see a diﬀerent image frame. long acquisition period. For example, Monserrat et al. employed the 3D ultrasonic scanner (SAC GP10, Smart Each pair of image frame was shifted horizontally for creating the depth information. A diﬀerent 3D viewing device called a EDDY System, USA) for capturing the 3D outside surface of the simulated object based on the boundary element 3D stereo visor was used to visualize a simulated model in the modeling method [4]. The combination between surface virtual reality myringotomy simulation in the study of Ho et al. [6]. This device included two diﬀerent high-resolution scanners and structural scanners was also proven to be eﬀec- tive for accurate reconstructing both surface and structural screens for displaying two diﬀerent image frames at the same time. details. Wang et al. combined a 3D laser scanner with the lat- eral X-ray scanners in their methods [48]. In fact, the 3D laser images containing both 3D geometrical point cloud 3.6.5. 2D and 3D Optical Cameras. The 2D optical cameras are the input interaction devices having the functions of and colors of the human face were transformed to the lateral X-ray image for comparing and cutting the nose part on the acquiring 2D image frames of object surfaces. In the applica- face. This combination provided a high-quality and patient- tion of facial expression recognition, Chandrasiri et al. speciﬁc model of the human face appearance. mounted a complementary metal oxide semiconductor (CMOS) camera to a headphone to capture 2D color video frames of a user face [42]. An ordinary web camera available 3.6.2. Biomechanical Sensors. Most biomechanical sensors used in soft-tissue modeling systems are electromagnetic sen- on a mobile device was also used by Weng et al. in the appli- cation of real-time facial animations [52]. In particular, the sors, force sensors, and electromyography sensors. Brown et al. used an electromagnetic tracker (miniBRID of Ascen- oﬄine 2D images acquired from a camera were also analyzed sion Technology Cooperation) to track behaviors of a real for detecting the facial expressions and cloning them to other 2D facial images in the study of Zhang et al. [51]. One of the surgical forceps [2]. Sedef et al. attached a force sensor (ATI Industrial Automation’s Nano 17) at the end of a real most drawbacks of 2D optical cameras is the inability of surgical probe for measuring forces inside a surgical trocar reconstructing depth information from a single view of so that the user could feel like being in a real minimally inva- vision, so multiple optical cameras have been cooperated to sive surgery [19]. Yarnitzky et al. employed ultra-thin force form a 3D optical camera system for detecting the 3D data. A motion capture device was utilized to capture 3D motions sensors arranged under the bony prominences of each foot for measuring the forces under the calcaneus, metatarsal of a human during dynamic movements in the study of heads, and phalanges in real time in an application of Murai et al. [5]. A stereo optical motion capture was also 22 Applied Bionics and Biomechanics liver simulation system [3]. The main thread called haptic combined with facial markers in the study of Wan et al. for detecting facial animations [49]. Over 36 facial markers were thread is acquiring positions of a haptic device, detecting col- detected and followed by mocap, and their motions were lisions, calculating haptic forces, and computing model’s deformations. The simulation system designed in the study then converted to MPEG-4 standard’sdeﬁnition of facial ani- mations. Woodward et al. used an oﬀ-the-shelf stereo web- of Goulette et al. also included multiple computation mod- cam for marker-based facial animation application [55]. ules for accelerating the system execution [53]. Two modules Applied to minimally invasive surgeries in the study of were threaded to execute in parallel on an Intel Core 2 Duo at Moroka et al., the 3D optical camera was integrated into a 2.40 GHz, 3.45 GB of RAM. As a result, the visual rate could be reached up to 47 FPS. Audette et al. designed a surgical stereo endoscopy whose size was small enough to be used in restricted navigation spaces [31]. simulation system which included their own developed hap- tic device with 7 DOFs [18]. To control this haptic device, an 3.7. System Architectures. Computational approaches and intelligent I/O board, called the DAP5216a/626, operating interaction devices have been developed throughout the individually with the computer system was proposed. Another scheme for system execution called a multimodel literature to improve computational accuracy and speed of soft-tissue deformation models, but they will not operate representation, was proposed by Allard et al. within the eﬀectively and robustly in real time if soft-tissue models SOFA framework [13]. In this scheme, each soft-tissue simu- and interaction devices are not well-cooperated in a system lation components could be represented by multiple model- architecture. This section will synthesize system execution ing methods related to real-time deformation simulation, accurate collision detection, or realistic interaction computa- schemes and programming frameworks of the system archi- tectures developed in the literature. tion. Finally, the task of programmers was to design a switcher to eﬀectively alternate modeling methods according 3.7.1. System Execution Schemes. Cotin et al. designed the to each appropriate simulation issue. ﬁrst execution scheme called the distributed execution 3.7.2. System Frameworks. System development frameworks scheme in which a computer system and a Dec AlphaStation were closely cooperated [12]. The computer system is aimed must be selected carefully so that the system could be devel- at computing the haptic forces and exchanging data with the oped both in high productivity and short time-to-market. haptic device while the Dec AlphaStation visualized the Generally speaking, a software framework is a generalization deformations of this soft-tissue model in real time. The com- software structure in which programmers can contribute their written codes to modify this structure to a speciﬁc appli- munication environment between two computer systems was the ethernet connection. Chen et al. distributed a developed cation. Taking advantages of available conﬁgurations and haptic surgery simulation onto two computing systems prebuilt libraries, simulation systems could be developed much more ﬂexibly and faster than in traditional develop- [44]. While an SGI Prism Visualization Server with 4 ATI FireGL GPUs covered graphical simulations, a Windows ment procedures. The system frameworks can be divided into four groups: the input/output interaction frameworks, computer system controlled the haptic devices and simulated the haptic feedbacks. The system could manage more than the graphic interaction frameworks, the modelling frame- one simulated model by using a new peripheral protocol works, and the hybrid frameworks. Regarding the input/out- put interaction frameworks, haptic devices have been called virtual reality peripheral network (VRPN) developed by the University of North Carolina. Two workstation sys- commonly used in the literature, and they are often inter- faced with computer systems by GHOST [3, 19, 44] and tems were also cooperated on a simulation system for the minimally invasive surgery proposed by Morooka et al. PHANTOM [44] input interaction framework. These input [31]. All model computations were performed by the ﬁrst interaction frameworks are all free and open source. More- over, other standard input interaction devices, such as key- workstation while the second workstation only performs visualization of deformations and virtual tools in the form boards, PC mouse, web cameras, and microphones, can of a 3D stereo vision for improving depth sensations. Note also interface with computer systems through application that the limitation of transmission bandwidth leaded to the programming interfaces (APIs) supported by Microsoft latency between the visualization force-feedback. To solve Windows systems [42, 45, 48]. Regarding the graphic inter- action frameworks, the most employed graphic framework this issue, the multithread execution scheme was proposed, Brown et al., which distributed two tasks of deformation was OpenGL in which 2D and 3D vector graphics can be rapidly rendered by GPU-platform boards. The rendering visualization and collision detection on two diﬀerent execu- tion threads on a single dual-processor machine (Sun Ultra tasks can be executed on a separate computer system or on 60 with two 450 MHz processors) [2]. The system included a local thread [3, 6, 18, 19, 38, 44, 45, 59]. In particular, the OpenGL framework can be embedded in multiple types three intercooperative simulators: a deformable object simu- lator, a tool simulator, and a collision detection module. The of operating systems such as Android, iOS, Linux, Windows, idea of multiple-thread executing on a single computer sys- and various embedded operating systems. Moreover, it can tem was also applied by Sedef et al. in a soft-tissue simulation also support for writing in multiple programming languages system including a phantom haptic device, a computer (e.g., C++, Python, C#, and Cg). In addition, the CUDA™ graphic framework was developed by Nvidia Corporation. screen, and a simulated model [19]. Peterlik et al. also imple- mented two asynchronous computation threads executed on There have been numerous studies using CUDA framework an AMD Opteron Processor 250 (2 GHz) PC to operate a for implementing their simulation system on of-the-shelf Applied Bionics and Biomechanics 23 tom could be used to give the validating data for the geo- graphic GPU boards and achieving great beneﬁts from parallel execution structure in real-time computations metrical validation [7]. Regarding the model behavior [17, 23, 24, 27, 32, 34]. Another general graphic framework validations, the physical characteristics of the simulated models must be assessed with the real physical data at diﬀer- called OpenCL™ was also developed for ﬂexible parallel implementation. An image processing framework called ent deforming states. One of the most popular schemes is to Virtual Place (AZE Co.) was also used for converting 3D use the calculated data from a standard commercial simula- deformations to stereo video frames for creating depth feel- tion software as baseline data. For example, a model using ing on human visions [31]. Regarding the modelling frame- linear viscoelastic FEM was validated through a compression test solved by both the proposed computation approach and works, GHS3D [26], TetGen [47], and CDAJ-Modeler [31] were employed for generating mesh models from CT/MRI the ANSYS ﬁnite element software package. Obtained results images. Additionally, Maxilim software could also support showed that the maximum error of displacement was less for boundary condition simulations [43]. Moreover, the than 1% [19]. The ANASYS software package was also used CHOLMOD open source library could also be used for solv- for validating a model based on a machine learning-based FEM method [36]. Recently, the Marquardt-based model ing the linear systems in real time [50]. Finally, the combina- tional frameworks have been developed to provide a much has also been validated by ANSYS in a liver simulation sys- more ﬂexible and multifunctional environment for develop- tem [56]. The Abaqus software was also used for validation ing a whole system. MATLAB is a powerful combinational purpose [17, 24, 32]. Other used FE packages relate to MSC framework including a facial analysis toolbox used for facial NASTRAN 2003 which was used by Yarnitzky et al. [58] and LS-DYNA™ which was used by Joldes et al. [15]. Note expression analysis [42]; a toolbox called iso2mesh was used to generate a tetrahedral mesh of a brain and its tumor [7]; an that open source packages were also employed. The SOFA artiﬁcial neuro network toolbox was employed to train the framework was the execution environment for performance force-deformation data [7]; an optimization toolbox was evaluation between the developed method and the previous used to obtain optimal parameters of simulation models that modelling methods [26]. In addition to geometrical and model validations, user acceptability/safety validation needs represent for simulated physical quantities; the OpenGL graphic library could be supported in the MATLAB environ- to be performed to evaluate the quality of interfaces between ment for simulating interaction between soft-tissue model the system and its users in real clinical applications. One of and surgical tools [47]. An Android programming platform the most popular schemes of this validation level is to collect was also used [52]. Additionally, supporting for FEM physi- feedbacks from experts and patients who have been experi- enced with the developed system. Ho et al. validated their vir- cal modelling, the Fast FE Modelling Software Platform [14] and GetFEM++ [3] were also employed. For parallel tual reality myringotomy simulation system by a face-validity threading, the RTAI-patched Linux was used for satisfying study in which a validated questionnaire was delivered to eight otolaryngologists and four senior otolaryngology res- the hard real-time requirements [18]. Other powerful and more multifunctional system frameworks are the SOFA idents for evaluating the system after a long period inter- action with the simulator [6]. Tonutti et al. conducted [13, 26, 29, 33] and CHAI3D [6] frameworks. In fact, they support various libraries and modules for implementing a their validation procedure on surgeons with and without complete simulation system including input/output inter- implementing the developed system, and the diﬀerence results were evaluated for proving the eﬀectiveness of the action device drivers, geometrical model libraries, model- ling algorithms libraries, and graphic rendering modules. simulation system [7]. 3.8. Clinical Validations. To translate the outcomes of the 4. Discussion developed simulation systems into clinical routine practices, a systematic validation must be required. All validation 4.1. Computational Approaches. The FE modelling methods eﬀorts done in the literature were ordered as three validation and its variations have been commonly used for developing levels: geometrical validations, model behavior validations, soft-tissue deformation models. However, the trade-oﬀ and user acceptability/safety validations. between system accuracy and computation speed remains a The most important components in a simulation system challenging issue. The FEM can simulate deformations for are the geometrical and physical models. To accurately simu- complex soft tissues [67, 68]. In particular, a commercial late the target soft tissues, the geometrical appearances of FE solver was commonly used to evaluate the accuracy of a both the simulated models and real objects must be well- new FE algorithm [21, 22, 24, 32]. However, using the FE ﬁtted. Geometries at a speciﬁc state are commonly compared method, real-time requirements are only satisﬁed if being with the real in vivo/in vitro data acquired from a relatively modelled with a smaller number of elements [30, 35] or being accurate measurement method. CT/MRI were usually used accelerated by GPU implementation [23, 24]. In general, the to reconstruct the real 3D geometrical models of the tissues computational cost of the FEM increases exponentially with of interest. Due to expensive processing time and resources, the expansion of the number of nodes especially in case of this scheme is just suitable for oﬄine geometrical validation. simulation of the nonlinear materials. Consequently, various For example, real data from patients under maxillofacial development techniques have been developed to improve surgeries were stored and compared with the predicting their computation speeds. The most used approach was the appearances for improving the reproducibility capacity of precomputation-based technique. This approach has been the simulation system [43]. Moreover, the soft-tissue phan- proven as a fast and robust technique for simulating 24 Applied Bionics and Biomechanics This method could be considered for the compromising solu- deformations in real time, but large deformations with com- plex material properties and constitutive laws and topological tion between the biomechanical accuracy and computation changes on the ﬂy could not be handled during the system eﬃciency in real time. A diﬀerent technique was point collocation-based method of ﬁnite sphere (PCMFS) [46]. iterations because the trained model cannot be updated online [3, 7, 8, 12, 19, 20, 31, 36, 37]. Other FEM variations By just focusing on the local region of interests, the simula- could signiﬁcantly improve the performance of the FEM. tion time could be decreased signiﬁcantly, but the method One of them was the case of linearizing the kinematic of for detecting the region of interest was still not deﬁned eﬀec- the simulated object [14] in which the model could be cut fas- tively. More globally, the method called inverse dynamic computation (IDC) [5] could compute the muscle tensions ter than the original model using FEM, but the speed was not fast enough for realistic visualization in medical applications. based on the external data from sensors. Despite of the The idea of dividing a FEM mesh into multiple submeshes to acceptable accuracy, the method could not analyze a single be executed in parallel [18] could initially increase the com- muscle. This idea could be found in the method called elas- putation speeds, but this method leaded to the limited num- tic-plus-muscle-distribution-based (E+MD) [51] in which the facial expression could be used for investigating the inter- ber of threads being able to handle on a real-time operation system. Other development methods such as matrix system nal muscle tensions. A recent method called time-saving reduction (MSR-FEM) [21] and the order reduction method volume-energy-conserved ChainMail (TSVE-ChainMail) (ORM-FEM) [29] based on the reduction of the FEM’s stiﬀ- [54] was powerful in handling both topological changes and ness matrix could improve the processing time, but they just simulating the isotropic, anisotropic, and heterogeneous materials at the real-time rate. At this stage, the real-time simulated small deformations. The total Lagrangian algo- rithm in conjunction with FEM [22] and its modiﬁcation deformations could be achieved but the interactions among known as total Lagrangian explicit dynamic (TLED-FEM) the simulated objects remains a diﬃcult task. To overcome [16] allowed element precomputations, so a less computation this drawback surface-based methods (e.g., boundary ele- cost would be required for each time step. It is important to ment method (BEM) [4], Laplacian surface deformation (LSD) [41], and Marquardt radial basis meshless method note that hyperelastic and viscoeleastic constitutive models were implemented with explicit time integration schemes in (MRM) [56]) have been proposed. These approaches could a straightforward manner using homemade or commercial estimate the internal deformations based on the surface changes, and it could handle the interaction between mod- FE solvers. Moreover, multiplicative Jacobian Energy Decom- position (MJED-FEM) [26] with implicit time integration elled soft tissues through surface interactions, but they were not able to simulate inhomogeneous materials, nonlinear schemes could be used to model hyperelastic, viscoelastic, and poroelastic behaviors of the soft tissues. However, these elastics, and topological changes on the ﬂy. In addition, methods could not handle interactions with other simulated model cutting and needle penetration issues were also stud- ied using the extended ﬁnite element method (XFEM) and objects and topological changes. The topological changes could be handled in the method proposed by Turkiyyard meshfree-based approaches for soft tissues. In particular, the extended ﬁnite element method (XFEM) has been used et al. [28], but they could not solve eﬀectively the cured cuts, partial cuts, and multiple cuts inside elements. More eﬀec- to study complex hard tissue (tooth [69], maxillary molar, tively for simulating the topological changes was element- and endodontic cavities [70]) models with fracture and crack propagation behaviors and soft tissue (cornea [71]) models by-element precondition conjugate gradient FEM (EbE PCG-FEM) [32] method, but it was not suitable for simulating with cutting simulation. This open new avenue to model bio- logical tissues with more complex interaction behaviors. the heterogeneous materials. Another potential method called preconditioning FEM (pre-cond FEM) [33] could solve this In addition, many studies have been conducted for the problem dramatically. It could both simulate the topological implemented model based on developed soft-tissue deforma- tion method on to the GPU-parallel computing platform, changes and the haptic feedback of homogeneous and hetero- geneous materials with acceptable accuracy and real-time and they can all achieve much better accelerations when frame rates. compared with the conventional developing approach. Not On the other hand, the meshfree-based techniques have all developed modeling methods are suitable for this been achieved great attention in the recent years. All the approach, so the implicit time integration of nonlinear FEM method has been proven to be the most suitable for par- meshfree-based methods have been very fast and highly adaptive to topological changes, but they are less realistic allel implementation. Furthermore, the additional reconﬁ- than the mesh-based methods from biomechanical point of gurations must be approved to the current methods to view. The most popular meshfree-based method was mass- adapt with the implemented hardware platforms. When the spring system (MSM) [2, 6, 38, 44, 45]. This method could model developing approach reaches its limitation, new implementation strategies will be necessary for accelerating handle the deformations and topological changes, but it could not simulate accurately with nonlinear material char- their current computation performances. acteristics. The improvements of MSM method were the It is important to note that the computation speed and mass-tensor method (MTM) [43] and mass-spring-damper resources depend on each particular application (e.g., surgi- (MSD) method [47]. They could handle the nonlinear mate- cal planning or surgical simulation) of soft-tissue deforma- tion systems. For example, real-time soft-tissue deformation rial more eﬀectively than the MSM due to the use of nonlin- ear mass springs in the method. Another improvement of behavior, high-speed device interaction, and skill-based MSM was the hyperelastic mass link (HEML) method [53]. training ability could be more important criteria to be Applied Bionics and Biomechanics 25 and right scenes [2, 6]. Even more realistically, the haptic achieved for a computer-aided surgical simulation system. Besides, surgical planning system focuses on the whole feedback devices receive calculated haptic forces from sim- workﬂow from data acquisition and previsualization of a ulated models to create collision feeling for human tactile [3, 12, 18, 20, 27, 44, 53, 57]. Consequently, the cooperation speciﬁc surgical intervention and then predeﬁne the optimal surgical steps. between the 3D viewers and the haptic feedback devices will Generally speaking, a large range of methods were devel- become much more powerful in generating realistic sensa- oped to simulate the soft-tissue deformations. Each method tions for humans [2, 6]. In particular, force-feedback devices showed its robustness and accuracy for a speciﬁc case study. have been commonly used for many medical applications (e.g., surgical simulation, surgical trainings, or minimally There exists no universal methods, and the selection of the methods depends directly on the application. It is important invasive surgeries [18, 44, 53]). Force-feedback devices have to note that real-time deformations with topological changes been ﬂexibly cooperated with various types of virtual surgical on the ﬂy, and accurate object interactions remain challeng- tools (e.g., virtual haptic interface point (HIP), the virtual ing issues. One of the potential solutions relates to the use blade, or the virtual scalpel). The most widely haptic device used in the literature is the SensAble™ PHANTOM Desk- of multiple modeling methods in a whole simulation system. However, an eﬀective cooperation strategy should be estab- top™ haptic device, and its ﬂexibilities are dependent on the lished, and the requirement of advanced computational number of DOFs. It is important to note that to simulate resources needs to be satisﬁed. force feedbacks realistically, the haptic forces must be esti- Finally, the computation speed of a soft-tissue simulation mated and transferred to the force-feedback device at speeds from 500 Hz to 1000 Hz, so this means that the computation system depends strongly on the use of constitutive behavior laws for modeling soft-tissue physiology. Elastic, hyperelas- speeds of simulated models must be faster than those speeds tic, and viscoelastic laws were commonly used in the devel- [18, 57]. Furthermore, a separate controller must be installed oped real-time simulation systems for the upper/lower limb and executed one or multiple computer system to keep the muscle, facial muscle, liver, and skin tissues. It is important real-time computation speed [3, 19, 27, 44, 53]. On the other hand, several biomechanical quantities have to note that more complex constitutive laws such as electro- mechanical models could be used in general for modeling not been measured directly from the biomechanical sensors, the skeletal muscle [68] or myocardium [72, 73]. However, so simulated models are often used to infer internal physical these complex models deal with additional computational characteristics based on external knowledge. For example, in need and requirements to reach a real-time ability for medi- the case of musculoskeletal tracking, EMG sensors are often fused with musculoskeletal models for inferring individual cal simulation systems. Linear and nonlinear stress-strain relationships were described in the elastic material. Hypere- muscle tensions according to the markers’ motions, which lastic material was described using Neo-Hookean and are tracked by the 3D optical camera system [5, 58]. The soft-tissue physical parameters can also be inferred from Mooney-Rivlin formulations. It is important to note that some additional components were integrated into linear elas- soft-tissue deformation models by the movements of surface markers instead of direct measurements from the sensors. tic law to improve the computation speed and model accu- racy. For example, the combination of a linear elastic law The surface makers are proven to be very robust and ﬂexible with a corotational method was performed (Courtecuisse for estimating outside deformations, but the limited number of markers being able to put on a soft-tissue surface leads to et al. [33]) or an extra mass-spring model was integrated into a linear elastic law (Zhu and Gu [59]). Regarding all analyzed decrease the estimated deformation resolutions and so are the resultant calculations [31, 49, 55]. The 3D scanners such simulation systems for soft tissues, the most used law is the linear elastic one. The use of more complex laws (hyperelastic as MRI/CT scanners [8, 12, 26] and 3D ultrasonic scanners and viscoelastic) leads to a larger number of model parame- [4] have been employed for detail surface reconstruction in 3D spaces, but their slow acquisition times (in case of ters and of course computation speed. MRI/CT scanners), lacking of surface characteristics (in case 4.2. Interaction Devices in Real-Time Simulation Systems. of 3D laser scanners and ultrasonic scanners), and harmful There have been various kinds of interaction devices contrib- infections to human health (in case of CT and laser scanners) uting diﬀerently to the system’s reality and computation per- make them not suitable for tracking external deformations of soft tissue in real time and in long-term use. This issue was formance. While the output interaction devices mainly provide realistic visualizations and reactions to human initially resolved by the combination between the 2D optical senses, the input interactions have the fundamental involve- cameras and the X-ray images for adding more surface char- ment to the computation performance, especially in both acteristics [48], but the appearances were static and could not model accuracy and computation speed. Regarding the out- estimate deformations on the ﬂy. Consequently, other devices having the ability of acquiring both detail surface put interaction devices, the computer screens display appear- ances of simulated models and their deformations when deformations in 3D spaces and surface characteristics online interacted with virtual surgical instruments and/or other sur- are substantially required for improving computation speeds rounding structures [6]. However, their lacking of depth and model accuracy of soft-tissue simulation systems. information makes visualizations possible only in 2D space. 4.3. Suitable Execution Scheme in Simulation Systems. To The 3D viewers can complement this drawback. Like human visions, this interaction devices can create 3D virtual sensa- manage the data transmission from/to I/O interaction tion for human vision based on the diﬀerences between left devices and to compute the simulated model in an optimal 26 Applied Bionics and Biomechanics Finally, the user acceptability/safety validation was per- way, a suitable system execution scheme must be developed. There are two main system execution schemes. In the formed with the end users including patients, trainees, and distribution-based scheme [12, 31, 44], system tasks are experts through questionnaires. Note that this approach is relatively subjective and qualitative. In addition to these val- highly parallelized in multiple computing machines, which are interfaced through a limited bandwidth and slow- idation levels, system validation should be performed in transmission environments. This scheme allows system tasks which the whole system was evaluated rather than each sys- to execute independently and take advantages of multiple tem’s components. This stage targets at analyzing system computing hardware, but the problems appear when having functions, system robustness, and system computation per- formances during short-term and/or long-term working delays in communication between multiple machines. Thus, data transmissions are still not fast enough for eﬀectively durations. While the system functions are relatively easy to communicating among multiple computer systems. This verify by comparing with the proposed development func- issue has been initially solved by numerous attempts such tions at the designing stage, the system robustness and com- as high-speed ethernets and high-speed data transmission putation performances must be tested after short-term and long-term working durations. Although this validation pro- protocols, but they are not eﬃcient enough for transmitting large information in real-time. In the multithread scheme cess is necessary for a stable and robust system, rarely, studies [2, 3, 19, 53], system tasks are executed on multiple comput- in the literature conduct this validation. ing threads. Because all threads are connected through a very high-speed internal bus, there is nearly a delay in data trans- 5. Current Trends, Limitations and mission among threads. However, because of the limitation Future Recommendations of computation strength and memory capacity of a single thread inside a computer system, the simulation task(s) must The trends of the current computational approaches relate to be simpliﬁed and optimized to be able to execute on a single (1) mathematical formulation of physical laws applicable on thread. This can be a challenging task for model develop- image-based soft tissue geometries, (2) real-time simulation ments and implementations. Fortunately, this challenge can achievement of soft-tissue deformation with simple constitu- be easily resolved by the development of hardware technol- tive laws, and (3) model implementation on speciﬁc hard- ogy with more threads integrated on a single CPU or even ware conﬁguration to speed up the computational time. more CPU facilitated on a single computer system. In addi- However, soft-tissue behavior is commonly anisotropic, vis- tion, cooperation of the two execution schemes was also coelastic, inhomogeneous, and nearly incompressible with found in the literature [13, 18]. In this case, various types of large deformation. In fact, the consideration of all physiolog- data acquisition boards have been developed to fast manage ical aspects is practically diﬃcult, particularly for a real-time the input/output data streams. These boards are designed in simulation system. Thus, modeling assumptions related to a mobile hardware and can be easily plugged in to a comput- constitutive laws, geometrical discretization, and boundary ing machine through a speciﬁc high-speed transmission and loading conditions were commonly performed for a spe- channel and a software driver. Consequently, this system ciﬁc application. Further studies need to be investigated to conﬁguration can take advantages of both distribution- develop more accurate computational approaches for simu- based and multithread-based execution schemes. lating complex soft-tissue behaviors in real-time conditions. The hybrid modelling approach in combining several 4.4. Clinical Validations. Generally speaking, the clinical val- methods is a potential solution leading to maximize the idation is the ﬁnal system development stage to determine advantage of each method and overcome the limitations of whether the simulation system is acceptably suitable for clin- the other. ical routine practices. Current clinical validation procedures Concerning the interaction devices, the ability of acquir- were grouped into three levels: geometrical validations, ing multiple types of data both in real-time and accurate model behavior validations, and user acceptability/safety val- manner and the portability of sensors are the current trends. idations. Regarding geometrical and model behavior valida- Multiple sensors could be embedded into a single well- tions, the validation data have been commonly acquired calibrated structure and worked as an independent conﬁgu- from standard simulation software, phantom soft-tissue ration. These types of sensors, therefore, are more accurate organs, or postoperation data. There is a lack of in vivo data and faster than manually calibrated sensor systems. In fact, for accurately validating the simulation system in real medi- some multiple function sensors such as the KINECT™ devel- cal environments. The use of accurate CT/MRI data is prom- oped by Microsoft®, XTION™ developed by Asus, and other ising, but this approach is not suitable for online validation. well calibrated stereo cameras are good recommendations for The use of standard simulation software to validate the phys- this requirement. However, current sensors are diﬃcult to ical behaviors of simulated models also faces some problems. acquire deep information on the soft tissues, which are cru- It is important to note that most of soft-tissue materials (e.g., cial for in vivo modeling and simulation. In particular, there muscle, fat, and skin) are unavailable in these types of soft- have been no sensors having the ability of acquiring these ware. Thus, only simpliﬁed behaviors were validated with data in real time, so there is a need for a new type of sensor classical mechanical laws (e.g., linear elastic or hyperelastic that can get the internal structures and/or textures in real laws). Consequently, more experimental protocols should time. In fact, complex data processing schemes need to be be investigated to characterize the soft-tissue behaviors investigated in the future to study the external-internal rela- and use them for enhancing model behavior validations. tionship of the soft tissues leading to a predictive solution of Applied Bionics and Biomechanics 27 tions. This review provides useful information to describe internal structures from external information. Statistical shape modeling (SSM) or artiﬁcial intelligence- (AI-) based how each aspect has been developed and how they have been approaches are potential methods for such complex objective. cooperated for both executing in real time and keeping real- Regarding the system architecture and execution scheme, istic behaviors of soft tissues. By clearly analysing advan- the availability of powerful and open frameworks for medical tages and drawbacks in each system development aspect, imaging processing (e.g., 3D Slicer), data visualization (e.g., this review paper can be used as a reference guideline for OpenGL, VTK), and simulation (e.g. SOFA) is the current system developers to choose their suitable system’s compo- trend, which can speed up the development of new systems. nents while developing soft-tissue simulation systems. However, the compatibility between these frameworks Finally, this review paper identiﬁed some recommendations becomes a potential drawback. To deal with this obstacle, for future researches. the community should work together to deﬁne a common computational protocol and promote its use within any sys- Conflicts of Interest tem development for future applications. In addition, all developed system execution schemes are very hard to pro- The authors declare that there is no conﬂict of interest gram without the help of system frameworks. Most used regarding the publication of this paper. system frameworks mainly supported for programming mul- tithread schemes rather than distributed schemes and combi- Acknowledgments nation schemes. Moreover, they did not well manage the memories between internal threads. For further recommen- This work was carried out and funded in the framework of dations, more system frameworks should be developed for the Labex MS2T. It was supported by the French Govern- supporting the communication between threads. Further- ment, through the program “Investments for the future” more, frameworks for programming, the distributed schemes managed by the National Agency for Research (Reference also need to be investigated for supporting connection and ANR-11-IDEX-0004-02). We acknowledged also the Région data transmission between multiple computing machines. Hauts-de-France for funding. More software development kits (SDKs) should be developed for general sensors such as single cameras, stereo cameras, References and laser scanners so that deeper information could be extracted and estimated. In addition, to translate developed [1] W. Sun and P. Lal, “Recent development on computer aided systems into clinical routine practices, software development tissue engineering — a review,” Computer Methods and Pro- workﬂow dedicated for medical application should be grams in Biomedicine, vol. 67, no. 2, pp. 85–103, 2002. followed to ensure a high-quality and reliable medical soft- [2] J. Brown, S. Sorkin, J.-C. Latombe, K. Montgomery, and ware for the beneﬁt of involved patients and clinicians. M. Stephanides, “Algorithmic tools for real-time microsurgery Finally, multilevel validation becomes an avoidable task simulation,” Medical Image Analysis, vol. 6, no. 3, pp. 289–300, when developing a real-time medical simulation with soft- tissue deformation. Quantitative assessment of accuracy is [3] I. Peterlík, M. Sedef, C. Basdogan, and L. Matyska, “Real-time also focused. However, when developing a soft-tissue sim- visio-haptic interaction with static soft tissue models having ulation system, a systematic validation procedure must be geometric and material nonlinearity,” Computers & Graphics, simultaneously proposed. Speciﬁcally, a system must be vol. 34, no. 1, pp. 43–54, 2010. sequentially validated through all validation levels. The [4] C. Monserrat, U. Meier, M. Alcaniz, F. Chinesta, and M. C. type of validating data for each validation process should be Juan, “A new approach for the real-time simulation of tissue clearly deﬁned, and acquisition processes should also be deformations in surgery simulation,” Computer Methods and Programs in Biomedicine, vol. 64, no. 2, pp. 77–85, 2001. planned. Moreover, an assessment program could be included in the system to evaluate the development progress of the [5] A. Murai, K. Kurosaki, K. Yamane, and Y. Nakamura, “Muscu- loskeletal-see-through mirror: computational modeling and users during each level of trainings. In user acceptability/saf- algorithm for whole-body muscle activity visualization in real ety validations, a supervising task programmed to run simul- time,” Progress in Biophysics and Molecular Biology, vol. 103, taneously with the simulation system could be used to track no. 2-3, pp. 310–317, 2010. users’ behaviors. Additionally, a function that include ques- [6] A. K. Ho, H. Alsaﬀar, P. C. Doyle, H. M. Ladak, and S. K. tionnaires for evaluating user acceptability could be included Agrawal, “Virtual reality myringotomy simulation with real- in the system’s functions, so the results are automatically time deformation: development and validity testing,” The processed and sent to developers for further development. Laryngoscope, vol. 122, no. 8, pp. 1844–1851, 2012. [7] M. Tonutti, G. Gras, and G. Z. Yang, “A machine learning 6. Conclusion approach for real-time modelling of tissue deformation in image-guided neurosurgery,” Artiﬁcial Intelligence in Medi- The present review paper was conducted to summarize the cine, vol. 80, pp. 39–47, 2017. literature works about real-time soft-tissue deformation sys- [8] D. Lorente, F. Martínez-Martínez, M. J. Rupérez et al., “A tems. Throughout this review, studies relating to real-time framework for modelling the biomechanical behaviour of the soft-tissue simulations have been analysed according to four human liver during breathing in real time using machine system engineering aspects: computational approaches, learning,” Expert Systems with Applications, vol. 71, pp. 342– interaction devices, system architectures, and clinical valida- 357, 2017. 28 Applied Bionics and Biomechanics [24] A. Wittek, G. Joldes, M. Couton, S. K. Warﬁeld, and K. Miller, [9] H. Delingette, “Toward realistic soft-tissue modeling in medical simulation,” Proceedings of the IEEE, vol. 86, no. 3, “Patient-speciﬁc non-linear ﬁnite element modelling for pre- pp. 512–523, 1998. dicting soft organ deformation in real-time; application to non-rigid neuroimage registration,” Progress in Biophysics [10] A. Nealen, M. Müller, R. Keiser, E. Boxerman, and M. Carlson, and Molecular Biology, vol. 103, no. 2-3, pp. 292–303, 2010. “Physically based deformable models in computer graphics,” Computer Graphics Forum, vol. 25, no. 4, pp. 809–836, [25] R. J. Lapeer, P. D. Gasson, and V. Karri, “Simulating plastic 2006. surgery: from human skin tensile tests, through hyperelastic ﬁnite element models to real-time haptics,” Progress in Bio- [11] D. Moher, A. Liberati, J. Tetzlaﬀ, and D. G. Altman, “Preferred physics and Molecular Biology, vol. 103, no. 2-3, pp. 208–216, reporting items for systematic reviews and meta-analyses: the PRISMA statement,” International Journal of Surgery, vol. 8, no. 5, pp. 336–341, 2010. [26] S. Marchesseau, T. Heimann, S. Chatelin, R. Willinger, and H. Delingette, “Fast porous visco-hyperelastic soft tissue [12] S. Cotin, H. Delingette, and N. Ayache, “Real-time elastic model for surgery simulation: application to liver surgery,” deformations of soft tissues for surgery simulation,” IEEE Progress in Biophysics and Molecular Biology, vol. 103, Transactions on Visualization and Computer Graphics, vol. 5, no. 2-3, pp. 185–196, 2010. no. 1, pp. 62–73, 1999. [27] H. Courtecuisse, H. Jung, J. Allard, C. Duriez, D. Y. Lee, and [13] J. Allard, S. Cotin, F. Faure et al., “SOFA - an open source S. Cotin, “GPU-based real-time soft tissue deformation with framework for medical simulation,” Stud Health Technol cutting and haptic feedback,” Progress in Biophysics and Inform, vol. 125, pp. 13–18, 2007. Molecular Biology, vol. 103, no. 2-3, pp. 159–168, 2010. [14] J. Berkley, G. Turkiyyah, D. Berg, M. Ganter, and S. Weghorst, [28] G. M. Turkiyyah, W. B. Karam, Z. Ajami, and A. Nasri, “Mesh “Real-time ﬁnite element modeling for surgery simulation: an cutting during real-time physical simulation,” Computer- application to virtual suturing,” IEEE Transactions on Visual- Aided Design, vol. 43, no. 7, pp. 809–819, 2011. ization and Computer Graphics, vol. 10, no. 3, pp. 314–325, 2004. [29] S. Niroomandi, I. Alfaro, E. Cueto, and F. Chinesta, “Account- ing for large deformations in real-time simulations of soft tis- [15] G. R. Joldes, A. Wittek, and K. Miller, “Suite of ﬁnite element sues based on reduced-order models,” Computer Methods algorithms for accurate computation of soft tissue deformation and Programs in Biomedicine, vol. 105, no. 1, pp. 1–12, for surgical simulation,” Medical Image Analysis, vol. 13, no. 6, pp. 912–919, 2009. [30] T. Wu, A. P. L. Hung, P. Hunter, and K. Mithraratne, “Model- [16] K. Miller, G. Joldes, D. Lance, and A. Wittek, “Total Lagrang- ling facial expressions: a framework for simulating nonlinear ian explicit dynamics ﬁnite element algorithm for computing soft tissue deformations using embedded 3D muscles,” Finite soft tissue deformation,” Communications in Numerical Elements in Analysis and Design, vol. 76, pp. 63–70, 2013. Methods in Engineering, vol. 23, no. 2, pp. 121–134, 2007. [31] K. Morooka, Y. Nakasuka, R. Kurazume, X. Chen, [17] Z. A. Taylor, M. Cheng, and S. Ourselin, “High-speed nonlin- T. Hasegawa, and M. Hashizume, “Navigation system with ear ﬁnite element analysis for surgical simulation using real-time ﬁnite element analysis for minimally invasive sur- graphics processing units,” IEEE Transactions on Medical gery,” in 2013 35th Annual International Conference of the Imaging, vol. 27, no. 5, pp. 650–663, 2008. IEEE Engineering in Medicine and Biology Society (EMBC), [18] M. A. Audette, V. Hayward, O. Astley, M. Doyon, G. A. pp. 2996–2999, Osaka, Japan, 2013. McCallister, and K. Chinzei, “A PC-based system architecture [32] R. Maﬁ and S. Sirouspour, “GPU-based acceleration of com- for real-time ﬁnite element-based tool-speciﬁc surgical simula- putations in nonlinear ﬁnite element deformation analysis,” tion,” International Congress Series, vol. 1268, pp. 378–383, International Journal for Numerical Methods in Biomedical Engineering, vol. 30, no. 3, pp. 365–381, 2014. [19] M. Sedef, E. Samur, and C. Basdogan, “Real-time ﬁnite- element simulation of linear viscoelastic tissue behavior based [33] H. Courtecuisse, J. Allard, P. Kerfriden, S. P. A. Bordas, S. Cotin, and C. Duriez, “Real-time simulation of contact and on experimental data,” IEEE Computer Graphics and Applica- tions, vol. 26, no. 6, pp. 58–68, 2006. cutting of heterogeneous soft-tissues,” Medical Image Analysis, vol. 18, no. 2, pp. 394–410, 2014. [20] G. Sela, J. Subag, A. Lindblad, D. Albocher, S. Schein, and [34] V. Strbac, J. Vander Sloten, and N. Famaey, “Analyzing the G. Elber, “Real-time haptic incision simulation using FEM- based discontinuous free-form deformation,” Computer-Aided potential of GPGPUs for real-time explicit ﬁnite element anal- ysis of soft tissue deformation using CUDA,” Finite Elements Design, vol. 39, no. 8, pp. 685–693, 2007. in Analysis and Design, vol. 105, pp. 79–89, 2015. [21] M. García, O. D. Robles, L. Pastor, and A. Rodríguez, “MSRS: a [35] A. Karami, M. Eghtesad, and S. A. Haghpanah, “Prediction of fast linear solver for the real-time simulation of deformable objects,” Computers & Graphics, vol. 32, no. 3, pp. 293–306, muscle activation for an eye movement with ﬁnite element modeling,” Computers in Biology and Medicine, vol. 89, pp. 368–378, 2017. [22] G. R. Joldes, A. Wittek, and K. Miller, “An eﬃcient hourglass control implementation for the uniform strain hexahedron [36] F. Martínez-Martínez, M. J. Rupérez-Moreno, M. Martínez- using the total Lagrangian formulation,” Communications in Sober et al., “A ﬁnite element-based machine learning Numerical Methods in Engineering, vol. 24, no. 11, pp. 1315– approach for modeling the mechanical behavior of the breast 1323, 2008. tissues under compression in real-time,” Computers in Biology and Medicine, vol. 90, pp. 116–124, 2017. [23] G. R. Joldes, A. Wittek, and K. Miller, “Real-time nonlinear ﬁnite element computations on GPU – application to neuro- [37] V. Luboz, M. Bailet, C. Boichon Grivot et al., “Personalized surgical simulation,” Computer Methods in Applied Mechanics modeling for real-time pressure ulcer prevention in sitting pos- and Engineering, vol. 199, no. 49-52, pp. 3305–3314, 2010. ture,” Journal of Tissue Viability, vol. 27, no. 1, pp. 54–58, 2018. Applied Bionics and Biomechanics 29 [53] F. Goulette and Z.-W. Chen, “Fast computation of soft tissue [38] L. P. Nedel and D. Thalmann, “Real time muscle deforma- tions using mass-spring systems,” in Proceedings. Computer deformations in real-time simulation with hyper-elastic mass Graphics International (Cat. No.98EX149), pp. 156–165, links,” Computer Methods in Applied Mechanics and Engineer- Hannover, Germany, 1998. ing, vol. 295, pp. 18–38, 2015. [39] T. Goto, W. S. Lee, and N. Magnenat-thalmann, “Facial feature [54] J. Zhang, Y. Zhong, J. Smith, and C. Gu, “A new ChainMail extraction for quick 3D face modeling,” Signal Processing: approach for real-time soft tissue simulation,” Bioengineered, Image Communication, vol. 17, no. 3, pp. 243–259, 2002. vol. 7, no. 4, pp. 246–252, 2016. [40] C. Bonamico, M. Costa, F. Lavagetto, and R. Pockaj, “Real- [55] A. Woodward, Y. H. Chan, R. Gong et al., “A low cost frame- time MPEG-4 facial animation with 3D scalable meshes,” work for real-time marker based 3-D human expression Signal Processing: Image Communication, vol. 17, no. 9, modeling,” Journal of Applied Research and Technology, pp. 743–757, 2002. vol. 15, no. 1, pp. 61–77, 2017. [41] O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rössl, and [56] J. Zhou, Z. Luo, C. Li, and M. Deng, “Real-time deformation of H.-P. Seidel, “Laplacian surface editing,” in Proceedings of the human soft tissues: A radial basis meshless 3D model based on 2004 Eurographics/ACM SIGGRAPH symposium on Geometry Marquardt's algorithm,” Computer Methods and Programs in processing - SGP '04, pp. 175–184, New York, NY, USA, 2004. Biomedicine, vol. 153, pp. 237–252, 2018. [42] N. P. Chandrasiri, T. Naemura, M. Ishizuka, H. Harashima, [57] S. Cotin, H. Delingette, and N. Ayache, “A hybrid elastic model and I. Barakonyi, “Internet communication using real-time for real-time cutting, deformations, and force feedback for sur- facial expression analysis and synthesis,” IEEE Multimedia, gery training and simulation,” The Visual Computer, vol. 16, vol. 11, no. 3, pp. 20–29, 2004. no. 8, pp. 437–452, 2000. [43] W. Mollemans, F. Schutyser, N. Nadjmi, and P. Suetens, “Very [58] G. Yarnitzky, Z. Yizhar, and A. Gefen, “Real-time subject- fast soft tissue predictions with mass tensor model for maxillo- speciﬁc monitoring of internal deformations and stresses in facial surgery planning systems,” International Congress Series, the soft tissues of the foot: a new approach in gait analysis,” vol. 1281, pp. 491–496, 2005. Journal of Biomechanics, vol. 39, no. 14, pp. 2673–2689, 2006. [44] P. Chen, K. E. Barner, and K. V. Steiner, “A displacement [59] B. Zhu and L. Gu, “A hybrid deformable model for real-time driven real-time deformable model for haptic surgery simu- surgical simulation,” Computerized Medical Imaging and lation,” in 2006 14th Symposium on Haptic Interfaces for Graphics, vol. 36, no. 5, pp. 356–365, 2012. Virtual Environment and Teleoperator Systems, pp. 499–505, [60] I. S. Pandzic and R. Forchheimer, MPEG-4 Facial Animation: Alexandria, VA, USA, 2006. The Standard, Implementation and Applications, John Wiley [45] M. López-Cano, J. Rodríguez-Navarro, A. Rodríguez-Baeza, & Sons, 2003. M. Armengol-Carrasco, and A. Susín, “A real-time dynamic [61] M. Doblaré, E. Cueto, B. Calvo, M. A. Martínez, J. M. Garcia, 3D model of the human inguinal region for surgical educa- and J. Cegoñino, “On the employ of meshless methods in bio- tion,” Computers in Biology and Medicine, vol. 37, no. 9, mechanics,” Computer Methods in Applied Mechanics and pp. 1321–1326, 2007. Engineering, vol. 194, no. 6-8, pp. 801–821, 2005. [46] Y. J. Lim and S. De, “Real time simulation of nonlinear tissue [62] G. Y. Zhang, A. Wittek, G. R. Joldes, X. Jin, and K. Miller, “A response in virtual surgery using the point collocation-based three-dimensional nonlinear meshfree algorithm for simulat- method of ﬁnite spheres,” Computer Methods in Applied ing mechanical responses of soft tissue,” Engineering Analysis Mechanics and Engineering, vol. 196, no. 31-32, pp. 3011– with Boundary Elements, vol. 42, pp. 60–66, 2014. 3024, 2007. [63] M. H. Doweidar, B. Calvo, I. Alfaro, P. Groenenboom, and [47] E. Basafa and F. Farahmand, “Real-time simulation of the non- M. Doblaré, “A comparison of implicit and explicit natural ele- linear visco-elastic deformations of soft tissues,” International ment methods in large strains problems: application to soft Journal of Computer Assisted Radiology and Surgery, vol. 6, biological tissues modeling,” Computer Methods in Applied no. 3, pp. 297–307, 2011. Mechanics and Engineering, vol. 199, no. 25-28, pp. 1691– [48] J. Wang, S. Liao, X. Zhu et al., “Real time 3D simulation for 1700, 2010. nose surgery and automatic individual prosthesis design,” [64] F. Roohbakhshan and R. A. Sauer, “Eﬃcient isogeometric thin Computer Methods and Programs in Biomedicine, vol. 104, shell formulations for soft biological materials,” Biomechanics no. 3, pp. 472–479, 2011. and Modeling in Mechanobiology, vol. 16, no. 5, pp. 1569– [49] X. Wan, S. Liu, J. X. Chen, and X. Jin, “Geodesic distance- 1597, 2017. based realistic facial animation using RBF interpolation,” [65] D. Kamensky, F. Xu, C. H. Lee, J. Yan, Y. Bazilevs, and M. C. Computing in Science & Engineering, vol. 14, no. 5, Hsu, “A contact formulation based on a volumetric potential: pp. 49–55, 2012. application to isogeometric simulations of atrioventricular [50] B. H. Le, M. Zhu, and Z. Deng, “Marker optimization for facial valves,” Computer Methods in Applied Mechanics and Engi- motion acquisition and deformation,” IEEE Transactions on neering, vol. 330, pp. 522–546, 2018. Visualization and Computer Graphics, vol. 19, no. 11, [66] T. S. Sørensen and J. Mosegaard, “An introduction to GPU pp. 1859–1871, 2013. accelerated surgical simulation,” in Biomedical Simulation, [51] Y. Zhang, W. Lin, B. Zhou, Z. Chen, B. Sheng, and J. Wu, M. Harders and G. Székely, Eds., pp. 93–104, Springer, Berlin, “Facial expression cloning with elastic and muscle models,” Heidelberg, 2006. Journal of Visual Communication and Image Representation, [67] T. T. Dao, A.-X. Fan, S. Dakpé, P. Pouletaut, M. Rachik, and vol. 25, no. 5, pp. 916–927, 2014. M. C. H. B. Tho, “Image-based skeletal muscle coordination: [52] Y. Weng, C. Cao, Q. Hou, and K. Zhou, “Real-time facial ani- case study on a subject speciﬁc facial mimic simulation,” Jour- mation on mobile devices,” Graphical Models, vol. 76, no. 3, nal of Mechanics in Medicine and Biology, vol. 18, no. 2, article pp. 172–179, 2014. 1850020, 2018. 30 Applied Bionics and Biomechanics [68] T. T. Dao and M.-C. H. B. Tho, “A systematic review of contin- uum modeling of skeletal muscles: current trends, limitations, and recommendations,” Applied Bionics and Biomechanics, vol. 2018, Article ID 7631818, 17 pages, 2018. [69] B. Wan, M. Shahmoradi, Z. Zhang et al., “Modelling of stress distribution and fracture in dental occlusal ﬁssures,” Scientiﬁc Reports, vol. 9, no. 1, p. 4682, 2019. [70] Y. Zhang, Y. Liu, Y. She, Y. Liang, F. Xu, and C. Fang, “The eﬀect of endodontic access cavities on fracture resistance of ﬁrst maxillary molar using the extended ﬁnite element method,” Journal of Endodontia, vol. 45, no. 3, pp. 316–321, [71] S. Niroomandi, I. Alfaro, D. González, E. Cueto, and F. Chinesta, “Real-time simulation of surgery by reduced- order modeling and X-FEM techniques,” International Journal for Numerical Methods in Biomedical Engineering, vol. 28, no. 5, pp. 574–588, 2012. [72] R. Avazmohammadi, D. S. Li, T. Leahy et al., “An integrated inverse model-experimental approach to determine soft tissue three-dimensional constitutive parameters: application to post-infarcted myocardium,” Biomechanics and Modeling in Mechanobiology, vol. 17, no. 1, pp. 31–53, 2018. [73] R. Avazmohammadi, J. S. Soares, D. S. Li, S. S. Raut, R. C. Gorman, and M. S. Sacks, “A contemporary look at biome- chanical models of myocardium,” Annual Review of Biomedi- cal Engineering, vol. 21, pp. 417–442, 2019.

Journal

Applied Bionics and Biomechanics – Hindawi Publishing Corporation

Published: Feb 20, 2020

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

References (76)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies