Access the full text.
Sign up today, get DeepDyve free for 14 days.
Context: Neural Network (NN) algorithms have been successfully adopted in a number of Safety-Critical Cyber- Physical Systems (SCCPSs). Testing and Veriﬁcation (T&V) of NN-based control software in safety-critical domains are gaining interest and attention from both software engineering and safety engineering researchers and practition- ers. Objective: With the increase in studies on the T&V of NN-based control software in safety-critical domains, it is important to systematically review the state-of-the-art T&V methodologies, to classify approaches and tools that are invented, and to identify challenges and gaps for future studies. Method: By searching the six most relevant digital libraries, we retrieved 950 papers on the T&V of NN-based Safety-Critical Control Software (SCCS). Then we ﬁltered the papers based on the predeﬁned inclusion and exclusion criteria and applied snowballing to identify new relevant papers. Results: To reach our result, we selected 83 primary papers published between 2011 and 2018, applied the thematic analysis approach for analyzing the data extracted from the selected papers, presented the classiﬁcation of approaches, and identiﬁed challenges. Conclusion: The approaches were categorized into ﬁve high-order themes, namely, assuring robustness of NNs, improving the failure resilience of NNs, measuring and ensuring test completeness, assuring safety properties of NN- based control software, and improving the interpretability of NNs. From the industry perspective, improving the interpretability of NNs is a crucial need in safety-critical applications. We also investigated nine safety integrity prop- erties within four major safety lifecycle phases to investigate the achievement level of T&V goals in IEC 61508-3. Results show that correctness, completeness, freedom from intrinsic faults, and fault tolerance have drawn most atten- tion from the research community. However, little eort has been invested in achieving repeatability, and no reviewed study focused on precisely deﬁned testing conﬁguration or defense against common cause failure. Keywords: Software testing and veriﬁcation, Neural network, Safety-critical control software, Systematic literature review 1. Introduction Cyber-Physical Systems (CPSs) are systems involving networks of embedded systems and strong human-machine interactions . Safety-critical CPSs (SCCPSs) are a type of CPSs that highlights the severe non-functional constraints (e.g., safety and dependability). The failure of SCCPSs could result in loss of life or signiﬁcant damage (e.g., property and environmental damage). Typical applications of SCCPSs are in nuclear systems, aircraft ﬂight control systems, automotive systems, smart grids, and healthcare systems. In the last few years, advances in Neural Networks (NNs) have boosted the development and deployment of SCCPSs. The NN is considered the most viable approach to meet the complexity requirements of Safety-Critical Corresponding author Email addresses: firstname.lastname@example.org (Jin Zhang), email@example.com (Jingyue Li) Accepted 6 March 2020 by Information and Software Technology arXiv:1910.06715v2 [cs.LG] 20 Mar 2020 Control Softwares (SCCSs) [2, 3]. In this study, we refer to NN-based SCCS as SCCS that heavily use NNs (e.g., to implement controller). For example, in the transportation industry, deep-learning-based NNs have been widely used to developing self-driving cars  and collision avoidance systems . It is also worth noting that several safety incidents caused by autonomous vehicles have been presented in media, e.g., Uber car’s fatal incident , Tesla’s fatal Autopilot crash , and Google’s self-driving car crash . In addition to the safety incidents caused by failures of the autonomous system, security breaches of autonomous vehicles can potentially lead to safety issues, e.g., a demo showed that autonomous vehicles can be remotely controlled and hijacked . How can we ensure that an SCCS containing NN technology will behave correctly and consistently when system failures or malicious attacks occur? Increasing interest in the migration of Industrial Control Systems (ICSs) towards SCCPSs has encouraged research in the area of safety analysis of SCCPSs. Kriaa et al.  surveyed existing approaches for an integrated safety and security analysis of ICSs. The approaches cover both the design stage and the operational stage of the system lifecycle. Some approaches (such as [11, 12]) are aimed at combining safety and security techniques into a single methodology. Others (such as [13, 14]) are trying to align safety and security techniques. These approaches are either generic, which consider both safety and security at a very high level, or model-based, which build upon the formal or semi-formal representation of the system’s functions. There are many studies that focus on the T&V of NNs in the past decade. Several review articles [15, 16, 17, 18] on this topic have been published. Studies [15, 19] have reviewed methods focusing on veriﬁcation and validation of NNs for aerospace systems. Studies [17, 18] are limited in automotive applications. None of these review articles have applied the Systematic Literature Review (SLR)  approach. Recently there has been more concern about Artiﬁcial Intelligence (AI) safety. The state-of-the-art advancements in the T&V of NN-based SCCS are increasingly important; hence, there is a need to have a thorough understanding of present studies to incentivize further discussion. This study aimed to summarize the current research on T&V methods for NN-based control software in SCCPSs. We have systematically identiﬁed and reviewed 83 papers focusing on the T&V of NN-based SCCSs and synthesized the data extracted from those papers to answer three research questions. RQ1 What are the proﬁles of the studies focusing on testing and verifying NN-based SCCSs? RQ2 What approaches and associated tools have been proposed to test and verify NN-based SCCSs? RQ3 What are the limitations of current studies with respect to testing and verifying NN-based SCCSs? To our best knowledge, our study is the ﬁrst SLR on testing and verifying NN-based control software in SCCPSs. The results of these research questions can help researchers identify the research gaps in this area, and help industrial practitioners choose proper veriﬁcation and certiﬁcation methods. The main contributions of this work are: We made a classiﬁcation of T&V approaches in both academia and industry for NN-based SCCSs. We identiﬁed and proposed challenges for advancing state-of-the-art T&V for NN-based SCCSs. The remainder of this paper is organized as follows: In section 2, we deﬁne terminologies related to NN-based SCCPSs and summarize related work from academia and industry. Section 3 describes the SLR process and the review protocol. The results of the research questions are reported in Section 4. Section 5 discusses the industry practice of T&V of NN-based SCCSs, and the threats to validity of our study. Section 6 concludes the study. 2. Background In this section, we ﬁrst introduce terminology related to CPSs and modern NNs and show how NN algorithms have been used in SCCPSs. Then, we present the current state of practice of T&V of SCCSs. 2.1. Cyber-physical systems As deﬁned in , “cyber-physical systems (CPSs) are physical and engineered systems whose operations are monitored, coordinated, controlled and integrated by a computing and communication core.” Several other systems, such as Internet of Things (IoTs) and ICSs have very similar features compared to CPSs, since they are all systems 2 used to monitor and control the physical world with embedded sensor and actuator networks. In general, CPSs are perceived as the new generation of embedded control systems, which can involve IoTs and ICSs [21, 22]. In this SLR, we adopted the CPS conceptual model in  as a high-level abstraction of CPSs to describe the dierent perspectives of CPSs and the potential interactions of devices and systems in a system of systems (SoS) as shown in Fig. 1. From the perspective of unit level, a CPS at least includes one or several controllers, many actuators, and sensors. A CPS can also be a system consisting of one or more cyber-physical devices. From the SoS perspective, a CPS is composed of multiple systems that include multiple devices. In general, a CPS must contain the decision ﬂow (from controller to actuators), information ﬂow (from sensors to controller), and action ﬂow (actuators impacting the physical state of the physical world). System of Systems level System level Unit level (cyber-physical device) Human Controller Decision Information Actuator(s) Sensor(s) Action Physical state Figure 1: CPS conceptual model In the context of SCCPS, safety and performance are dependent on the system (to be more speciﬁc, the controller of the system) making the right decision according to the measurement of the sensors, and operating the actuators to take the right action at the right time. Thus, veriﬁcation of the process of decision-making is vital for a SCCPS. 2.2. Modern neural networks The concept of neural network was ﬁrst proposed in 1943 by Warren McCullough and Walter Pitts , and Frank Rosenblatt in 1957 designed the ﬁrst trainable neural network called the Perceptron”. A perceptron is a simple binary classiﬁcation algorithm with only one layer and output decision of 0 or 1. By the 1980s, neural nets with more than one layer were proposed to solve more complex problems, i.e., multilayer perceptron (MLP). In this SLR, we regard multilayer NNs that emerged after the 1980s as modern NNs. Artiﬁcial Neural Network (ANN) is the general name of computing systems designed to mimic how the human brain processes information . An ANN is composed of a collection of interconnected computation nodes (namely artiﬁcial neurons), which are organized in layers. Depending on the directions of the signal ﬂow, an ANN can have feed-forward or feedback architectures. Fig. 2 shows a simpliﬁed feed-forward ANN architecture with multiple hidden layers. Each artiﬁcial neuron has weighted inputs, an activation function, and one output. The weights of the interconnections are adjusted based on the learning rules. There are three main models of learning rules, which are unsupervised learning, supervised learning, and reinforcement learning. The choice of learning rules corresponds to the particular learning task. The common activation functions contain sigmoid, hyperbolic tangent, radial bases function (RBF), and piece-wise linear transfer function, such as Rectiﬁed Linear Unit (ReLU) . In a word, an 3 ANN can be deﬁned by three factors: the interconnection structure between dierent layers, activation function type, and procedure for updating the weights. Input Hidden Output layer layer(s) layer …….. …….. …….. Figure 2: A simpliﬁed feed-forward ANN architecture Multi-Layer Perceptron (MLP ) represents a class of feed-forward ANN. An MLP consists of an input layer, one or several hidden layers, and an output layer. Each neuron of MLP in one layer is fully connected with every node in the following layer. An MLP employs a back-propagation technique (which belongs to supervised learning) for training. Convolutional Neural Network (CNN ) is a special type of multi-layer NN with one or more convolutional layers. A convolutional layer includes “several feature maps with dierent weight vectors. A sequential implementa- tion of a feature map would scan the input image with a single unit that has a local receptive ﬁeld, and store the states of this unit at corresponding locations in the feature map. This operation is equivalent to a convolution, followed by an additive bias and squashing function, hence the name convolutional network”. CNNs are superior for pro- cessing two-dimensional data (particular camera images) because of the convolution operations, which are capable of detecting features in images. CNNs are now widely applied to develop partially-autonomous and fully-autonomous vehicles. Deep Neural Networks (DNNs ) represent an ANN with multiple hidden layers between the input and output layers. DNNs (e.g., a MLP with more than three layers or a CNN) dier from shallow NNs (e.g., a three-layer MLP) in the number of layers, the activation functions that can be employed, and the arrangement of the hidden layer. Compared to shallow NNs, DNNs can be trained more in-depth to ﬁnd patterns with high performance even for complex nonlinear relationships. An NN could be trained oine or online. An NN trained oine means it only learns during development. After training, the weights of the NN will be ﬁxed and the NN will act deterministically. Therefore static veriﬁcation meth- ods could be possible. In contrast, online training will allow the NN to keep learning and evolving during operation, which requires run-time veriﬁcation methods. In some applications, such as the Intelligent Flight Control System developed by NASA , both oine and online training strategies are employed to meet the system requirements. NNs are fundamentally dierent with algorithmic programs, but a formal development methodology can still be derived for an NN system. Development process of an NN system can include six phases : 1. Formulation of requirements and goals; 2. Selection of training and test data sets; 3. Selection of the NN architecture; 4. Training of the network; 5. Testing of the network; and 6. Acceptance and use by the customer. 4 Like , Falcini et al. introduced a similar development lifecycle for DNNs in automotive software  and proposed a W-model integrated data development with standard software development to highlight the importance of data-driven in DNN development. Falcini et al.  also summarized that the DNN’s functional behavior depends on both its architecture and its learning outcome through training. 2.3. The trends of using NN algorithm in SCCPSs From 1940s automated range ﬁnders (developed by Norbert Wiener for anti-aircraft guns)  to today’s self- driving cars, AI, especially NN algorithms, is widely applied in both civilian (e.g., autonomous cars) and military domains (e.g., military drones). Boosted by the advances of AI, state-of-the-art CPSs can plan and execute more and more complex operations with less human interaction. Here we present the applications of NNs in the following four representative SCCPSs. 2.3.1. Autonomous cars For automobile, the Society of Automotive Engineers (SAE) proposed six levels of autonomous driving . A level 0 vehicle has no autonomous capabilities, and the human driver is responsible for all aspects of the driving task. For level 5 vehicle, the driving tasks are only managed by the autonomous driving system. When developing autonomous vehicles targeting a high level of autonomy, one industry trend is to use DNNs to implement vehicle control algorithms. The deep-learning-based approach enables vehicles to learn meaningful road features from raw input data automatically and then output driving actions. The so-called end-to-end learning approach can be applied to resolve complex real-world driving tasks. When using deep-learning-based approaches, the ﬁrst step is to use a large number of training data sets (images or other sensor data) to train a DNN. Then a simulator is used to evaluate the performance of the trained network. After that, the DNN-based autonomous vehicle will be able to “execute recognition, prediction, and planning” driving tasks in diverse conditions . Nowadays, CNNs are the most widely adopted deep-learning model for fully autonomous vehicles [5, 6, 7, 8]. NVIDIA introduced an AI supercomputer for autonomy . The development ﬂow using NVIDIA DRIVE PX includes four stages: 1) data acquisition to train the DNN, 2) deployment of the output of a DNN in a car, 3) autonomous application development, and 4) testing in-vehicle or with simulation. One essential characteristic of deep-learning-based autonomy is that the decision-making part of the vehicle is almost a black box. This means that in most cases, we as human drivers must trust the decisions made by the deep- learning algorithms without knowing exactly why and how the decisions are made. 2.3.2. Industrial control systems Industrial Control System (ICS) is the general term for control systems, also called Supervisory Control and Data Acquisition (SCADA) systems. ICSs make decisions based on the speciﬁc control law (such as lookup table and non-linear mathematical model) formulated by human designers. In contrast to the classical design procedure of control law, reinforcement-learning-based approaches learn the control law simply from the interaction between the controller and the process, and then incrementally improving control behavior. Such approaches and NNs have been used in process control two decades ago . Concerning the recent progress in AI and the success of DNNs in making complex decisions, there are high expectations for the application of DNNs in ICSs. For instance, DNNs and reinforcement learning can be combined to develop continuous control . Spielberg et al. extended the work in  to design control policy for process control . Even though the proposed approach in  is only tested on linear systems, it shows a practical solution for applying DNNs in non-linear ICSs. 2.3.3. Smart grid systems The smart grid is designed as the next generation of electric power system, dependent on information communi- cations technology (ICT). There is tremendous initiative of research activities in automated smart grid applications, such as FLISR (which is a smart grid multi-agent automation architecture based on decentralized intelligent decision- making nodes) . NNs have been considered for solving many pattern recognition and optimization problems, such as fault diagnosis , and control and estimation of ﬂux, speed , and economical electricity distribution to consumers. MLP is one of the most commonly used topology in power electronics and motor drives . 5 2.3.4. Healthcare Medical devices is another emerging area where research and industry practitioners are seeking to integrate AI technologies to improve accuracy and automation. ANNs and other machine learning approaches have been proposed to improve the control algorithms for diabetes treatment in recent decades [41, 42]. In 2017, an AI-powered device for automated and continuous delivery of basal insulin (named MiniMed 670G system ) was approved by the U.S. Food and Drug Administration. In the same year, it was reported that GE Healthcare had integrated the NVIDIA AI platform into their computerized tomography scanner to improve speed and accuracy for the detection of liver and kidney lesions . Using deep learning solutions, such as CNNs, in the medical computing ﬁeld has proven to be eective since CNNs have excellent performance in object recognition and localization in medical images . 2.4. Testing and veriﬁcation of safety-critical control software IEC 61508 and ISO 26262 are two standards highly relevant to the T&V of SCCS. IEC 61508 is an international standard concerning Functional safety of electrical/electronic/programmable electronic safety-related systems. It de- ﬁnes four safety integrity levels (SILs) for safety-critical systems . The higher the SIL level a SCCPS requires, the more time and eort for veriﬁcation are needed. In IEC 61508, formal methods are highly recommended tech- niques for verifying high SIL systems. Because formal methods can be used to construct the speciﬁcation and provide a mathematical proof that the system matches some formal requirements, this is quite a strong commitment for the correctness of a system. ISO 26262, titled Road vehicles functional safety, is an international standard for the functional safety of electrical and/or electronic systems in production automobiles . Besides using classical safety analysis methods such as Fault Tree Analysis (FTA) and Failure Mode and Eects Analysis (FMEA), ISO 26262 explicitly states that the production of a safety case is mandated to assure system safety. It deﬁnes a safety case as an argument that the safety requirements for an item are complete and satisﬁed by evidence compiled from work products of the safety activities during development . The development of suitable approaches, which can verify the system behavior and misbehavior of a SCCPS is always challenging. Not to mention that the architecture of NNs (especially DNNs) makes it even harder to decipher how the algorithmic decisions were made. The current version of IEC 61508 is not applicable for the veriﬁcation of NN-based SCCSs because AI technologies are not recommended there. The latest version of ISO 26262 and its extension, ISO/PAS 21448, which is also known as safety of the intended functionality (SOTIF) , will likely provide a way to handle the development of autonomous vehicles. However, SOTIF will only provide guidelines associated with SAE Level 0–2 autonomous vehicles , which are not ready for the veriﬁcation of NN-based autonomous vehicles. In practice, in order to reduce test and validation costs, high-ﬁdelity simulation is a commonly used approach in the automotive domain. The purpose of using a simulator is to predict the behavior of an autonomous car in a mimicked environment. NVIDIA and Apollo distributed their high-ﬁdelity simulation platforms for testing autonomous vehicles. CARLA  and Udacity’s Self-Driving Car Simulator  are two popular open-source simulators for autonomous driving research and testing. 3. Research method We conducted our SLR by following the SLR guidelines in  as well as consulting other relevant guidelines in  and [53, 54]. Our review protocol consisted of four parts: 1) search strategy, 2) inclusion and exclusion criteria, 3) selection process, and 4) data extraction and synthesis. 3.1. Search strategy Based on guidelines provided in , we use the Population, Intervention, Outcome, Context (PIOC) criteria to formulate search terms. In this SLR, The population should be an application area (e.g., general CPS) or speciﬁc CPS (e.g., self-driving car). The intervention is methodology, tools and technology that address system/component testing or veriﬁcation. 6 The outcome is the improved safety or functional safety of CPSs. The context is the NN-based SCCPSs in which the T&V take place. Fig. 3 shows the search terms formulated based on the PIOC criteria. We ﬁrst used these search terms to run a series of trial searches and verify the relevance of the resulting papers. We then revised the search string to form the ﬁnal search terms. The ﬁnal search terms were composed of synonyms and related terms. Population: “Cyber-physical system*” or “Cyber physical TITLE-ABS-KEY(("Cyber-physical system*" or "Cyber-physical system*" or CPS* or "Smart grid" or "Smart car" or "Automotive system*” or CPS* or “Smart grid” or “Smart car” or “Automotive cyber-physical system*" or "Self-driving car*" or "Autonomous cyber-physical system*” or “Self-driving car*” or “Autonomous vehicle*" or "Autonomous driving system*" or "Automotive vehicle*” or “Autonomous driving system*” or “Automotive electronic control system*" or "Automotive embedded system*" or electronic control system*” or “Automotive embedded system*” "Unmanned Aerial Vehicles" or "aircraft collision avoidance Intervention: “Risk assessment” or “verification” or “test” or system*")AND(“Risk assessment” or “verification” or “test” or ”testing” or “analysis” or “Certification” or “assurance” ” t e s t i n g ” o r “ a n a l y s i s ” o r “ C e r t i f i c a t i o n ” o r Outcome: “Safety” or “Functional safety” “assurance”) AND ( “ S a f e t y ” o r “ F u n c t i o n a l safety”)AND(“Autonomous decision” or “Autonomous agent*” or Context: “Deep learning” or “Deep neural networks” or “Deep learning” or “Deep neural networks”) “Autonomous decision” or “Autonomous agent” Figure 3: Search terms We executed automated searches in six digital libraries, namely, Scopus, IEEE Xplore, Compendex EI, ACM Digital library, SpringerLink, and Web of Science (ISI). 3.2. Inclusion and exclusion criteria Table 1 presents our inclusion and exclusion criteria. We set three inclusion criteria to restrict the application domain, context, and outcome type. We excluded papers that were not peer-reviewed, such as keynotes, books, and dissertations, and papers not written in English. It should be clariﬁed that, unlike most other SLR studies, we did not directly exclude short papers (less than six pages), work-in-progress papers, and pre-print papers. The reason is that this research area is far from mature, so many initial thoughts or in-progress papers are still valuable to review. Table 1: Inclusion and Exclusion criteria Inclusion criteria I1 The paper must have a context in SCCPSs, either in general or in a speciﬁc application domain I2 The paper must be aimed at testing/veriﬁcation approaches for NN-based SCCSs I3 The paper must be aimed at modern neural networks Exclusion criteria E1 Papers not peer-reviewed E2 Not written in English E3 Full-text is not available E4 Not relevant to modern neural networks 3.3. Selection process We used the inclusion and exclusion criteria to ﬁlter the papers in the following steps. We covered papers from January 2011 to November 2018. Fig. 4 shows the whole search and ﬁltering process. Stage 1: Ran the search string on the six digital libraries and retrieved 1046 papers. After removing those dupli- cated papers, we had 950 papers. Stage 2: Excluded studies by reading title and keywords. If it was not excluded simply by reading titles and keywords, the paper was kept for further investigation. At the end of this stage, we selected 254 papers. Stage 3: Further ﬁltered the papers by reading abstracts and found 105 potential papers with high relevance to the research goal of our SLR. 7 Stage 1 Data Sources: Scopus, IEEE Xplore, Compendex Results=950 EI, ACM Digital library, SpringerLink, and Web of Science(ISI) Stage 2 Results=254 Apply inclusion and exclusion criteria by reading title and keywords Stage 3 Results=105 Apply inclusion and exclusion criteria by reading abstract Stage 4 Results=27 Apply inclusion and exclusion criteria by reading introduction and conclusion Stage 5 Results=83 Snowballing: Read full paper got in 4th stage and scan the reference Figure 4: Search process Stage 4: Read the introduction and conclusion to decide on selection. We recorded the reasons for exclusion for each excluded paper. We excluded the papers that were irrelevant, or whose full texts were not available. Furthermore, we critically examined the quality of primary studies to exclude those that lacked sucient information. We ended up with 27 papers. Stage 5: Read full text of the selected studies from the fourth stage, applied snowballing by scanning the reference of the selected papers. The snowballing process can be implemented in two directions: backwards (which means scanning the references of a selected paper and ﬁnd any other relevant papers published before the selected paper), and forwards (which means checking if any other relevant paper was published after the selected paper and cited the selected paper). In our SLR, we adapted mainly backward snowballing to include additional papers. To limit the scope of the snowballing, we covered only references published between 2011 and 2018. From snowballing, we found 56 new relevant papers. Finally, we selected 83 papers as primary studies for detailed analysis. We listed all of the selected studies in Appendix A. The ﬁrst author conducted the selection process with face-to-face discussions with the second author. The second author performed a cross-check of each step and read all the ﬁnal selected papers to conﬁrm the selection of the papers. 8 3.4. Data extraction and synthesis Data Extraction: We extracted two kinds of information from the selected papers. To answer RQ1, we extracted information for statistical analysis, e.g., publication year and research type. To answer RQ2 and RQ3, we collected information to identify key features (such as research goal, technique and tools, major contribution and limitation) of T&V approaches. Synthesis: We used descriptive statistics to analyze the data for answering RQ1. To answer RQ2 and RQ3, we analyzed the data using the qualitative analysis method by following the ﬁve steps of thematic analysis : 1) extracting data, 2) coding data, 3) translating codes into themes, 4) creating a model of higher-order themes, and 5) assessing the trustworthiness of the synthesis. 4. Result 4.1. RQ1. What are the proﬁles of the studies focusing on testing and verifying NN-based SCCSs? Studies distributions: Fig. 5 shows the distribution of selected papers based on publication year and the types of work. There has been 68 papers (81.9%) published since 2016, indicating that researchers are paying more attention to the T&V of NN-based SCCSs. Conference was the most popular publication type with 48 papers (57.8%), followed by pre-print (25 papers, 30.1%), workshop (6 papers, 7.2%), and journal (4 papers, 4.8%). 2011 2012 2013 2014 2015 2016 2017 2018 Pre-print 0 0 2 0 3 0 13 7 Wo rk sh op 0 0 0 1 1 0 3 1 Journal 0 1 0 0 1 0 1 1 Conference 1 0 0 2 3 12 20 10 Conference Journal Wo rk sh op Pre-print Figure 5: Publication year and types of selected papers We also investigated the geographic distribution of the reviewed studies. It allowed us to identify which countries are leading the research in this domain. We considered a study to be conducted in one country if the aliation of at least one author is in that country. Moreover, the involvement of industry would be an indicator of industry’s interest in this domain. We classiﬁed the reviewed papers as industry if at least one author came from industry or the study used real-world industrial systems to test/verify the proposed approach. A paper would be categorized as academia if all authors came from academia. It shows that researchers based in the USA have been involved in the most primary studies for testing or veriﬁcation of NN-based SCCSs with 56 publications, followed by the researchers based in Germany and the UK with 10 and 9 publications, respectively. It is worth noting that 47 of 83 (56.6%) publications have involvement from industry. Research types: We classiﬁed the selected papers based on the criteria proposed by Kai et al.  (See Table 2). According to Table 2, the research type of the paper is governed by rules (i.e., R1-R6). Each rule is a combination of Number of studies several conditions. The six research types (i.e., evaluation research, solution proposal, validation research, philosoph- ical papers, opinion papers, and experience papers) correspond to R1-R6, respectively. For example, both evaluation research (corresponding to R1) and validation research (corresponding to R4) must present empirical evaluation. The dierence between evaluation and validation research is that validation is not used in practice (e.g., experimental or simulation-based approaches), whereas evaluation studies should be conducted in a real-world context. Solution pro- posal means that it has to propose a new solution that may or may not be used in practice. We found that evaluation and validation research are the majority of the selected papers, corresponding to 31.3% (26 papers) and 61.4% (51 papers) of the selected papers, respectively. The low percentage of the solution proposal (6 papers) was not surprising because a majority of the reviewed papers presented and demonstrated their T&V approaches through academic and industrial case studies, simulation, and controlled experiments. The other three types of research papers (i.e., philo- sophical papers, opinion papers, and experience papers) do not exist in selected studies because we only included papers that aimed at testing/veriﬁcation approaches (refer to inclusion criteria I2). Table 2: Research type classiﬁcation (T = True, F = False, l = irrelevant or not applicable, R1R6 refer to rules). R1 R2 R3 R4 R5 R6 Conditions Used in practice T l T F F F Novel solution l T F l F F Empirical evaluation T F F T F F Conceptual framework l l l l T F Opinion about something F F F F F T Authors’ experience l l T l F F Decisions Evaluation research 3 l l l l l Solution proposal l 3 l l l l Validation research l l l 3 l l Philosophical papers l l l l 3 l Opinion papers l l l l l 3 Experience papers l l 3 l l l Note: Reprinted from ,Copyright 2015 by the Elsevier. Application domains: We analyzed the application domain of selected studies to provide useful information for researchers and practitioners who are interested in the domain-speciﬁc aspects of the approaches. The results are shown in Table 3. We found that considerable eort is now being put into using NN algorithms to accomplish control logic for general purpose (59 papers, 71.1%), automotive CPSs, such as autonomous vehicles (13 papers, 15.7), and autonomous aerial systems, such as airborne collision avoidance systems for unmanned aircrafts (5 papers, 6%). Table 3: Distribution of application domains of the selected studies Application domain No. of studies General SCCPSs 59 Automotive CPSs 13 Autonomous aerial systems 5 Robot system 5 Health care 1 4.2. RQ2. What approaches and associated tools have been proposed to test and verify NN-based SCCSs? As 4 of the 83 papers focused mainly on high-level ideas and concepts without presenting detailed approaches or tools, we did not include them to answer RQ2. For the remaining 79 out of 83 (95.2%) papers, we applied the thematic 10 analysis approach  and identiﬁed ﬁve high-order themes and some sub-themes. Some papers contain more than one themes. In order to balance the accuracy and the simplicity of categorization, we decided to assign each study only one category based on its major contribution. Table 4 presents the themes, sub-themes, and the corresponding papers. Fig. 6 compares the interests dierence of academia and industry for the ﬁve identiﬁed themes. Table 4: A classiﬁcation of approaches to test and verify NN-based SCCSs Themes Sub-themes Papers # Understanding the characteristics and impacts of adver- ,,,,, ,  17 sarial examples Assuring robustness of NNs Detect adversarial examples ,, , , ,  Mitigate impact of adversarial examples ,  Improving robustness of NNs through using adversarial ,  examples Improving failure resilience of NNs ,,,,,,,,,, 11 Measuring and ensuring test com- ,,,,,, 7 pleteness Assuring safety properties of NN- ,,,,,,,,, 13 based CPSs ,,, ,,,,,,,, Understand how a speciﬁc decision is made ,,,,,,, , Improving interpretability ,, of NNs Facilitate understanding of the internal logic of NNs ,,,,, 31 Visualizing internal layers of NNs to help identify errors ,,,,, in NNs Assuring robustness of NN Improving failure resilience of NN Me asuring and ens uring t est completeness Assuring safety properties of NN-based SCCPS Improving interpretability of NN 0 5 10 15 20 25 30 35 ACADEMIA INDUSTRY Figure 6: Comparing the interests dierence of academia and industry 4.2.1. CA1: Assuring robustness of NNs One high-order theme of the studies is to assure the robustness of NNs. Robustness of an NN is its ability to cope with erroneous inputs. The erroneous inputs can be an adversarial example (i.e., an input that adds small perturbation intentionally to mislead classiﬁcation of an NN), or benign but wrong input data. Methods under this theme can be further classiﬁed into four sub-themes. Studies focusing on understanding the characteristics and impacts of adversarial examples. Some studies tried to identify the characteristics and impacts of adversarial examples. The study  found the characteristics, such as the linear nature, of adversarial examples. The study  measured the impact of adversarial examples by counting 11 their frequencies and severities. Nguyen et al.  found that a CNN trained on ImageNet  is vulnerable to adversarial examples generated through Evolutionary Algorithms (EAs) or gradient ascent. A few other studies, such as [58, 60, 61, 62], tried to understand the characteristics of robust NNs. Cisse et al.  introduced a particular form of DNN, namely Parseval Networks, that is intrinsically robust to adversarial noise. Gu et al.  concluded that some training strategies, for example, training using adversarial examples or imposing contractive penalty layer by layer, are robust to certain structures of adversarial examples (e.g., inputs corrupted by Gaussian additive noises or blurring). Higher-conﬁdence adversarial examples (i.e., adversarial instances that are extremely easy to classify into the wrong category) were used to evaluate the robustness of the state-of-the-art NN in  and the robot-vision system in . Studies focusing on methods to detect adversarial examples. Detecting adversarial examples that are already inserted into training or testing data set are the primary targets of [63, 65, 66, 67, 68]. Wicker et al.  and  formulated the adversarial examples detection as a two-player stochastic game and used the Monte Carlo Tree Search to identify adversarial examples. Reuben  applied density estimates, and Bayesian uncertainty estimates to detect adversarial samples. Xu et al.  proposed a feature squeezing framework to detect adversarial examples, which are generated by seven state-of-the-art methods. According to , an advantage of feature squeezing is that it did not change the underlying model. Therefore, it can easily be integrated with other defenses methods. Metzen et al.  embedded DNNs with a subnetwork (called “detector”) to detect adversarial perturbations. The Deepsafe presented in  used clustering technology to ﬁnd candidate-safe regions ﬁrst and then veriﬁed whether the candidates were safe using counter-examples as a proof. Studies focusing on methods to mitigate impact of adversarial examples. Papemot et al.  adopted defen- sive distillation as a defense strategy to train DNN-based classiﬁers against adversarial examples. However, several powerful attacks have been proposed to defeat defensive distillation and have demonstrated that defensive distilla- tion does not actually eliminate adversarial examples . Papemot et al.  revisited defensive distillation and proposed a more eective way to defend against three recently discovered attack strategies, i.e., the Fast Gradient Method (FGM) , the Jacobian Saliency Map Approach (JSMA) , and the AdaDelta optimization strategy (AdaDelta) . Studies focusing on increasing robustness of NNs through using adversarial examples. In studies  and , the authors proposed methods to leverage adversarial training (e.g., generating a large amount of adversarial examples and then training the NN not to be fooled by these adversarial examples) to increase the robustness of NNs. 4.2.2. CA2: Improving failure resilience of NNs Studies under this theme focused on improving the resilience of NNs, so that the NN-based CPSs are more tolerant of possible hardware and software failures. Studies [75, 77, 78] investigated error detection and mitigation mechanisms, while studies [76, 80] focused on understanding error propagation in DNN accelerators. Vialatte et al. demonstrated that faulty computations can be addressed by increasing the size of NNs. Santos et al.  proposed an algorithm-based fault tolerance (ABFT) strategy to detect and correct radiation-induced errors. In , a binary classiﬁcation algorithm based on temporal and stereo inconsistencies was applied to identify errors caused by single frame object detectors. Li et al.  developed a general-purpose GPU (GPGPU) fault injection tool  to investigate error propagation patterns in twelve GPGPU applications. Later, Li et al. revealed that the error resilience of DNN accelerators depends on “the data types, values, data reuse, and the types of layers in the design ”. Based on this ﬁnding, they devised guidelines for designing resilient DNN systems and proposed two DNN protection techniques, namely Symptom-based Error Detectors (SED) and Selective Latch Hardening (SLH) to mitigate soft errors that are typically caused by high-energy particles in hardware systems . Mhamdi et al. explored error propagation mechanism in an NN , and they theoretically and empirically proved that the key parameters that can be used to estimate the robustness of an NN are: “Lipschitz coecient of the activation function, distribution of large synaptic weights, and depth of the network”. The study  characterized the faults propagation through an open-source autonomous vehicle control software (i.e., openpilot) to assess the failure resilience of the system. The Systems-Theoretic Process Analysis (STPA)  hazard analysis technique was used to guide fault injection. Existing work in  showed that STPA is suited for an in-depth identiﬁcation of unsafe scenarios, and thus, the fault injection space was reduced. 12 Based on the diversiﬁed redundancy strategies, the study  developed diverse networks in the context of dif- ferent training data sets, dierent network parameters, and dierent classiﬁcation mechanisms to strengthen the fault tolerance of the DNN architecture. Studies [73, 74] tried to improve computation eciency without compromising error resilience. Studies [73, 74] also predicted the error resilience of DNN accelerators to make reconﬁgurable NN accelerators. The study  demonstrated a more accurate neuron resilience assignment than the state-of-the-art techniques and provided the possibility of moving parts of the neuron computations to unreliable hardware at the given quality constraint. Zhang et al.  proposed a framework to increase eciency of computation by approximating the computation of certain less critical neurons. Daftry et al.  provided an interesting idea about “how to enable a robot to know when it does not know?” The idea of  is to utilize the resulting features of the controller, which are learned from a CNN to predict the failure of the controller, and then let the system self-evaluate and decide whether to execute or discard an action. 4.2.3. CA3: Measuring and ensuring test completeness The approaches and tools under this theme aim to ensure good coverage when testing NNs. The testing approaches include black-box testing (i.e., focusing on whether the tests cover all possible usage scenarios), white-box testing (i.e., focusing on whether the tests cover every neuron in the NN), and metamorphic testing, which focuses on both test case generation and result veriﬁcation . O’Kelly et al. proposed methods to ensure good usage coverage through ﬁrst making a formal Scenario De- scription Language (SDL) to create driving scenarios, and then translating the scenarios to a speciﬁcation-guided automatic test generation tool named S-TALIRO to generate and run the tests. Raj et al.  proved the possibil- ity of speeding up the generation of new and interesting counterexamples by introducing fuzzing patterns obtained from an unrelated DNN on a dierent image database, although the proposed method provides no guarantee of test completeness. DeepXplore  ﬁrst introduced neuron coverage as a testing metric for DNNs, and then used multiple dierent DNNs with similar functionality to identify erroneous corner cases. Compared to , DeepTest  and DLFuzz  aimed at maximizing the neuron coverage without requiring multiple DNNs. The study  employed meta- morphic relations to identify erroneous behaviors. The study  proposed a dierential fuzzing testing framework to generate adversarial inputs. However, methods proposed in [85, 86, 90] cannot guarantee the generation of test cases that can precisely reﬂect real-world cases (e.g., driving scenes in various weather conditions when taking a DNN-based autonomous driving system). DeepRoad  employed Generative Adversarial Network (GAN) based techniques and metamorphic testing to synthesize diverse real driving scenes, and to test inconsistent behaviors in DNN-based autonomous driving systems. In contrast to earlier works, DeepGauge  argued that the testing criteria for traditional software are no longer applicable for DNNs. Ma et al.  proposed neuron-level and layer-level coverage criteria for testing DNNs and for measuring the testing quality. 4.2.4. CA4: Assuring safety property of NN-based SCCPSs Formal veriﬁcation can provide a mathematical proof that a system satisﬁes some desired safety properties (e.g., the system should always stay within some allowed region, namely a safe region). Formal veriﬁcation usually presents NNs as models and then apply a model checker, such as Boolean satisﬁability (SAT) solvers (e.g., Cha , SATO , GRASP ) to verify the safety property. Pulina et al.  developed NeVer (Neural networks Veriﬁer), which solves Boolean combinations of linear arithmetic constraints, to verify safe regions of MLPs. Through adopting an abstraction-reﬁnement mechanism, NeVer can verify real-world MLPs automatically. As an extended experiment analysis of results of ,  compared the performance (e.g., competition-style and scalability) of state-of-the-art Satisﬁability Modulo Theories (SMT) solvers , and demonstrated that scalability and ﬁne-grained abstractions remain challenges for realistic size networks. The studies  and  veriﬁed the “feed-forward NNs with piece- wise linear activation functions” by encoding veriﬁcation problems into solving a linear approximation exploring network behavior in a SMT solver. The next generation of collision avoidance systems for unmanned aircrafts (ACAS Xu) adopted DNNs to compress large score table . Julian et al.  explored the performance of ACAS Xu by measuring a set of safety and performance metrics. A simulation in study  shows that the system based on DNNs performed as correctly as the original large score table but with better performance. Reluplex  had successfully been used to verify the safety 13 property of a DNN for the prototype of ACAS Xu. Although the outcomes of Reluplex  are limited to verifying the correctness of NNs with speciﬁc type of activation functions (i.e., ReLUs and max-pooling layers), the study sheds a light on which types of NN architectures are easier to verify, and thus paves the way for verifying real-world DNN-based controllers. The method proposed in studies  and  veriﬁed that Binarized Neural Networks (BNNs) are ecient and scalable to moderate-sized BNNs. Study  represented BNNs as boolean formulas, and then veriﬁed the robustness of BNNs against adversarial perturbations. In study , BNNs and their input-output speciﬁcations were transferred into equivalence hardware circuits. The equivalence hardware circuits consist of a BNN structure module and a BNN property module. The authors of  then applied a SAT solver to verify the properties (e.g., simultaneously classify an image as a priority road sign and as a stop sign with high conﬁdence) of the BNN in order to identify the risk behavior of the BNN. When verifying a SCCS, one of the fundamental concerns is to make sure that the SCCS will never violate a safety property. An example of a safety property is that the system should never reach an unsafe region. The main ideas of studies under this sub-theme are to calculate the output reachable set of MLPs, such as in studies  and , or DNNs in study , to verify if unsafe regions will be reached. Xiang et al.  proposed a layer-by-layer approach to compute the output reachable set assisted by polyhedron computation tools. The safety veriﬁcation of a ReLU MLP is turned into checking if a non-empty intersection exists between the output reachable set and the unsafe regions. In a later work of Xiang et al. , they introduced maximum sensitivity to perform a simulation-based reachable set estimation with few restrictions on the activation functions. By combining local search and linear programming problems, Dutta et al.  developed an output bound searching approach for DNNs with ReLU activation functions, which is implemented in a tool called SHERLOCK to check whether the unsafe region is reached. Study  focused on the safety veriﬁcation of image classiﬁcation decisions. In , Huang et al. employed discretization to enable a ﬁnite exhaustive search for adversarial misclassiﬁcations. If no misclassiﬁcations are found in all layers after the exhaustive search, the NN is regarded as safe. The idea of  was to formulate the formal veriﬁcation of temporal logic properties of a CPS with Machine Learning (ML) components as the falsiﬁcation problem (ﬁnding a counterexample that does not satisfy system spec- iﬁcation). The study  adopted an ML analyzer to abstract the feature space of ML components (which ap- proximately represents the ML classiﬁers). The identiﬁed misclassifying features are then used to drive the process of falsiﬁcation. The introduction of the ML analyzer narrowed down the searching space for counterexamples and established a connection between the ML component and the rest of the system. Another direction to make sure the system will not violate safety properties is to use run-time monitoring. The study  envisioned an approach named WISEML, which combines reinforcement learning and run-time monitor- ing technique, to detect invariants violations. The purpose of this work was to create a safety envelope around the NN-based SCCPSs. 4.2.5. CA5: Improving interpretability of NNs NNs have proved to be eective ways to generalize the relationship between inputs and outputs. As the models of NNs are learned from training data sets without human intervention, the relationship between the inputs and outputs of NNs is like a black box. Due to the black-box nature of NNs, it is dicult for people to understand and explain how an NN works. Studies under this theme focus on facilitating the understanding on how NNs generate outputs from inputs. Studies in this theme can be classiﬁed into the following three sub-themes, which can be overlapped. However, this can be a way to capture the dierent motivations for the interpretability of NNs. Studies focusing on understanding how a speciﬁc decision is made. This line of work mainly focuses on providing explanations for individual predictions (also deﬁned as local interpretability). One study is called Local Interpretable Model-agnostic Explanations (LIME) . LIME can approximate the original NN model locally to provide an explanation for a speciﬁc prediction of interest. The problem of LIME is that it assumes the local linearity of the classiﬁcation boundary, which is not true for most complex NNs. The creators of LIME later extended their work by introducing high-precision rules (i.e., if-then rules), which they called anchors . The study  developed an explanation system named LEMNA for security applications and Recurrent Neural Networks (RNNs). LEMNA can locally approximate a non-linear classiﬁcation boundary and handle feature dependency problems and therefore is able to provide a high ﬁdelity explanation. 14 In the case of an image classiﬁer, it is also common to use gradient measurements to estimate the importance value of each pixel for the ﬁnal classiﬁcation. DeepLIFT , Integrated Gradients , and more recently, SmoothGrad  fall into this category. The study  proposed a uniﬁed framework, SHapley Additive exPlanations (SHAP), by integrating six existing methods (LIME , DeepLIFT , Layer-Wise Relevance Propagation, Shapley regression values, Shapley sampling values, and Quantitative Input Inﬂuence) to measure feature importance. Several approaches attempted to decompose the classiﬁcation decision (output) into the contributions of individual components of an input based on speciﬁc local decomposition rules (i.e., Pixel-Wise decomposition [107, 117], and deep Taylor decomposition ). Szegedy et al.  investigated the semantic meaning of individual units and the stability of DNNs while small perturbations were added to the input. They pointed out that the individual neurons did not contain the semantic infor- mation, while the entire space of activations does. They also experimentally proved that the same small perturbation of input can cause dierent DNN models (e.g., trained with dierent hyperparameters) to generate wrong predictions. There are several methods for improving local explanations for NN models compared to the above-mentioned approaches. The study  argued that explanation approaches for NN models should provide sound theoretical support. Ross et al.  presented their idea as “Right for the right reasons,” which means that the output of NN models should be right with the right explanation. In Ross et al. , incorrect explanations for particular inputs can be identiﬁed, and NN models can be guided to learn alternate explanations. Both [114, 118] made eorts on real-time explanations since their approaches can generate accurate explanations quickly enough. Studies focusing on facilitating understanding of the internal logic of NNs. Studies in this sub-theme are also known as global interpretability. To help interpret how NN models work, model distillation is used in , , , and . The initial intention of distillation was to reduce the computational cost. For example, Hinton et al.  distilled a collection of DNN models into a single model to facilitate deployment. The knowledge distilled from NN models has later been applied for interpretability. Some studies compressed information (e.g., decision rules) from deep learning models into transparent models such as decision trees [123, 132] and gradient boosting trees  to mimic the performance of models. Others tended to explain the inner mechanisms of NN models through analyzing the feature space. Study  distilled the relationship between input features and model predictions (outputs of the model) as a feature shape to evaluate the feature contribution to the model. Another attempt to produce global interpretability is to reveal the features learned by each neuron. For example, in , the authors leveraged deep generator networks to synthesized the input (i.e., image) that highly activates a neuron. Dong et al.  adopted an attentive encoder-decoder network to learn interpretable features, and then proposed an algorithm called prediction dierence maximization to interpret the features learned by each neuron. One interesting work  used an additional NN module that is ﬁt for relational reasoning to reason the relations between the input and response of the NN models. There is also another promising line of work (e.g., , ) that combined local and global interpretability to explain NN models. Studies focusing on visualizing internal layers of NNs to help identify errors in NNs. In study, activities, such as the operation of the classiﬁer and the function of intermesdiate feature layers within the CNN model, were visualized by using a multi-layered deconvolutional network (named DeconvNet). These visualizations are useful to interpret model problems. Unlike , which visually depicted neurons in a convolutional layer, the study  visualized neurons in a fully connected layer. Zhou et al.  proposed Class Activation Mapping (CAM) for CNNs to visualize the discriminative object parts on any given image. Fong and Vedaldi  highlighted the most respon- sible part of an image for a decision by perturbing meaningful images. DarkSight  combined the ideas of model distillation and visualization to visualize the prediction of an NN model. Thiagarajan et al.  built a TreeView rep- resentation via feature-space partitioning to interpret the prediction of an NN. Mahendran et al.  reconstructed semantic information (images) in each layer of CNNs by using information from the image representation. 4.3. RQ3. What are the limitations of current research with respect to testing and verifying NN-based SCCSs? Analyzing failure modes and how the system reacts to failures are crucial parts of the safety analysis, especially in safety-critical domains. When testing and verifying the safety of NN-based SCCPSs, we need to rethink how to perform failure mode and eect analysis, how to analyze inter-dependencies between sub-systems of SCCPSs, and how to analyze the resilience of the system. We need to ensure that even if some of the system’s hardware or software do not behave as expected, the system can sense the risk, avoid the risk before the incident, and mitigate the risk 15 eectively when an incident happens. Looking into T&V activities through software development, the ideal situation is that we would ﬁnd appropriate T&V methods to verify whether the design and implementation are consistent with the requirements, construct complete test criteria and test oracle, and generate test data and test any objects (such as code modules, data structures) that are necessary for the correct development of software . Unfortunately, the fact is that complete T&V is hard to guarantee. In order to investigate the gap between industry needs for T&V of NN-based SCCPS and state-of-the-art T&V methods, we performed a mapping of identiﬁed approaches to the relevant standard. 4.3.1. Mapping of reviewed approaches to the software safety lifecycles in IEC 61508 An increased interest in the application of NNs within safety-critical domains has encouraged research in the area of T&V of NN-based SCCSs. Research institutions and industry T&V practitioners are working on dierent aspects of this problem. However, we have not found strong connections between those potentially useful methods for T&V of NNs and relevant safety standards (such as IEC 61508  and ISO 26262 ). We hereby adopt IEC 61508  as a reference standard to execute the mapping analysis since ISO 26262  is the adaptation of IEC 61508 . We found that the major T&V activities listed in the software safety lifecycles of IEC 61508-3 (including evaluation of software architecture design, software module testing and integration, programmable electronics integration, and software veriﬁcation) are still valid when conducting T&V for NN-based SCCSs. But for most of them, new techniques/measures for supporting the T&V of NN-based software are demanded. Therefore, we decided to employ safety integrity properties (which are explained in IEC 61508-3 Annex C and Annex F of IEC 61508-7) as indicators to justify to what extent these desirable properties have been achieved by the state-of-the-art methods for T&V of NN-based SCCSs. The detailed mapping information can be found in Table 5. Table 5: A mapping of reviewed approaches to IEC 61508 safety lifecycle Relevant primary Phase Property Remaining challenges studies Software Completeness None N/A architecture Correctness  CA4 Training process of NN-based algo- design rithm is time-consuming. [57, 59, 60, 62, 66], Freedom from in- CA1 ¶ Limited to speciﬁc model  -  trinsic faults classes, or tasks (e.g., image classi- ﬁer), or small size NNs ; · Not immune to adversarial adaptation ; ¸ Lack of understanding on how system can be free from dierent kinds of attacks other than adversarial examples. Understand-  -  CA5 ¶ Limited to speciﬁc model ability classes, or tasks (e.g., image clas- siﬁer), or small size NN models ; · Not able to provide real-time explanations; ¸ Lack of evaluation method for the explanation of NNs. Veriﬁable and  CA3 ¶ Lack of integrated computer- testable design aided toolchains to support the ver- iﬁcation activities; · Limited to speciﬁc models, tasks or NN size.  CA4 ¶ Limited to speciﬁc NN archi- tectures (i.e., piece-wise linear ac- tivation functions), need better un- derstanding of NN architectures; · Trade-o between ecient veriﬁca- tion and linear approximation of the NN behavior is not studied su- ciently. Category Table 5 – continued from the previous page Relevant primary Phase Property Remaining challenges studies Fault [74, 75, 79, 82, CA2 ¶ Decouple the fault tolerance from tolerance 83] the classiﬁcation performance ; · Lack of studies on unexpected environmental failures. Defense against None N/A common cause failure Software module testing and Completeness [61, 72] CA1 Lack of comprehensive criteria to integration evaluate testing adequacy.  -  CA3 Low ﬁdelity of testing cases com- pared with real-world cases . [56, 58, 61, 63, 64] Correctness CA1 ¶ Vulnerable to the variation of ad- [65, 67] versarial examples; · Limited to speciﬁc NN model classes or tasks.  CA2 Insucient validation of input raw data. Repeatability [84, 85, 86] CA3 Testing cases generated by auto- mated tools may be biased. Precisely deﬁned None N/A testing conﬁgura- tion Programm- able electronics integration Completeness None N/A (hardware and software) Correctness [73, 76, 77, 80] CA2 Insucient testing of hardware ac- celerator. Repeatability None N/A Precisely deﬁned None N/A testing conﬁgura- tion Software Completeness [95, 97] CA4 ¶ Limited to speciﬁc NN models; veriﬁcation · Lack of scalability. Correctness  CA2 ¶ Automatic generation of com- plete testing scenarios sets. [91, 93, 94, 98, 99] CA4 ¶ Scalability and computational  -  performance need to improve; · SMT encoding for large-scale NN model; ¸ Lack of model-agnostic veriﬁcation methods; ¹ Automatic generation of feature space abstrac- tions . Repeatability None N/A Precisely deﬁned None N/A testing conﬁgura- tion In Table 5, we mapped existing T&V methods for NN-based SCCSs (column 3 and column 4) into relevant properties (column 2) of four major T&V phases (column 1) in the software safety lifecycles of IEC 61508-3. For column 5 in Table 5, we summarized the remaining challenges in testing and verifying NN-based SCCSs based on reviewed papers. The overviews of these remaining challenges can potentially inspire researchers to look for a focus in the future. Category 4.3.2. Limitations and suggestions for testing and verifying NN-based SCCSs In Table 5, we show the limitations and gaps of state-of-the-art T&V approaches for NN-based SCCSs. In this section, we will take two T&V phases (evaluation of software architecture design and software module testing and integration) as examples to provide detailed analysis of identiﬁed limitations and corresponding suggestions on the basis of required safety integrity properties. For the other two T&V phases (programmable electronics integration and software veriﬁcation), only summaries of limitations and suggestions will be presented to avoid duplication. Evaluation of software architecture design. The top three properties that have been addressed are: simplicity and understandability (31 papers), freedom from intrinsic design faults (10 papers), and fault tolerance (5 papers). Cor- rectness with respect to software safety requirements speciﬁcation (1 paper) and veriﬁable and testable design have drawn little attention (2 papers) for reviewed studies. There are two properties, i.e., completeness with respect to soft- ware safety requirements speciﬁcation and Defense against common cause failure from external events, which have not been addressed in reviewed papers. Completeness with respect to software safety requirements speciﬁcation No study contributes to the achieve- ment of completeness, which requires the architecture design to be able to address all the safety needs and constraints. The achievement of completeness depends on the achievement of other properties, such as fully understanding the behavior of NN models. The design and deployment of NN-based SCCSs are in its infancy stage. When NN-based SCCS design becomes more practical, more studies may address this topic. Correctness with respect to software safety requirements speciﬁcation To achieve correctness, software ar- chitecture design needs to respond to the speciﬁed software safety requirements appropriately. Study  reported their successful design of a DNN-based compression algorithm for aircraft collision avoidance systems. Even though they demonstrated that the DNN-based algorithm preserves the required safety performance, the training process is still time-consuming. Freedom from intrinsic design faults Intrinsic design faults can be interpreted as failures derived from the design itself. State-of-the-art NNs have proved to be vulnerable to adversarial perturbations due to some intriguing properties of NNs . Most of the studies in this category were aimed at understanding, detecting, and mitigating adversarial examples. Study  reported that their approach could generalize well on several state-of-the-art NNs to ﬁnd adversarial examples successfully. However, the veriﬁcation process of founded features is time-consuming, especially for larger images. In this sense, the scalability and computational performance of adversarial robustness are expected to be addressed in the future. In addition, adversarial robustness does not imply that the NN model is truly free from intrinsic design faults. How to assure freedom from interferences (e.g., signal-noise ratio degradation) other than adversarial perturbations is a research gap that needs to be ﬁlled. Understandability This property can be interpreted as the predictability of system behavior, even in erroneous and failure situations. In this category, studies focusing on providing explanations for individual prediction (e.g., ) and on visualizing internal layers of NN (e.g., [129, 130, 131]) are not meaningful for safety assurance. Stud- ies focusing on facilitating understanding of the internal logic of NNs (such as presenting NNs as decision trees ) could be a solution to improve the understandability of NN-based architecture design. However, this line of work is rare, and most methods are only applied to small-scale DNNs with image input, or speciﬁc NN models. Besides, assuming the explanation of NN is available, conﬁrming the correctness of the explanation is still a challenge. Inter- pretability of NNs is undoubtedly a crucial need in safety-critical applications. Methods in this line should capable of explaining dierent types of sensor data (e.g., image, text, and point data) and both local and global decisions. Veriﬁable and testable design The evaluation metrics of veriﬁable and testable design may be derived from modularity, simplicity, provability, and so on. We observed that existing veriﬁable and testable designs are limited to speciﬁc NN architectures (e.g., ) or speciﬁc tasks (e.g., ). There is no standard procedure for determining which type of NNs will be easier to verify. Ehlers  argued that NNs that adopt piece-wise linear activation 18 functions are easier to verify, but their method still need to face the conﬂict between ecient veriﬁcation and accuracy of linear approximation for the NN behavior. Fault tolerance Fault tolerance implies that the architecture design can assure the safe behavior of the software whenever internal or external errors occur. To achieve fault tolerance, features like failure detection and failure impact mitigation of both internal and external errors should be included in the design. Existing methods showed that unexpected environmental failures are hard to detect and mitigate. Besides, many of the proposed approaches in this category have not yet been evaluated in the real-world. Some studies formulated approximated computational models to represent real-world systems (e.g., ). The study  did not use any test oracle when executing system ﬂight tests. Some studies used simulation models to verify the performance of the original NN (e.g., ). They are not able to prove the ﬁdelity of the model compared with the real-world system. Defense against common cause failure from external events Software common cause failure is a type of concurrent failure of two or more modules in the software, which is caused by software design defects and triggered by external events such as time, unexpected input data, or hardware abnormalities . Many safety critical systems adopt redundant architectures (meaning two or more independent subsystems have identical functions to back-up each other) to prevent a single point of failure. However, redundant architectures are vulnerable considering common cause failure. In the context of NN-based SCCSs, it is common to employ multiple NNs with similar architectures in order to improve the accuracy of prediction. If a common cause failure occurs in this kind of software design, the prediction might be totally wrong, and thus the control software might make the wrong decision. DeepXplore, reported in , used more than two dierent DNNs with the same functionality to automatically generate a test case. If all the DNNs in DeepXplore are aected by common cause failure, such as if a sensor failure causes all the DNNs to make the same misclassiﬁcation, then it will not be able to generate the corresponding test case. No method is found in reviewed papers that can identify common cause failure modes and defend against such failures. In order to eectively defend against common cause failure, designers need to inspect the completeness and correctness of the safety requirements speciﬁcation, trace the implementation of the safety requirements speciﬁcation, and make a thorough T&V plan to reveal the common cause failure modes in the early stage. Software module testing and integration. The top two properties that have been addressed are: completeness of testing and integration with respect to the design speciﬁcations (9 papers) and correctness of testing and integration with respect to the design speciﬁcations (8 papers). Repeatability has drawn little attention (3 papers) from the reviewed studies. There is one property, precisely deﬁned testing conﬁguration, which has not been addressed in the reviewed papers. This property aims to evaluate the precision of T&V procedures, which is not in the scope of our selected papers. Therefore, we will not give more explanation on this property. Completeness of testing and integration with respect to the design speciﬁcations We observed some eorts that tried to ﬁnd a systematic way to generate testing cases (e.g., [86, 89]) to measure testing quality (e.g., ) or to connect dierent T&V stages in the development of SCCSs (e.g., ). As analyzed in Section 4.2, we can infer that an NN-based control software is instinctually dierent in design workﬂow and software development compared to the design of traditional control software. We suggest that the testing criteria should thoroughly align with the software design. To be more speciﬁc, the instinctive features of NN-based softwares (e.g., NN model’s architectural details and the working mechanism of NNs) should be carefully considered when setting the testing criteria. That is testing criteria should be deﬁned comprehensively and explicitly under the consideration of not only test case coverage but also the robustness of NN-based system performance (for instance, test how an NN will respond when input data change slightly) and the features of training data sets, such as the data density issue mentioned in . Correctness of testing and integration with respect to the design speciﬁcations Several studies (e.g., [56, 63, 64]) reported that their methods are vulnerable to the variation of adversarial examples. Another common limitation is that most methods are model-speciﬁc, meaning that they can only apply to a single type or class of NN model. To achieve correctness of testing and integration, the module testing task should be completed, which means the testing 19 should cover both NN models and external input. However, few studies focused on the validation of input data. One study  identiﬁed that sucient validation of input raw data remains a challenge. Repeatability The complexity and un-interpretable feature of NNs make manual testing almost infeasible. In order to be able to generate consistent results from testing repeatedly, some studies were dedicated to achieving automatic test execution or even automatic test generation. We found three papers (i.e., [84, 85, 86]) addressing automatic test generation. However, generating test cases automatically is still a challenge. For instance, studies [85, 86] claimed that the test cases generated by an automated testing tool may not cover all real-world cases. Programmable electronics integration. The major limitation of this line of work is insucient testing for hardware accelerators. NN-based SCCPSs requires typically high-performance computing systems, such as Graphics Process- ing Units (GPUs). Some industry participants have provided specialized hardware accelerators to accelerate NN-based computations. For example, Google deployed a DNN accelerator (called Tensor Processing Unit) in its data centers for DNN applications . NVIDIA introduced an automotive supercomputing platform named DRIVE PX 2 , which now has been used by over 370 companies and research institutions in the automotive industry . However, little research eort has been put into the T&V of the reliability of using hardware accelerators for NN applications. We found seven studies (i.e. [73, 74, 75, 77, 76, 78, 80]) addressing the evaluation of the error resilience of hard- ware accelerators. However, the testing is limited to speciﬁc type errors (e.g., radiation-induced soft errors, which are presented in [73, 77, 80]). The mitigation method proposed in  (called ABFT: Algorithm-Based Fault Tolerance) can only protect portions of the accelerator (e.g., sgemm kernels, which is one kind of matrix multiplication kernels). The study  identiﬁed errors made by single frame object detectors, but the result showed that the method is not capable of detecting all mistakes. The studies [73, 80] investigated the propagation characteristic of soft errors in the DNN system, but they used a DNN simulator instead of a real DNN accelerator for fault injection. Software veriﬁcation. In general, there is a lack of a comprehensive and standardized framework for verifying the safety of NN-based SCCSs. Formal veriﬁcation procedures are highly demanding. The common limitation of formal veriﬁcation approaches is the scalability issues. Most proposed methods are limited to a speciﬁc NN structure and size (e.g., [92, 93, 98, 100, 101]). The study  reported that their approaches can only verify small-scale systems (i.e., the layer of NN is 3 and the maximum amount of input neurons is 64). One approach reported in  can verify medium size NNs. The veriﬁcation of large-scale NNs is still a challenge. Another limitation is that proposed approaches are not robust to NN variations. For example, veriﬁcation methods in studies [92, 98] are only adapted to speciﬁc network types and sizes. 5. Discussion In this section, we ﬁrst discuss industry practices for T&V of NN-based SCCPSs. Then, we compare this SLR with related works. At the end of this section, we present the threats to the validity of our study. 5.1. Industry practice Our ﬁndings on the research questions (RQ1 to RQ3) mainly reﬂected the academic eorts addressing T&V of NN-based SCCPSs. NN-based applications have drawn a lot of attention from industry practitioners. Taking the automotive industry as an example, several car makers (e.g., GM, BMW, and Tesla) and some high technology companies (e.g., Waymo and Baidu) are leading the revolution in autonomous driving safety. 5.1.1. Safety of the intended functionality At the beginning of this year, ISO/PAS 21448:2019  was published. It listed recommended methods for deriving veriﬁcation and validation activities (See ISO/PAS 21448:2019 Table 4). In Table 6, we highlighted six of the recommended methods, which shared similar veriﬁcation interests with existing academic eorts. 20 Table 6: Shared veriﬁcation interests of ISO/PAS 21448 and academic eorts ISO/PAS 21448 Academic eorts Analysis of triggering events CA1: Assuring robustness of NNs Analysis of sensors design and their known potential limitations CA2: Improving failure resilience of NNs Analysis of environmental conditions and operational use cases CA3: Measuring and ensuring test completeness Analysis of boundary values CA4: Assuring safety property of NN-based SCCPSs Analysis of algorithms and their decision paths CA5: Improving interpretability of NNs Analysis of system architecture CA1-CA5 5.1.2. Safety reports In 2018, three companies (Waymo, General Motor, and Baidu Apollo) published their annual safety reports. As a pioneer in the development of self-driving cars, Waymo proposed the “Safety by Design”  approach, which entails the processes and techniques they used to face safety challenges of a level 4 autonomous car on the road. For the cybersecurity consideration, Waymo adopted Google’s security framework  as the foundation. Af- ter that, General Motor (GM) released their safety report  for Cruise AV (also level 4). GM’s safety process combined conventional system validation (such as vehicle performance tests, fault injection testing, intrusive test- ing, and simulation-based software validation) with SOTIF validation through iterative design. Baidu adopted the Responsibility-Sensitive Safety model  proposed by Mobileye  (an Intel company) to design the safety process for the Apollo Pilot for a passenger car (level 3). In addition, we noticed that Tesla started releasing quarterly safety data since October 2018 . It seemed that Tesla has a completely dierent approach to self-driving cars than other companies. According to TESLA NEWS , AutoPilot will rely for its self-driving function on cameras, not on LIDAR; the AutoPilot software is trained online (which means that the NN keeps learning and evolving during operation). The Autopilots safety features are continuously evolved and enhanced through understanding real-world driving data from every Tesla. Referring to these safety reports of existing autonomous cars, we should be aware that when testing DNN-based control software (the core part of autonomous vehicles), black-box system level testing (by observing inputs and its corresponding outputs, e.g., closed course testing and real-world driving) is still the leading method. More systematic T&V criteria and approaches are needed for more complete and reliable testing results. 5.2. Comparison with related work 5.2.1. Veriﬁcation and validation of NNs Taylor et al.  conducted a survey on the Veriﬁcation and Validation (V&V) of NNs used in safety-critical domains in 2003. Study  is the closest work we found, although they did not adopt an SLR approach. Our study covered new studies from 2011-2018. The authors of  also made a classiﬁcation of methods for the V&V of NNs. They grouped the methods into ﬁve traditional V&V technique categories, namely, automated testing and testing data generation methods, run-time monitoring, formal methods, cross validation, and visualization. In contrast to , our study adopted a thematic analysis approach  and identiﬁed ﬁve themes based on the research goals of the selected studies. We thought it was better to classify the proposed T&V methods of NNs based on their aims rather than on the traditional technique categories since many traditional V&V techniques are no longer eective for verifying NNs in many cases. New methods and tools should be explored and developed without being limited by the traditional V&V categories. Another dierence is our study specialized more in the T&V of modern NNs, such as MLP and DNN, whereas the study  provided more in-depth analysis of V&V methodologies for NNs used in ﬂight control system, such as Pre-Trained Neural Network (PTNN) and Online Learning Neural Network (OLNN). Our study and  have some common ﬁndings. For example, one category, named Visualization in , falls into our category CA5 Improving interpretability of NNs. 5.2.2. Surveys of security, safety, and productivity for Deep Learning (DL) systems development Hains et al.  surveyed existing work on “attacks against DL systems; testing, training, and monitoring DL systems for safety; and the veriﬁcation of DL systems.” Our study and  shared a similar motivation. The critical dierence between our SLR and  are threefold: 1) We conducted our literature review on 83 selected papers based 21 on speciﬁc SLR guidelines, while they used an ad hoc literature review (ALR) approach and reviewed only 21 primary papers. 2) They only focused on DL systems, whereas our scope covered modern NN-based software systems, which embodies DL-based software systems. 3) They inferred that formal methods and automation veriﬁcation are the two promising research directions based on the reviewed works. In contrast, we focused more on safety issues, and found more categories to be addressed for safety purposes. 5.2.3. Surveys of certiﬁcation of AI technologies in automotive Falcini et al. [17, 18] reviewed the existing standards in the automotive industry and pointed out the related applicability issues of automotive software development standards to deep learning. Although our SLR takes the automotive industry as an example, we are concerned with SCCPSs in general. This concern is reﬂected in the distribution of the selected papers (only 13 of the 83 selected papers are oriented to automotive CPSs). 5.2.4. SLR of Explainable Artiﬁcial Intelligence (XAI) There are two very recent SLRs,  and , on the interpretation of artiﬁcial intelligence. Both  and  employed similar commonly accepted guidelines to conduct their SLRs. The fundamental dierence between our study and [159, 158] is the scope.  reviewed 381 papers on existing XAI approaches from interdisciplinary perspectives. As reported in , the scope of their SLR is visualization and visual analytics for deep learning. The study  focused on studies that adopted visual analytics to explain NN decisions. Our study has a more comprehensive coverage of T&V approaches that were employed to not only interpret NN behaviors but also to assure the robustness of NNs, to improve the failure resilience of NNs, to ensure test completeness, and to assure the safety property of NN-based SCCPSs. In a summary, our SLR tried to provide an overview of key aspects related to T&V activities for NN-based SCCSs. 5.3. Threats to validity In this section, we discuss some threats to the validity of our study. 5.3.1. Search strategy The most possible threat in this step is missing or excluding relevant papers. To mitigate this threat, we used six of the most relevant digital libraries to retrieve papers. Additionally, we employed two strategies to mitigate potential limitations in the search terms: 1) adopted an PIOC criteria to ensure the coverage of search terms; and 2) improved search terms iteratively. Further, we conducted an extensive snowballing process on references of the selected papers to identify related papers. The search keywords were cross-checked and agreed on by both authors. 5.3.2. Study selection Researchers’ subjective judgment could be a threat to the study selection. We strictly followed the pre-deﬁned review protocol to mitigate this threat. For example, we started recording the inclusion and exclusion reasons from the 3rd stage. We validated the inclusion and exclusion criteria with two authors on the basis of the pilot search. Furthermore, the second author performed a cross-check of all selected papers. Any paper that raised doubts about its inclusion or exclusion decision was discussed between the ﬁrst and second authors. For example, the “smart grid” is included in the search term, but no relevant papers were found after the 3rd stage. Then, we conducted a snowballing search to identify papers that presented how to use NNs in smart grids. We found out that AI is mainly used to solve the economically relevant problems  of the smart grid system (e.g., prediction of energy usage and ecient use of resources). AI is not involved in the safety-critical applications (e.g., decision making on optimal provision of power) of smart grids. Therefore, there were no relevant papers addressing safety analysis or testing/veriﬁcation (refer to Inclusion criteria I2). 5.3.3. Data extraction The ﬁrst author was responsible for designing the data extraction form and conducting the data extraction from selected papers. In order to avoid the ﬁrst author’s bias in data extraction, the two authors continuously discussed the data extraction issues. The extracted data were veriﬁed by the second author. 22 5.3.4. Data synthesis Data analysis outcomes could vary with dierent researchers. To reduce the subjective impact on data synthesis, besides strictly following the thematic synthesis steps , the data synthesis was ﬁrst agreed on by both authors. We disseminated our preliminary ﬁndings to two internal research groups at our university (i.e., the autonomous vehicle lab and autonomous ships lab) and presented at a Ph.D. seminar on IoT, Machine Learning, Security, and Privacy for comments and feedback. In summary, the audiences agreed with our research design and results, and they thought that the mapping of reviewed approaches to the IEC61508 is a valuable attempt. Several researchers working in formal veriﬁcation and safety veriﬁcation thought that safety cases would be a promising direction to address the challenges of T&V of NN-based SCCSs. One suggestion is adding information about self-driving car simulators. Based on these comments and feedback, we revised our paper accordingly. 6. Conclusion and future work In this paper, we have presented the results of a Systematic Literature Review (SLR) of existing approaches and practices on T&V methods for neural-network-based safety critical control software (NN-based SCCS). The motivation of this study was to provide an overview of the state-of-the-art T&V of safety-critical NN-based SCCSs and to shed some light on potential research directions. Based on pre-deﬁned inclusion and exclusion criteria, we selected 83 papers that were published between 2011 and 2018. A systematic analysis and synthesis of the data extracted from the papers and comprehensive reviews of industry practices (e.g., technical reports, standards, and white papers) related to our RQs were performed. Results of the study show that: 1. The research on T&V of NN-based SCCSs is gaining interest and attention from both software engineering and safety engineering researchers/practitioners according to the impressive upward trend in the number of papers on T&V of NN-based SCCSs (See Fig. 5). Most of the reviewed papers (68/83, 81.9%) have been published in the last three years. 2. The approaches and tools reported for the T&V of NN-based control software have been applied to a wide variety of safety-critical domains, among which automotive CPSs has received the most attention. 3. The approaches can be classiﬁed into ﬁve high-order themes, namely, assuring robustness of NNs, improving failure resilience of NNs, measuring and ensuring test completeness, assuring safety properties of NN-based SCCPSs, and improving interpretability of NNs. 4. The activities listed in the software safety lifecycles of IEC 61508-3 are still valid when conducting safety veriﬁcation for NN-based control software. However, most of the activities need new techniques/measures to deal with the new characteristics of NNs. 5. Four safety integrity properties within the four major safety lifecycle phases, namely, correctness, completeness, freedom from intrinsic faults, and fault tolerance, have drawn the most attention from the research community. Little eort has been put on achieving repeatability. No reviewed study focused on precisely deﬁned testing conﬁguration and defense against common cause failure, which are extremely crucial for assuring the safety of a production-ready NN-based SCCS . 6. It is common to combine standard testing techniques with formal veriﬁcation when testing and verifying large- scale, complex safety-critical software [15, 145]. As explained in section 4.3, we found that an increasing concern of the reviewed works is the integration of dierent T&V techniques in a systematic manner to gain assurance for the whole lifecycle of the NN-based control software. This SLR is just a starting point in our studies to test and verify NN-based SCCPSs. In the future, we will focus on improving the interpretability of NNs. To be more speciﬁc, we plan to develop a method for explaining why an NN model is more (or less) robust than other models. It can guide software designers to design an NN model with an appropriate robustness level, which will greatly support safety by design. Acknowledgments The authors would like to thank Weifeng Liu for commenting on and improving this paper. This work is supported by the Safety, autonomy, remote control and operations of industrial transport systems (SAREPTA) project, which is 23 ﬁnanced by the Norwegian Research Council with Grant No. 267860. This work is also supported by the Management of safety and security risks for cyber-physical systems project, which is ﬁnanced by the Norwegian University of Science and Technology and the Technical University of Denmark. References  R. Rajkumar, I. Lee, L. Sha, J. Stankovic, Cyber-physical systems: the next computing revolution, in: Design Automation Conference (DAC), 2010 47th ACM/IEEE, IEEE, pp. 731–736.  B. K. Bose, Neural network applications in power electronics and motor drivesan introduction and perspective, IEEE Transactions on Industrial Electronics 54 (2007) 14–33.  P. Ongsulee, Artiﬁcial intelligence, machine learning and deep learning, in: ICT and Knowledge Engineering (ICT&KE), 2017 15th International Conference on, pp. 1–6.  M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., End to end learning for self-driving cars, arXiv preprint arXiv:1604.07316 (2016).  K. D. Julian, J. Lopez, J. S. Brush, M. P. Owen, M. J. Kochenderfer, Policy compression for aircraft collision avoidance systems, in: Digital Avionics Systems Conference (DASC), 2016 IEEE/AIAA 35th, IEEE, pp. 1–10.  S. Levin, J. C. Wong, Self-driving uber kills arizona woman in ﬁrst fatal crash involving pedestrian, https://www.theguardian.com/ technology/2018/mar/19/uber-self-driving-car-kills-woman-arizona-tempe, 2018. Accessed: 2018-07-27.  D. Yadron, D. Tynan, Tesla driver dies in ﬁrst fatal crash while using autopilot mode, https://www.theguardian.com/technology/ 2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk, 2016. Accessed: 2018-07-27.  D. Lee, Google self-driving car hits a bus, https://www.bbc.com/news/technology-35692845, 2016. Accessed:18-12-2018.  Valasek, Chris, Miller, Charlie, Who’s behind the wheel? exposing the vulnerabilities and risks of high tech vehicles, https://trid.trb. org/view/1370158, 2015. Accessed: 2018-07-27.  S. Kriaa, L. Pietre-Cambacedes, M. Bouissou, Y. Halgand, A survey of approaches combining safety and security for industrial control systems, Reliability Engineering & System Safety 139 (2015) 156–178.  T. Aven, A uniﬁed framework for risk and vulnerability analysis covering both safety and security, Reliability Engineering & System Safety 92 (2007) 745 – 754.  G. Stoneburner, Toward a uniﬁed security-safety model, Computer 39 (2006) 96–97.  T. Novak, A. Treytl, Functional safety and system security in automation systems - a life cycle model, in: 2008 IEEE International Conference on Emerging Technologies and Factory Automation, pp. 311–318.  P. Bieber, J.-P. Blanquart, G. Descargues, M. Dulucq, Y. Fourastier, E. Hazane, M. Julien, L. Leonardon, ´ G. Sarouille, Security and safety assurance for aerospace embedded systems, in: Proceedings of the 6th International Conference on Embedded Real Time Software and Systems, Toulouse, France, pp. 1–10.  B. J. Taylor, M. A. Darrah, C. D. Moats, Veriﬁcation and validation of neural networks: a sampling of research in progress, in: Intelligent Computing: Theory and Applications, volume 5103, International Society for Optics and Photonics, pp. 8–17.  G. Hains, A. Jakobsson, Y. Khmelevsky, Towards formal methods and software engineering for deep learning: Security, safety and produc- tivity for dl systems development, in: Systems Conference (SysCon), 2018 Annual IEEE International, IEEE, pp. 1–5.  F. Falcini, G. Lami, Challenges in certiﬁcation of autonomous driving systems, in: 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 286–293.  F. Falcini, G. Lami, Deep learning in automotive: Challenges and opportunities, in: A. Mas, A. Mesquida, R. V. O’Connor, T. Rout, A. Dorling (Eds.), Software Process Improvement and Capability Determination, Springer International Publishing, 2017, pp. 279–288.  P. Van Wesel, A. E. Goodloe, Challenges in the Veriﬁcation of Reinforcement Learning Algorithms, Technical Report, 2017. https: //ntrs.nasa.gov/search.jsp?R=20170007190.  B. Kitchenham, S. Charters, Guidelines for performing systematic literature reviews in software engineering, 2007.  E. Lee, The past, present and future of cyber-physical systems: A focus on models, Sensors 15 (2015) 4837–4869.  A. Humayed, J. Lin, F. Li, B. Luo, Cyber-physical systems security - a survey, IEEE Internet of Things Journal 4 (2017) 1802–1831.  E. R. Grior, C. Greer, D. A. Wollman, M. J. Burns, Framework for Cyber-Physical Systems: Volume 1, Overview, Technical Report, 2017. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1500-201.pdf.  W. S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity, The bulletin of mathematical biophysics 5 (1943) 115–133.  F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain., Psychological review 65 (1958)  G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochenderfer, Reluplex: An ecient smt solver for verifying deep neural networks, in: International Conference on Computer Aided Veriﬁcation, Springer, pp. 97–117.  R. Kruse, C. Borgelt, F. Klawonn, C. Moewes, M. Steinbrecher, P. Held, Multi-layer perceptrons, in: Computational Intelligence, Springer, 2013, pp. 47–81.  Y. LeCun, Y. Bengio, G. Hinton, Deep learning, nature 521 (2015) 436.  Y. LeCun, L. Bottou, Y. Bengio, P. Haner, et al., Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (1998) 2278–2324.  M. van Gerven, S. Bohte, Artiﬁcial neural networks as models of neural information processing, Frontiers Media SA, 2018.  D. M. Rodvold, A software development process model for artiﬁcial neural networks in critical applications, in: IJCNN’99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339), volume 5, pp. 3317–3322.  F. Falcini, G. Lami, A. M. Costanza, Deep learning in automotive software, IEEE Software 34 (2017) 56–63.  J. L. Heilbron, The Oxford companion to the history of modern science, Oxford University Press, 2003. 24  SAE, J3016:Taxonomy and Deﬁnitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, Standard, 2014.  NVIDIA, Nvidia drive:scalable ai platform for autonomous driving, https://www.nvidia.com/en-us/self-driving-cars/drive- platform/, 2018. Accessed:18-12-2018.  J. C. Hoskins, D. M. Himmelblau, Process control via artiﬁcial neural networks and reinforcement learning, Computers & Chemical Engineering 16 (1992) 241–251.  T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971 (2015).  S. P. K. Spielberg, R. B. Gopaluni, P. D. Loewen, Deep reinforcement learning approaches for process control, in: 2017 6th International Symposium on Advanced Control of Industrial Processes (AdCONIP), pp. 201–206.  G. Zhabelova, V. Vyatkin, Multiagent smart grid automation architecture based on iec 61850/61499 intelligent logical nodes, IEEE Transactions on Industrial Electronics 59 (2012) 2351–2362.  B. K. Bose, Artiﬁcial intelligence techniques in smart grid and renewable energy systemssome example applications, Proceedings of the IEEE 105 (2017) 2262–2273.  G. Robertson, E. D. Lehmann, W. Sandham, D. Hamilton, Blood glucose prediction using artiﬁcial neural networks trained with the aida diabetes simulator: a proof-of-concept pilot study, Journal of Electrical and Computer Engineering 2011 (2011) 1–11.  M. K. Bothe, L. Dickens, K. Reichel, A. Tellmann, B. Ellger, M. Westphal, A. A. Faisal, The use of reinforcement learning algorithms to meet the challenges of an artiﬁcial pancreas, Expert review of medical devices 10 (2013) 661–673.  Medtronic, Medtronic initiates u.s. launch of world’s ﬁrst hybrid closed loop system for type 1 diabetes, http://newsroom.medtronic. com/phoenix.zhtml?c=251324&p=irol-newsArticle&ID=2279529, 2017. Accessed: 2018-08-25.  K. Sennaar, Ai in medical devices three emerging industry applications, https://www.techemergence.com/ai-medical-devices- three-emerging-industry-applications/, 2018. Accessed: 2018-08-16.  H. Greenspan, B. Van Ginneken, R. M. Summers, Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique, IEEE Transactions on Medical Imaging 35 (2016) 1153–1159.  IEC61508:2005, Functional safety of electrical/electronic/programmable electronic safety-related systems, Standard, International Elec- trotechnical Commission, 2005.  ISO 26262:2011, Road vehicles – Functional safety, Standard, International Organization for Standardization, 2011.  G. Griessnig, A. Schnellbach, Development of the 2nd edition of the iso26262, in: J. Stolfa, S. Stolfa, R. V. O’Connor, R. Messnarz (Eds.), Systems, Software and Services Process Improvement, Springer International Publishing, 2017, pp. 535–546.  Hansen, Standardization Eorts on Autonomous Driving Safety Barely Under Way, Technical Report, 2017.  A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, Carla: An open urban driving simulator, arXiv preprint arXiv:1711.03938 (2017).  Udacity, An open source self-driving car, https://github.com/udacity/self-driving-car, 2016. Accessed:2018-12-19.  K. Petersen, S. Vakkalanka, L. Kuzniarz, Guidelines for conducting systematic mapping studies in software engineering: An update, Information and Software Technology 64 (2015) 1–18.  M. Shahin, M. A. Babar, L. Zhu, Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices, IEEE Access 5 (2017) 3909–3943.  P. H. Nguyen, S. Ali, T. Yue, Model-based security engineering for cyber-physical systems: A systematic mapping study, Information and Software Technology 83 (2017) 116–135.  D. S. Cruzes, T. Dyba, Recommended steps for thematic synthesis in software engineering, in: 2011 International Symposium on Empirical Software Engineering and Measurement, pp. 275–284.  A. Nguyen, J. Yosinski, J. Clune, Deep neural networks are easily fooled: High conﬁdence predictions for unrecognizable images, pp. 427–436.  I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 (2014).  M. Melis, A. Demontis, B. Biggio, G. Brown, G. Fumera, F. Roli, Is deep learning safe for robot vision? adversarial examples against the icub humanoid, in: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 751–759.  O. Bastani, Y. Ioannou, L. Lampropoulos, D. Vytiniotis, A. Nori, A. Criminisi, Measuring neural net robustness with constraints, pp. 2613–2621.  M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, N. Usunier, Parseval networks: Improving robustness to adversarial examples, arXiv preprint arXiv:1704.08847 (2017).  N. Carlini, D. Wagner, Towards evaluating the robustness of neural networks, in: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57.  S. Gu, L. Rigazio, Towards deep neural network architectures robust to adversarial examples, arXiv preprint arXiv:1412.5068 (2014).  M. Wu, M. Wicker, W. Ruan, X. Huang, M. Kwiatkowska, A game-based approximate veriﬁcation of deep neural networks with provable guarantees, arXiv preprint arXiv:1807.03571 (2018).  D. Gopinath, G. Katz, C. S. Pasareanu, C. Barrett, Deepsafe: A data-driven approach for checking adversarial robustness in neural networks, arXiv preprint arXiv:1710.00486 (2017).  F. Reuben, R. R. Curtin, S. Saurabh, A. B. Gardner, Detecting adversarial samples from artifacts, arXiv preprint arXiv:1703.00410 (2017).  W. Xu, D. Evans, Y. Qi, Feature squeezing: Detecting adversarial examples in deep neural networks, arXiv preprint arXiv:1704.01155 (2017).  M. Wicker, X. Huang, M. Kwiatkowska, Feature-guided black-box safety testing of deep neural networks, volume 10805 LNCS, Springer Verlag, 2018, pp. 408–426.  J. H. Metzen, T. Genewein, V. Fischer, B. Bischo, On detecting adversarial perturbations, arXiv preprint arXiv:1702.04267 (2017).  N. Papernot, P. McDaniel, X. Wu, S. Jha, A. Swami, Distillation as a defense to adversarial perturbations against deep neural networks, arXiv preprint arXiv:1511.04508 (2015).  N. Papernot, P. McDaniel, Extending defensive distillation, arXiv preprint arXiv:1705.05264 (2017). 25  S. Zheng, Y. Song, T. Leung, I. Goodfellow, Improving the robustness of deep neural networks via stability training, pp. 4480–4488.  U. Shaham, Y. Yamada, S. Negahban, Understanding adversarial training: Increasing local stability of neural nets through robust optimiza- tion, arXiv preprint arXiv:1511.05432 (2015).  C. Schorn, A. Guntoro, G. Ascheid, Accurate neuron resilience prediction for a ﬂexible reliability management in neural network accelera- tors, in: 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 979–984.  Q. Zhang, T. Wang, Y. Tian, F. Yuan, Q. Xu, Approxann: an approximate computing framework for artiﬁcial neural network, EDA Consortium, 2015, pp. 701–706.  J.-C. Vialatte, F. Leduc-Primeau, A study of deep learning robustness against computation failures, arXiv preprint arXiv:1704.05396 (2017).  G. Li, K. Pattabiraman, C.-Y. Cher, P. Bose, Understanding error propagation in gpgpu applications, IEEE, 2016, pp. 240–251.  F. F. d. Santos, L. Draghetti, L. Weigel, L. Carro, P. Navaux, P. Rech, Evaluation and mitigation of soft-errors in neural network-based object detection in three gpu architectures, in: 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pp. 169–176.  S. R. Manikandasriram, C. Anderson, R. Vasudevan, M. Johnson-Roberson, Failing to learn: autonomously identifying perception failures for self-driving cars [arxiv], arXiv arXiv:1707.00051 (2017) 8 pp.  E. M. E. Mhamdi, R. Guerraoui, S. Rouault, On the robustness of a neural network, in: 2017 IEEE 36th Symposium on Reliable Distributed Systems (SRDS), pp. 84–93.  G. Li, S. K. S. Hari, M. Sullivan, T. Tsai, K. Pattabiraman, J. Emer, S. W. Keckler, Understanding error propagation in deep learning neural network (dnn) accelerators and applications, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ACM, p. 8.  A. H. M. Rubaiyat, Y. Qin, H. Alemzadeh, Experimental resilience assessment of an open-source driving agent, CoRR abs/1807.06172 (2018).  K. Rhazali, B. Lussier, W. Schn, S. Geronimi, Fault tolerant deep neural networks for detection of unrecognizable situations, IFAC- PapersOnLine 51 (2018) 31–37.  S. Daftry, S. Zeng, J. A. Bagnell, M. Hebert, Introspective perception: Learning to predict failures in vision systems, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1743–1750.  M. O’Kelly, H. Abbas, R. Mangharam, Computer-aided design for safe autonomous vehicles, in: Resilience Week (RWS) 2017, 2017 Resilience Week (RWS), IEEE, 2017, pp. 90–6.  K. Pei, Y. Cao, J. Yang, S. Jana, Deepxplore: Automated whitebox testing of deep learning systems, Association for Computing Machinery, Inc, 2017, pp. 1–18.  Y. Tian, K. Pei, S. Jana, B. Ray, Deeptest: Automated testing of deep-neural-network-driven autonomous cars, in: Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, ACM, New York, NY, USA, 2018, pp. 303–314.  S. Raj, S. K. Jha, A. Ramanathan, L. L. Pullum, Work-in-progress: testing autonomous cyber-physical systems using fuzzing features from convolutional neural networks, in: 2017 International Conference on Embedded Software (EMSOFT), pp. 1–2.  L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, J. Zhao, Y. Wang, Deepgauge: Multi-granularity testing criteria for deep learning systems, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ACM, 2018, pp. 120–131.  M. Zhang, Y. Zhang, L. Zhang, C. Liu, S. Khurshid, Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ACM, pp. 132–142.  J. Guo, Y. Jiang, Y. Zhao, Q. Chen, J. Sun, Dlfuzz: dierential fuzzing testing of deep learning systems, in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, pp. 739–743.  L. Pulina, A. Tacchella, Challenging smt solvers to verify neural networks, AI Communications 25 (2012) 117–135.  R. Ehlers, Formal veriﬁcation of piece-wise linear feed-forward neural networks, Springer, 2017, pp. 269–286.  L. Pulina, A. Tacchella, N e v er: a tool for artiﬁcial neural networks veriﬁcation, Annals of Mathematics and Artiﬁcial Intelligence 62 (2011) 403–425.  S. Dutta, S. Jha, S. Sanakaranarayanan, A. Tiwari, Output range analysis for deep neural networks, arXiv preprint arXiv:1709.09130 (2017).  W. Xiang, H. D. Tran, T. T. Johnson, Output reachable set estimation and veriﬁcation for multilayer neural networks, IEEE Transactions on Neural Networks and Learning Systems (2018) 1–7.  K. D. Julian, J. Lopez, J. S. Brush, M. P. Owen, M. J. Kochenderfer, Policy compression for aircraft collision avoidance systems, IEEE, 2016, pp. 1–10.  W. Xiang, H.-D. Tran, T. T. Johnson, Reachable set computation and safety veriﬁcation for neural networks with relu activations, arXiv preprint arXiv:1712.08163 (2017).  G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochenderfer, Reluplex: An ecient smt solver for verifying deep neural networks, in: Computer Aided Veriﬁcation. CAV 2017, Springer, 2017, pp. 97–117.  X. Huang, M. Kwiatkowska, S. Wang, M. Wu, Safety veriﬁcation of deep neural networks, in: International Conference on Computer Aided Veriﬁcation, Springer, pp. 3–29.  N. Narodytska, S. P. Kasiviswanathan, L. Ryzhyk, M. Sagiv, T. Walsh, Verifying properties of binarized deep neural networks, arXiv preprint arXiv:1709.06662 (2017).  C.-H. Cheng, G. Nhrenberg, H. Ruess, Veriﬁcation of binarized neural networks, arXiv preprint arXiv:1710.03107 (2018).  T. Dreossi, A. Donz, S. A. Seshia, Compositional falsiﬁcation of cyber-physical systems with machine learning components, NASA Formal Methods, Springer International Publishing, 2017, pp. 357–372.  P. Mallozzi, P. Pelliccione, C. Menghi, Keeping intelligence under control, in: Proceedings of the 1st International Workshop on Software Engineering for Cognitive Services, ACM, 2018, pp. 37–40.  C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, arXiv preprint 26 arXiv:1312.6199 (2013).  M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: High-precision model-agnostic explanations, in: Proceedings of the 32rd AAAI Conference on Artiﬁcial Intelligence.  M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3319–3328.  S. Bach, A. Binder, K.-R. Mller, W. Samek, Controlling explanatory heatmap resolution and semantics via decomposition depth, in: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2271–2275.  K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: Visualising image classiﬁcation models and saliency maps, arXiv preprint arXiv:1312.6034 (2013).  G. Montavon, S. Lapuschkin, A. Binder, W. Samek, K.-R. Mller, Explaining nonlinear classiﬁcation decisions with deep taylor decomposi- tion, Pattern Recognition 65 (2017) 211–222.  D. Linsley, D. Scheibler, S. Eberhardt, T. Serre, Global-and-local attention networks for visual recognition, arXiv preprint arXiv:1805.08819 (2018).  Y. Dong, H. Su, J. Zhu, B. Zhang, Improving interpretability of deep neural networks with semantic information, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4306–4314.  R. C. Fong, A. Vedaldi, Interpretable explanations of black boxes by meaningful perturbation, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 3429–3437.  B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921–2929.  M. A. K.-R. M. Dumitru, E. B. K. S. D. Pieter, J. Kindermans, K. T. Schtt, Learning how to explain neural networks: Patternnet and patternattribution, in: Proceedings of the International Conference on Learning Representations (2018).  R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, F. Giannotti, Local rule-based explanations of black box decision systems, arXiv preprint arXiv:1805.10820 (2018).  A. Shrikumar, P. Greenside, A. Kundaje, Learning important features through propagating activation dierences, in: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp. 3145–3153.  S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Mller, W. Samek, On pixel-wise explanations for non-linear classiﬁer decisions by layer-wise relevance propagation, PLOS ONE 10 (2015) 1–46.  P. Dabkowski, Y. Gal, Real time image saliency for black box classiﬁers, in: Advances in Neural Information Processing Systems, pp. 6967–6976.  A. S. Ross, M. C. Hughes, F. Doshi-Velez, Right for the right reasons: Training dierentiable models by constraining their explanations, arXiv preprint arXiv:1703.03717 (2017).  A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, T. Lillicrap, A simple neural network module for relational reasoning, in: Advances in neural information processing systems, pp. 4967–4976.  D. Smilkov, N. Thorat, B. Kim, F. Vigas, M. Wattenberg, Smoothgrad: removing noise by adding noise, arXiv preprint arXiv:1706.03825 (2017).  S. M. Lundberg, S.-I. Lee, A uniﬁed approach to interpreting model predictions, in: Advances in Neural Information Processing Systems, pp. 4765–4774.  N. Frosst, G. Hinton, Distilling a neural network into a soft decision tree, arXiv preprint arXiv:1711.09784 (2017).  Z. Che, S. Purushotham, R. Khemani, Y. Liu, Distilling knowledge from deep networks with applications to healthcare domain, arXiv preprint arXiv:1512.03542 (2015).  G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 (2015).  K. Xu, D. H. Park, C. Yi, C. Sutton, Interpreting deep classiﬁer by visual distillation of dark knowledge, arXiv preprint arXiv:1803.04042 (2018).  S. Tan, R. Caruana, G. Hooker, P. Koch, A. Gordo, Learning global additive explanations for neural nets using model distillation, arXiv preprint arXiv:1801.08640 (2018).  A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, J. Clune, Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, in: Advances in Neural Information Processing Systems, pp. 3387–3395.  M. D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, Springer, 2014, pp. 818–833.  M. T. Ribeiro, S. Singh, C. Guestrin, Why should i trust you?: Explaining the predictions of any classiﬁer, ACM, 2016, pp. 1135–1144.  W. Guo, D. Mu, J. Xu, P. Su, G. Wang, X. Xing, Lemna: Explaining deep learning based security applications, in: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, ACM, pp. 364–379.  O. Bastani, C. Kim, H. Bastani, Interpretability via model extraction, arXiv preprint arXiv:1706.09773 (2017).  J. J. Thiagarajan, B. Kailkhura, P. Sattigeri, K. N. Ramamurthy, Treeview: Peeking into deep neural networks via feature-space partitioning, arXiv preprint arXiv:1611.07429 (2016).  A. Mahendran, A. Vedaldi, Understanding deep image representations by inverting them, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188–5196.  J. Deng, W. Dong, R. Socher, L. Li, and, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255.  N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, A. Swami, The limitations of deep learning in adversarial settings, IEEE, 2016, pp. 372–387.  Q. Lu, M. Farahani, J. Wei, A. Thomas, K. Pattabiraman, Llﬁ: An intermediate code-level fault injection tool for hardware faults, in: 2015 IEEE International Conference on Software Quality, Reliability and Security, IEEE, pp. 11–16.  S. Borkar, Designing reliable systems from unreliable components: the challenges of transistor variability and degradation, IEEE Micro 25 (2005) 10–16.  N. Leveson, Engineering a safer world: Systems thinking applied to safety, MIT press, 2011. 27  T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, Z. Q. Zhou, Metamorphic testing: A review of challenges and opportunities, ACM Comput. Surv. 51 (2018) 4:1–4:27.  M. W. Moskewicz, C. F. Madigan, Y. Zhao, L. Zhang, S. Malik, Cha: Engineering an ecient sat solver, ACM, 2001, pp. 530–535.  H. Zhang, Sato: An ecient prepositional prover, in: International Conference on Automated Deduction, Springer, pp. 272–275.  J. P. Marques-Silva, K. A. Sakallah, Grasp: A search algorithm for propositional satisﬁability, IEEE Transactions on Computers 48 (1999) 506–521.  C. Barrett, C. Tinelli, Satisﬁability modulo theories, Springer, pp. 305–343.  W. R. Adrion, M. A. Branstad, J. C. Cherniavsky, Validation, veriﬁcation, and testing of computer software, ACM Computing Surveys (CSUR) 14 (1982) 159–192.  Protecting Against Common Cause Failures in Digital I&C Systems of Nuclear Power Plants, number NP-T-1.5 in Nuclear Energy Series, International Atomic Energy Agency, Vienna, 2009.  H. J. Vishnukumar, B. Butting, C. Muller, E. Sax, Machine learning and deep neural network - artiﬁcial intelligence core for lab and real- world test and validation for adas and autonomous vehicles: Ai for ecient and quality test and validation, in: 2017 Intelligent Systems Conference, pp. 714–21.  R. Ashmore, M. Hill, boxing clever: Practical techniques for gaining insights into training data and monitoring distribution shift, in: International Conference on Computer Safety, Reliability, and Security, Springer, 2018, pp. 393–405.  N. Jouppi, Google supercharges machine learning tasks with tpu custom chip, https://cloud.google.com/blog/products/gcp/ google-supercharges-machine-learning-tasks-with-custom-chip, 2017. Accessed: 2018-08-25.  NVIDIA, Partner innovation:accelerating automotive breakthroughs, https://www.nvidia.com/en-us/self-driving-cars/ partners/, 2018. Accessed:2018-12-19.  WAYMO, Waymo Safety Report : On the Road to Fully Self-Driving, Technical Report, 2017. https://www.bbc.com/news/ technology-35692845.  GoogleCloud, Google Infrastructure Security Design Overview, Technical Report, 2017. https://cloud.google.com/security/ infrastructure/design/.  GM, Self-driving safety report, Technical Report, 2018. https://www.gm.com/our-stories/self-driving-cars.html.  S. Shalev-Shwartz, S. Shammah, A. Shashua, On a Formal Model of Safe and Scalable Self-driving Cars, arXiv e-prints (2017) arXiv:1708.06374.  Mobileye, Mobileye:sensing the future, https://www.mobileye.com/, 2018. Accessed:2018-12-19.  Tesla, Tesla vehicle safety report, https://www.tesla.com/VehicleSafetyReport, 2018. Accessed: 2019-11-01.  Tesla, Your tesla is learning to drive by itself, https://evannex.com/blogs/news, 2019. Accessed: 2019-11-01.  A. Adadi, M. Berrada, Peeking inside the black-box: A survey on explainable artiﬁcial intelligence (xai), IEEE Access 6 (2018) 52138–  F. M. Hohman, M. Kahng, R. Pienta, D. H. Chau, Visual analytics in deep learning: An interrogative survey for the next frontiers, IEEE Transactions on Visualization and Computer Graphics (2018) 1–1.  S. Khan, D. Paul, P. Momtahan, M. Aloqaily, Artiﬁcial intelligence framework for smart city microgrids: State of the art, challenges, and opportunities, Institute of Electrical and Electronics Engineers Inc., 2018, pp. 283–288.  A. Arpteg, B. Brinne, L. Crnkovic-Friis, J. Bosch, Software engineering challenges of deep learning, in: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50–59.  K. Scheibler, L. Winterer, R. Wimmer, B. Becker, Towards veriﬁcation of artiﬁcial neural networks, pp. 30–40.  G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochenderfer, Towards proving the adversarial robustness of deep neural networks, arXiv preprint arXiv:1709.02802 (2017).  L. Kuper, G. Katz, J. Gottschlich, K. Julian, C. Barrett, M. Kochenderfer, Toward scalable veriﬁcation for safety-critical deep networks, arXiv preprint arXiv:1801.05950 (2018). 28 Appendix A. Selected studies (sorted based on publication year) S ID Author(s) Year Title Publication Venue  Pulina, L. and A. Tacchella 2011 NeVer: a tool for artiﬁcial neural networks veriﬁca- Annals of Mathematics and Artiﬁ- tion cial Intelligence  Pulina, L. and A. Tacchella 2012 Challenging SMT solvers to verify neural networks AI Communications  Simonyan, K., A. Vedaldi 2013 Deep inside convolutional networks: Visualising im- arXiv preprint and A. Zisserman age classiﬁcation models and saliency maps  Szegedy, C., W. Zaremba, 2013 Intriguing properties of neural networks arXiv preprint I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow and R. Fergus  Goodfellow, I. J., J. Shlens 2014 Explaining and Harnessing Adversarial Examples International Conference on Learn- and C. Szegedy ing Representations (ICLR)  Gu, S. and L. Rigazio 2014 Towards deep neural network architectures robust to International Conference on Learn- adversarial examples ing Representations (ICLR)  Zeiler, M. D. and R. Fer- 2014 Visualizing and understanding convolutional net- European conference on computer gus works vision  Zhang, Q., T. Wang, Y. 2015 ApproxANN: an approximate computing framework Design, Automation & Test in Eu- Tian, F. Yuan and Q. Xu for artiﬁcial neural network rope Conference & Exhibition  Che, Z., S. Purushotham, 2015 Distilling knowledge from deep networks with appli- arXiv preprint R. Khemani and Y. Liu cations to healthcare domain  Hinton, G., O. Vinyals and 2015 Distilling the knowledge in a neural network arXiv preprint J. Dean  Nguyen, A., J. Yosinski 2015 Deep neural networks are easily fooled: High conﬁ- IEEE Conference on Computer and J. Clune dence predictions for unrecognizable images Vision and Pattern Recognition (CVPR)  Bach, S., A. Binder, G. 2015 On pixel-wise explanations for non-linear classiﬁer PloS one Montavon, F. Klauschen, decisions by layer-wise relevance propagation K.-R. Mller and W. Samek  Scheibler, K., L. Winterer, 2015 Towards Veriﬁcation of Artiﬁcial Neural Networks Workshop on Methods and De- R. Wimmer and B. Becker scription Languages for Modeling and Veriﬁcation of Circuits and Systems (MBMV)  Shaham, U., Y. Yamada 2015 Understanding adversarial training: Increasing local arXiv preprint and S. Negahban stability of neural nets through robust optimization  Mahendran, A. and A. 2015 Understanding deep image representations by invert- IEEE conference on computer vi- Vedaldi ing them sion and pattern recognition  Bach, S., A. Binder, K.-R. 2016 Controlling explanatory heatmap resolution and se- IEEE International Conference on Mller and W. Samek mantics via decomposition depth Image Processing (ICIP)  Papernot, N., P. McDaniel, 2016 Distillation as a defense to adversarial perturbations IEEE Symposium on Security & X. Wu, S. Jha and A. against deep neural networks Privacy Swami  Zheng, S., Y. Song, T. Le- 2016 Improving the robustness of deep neural networks via IEEE conference on computer vi- ung and I. Goodfellow stability training sion and pattern recognition  Daftry, S., S. Zeng, J. A. 2016 Introspective perception: Learning to predict failures IEEE/RSJ International Conference Bagnell and M. Hebert in vision systems on Intelligent Robots and Systems (IROS)  Zhou, B., A. Khosla, A. 2016 Learning deep features for discriminative localization IEEE conference on computer vi- Lapedriza, A. Oliva and A. sion and pattern recognition Torralba  Bastani, O., Y. Ioannou, L. 2016 Measuring neural net robustness with constraints Advances in neural information Lampropoulos, D. Vytini- processing systems otis, A. Nori and A. Crim- inisi  Shrikumar, A., P. Green- 2016 Not just a black box: Interpretable deep learning by arXiv Preprint side, A. Shcherbina and A. propagating activation dierences Kundaje  Julian, K. D., J. Lopez, J. 2016 Policy compression for aircraft collision avoidance IEEE/AIAA international confer- S. Brush, M. P. Owen and systems ence on Digital Avionics Systems M. J. Kochenderfer Conference (DASC)  Nguyen, A., A. Dosovit- 2016 Synthesizing the preferred inputs for neurons in neu- Advances in Neural Information skiy, J. Yosinski, T. Brox ral networks via deep generator networks Processing Systems and J. Clune 29 S ID Author(s) Year Title Publication Venue  Thiagarajan, J. J., B. 2016 TreeView: Peeking into deep neural networks via arXiv preprint Kailkhura, P. Sattigeri and feature-space partitioning K. N. Ramamurthy  Li, G., K. Pattabiraman, 2016 Understanding error propagation in GPGPU applica- International Conference on High C.-Y. Cher and P. Bose tions Performance Computing, Network- ing, Storage and Analysis  Ribeiro, M. T., S. Singh 2016 Why should i trust you?: Explaining the predictions ACM SIGKDD International Con- and C. Guestrin of any classiﬁer ference on Knowledge Discovery and Data Mining  Sundararajan, M., A. Taly 2017 Axiomatic attribution for deep networks International Conference on Ma- and Q. Yan chine Learning  O’Kelly, M., H. Abbas and 2017 Computer-aided design for safe autonomous vehicles Resilience Week (RWS) R. Mangharam  Tommaso DreossiAlexan- 2017 Compositional Falsiﬁcation of Cyber-Physical Sys- NASA Formal Methods dre DonzSanjit A. Seshia tems with Machine Learning Components  Tian, Y., K. Pei, S. Jana 2017 DeepTest: Automated testing of deep-neural- arXiv preprint and B. Ray network-driven autonomous cars  Reuben, F., R. R. Curtin, 2017 Detecting Adversarial Samples from Artifacts arXiv preprint S. Saurabh and A. B. Gardner  Frosst, N. and G. Hinton 2017 Distilling a Neural Network Into a Soft Decision Tree arXiv preprint  Pei, K., Y. Cao, J. Yang 2017 DeepXplore: Automated Whitebox Testing of Deep ACM Symposium on Operating and S. Jana Learning Systems Systems Principles (SOSP)  Gopinath, D., G. Katz, C. 2017 Deepsafe: A data-driven approach for checking ad- arXiv preprint S. Pasareanu and C. Bar- versarial robustness in neural networks rett  Montavon, G., S. La- 2017 Explaining nonlinear classiﬁcation decisions with Pattern Recognition puschkin, A. Binder, W. deep Taylor decomposition Samek and K.-R. Mller  Santos, F. F. d., L. 2017 Evaluation and Mitigation of Soft-Errors in Neural IEEE/IFIP International Confer- Draghetti, L. Weigel, L. Network-Based Object Detection in Three GPU Ar- ence on Dependable Systems and Carro, P. Navaux and P. chitectures Networks Workshops (DSN-W) Rech  Papernot, N. and P. Mc- 2017 Extending defensive distillation arXiv preprint Daniel  Ehlers, R. 2017 Formal veriﬁcation of piece-wise linear feed-forward International Symposium on Auto- neural networks mated Technology for Veriﬁcation and Analysis  Manikandasriram, S. R., 2017 Failing to learn: autonomously identifying perception arXiv preprint C. Anderson, R. Vasude- failures for self-driving cars van and M. Johnson- Roberson  Xu, W., D. Evans and Y. Qi 2017 Feature squeezing: Detecting adversarial examples in Network and Distributed Systems deep neural networks Security Symposium (NDSS)  Dong, Y., H. Su, J. Zhu 2017 Improving interpretability of deep neural networks IEEE Conference on Computer Vi- and B. Zhang with semantic information sion and Pattern Recognition  Bastani, O., C. Kim and H. 2017 Interpretability via model extraction arXiv preprint Bastani  Fong, R. C. and A. Vedaldi 2017 Interpretable explanations of black boxes by mean- IEEE International Conference on ingful perturbation Computer Vision  Melis, M., A. Demontis, 2017 Is Deep Learning Safe for Robot Vision? Adversarial IEEE International Conference on B. Biggio, G. Brown, G. Examples Against the iCub Humanoid Computer Vision Workshops (IC- Fumera and F. Roli CVW)  Vishnukumar, H. J., B. 2017 Machine learning and deep neural network - artiﬁcial Intelligent Systems Conference (In- Butting, C. Muller and E. intelligence core for lab and real-world test and val- telliSys) Sax idation for ADAS and autonomous vehicles: AI for ecient and quality test and validation  Mhamdi, E. M. E., R. 2017 On the Robustness of a Neural Network IEEE Symposium on Reliable Dis- Guerraoui and S. Rouault tributed Systems (SRDS)  Metzen, J. H., T. Ge- 2017 On detecting adversarial perturbations International Conference on Learn- newein, V. Fischer and B. ing Representations (ICLR) Bischo 30 S ID Author(s) Year Title Publication Venue  Dutta, S., S. Jha, S. 2017 Output range analysis for deep neural networks arXiv preprint Sanakaranarayanan and A. Tiwari  Cisse, M., P. Bojanowski, 2017 Parseval networks: Improving robustness to adversar- arXiv preprint E. Grave, Y. Dauphin and ial examples N. Usunier  Xiang, W., H.-D. Tran and 2017 Reachable set computation and safety veriﬁcation for arXiv preprint T. T. Johnson neural networks with ReLU activations  Katz, G., C. Barrett, D. L. 2017 Reluplex: An ecient SMT solver for verifying deep International Conference on Com- Dill, K. Julian and M. J. neural networks puter Aided Veriﬁcation (CAV) Kochenderfer  Dabkowski, P. and Y. Gal 2017 Real time image saliency for black box classiﬁers Advances in Neural Information Processing Systems (NIPS)  Ross, A. S., M. C. Hughes 2017 Right for the right reasons: Training dierentiable arXiv preprint and F. Doshi-Velez models by constraining their explanations  Vialatte, J.-C. and F. 2017 A Study of Deep Learning Robustness Against Com- arXiv preprint Leduc-Primeau putation Failures  Santoro, A., D. Raposo, 2017 A simple neural network module for relational rea- Advances in Neural Information D. G. Barrett, M. Ma- soning Processing Systems (NIPS) linowski, R. Pascanu, P. Battaglia and T. Lillicrap  Huang, X. W., M. 2017 Safety Veriﬁcation of Deep Neural Networks International Conference on Com- Kwiatkowska, S. Wang puter Aided Veriﬁcation and M. Wu  Smilkov, Daniel and Tho- 2017 Smoothgrad: removing noise by adding noise arXiv preprint rat, Nikhil and Kim, Been and Vigas, Fernanda and Wattenberg, Martin  Carlini, N. and D. Wagner 2017 Towards Evaluating the Robustness of Neural Net- IEEE Symposium on Security and works Privacy (SP)  Katz, G., C. Barrett, D. L. 2017 Towards proving the adversarial robustness of deep arXiv Preprint Dill, K. Julian and M. J. neural networks Kochenderfer  Li, G., S. K. S. Hari, M. 2017 Understanding error propagation in deep learning International Conference for High Sullivan, T. Tsai, K. Pat- neural network (DNN) accelerators and applications Performance Computing, Network- tabiraman, J. Emer and S. ing, Storage and Analysis W. Keckler  Lundberg, S. M. and S.-I. 2017 A uniﬁed approach to interpreting model predictions Advances in Neural Information Lee Processing Systems (NIPS)  Narodytska, N., S. P. Ka- 2017 Verifying properties of binarized deep neural net- arXiv preprint siviswanathan, L. Ryzhyk, works M. Sagiv and T. Walsh  Raj, S., S. K. Jha, A. Ra- 2017 Work-in-progress: testing autonomous cyber- International Conference on Em- manathan and L. L. Pul- physical systems using fuzzing features from bedded Software (EMSOFT) lum convolutional neural networks  Schorn, C., A. Guntoro 2018 Accurate neuron resilience prediction for a ﬂexible re- Design, Automation & Test in and G. Ascheid liability management in neural network accelerators Europe Conference & Exhibition (DATE)  Ribeiro, M. T., S. Singh 2018 Anchors: High-precision model-agnostic explana- AAAI Conference on Artiﬁcial In- and C. Guestrin tions telligence  Ma, L., F. Juefei-Xu, F. 2018 DeepGauge: multi-granularity testing criteria for ACM/IEEE International Confer- Zhang, J. Sun, M. Xue, B. deep learning systems ence on Automated Software Engi- Li, C. Chen, T. Su, L. Li neering and Y. Liu  Zhang, M., Y. Zhang, L. 2018 DeepRoad: GAN-based metamorphic testing and in- ACM/IEEE International Confer- Zhang, C. Liu and S. put validation framework for autonomous driving sys- ence on Automated Software Engi- Khurshid tems. neering  Guo, J., Y. Jiang, Y. Zhao, 2018 DLFuzz: dierential fuzzing testing of deep learning ACM Joint Meeting on European Q. Chen and J. Sun systems Software Engineering Conference and Symposium on the Foundations of Software Engineering  Rubaiyat, A. H. M., 2018 Experimental Resilience Assessment of An Open- arXiv preprint Q. Yongming and H. Source Driving Agent Alemzadeh 31 S ID Author(s) Year Title Publication Venue  Rhazali, K., B. Lussier, W. 2018 Fault Tolerant Deep Neural Networks for Detection IFAC-PapersOnLine Schn and S. Geronimi of Unrecognizable Situations  Wicker, M., X. Huang and 2018 Feature-guided black-box safety testing of deep neu- International Conference on Tools M. Kwiatkowska ral networks and Algorithms for the Construc- tion and Analysis of Systems (TACAS)  Linsley, D., D. Scheibler, 2018 Global-and-local attention networks for visual recog- arXiv preprint S. Eberhardt and T. Serre nition  Wu, M., M. Wicker, W. 2018 A Game-Based Approximate Veriﬁcation of Deep arXiv preprint Ruan, X. Huang and M. Neural Networks with Provable Guarantees Kwiatkowska  Xu, K., D. H. Park, C. Yi 2018 Interpreting Deep Classiﬁer by Visual Distillation of arXiv preprint and C. Sutton Dark Knowledge  Mallozzi, P., P. Pelliccione 2018 Keeping intelligence under control. International Workshop on Soft- and C. Menghi ware Engineering for Cognitive Services  Guidotti, R., A. Monreale, 2018 Local rule-based explanations of black box decision arXiv preprint S. Ruggieri, D. Pedreschi, systems F. Turini and F. Giannotti  Guo, W., D. Mu, J. Xu, P. 2018 LEMNA: Explaining Deep Learning based Security ACM SIGSAC Conference on Su, G. Wang and X. Xing Applications Computer and Communications Security  Tan, S., R. Caruana, G. 2018 Learning Global Additive Explanations for Neural arXiv preprint Hooker, P. Koch and A. Nets Using Model Distillation Gordo  Dumitru, M. A. K.-R. M., 2018 Learning how to explain neural networks: Patternnet International Conference on Learn- E. B. K. S. D. Pieter, and patternattribution ing Representations (ICLR) J. Kindermans and K. T. Schtt  Xiang, W., H. D. Tran and 2018 Output Reachable Set Estimation and Veriﬁcation for IEEE Transactions on Neural Net- T. T. Johnson Multilayer Neural Networks works and Learning Systems  Kuper, L., G. Katz, J. 2018 Toward scalable veriﬁcation for safety-critical deep arXiv preprint Gottschlich, K. Julian, C. networks Barrett and M. Kochender- fer  Cheng, C.-H., G. Nhren- 2018 Veriﬁcation of binarized neural networks arXiv preprint berg and H. Ruess
Electrical Engineering and Systems Science – arXiv (Cornell University)
Published: Oct 5, 2019
Access the full text.
Sign up today, get DeepDyve free for 14 days.