Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Yiming Xu; Yajie Zou; Jian Sun

doi:10.1108/jicv-01-2018-0002

Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Xu, Yiming; Zou, Yajie; Sun, Jian 2018-10-02 00:00:00 Purpose – It would take billions of miles’ ﬁeld road testing to demonstrate that the safety of automated vehicle is statistically signiﬁcantly higher than the safety of human driving because that the accident of vehicle is rare event. Design/methodology/approach – This paper proposes an accelerated testing method for automated vehicles safety evaluation based on improved importance sampling (IS) techniques. Taking the typical cut-in scenario as example, the proposed method extracts the critical variables of the scenario. Then, the distributions of critical variables are statistically ﬁtted. The genetic algorithm is used to calculate the optimal IS parameters by solving an optimization problem. Considering the error of distribution ﬁtting, the result is modiﬁed so that it can accurately reveal the safety beneﬁts of automated vehicles in the real world. Findings – Based on the naturalistic driving data in Shanghai, the proposed method is validated by simulation. The result shows that compared with the existing methods, the proposed method improves the test efﬁciency by 35 per cent, and the accuracy of accelerated test result is increased by 23 per cent. Originality/value – This paper has three contributions. First, the genetic algorithm is used to calculate IS parameters, which improves the efﬁciency of test. Second, the result of test is modiﬁed by the error correction parameter, which improves the accuracy of test result. Third, typical high-risk cut-in scenarios in China are analyzed, and the proposed method is validated by simulation. Keywords Genetic algorithm, Simulation, Automated vehicles, Importance sampling, Lane changing, Safety evaluation, High-risk scenarios Paper type Research paper the test time required is still very long without acceleration 1. Introduction because of the huge scenarios and long driving-testing distance. According to trafﬁc accident statistics reports, human factor is However, the existing studies pay more attention to the the main cause of trafﬁc accidents. It is estimated that over 90 construction of scenario libraries (e.g. Pegasus [Federal per cent accidents in motor crashes are because of drivers’ error Ministry for Economic Affairs and Energy (BMWi), 2016], (National Highway Trafﬁc Safety Administration, 2015). enableS3 (The Enable-S3 Consortium, 2016), etc.) and test Therefore, the automated vehicle (AV) is considered as the key tools development (e.g. software-in-loop (Russo et al.,2007), technique to improve trafﬁc safety. When an AV drives in real- hardware-in-loop (Gietelink et al.,2006) and vehicle in the world roads, it needs to deal with various and complicated loop (Berg et al., 2016)). The studies of accelerated testing trafﬁc conditions. Therefore, AVs must be tested before they method are neglected. Therefore, the accelerated method of can be permitted to travel on the road, otherwise the AVs will be automated loading of the re-sampled driving scenarios athreat to other trafﬁc participants. However, according to the study by Kalra and Paddock (2016), it would take approximately 5 billion miles to demonstrate that the safety of AV is statistically signiﬁcantly higher than the safety of human © Yiming Xu, Yajie Zou and Jian Sun. Published in Journal of Intelligent driving because of the very small probability of human driving and Connected Vehicles. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY accidents. And with a ﬂeet of 100 AVs being ﬁeld test-driven 4.0) licence. Anyone may reproduce, distribute, translate and create 24 h a day, 365 days a year at an average speed of 25 miles per derivative works of this article (for both commercial and non- hour, this would take about 225 years. Even by simulation, commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode The current issue and full text archive of this journal is available on The authors would like to thank the Natural Science Foundation of China Emerald Insight at: www.emeraldinsight.com/2399-9802.htm (U1764261, 51422812), and the Shanghai Science and technology project of international cooperation (16510711400) for supporting this research. Received 25 January 2018 Journal of Intelligent and Connected Vehicles Revised 20 March 2018 1/1 (2018) 28–38 2 May 2018 Emerald Publishing Limited [ISSN 2399-9802] [DOI 10.1108/JICV-01-2018-0002] Accepted 9 May 2018 28 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 should be concerned so that the safety beneﬁts of AVs can be distribution. With the advent of naturalistic driving projects effectively evaluated. This scenario acceleration method can around the world in recent years, some studies began to build be applied to simulation test and provide scenarios for stochastic models based on naturalistic driving big data and hardware-in-loop test, driving simulator test and controlled carry out Monte Carlo simulations to evaluate AVs. Yang and test bed (e.g. Mcity). Peng (2010); Lee (2004) and Woodrooffe et al. (2014) all This paper proposes an accelerated testing method for AVs evaluated the collision avoidance systems of vehicles by (Level 2 to Level 5) safety evaluation based on improved Monte Carlo simulation based on naturalistic driving big data. The advantage of Monte Carlo simulation is stochastic importance sampling (IS) techniques. This paper uses the scenarios. However, in the simulation of rare events (such as occurrence of high-risk event, such as conﬂict, crash and injury, collision), the number of tests required will be very large to evaluate safety of AV. The high-risk events are identiﬁed by because of randomness. indirect indicators such as Time to Collision (TTC) and Test matrix method is to predeﬁne a “test matrix” consisting headway. Taking the typical cut-in scenarios as example, the proposed method extracts the critical variables of the scenario. of a variety of typical scenarios based on past crash data and expert knowledge. Then the test matrix is used to test some Based on IS techniques, the scenarios with higher probability of properties (such as safety beneﬁt) of AVs. The test matrix occurrence of high-risk events are reconstructed to test the method is the basis of many test studies, such as autonomous AVs. Theresultoftestismodiﬁed by the error correction emergency braking (AEB) protocol (Euro, N.C.A.P., 2013), parameter which is calibrated by the empirical data, so that CAMP (Deering, 2002), HASTE (Carsten et al., 2005), AIDE the safety beneﬁts of AVs in the real world can be revealed. (Kussmann et al.,2004), TRACE (Karabatsou et al.,2007), Finally, based on the naturalistic driving data in Shanghai, APROSYS (Wohllebe et al.,2004) and ASSESS (Bühne et al., the proposed method is validated by simulation. Besides the 2012). The advantage of the test matrix method is efﬁcient, IS accelerated testing method, the detailed contributions of credible and repeatable. Applying test matrix method to this paper are as follows. First, the genetic algorithm (GA) is evaluate low-level AVs is straightforward and it might used to calculate the optimal IS parameters by solving an continue to be the selected approach in the near future optimization problem, which improves the efﬁciency of (Zhao et al., 2017a). However, test scenarios in the test accelerated testing. Second, based on the empirical data, the matrix are predeﬁned and ﬁxed. The scenarios predeﬁned result of test is modiﬁed by the error correction parameter, based on crash data cannot test the AVs comprehensively. which solves the problem in existing studies that the conﬂict And the test cannot reveal the properties of AVs in real- rate in accelerated testing result is inconsistent with the world conditions, especially in complicated mixed trafﬁc conﬂict rate calculated by the empirical data. Third, based ﬂows. on the naturalistic driving data in Shanghai, typical high-risk WCSE methodology evaluates the control system of a vehicle cut-in scenarios in China are analyzed, and the proposed by generating worst scenario (Ma and Huei, 1999; Kou, 2010). method is validated by simulation. WCSE can be expressed as an optimization problem that The rest of this paper is organized as follows. Section 2 searches for the worst scenario. The scenario is quantitatively presents the literature review of existing methods of AVs evaluated by a cost function, and the goal of WCSE is to search testing. Section 3 introduces the proposed accelerated testing for the scenario with the largest cost. WCSE can identify the method for AVs safety evaluation in detail. Section 4 veriﬁes the weakness in the control system of an AV. But WCSE does not proposed method by simulation based on naturalistic driving correlate the worst scenario with real-world scenarios, and the data in Shanghai. Section 5 presents the conclusion and future probability of occurrence of worst-case scenario in real world research needs. cannot be identiﬁed. Accelerated evaluation is a data (such as naturalistic 2. Literature review driving data)-based testing method. High-risk scenarios are The test of AV requires a combination of test tools and test more efﬁcient than normal scenarios in AV testing. methods. Test tools provide facilities and test methods provide However, high-risk scenarios are rare events in naturalistic theoretical guidance for the test. This paper focuses on test driving data. Accelerated evaluation method modiﬁes the methods because of the larger potential for accelerated testing distribution of real-world data so that the high-risk scenarios and their theory signiﬁcance. In existing studies, there are four have a higher probability of occurrence. Zhao et al. (2015) main methods for AVs testing: Monte Carlo simulation, test extracted the car-following scenarios in naturalistic driving matrix, worst-case scenario evaluation (WCSE) and accelerated data and used models to ﬁt the critical variables in the evaluation. The above four methods are reviewed in this section. scenarios. The most frequent scenarios are deleted to Monte Carlo simulation is a stochastic method. The AVs are increase the overall exposure rate for critical scenarios. tested in the scenarios generated stochastically based on a However, this method cannot evaluate the safety beneﬁts of certain distribution. Touran et al. (1999) evaluated the safety of AVs in real world. Considering this problem, Zhao et al. autonomous intelligent cruise control model by Monte Carlo (2017a, 2017b) applied the IS technique to AVs testing and simulation. The values of some parameters in the model were studied the cut-in and car-following scenarios. The obtained randomly based on a certain distribution. Althoff and distribution of critical variables was modiﬁed to increase the Mergel (2011) evaluated the collision risk for autonomous probability of occurrence of high-risk scenarios and reduce vehicles when executing a planned maneuver by Monte Carlo the number of required tests. Then, the IS technique was simulation. The initial states in simulation were generated used to modify the test result so that the result can reveal the randomly according to a piecewise constant probability safety beneﬁts of AVs in real world. The most important 29 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 advantage of accelerated evaluation is that it can describe random sample in X. Let I (x) be the indicator function of the real-world beneﬁts (such as crash rate) of AVs. And the high-risk event «.I (x)isdeﬁned as: method also improves the efﬁciency of testing. But in 1; if x « existing studies, the error in critical variable distribution I ðÞ x ¼ (1) ﬁtting is ignored, which leads to the ﬁnal result deviating 0; otherwise from actual. This phenomenon will be discussed in detail in Section 4.4.3. Let g denote the probability P(«). g can be estimated via In summary, the test matrix method and WCSE method simulation by generating independent samples (x , x ,.. ., x ). 1 2 n can hardly reveal the probability of AVs being exposed to ^ ^ Let g denote an estimator of g. g can be calculated by: n n risk in real-world scenarios, and the Monte Carlo simulation is inefﬁcient. The accelerated evaluation method can reveal the safety beneﬁts of AVs in real world and has the advantage g ^ ¼ I x (2) «ðÞ i of high testing efﬁciency. However, existing studies i¼1 overlooked the ﬁtting errors of critical variables, resulting in the deviation of test result from the ﬁeld operation. To solve According to the law of large numbers, g ^ ! g as n !1, that this problem, this paper proposes an accelerated testing is, when n is large enough, g ^ converges to g. method for AVs safety evaluation based on improved IS To ensure the reliability of g ^ , the central limit theorem techniques. Considering the ﬁtting error of critical variables, proves useful in developing a conﬁdence interval (CI) for the GA is used to calculate the optimal IS parameters to estimate and is used to determine the necessary n for obtain a better acceleration efﬁciency. And the error accurate estimation. For a sufﬁciently large n, the variance correction parameter is used to correct the test result to of g ^ is: make the result consistent with the result calculated by 0 1 empirical data. X @ A s ðÞ g ^ ¼ Var I x «ðÞ i i¼1 3. Methodology In real-world data, high-risk events of vehicles belong to rare (3) ¼ Var I x «ðÞ i events, and a reliable evaluation of the probability of this 2 i¼1 event requires a large number of tests. The proposed gðÞ 1 g accelerated testing method based on improved IS techniques can reduce the number of tests and obtain a reliable test result. The proposed method can be divided With a conﬁdence level at 100(1 – a) per cent, the CI of g ^ is: into four steps (Figure 1). First, basedonthe real-world data, the scenarios to be analyzed are extracted, and the g ^ z sðÞ g ^ ; g ^ 1 z sðÞ g ^ (4) n a=2 n n a=2 n critical variables of these scenarios are deﬁned and obtained. Second, based on the extracted scenarios and variables, the where, z is deﬁned as: a/2 optimal IS parameters are calculated to generate accelerated scenarios, and the IS technique and simulation are used to 1 z ¼ UðÞ 1 a=2 (5) a=2 calculate the safety beneﬁts of AVs. Third, the error correction parameter is calibrated by the real-world data and 1 where, U is the inverse cumulative distribution function of safety beneﬁts. Finally, the test result is corrected by the normal distribution N(0,1). error correction parameter, and the ﬁnal safety beneﬁts of The half-width of CI is: AVs in real world are obtained. We will discuss this framework in the following sections in detail. l ¼ z s g ^ (6) ðÞ a=2 a=2 n 3.1 Probability of high-risk-event As the value of g ^ is small, the relative half-width l is Consider a sample space X with a probability measure P. Let used to indicate the accuracy of the estimation. l is P(«) denote the probability of a high-risk event. Let x be a deﬁned as: Figure 1 Framework of proposed accelerated testing method 30 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 l ¼ l =g (7) 2 2 r a=2 z ðÞ ðÞ E I x L x f a=2 n 1 (13) 2 2 g b To ensure that l is smaller than a constant b we need: sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ Therefore, to obtain a reliable result through a small number of l z sðÞ g ^ 1 g a=2 a=2 tests, the density function f (x) needs to be properly chosen to l ¼ ¼ ¼ z b (8) r a=2 g g gn 2 2 2 ðÞ ðÞ make E I x L x close to g . That is: 3.3 High-risk scenarios: cut-in events The proposed method can be used in a variety of scenarios such 1 g a=2 as lane changing scenarios, car-following scenarios and crossing n (9) g b scenarios. In the typical trafﬁc environment, the cut-in scenario occurs more frequently and has greater risk on human driving. The probability of high-risk events in real-world driving is Therefore, this study focuses on the cut-in scenario which very small. Therefore, the sample size n needs to be huge to refers to the situation that other vehicles move into the lane ensure the reliability of the estimation. This means that a where AV located from an adjacent lane in front of the AV. huge number of tests are required to evaluate the probability The critical variables of cut-in scenario are identiﬁed as: the velocity of lane changing vehicle (LCV) v , the range R deﬁned of high-risk events in AV driving if using the Crude Monte Carlo method. as the distance between the rear edge of the LCV and the front edge of AV, and TTC. The TTC is deﬁned as: 3.2 Importance sampling IS is one of the classical variance reduction techniques for TTC ¼ (14) increasing the efﬁciency of Monte Carlo algorithms (Glynn and Iglehart, 1989). The IS has been successfully used to evaluate where, R is the range, R is the derivative of R. reliability (Heidelberger, 1995) and critical events in ﬁnance During the driving process, the high-risk events tend to (Glasserman and Li, 2005), insurance (Asmussen and Albrecher, correspond to small R and TTC. Smaller R and TTC indicate 2010) and telecommunication networks (Chang et al.,1994). that the event is rarer and less safe. Therefore, the reciprocal of General overviews about IS can be found in Bucklew (2013) and R and TTC are used to put the rare events in the tail of the Blanchet and Lam (2012). The basic idea of IS used in distribution. The Pareto distribution is used to ﬁt R and the evaluation of AVs is to replace the original distribution density Exponential distribution is used to ﬁt TTC . The distribution function f(x)bya newone f (x) to generate event x,which leads of v is not ﬁtted and the empirical distribution is directly used to a higher probability of occurrence of high-risk events. And to generate events. then the risk calculation function is modiﬁed to obtain the safety 1 The density function of the distribution of R can be beneﬁts of AVs. expressed as: Let x be the event generated by f (x), the estimator g ^ can be expressed as: 1 1 1 x u R f 1ðÞ xjk 1; s 1; u 1 ¼ 11 k 1 R R R R R n s 1 s 1 X R R (15) g ^ ¼ I x Lx (10) n i i i¼1 where, k 1 is the shape parameter; s 1 is the scale parameter; R R u 1 is the threshold parameter. where,LxðÞ is the likelihood ratio (Radon–Nikodym derivative R The density function of the distribution of TTC can be (Royden and Fitzpatrick, 1988)) deﬁned as: expressed as: fx ðÞ Lx ¼ (11) i 1 x=l f ðÞ x 1 TTC i 1 1 f ðÞ xjl ¼ e (16) TTC TTC l 1 TTC The relative half-width of CI constructed by IS can be where, l 1 is the rate parameter. TTC expressed as: Therefore, every lane changing event can be expressed as: l z s g ^ ðÞ a=2 a=2 1 1 l ¼ ¼ x ¼ v ; R ; TTC (17) r i l g g qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ 2 2 z E ðÞ g ^ E ðÞ g ^ a=2 n n pﬃﬃﬃ (12) 3.4 Accelerated Evaluation g n sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ According to IS technique, a new distribution f is used to 2 2 ðÞ ðÞ z E I x L x a=2 f « replace f 1, and a new distribution f is used to replace TTC ¼ pﬃﬃﬃ 1 n g f 1.AsR obeys Pareto distribution, we need to TTC construct a new exponential distribution f 1 before To ensure l b, the necessary test number n is: replacing f 1 by f to reduce the computation complexity r R 31 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 1 x in later steps (Zhao et al., 2017b). f 1 has the smallest least f 1ðÞ x ¼ exp TTC square error to f 1. f 1 is deﬁned as: l 1 # 1 l 1 # 1 R R TTC TTC TTC TTC (30) 1 x f 1ðÞ x ¼ exp (18) l 1 l 1 R R f 1 m f 1 w ðÞ ðÞ R TTC 1 1 LR ¼ m; TTC ¼ w ¼ (31) ~ f m f w ðÞ ðÞ 1 1 Apply exponential change of measure to f 1 and f 1, we get R TTC TTC f and f : 1 1 R TTC In this optimization problem, the objective function is number 1 x of tests n, and the decision variables are IS parameters # 1 and f 1ðÞ xj# 1 ¼ exp l 1 # 1 l 1 # 1 # 1. In the constrains, I is the indicator function of high- R R R R TTC « risk event «; L(x ) is the likelihood; g is the probability of high- (19) risk events; z is given by equation (5); b is the threshold of a/2 relative half-width; x is a matrix of a series of samples x , x ,.. ., 1 2 1 x x,and x is given by equation (17); f 1ðÞ x is the density f ðÞ xj# 1 ¼ exp 1 i i R TTC TTC l 1 # 1 l 1 # 1 1 TTC TTC TTC TTC function of distribution of R ; f 1ðÞ x is the density function TTC (20) of distribution of TTC ; x is a matrix of a series of samples 1 1 x ; x ; .. . ; x and x ¼ v ; R ; TTC ; f x is the density ðÞ l 1 1 2 i i R Therefore, the likelihood is: function of distribution of R ; f ðÞ x is the density function TTC 1s of distribution of TTC . f 1ðÞ x f 1 y ðÞ R TTC 1 1 GA is oneofthe effectivemethods to solvethe optimization LR ¼ x; TTC ¼ y ¼ (21) f ðÞ x f y 1 1ðÞ R TTC problem (Goldberg, 1989; Michalewicz, 2013). GA is an adaptive global search algorithm, which has the advantages of short The probability of high-risk events is: calculation time and high robustness. GA is widely used in multi- objective optimization, industrial engineering, management PðÞ « ¼ EðÞ I ðÞ x ¼ EðÞ I ðÞ x LxðÞ (22) f « f « science, artiﬁcial intelligence and so on (Gen and Cheng, 2000). The optimization problem can be solved by GA to calculate the Therefore, proper f and f need to be constructed to 1 1 optimal IS parameter. With the assistant of GA tool in MATLAB, R TTC calculate P(«)efﬁciently. In other words, optimal IS parameters the optimization problem can be easily solved. # 1 and # 1 need to be obtained to get the reliable test result R TTC through the minimum number of tests. 3.6 Error correction 1 1 The error in distribution ﬁtting of R and TTC is inevitable during the accelerated testing. Through the comparison with 3.5 Searching for optimal importance sampling the empirical data, it is obvious that these ﬁtting errors will lead parameters with genetic algorithm to the ﬁnal result g ^ deviating from the actual probability of The efﬁciency of accelerated evaluation is closely related to the risk g. However, the existing studies only compare the IS parameters. To obtain the optimal IS parameters, consider accelerated testing result with Monte Carlo simulation result. The data used in these two methods strictly follow the well- the following optimization problem: ﬁtted distributions, and the error is overlooked. Fortunately, 2 2 ðÞ ðÞ E I x L x f a=2 this study found this issue by comparing the result with the min n ¼ 1 (23) 2 2 g b actual data and corrected the error. The error is corrected by error correction parameter t,let: s:t g ¼ EðÞ I ðÞ x (24) f « g ¼ g ^ t (32) 1 1 where, g is the probability of risk calculated by empirical data; x ¼ v ; R ; TTC (25) g ^ is the estimator obtained by accelerated evaluation. The ﬁtting error can be reﬂected by the difference between mean of 1 x u 1 R the ﬁtted distributions and empirical distributions. Therefore, f 1ðÞ x ¼ 11 k 1 (26) R R s 1 s 1 R R t is deﬁned as: t ¼ k m m 1 k m m (33) 1 2 ðÞ R1 R0 ðÞ TTC1 TTC0 x=l TTC f 1ðÞ x ¼ e (27) TTC l 1 TTC 1 where, m is the mean of ﬁtted R distribution; m is the R1 R0 mean of empirical R distribution; m is the mean of ﬁtted TTC1 1 1 1 x ¼ v ; R ; TTC (28) TTC distribution; m is the mean of empirical TTC l TTC0 distribution; k and k parameters need to be calibrated. 1 2 The signiﬁcance of error correction parameter is to modify 1 x f x ¼ exp (29) 1ðÞ the test result to make it closer to empirical result rather than l 1 # 1 l 1 # 1 R R R R the result of Monte Carlo simulation. The revised result can 32 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 better reveal the safety beneﬁts of AVs in real world, and it is Figure 2 The trajectories of naturalistic driving vehicles more persuasive. 3.7 Algorithm of the proposed method The execution algorithm of the proposed method is as follows: Execution Algorithm of the Proposed Method Step 1 Critical Variables Extraction Step 1.1: Input the real-world data X. Extract scenario W to be analyzed. Take cut-in scenario as example. Step 1.2: Input the scenarios W.Deﬁne the critical variables j , j ,.. .,j . Extract j ,j ,.. .,j . from W. The critical 1 2 i 1 2 i variables of cut-in scenario are v , R and TTC. Step 2 Accelerated Distribution Generating Step 2.1: Input the critical variables R and TTC. Fit the distribution of R based on equation (15). Fit the distribution of TTC based on equation (16). Figure 3 Data collected by data acquisition system Step 2.2: Input the distribution of R . Generate f 1 based on equation (18). Step 2.3: Input the distribution f 1 and f 1. Generate the R TTC optimal IS parameters # 1 and # 1 based on equations R TTC (23)-(31). Step 2.4: Input the parameters # 1 and # 1. Generate R TTC accelerated distribution f and f based on equations 1 1 R TTC (19)-(20). Step 3 Error Correction Parameter Calibration Input the real-world data X . Where X is a subset ofX. 1 1 Input the accelerated distribution f and f . Generate 1 1 R TTC parameter k and k based on equations (32) and (33). 1 2 Step 4 Test of AVs Step 4.1: Input the accelerated distribution f and f , 1 1 R TTC and the empirical distribution of v . Generate the accelerated scenarios xi based on equation (17). Step 4.2: Input accelerated scenarios xi, and the parameter k and k . Calculate the test result based on equations (21) (22) (32) (33). of lane changing moment are extracted. The variables include 4. Simulation v , v and R (Figure 4). Where v is the velocity of LCV, v is the l l velocity of evaluated vehicle and R is the range deﬁned as the 4.1 Data distance between the rear edge of the LCV and the front edge of The data used in this research are from Shanghai Naturalistic evaluated vehicle. In Figure 4, the evaluated vehicle is AV, and Driving Research project. The project is the ﬁrst naturalistic the LCV can be AV or manual-driving vehicle. driving data collection project using real vehicles and high- For comparison analysis, the same criteria as Zhao et al. precision equipment in China. The project aims to collect real- (2017b) were applied: v [ (2m/s,40m/s); v [ (2m/s,40m/s); and world trafﬁc data and study the behavioral characteristics of R [ (0.1m,75m). Finally, 32,104 cut-in scenarios were extracted Chinese drivers. It recorded over 500,000 km naturalistic from the data. driving data from December 2012 to December 2015. The driving trajectories of vehicles basically covered the main roads 4.2 Evaluated models in Shanghai (Figure 2). The vehicles are equipped with For comparison analysis, the same AV model and parameters Mobileye vehicle active safety system and SHARP2 NextGen as that of Zhao et al. (2014b) were used in the simulation. The data acquisition system. The data collected include AV is equipped with adaptive cruise control (ACC) and AEB information such as vehicle position, velocity and distance to surrounding vehicles (Figure 3). The simulation takes the cut-in scenario as an example to Figure 4 Extracted variables of cut-in scenario validate the proposed method. Therefore, the cut-in scenarios are extracted from the naturalistic driving data. The cut-in scenario refers to the situation that other vehicles move into the lane where AV located from an adjacent lane in front of the AV. The moment when the LCV crosses the lane line is deﬁned as lane changing moment. Then, the variables of cut-in scenario 33 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 system. When TTC TTC , the AV is controlled by ACC; Table I Parameters in evaluated models AEB and when TTC < TTC , the AV is controlled by AEB. AEB Parameter Value Unit Equation TTC is a function of vehicle velocity (Figure 5). AEB ACC 38.6 – (34) The ACC is approximated by a discrete proportional- K ACC ACC integral controller to achieve a desired time headway T 1.35 – (34) HW d i ACC (Ulsoy et al., 2012). The input of the controller is time headway T 2 s (35) HW Err T 0.1 s (34) error t , and the output is the command acceleration of the HW max 2 a 5 m/s (37) next time step which can be expressed as: ACC a 10 m/s (38) AEB ACC Err Err ðÞ ðÞ ðÞ ðÞ a k1 1 ¼ a k 1 K t k t k 1 d d r 16 m/s (38) AEB p HW HW T 0.5 s (38) ACC Err Err ðÞ ðÞ 1 K t k t k 1 T =2 (34) i HW HW s 0.0796 s – AV Err ACC t ¼ t T (35) HW HW HW Besides, the simulation analysis methods and conclusions of three kinds of events are both similar. Therefore, for simplicity, simulation analysis in this paper focuses on the conﬂict events. t ¼ R=v (36) HW A conﬂict event happens when an AV enters the proximity zone of the LCV during time t to t 1 T. The proximity zone is max ja j a (37) ACC deﬁned as the area in the adjacent lane from 1.2 m in front of the bumper of LCV to 9 m behind the rear bumper of LCV Err where, a is the command acceleration; t is the time HW (Lee et al.,2004). The deﬁnition of proximity zone is shown in ACC ACC headway error; K and K are constant gains; T is the Figure 6. p i ACC sampling time; T is the desired time headway; t is the HW HW max 4.4 Simulation result time headway; R is the range; v is the velocity of AV; and a is ACC Simulation can be used to evaluate the real-world safety beneﬁts the maximum acceleration. of AVs if the models are strictly calibrated by real-world data. In Once triggered, AEB aims to achieve an acceleration a . AEB this paper, the distributions of critical variables are ﬁtted based Let the triggered moment be 0, the AEB model can be on empirical data. The IS parameter is calculated based on a expressed as: subset of empirical data. The error correction parameter is calibrated by a subset of empirical data. The AEB system was 0; if t T > a extracted from a 2011 Volvo V60, based on a test conducted by ðÞ atðÞ ¼ r t T ; if t > T and aðÞ t a (38) AEB a a AEB > ADAC (Allgemeiner Deutscher Automobil-Club e.V.) a ; else AEB (Gorman, 2013). Therefore, the results of simulation can reveal the real-world safety beneﬁts of tested AV. where, a(t) is the acceleration of vehicle at t moment; r is the AEB 4.4.1 Distribution of critical variables derivative of acceleration; T is the action time; and a is the a AEB As is mentioned in Section 3.3, the critical variables of cut-in acceleration of emergency braking. scenario are the velocity of LCV v , the range R and time to A time t is needed to model the transfer function from the AV collision TTC. The distribution of v is not ﬁtted and the commanded acceleration to the actual acceleration. For empirical distribution is shown in Figure 7. simplicity, let t be a constant. AV 1 The Pareto distribution is used to ﬁt the distribution of R . TheparametersofAVmodelinsimulationarelistedinTableI. The ﬁtting result is shown in Figure 8 and Table II. The exponential distribution is used to ﬁt the distribution of 4.3 Analyzed event 1 TTC . Fitting result is shown in Figure 9. The estimate of There are three kinds of high-risk events: conﬂict, crash and ðÞ parameter is l 1 ¼ 0:0647 Std: Err: ¼ 0:0004 . TTC injury. Crash and injury events are included in conﬂict events. 4.4.2 Result of accelerated evaluation The optimal IS parameters # 1 and # 1 are calculated by R TTC Figure 5 TTC as a function of vehicle velocity AEB solving the optimization problem in equations (22)-(30). The GA tool of MATLAB is used to solve this problem. The Figure 6 Deﬁnition of proximity zone 34 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Figure 7 The empirical distribution of v Figure 9 Fitting result of TTC using exponential distribution Figure 10 Fitness value in solution searching process Figure 8 Fitting result of R using Pareto distribution Table II Parameters in distribution of R Parameter k 1 s 1 u 1 R R R Figure 11 Result of traversing calculation Estimate 0.1987 0.0180 0.0133 Std. Err. 0.0066 0.0002 0 value of ﬁtness function in solution searching process is shown in Figure 10. The result is # 1 ¼0:3419, and # 1 ¼0:0120. Traversing calculation may determine TTC the approximate region of the optimal solution. Therefore, to verify the validity of the GA optimal solution, different combinations of # 1 and # 1 values are traversed R TTC [Figure 11]. The result shows that the GA optimal solution is reliable. Then, the conﬂict rate is calculated using the optimal # 1 and # . The convergence is reached when the relative half- TTC width l < 0.2 with 80 per cent conﬁdence. And the test number is calculated when the conﬂict rate reaches convergence. As the simulation is stochastic, the required test number may ﬂuctuate within a certain range. To ensure the credibility of the result, certain times of simulations are required. Therefore, the simulation was done ten times using the average required test number of proposed method is 286. 1 1 And the required test number of non-accelerated simulation optimal # and # . The average test number of ten R TTC simulations was compared with the result of non-accelerated is 10,391. The result of one of the ten simulations is shown in simulation based on empirical data. The result shows that the Figures 12 and 13. 35 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Figure 12 Estimation of the conﬂict rate Figure 14 Conﬂict rate calculated by different methods Figure 13 Relative half-width of conﬂict rate As mentioned in Section 3, the main advantage of accelerated evaluation is that the method can reveal the safety beneﬁts of real world. However, if the error is not corrected, the accelerated testing cannot accurately reveal the conﬂict rate in real world and cannot provide the correct reference to the safety of AVs. Therefore, error correction is of great importance to accelerated testing, which is an indispensable step. 5. Conclusion The rapid development of automated driving technology constantly puts forward new requirements for the testing technique of AVs. This paper proposes an accelerated testing method for AVs safety evaluation based on improved IS techniques. Based on the real-world data, the critical variables of research scenarios are extracted, and the distributions of Based on the same data, the method proposed in Zhao et al. these variables are ﬁtted. Then the optimal IS parameters are (2017b) was also used in simulation. The simulation was done calculated by solving an optimization problem with GAs to ten times too, and the average test number was compared with generate accelerated scenarios, and the AVs are tested in these the result obtained by the proposed method. The result shows scenarios to obtain the safety beneﬁts. Finally, the testing result that the average test number of method in Zhao et al. (2017b)is is modiﬁed by the error correction parameter which is calibrated 435. Therefore, the proposed method is more efﬁcient. by the real-world data and safety beneﬁts, and the ﬁnal result of Compared with the method in Zhao et al. (2017b), the accelerated testing is obtained. Focusing on the cut-in scenario, proposed method improves the test efﬁciency by 35 per cent. the proposed method is validated by simulation based on the Shanghai naturalistic driving data. The result shows that 4.5 Error correction compared with the existing IS technique, the proposed method 1 1 The error in distribution ﬁtting of R and TTC is improves the test efﬁciency by 35 per cent and increases the inevitable. However, the data used in Monte Carlo accuracy of accelerated test result by 23 per cent. simulation and existing accelerated evaluation method Compared with the existing IS technique, the proposed strictly follow the well-ﬁtted distributions, and the error is improved importance technique method has the following overlooked, which leads to the ﬁnal conﬂict rate deviating contributions: First, GA is used to calculate the optimal IS from the actual. Therefore, the error correction is necessary. parameters by solving an optimization problem, which As shown in Figure 14,the conﬂict rate calculated by the improves efﬁciency of test. Second, based on the empirical proposed method converges to the similar value of actual data, the result of test is modiﬁed by the error correction conﬂict rate. However, the conﬂict rate calculated by Monte parameter, which solves the problem in existing studies that the Carlo method and the method in Zhao et al. (2017b) conﬂict rate in accelerated testing result is inconsistent with the converges to another value. By error correction, the accuracy conﬂict rate calculated by the empirical data. Third, based on of accelerated test result was increased by 23 per cent. the naturalistic driving data in Shanghai, typical high-risk 36 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Goldberg,D.E.(1989), GeneticAlgorithms in Search, Optimization, cut-in scenarios in China are analyzed, and the proposed method is validated by simulation. andMachineLearning,1989,Addison-Wesley,Reading. As a new method for AVs testing, the accelerated testing Gorman, T.I. (2013), Prospects for the Collision-Free Car: The method has broad application prospects. Further research Effectiveness of Five Competing Forward Collision Avoidance needs to focus on the following two aspects. One is the Systems, Virginia Polytechnic Institute and State University. deﬁnition and testing method of the more complex high-risk Blacksburg, VA. behaviors, especially the multi-object interactive behavior in Heidelberger, P. (1995), “Fast simulation of rare events in mixed trafﬁc ﬂow. The other aspect is the further integration Queueing and reliability models”, ACM Transactions on of testing tools and accelerated testing methods. Modeling and Computer Simulation (TOMACS),Vol.5No.1, pp. 43-85. Kalra, N. and Paddock, S.M. (2016), “Driving to safety: how References many miles of driving would it take to demonstrate autonomous vehicle reliability?”, Transportation Research Part Althoff, M. and Mergel, A. (2011), “Comparison of Markov chain A: Policy and Practice, Vol. 94, pp. 182-193. abstraction and Monte Carlo simulation for the safety Karabatsou, V., Pappas, M., van Elslande, P., Fouquet, K. and assessment of autonomous cars”, IEEE Transactions on Intelligent Stanzel, M. (2007), A-Priori Evaluation of Safety Functions Transportation Systems, Vol. 12 No. 4, pp. 1237-1247. Effectiveness-Methodologies Table of Contents,Trafﬁc Accident Asmussen, S. and Albrecher, H. (2010), Ruin Probabilities, Causation Eur. D, Paris, Vol. 4, pp. 1-3. World Scientiﬁc Publishing Co Pte Ltd. Kou, Y. (2010), Development and Evaluation of Integrated Berg, G., Nitsch, V. and Färber, B. (2016), “Vehicle in Chassis Control Systems, University of Michigan, Ann Arbor. the loop”, Handbook of Driver Assistance Systems: Basic Kussmann,H.,Modler,H.,Engstrom,J.,Agnvall,A.,Piamonte, Information, Components and Systems for Active Safety and P., Markkula’s, G., Amditis, A., Bolovinou, A., Andreone, L., Comfort, pp. 199-210. Deregibus, E. and Kompfner, P. (2004), “Requirements for Blanchet, J. and Lam, H. (2012), “State-dependent importance AIDE HMI and safety functions”, AIDE project, March. sampling for rare-event simulation: an overview and recent Lee, K. (2004), Longitudinal Driver Model and Collision Warning advances”, Surveys in Operations Research and Management and Avoidance Algorithms Based on Human Driving Databases, Science, Vol. 17 No. 1, pp. 38-59. University of Michigan, Ann Arbor. Bucklew, J. (2013), Introduction to Rare Event Simulation, Lee, S.E., Olsen, E.C. and Wierwille, W.W. (2004), “A Springer Science & Business Media, Berlin. comprehensive examination of naturalistic lane-changes”, Bühne, J.A., Lüdeke, A., Schönebeck, S., Dobberstein, J., No. HS-809 702. Fagerlind, H., Bálint, A. and McCarthy, M. (2012), Ma, W.H. and Huei, P. (1999), “A worst-case evaluation “Assessment of integrated vehicle safety systems for method for dynamic systems”, Transactions-American Society improved vehicle”, ASSESS D2, Vol. 2 Nos 2/2. of Mechanical Engineers Journal of Dynamic Systems Carsten, O., Merat, N., Janssen, W.H., Johansson, E., Fowkes, Measurement and Control, Vol. 121, pp. 191-199. M. and Brookhuis, K.A. (2005), “Human machine Michalewicz, Z. (2013), Genetic Algorithms 1 Data Structures = interaction and the safety of trafﬁc in Europe ﬁnal report”, Evolution Programs, Springer Science & Business Media. Portal, Leeds, Transp. Res. Innov. Royden, H.L. and Fitzpatrick, P. (1988), Real Analysis, Chang, C.S., Heidelberger, P., Juneja, S. and Shahabuddin, Macmillan, New York. P. (1994), “Effective bandwidth and fast simulation of Russo, R., Terzo, M. and Timpone, F. (2007), “Software-in- ATM INTREE networks”, Performance Evaluation,Vol. 20 the-loop development and validation of a cornering brake Nos 1/3, pp. 45-65. control logic”, Vehicle System Dynamics, Vol. 45 No. 2, Deering, R.K. (2002), “Annual report of the crash avoidance pp. 149-163. metrics partnership, April 2001-March 2002”,No. HS- The Enable-S3 Consortium (2016), “Enable-S3 European 809 531. project”, available at: www.enable-s3.eu/ (accessed 22 Euro, N.C.A.P. (2013), Test Protocol – AEB Systems, Eur. New January 2018). Car Assess. Programme (Euro NCAP), Brussels. Touran, A., Brackstone, M.A. and McDonald, M. (1999), “A Federal Ministry for Economic Affairs and Energy (BMWi) (2016), “Pegasus research project”, available at: www. collision model for safety evaluation of autonomous pegasus-projekt.info/en/ intelligent cruise control”, Accident Analysis & Prevention, (accessed 22 January 2018). Vol. 31 No. 5, pp. 567-578. Gen, M. and Cheng, R. (2000), Genetic Algorithms and Engineering Ulsoy, A.G., Peng, H. and Çakmakci, M. (2012), Automotive Optimization,JohnWiley&Sons,Hoboken,NJ,Vol.7. Control Systems, Cambridge University Press, Cambridge. Gietelink, O., Ploeg, J., De Schutter, B. and Verhaegen, M. Wohllebe, T., Vetter, J., Mayer, C., McCarthy, M. and de (2006), “Development of advanced driver assistance systems Lange, R. (2004), “Integrated project on advanced with vehicle hardware-in-the-loop simulations”, Vehicle protection systems”, AP-SP13-0035 Project, Chalmers, System Dynamics, Vol. 44 No. 7, pp. 569-590. Gothenburg. Glasserman, P. and Li, J. (2005), “Importance sampling for Woodrooffe, J., Blower, D., Bao, S., Bogard, S., Flannagan, portfolio credit risk”, Management Science, Vol. 51 No. 11, pp. 1643-1656. C., Green, P.E. and LeBlanc, D. (2014), “Performance Glynn, P.W. and Iglehart, D.L. (1989), “Importance sampling characterization and safety effectiveness estimates of forward for stochastic simulations”, Management Science, Vol. 35 collision avoidance and mitigation systems for medium/ No. 11, pp. 1367-1392. heavy commercial vehicles”, UMTRI-2011-36. 37 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Yang, H.H. and Peng, H. (2010), “Development and mixture models”, IEEE Transactions on Intelligent Transportation Systems. evaluation of collision warning/collision avoidance algorithms Juneja, S. and Shahabuddin, P. (2006), “Rare-event simulation using an errable driver model”, Vehicle System Dynamics, techniques: an introduction and recent advances”, Handbooks Vol. 48 No. S1, pp. 525-535. in Operations Research and Management Science,Vol. 13, Zhao, D., Huang, X., Peng, H., Lam, H. and LeBlanc, D.J. pp. 291-350. (2017a), “Accelerated evaluation of automated vehicles in Kaempchen, N., Schiele, B. and Dietmayer, K. (2009), car-following maneuvers”, IEEE Transactions on Intelligent “Situation assessment of an autonomous emergency brake Transportation Systems, Vol. 19 No. 3. for arbitrary vehicle-to-vehicle collision scenarios”, IEEE Zhao, D., Lam, H., Peng, H., Bao, S., LeBlanc, D.J., Transactions on Intelligent Transportation Systems,Vol. 10 Nobukawa, K. and Pan, C.S. (2017b), “Accelerated No. 4, pp. 678-687. evaluation of automated vehicles safety in lane-change Kollman, C., Baggerly, K., Cox, D. and Picard, R. (1999), scenarios based on importance sampling techniques”, IEEE “Adaptive importance sampling on discrete Markov chains”, Transactions on Intelligent Transportation Systems,Vol. 18 Annals of Applied Probability, Vol. 9 No. 2, pp. 391-412. No. 3, pp. 595-607. Marsden, G., McDonald, M. and Brackstone, M. (2001), “Towards an understanding of adaptive cruise control”, Further reading Transportation Research Part C: Emerging Technologies,Vol.9 No. 1, pp. 33-51. Blincoe, L., Miller, T.R., Zaloshnja, E. and Lawrence, B.A. Moridpour, S., Sarvi, M. and Rose, G. (2010), “Lane changing (2015), “The economic and societal impact of motor vehicle models: a critical review”, Transportation Letters,Vol. 2 crashes, 2010 (revised) 1”, Annals of Emergency Medicine, No. 3, pp. 157-173. Vol. 66 No. 2, pp. 194-196. Sayer, J., LeBlanc, D., Bogard, S., Funkhouser, D., Bao, S., Fildes, B., Keall, M., Bos, N., Lie, A., Page, Y., Pastor, C., Buonarosa, M.L. and Blankespoor, A. (2011), “Integrated Pennisi, L., Rizzi, M., Thomas, P. and Tingvall, C. (2015), vehicle-based safety systems ﬁeld operational test ﬁnal “Effectiveness of low speed autonomous emergency braking in program report”. real-world rear-end crashes”, Accident Analysis & Prevention, Vahidi, A. and Eskandarian, A. (2003), “Research advances in Vol. 81, pp. 24-29. intelligent collision avoidance and adaptive cruise control”, Huang, Z., Lam, H. and Zhao, D. (2017a), “Towards IEEE Transactions on Intelligent Transportation Systems, Vol. 4 affordable on-track testing for autonomous vehicle – a No. 3, pp. 143-153. Kriging-based statistical approach”, Proceedings of the IEEE Zhao, D., Peng, H., Bao, S., Nobukawa, K., LeBlanc, D.J. and 20th International Intelligent Transportation Systems Conference Pan, C.S. (2016), “Accelerated evaluation of automated (ITSC), Yokohama, 16-19 October 2017. vehicles using extracted naturalistic driving data”, The Huang, Z., Lam, H. and Zhao, D. (2017b), “An accelerated Dynamics of Vehicles on Roads and Tracks: Proceedings of the testing approach for automated vehicles with background 24th Symposium of the International Association for Vehicle trafﬁc described by joint distributions”, Proceedings of the System Dynamics (IAVSD 2015), Graz, Austria, 17-21 August IEEE 20th International Intelligent Transportation Systems 2015, CRC Press, Florida,p.287. Conference (ITSC), Yokohama, 16-19 October 2017. Huang, Z., Lam, H., LeBlanc, D.J. and Zhao, D. (2017c), Corresponding author “Accelerated evaluation of automated vehicles using piecewise Jian Sun can be contacted at: sunjian@tongji.edu.cn For instructions on how to order reprints of this article, please visit our website: www.emeraldgrouppublishing.com/licensing/reprints.htm Or contact us for further details: permissions@emeraldinsight.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Intelligent and Connected Vehicles Emerald Publishing http://www.deepdyve.com/lp/emerald-publishing/accelerated-testing-for-automated-vehicles-safety-evaluation-in-cut-in-xEFGBL8yJw

Loading next page...

Publisher: Emerald Publishing
Copyright: © Yiming Xu, Jianqiang Wang, Yajie Zou and Jian Sun.
ISSN: 2399-9802
DOI: 10.1108/jicv-01-2018-0002
Publisher site: See Article on Publisher Site

Abstract

Purpose – It would take billions of miles’ ﬁeld road testing to demonstrate that the safety of automated vehicle is statistically signiﬁcantly higher than the safety of human driving because that the accident of vehicle is rare event. Design/methodology/approach – This paper proposes an accelerated testing method for automated vehicles safety evaluation based on improved importance sampling (IS) techniques. Taking the typical cut-in scenario as example, the proposed method extracts the critical variables of the scenario. Then, the distributions of critical variables are statistically ﬁtted. The genetic algorithm is used to calculate the optimal IS parameters by solving an optimization problem. Considering the error of distribution ﬁtting, the result is modiﬁed so that it can accurately reveal the safety beneﬁts of automated vehicles in the real world. Findings – Based on the naturalistic driving data in Shanghai, the proposed method is validated by simulation. The result shows that compared with the existing methods, the proposed method improves the test efﬁciency by 35 per cent, and the accuracy of accelerated test result is increased by 23 per cent. Originality/value – This paper has three contributions. First, the genetic algorithm is used to calculate IS parameters, which improves the efﬁciency of test. Second, the result of test is modiﬁed by the error correction parameter, which improves the accuracy of test result. Third, typical high-risk cut-in scenarios in China are analyzed, and the proposed method is validated by simulation. Keywords Genetic algorithm, Simulation, Automated vehicles, Importance sampling, Lane changing, Safety evaluation, High-risk scenarios Paper type Research paper the test time required is still very long without acceleration 1. Introduction because of the huge scenarios and long driving-testing distance. According to trafﬁc accident statistics reports, human factor is However, the existing studies pay more attention to the the main cause of trafﬁc accidents. It is estimated that over 90 construction of scenario libraries (e.g. Pegasus [Federal per cent accidents in motor crashes are because of drivers’ error Ministry for Economic Affairs and Energy (BMWi), 2016], (National Highway Trafﬁc Safety Administration, 2015). enableS3 (The Enable-S3 Consortium, 2016), etc.) and test Therefore, the automated vehicle (AV) is considered as the key tools development (e.g. software-in-loop (Russo et al.,2007), technique to improve trafﬁc safety. When an AV drives in real- hardware-in-loop (Gietelink et al.,2006) and vehicle in the world roads, it needs to deal with various and complicated loop (Berg et al., 2016)). The studies of accelerated testing trafﬁc conditions. Therefore, AVs must be tested before they method are neglected. Therefore, the accelerated method of can be permitted to travel on the road, otherwise the AVs will be automated loading of the re-sampled driving scenarios athreat to other trafﬁc participants. However, according to the study by Kalra and Paddock (2016), it would take approximately 5 billion miles to demonstrate that the safety of AV is statistically signiﬁcantly higher than the safety of human © Yiming Xu, Yajie Zou and Jian Sun. Published in Journal of Intelligent driving because of the very small probability of human driving and Connected Vehicles. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY accidents. And with a ﬂeet of 100 AVs being ﬁeld test-driven 4.0) licence. Anyone may reproduce, distribute, translate and create 24 h a day, 365 days a year at an average speed of 25 miles per derivative works of this article (for both commercial and non- hour, this would take about 225 years. Even by simulation, commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode The current issue and full text archive of this journal is available on The authors would like to thank the Natural Science Foundation of China Emerald Insight at: www.emeraldinsight.com/2399-9802.htm (U1764261, 51422812), and the Shanghai Science and technology project of international cooperation (16510711400) for supporting this research. Received 25 January 2018 Journal of Intelligent and Connected Vehicles Revised 20 March 2018 1/1 (2018) 28–38 2 May 2018 Emerald Publishing Limited [ISSN 2399-9802] [DOI 10.1108/JICV-01-2018-0002] Accepted 9 May 2018 28 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 should be concerned so that the safety beneﬁts of AVs can be distribution. With the advent of naturalistic driving projects effectively evaluated. This scenario acceleration method can around the world in recent years, some studies began to build be applied to simulation test and provide scenarios for stochastic models based on naturalistic driving big data and hardware-in-loop test, driving simulator test and controlled carry out Monte Carlo simulations to evaluate AVs. Yang and test bed (e.g. Mcity). Peng (2010); Lee (2004) and Woodrooffe et al. (2014) all This paper proposes an accelerated testing method for AVs evaluated the collision avoidance systems of vehicles by (Level 2 to Level 5) safety evaluation based on improved Monte Carlo simulation based on naturalistic driving big data. The advantage of Monte Carlo simulation is stochastic importance sampling (IS) techniques. This paper uses the scenarios. However, in the simulation of rare events (such as occurrence of high-risk event, such as conﬂict, crash and injury, collision), the number of tests required will be very large to evaluate safety of AV. The high-risk events are identiﬁed by because of randomness. indirect indicators such as Time to Collision (TTC) and Test matrix method is to predeﬁne a “test matrix” consisting headway. Taking the typical cut-in scenarios as example, the proposed method extracts the critical variables of the scenario. of a variety of typical scenarios based on past crash data and expert knowledge. Then the test matrix is used to test some Based on IS techniques, the scenarios with higher probability of properties (such as safety beneﬁt) of AVs. The test matrix occurrence of high-risk events are reconstructed to test the method is the basis of many test studies, such as autonomous AVs. Theresultoftestismodiﬁed by the error correction emergency braking (AEB) protocol (Euro, N.C.A.P., 2013), parameter which is calibrated by the empirical data, so that CAMP (Deering, 2002), HASTE (Carsten et al., 2005), AIDE the safety beneﬁts of AVs in the real world can be revealed. (Kussmann et al.,2004), TRACE (Karabatsou et al.,2007), Finally, based on the naturalistic driving data in Shanghai, APROSYS (Wohllebe et al.,2004) and ASSESS (Bühne et al., the proposed method is validated by simulation. Besides the 2012). The advantage of the test matrix method is efﬁcient, IS accelerated testing method, the detailed contributions of credible and repeatable. Applying test matrix method to this paper are as follows. First, the genetic algorithm (GA) is evaluate low-level AVs is straightforward and it might used to calculate the optimal IS parameters by solving an continue to be the selected approach in the near future optimization problem, which improves the efﬁciency of (Zhao et al., 2017a). However, test scenarios in the test accelerated testing. Second, based on the empirical data, the matrix are predeﬁned and ﬁxed. The scenarios predeﬁned result of test is modiﬁed by the error correction parameter, based on crash data cannot test the AVs comprehensively. which solves the problem in existing studies that the conﬂict And the test cannot reveal the properties of AVs in real- rate in accelerated testing result is inconsistent with the world conditions, especially in complicated mixed trafﬁc conﬂict rate calculated by the empirical data. Third, based ﬂows. on the naturalistic driving data in Shanghai, typical high-risk WCSE methodology evaluates the control system of a vehicle cut-in scenarios in China are analyzed, and the proposed by generating worst scenario (Ma and Huei, 1999; Kou, 2010). method is validated by simulation. WCSE can be expressed as an optimization problem that The rest of this paper is organized as follows. Section 2 searches for the worst scenario. The scenario is quantitatively presents the literature review of existing methods of AVs evaluated by a cost function, and the goal of WCSE is to search testing. Section 3 introduces the proposed accelerated testing for the scenario with the largest cost. WCSE can identify the method for AVs safety evaluation in detail. Section 4 veriﬁes the weakness in the control system of an AV. But WCSE does not proposed method by simulation based on naturalistic driving correlate the worst scenario with real-world scenarios, and the data in Shanghai. Section 5 presents the conclusion and future probability of occurrence of worst-case scenario in real world research needs. cannot be identiﬁed. Accelerated evaluation is a data (such as naturalistic 2. Literature review driving data)-based testing method. High-risk scenarios are The test of AV requires a combination of test tools and test more efﬁcient than normal scenarios in AV testing. methods. Test tools provide facilities and test methods provide However, high-risk scenarios are rare events in naturalistic theoretical guidance for the test. This paper focuses on test driving data. Accelerated evaluation method modiﬁes the methods because of the larger potential for accelerated testing distribution of real-world data so that the high-risk scenarios and their theory signiﬁcance. In existing studies, there are four have a higher probability of occurrence. Zhao et al. (2015) main methods for AVs testing: Monte Carlo simulation, test extracted the car-following scenarios in naturalistic driving matrix, worst-case scenario evaluation (WCSE) and accelerated data and used models to ﬁt the critical variables in the evaluation. The above four methods are reviewed in this section. scenarios. The most frequent scenarios are deleted to Monte Carlo simulation is a stochastic method. The AVs are increase the overall exposure rate for critical scenarios. tested in the scenarios generated stochastically based on a However, this method cannot evaluate the safety beneﬁts of certain distribution. Touran et al. (1999) evaluated the safety of AVs in real world. Considering this problem, Zhao et al. autonomous intelligent cruise control model by Monte Carlo (2017a, 2017b) applied the IS technique to AVs testing and simulation. The values of some parameters in the model were studied the cut-in and car-following scenarios. The obtained randomly based on a certain distribution. Althoff and distribution of critical variables was modiﬁed to increase the Mergel (2011) evaluated the collision risk for autonomous probability of occurrence of high-risk scenarios and reduce vehicles when executing a planned maneuver by Monte Carlo the number of required tests. Then, the IS technique was simulation. The initial states in simulation were generated used to modify the test result so that the result can reveal the randomly according to a piecewise constant probability safety beneﬁts of AVs in real world. The most important 29 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 advantage of accelerated evaluation is that it can describe random sample in X. Let I (x) be the indicator function of the real-world beneﬁts (such as crash rate) of AVs. And the high-risk event «.I (x)isdeﬁned as: method also improves the efﬁciency of testing. But in 1; if x « existing studies, the error in critical variable distribution I ðÞ x ¼ (1) ﬁtting is ignored, which leads to the ﬁnal result deviating 0; otherwise from actual. This phenomenon will be discussed in detail in Section 4.4.3. Let g denote the probability P(«). g can be estimated via In summary, the test matrix method and WCSE method simulation by generating independent samples (x , x ,.. ., x ). 1 2 n can hardly reveal the probability of AVs being exposed to ^ ^ Let g denote an estimator of g. g can be calculated by: n n risk in real-world scenarios, and the Monte Carlo simulation is inefﬁcient. The accelerated evaluation method can reveal the safety beneﬁts of AVs in real world and has the advantage g ^ ¼ I x (2) «ðÞ i of high testing efﬁciency. However, existing studies i¼1 overlooked the ﬁtting errors of critical variables, resulting in the deviation of test result from the ﬁeld operation. To solve According to the law of large numbers, g ^ ! g as n !1, that this problem, this paper proposes an accelerated testing is, when n is large enough, g ^ converges to g. method for AVs safety evaluation based on improved IS To ensure the reliability of g ^ , the central limit theorem techniques. Considering the ﬁtting error of critical variables, proves useful in developing a conﬁdence interval (CI) for the GA is used to calculate the optimal IS parameters to estimate and is used to determine the necessary n for obtain a better acceleration efﬁciency. And the error accurate estimation. For a sufﬁciently large n, the variance correction parameter is used to correct the test result to of g ^ is: make the result consistent with the result calculated by 0 1 empirical data. X @ A s ðÞ g ^ ¼ Var I x «ðÞ i i¼1 3. Methodology In real-world data, high-risk events of vehicles belong to rare (3) ¼ Var I x «ðÞ i events, and a reliable evaluation of the probability of this 2 i¼1 event requires a large number of tests. The proposed gðÞ 1 g accelerated testing method based on improved IS techniques can reduce the number of tests and obtain a reliable test result. The proposed method can be divided With a conﬁdence level at 100(1 – a) per cent, the CI of g ^ is: into four steps (Figure 1). First, basedonthe real-world data, the scenarios to be analyzed are extracted, and the g ^ z sðÞ g ^ ; g ^ 1 z sðÞ g ^ (4) n a=2 n n a=2 n critical variables of these scenarios are deﬁned and obtained. Second, based on the extracted scenarios and variables, the where, z is deﬁned as: a/2 optimal IS parameters are calculated to generate accelerated scenarios, and the IS technique and simulation are used to 1 z ¼ UðÞ 1 a=2 (5) a=2 calculate the safety beneﬁts of AVs. Third, the error correction parameter is calibrated by the real-world data and 1 where, U is the inverse cumulative distribution function of safety beneﬁts. Finally, the test result is corrected by the normal distribution N(0,1). error correction parameter, and the ﬁnal safety beneﬁts of The half-width of CI is: AVs in real world are obtained. We will discuss this framework in the following sections in detail. l ¼ z s g ^ (6) ðÞ a=2 a=2 n 3.1 Probability of high-risk-event As the value of g ^ is small, the relative half-width l is Consider a sample space X with a probability measure P. Let used to indicate the accuracy of the estimation. l is P(«) denote the probability of a high-risk event. Let x be a deﬁned as: Figure 1 Framework of proposed accelerated testing method 30 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 l ¼ l =g (7) 2 2 r a=2 z ðÞ ðÞ E I x L x f a=2 n 1 (13) 2 2 g b To ensure that l is smaller than a constant b we need: sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ Therefore, to obtain a reliable result through a small number of l z sðÞ g ^ 1 g a=2 a=2 tests, the density function f (x) needs to be properly chosen to l ¼ ¼ ¼ z b (8) r a=2 g g gn 2 2 2 ðÞ ðÞ make E I x L x close to g . That is: 3.3 High-risk scenarios: cut-in events The proposed method can be used in a variety of scenarios such 1 g a=2 as lane changing scenarios, car-following scenarios and crossing n (9) g b scenarios. In the typical trafﬁc environment, the cut-in scenario occurs more frequently and has greater risk on human driving. The probability of high-risk events in real-world driving is Therefore, this study focuses on the cut-in scenario which very small. Therefore, the sample size n needs to be huge to refers to the situation that other vehicles move into the lane ensure the reliability of the estimation. This means that a where AV located from an adjacent lane in front of the AV. huge number of tests are required to evaluate the probability The critical variables of cut-in scenario are identiﬁed as: the velocity of lane changing vehicle (LCV) v , the range R deﬁned of high-risk events in AV driving if using the Crude Monte Carlo method. as the distance between the rear edge of the LCV and the front edge of AV, and TTC. The TTC is deﬁned as: 3.2 Importance sampling IS is one of the classical variance reduction techniques for TTC ¼ (14) increasing the efﬁciency of Monte Carlo algorithms (Glynn and Iglehart, 1989). The IS has been successfully used to evaluate where, R is the range, R is the derivative of R. reliability (Heidelberger, 1995) and critical events in ﬁnance During the driving process, the high-risk events tend to (Glasserman and Li, 2005), insurance (Asmussen and Albrecher, correspond to small R and TTC. Smaller R and TTC indicate 2010) and telecommunication networks (Chang et al.,1994). that the event is rarer and less safe. Therefore, the reciprocal of General overviews about IS can be found in Bucklew (2013) and R and TTC are used to put the rare events in the tail of the Blanchet and Lam (2012). The basic idea of IS used in distribution. The Pareto distribution is used to ﬁt R and the evaluation of AVs is to replace the original distribution density Exponential distribution is used to ﬁt TTC . The distribution function f(x)bya newone f (x) to generate event x,which leads of v is not ﬁtted and the empirical distribution is directly used to a higher probability of occurrence of high-risk events. And to generate events. then the risk calculation function is modiﬁed to obtain the safety 1 The density function of the distribution of R can be beneﬁts of AVs. expressed as: Let x be the event generated by f (x), the estimator g ^ can be expressed as: 1 1 1 x u R f 1ðÞ xjk 1; s 1; u 1 ¼ 11 k 1 R R R R R n s 1 s 1 X R R (15) g ^ ¼ I x Lx (10) n i i i¼1 where, k 1 is the shape parameter; s 1 is the scale parameter; R R u 1 is the threshold parameter. where,LxðÞ is the likelihood ratio (Radon–Nikodym derivative R The density function of the distribution of TTC can be (Royden and Fitzpatrick, 1988)) deﬁned as: expressed as: fx ðÞ Lx ¼ (11) i 1 x=l f ðÞ x 1 TTC i 1 1 f ðÞ xjl ¼ e (16) TTC TTC l 1 TTC The relative half-width of CI constructed by IS can be where, l 1 is the rate parameter. TTC expressed as: Therefore, every lane changing event can be expressed as: l z s g ^ ðÞ a=2 a=2 1 1 l ¼ ¼ x ¼ v ; R ; TTC (17) r i l g g qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ 2 2 z E ðÞ g ^ E ðÞ g ^ a=2 n n pﬃﬃﬃ (12) 3.4 Accelerated Evaluation g n sﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ According to IS technique, a new distribution f is used to 2 2 ðÞ ðÞ z E I x L x a=2 f « replace f 1, and a new distribution f is used to replace TTC ¼ pﬃﬃﬃ 1 n g f 1.AsR obeys Pareto distribution, we need to TTC construct a new exponential distribution f 1 before To ensure l b, the necessary test number n is: replacing f 1 by f to reduce the computation complexity r R 31 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 1 x in later steps (Zhao et al., 2017b). f 1 has the smallest least f 1ðÞ x ¼ exp TTC square error to f 1. f 1 is deﬁned as: l 1 # 1 l 1 # 1 R R TTC TTC TTC TTC (30) 1 x f 1ðÞ x ¼ exp (18) l 1 l 1 R R f 1 m f 1 w ðÞ ðÞ R TTC 1 1 LR ¼ m; TTC ¼ w ¼ (31) ~ f m f w ðÞ ðÞ 1 1 Apply exponential change of measure to f 1 and f 1, we get R TTC TTC f and f : 1 1 R TTC In this optimization problem, the objective function is number 1 x of tests n, and the decision variables are IS parameters # 1 and f 1ðÞ xj# 1 ¼ exp l 1 # 1 l 1 # 1 # 1. In the constrains, I is the indicator function of high- R R R R TTC « risk event «; L(x ) is the likelihood; g is the probability of high- (19) risk events; z is given by equation (5); b is the threshold of a/2 relative half-width; x is a matrix of a series of samples x , x ,.. ., 1 2 1 x x,and x is given by equation (17); f 1ðÞ x is the density f ðÞ xj# 1 ¼ exp 1 i i R TTC TTC l 1 # 1 l 1 # 1 1 TTC TTC TTC TTC function of distribution of R ; f 1ðÞ x is the density function TTC (20) of distribution of TTC ; x is a matrix of a series of samples 1 1 x ; x ; .. . ; x and x ¼ v ; R ; TTC ; f x is the density ðÞ l 1 1 2 i i R Therefore, the likelihood is: function of distribution of R ; f ðÞ x is the density function TTC 1s of distribution of TTC . f 1ðÞ x f 1 y ðÞ R TTC 1 1 GA is oneofthe effectivemethods to solvethe optimization LR ¼ x; TTC ¼ y ¼ (21) f ðÞ x f y 1 1ðÞ R TTC problem (Goldberg, 1989; Michalewicz, 2013). GA is an adaptive global search algorithm, which has the advantages of short The probability of high-risk events is: calculation time and high robustness. GA is widely used in multi- objective optimization, industrial engineering, management PðÞ « ¼ EðÞ I ðÞ x ¼ EðÞ I ðÞ x LxðÞ (22) f « f « science, artiﬁcial intelligence and so on (Gen and Cheng, 2000). The optimization problem can be solved by GA to calculate the Therefore, proper f and f need to be constructed to 1 1 optimal IS parameter. With the assistant of GA tool in MATLAB, R TTC calculate P(«)efﬁciently. In other words, optimal IS parameters the optimization problem can be easily solved. # 1 and # 1 need to be obtained to get the reliable test result R TTC through the minimum number of tests. 3.6 Error correction 1 1 The error in distribution ﬁtting of R and TTC is inevitable during the accelerated testing. Through the comparison with 3.5 Searching for optimal importance sampling the empirical data, it is obvious that these ﬁtting errors will lead parameters with genetic algorithm to the ﬁnal result g ^ deviating from the actual probability of The efﬁciency of accelerated evaluation is closely related to the risk g. However, the existing studies only compare the IS parameters. To obtain the optimal IS parameters, consider accelerated testing result with Monte Carlo simulation result. The data used in these two methods strictly follow the well- the following optimization problem: ﬁtted distributions, and the error is overlooked. Fortunately, 2 2 ðÞ ðÞ E I x L x f a=2 this study found this issue by comparing the result with the min n ¼ 1 (23) 2 2 g b actual data and corrected the error. The error is corrected by error correction parameter t,let: s:t g ¼ EðÞ I ðÞ x (24) f « g ¼ g ^ t (32) 1 1 where, g is the probability of risk calculated by empirical data; x ¼ v ; R ; TTC (25) g ^ is the estimator obtained by accelerated evaluation. The ﬁtting error can be reﬂected by the difference between mean of 1 x u 1 R the ﬁtted distributions and empirical distributions. Therefore, f 1ðÞ x ¼ 11 k 1 (26) R R s 1 s 1 R R t is deﬁned as: t ¼ k m m 1 k m m (33) 1 2 ðÞ R1 R0 ðÞ TTC1 TTC0 x=l TTC f 1ðÞ x ¼ e (27) TTC l 1 TTC 1 where, m is the mean of ﬁtted R distribution; m is the R1 R0 mean of empirical R distribution; m is the mean of ﬁtted TTC1 1 1 1 x ¼ v ; R ; TTC (28) TTC distribution; m is the mean of empirical TTC l TTC0 distribution; k and k parameters need to be calibrated. 1 2 The signiﬁcance of error correction parameter is to modify 1 x f x ¼ exp (29) 1ðÞ the test result to make it closer to empirical result rather than l 1 # 1 l 1 # 1 R R R R the result of Monte Carlo simulation. The revised result can 32 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 better reveal the safety beneﬁts of AVs in real world, and it is Figure 2 The trajectories of naturalistic driving vehicles more persuasive. 3.7 Algorithm of the proposed method The execution algorithm of the proposed method is as follows: Execution Algorithm of the Proposed Method Step 1 Critical Variables Extraction Step 1.1: Input the real-world data X. Extract scenario W to be analyzed. Take cut-in scenario as example. Step 1.2: Input the scenarios W.Deﬁne the critical variables j , j ,.. .,j . Extract j ,j ,.. .,j . from W. The critical 1 2 i 1 2 i variables of cut-in scenario are v , R and TTC. Step 2 Accelerated Distribution Generating Step 2.1: Input the critical variables R and TTC. Fit the distribution of R based on equation (15). Fit the distribution of TTC based on equation (16). Figure 3 Data collected by data acquisition system Step 2.2: Input the distribution of R . Generate f 1 based on equation (18). Step 2.3: Input the distribution f 1 and f 1. Generate the R TTC optimal IS parameters # 1 and # 1 based on equations R TTC (23)-(31). Step 2.4: Input the parameters # 1 and # 1. Generate R TTC accelerated distribution f and f based on equations 1 1 R TTC (19)-(20). Step 3 Error Correction Parameter Calibration Input the real-world data X . Where X is a subset ofX. 1 1 Input the accelerated distribution f and f . Generate 1 1 R TTC parameter k and k based on equations (32) and (33). 1 2 Step 4 Test of AVs Step 4.1: Input the accelerated distribution f and f , 1 1 R TTC and the empirical distribution of v . Generate the accelerated scenarios xi based on equation (17). Step 4.2: Input accelerated scenarios xi, and the parameter k and k . Calculate the test result based on equations (21) (22) (32) (33). of lane changing moment are extracted. The variables include 4. Simulation v , v and R (Figure 4). Where v is the velocity of LCV, v is the l l velocity of evaluated vehicle and R is the range deﬁned as the 4.1 Data distance between the rear edge of the LCV and the front edge of The data used in this research are from Shanghai Naturalistic evaluated vehicle. In Figure 4, the evaluated vehicle is AV, and Driving Research project. The project is the ﬁrst naturalistic the LCV can be AV or manual-driving vehicle. driving data collection project using real vehicles and high- For comparison analysis, the same criteria as Zhao et al. precision equipment in China. The project aims to collect real- (2017b) were applied: v [ (2m/s,40m/s); v [ (2m/s,40m/s); and world trafﬁc data and study the behavioral characteristics of R [ (0.1m,75m). Finally, 32,104 cut-in scenarios were extracted Chinese drivers. It recorded over 500,000 km naturalistic from the data. driving data from December 2012 to December 2015. The driving trajectories of vehicles basically covered the main roads 4.2 Evaluated models in Shanghai (Figure 2). The vehicles are equipped with For comparison analysis, the same AV model and parameters Mobileye vehicle active safety system and SHARP2 NextGen as that of Zhao et al. (2014b) were used in the simulation. The data acquisition system. The data collected include AV is equipped with adaptive cruise control (ACC) and AEB information such as vehicle position, velocity and distance to surrounding vehicles (Figure 3). The simulation takes the cut-in scenario as an example to Figure 4 Extracted variables of cut-in scenario validate the proposed method. Therefore, the cut-in scenarios are extracted from the naturalistic driving data. The cut-in scenario refers to the situation that other vehicles move into the lane where AV located from an adjacent lane in front of the AV. The moment when the LCV crosses the lane line is deﬁned as lane changing moment. Then, the variables of cut-in scenario 33 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 system. When TTC TTC , the AV is controlled by ACC; Table I Parameters in evaluated models AEB and when TTC < TTC , the AV is controlled by AEB. AEB Parameter Value Unit Equation TTC is a function of vehicle velocity (Figure 5). AEB ACC 38.6 – (34) The ACC is approximated by a discrete proportional- K ACC ACC integral controller to achieve a desired time headway T 1.35 – (34) HW d i ACC (Ulsoy et al., 2012). The input of the controller is time headway T 2 s (35) HW Err T 0.1 s (34) error t , and the output is the command acceleration of the HW max 2 a 5 m/s (37) next time step which can be expressed as: ACC a 10 m/s (38) AEB ACC Err Err ðÞ ðÞ ðÞ ðÞ a k1 1 ¼ a k 1 K t k t k 1 d d r 16 m/s (38) AEB p HW HW T 0.5 s (38) ACC Err Err ðÞ ðÞ 1 K t k t k 1 T =2 (34) i HW HW s 0.0796 s – AV Err ACC t ¼ t T (35) HW HW HW Besides, the simulation analysis methods and conclusions of three kinds of events are both similar. Therefore, for simplicity, simulation analysis in this paper focuses on the conﬂict events. t ¼ R=v (36) HW A conﬂict event happens when an AV enters the proximity zone of the LCV during time t to t 1 T. The proximity zone is max ja j a (37) ACC deﬁned as the area in the adjacent lane from 1.2 m in front of the bumper of LCV to 9 m behind the rear bumper of LCV Err where, a is the command acceleration; t is the time HW (Lee et al.,2004). The deﬁnition of proximity zone is shown in ACC ACC headway error; K and K are constant gains; T is the Figure 6. p i ACC sampling time; T is the desired time headway; t is the HW HW max 4.4 Simulation result time headway; R is the range; v is the velocity of AV; and a is ACC Simulation can be used to evaluate the real-world safety beneﬁts the maximum acceleration. of AVs if the models are strictly calibrated by real-world data. In Once triggered, AEB aims to achieve an acceleration a . AEB this paper, the distributions of critical variables are ﬁtted based Let the triggered moment be 0, the AEB model can be on empirical data. The IS parameter is calculated based on a expressed as: subset of empirical data. The error correction parameter is calibrated by a subset of empirical data. The AEB system was 0; if t T > a extracted from a 2011 Volvo V60, based on a test conducted by ðÞ atðÞ ¼ r t T ; if t > T and aðÞ t a (38) AEB a a AEB > ADAC (Allgemeiner Deutscher Automobil-Club e.V.) a ; else AEB (Gorman, 2013). Therefore, the results of simulation can reveal the real-world safety beneﬁts of tested AV. where, a(t) is the acceleration of vehicle at t moment; r is the AEB 4.4.1 Distribution of critical variables derivative of acceleration; T is the action time; and a is the a AEB As is mentioned in Section 3.3, the critical variables of cut-in acceleration of emergency braking. scenario are the velocity of LCV v , the range R and time to A time t is needed to model the transfer function from the AV collision TTC. The distribution of v is not ﬁtted and the commanded acceleration to the actual acceleration. For empirical distribution is shown in Figure 7. simplicity, let t be a constant. AV 1 The Pareto distribution is used to ﬁt the distribution of R . TheparametersofAVmodelinsimulationarelistedinTableI. The ﬁtting result is shown in Figure 8 and Table II. The exponential distribution is used to ﬁt the distribution of 4.3 Analyzed event 1 TTC . Fitting result is shown in Figure 9. The estimate of There are three kinds of high-risk events: conﬂict, crash and ðÞ parameter is l 1 ¼ 0:0647 Std: Err: ¼ 0:0004 . TTC injury. Crash and injury events are included in conﬂict events. 4.4.2 Result of accelerated evaluation The optimal IS parameters # 1 and # 1 are calculated by R TTC Figure 5 TTC as a function of vehicle velocity AEB solving the optimization problem in equations (22)-(30). The GA tool of MATLAB is used to solve this problem. The Figure 6 Deﬁnition of proximity zone 34 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Figure 7 The empirical distribution of v Figure 9 Fitting result of TTC using exponential distribution Figure 10 Fitness value in solution searching process Figure 8 Fitting result of R using Pareto distribution Table II Parameters in distribution of R Parameter k 1 s 1 u 1 R R R Figure 11 Result of traversing calculation Estimate 0.1987 0.0180 0.0133 Std. Err. 0.0066 0.0002 0 value of ﬁtness function in solution searching process is shown in Figure 10. The result is # 1 ¼0:3419, and # 1 ¼0:0120. Traversing calculation may determine TTC the approximate region of the optimal solution. Therefore, to verify the validity of the GA optimal solution, different combinations of # 1 and # 1 values are traversed R TTC [Figure 11]. The result shows that the GA optimal solution is reliable. Then, the conﬂict rate is calculated using the optimal # 1 and # . The convergence is reached when the relative half- TTC width l < 0.2 with 80 per cent conﬁdence. And the test number is calculated when the conﬂict rate reaches convergence. As the simulation is stochastic, the required test number may ﬂuctuate within a certain range. To ensure the credibility of the result, certain times of simulations are required. Therefore, the simulation was done ten times using the average required test number of proposed method is 286. 1 1 And the required test number of non-accelerated simulation optimal # and # . The average test number of ten R TTC simulations was compared with the result of non-accelerated is 10,391. The result of one of the ten simulations is shown in simulation based on empirical data. The result shows that the Figures 12 and 13. 35 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Figure 12 Estimation of the conﬂict rate Figure 14 Conﬂict rate calculated by different methods Figure 13 Relative half-width of conﬂict rate As mentioned in Section 3, the main advantage of accelerated evaluation is that the method can reveal the safety beneﬁts of real world. However, if the error is not corrected, the accelerated testing cannot accurately reveal the conﬂict rate in real world and cannot provide the correct reference to the safety of AVs. Therefore, error correction is of great importance to accelerated testing, which is an indispensable step. 5. Conclusion The rapid development of automated driving technology constantly puts forward new requirements for the testing technique of AVs. This paper proposes an accelerated testing method for AVs safety evaluation based on improved IS techniques. Based on the real-world data, the critical variables of research scenarios are extracted, and the distributions of Based on the same data, the method proposed in Zhao et al. these variables are ﬁtted. Then the optimal IS parameters are (2017b) was also used in simulation. The simulation was done calculated by solving an optimization problem with GAs to ten times too, and the average test number was compared with generate accelerated scenarios, and the AVs are tested in these the result obtained by the proposed method. The result shows scenarios to obtain the safety beneﬁts. Finally, the testing result that the average test number of method in Zhao et al. (2017b)is is modiﬁed by the error correction parameter which is calibrated 435. Therefore, the proposed method is more efﬁcient. by the real-world data and safety beneﬁts, and the ﬁnal result of Compared with the method in Zhao et al. (2017b), the accelerated testing is obtained. Focusing on the cut-in scenario, proposed method improves the test efﬁciency by 35 per cent. the proposed method is validated by simulation based on the Shanghai naturalistic driving data. The result shows that 4.5 Error correction compared with the existing IS technique, the proposed method 1 1 The error in distribution ﬁtting of R and TTC is improves the test efﬁciency by 35 per cent and increases the inevitable. However, the data used in Monte Carlo accuracy of accelerated test result by 23 per cent. simulation and existing accelerated evaluation method Compared with the existing IS technique, the proposed strictly follow the well-ﬁtted distributions, and the error is improved importance technique method has the following overlooked, which leads to the ﬁnal conﬂict rate deviating contributions: First, GA is used to calculate the optimal IS from the actual. Therefore, the error correction is necessary. parameters by solving an optimization problem, which As shown in Figure 14,the conﬂict rate calculated by the improves efﬁciency of test. Second, based on the empirical proposed method converges to the similar value of actual data, the result of test is modiﬁed by the error correction conﬂict rate. However, the conﬂict rate calculated by Monte parameter, which solves the problem in existing studies that the Carlo method and the method in Zhao et al. (2017b) conﬂict rate in accelerated testing result is inconsistent with the converges to another value. By error correction, the accuracy conﬂict rate calculated by the empirical data. Third, based on of accelerated test result was increased by 23 per cent. the naturalistic driving data in Shanghai, typical high-risk 36 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Goldberg,D.E.(1989), GeneticAlgorithms in Search, Optimization, cut-in scenarios in China are analyzed, and the proposed method is validated by simulation. andMachineLearning,1989,Addison-Wesley,Reading. As a new method for AVs testing, the accelerated testing Gorman, T.I. (2013), Prospects for the Collision-Free Car: The method has broad application prospects. Further research Effectiveness of Five Competing Forward Collision Avoidance needs to focus on the following two aspects. One is the Systems, Virginia Polytechnic Institute and State University. deﬁnition and testing method of the more complex high-risk Blacksburg, VA. behaviors, especially the multi-object interactive behavior in Heidelberger, P. (1995), “Fast simulation of rare events in mixed trafﬁc ﬂow. The other aspect is the further integration Queueing and reliability models”, ACM Transactions on of testing tools and accelerated testing methods. Modeling and Computer Simulation (TOMACS),Vol.5No.1, pp. 43-85. Kalra, N. and Paddock, S.M. (2016), “Driving to safety: how References many miles of driving would it take to demonstrate autonomous vehicle reliability?”, Transportation Research Part Althoff, M. and Mergel, A. (2011), “Comparison of Markov chain A: Policy and Practice, Vol. 94, pp. 182-193. abstraction and Monte Carlo simulation for the safety Karabatsou, V., Pappas, M., van Elslande, P., Fouquet, K. and assessment of autonomous cars”, IEEE Transactions on Intelligent Stanzel, M. (2007), A-Priori Evaluation of Safety Functions Transportation Systems, Vol. 12 No. 4, pp. 1237-1247. Effectiveness-Methodologies Table of Contents,Trafﬁc Accident Asmussen, S. and Albrecher, H. (2010), Ruin Probabilities, Causation Eur. D, Paris, Vol. 4, pp. 1-3. World Scientiﬁc Publishing Co Pte Ltd. Kou, Y. (2010), Development and Evaluation of Integrated Berg, G., Nitsch, V. and Färber, B. (2016), “Vehicle in Chassis Control Systems, University of Michigan, Ann Arbor. the loop”, Handbook of Driver Assistance Systems: Basic Kussmann,H.,Modler,H.,Engstrom,J.,Agnvall,A.,Piamonte, Information, Components and Systems for Active Safety and P., Markkula’s, G., Amditis, A., Bolovinou, A., Andreone, L., Comfort, pp. 199-210. Deregibus, E. and Kompfner, P. (2004), “Requirements for Blanchet, J. and Lam, H. (2012), “State-dependent importance AIDE HMI and safety functions”, AIDE project, March. sampling for rare-event simulation: an overview and recent Lee, K. (2004), Longitudinal Driver Model and Collision Warning advances”, Surveys in Operations Research and Management and Avoidance Algorithms Based on Human Driving Databases, Science, Vol. 17 No. 1, pp. 38-59. University of Michigan, Ann Arbor. Bucklew, J. (2013), Introduction to Rare Event Simulation, Lee, S.E., Olsen, E.C. and Wierwille, W.W. (2004), “A Springer Science & Business Media, Berlin. comprehensive examination of naturalistic lane-changes”, Bühne, J.A., Lüdeke, A., Schönebeck, S., Dobberstein, J., No. HS-809 702. Fagerlind, H., Bálint, A. and McCarthy, M. (2012), Ma, W.H. and Huei, P. (1999), “A worst-case evaluation “Assessment of integrated vehicle safety systems for method for dynamic systems”, Transactions-American Society improved vehicle”, ASSESS D2, Vol. 2 Nos 2/2. of Mechanical Engineers Journal of Dynamic Systems Carsten, O., Merat, N., Janssen, W.H., Johansson, E., Fowkes, Measurement and Control, Vol. 121, pp. 191-199. M. and Brookhuis, K.A. (2005), “Human machine Michalewicz, Z. (2013), Genetic Algorithms 1 Data Structures = interaction and the safety of trafﬁc in Europe ﬁnal report”, Evolution Programs, Springer Science & Business Media. Portal, Leeds, Transp. Res. Innov. Royden, H.L. and Fitzpatrick, P. (1988), Real Analysis, Chang, C.S., Heidelberger, P., Juneja, S. and Shahabuddin, Macmillan, New York. P. (1994), “Effective bandwidth and fast simulation of Russo, R., Terzo, M. and Timpone, F. (2007), “Software-in- ATM INTREE networks”, Performance Evaluation,Vol. 20 the-loop development and validation of a cornering brake Nos 1/3, pp. 45-65. control logic”, Vehicle System Dynamics, Vol. 45 No. 2, Deering, R.K. (2002), “Annual report of the crash avoidance pp. 149-163. metrics partnership, April 2001-March 2002”,No. HS- The Enable-S3 Consortium (2016), “Enable-S3 European 809 531. project”, available at: www.enable-s3.eu/ (accessed 22 Euro, N.C.A.P. (2013), Test Protocol – AEB Systems, Eur. New January 2018). Car Assess. Programme (Euro NCAP), Brussels. Touran, A., Brackstone, M.A. and McDonald, M. (1999), “A Federal Ministry for Economic Affairs and Energy (BMWi) (2016), “Pegasus research project”, available at: www. collision model for safety evaluation of autonomous pegasus-projekt.info/en/ intelligent cruise control”, Accident Analysis & Prevention, (accessed 22 January 2018). Vol. 31 No. 5, pp. 567-578. Gen, M. and Cheng, R. (2000), Genetic Algorithms and Engineering Ulsoy, A.G., Peng, H. and Çakmakci, M. (2012), Automotive Optimization,JohnWiley&Sons,Hoboken,NJ,Vol.7. Control Systems, Cambridge University Press, Cambridge. Gietelink, O., Ploeg, J., De Schutter, B. and Verhaegen, M. Wohllebe, T., Vetter, J., Mayer, C., McCarthy, M. and de (2006), “Development of advanced driver assistance systems Lange, R. (2004), “Integrated project on advanced with vehicle hardware-in-the-loop simulations”, Vehicle protection systems”, AP-SP13-0035 Project, Chalmers, System Dynamics, Vol. 44 No. 7, pp. 569-590. Gothenburg. Glasserman, P. and Li, J. (2005), “Importance sampling for Woodrooffe, J., Blower, D., Bao, S., Bogard, S., Flannagan, portfolio credit risk”, Management Science, Vol. 51 No. 11, pp. 1643-1656. C., Green, P.E. and LeBlanc, D. (2014), “Performance Glynn, P.W. and Iglehart, D.L. (1989), “Importance sampling characterization and safety effectiveness estimates of forward for stochastic simulations”, Management Science, Vol. 35 collision avoidance and mitigation systems for medium/ No. 11, pp. 1367-1392. heavy commercial vehicles”, UMTRI-2011-36. 37 Testing for automated vehicles Journal of Intelligent and Connected Vehicles Yiming Xu, Yajie Zou and Jian Sun Volume 1 · Number 1 · 2018 · 28–38 Yang, H.H. and Peng, H. (2010), “Development and mixture models”, IEEE Transactions on Intelligent Transportation Systems. evaluation of collision warning/collision avoidance algorithms Juneja, S. and Shahabuddin, P. (2006), “Rare-event simulation using an errable driver model”, Vehicle System Dynamics, techniques: an introduction and recent advances”, Handbooks Vol. 48 No. S1, pp. 525-535. in Operations Research and Management Science,Vol. 13, Zhao, D., Huang, X., Peng, H., Lam, H. and LeBlanc, D.J. pp. 291-350. (2017a), “Accelerated evaluation of automated vehicles in Kaempchen, N., Schiele, B. and Dietmayer, K. (2009), car-following maneuvers”, IEEE Transactions on Intelligent “Situation assessment of an autonomous emergency brake Transportation Systems, Vol. 19 No. 3. for arbitrary vehicle-to-vehicle collision scenarios”, IEEE Zhao, D., Lam, H., Peng, H., Bao, S., LeBlanc, D.J., Transactions on Intelligent Transportation Systems,Vol. 10 Nobukawa, K. and Pan, C.S. (2017b), “Accelerated No. 4, pp. 678-687. evaluation of automated vehicles safety in lane-change Kollman, C., Baggerly, K., Cox, D. and Picard, R. (1999), scenarios based on importance sampling techniques”, IEEE “Adaptive importance sampling on discrete Markov chains”, Transactions on Intelligent Transportation Systems,Vol. 18 Annals of Applied Probability, Vol. 9 No. 2, pp. 391-412. No. 3, pp. 595-607. Marsden, G., McDonald, M. and Brackstone, M. (2001), “Towards an understanding of adaptive cruise control”, Further reading Transportation Research Part C: Emerging Technologies,Vol.9 No. 1, pp. 33-51. Blincoe, L., Miller, T.R., Zaloshnja, E. and Lawrence, B.A. Moridpour, S., Sarvi, M. and Rose, G. (2010), “Lane changing (2015), “The economic and societal impact of motor vehicle models: a critical review”, Transportation Letters,Vol. 2 crashes, 2010 (revised) 1”, Annals of Emergency Medicine, No. 3, pp. 157-173. Vol. 66 No. 2, pp. 194-196. Sayer, J., LeBlanc, D., Bogard, S., Funkhouser, D., Bao, S., Fildes, B., Keall, M., Bos, N., Lie, A., Page, Y., Pastor, C., Buonarosa, M.L. and Blankespoor, A. (2011), “Integrated Pennisi, L., Rizzi, M., Thomas, P. and Tingvall, C. (2015), vehicle-based safety systems ﬁeld operational test ﬁnal “Effectiveness of low speed autonomous emergency braking in program report”. real-world rear-end crashes”, Accident Analysis & Prevention, Vahidi, A. and Eskandarian, A. (2003), “Research advances in Vol. 81, pp. 24-29. intelligent collision avoidance and adaptive cruise control”, Huang, Z., Lam, H. and Zhao, D. (2017a), “Towards IEEE Transactions on Intelligent Transportation Systems, Vol. 4 affordable on-track testing for autonomous vehicle – a No. 3, pp. 143-153. Kriging-based statistical approach”, Proceedings of the IEEE Zhao, D., Peng, H., Bao, S., Nobukawa, K., LeBlanc, D.J. and 20th International Intelligent Transportation Systems Conference Pan, C.S. (2016), “Accelerated evaluation of automated (ITSC), Yokohama, 16-19 October 2017. vehicles using extracted naturalistic driving data”, The Huang, Z., Lam, H. and Zhao, D. (2017b), “An accelerated Dynamics of Vehicles on Roads and Tracks: Proceedings of the testing approach for automated vehicles with background 24th Symposium of the International Association for Vehicle trafﬁc described by joint distributions”, Proceedings of the System Dynamics (IAVSD 2015), Graz, Austria, 17-21 August IEEE 20th International Intelligent Transportation Systems 2015, CRC Press, Florida,p.287. Conference (ITSC), Yokohama, 16-19 October 2017. Huang, Z., Lam, H., LeBlanc, D.J. and Zhao, D. (2017c), Corresponding author “Accelerated evaluation of automated vehicles using piecewise Jian Sun can be contacted at: sunjian@tongji.edu.cn For instructions on how to order reprints of this article, please visit our website: www.emeraldgrouppublishing.com/licensing/reprints.htm Or contact us for further details: permissions@emeraldinsight.com

Journal

Journal of Intelligent and Connected Vehicles – Emerald Publishing

Published: Oct 2, 2018

Keywords: Genetic algorithm; Simulation; Automated vehicles; Importance sampling; Lane changing; Safety evaluation; High-risk scenarios

References

Comparison of Markov chain abstraction and Monte Carlo simulation for the safety assessment of autonomous cars

Althoff, M.; Mergel, A.
Vehicle in the loop

Berg, G.; Nitsch, V.; Färber, B.
State-dependent importance sampling for rare-event simulation: an overview and recent advances

Blanchet, J.; Lam, H.
Assessment of integrated vehicle safety systems for improved vehicle

Bühne, J.A.; Lüdeke, A.; Schönebeck, S.; Dobberstein, J.; Fagerlind, H.; Bálint, A.; McCarthy, M.
Human machine interaction and the safety of traffic in Europe final report

Carsten, O.; Merat, N.; Janssen, W.H.; Johansson, E.; Fowkes, M.; Brookhuis, K.A.
Effective bandwidth and fast simulation of ATM INTREE networks

Chang, C.S.; Heidelberger, P.; Juneja, S.; Shahabuddin, P.
Annual report of the crash avoidance metrics partnership, April 2001-March 2002

Deering, R.K.
Pegasus research project
Development of advanced driver assistance systems with vehicle hardware-in-the-loop simulations

Gietelink, O.; Ploeg, J.; De Schutter, B.; Verhaegen, M.
Importance sampling for portfolio credit risk

Glasserman, P.; Li, J.
Importance sampling for stochastic simulations

Glynn, P.W.; Iglehart, D.L.
Fast simulation of rare events in Queueing and reliability models

Heidelberger, P.
Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?

Kalra, N.; Paddock, S.M.
Requirements for AIDE HMI and safety functions

Kussmann, H.; Modler, H.; Engstrom, J.; Agnvall, A.; Piamonte, P.; Markkula’s, G.; Amditis, A.; Bolovinou, A.; Andreone, L.; Deregibus, E.; Kompfner, P.
A comprehensive examination of naturalistic lane-changes

Lee, S.E.; Olsen, E.C.; Wierwille, W.W.
A worst-case evaluation method for dynamic systems

Ma, W.H.; Huei, P.
Software-in-the-loop development and validation of a cornering brake control logic

Russo, R.; Terzo, M.; Timpone, F.
Enable-S3 European project
A collision model for safety evaluation of autonomous intelligent cruise control

Touran, A.; Brackstone, M.A.; McDonald, M.
Integrated project on advanced protection systems

Wohllebe, T.; Vetter, J.; Mayer, C.; McCarthy, M.; de Lange, R.
Performance characterization and safety effectiveness estimates of forward collision avoidance and mitigation systems for medium/heavy commercial vehicles

Woodrooffe, J.; Blower, D.; Bao, S.; Bogard, S.; Flannagan, C.; Green, P.E.; LeBlanc, D.
Development and evaluation of collision warning/collision avoidance algorithms using an errable driver model

Yang, H.H.; Peng, H.
Accelerated evaluation of automated vehicles in car-following maneuvers

Zhao, D.; Huang, X.; Peng, H.; Lam, H.; LeBlanc, D.J.
Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques

Zhao, D.; Lam, H.; Peng, H.; Bao, S.; LeBlanc, D.J.; Nobukawa, K.; Pan, C.S.
The economic and societal impact of motor vehicle crashes, 2010 (revised) 1

Blincoe, L.; Miller, T.R.; Zaloshnja, E.; Lawrence, B.A.
Effectiveness of low speed autonomous emergency braking in real-world rear-end crashes

Fildes, B.; Keall, M.; Bos, N.; Lie, A.; Page, Y.; Pastor, C.; Pennisi, L.; Rizzi, M.; Thomas, P.; Tingvall, C.
Towards affordable on-track testing for autonomous vehicle – a Kriging-based statistical approach

Huang, Z.; Lam, H.; Zhao, D.
An accelerated testing approach for automated vehicles with background traffic described by joint distributions

Huang, Z.; Lam, H.; Zhao, D.
Accelerated evaluation of automated vehicles using piecewise mixture models

Huang, Z.; Lam, H.; LeBlanc, D.J.; Zhao, D.
Rare-event simulation techniques: an introduction and recent advances

Juneja, S.; Shahabuddin, P.
Situation assessment of an autonomous emergency brake for arbitrary vehicle-to-vehicle collision scenarios

Kaempchen, N.; Schiele, B.; Dietmayer, K.
Adaptive importance sampling on discrete Markov chains

Kollman, C.; Baggerly, K.; Cox, D.; Picard, R.
Towards an understanding of adaptive cruise control

Marsden, G.; McDonald, M.; Brackstone, M.
Lane changing models: a critical review

Moridpour, S.; Sarvi, M.; Rose, G.
Integrated vehicle-based safety systems field operational test final program report

Sayer, J.; LeBlanc, D.; Bogard, S.; Funkhouser, D.; Bao, S.; Buonarosa, M.L.; Blankespoor, A.
Research advances in intelligent collision avoidance and adaptive cruise control

Vahidi, A.; Eskandarian, A.
Accelerated evaluation of automated vehicles using extracted naturalistic driving data

Zhao, D.; Peng, H.; Bao, S.; Nobukawa, K.; LeBlanc, D.J.; Pan, C.S.

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Abstract

Journal

Recommended Articles

References

Our policy towards the use of cookies