A Rational Approach to Understanding and Evaluating Responsive Neurostimulation
A Rational Approach to Understanding and Evaluating Responsive Neurostimulation
Sisterson, Nathaniel D.; Wozny, Thomas A.; Kokkinos, Vasileios; Bagic, Anto; Urban, Alexandra P.; Richardson, R. Mark
2020-06-09 00:00:00
Closed-loop brain stimulation is increasingly used in level 4 epilepsy centers without an understanding of how the device behaves on a daily basis. This lack of insight is a barrier to improving closed-loop therapy and ultimately understanding why some patients never achieve seizure reduction. We aimed to quantify the accuracy of closed-loop seizure detection and stimulation on the RNS device through extrapolating information derived from manually reviewed ECoG recordings and comprehensive device logging information. RNS System event logging data were obtained, reviewed, and analyzed using a custom-built software package. Aweighted-means methodology was developed to adjust for bias and incompleteness in event logs and evaluated using Bland–Altman plots and Wilcoxon signed-rank tests to compare adjusted and non-weighted (standard method) results. Twelve patients implanted for a mean of 21.5 (interquartile range 13.5–31) months were reviewed. The mean seizure frequency reduction post-RNS implantation was 40.1% (interquartile range 0–96.2%). Three primary levels of event logging granularity were identified (ECoG recordings: 3.0% complete (interquartile range 0.3–1.8%); Event Lists: 72.9% complete (interquartile range 44.7–99.8%); Activity Logs: 100% complete; completeness measured with respect to Activity Logs). Bland–Altman interpre- tation confirmed non-equivalence with unpredictable differences in both magnitude and direction. Wilcoxon signed rank tests −6 demonstrated significant (p <10 ) differences in accuracy, sensitivity, and specificity at >5% absolute mean difference for extrapolated versus standard results. Device behavior logged by the RNS System should be used in conjunction with careful review of stored ECoG data to extrapolate metrics for detector performance and stimulation. . . . . . Keywords Closed-loop Neuromodulation Seizure detection Device configuration Extrapolation Drug-resistant epilepsy Introduction Electronic supplementary material The online version of this article As early as 1954, Penfield reported the modulatory effects of (https://doi.org/10.1007/s12021-019-09446-7) contains supplementary electrical stimulation on seizures, observed by electrocorticog- material, which is available to authorized users. raphy (ECoG) (Penfield and Jasper 1954). Based on this and * Nathaniel D. Sisterson subsequent observation (Durand 1986; Kinoshita et al. 2004, nds38@pitt.edu 2005; Kossoff et al. 2004), the NeuroPace RNS System was developed as a closed-loop brain modulation device capable of detecting and responding to abnormal brain activity by Department of Neurological Surgery, Brain Modulation Lab, University of Pittsburgh, Pittsburgh, PA, USA delivering programmable stimulation targeted to seizure foci, with the intention of disrupting epileptiform activity before a Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA seizure can develop (Heck et al. 2014). The ability of the device to respond to abnormal brain activity is contingent Department of Neurological Surgery, Massachusetts General Hospital, Boston, MA, USA upon the degree to which the physician-selected combination of detection settings are suited for the particular seizure onset Harvard Medical School, Boston, MA, USA 5 pattern(s) observed in ECoG recordings for each patient (Sun Department of Neurology, University of Pittsburgh, Pittsburgh, PA, et al. 2008). USA 6 The efficacy of the RNS System has been demonstrated by University of Pittsburgh Comprehensive Epilepsy Center, several multi-center outcomes studies, in which a median 55% Pittsburgh, PA, USA 366 Neuroinform (2020) 18:365–375 of patients experienced a 53% reduction in seizure frequency All subjects underwent intracranial monitoring via stereo- at 2 years and a median 70% of patients experienced a 78% electroencephalography (SEEG), prior to RNS therapy (RNS, seizure frequency reduction at 6 years (Bergey et al. 2015; NeuroPace, Mountain View, CA, USA), as recommended by Geller et al. 2017; Jobst et al. 2017). However, patients remain a multidisciplinary epilepsy surgery team. RNS implantations at increased risk for severe complications, such as severe in- occurred consecutively between January 2015 and November juries and sudden unexplained death (SUDEP), during the 2017. All related clinical data were abstracted from the elec- interval to seizure control (Banerjee et al. 2009). The increase tronic medical records systems or obtained by patient surveys. in responder rate and seizure reduction over time raises the ECoG recordings and associated metadata generated by the question of why some patients achieve a faster and greater RNS System were obtained from the PDMS (accessed be- response. tween November 2016 and December 2017). One approach to this question is to explore the characteris- tics of patients who are responders versus non-responders. Implantation and Programming This approach, however, is insufficient to disentangle whether poor patient response is due to suboptimal settings or intrinsic RNS electrodes were targeted to epileptogenic zones identi- patient characteristics, because it is severely confounded by fied during intracranial monitoring via SEEG. No neural stim- the heterogeneity of device behavior between patients. This ulation was delivered for approximately one month after im- heterogeneity results from the interaction between the detec- plantation, in order to observe baseline brain activity and de- tion and stimulation settings and an individual patient’sdis- tector performance. Patients, or their caregivers, were also tinct neurophysiological seizure onset patterns. Thus, we pro- asked to keep a seizure diary to log the time, duration, and pose that the first step to improving closed-loop therapy is to manifestations of seizures. Once the clinical team reached develop quantitative methods for evaluating device behavior, consensus on the presumed acceptable accuracy for the detec- defined by the type of neurophysiological event being stimu- tion parameters, initial stimulation settings were configured lated, the timing of stimulation relative to event onset, and the and enabled so that the device delivered detection-triggered rate of stimulation. This strategy extends significantly beyond stimulation therapy. the scope of current analysis tools available via the NeuroPace The RNS System parameter space can be separated into Patient Data Management System (PDMS). two main categories: detection parameters and stimulation pa- Here, we report our comprehensive analysis of logging rameters (S1 Fig). The three primary types of detectors are information generated by the RNS System. Based on findings bandpass, line length, and area, with additional built-in detec- of incomplete or overwritten information about detections and tors for saturation and noise. Two channels are selected to be stimulations and reporting bias, we hypothesized that extrap- detection channels for a total of two detecting electrode pairs. olation of information derived from manually reviewed ECoG Next, up to two first-order Patterns, referred to as Pattern A recordings using comprehensive aggregate logging informa- and Pattern B are configured. Each Pattern can be further tion (which by itself lacks details about specific detections and comprised of up to two second-order patterns, referred to as stimulations) would reveal a clinically significant and unpre- Pattern A1, Pattern A2, Pattern B1, and Pattern B2. Each of dictable difference in device behavior, as compared to current, these second-order Patterns corresponds to a single detection standard methods available in the PDMS (S1 Table). Finally, channel and detector. Stimulation can occur from all eight we report the results of the extrapolation method in the context electrodes, and settings are configured for up to five consecu- of clinical outcomes, to highlight the heterogeneity of device tive discretely triggered stimulations, referred to as therapies, behavior between patients. comprised of up to two consecutive bursts each. The cathode (positive) and anode (negative) electrode montage and current affect the volume of tissue treated and must be configured for each burst. The bursts of the first therapy only may be config- Material and Methods ured to respond differently to specific Patterns. The total num- ber of therapies in a given 24-h window are also limited by a Study Design and Cohort Selection programmable amount. On an approximately monthly basis, the clinical team We performed a retrospective chart review approved by the reviewed the information recorded by the device, as well as University of Pittsburgh Institutional Review Board (IRB). patient reported outcomes. The team decided whether, and First, we collected RNS System device data and performed how, to adjust the device detection or stimulation parameters an in-depth analysis of the logging capabilities. Second, we based on a combination of summary data obtained from the used the information generated by this analysis to perform a standard PDMS interface and clinical interviews. The period quantitative evaluation of device behavior (detections and of time during which detection and stimulation settings remain consistent is referred to as a programming epoch. The primary stimulations). Neuroinform (2020) 18:365–375 367 factors driving RNS adjustments were the approximate daily calculated the amount of data missing from each level of log- number of events and the number of long episodes and satu- ging, using the Activity Log as the source of truth for com- rations since the most recent programming epoch. This pro- pleteness for interrogation data (ECoG Reports, Event List, cess represents the current, or standard, method for evaluating and Activity Log), and the histogram_data_missing and device behavior. diagnostic_data_missing flags for the daily and hourly Neurostimulator History. Data Acquisition Device Behavior Analysis We obtained ECoG recordings from NeuroPace via an encrypted USB drive on a quarterly basis. To obtain the ma- To evaluate device behavior, we looked at whether or not the jority of the data needed for analysis, including device detec- neurophysiological event being stimulated was an tion and stimulation settings, event timestamps, total number electrographic ictal pattern (EIP). In this study, EIPs represent of events per interrogation, and summary logging data, we probable seizure events, but we use this term since the confir- created a custom HTML parsing tool in C#. NET to program- mation of a seizure event requires additional clinical evidence matically load and transform data from the PDMS into our not available through the RNS System. First, we reviewed local database on a weekly basis (Fig. 1). All data were loaded ECoG recordings and calculated sensitivity, specificity, and in a SQL Server (Microsoft, Redmond, Washington, USA) accuracy of the detectors with respect to EIPs by extrapolating database using extract, transform, and load (ETL) code devel- this information using logging data. Next, we calculated la- oped for SQL Server Integration Services. The database was tency to evaluate the timing of stimulation relative to seizure secured by schema, and the primary data set was de-identified onset. To look at rate of stimulation, we calculated the mean using an automated process. Patient identifiers were stored in number of stimulations delivered per day. We tested our hy- a separately secured database. This database and associated pothesis, that the weighted method would reveal a clinically software used in our subsequent analyses form the basis of the significant difference in device behavior as compared to the BRAINStim© (Biophysically Rational Analysis for Informed standard, non-weighted method, using Bland-Altman plots Neural Stimulation) platform. and Wilcoxon signed rank tests. Finally, we calculated the mean, median, interquartile range, and range of device behav- Logging Analysis iors generated by the extrapolated calculations to demonstrate the heterogeneity of device behavior across patients. We reviewed the different levels of logging available on the All ECoG recordings were reviewed by an experienced PDMS, including ECoG Reports, Event List, Activity Log,and neurophysiologist (VK) to identify the presence or absence Neurostimulator History. ECoG Reports, Event List,and of an EIP. True positive, false positive, true negative, and false Activity Log are complementary log summaries of each inter- negative rates were calculated for each episode from the man- rogation while the Neurostimulator History provides a histo- ually reviewed ECoG data using the initial second-order gram summarizing the same information, separately located Pattern (e.g. Pattern A1, Pattern A2, etc.) to trigger (S2 on the PDMS. We assessed the level of detail available and Fig). Each second-order Pattern was further grouped by Fig. 1 Flow diagram of data loading, pre-processing, manual review, and which are exported as .EDF files. d ECoG data are manually reviewed, calculations using the BRAINStim© platform. a Data crucial to the and EIP onset and laterality are annotated. All data are imported back into analysis of RNS System performance are loaded from the PDMS using the database and merged with the original files. e Weighted calculation a custom C#.NET HTML parsing tool. b Raw ECoG data, along with scripts are executed on the database. f The results of the scripts are loaded hardware diagnostic information, are loaded from files provided by back into the database and used to generate figures, as well as to facilitate NeuroPace. c ECoG data are sorted into groups by programming epoch, further analysis 368 Neuroinform (2020) 18:365–375 episode length (episode versus long episode). To perform EIP (true positive) were used, and recordings beginning with weighted calculations, standard ECoG calculations were an in-progress episode were excluded. The weighted mean weighted using the percentage of each trigger (e.g. Pattern paradigm was similarly used for each patient to calculate mean A1, Pattern A2, etc.) and pattern type (e.g. episode, long epi- number of stimulations per episode and EIPs per month. The sode), as reported by the daily Neurostimulator History.As mean pattern detection rate per hour, total events per hour, shown below, Pattern accuracy for all detectors stratified by percentage of days reaching the daily therapy limit, number episodes (A1_ E ) and long episodes (A1_ LE ) of programming epochs, and time to enabling stimulation ECoGAcc ECoGAcc were calculated first. Next, the weights were calculated using were calculated directly from log data (no weighting was the percentage of times a given Pattern triggered an episode necessary). multiplied by (A1_ E ) the percentage of Patterns for the The number of ECoG recordings stored on the device at ECoG entire epoch (A1 ). Long episode weights were calculated any given time is limited to approximate four 90-s 4-channel History by multiplying the percentage of times a given Pattern trig- recordings, which represent only a subset of all recorded gered a long episode by the percentage of long episodes for events due to constant overwriting of data. To determine the the entire epoch (LE ). Finally, the weights were applied extent to which stored ECoGs may adequately reflect true History to each detector stratified by episode versus long episode and overall device behavior, we applied a weighted mean method- summed, resulting in the weighted accuracy for the program- ology to ECoG-dependent calculations (detector sensitivity, ming epoch (PE ). specificity, accuracy, number of EIPs, and number of thera- Acc pies) and compared these results to those obtained using the standard (non-weighted) PDMS methodology using Bland– A1 E ¼ðÞ A1 E þ A1 E ECoG ECoG ECoG Acc TP TN Altman plots (Bland and Altman 1986;Giavarina 2015). To test our hypothesis, we performed Wilcoxon signed rank tests = A1 ðÞ E þ A1 E þ B1 E þ B2 E ECoG ECoG ECoG ECoG TP TN TP TN of extrapolated versus standard calculations for the accuracy, sensitivity, and specificity of each patient programming ep- och, with clinical significance set to >5%. A1 E ¼ A1 E =ðÞ A1 E þ A2 E þ B1 E þ B2 E WT ECoG ECoG ECoG ECoG ECoG We quantified heterogeneity of device behavior between History patients by calculating the mean, median, 25th and 75th per- A1 = A1 þ A2 þ B1 þ B2 History History History History History centiles, and range of the accuracy, sensitivity, specificity and latency obtained by the extrapolated calculations. We addi- tionally calculated the total number of stimulations delivered A1 LE WT History to the left versus right hemisphere at 8.5 months post- implantation for patients with bilateral lead placements, again ¼ A1 LE =ðÞ A1 LE þ A2 LE þ B1 LE þ B2 LE ECoG ECoG ECoG ECoG ECoG using the extrapolated results. LE = LE þ E History History History Clinical Data Acquisition and Analysis We performed a retrospective review of the UPMC electronic PE ¼ A1 E A1 E þ A2 E Acc ECoG WT ECoG Acc History Acc medical record to obtain medical history, surgical history, sei- A2 E þ B1 E B1 E WT ECoG WT History Acc History zure classifications, anti-epileptic drug regimens, and natural course of disease. Due to the incomplete and inconsistent doc- þ B2 E B2 E þ A1 LE ECoG WT ECoG Acc History Acc umentation of seizure activity, we evaluated clinical outcomes A1 LE þ A2 LE using two complementary patient reported outcomes surveys. WT ECoG History Acc To measure quality of life outcomes, we used the Personal A2 LE þ B1 LE WT ECoG History Acc Impact of Epilepsy Scale (PIES) instrument, a compact 25 question survey which detects changes in impact of seizures, B1 LE þ B2 LE WT ECoG History Acc medication side effects, impact of comorbidities, and overall B2 LE WT History quality of life (Fisher et al. 2015). To measure clinical seizure outcomes, we used as a custom questionnaire to document seizure type and frequency, duration, intensity, and loss of We repeated this weighted-mean methodology to calculate consciousness, as well as seizure diary and magnet swipe detector latency, defined as the number of seconds elapsed compliance (S3 Fig) (Sun and Morrell 2014). These question- between the manually marked EIP onset and the first detection naires were administered to RNS patients at their most recent event of the first episode of a given ECoG recording. Only clinic visit, at which time they reported both pre- and post- recordings in which the RNS System correctly identified an RNS outcomes. Neuroinform (2020) 18:365–375 369 We calculated the mean, median, interquartile range, and saturation, and magnet swipe events), was 3.0% (median range for age at time of implant, years of uncontrolled sei- 0.6%; interquartile range 0.3–1.8%; range 0–39.8%). The re- zures, months implanted with the RNS System, and number mainder of the files were either not configured for storage and/ of failed anti-seizure drugs (ASDs). Quality of life was quan- or overwritten due device storage constraints. tified using the PIES instrument. Both the percent and abso- The next most detailed source of information comes from lute improvement in pre- and post-RNS survey scores were the Event List, which includes the episode onset timestamp calculated and compared for each patient to determine wheth- and a summary of the second-order Patterns and therapies er quality of life improved, declined, or remained the same. that occurred. The Event List canhold up toapproximately Both the percent and absolute change in seizure frequency, 700 events between interrogations, after which storage capac- duration, and severity pre- and post-RNS survey results were ity is reached and additional new events are lost. Relative to compared for each patient. Finally, we used Pearson correla- total events, the Event Lists were 72.9% (median 80.0%; in- tion to quantify the degree to which EIPs correlate with patient terquartile range 44.7–99.8%; range 16.2–100%) complete in reported seizures for a corresponding one month period. All our cohort. Finally, the Activity Log contains summary data for statistical analyses were performed using MATLAB first-order Patterns during the time period since the previous (MathWorks, Natic, MA, USA) and R (R Core Team, interrogation. The mean number of hours between interroga- Vienna, Austria). tions was 30.3 (median 24; interquartile range 23–26; range 0–2583). The storage available for this information is suffi- cient such that the device is unlikely to ever lose any summary Results data. Separately, the Neurostimulator History contains daily summary data going back up to 10 years and hourly summary Demographics and Adverse Events data going back up to 180 days. Summary data is overwritten at 255 events in a one-hour period. Relative to total events, the Twelve patients were included in this analysis, with a mean Neurostimulator History was 84.9% (median 100%, inter- responsive stimulation therapy duration of 21.5 months (S2 quartile range 72.0–100%, range 37.8–100%) complete in Table; median 23.5; interquartile range 13.5–31; range 5–34). our cohort. Five out of 12 patients exhibited epilepsy of structural etiolo- gy, as defined by the ILAE classification system (S3 Table) Device Behavior Results (Berg et al. 2010;Fisher 2017). A total of 14,394 ECoG files were processed, encompassing a total of 198.9 RNS- The mean time to enabled neural stimulation therapy was implanted months. Analysis revealed a total number of 4827 45.9 days (median 37 days, interquartile range 29–66.75 days, EIPs, with a mean of 71 EIPs per programming epoch (median range 8–90 days). The mean number of programming events 24; interquartile range 6.5–98; range 1–780). One patient was 3.7 per year per patient (median 3.5, interquartile range underwent resection and electrode repositioning 366 days af- 2.8–4.3, range 1.8–6). Responsive stimulation activity was ter RNS-recorded ECoG revealed primarily unilateral seizure characterized by a mean of 39.5 events per hour per patient onset, and subsequent data were excluded from our analyses. (median 26.6; interquartile range 8.7–51.2; range 0–244.5). Another patient experienced throbbing headaches post- The mean number of stimulations per episode per patient implantation that resolved within one month. There were no was 1.1 (median 1.1, interquartile range 1.0–1.2, range 0–5), other adverse events recorded and no surgical infections. and the daily therapy limit (mean 2039.2, median 2000, inter- quartile range 1000–3000, range 1000–4000) was reached for Logging Results 5.6% of the total treatment days (median 1.2%, interquartile range 0.2–8.6%, range 0–27.9%). We found that the RNS System is limited in the granularity of To evaluate the difference between standard and extrap- information reliably captured and made available via the olated methods, we calculated both standard (S) means and PDMS (Fig. 2). The richest source of information comes from means weighted by Neurostimulator History data (W), for stored ECoG recordings, which includes up to 4 channels of detection accuracy (S = 90.0%, median 94.8%, interquar- ECoG and corresponding event timestamps identifying tile range 84.3–98.8%, range 63.0–100%; W = 85.1%, me- second-order Patterns and therapies. Because the onboard dian 89.7%, interquartile range 75.9–98.2%, range 45.0– storage space available for ECoG recordings on the device is 100%), sensitivity (S = 51.5%, median 50.0%, interquartile extremely limited, only a small subset of recordings is pre- range 48.7–50.1%, range 0–100%; W = 68.1%, median served relative to the continuous neural signal analyzed online 74.9%, interquartile range 48.4–94.5%, range 0–100%), by the device. The mean percentage of triggered ECoG re- and specificity (S = 93.2%, median 96.7%, interquartile cording files uploaded to the PDMS per programming epoch, range 90.6–99.6%, range 64.5–100%; W = 84.6%, median relative to total events (defined as the sum of pattern detection, 92.4%, interquartile range 75.9–99.6%, range 34.5–100%), 370 Neuroinform (2020) 18:365–375 3.0% 72.9% 100% 84.9% 87.7% Fig. 2 Levels of RNS System detailed and summary logging. logging data used to perform our analyses. Logging data (top) comes Completeness of each source of logging information relative to the from the Reports tab for ECoG recordings (most detailed) and the inter- Activity Log, which is the most complete but least detailed logging rogation Event List and Activity Log (most complete); additional summary source, is shown in pie charts. Representative screen captures of the data (bottom) comes from the Neurostimulator History tab for daily and NeuroPace PDMS provide a basic reference of the primary sources of hourly histogram data for each epoch between programming changes (S4 Table). stimulation preceded seizure onset in these cases. We used The weighted mean detector latency was negative for three Bland–Altman plots to evaluate the difference between of 71 total programming epochs evaluated (−1.073 s, standard and weighted average calculations, which re- −0.278 s, −0.596 s), indicating that detection and vealed a bias that was unpredictable in both magnitude Fig. 3 a Bland–Altman plots. For weighted accuracy, there was a nega- moderate dispersion from the bias with a positive trend between mean and tive bias with significant dispersion, and a slight positive trend between difference, with greater scatter as the mean decreases. b Absolute mean the mean and difference, with greater scatter as the mean decreases. For difference of standard and weighted calculations. Accuracy, sensitivity, weighted latency, there was a negative bias with minimal dispersion, and and specificity all have statistically, as well as clinically significant dif- some scattering at all values. For weighted sensitivity, there was a positive ferences (defined as a difference of >5%) in extrapolated versus standard bias with significant dispersion and no clear trend, but with less scatter calculations. The difference between extrapolated and standard latency above a mean of 80%. For weighted specificity, there was a negative bias calculations was not statistically significant Daily Histo. Hourly Histo. Ac vity Log Event List ECoG Neuroinform (2020) 18:365–375 371 and direction, with limits of agreement ≥8% for all mea- rate of stimulation therapy between patients, with some pa- surements (Fig. 3a). Wilcoxon signed rank tests were sig- tients receiving greater than 10 times the charge amount, or −6 nificant (p <10 ) with >5% difference between extrapo- dosage, as others (Fig. 4b). In half of the patients with a bilat- lated and standard methods for accuracy, sensitivity, and eral lead configuration, there was a difference in the rate at specificity (Fig. 3b). which stimulation therapy was delivered to the left and right The mean, median, 25th and 75th percentiles, and range for hemispheres of ≥10%, as well. accuracy, sensitivity, specificity, had a large spread with an interquartile range of >20% for all metrics (Fig. 4a). Latency Clinical Results had a tighter distribution but still with a large range (−1.1 s– 6.1 s). The mean total number of stimulations at 8.5 months The mean score for patient compliance to using the RNS magnet 2 2 post-implant was 355,681 μC/cm (median 218,710 μC/cm , to mark a seizure event was 2 (Almost never–Half of the time; interquartile range 165,976–510,244 μC/cm , range 73,556– median 1, interquartile range 1–4, range 1–5), and compliance 932,899 μC/cm ). The histogram revealed a difference in the with maintaining a seizure diary was 3 (Half of the time; median 2, interquartile range 1–3.75, range 1–5). One patient’s primary caretaker passed away and PIES could not be administered. Another patient moved away and could not be reached. A third patient does not currently upload device recordings, due to not having had any clinical seizures since the device was implanted. The mean patient reported seizure frequency was 59.5 per month (median 30.4, interquartile range 10.3–45.1, range 0– 327.4), prior to RNS implantation. The mean reduction in patient reported seizure frequency was 40.1% (median 66.7%, interquartile range 0–96.2%, range − 11.4–100%) with an absolute reduction of 10.9 seizures per month. The mean reduction in seizure duration was 35.8% (median 8.7%, interquartile range 0–82.0%, range − 9.1–100%), and the mean reduction in severity score was 31.6% (median 26.3%, interquartile range 1.7–80.8%, range 0–100%), with an abso- lute reduction of 1.9 points. The mean time to follow-up was 21.5 months (Table 1; median 22.3, interquartile range 8.8– 33.3, range 5.8–36.5). An Engel score of III or better was achieved in 7 patients (58.3%). Pearson correlation showed a trend towards positive correlation between the number of patient reported seizures and corresponding EIPs over a one month period (r = 0.61, p =0.06). Discussion We established an analysis pipeline to quantify the extent of bias in electrophysiological event data reported from the RNS System. We found significant bias in how events are reported, secondary to data storage limitations on the device. Our find- ings demonstrate that standard analysis methods available via Fig. 4 a Boxplots of mean, median, 25th and 75th percentiles, and range of accuracy, sensitivity, specificity, and latency. Wide interquartile ranges the PDMS to understand device behavior can provide mis- for accuracy, sensitivity, and specificity (left axis), with disagreement leading results. For this reason, we developed a weighted- between the mean (dashed) and median (solid), revealing a heterogeneous means methodology to extrapolate device behavior and par- and widely distributed differences in device behavior between patients. tially compensate for data incompleteness and bias. Latency (right axis) has a relatively narrower interquartile range, a wide range still exists. b Total stimulation per patient at 8.5 months post-im- plant. There is significant variability in the rate at which stimulation Logging Incompleteness and Bias therapy is delivered between patients, with some patients receiving great- er than 10 times the amount, or dosage, as others. For patients with a While it is understood that the RNS System has limited stor- bilateral lead configuration, the rate at which stimulation therapy is de- livered to the left and right hemispheres can be uneven, as well age capacity, the degree to which its recordings are incomplete 372 Neuroinform (2020) 18:365–375 Table 1 Patient reported clinical outcomes. Clinical outcomes were measured using a custom seizure questionnaire, the Personal Impact of Epilepsy Scale (PIES) tool developed by Fisher, et al, and Engel classification. Both the percent and absolute changes are reported. Better outcomes are indicated by negative numbers for the custom seizure questionnaire and positive number for PIES. Lead locations categorized as cortical dysplasia (CD), neocortical (NC), or mesial temporal (MT). Patient Implant Seizure Change % (Abs) PIES Change % (Abs) Engel Class (lead loc.) (months) Frequency Duration Severity Sec. A Sec. B Sec. C QOL Overall (per month) (seconds) (points) (SEZ) (MED) (COM) RNS1090 5.6 −98.3% 0% −33.3% +1.7% +7.3% 0% 0% +2.6% IIB (CD) (−45.2) (−2) (+1) (+3) (+4) RNS1556 5.3 0% 0% −4.3% −15.8% 0% 0% 0% −5.0% IIB (CD) (−1) (−9) (−9) RNS1836 19.3 −90.0% −92.7% −81.8% +13.3% +1.9% +6.7% +60.0% +7.7% IIIA (CD) (−39.1) (−38) (−9) (+8) (+1) (+2) (+3) (+11) RNS9536 33.6 −66.7% +100% +11.1% +15.9% +6.9% 0% 0% +6.9% IIIA (CD) (−34.7) (+20) (+1) (+4) (+4) (+15) RNS1597 20.5 0% 0% −26.3% – – ––– IVA (MT) (−5) RNS1529 23.9 0% 0% 0% 0% 0% −2.7% −33.3% −0.9% IVB (MT) (−1) (−1) (−1) RNS1534 10.8 0% 0% 0% 0% 0% 0%0%0%IVB (MT) RNS2227 – – – – – – ––– IVC (MT) RNS1603 27.4 −100% −100% −100% – −8.0% +9.3% +14.3% – IB (NC) (−6.5) (−10) (−3) (−4) (+7) (+1) RNS4098 31.8 −100% −100% (−1800) −100% +31.0% (+18) +5.7% + +66.7% + IB (NC) (−1) (−5) (+3) 53.6- (+4) 30.5- % % (+30) (+51) RNS8076 – – – – – – ––– IIA (NC) RNS1440 31.8 0% 0% 0% 0% 0% 0%0%0%IVB (NC) Neuroinform (2020) 18:365–375 373 and the potential impact on clinical evaluation of recorded and stimulation immediately preceded the EIP onset. One po- events previously has not been shown. This incompleteness tential explanation is that false positive stimulation may inad- can be due to overload of detected events (either true interictal vertently precipitate an EIP event. Such an occurrence does not activity or due overly sensitive detection settings), frequency necessarily correlate to a clinical seizure, and preemptive trig- of patient interrogations, insufficient uploads to the laptop, gering of a subclinical event could be one of the mechanisms by configuration of which recorded events to store, and when which the device therapeutically changes seizure dynamics. the event occurred relative to device interrogation, as only the most recent recordings are saved. The RNS System over- Clinical Response writes the oldest ECoG recordings once the storage limit is reached, which creates a temporally biased sample. In addi- Approximately half of our patients demonstrated a good clinical tion, selection bias is introduced when storage slots are allo- response to the RNS System at ≥20 months implanted. This cated to particular triggers, such as magnet swipes, scheduled finding further corroborates the RNS clinical trial results of a recordings, Pattern A, and Pattern B. Despite recent FDA 50% responder rate at 2 years. Changes in seizure frequency, approval of the second generation RNS System, which has duration, and severity may reflect the modulatory effects of the double the storage capacity compared to the current genera- RNS System, and these quantifiable endpoints should be moni- tion, these biases will persist. tored to evaluate treatment efficacy. It may be possible to detect these changes using data captured by the RNS System to extrap- Device Behavior Reliability and Heterogeneity olate the frequency, duration, and severity of EIPs. For example, patient reported seizure frequency correlated strongly with EIPs While standard detector performance often reflected the gen- (despite a borderline p value), further indicating that the latter eral nature of weighted detector performance, significant dif- may be a useful metric for evaluating patient response. ferences in magnitude of the detection values were observed. Interpretation of the Bland–Altman plots revealed non- equivalence in all calculations, indicating that the ECoG re- Limitations ports presented on the PDMS represent a significantly biased sample of overall RNS System activity. Further, weighted This study was undertaken to address the limitations of working means were not consistently higher or lower than standard with 90-s snippets of chronic data. While we believe our work means, demonstrating that general inferences about the direc- shows it is possible to improve upon current methods, we ac- tion or magnitude of bias cannot be made. Consequently, ob- knowledge it is not possible to fully compensate for missing data. servations made solely using ECoG reports on the PDMS can The RNS System’s ability to perform extended recordings is ex- mislead a clinician to unnecessarily modify a well-performing tremely limited, as the propriety wand must be continually pressed detector or continue a poorly performing detector. It is crucial to the scalp overlying the device. Because continuous monitoring to appreciate that the ECoG recordings may represent only a is not possible, the potential to miss EIPs (false negatives versus small and temporally biased view of what the RNS System is true negatives) is inherent due to reliance on detector performance, actually detecting and stimulating. so we cannot be certain EIPs, and by inference clinical seizures, Significant variability in detection accuracy was also ob- are not missed. In lieu of continuous recording, we used scheduled served, where the percentages of types of neurophysiological recordings to gain insight into brain activity that does not trigger events being stimulated varied between patients. Likewise, we the detector. In the future, physicians may consider continuous observed extensive variability in the rate of stimulation, with scalp EEG post-implantation to augment weighted calculations some patients receiving stimulation at greater than 10 times the with a more complete neurophysiologic record. frequency of other patients. This heterogeneity with regard to The PDMS does not display raw data but rather is an ab- both what is being stimulated and how frequently the stimula- straction layer that transforms the raw data into something tion is occurring should be considered when adjusting detection readable by a clinician. As a result, there are sometimes in- and stimulation parameters, particularly for patients who do not consistencies and inaccuracies in what is reported on the achieve the reported average seizure reduction rates. PDMS. For example, a magnet swipe may be recorded during Additionally, the challenge of adjusting detection thresholds is an ECoG episode but not displayed in the Event List.In an- made more complex by the fact that seizure networks are evolv- other scenario, programming changes made to RNS System ing both via natural history of disease and by modulatory effects may be missing from the Programming Epochs tab if they are of stimulation (Kokkinos et al. 2019). As a result, there is an not properly uploaded from the programming laptop. expected need to regularly optimize detection and stimulation The number of patients in this cohort limits statistical anal- based on monitoring of the closed-loop system. ysis. However, the primary focus of this study is to show that The finding of a negative detector latency in three of 71 current methods for evaluating device behavior are biased and programming epochs is unexpected as it indicates that detection potentially misleading. A larger cohort will be necessary to 374 Neuroinform (2020) 18:365–375 address the question of how device behavior correlates with References closed-loop therapy outcomes. Other potential confounders in Banerjee, P. N., Filippi, D., & Hauser, W. A. (2009). The descriptive correlating RNS System detector performance and stimulation epidemiology of epilepsy - a review. Epilepsy Research, 85(1), settings with outcomes are changes to the patient’s ASD reg- 31–45. https://doi.org/10.1016/j.eplepsyres.2009.03.003.The. imen, heterogeneity of seizure foci location and etiology, and Baud, M. O., Kleen, J. K., Mirro, E. A., Andrechak, J. C., King-Stephens, the effect of multidien rhythms in EIP activity on the temporal D., Chang, E. F., & Rao, V. R. (2018). Multi-day rhythms modulate seizure risk in epilepsy. Nature Communications, 9(1), 88. https:// bias of ECoG recordings (Baud et al. 2018). doi.org/10.1038/s41467-017-02577-y. Berg,A.T., Berkovic,S.F., Brodie, M. J., Buchhalter, J., Cross, J. H.,Van Emde Boas, W., et al. (2010). Revised terminology and concepts for organization of seizures and epilepsies: Report of the ILAE commission Conclusions on classification and terminology, 2005-2009. Epilepsia, 51(4), 676– 685. https://doi.org/10.1111/j.1528-1167.2010.02522.x. Bergey, G. K., Morrell, M. J., Mizrahi, E. M., Goldman, A., King- In ordertoestablishamore sophisticated knowledge base from Stephens, D., Nair, D., et al. (2015). Long-term outcomes of treat- which to ground RNS System programming, we undertook a ment with responsive brain stimulation in adults with refractory detailed evaluation of the device capabilities and then built a partial seizures. Neurology, 84(8), 810–817. https://doi.org/10. novel data acquisition and analysis platform to aid in 1212/WNL.0000000000001280. Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing interpreting clinical and ECoG results in the context of the vast agreement between two methods of clinical measurement. Lancet parameter space that exists for recording and stimulating. (London, England), 1(8476), 307–310. https://doi.org/10.1016/ Activity logged by the RNS System is biased and incomplete S0140-6736(86)90837-8. but can be used, in conjunction with manual review of stored Durand, D. (1986). Electrical stimulation can inhibit synchronized neu- ronal activity. Brain Research, 382(1), 139–144. ECoG data, to extrapolate metrics for understanding device Fisher, R. S. (2017). The new classification of seizures by the internation- behavior. These extrapolated metrics can be unpredictably and al league against epilepsy 2017. Current Neurology and significantly different from those derived from currently avail- Neuroscience Reports, 17(6), 1–6. https://doi.org/10.1007/s11910- able metrics on the PDMS. This new framework is useful as an 017-0758-6. endpoint for device behavior, especially in the context of mon- Fisher, R. S., Nune, G., Roberts, S. E., & Cramer, J. A. (2015). The personal impact of epilepsy scale (PIES). Epilepsy & Behavior, 42, itoring patient response to parameter changes, thereby provid- 140–146. https://doi.org/10.1016/j.yebeh.2014.09.060. ing basis for adjusting detection and stimulation settings. Geller, E. B., Skarpaas, T. L., Gross, R. E., Goodman, R. R., Barkley, G. L., Bazil, C. W., et al. (2017). Brain-responsive neurostimulation in patients with medically intractable mesial temporal lobe epilepsy. Information Sharing Statement The MS SQL Server database Epilepsia, 58(6), 994–1004. https://doi.org/10.1111/epi.13740. Giavarina, D. (2015). Understanding Bland Altman analysis. Biochemia DDL, query statements used to process and analyze data for Medica, 25(2), 141–151. https://doi.org/10.11613/BM.2015.015. this manuscript, as well as MATLAB 2017a scripts used for Heck, C. N., King-Stephens, D., Massey, A. D., Nair, D. R., Jobst, B. C., analysis are available to the community at https://github.com/ Barkley, G. L., et al. (2014). Two-year seizure reduction in adults Brain-Modulation-Lab/rational-approach. For assistance with with medically intractable partial onset epilepsy treated with respon- obtaining RNS System generated data, or loading data into the sive neurostimulation: Final results of the RNS system pivotal trial. Epilepsia, 55(3), 432–441. https://doi.org/10.1111/epi.12534. DDL, please feel free to contact us. Jobst,B.C.,Kapur,R.,Barkley,G.L.,Bazil,C.W.,Berg,M.J.,Bergey,G.K., et al. (2017). Brain-responsive neurostimulation in patients with medical- ly intractable seizures arising from eloquent and other neocortical areas. Acknowledgements NDS and TAW were trainees in the Physician Scientist Epilepsia, 58(6), 1005–1014. https://doi.org/10.1111/epi.13739. Training Program (PSTP) at the University of Pittsburgh School of Medicine. Kinoshita, M., Ikeda, A., Matsumoto, R., Begum, T., Usui, K., The authors thank NeuroPace, Inc. for assistance with data transfers and for Yamamoto, J., et al. (2004). Electric stimulation on human cortex clarifying ambiguities in RNS System documentation. suppresses fast cortical activity and epileptic spikes. Epilepsia, 45(7), 787–791. https://doi.org/10.1111/j.0013-9580.2004.60203.x. Kinoshita, M., Ikeda, A., Matsuhashi, M., Matsumoto, R., Hitomi, T., Begum, T., et al. (2005). Electric cortical stimulation suppresses Open Access This article is licensed under a Creative Commons epileptic and background activities in neocortical epilepsy and me- Attribution 4.0 International License, which permits use, sharing, sial temporal lobe epilepsy. Clinical Neurophysiology, 116(6), adaptation, distribution and reproduction in any medium or format, as 1291–1299. https://doi.org/10.1016/j.clinph.2005.02.010. long as you give appropriate credit to the original author(s) and the Kokkinos, V., Sisterson, N. D., Wozny, T. A., & Richardson, R. M. source, provide a link to the Creative Commons licence, and indicate if (2019). Association of Closed-Loop Brain Stimulation changes were made. The images or other third party material in this article Neurophysiological Features with Seizure Control among Patients are included in the article's Creative Commons licence, unless indicated with Focal Epilepsy. JAMA Neurology, 76(7), 800–808. https://doi. otherwise in a credit line to the material. If material is not included in the org/10.1001/jamaneurol.2019.0658. article's Creative Commons licence and your intended use is not Kossoff, E. H., Ritzl, E. K., Politsky, J. M., Murro, A. M., Smith, J. R., permitted by statutory regulation or exceeds the permitted use, you will Duckrow, R. B., et al. (2004). Effect of an external responsive need to obtain permission directly from the copyright holder. To view a neurostimulator on seizures and electrographic discharges during copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Neuroinform (2020) 18:365–375 375 subdural electrode monitoring. Epilepsia, 45(12), 1560–1567. Sun, F. T., Morrell, M. J., & Wharen, R. E. (2008). Responsive cortical https://doi.org/10.1111/j.0013-9580.2004.26104.x. stimulation for the treatment of epilepsy. Neurotherapeutics : the journal of the American Society for Experimental NeuroTherapeutics, 5(1), 68– Penfield, W., & Jasper, H. (1954). Epilepsy and the functional anatomy of 74. https://doi.org/10.1016/j.nurt.2007.10.069. the human brain. Boston: Little Brown. Sun, F. T., & Morrell, M. J. (2014). Closed-loop Neurostimulation: The clinical experience. Neurotherapeutics, 11(3), 553–563. https://doi. Publisher’sNote Springer Nature remains neutral with regard to juris- org/10.1007/s13311-014-0280-3. dictional claims in published maps and institutional affiliations.
http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png
Neuroinformatics
Springer Journals
http://www.deepdyve.com/lp/springer-journals/a-rational-approach-to-understanding-and-evaluating-responsive-xXCYX1cF3h