Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device

Brain-computer interface for the communication of acute patients: a feasibility study and a... Brain-Computer interfa Ces, 2016 VoL. 3, no . 4, 197–215 http://dx.doi.org/10.1080/2326263X.2016.1254403 OPEN ACCESS Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device a,b,c,d c a,b,c h Louis Mayaud   , Salvador Cabanilles , Aurélien Van Langhenhove , Marco Congedo   , h a,b,c e f g b,c Alexandre Barachant , Samuel Pouplin , Sabine Filipe , Lucie Pétégnief , Olivier Rochecouste , Eric Azabou , c a a,b,c a,b,c Caroline Hugeron , Michèle Lejaille , David Orlikowski and Djillali Annane a b inserm, Centre d’i nvestigation Clinique et d’innovation technologique (CiC-it ), umr805, Garches, f rance; inserm, equipes t hérapeutiques innovantes et t echnologies appliquées aux troubles neuromoteurs, u . 1179, Garches, f rance; Hôpital r aymond poincaré, apHp, Garches, f rance; d e f mensia technologies sa, paris, f rance; Department DtBs, Cea/Leti, Grenoble, f rance; DiXi microtechniques medical, Besançon, f rance; g h Davidson Consulting, rennes, f rance; Gipsa-Lab, Cnrs, university of Grenoble- alpes, Grenoble i nstitute of t echnology, Grenoble, f rance ABSTRACT ARTICLE HISTORY received 12 m arch 2016 This study presents the outcome of the 5-year-long French national project aiming at the a ccepted 26 o ctober 2016 development and evaluation of an effective brain-computer interface (BCI) prototype for the communication of patients with acute motor disabilities. It presents results from two clinical studies: KEYWORDS a clinical feasibility study carried out partly in the intensive care unit (ICU) and the clinical evaluation assistive technology; of an innovative BCI prototype. In this second study the BCI performance of patients was compared communication to that of healthy volunteers and benchmarked against a traditional assistive technology (scanning aids; event-related device). Altogether, 15 of 22 patients could control the BCI system with an accuracy significantly potentials, p300 speller; electroencephalography; above the chance level. The bit-rate of the traditional assistive technology proved superior, even r iemannian geometry though an equivalent bit-rate could be achieved using personalized parameters for the BCI. Fatigue was found to be the primary limitation factor, which was particularly true for patients and during the use of the BCI. A classifier based on Riemannian geometry was found to contribute significantly to the accuracy of the BCI system. This study demonstrates that the communication of patients with severe motor impairments can be effectively restored using an adequately designed BCI system. All electrophysiological data are freely available at the Physionet.org platform. Introduction facilitate computer access are available [3]. In particular, occupational therapists equip patients with ‘scanning’ Lack of communication in the hospital may be a great systems [4–6] enabling interaction with computers or source of distress for both patients and caregivers, which other devices such as speech synthesizers. The assistive is particularly exacerbated in the acute context of inten- technology (AT) in this case consists of a ‘click interface’ sive care units (ICUs) [1]. Several neurological disorders (or switch) connected to an application where a limited impair communication abilities while leaving cognitive number of options are visually ‘scanned’ (highlighted) for capacities almost untouched. This has long been reported the patient. er Th e are many types of available switches to for chronic conditions such as myopathies, spinal cord control a scanning system [7]: muscular switches, tactile injuries, amyotrophic lateral sclerosis (ALS), and multiple switches, puff switches, and mechanical switches, the most sclerosis (MS), but is also true for acute neuropathies such common type in France, which consist of a press-down as the Guillain-Barré syndrome (GBS). Moreover, quad- button whose activation force, size, location (feet, near the riplegic or locked-in syndrome (LIS) patients undergoing head, thumb), and type of activation (validation on press- invasive mechanical ventilation (via endotracheal tube or down or release) are set according to the patient’s specific tracheotomy) see their communication abilities impaired condition. Whenever a reliable switch command can be by the sudden loss of speech [2]. obtained, the system is connected to a software interface. e u Th se of computers in this context can facili- Typically, this consists of a grid of symbols displayed on a tate communication and many technical solutions to CONTACT Louis mayaud louis.mayaud@gmail.com © 2016 t he a uthor(s). published by informa uK Limited, trading as t aylor & f rancis Group. t his is an open a ccess article distributed under the terms of the Creative Commons a ttribution-nonCommercial-noDerivatives License (http://creativecommons.org/licenses/by-nc- nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 198 L. MAYAUD ET AL. screen, which are flashed at a regular pace (i.e. the ‘scan- path to low-cost EEG technology for the general public, ning’) until the patient selects one. The configuration of although the quality of the signal (hence the overall BCI both the physical and the software interface requires a performance) has proved inferior to research-grade EEG tremendous amount of time and expertise, which is one acquisition systems [25,26]. Thus, to date, a state-of-the-art reason why, as a matter of fact, patients with severe motor EEG-based BCI requires the support of a technical and disabilities in acute contexts such as the ICU are oen ft left clinical team composed of highly qualified professionals unequipped. [27–29]. A major concern for the present study is the e b Th rain-computer interface (BCI) is a longstanding possibility of generating and processing visual ERPs in an technology [8] that translates the brain electrical activ- ICU environment overflowing with uncontrolled sources ity into a command for a device [9,10]. BCIs are usually of noise, both acoustic (alarms, sta, ff mechanical ventila- characterized by: (1) a brain activity recording modality, tion) and electromagnetic (bedside monitors, automated (2) a paradigm that relates sensorial stimulations to some syringes, mechanical ventilation). While EEG applications specific brain activity, and (3) an interface connecting to are known to be particularly sensitive to non-controlled the front-end application by means of pattern-recogni- environments [16,30], to the best of our knowledge no tion techniques. Electroencephalography (EEG)-based report exists about their use in an ICU. BCIs [11,12] have been extensively studied since EEG is P300-based BCI also have specific limitations. First of a non-invasive and portable neuroimaging modality. In all, since several ERPs need to be averaged to obtain a this article, we focus on BCI technology based on the P300 sufficient SNR, the overall bit-rate is low in practice (up event-related potential (ERP). A P300 is a positive ERP to 1 minute per letter in a P300 speller) [31,32]. The signal occurring 300 to 500 ms aer t ft he presentation of a rare processing and machine learning research communities meaningful stimulus presented in a flow of many irrel- actively address this issue by improving ERP detection evant stimuli [13]. e Th repeated presentation of several methods [33–35]. Also, the use of a P300 speller requires stimuli among which there is a possible target for the user constant concentration and there is still a debate on is referred to as the ‘oddball paradigm’ [13–15]. The ‘P300 whether some clinical populations possesses the necessary speller’ is currently the best-known and most widespread attention span to effectively control a P300 speller, espe- ERP-based BCI communication interface. In a typical cially if the session is prolonged [36,37]. This is particu- P300 speller a regular grid of symbols is displayed on the larly true for populations of patients admitted to the ICU screen while lines and columns are flashed at random. The where central nervous system (CNS) depressant drugs participant is asked to concentrate on a target symbol. A are usually administered as they are known to influence P300 is elicited in response to the flash of the line and the vigilance and attention [38]. It is worth noting that while column containing the target symbol. Since the ampli- P300 speller BCI systems have traditionally been clinically tude of the P300 is much smaller as compared to the EEG evaluated in populations of chronic patients such as those background activity and EEG artifacts, the ERP is elicited suffering from ALS [27,28,30,39], its potential benefit in several times and averaged in order to increase the signal- acute nervous conditions, such as GBS, have not yet been to-noise (SNR). The averaging procedure increases the reported. discrimination ability (target ERPs vs. non-target ERPs), To complete the technological transfer from ‘bench to effectively resulting in a trade-off between accuracy and bedside’, BCI must gain ease of use and robustness in terms speed of selection. The P300 speller represents a great of both algorithms and interface (signal-processing and hope for severely disabled patients, especially for those applications). Overcoming the aforementioned technical who are not able to use traditional AT. challenges appears important, particularly in the context Despite the tremendous amount of research over the of a better assessment of pain and cognitive function past 20 years, EEG-based BCIs still sue ff r from signic fi ant (perhaps identification of delirium) of non-communicat - drawbacks such as setup time and sensitivity to noise [16]. ing patients. This would certainly result in an improved For instance, the placement of electrodes connected to the quality of ICU stay. The Robust Brain -computer Interface scalp skin through a conductive gel is a time-consuming for virtual Keyboard (RoBIK) project [40] aimed at the process, certainly prohibitive for everyday use in both development of a BCI system for the communication of hospitals and at the patient’s home. Moreover, caregivers patients that could be used on a daily basis in the hospital are usually not familiar with EEG technology. Attempts with limited external intervention. In order to achieve this to overcome this limitation include the use of caps with goal a multidisciplinary approach was chosen and devel- pre-positioned gel-based [16,17] or dry [18–20] electrodes, opments were carefully framed by clinical specifications possibly interfaced with active recording systems [21,22] and validation, according to a user-centered design. The Recently, the EPOC headset [23,24] has opened the project resulted in two clinical studies: BRAIN-COMPUTER INTERFACES 199 Table 1. summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-val- idation procedures. abbreviations: minimum distance to mean (mDm) is a r iemannian classifier; support vector machine (sVm) is a classifier. xD a Wn is a spatial filter for event-related potentials. Population Hardware Software Algorithms Cross-validation s tudy i• xDaWn• 20-fold (training) • patients (p1)• tmsi• openViBe • sVm• Cross-session s tudy 2• patients (p2)• openViBe• riemannian potato• 5-fold (training) • roBiK • Volunteers (V2)• python 2.7• mDm• Cross-session • Study 1: Identification of the clinical and technical boards (‘Comité de protection des personnes’, CPP, Saint limitations aeff cting the use of BCI applications Germain-en-Laye, 2009/10/15). Quadriplegic patients for the communication of acute patients in a clin- admitted to ICU and rehabilitation unit at a tertiary-care ical setting; a questionnaire was administered to hospital were enrolled aer g ft iving their informed consent. patients and caregivers [41] in order to identify Pregnant women, illiterate patients, patients under judi- appropriate user specifications; cial protection or without social security as well as patients • Study 2: the clinical validation of a BCI system with a history of epilepsy were excluded from the study. developed with specifications derived from Study I, e p Th rotocol consisted of three P300 speller which consisted of sessions [15]. ◦ An EEG headset with portable electronics (1) A training session was given during which designed for convenience of use in the hospital patients were instructed to spell two words (Appendix 1), (‘OCEAN’ and ‘NUAGES’, meaning in French ◦ dedicated sowa ft re comprising a user-friendly ‘Ocean’ and ‘Clouds’, respectively) and for interface and a classification algorithm based on which no feedback was provided. Signals col- Riemannian geometry (Appendix 2); the perfor- lected were then used to train a state-of-the-art mance of this BCI prototype on clinical samples pattern-recognition algorithm based on a spa- was compared with a state-of-the-art assistive tial filter and a classifier, detailed in the next device for the communication of patients (scan- section [33]. ning device), in order to assess the actual clinical (2) e c Th lassifier performance was evaluated dur - value of the two technologies, and withhealthy ing an ‘online’ session during which instruc- volunteers, in order to quantify the drop in per- tions were given for three words (‘OEUFS’, formance induced by the patients’ condition. ‘AVION,’ and ‘OASIS’, meaning in French ‘eggs’, Finally, an offline analysis of the data collected during ‘airplane’, and ‘oasis’, respectively). The letters these two clinical trials was carried out in order to quan- identified by the algorithm aer ft each sequence tify the contribution of each system’s element (hardware of stimulations were displayed on the screen as and software) to the whole and to estimate the perfor - a feedback. mance of a personalized BCI. The details of this analysis (3) Finally, an optional ‘free-spelling’ session was are available in Appendix 3. oer ff ed to the patient, during which no instruc- tion was given and patients could spell what- Materials and methods ever they wished. Table 1 summarizes the populations, hardware, software, At the end of the session patients were requested to algorithms, and cross-validation techniques used in each rate their satisfaction with the BCI system as a commu- clinical trial. nication tool by means of a visual analogue scale (VAS) ranking from 0 (completely unsatisfied) to 10 (completely satisfied). Study 1: clinical feasibility e Th aim of this study was to evaluate the usability of a Data acquisition and processing state-of-the-art P300 speller BCI for communication [15] Twenty-four silver chloride (AgCl) disk-electrodes in the challenging environment of intensive care and reha- were connected by active-shielded coaxial cables to a bilitation units. Porti32 EEG amplifier (TMSi, Twente, Netherlands) sampling at 512  Hz. Signal acquisition, processing, and Study design and protocol storage were performed using the open-source platform e p Th rotocol was registered on ClinicalTrials.gov OpenViBE [42]. Raw EEG signals were stored as GDF (NCT01005524) and received clearance from local ethics 200 L. MAYAUD ET AL. format [43] including the type and location of visual which were implemented as described in this section and stimulations. This data-set is freely available on the used for the second clinical study. Technical details can be Physionet platform . found in Appendices 1 and 2. As pre-processing, all EEG signals were band-pass-fil- tered between 0.01 and 30 Hz by means of a Butterworth The EEG headset fourth-order filter with linear phase response. e Th e R Th oBIK EEG headset, shown in Figure 1 (left), has xDAWN spatial filter [33] was trained on the training been designed to be an effective means of recording EEG session. xDAWN is a spatial filter specifically designed activity usable by people not familiar with EEG technol- for ERP data [44]. The aim of a spatial filter in this context ogy, while oer ff ing performance equivalent to that of a is to enhance the signal of interest (ERPs) while suppressing clinical-grade EEG system. The resulting headset has silver the background noise [45]. It also dramatically reduces chloride (AgCl) electrodes mounted at the following 14 the dimensionality of EEG data, since a few filters suffice standard 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, to summarize the signal of interest (ERPs). e Th spatially P7, Fp2, F3, P8, Po8, and Cz. Connection to the scalp skin filtered signal is fed to an ensemble of support vector is obtained by means of a cotton pad soaked with physi- machine (SVM) classifiers for identification of target and ological saline solution. e Th signal is sampled at 580 Hz non-target stimulation [46]. with 12-bit resolution. Cross-validation scheme The user interface For each participant the accuracy was assessed with a e a Th pplication and generation of visual stimulations were 20-fold cross-validation procedure on the training ses- handled by a dedicated application called ‘Brainmium’ sion. A K-fold validation [47] is a technique oen u ft sed (Figure 1, right). It was designed to be user-friendly so that to estimate the performance of a classification technique a caregiver with basic knowledge of informatics can oper- (and its variability) on ‘unseen’ data. At each fold, 20% ate it aer a s ft hort training. Brainmium implements state- of the data was left aside for model validation (the ‘test’ of-the-art P300 paradigms, using random inter-stimulus set) while the remaining 80% (the ‘training set’) was used interval (ISI) and random group flashing (different from to design the model (the xDAWN spatial filter and the the line-column paradigm), which are meant to reduce ensemble of SVMs). Online letter-recognition accuracy visual fatigue and increase discriminatory power. Details during the test session was simply obtained by applying can be found in Appendix 2. models derived from the training session, live, to the online data. Online artifact rejection In this work we use a signal quality index (SQI) in order to reject trials contaminated by excessive artifacts during Technical developments both the training and the online phase of the experiment. Following this feasibility study and results from a survey e Th SQI is an improvement of the ‘Riemannian potato’ conducted with patients and caregivers [41], we identi- methods [48] based on the suggestion from Congedo [49]. fied optimal hardware and software BCI specifications, Technical details are given in Appendix 2. Figure 1. t he roBiK prototype: the roBiK headset (left) with 14 wet eeG channels and the virtual keyboard interface ‘Brainmium’ (right). BRAIN-COMPUTER INTERFACES 201 Online classifier e o Th nline classification was performed by means of the Riemannian minimum distance to means (MDM) classi- fier [50] as applied to P300 data [49]. The standard method was complemented with a new logistic decision function as detailed in Appendix 2. Study 2: clinical evaluation e p Th rimary objective of this study was to evaluate the performance of the RoBIK BCI prototype and to com- pare its performance to that of a traditional assistive Figure 2.  patients were randomly assigned to try first the roBiK technology. The second objective was to compare the BCi or the scanning device. t he roBiK prototype was evaluated performance of patients using the BCI prototype to that within half a day while the scanning device was evaluated over of healthy volunteers. three sessions in order to account for a possible learning effect. f our different texts of equivalent difficulty were selected so that the roBiK device was evaluated with text 1 or 4. Study design and setting The protocol was registered on ClinicalTrials.gov (NCT01707498) and received clearance from the Bedside procedure local ethics board (CPP de Saint Germain-en-Laye, e R Th oBIK prototype was setup by an occupational ther - 2012/07/05) and the regulatory agency for the use of a apist who had no specific training in electrophysiology. non-CE-marked device (Agence Nationale de Sécurité e p Th rotocol consisted of the following steps: des Médicaments, ANSM, 2012/09/14, 2012-A00613– (1) e Th EEG headset was set up and time was 40). The clinical protocol inclusion criterion for patients counted from the beginning of the installation was the presence of functional quadriplegia. The inclu- to the beginning of step 2; the operator was sion criterion for healthy participants was age greater instructed to reduce the presence of power line than 18 years. Patients were enrolled in the intensive contaminations (50 Hz) by adjusting electrode care unit (ICU) and rehabilitation units at a tertiary care positioning until the presence of eye blinks, hospital after giving their informed consent. Healthy jaw muscular artifacts, and alpha waves could volunteers were enrolled at the Center for Clinical be observed when the patient was instructed to Investigation and Technological Innovation (CIC-IT) blink, clamp his jaw, and close his eyes, respec- after giving their informed consent. Pregnant women, tively. The impedance between each electrode patients with hemodynamic instability, illiterate par- and the reference was measured at the begin- ticipants, patients under judicial protection or without ning and end of each session and maintained social security, epileptic patients, and participants show- below 5 kΩ. ing skin/scalp sensitivity or severe visual impairment or (2) e in Th terface was presented to the patient and aged less than 18 years were excluded from the study. directions for the training sessions were given. e exp Th erimental design is shown in Figure 2; patients When possible, the participant was asked to were randomly assigned to try the RoBIK prototype or re-formulate to make sure the task was cor- the scanning device (described below) first. Because there rectly understood. are no well-documented learning effects in P300-based (3) e Th training session consisted of the spelling of BCIs [51], the RoBiK prototype was evaluated only over 10 consecutive characters. The participant was a single half-a-day session. Instead, the scanning speller instructed to concentrate on the chosen letter performance was estimated over three sessions in order to and count the number of times it flashed. Each estimate a possible learning effect [52]. Each session was letter was flashed 20 times. The target letter was separated by at least a half-day wash-out period in order continuously indicated in the upper section of to avoid a possible bias due to fatigue. the screen and each new letter was notified by At the end of each session we assessed the fatigue and printing in blue the corresponding key on the satisfaction of patients by means of a VAS ranging from 0 virtual keyboard before each new sequence of (very tired / very unsatisfied) to 10 (not tired at all / very random stimulation. satisfied). In both cases participants were asked to fill a (4) Data collected during this training session questionnaire to express their opinion on both techniques were automatically passed down the processing (including comfort and fatigue). 202 L. MAYAUD ET AL. pipeline for training to the MDM classification Statistical analysis algorithm. The data from the training sessions For all tests, samples were tested for normality by means were split into five distinct training and test of the Jarque-Bera test (significance level at 5%) and ana- sets in order to estimate the ‘real accuracy’ lyzed with non-parametric or parametric tests depending (percent of correctly identified characters on whether the hypothesis of normal distribution of the within 36 choices) on unseen data. e p Th atient data was rejected or not, respectively. Tests for comparing was allowed to continue the protocol and pass central location parameters (means) were chosen paired to the ‘online’ session if the estimated accu- or unpaired according to the test at hand. Differences racy was above 70%, meaning that, on average, between the healthy volunteers and patients (unpaired) three spelling errors every 10 characters were for the variables age, setup time, comfort, performance tolerated, otherwise the participant was dis- during the training and online sessions were tested using charged from the study. the Student t-test or the Mann Whitney U-test for nor- (5) e ‘ Th online’ session was equivalent to the pre- mally and non-normally distributed samples, respectively. vious one except that visual feedback was Comparison of fatigue, setup time, and spelling accuracy provided to the patients by the classification in patients (paired) using the RoBIK and the scanning sys- algorithm. The number of repetitions for tem (paired statistics) was assessed with a paired Student the online session was chosen to maximize t-test or a Wilcoxon test depending on whether data were the estimated accuracy on the training data. found to be normally distributed or not. Comparison of e Th online session was timed to last exactly fatigue in patients for each interface was estimated with 10 minutes, during which participants were a repeated-measures ANOVA that uses a multivariate instructed to spell as many letters as possible framework (Hotelling T-square) to account for cor- without corrections. relation between measures [53], which therefore does (6) Finally, patients could use the interface during not need correction for sphericity [54]. Comparison of a ‘free-spelling’ session where no instruction fatigue between healthy volunteers and patients using the was given, but an output was still provided. RoBIK interface was estimated with the same technique es Th e data are freely available on the Physionet considering two independent groups of participants. For platform . all statistical tests the tolerance for type I error was set to 0.05. All analyses were performed using Matlab (Version 8.0.0.783 - R2012b) and toolboxes referenced. The scanning system An occupational therapist, as part of every day’s clinical Results activity, was in charge of equipping patients with a ‘switch interface’ (‘Buddy button’ switch, Ablenet, Roseville, MN, Study 1: clinical feasibility USA). The nominal activation force of the switch varied Patients from 10 to 600 grams. Depending on the specific condi- Twelve quadriplegic patients admitted in adult medical tion of the patient the following parameters were tuned by ICU [7] and rehabilitation units [5] met the inclusion cri- the occupational therapist: activation force, size, location teria and were consecutively included in the study aer ft (feet, near the head, thumb), type of activation (valida- giving their informed consent. Table 1 summarizes the tion on press-down or release), and possible filtering of sample, composed of eight men (67%) and four women, double-clicks. Once a reliable switch command could be aged from 22 to 63 years old. e Th tolerance was good in obtained, the system was connected via the ‘Joycable’ USB all patients. Four patients (33%) did not complete the pro- interface (Sensory software, Malvern, Worcestershire, tocol. Access to the occipital area was complicated by a UK) to the KEYVIT virtual keyboard (Jabbla, Ghent, central catheter in the first patient (#01). In one patient Belgium). On this interface, a regular grid of symbols a technical issue interrupted the inclusion (#02). One flashes lines at a regular pace (i.e. the ‘scanning’) until patient fell asleep during the training session (#10). The the patient selects one line by activating the switch. Once last patient (#11) was suspected to have eyesight problems, a line is selected, each element of the line is then flashed although this could not fully be confirmed by medical files. until a second click selects the desired symbol or letter. Aer in ft stallation of the interface, patients were asked to P300 speller for communication spell as many characters as they could within 10 min. The The results of all patients included in Study 1 are pre- instruction text was printed big enough so that it could sented in Table 2. Out of 10 patients who completed be seen as displayed on an A4 sheet of paper next to the the training session, eight used the system during a test screen. BRAIN-COMPUTER INTERFACES 203 Table 2. Description of the population included in s tudy 1 with results: a C, access to computer; Hm, voluntary head mobility; oe, oral expression; mV, mechanical ventilation, et , endotracheal intubation. ‘t reatments’ details drugs administered to patients during the 48 h preceding the inclusion in the study. t raining session performance is evaluated with a 20-fold cross-validation procedure on the training data using a combination of xDaWn and sVm. i t shows discrimination – area under the receiver operating curve (auroC) – between targets (33% of stimuli, i.e. one line and one column) and non-targets (random classifier is 50%). t esting and online performance shows accuracy in letter selection (1 symbol in 36; chance is 2.7%). t he last lines of the table summarize the distributions: categorical variables are represented by the proportion of each category (noted in parenthesis) and continuous variables are represented by the mean and standard deviation; whenever relevant, the sample size is indicated in parenthesis. f or speller performance, brackets indicate the average number of letters per patient. P-values are computed with a mann-Whiteney U-test (*). Free spelling Training Testing session Session Percent Type of Treatments session AUROC Percent Accuracy Accuracy (%) Satisfac- ID Service Age Gender Origin of handicap HM OE AC MV ET (up to 48 hours) (# letters) (# letters) (# letters) tion EVA Details 1 iCu 21 f emale f unctional post-surgery Y Y n Y Y sufentanyl, m ida- n.a.     n.a. eeG leads could not Quadriplegia (severe zolam, Gabapentine, be setup idiopathic scoliosis) Hydroxyzine 2 iCu 62 f emale post-traumatic quadriplegia Y n n Y Y alprazolam, 80.0% (20) n.a. t echnical issue C5-C6 Hydroxyxine, with acquisition prégabaline, software escitalopram, alimemazine 3 iCu 30 male post traumatic spinal cord n n Y Y Y Baclofene, 73.0% (15) 60.00% (5)   n.a.   hematoma C1-C5 Clonazepam, s odium Valproate, Hydroxyzine 4 iCu 25 male Duchene’s muscular n n Y Y Y t ramadol , alpra- 85.0% (10) 100.00% (15) 100.00% (8) 9 dystrophy zolam, mirtazapine, Hydroxyzine 5 iCu 22 male Duchene’s muscular Y n Y Y Y none 68.0% (10) 73.30% (11) 72.70% (11) 7.6   dystrophy 6 rehab 32 male post-traumatic Quadriple- n n Y n Y Zopiclone , 89.0% (10) 100.00% (15) 100.00% (6) 9 gia C4 escitalopram, alprazolam 7 rehab 58 male Locked-in s yndrome (Lis, Y Y Y n Y Bromazepam, 77.0% (10) 93.30% (14) 100.00% (15) 9.9   s troke) Baclofene 8 rehab 36 male Lis (Head injury) Y Y Y n Y Baclofene 67.0% (10) 86.70% (13) 53.80% (13) 8.5 9 rehab 46 female Lis (s troke) Y Y Y n Y Baclofene 69.0% (10) 40.00% (6)   5.5 problem of concen- tration related to swallowing issues 10 iCu 34 female Guillain-Barré s yndrome n Y n Y n t ramadol, n.a. n.a. painful Guillain Barré (sGB) Hydroxyzine, s yndrome with Gabapentine, sleep deprivation Zopiclone, Clonazepam 11 rehab 59 male Lis (s troke); Y Y Y n Y Zolpidem, 65.0% (10)     3.8 possibly bad eyesight Gabapentine 12 iCu 22 male Duchene’s muscular n n n Y Y none 75.0% (10) 80.00% (4)   4.8 patient was bored dystrophy. and interrupted the protocol   58% (ICU) 37.7±15.3 66% Neuromuscular (33%) 58% 50% 67% 58% 92% Sedated (8%) 74.8±8.0 (n=10) 79.2±20.9 (n=8) 85.3±21.2 (n=5) 7.3 Mean ± std (male) (yes) (yes) (yes) (yes) (yes) [11] [12] [10] Stroke (15%) Use of CNS 76.2±6.5 (n=5) 78.3±16.6 (n=4) 86.4±19.3 (n=2)   iCu depressant (75%) Trauma (42%)   73.4±9.8 (n=5) 80.0±27.2 (n=4) 84.6±26.7 (n=3)   rehabilitation 0.46 0.62 -   p-value* 204 L. MAYAUD ET AL. session with accuracy significantly above chance (2.6%) Study 2: clinical evaluation level (rank sum test, W = 100, p < .001); seven could use Participants the system with more than half the symbols correctly Table 3 summarizes demographic and other variables of identified and an average accuracy of 84.7%. One patient the participants. had an accuracy of 40% and the BCI technique did not As seen in Table 3, the patients recruited in this study work for two patients. Five out of eight patients (62.5%) were on average 46.5 years old (37.0–56.0) and the healthy who co mpleted the test session asked for the optional free volunteers were on average 28.0 years old (21.8–32.2). The spelling session and spelled on average 10 characters with age difference was statically significant (rank sum test, W a median accuracy of 100.0% (72.2,100). Interestingly, = 136, p = .002). One patient was discharged prematurely the spelled words were all the first names of relatives. from the study on his request. Two additional patients did The overall satisfaction for the technique amongst not meet the threshold of performance (cross-validated participants who completed the training sessions was accuracy on training session above 70%) and were dis- high (7.3/10). charged from the study. Table 3. participants enrolled in s tudy 2 (patients and healthy volunteers). abbreviations: aBp , arterial blood pressure – systolic (s) and diastolic (d); bpm, beats per minute; f , female; GBs, Guillain-barré syndrome; Hr, heart rate; Lis, locked-in syndrome, m, male; mmHg, millimetres of mercury; ms, mulitple sclerosis; Q, quadriplegic; s, scanning device. Patients age 46 47 56 60 39 35 64 30 52 37 Gender m m f m f m m f f m pathology Lis GBs ms Lis Lis Q Q Q Q GBs Hr (bpm) 64 76 – 65 101 47 60 58 – 96 aBp s / aBp d (mmHg) 130/80 111/72 – 150/110 111/73 58/78 130/60 86/46 – 85/67 r andomization roBiK s roBiK s roBiK s roBiK roBiK s s Healthy volunteers age 33 50 26 31 32 28 21 21 22 Gender m f m f f m m m f Hr(bpm) 51 75 26 31 – 77 57 61 69 aBp s / aBp d(mmHg) 123/75 111/84 123/69 112/82 – 108/82 122/65 111/66 130/87 Figure 3.  Comparison of setup time (min), self-evaluated comfort, spelling accuracy (%), and bit-rates (bit/min) between the roBiK system (online results) and the three sessions of the scanning device. BRAIN-COMPUTER INTERFACES 205 Figure 4. Comparison of the evolution of fatigue (left) and information rate (right) for patients (dark line) and healthy volunteers (bright line) over three times of the BCi protocol (train, online, free). Comparison of patients and healthy volunteers the scanning device, respectively, which was found stat- populations ically significant (Kruskal-Wallis, H(1) = 4.90, p = .027). For the RoBIK system, the setup time was 11.0 min (8.5– Likewise, the device speed for the RoBIK system was 0.5 15.5) for patients and 13.0 min (10.5–14.2) for healthy characters per minutes (0.5–0.6) against 4.5 characters per volunteers (Figure 3). The difference was not found statisti - minute (3.5–5.0) for the scanning device (Student t-test, cally significant (rank sum test, W = 66, p = .58). Likewise, t(4) = −14.78, p < .001). Naturally these translated into the experience was rated equally pleasant (rank sum test, the significant superiority of the scanning system over W = 85, p = .23), with patients reporting an average rating the RoBIK system, with bit-rates of 28.3 bpm (19.0–31.1) of 2.5 (1.4–3.0) and healthy participants an average rating and 1.7 bpm (1.1–2.1), respectively (Student t-test, t(4) = of 1.5 (1.2–1.9). The comparison of information trans- −12.10, p < .001). fer rate (bit-rate) between patients and healthy controls In terms of self-reported fatigue in the patient group, (Figure 4) did not reveal significant differences (F(1,18) = it was found significantly greater while using the RoBIK 3.90, p = .07) even though a significant time factor (F (2,18) system, 2.7 (2.6–3.0), as compared to using the scan- = 24.46, p < .001) and time-group interaction effect was ning device, 2.0 (1.1–2.9) (F(1,29) = 49.83, p  <  .001). found (F(2,18) = 0.55, p < .001), indicating that the drop Interestingly, the repeated-measures ANOVA did not in EEG discriminatory power between the calibration and identify the time factor (measured before and aer in ft ter - the test session was greater in patients as compared to vention) as significant (F(1,29) = 1.96, p = .172), instead healthy volunteers. This might indicate that the evoked suggesting a strongly significant time-technique interac- potential amplitude (P300) of patients decreases faster tion effect (F(1,29) = 16.33, p < .001), indicating that the over time, which in turn might relate to changes in atten- BCI system was associated with greater fatigability. These tion. This interpretation is corroborated by the results on results, however, did not translate into a decreased self- self-reported fatigue, showing that although no difference reported comfort, which was found equally good for the two between the populations can be found (F(3,15) = 1.00, techniques: 4.3 (2.2–6.8) for BCI and 5.0 (4.3–6.2) for the p = .33) and both samples get tired during sessions (time scanning device (Student t-test; t(7) = −0.60, p = .565). factor, F(3,15) = 9.85, p = .001), the effect is significantly stronger in patients (interaction factor time-population, Discussion F(3,15) = 3.44, p = .049). Study 1: clinical feasibility Feasibility studies and evaluation of performance of Comparison of RoBIK prototype with scanning device P300 speller systems so far have concentrated on rela- e Th RoBIK system was associated in patients with a setup tively small samples (n < 10) of late-stage ALS or acquired time of 11.0 min (8.5–15.5) against only 3.0 min (2.0–4.2) brain injury patients using the system at home or in a for the scanning device, a difference that was found statis - controlled environment [16,27,36,55,56]. A comparison tically significant (Kruskal-Wallis, H (1) = 11.48, p = .001). of performance between normal and severely disabled e p Th erformance of the RoBIK system was signif- participants showed significantly better performance in icantly lower as compared to the performance of the the healthy population [29], which is a strong rationale spelling device in the group of patients (the only group for the evaluation of this technology with a larger variety having evaluated both techniques). The accuracy was 0.8 of patients and preferably in a natural environment. A (0.5–0.8) and 1.0 (1.0–1.0) for the RoBIK system and 206 L. MAYAUD ET AL. recent study on a bigger cohort (n = 27) of ALS patients GBS accompanied by sleep deprivation. There was no at home mentions mechanical ventilation, but it is unclear obvious reason for the low performance found in patient if patients where using the ventilators during the sessions 11 apart from a light visual impairment. This patient may [30]. The Study 1 aimed at the investigation of the use of be a ‘BCI illiterate’ [58]. BCI by acute patients in their real environment of use To the best of our knowledge, this is the first compre- (here, the ICU). hensive attempt to explore the use of a brain-computer As reported in Table 2, we found no statistically interface in an adverse clinical context. Four key factors significant mean difference of performance between were explored: first, a non-controlled clinical environ- mechanically ventilated patients (in the ICU, n = 7) and ment and the particularly adverse setup of the ICU (58%); non-ventilated patients (in the rehabilitation unit, n = 5) second, all ICU patients were evaluated during invasive over the different sessions: training (p = .46) and test mechanical ventilation; third, patients with myopathy or (p = .62). These results have to be handled with care, as the Guillain-Barré syndrome (42%) are populations that have study was certainly not designed to study such an effect. not yet been reported to use BCI; finally, this study pro- However, the individual performance reported in this vides an initial insight into the impact of central nervous study show that neither mechanical ventilation, nor the system (CNS) depressants on concomitant use of a P300 origin of tetraplegia, nor the use of CNS depressants may ERP-based BCI for communication. While this study was be reliably related to failure at controlling the BCI, which clearly not dimensioned to fully explore this phenomenon, certainly is encouraging. we welcome the successful use of the BCI by some patients For three patients who could not achieve adequate under high levels of CNS depressant, which could have performance during the test session, several possible primarily been thought to be prohibitive. Hence, further explanations were identified. A technical problem on the research is required to exactly understand the extent to acquisition module of the software degraded the system which this kind of medication influences the performance performance for patients 2 and 3 (which was subsequently of a BCI. fixed on the open-source platform). This illustrates a well- known limitation of existing BCI systems (usability of Study 2: clinical evaluation hardware and software components) when it comes to their transfer to the patients’ bedside [57]; we believe that The comparison of the BCI performance of healthy the joint efforts of the scientific and industrial commu- volunteers and patients showed no statistical difference nities is gaining momentum to address these limitations. in the primary performance outcome (bit-rate). Our find - Patient 9 reported difficulties in simultaneously swallow- ing adds to the small and contrasted available literature ing and concentrating on the task, finally achieving low [57,59]. However, the progression of bit-rate over time performance. Patient 10 fell asleep during the training within the same day, in particular for patients, was session despite initially showing interest and motivation found significant and corroborated the evolution of self- for the protocol; this was possibly explained by painful reported fatigue, which increased in both groups, but was Figure 5. example of two event-related potentials (erp s) elicited from the interface used in the pilot study (left) and the roBiK prototype (right); the roBiK prototype does not show the typical steady-state visual evoked potential (ssVep) in response to non-target stimulations that can be seen on the non-target response (left). BRAIN-COMPUTER INTERFACES 207 stronger in patients. The inclusion of fatigue and time Put together these innovative steps in EEG data col- in the analysis might in part explain the discrepancies lection and analysis have the potential to dramatically within the literature. We have included self-reported reduce the setup and calibration time. This would limit fatigue in this study because we knew that the typical the accumulation of fatigue prior to the actual use of the population of quadriplegic patients admitted to the ICU communication interface and thereby improve overall sys- is prone to fatigue and because the outcome of Study 1 tem performance, which was shown to be greatly ae ff cted indicated that fatigue was a recurrent complainr about by fatigue. These results are encouraging for the future use the system. As a consequence, we hypothesized it could of P300 spellers for the communication of patients. Apart constitute an important limitation for the use of BCIs in from patients for whom a reliable neural interface could this population. This was despite the fact that some ele- not be found (so called ‘BCI illiterates’), patients’ perfor- ments of the RoBIK prototype were specifically designed mance appeared not to differ significantly from that of to reduce fatigue: for instance, visual fatigue was limited healthy participants. Most importantly, adequately chosen by the use variable ISI, which is illustrated by Figure 5. parameters for the BCI interface lead to an estimated per- Since the patient population was significantly older it is formance that was not found to differ significantly from not possible to disentangle the effect of age from that of that of a traditional assistive technology device (t = 0.56, fatigue. This certainly constitutes a significant limitation permuted p-value = .33). Again, these results should be of our study, even though one can argue that the direction interpreted with extreme care because this study was not of this effect is nonetheless unfavorable to the patient statistically powered to answer these specific questions population since fatigability is expected to increase with and because we excluded three of ten patients (from Study age. Such a hypothesis is, however, in contrast with evi- 2) who could not use the BCI system. dence of a positive correlation between age and perfor- mance in a P300 speller, which was reported in healthy Conclusions volunteers as well as in patients [59,60]. The retrospective Some neurological disorders leave patients with baseline analysis of existing cohorts could help investigate this cognitive status and no or little communication capacity. correlation further. LIS patients – whatever their etiology – are a well-known e co Th mparison of online performance for the use of illustration of such a condition. Quadriplegic patients BCI and the scanning system in the patient population with invasive mechanical ventilation are also temporar- showed that the BCI did not compare advantageously to ily deprived of communication, resulting in additional the traditional assistive technology. All measured indi- stress for all people involved with the management of cators (with the exception of self-reported comfort and their disease. Traditional assistive technologies for chronic satisfaction – non-significant) were found to be in favor patients, such as those suffering from neurodegenerative of the scanning system: setup time was longer for the BCI; disorders, have long been used and most severe cases are accuracy, speed, and bit-rate were all found better for the equipped, whenever possible, with a ‘scanning’ spelling scanning device. Again, fatigue was strongly associated with device interfaced with simple ‘click’ contactors. However, the use of the BCI system, since both the technique and the use of these systems requires the intervention of an the time × technique interaction factors were significant. experienced occupational therapist who must find the optimal set of parameters for each patient. BCIs, on the Offline analysis of clinical data other hand, have long been promised as a potential ‘uni- versal’ assistive device for chronic patients such as those e a Th nalysis of the EEG data collected during the two clin - with ALS. So far BCI technology has consistently shown ical trials described in this paper is reported in Appendix major limitations such as  low information transfer rate 3. e Th results we obtained stress the added value of the (low spelling rate and accuracy) and the need for cum- Riemannian approach over state-of-the art classification bersome and expensive setups for the recording of the techniques, which was found particularly relevant for the EEG and its processing. However, recent advances in EEG development of calibration-free BCIs (also referred to as hardware and software appear to open a new era where ‘cross-learning’). Conversely, the analysis did not identify these limitations will be overcome. the presence of a Riemannian SQI (artifact rejection) as A feasibility study was carried out in the ICU and in a a factor contributing positively to the BCI performance, rehabilitation unit in a population of quadriplegic patients which was surprising. On the hardware side, the offline using a traditional BCI system for communication (the analysis revealed that the gain in setup time oer ff ed by the ‘P300 speller’). Twelve patients were included in the study. hardware prototype developed was not associated with a Among them, eight could successfully control the sys- noticeable drop in system performance. tem with above-chance accuracy. Performance could not 208 L. MAYAUD ET AL. directly be related to the presence of either mechanical To conclude, this study demonstrates that caregivers ventilation or sedation. In contrast, eyesight and fatigue who are unfamiliar with EEG and BCIs in general can were identified as possible limitation factors. restore some form of communication in severely disabled  Based on this pilot phase, technical specifications and patients by means of an adequately designed BCI system. user requirements were drae ft d for the development of However, the population of patients who could immedi- adequate bedside BCI prototypes for communication. e Th ately benefit from it was found to be much smaller than hardware was a 14-channel EEG system that could be set initially expected, since traditional assistive techniques are up in approximately 10 min. The software was developed remarkably effective and compare very favorably to the use with a user-friendly interface and state-of-the art pro- of P300-based spellers. Today, this is probably – more than cessing/classification techniques (based on Riemannian ever – due to the lack of ao ff rdable, convenient, and reli- geometry) in the back-end. able EEG systems, while other technological limitations e p Th rototype’s performance was evaluated in a clinical are finally being overcome. Algorithms, for instance, are trial. Its performance was compared to the performance probably reaching a form of maturity and little further of traditional assistive technology (a scanning spelling improvement can be foreseen for the coming few years; in device) in patients. The comparison was carried out using our opinion, adaptive techniques based on cross-subject bit-rate, accuracy, setup time, comfort, and fatigue, con- and cross-session initialization will play a dominant role. firming the superiority of the traditional AT technique in Fortunately, the ongoing transition of the EEG field from terms of simplicity of use (setup time), cost, performance, cottage industry to a more financially structured sector and – most importantly – patient fatigue. The BCI was also will soon translate into better and cheaper EEG systems. evaluated in a group of healthy volunteers to benchmark In turn, BCIs will cover the needs of an increasing pro- performance, showing that, if both groups performed portion of severely disabled patients and ultimately find equally, a stronger decrease in performance over time was legitimate room next to other assistive technologies on observed in patients. This further strengthens the impor - the occupational therapist’s shelf. tance of fatigue in this population. Finally, three offline analyses were carried out using Notes the data collected during phases 1 and 3: 1. https://physionet.org/works/P300SpellerBCIICU/. • First, the ‘personalized’ estimated performance of 2. https://physionet.org/works/P300SpellerBCIICU/. the BCI rate was derived from the training data, 3. h t t ps://g i t h u b .co m/a lexa n dr e b a rac h a n t/ covariancetoolbox/commit/bcccb4d750b2ad9ae6dc3 pseudo-prospectively applied to the test set (online dd76e7e6633f98bc1f3. session) and the results were compared to those of the scanning device. The analysis confirmed that the bit-rate is greatly improved when an optimal Acknowledgements and patient-specific number of repetitions is cho- We would like to thank the medical staff who have been in- sen. The resulting performance was found compa- volved on this project, and in particular Marjorie Figère, rable to that of the scanning spelling device. Marjorie Dezeaux, and Sandra Potier from the CICIT, • Second, the individual added value of each element Jean-Marie, Gilles, Prof. Lofaso, and Prof. Herault from the Functional Exploration Unit, Justine Bouteille from the New of the design was quantified in an offline analysis. It Technologies Platform (PFNT), as well as many others involved showed that the gain in ease of use generated by the in the project. new headset came at no cost in performance. It also confirmed the superiority of Riemannian methods over state-of-the-art techniques for ERP detection. Funding However, the rejection of artifactual segments of We would like to thank the National Research Agency (ANR) the data did not increase the performance above the and the General Directorate for Armament (DGA) for fund- threshold of statistical significance.  ing this project (Project ANR- 09-TECS-013–01-RoBIK). We • Last but not least, we investigated possible model would like to thank the French Association for Myopathies (AFM) for partly funding this project. initialization using an existing database. The ration- ale was to reduce fatigue by removing the tiring calibration session. The analysis revealed that the ORCID Riemannian methods enjoy superior cross-subject Louis Mayaud   http://orcid.org/0000-0002-3187-8030 generalization, providing a good initialization that Marco Congedo   http://orcid.org/0000-0003-2196-0409 needs to be adapted online [44].  BRAIN-COMPUTER INTERFACES 209 [19] Popescu F, Fazli S, Badower Y, et al. Single trial References classification of motor imagination using 6 dry EEG [1] Vincent JL. Communication in the ICU. Intensive Care electrodes. PLoS ONE. 2007;2(7):e637. Med. 1997;23(10):1093–1098. [20] Sellers E, Turner P, Sarnacki W, et al. A novel dry electrode [2] Haig AJ, Katz R, Sahgal V. Mortality and complications for brain-computer interface. Human-Computer of the locked-in syndrome. Arch Phys Med Rehabil. Interaction Novel Interaction Methods Tech. 2009: 1987;68(1):24–27. 623-631. [3] Frank Lopresti EF, Mihailidis A, Kirsch N. Assistive [21] Hornecker A. Dear customers, dear friends of brain technology for cognitive rehabilitation: state of the art. products. Brain. 2009;33. Neuropsychol Rehabil. 2004;14(1-2):5–39. [22] Renard Y. In the focus. Brain. 2010;37. [4] Simpson R, Koester H, LoPresti E. Evaluation of an [23] Emotiv E. Software Development Kit. 2010. adaptive row/column scanning system. Technol Disabil. [24] Liu Y, Jiang X, Cao T, et al., editors. Implementation 2006;18(3):127–138. of SSVEP based BCI with emotiv EPOC. Virtual [5] Simpson RC, Koester HH. Adaptive one-switch Environments Human-Computer Interfaces and row-column scanning. IEEE Trans Rehabil Eng. Measurement Systems (VECIMS), 2012 IEEE 1999;7(4):464–473. International Conference on; 2012: IEEE. [6] Biswas P, Robinson P. A new screen scanning system [25] Duvinage M, Castermans T, Dutoit T, et al., editors. A based on clustering screen objects. J Assist Technol. P300-based quantitative comparison between the emotiv 2008;2(3):24–31. epoc headset and a medical eeg device. biomedical [7] Cook AM, Polgar JM. Assistive technologies: principles engineering/765: telehealth/766: assistive technologies. and practice. Elsevier Health Sciences; 2014. ACTA Press; 2012. [8] Vidal JJ. Toward direct brain-computer communication. [26] Stytsenko K, Jablonskis E, Prahm C, editors. Evaluation Annu Rev Biophys Bioeng. 1973;2:157-180. Epub of consumer EEG device emotiv epoc. MEi: CogSci 1973/01/01. doi: http://dx.doi.org/10.1146/annurev.bb.02. Conference 2011, Ljubljana; 2011. 060173.001105. PubMed PMID: 4583653. [27] Sellers EW, Donchin E. A P300-based brain–computer [9] Birbaumer N. Breaking the silence: brain?computer interface: initial tests by ALS patients. Clin Neurophysiol. interfaces (bci) for communication and motor control. 2006;117(3):538–548. Psychophysiology. 2006;43(6):517-532. Epub 2006/11/02. [28] Kubler A, Nijboer F, Mellinger J, et al. Patients with doi: http://dx.doi.org/10.1111/j.1469-8986.2006.00456.x. ALS can use sensorimotor rhythms to operate a brain- PubMed PMID: 17076808. computer interface. Neurology. 2005;64(10):1775-1777, [10] Wolpaw JR, Birbaumer N, Heetderks WJ, et al. Brain- 0028-3878. computer interface technology: a review of the first [29] Piccione F, Giorgi F, Tonin P, et al. P300-based brain international meeting. IEEE Trans Rehabil Eng. computer interface: reliability and performance in 2000;8(2):164–173. Epub 2000/07/15 PubMed PMID: healthy and paralysed participants. Clin Neurophysiol. 2006;117(3):531–537. [11] Teplan M. Fundamentals of EEG measurement. Meas Sci [30] McCane L, Vaughan TM, McFarland DJ, et al. Evaluation Rev. 2002;2(2):1–11. of individuals with ALS for in-home use of a P300 brain- [12] Wolpaw JR, McFarland DJ, Neat GW, et al. An computer interface. Soc Neurosci, Submitted. 2009. EEG-based brain-computer interface for cursor [31] Li K, Narayan Raju VN, Sankar R, Arbel Y, Donchin E. control. Electroencephalogr Clin Neurophysiol. Advances and challenges in signal analysis for single 1991;78(3):252–259. Epub 1991/03/01 PubMed PMID: trial P300-BCI. foundations of augmented cognition directing the future of adaptive systems. Springer; 2011. [13] Picton TW. The P300 wave of the human event-related p. 87–94. potential. J Clin Neurophysiol. 1992;9(4):456–479. Epub [32] Khan OI, Kim S-H, Rasheed T, Khan A, Kim T-S, editors. 1992/10/01 PubMed PMID: 1464675. Extraction of P300 using constrained independent [14] Kaper M, Meinicke P, Grossekathoefer U, et al. BCI component analysis. Engineering in Medicine and competition 2003—data set iib: support vector machines Biology Society, 2009 EMBC 2009 Annual International for the p300 Speller Paradigm. IEEE Trans Biomed Eng. Conference of the IEEE. IEEE; 2009. 2004;51(6):1073-1076, 0018-9294. [33] Rivet B, Souloumiac A, Attina V, et al. xDAWN [15] Krusienski DJ, Sellers EW, McFarland DJ, et al. Toward algorithm to enhance evoked potentials: application enhanced P300 speller performance. J Neurosci Methods. to brain–computer interface. IEEE Trans Biomed Eng. 2008;167(1):15. 2009;56(8):2035–2043. [16] Sellers EW, Vaughan TM, Wolpaw JR. A brain-computer [34] Rakotomamonjy A, Guigue V, Mallet G, et al. Ensemble interface for long-term independent home use. of SVMs for improving brain computer interface P300 Amyotroph Lateral Scler. 2010;11(5):449–455. speller performances. Artificial Neural Netw: Biol [17] Müller K-R, Blankertz B. Toward non-invasive brain- Inspirations–ICANN 2005. 2005:45-50. computer interfaces. 2006. [35] Patrone M, Lecumberry F, Martín Á, Ramirez I, Seroussi [18] Grozea C, Voinescu CD, Fazli S. Bristle-sensors–low-cost G. EEG Signal Pre-processing for the p300 speller. flexible passive dry EEG electrodes for neurofeedback progress in pattern recognition, image analysis, computer and BCI applications. J Neural Eng. 2011;8(2):025008. vision, and applications. Springer; 2015. p. 559-66%@ Epub 2011/03/26. doi: http://dx.doi.org/10.1088/1741- 2560/8/2/025008. PubMed PMID: 21436526. 210 L. MAYAUD ET AL. [36] Kübler A, Furdea A, Halder S, et al. A Brain-computer [52] Stewart H, Wilcock A. Improving the communication interface controlled auditory event-related potential rate for symbol based, scanning voice output device (P300) spelling system for locked-in patients. Ann N Y users. Technol Disabil. 2000;13(3):141–150. Acad Sci. 2009;1157(1):90–100. [53] Rencher X. Methods of multivariate data. Analysis. 1995. [37] Guo M, Xu G, Wang L, Wang J, editors. Research on [54] Pernet CR, Chauveau N, Gaspar C, et al. Limo auditory BCI based on wavelet transform. Virtual EEG: a toolbox for hierarchical linear modeling of Environments Human-Computer Interfaces and electroencephalographic data. Comput Intell Neurosci. Measurement Systems (VECIMS), 2012 IEEE 2011;2011:3. International Conference on. IEEE; 2012. [55] Nijboer F, Sellers E, Mellinger J, et al. A P300-based brain– [38] Vienne J, Lecciso G, Constantinescu IO, et al. Differential computer interface for people with amyotrophic lateral effects of sodium oxybate and baclofen on EEG, sleep, sclerosis. Clin Neurophysiol. 2008;119(8):1909–1916. neurobehavioral performance, and memory. Sleep. [56] Riccio A, Leotta F, Tiripicchio S, et al. Can severe Acquired 2012;35(8):1071–1083. 1071-83%@ 0161-8105. Brain Injury users control a communication application [39] Mugler EM, Ruf CA, Halder S, et al. Design and operated through a P300-based brain computer interface? implementation of a P300-based brain-computer interface Age. 2011;41(46):24. for controlling an internet browser. IEEE Trans Neural [57] Cipresso P, Carelli L, Solca F, et al. The use of P300‐based Syst Rehabil Eng 2010;18(6):599-609. Epub 2010/09/02. BCIs in amyotrophic lateral sclerosis: from augmentative doi: http://dx.doi.org/10.1109/TNSRE.2010.2068059. and alternative communication to cognitive assessment. PubMed PMID: 20805058. Brain Behav. 2012; 2:479–498. [40] Mayaud L, Filipe S, Pétégnief L, et al. Robust brain- [58] Guger C, Daban S, Sellers E, et al. How many people are computer interface for virtual keyboard (RoBIK): project able to control a P300-based brain? computer interface results. IRBM; 2013. p. 131–138. (BCI)? Neurosci Lett. 2009;462(1):94–98. [41] Mayaud L, Cabanilles S, Azabou E. Patient needs analysis [59] Silvoni S, Volpato C, Cavinato M, et al. P300-based for brain-computer interfaces. In: Clerc M, Bougrain L, brain–computer interface communication: evaluation Lotte F, editors. Brain computer interfaces. Great Britain: and follow-up in amyotrophic lateral sclerosis. Front ISTE Editions. Croydon, CR0 4YY; 2016. Neurosci 2009;3(60). [42] Renard Y, Lotte F, Gibert G, et al. OpenViBE: an open- [60] Kübler A, Birbaumer N. Brain–computer interfaces source software platform to design, test, and use brain– and com- munication in paralysis: Extinction of goal computer interfaces in real and virtual environments. directed thinking in completely paralysed patients? Clin Presence-Teleop Virt Environ. 2010;19(1):35–53. Neurophysiol. 2008;119:2658–2666. [43] Schlögl A. GDF-A general dataformat for biosignals. [61] Cecotti H, Rivet B, Congedo M, et al. A robust sensor- arXiv preprint cs/0608052. 2006. selection method for P300 brain–computer interfaces. J [44] Congedo M, Korczowski L, Delorme A. Spatio-temporal Neural Eng. 2011;8(1):016001. common pattern: a companion method for ERP analysis [62] Oostenveld R, Fries P, Maris E, et al. FieldTrip: open in the time domain. J Neurosci Methods. 2016;267:74– source software for advanced analysis of MEG, EEG, 88. 74-88%@ 0165-270. and invasive electrophysiological data. Comput Intell [45] Congedo M, Gouy-Pailler C, Jutten C. On the blind Neurosci 2011;2011:156869. Epub 2011/01/22. doi: source separation of human electroencephalogram http://dx.doi.org/10.1155/2011/156869. PubMed PMID: by approximate joint diagonalization of second order 21253357; PubMed Central PMCID: PMC3021840. statistics. Clin Neurophysiol. 2008;119(12):2677–2686. [63] Congedo M, Goyat M, Tarrin N, et al., editors. Brain [46] R akotomamonjy A, Guigue V, Mallet G, et al. Ensemble Invaders’: a prototype of an open-source P300-based of SVMs for improving brain computer interface video game working with the OpenViBE platform. P300 speller performances. Lect Notes Comput Sc. Proceedings of the 5th International Brain-Computer 2005;3696:45–50. PubMed PMID: ISI:000232193800008. Interface Conference 2011; 2011. [47] Lotte F, Congedo M, Lécuyer A, et al. A review of [64] Jin SH, Lin P, Hallett M. Abnormal reorganization classification algorithms for EEG-based brain–computer of functional cortical small-world networks in focal interfaces. J. Neural Eng. 2007;4. hand dystonia. PLoS ONE. 2011;6(12):e28682. Epub [48] Barachant A, Andreev A, Congedo M, editors. The 2011/12/17. doi: http://dx.doi.org/10.1371/journal. Riemannian Potato: an automatic and adaptive artifact pone.0028682. PubMed PMID: 22174867; PubMed detection method for online experiments using Central PMCID: PMC3236757. Riemannian geometry. TOBI Workshop lV. 2013. [65] Townsend G, LaPallo BK, Boulay CB, et al. A novel P300- [49] Congedo M. EEG Source Analysis Université de based brain–computer interface stimulus presentation Grenoble; 2013. paradigm: moving beyond rows and columns Clin [50] Barachant A, Bonnet S, Congedo M, et al. Multiclass Neurophysiol 2010;121(7):1109-1120. Epub 2010/03/30. brain-computer interface classification by Riemannian doi: http://dx.doi.org/10.1016/j.clinph.2010.01.030. PubMed geometry. IEEE Trans Biomed Eng 2012;59(4):920- PMID: 20347387; PubMed Central PMCID: PMC2879474. 928. Epub 2011/10/20. doi: http://dx.doi.org/10.1109/ [66] Congedo M. Introducing the logistic discriminant TBME.2011.2172210. PubMed PMID: 22010143. function in electroencephalography. J Neurother. [51] Nijboer F, Birbaumer N, Kübler A. The influence of 2003;7(2):5–23. psychological state and motivation on brain–computer [67] Serby H, Yom-Tov E, Inbar GF. An improved P300- interface performance in patients with amyotrophic lateral based brain-computer interface. IEEE Trans Neural Syst sclerosis–a longitudinal study. Front Neurosci. 2010;4. Rehabil Eng. 2005;13(1):89–98. BRAIN-COMPUTER INTERFACES 211 [68] Wolpaw JR, Ramoser H, McFarland DJ, et al. EEG- Appendix 2. Details of software development based communication: improved accuracy by response Graphical user interface verification. IEEE Trans Rehabil Eng. 1998;6(3):326–333. [69] Hosmer DW Jr, Wang CY, Lin IC, et al. A computer e Th Brainmium speller interface was designed to automate the program for stepwise logistic regression using calibration of machine learning algorithms implemented in maximum likelihood estimation. Comput Prog Biomed. OpenViBE and to provide a user-friendly interface for patients 1978;8(2):121–134. Epub 1978/06/01 PubMed PMID: and operators. Rather than leaving the application synchroniz- ing the received EEG data with the precise temporal occurrence [70] Congedo M, Barachant A, Andreev A A New generation of visual stimulations, which would be prone to delays and jitter of brain-computer interface based on riemannian induced by the operating system (Windows 7 professional edi- geometry. arXiv preprint arXiv:13108115. 2013. tion, Microsoft Corporation, Redmond, WA, USA), we decided [71] Guger C, Krausz G, Allison BZ, et al. Comparison of to send these stimulations through the USB directly to the EEG dry and gel based electrodes for P300 brain–computer electronic acquisition unit to obtain accurate timestamps for all interfaces. Front Neurosci. 2012;6. stimulation events. Flashes had a duration of 75 ms [55] and ran- dom inter-stimulus intervals (ISI) were drawn from an exponen- tial distribution with mean  = 0.15s. In previous research by Appendix 1. Details of hardware developments our consortium, an exponential ISI was found to reduce the cor- tical fatigue associated with stimulation at fixed frequency while e EEG h Th eadset was designed to minimize the need for EEG enhancing the ERP single-trial estimation [63]. In the original recording preparation: as a primary goal in the design phase, P300 speller paradigm symbols flash by rows and columns. Of- skin preparation with abrasive paste, electrode positioning with ten detection errors arise because of the ‘adjacency-distraction’ gel, and retention with tape had to be avoided. The resulting phenomenon [64,65], according to which non-target symbols in headset is a 3D-printed polyamide structure holding 14 Ag- rows or columns adjacent to the target attract the user’s attention Cl electrodes (Neuroservices, Evry, France) located at standard when they flash, producing evoked activity similar to the P300, 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, P7, Fp2, F3, making the detection of the target P300 more difficult. To miti- P8, Po8, and Cz. These locations were selected in order to max- gate this effect we flash the symbols by random groups [63]. Not imize the chance of capturing P300 ERPs using as reference only is the ‘adjacency-distraction’ effect mitigated, we also find and ground the right and left mastoids, respectively, according that the pattern of flashing becomes totally unpredictable, which to previous studies by the RoBIK consortium [61]. Each elec- is expected to sustain the attention of the user and to enhance trode was mounted on a polyethurane wheel connected to a the P300. More importantly, random-group flashing allows arbi- spring in order to control the pressure applied to the scalp with trary positioning of the symbols on the screen (no more need to the electrode. The headset was designed to meet the need of pa- arrange symbols on a grid), which greatly expands the usability tients and safety requirements, as well as to be compliant with of the P300 paradigm. the ICU environment. In particular, the headset could be used in a lying position or in the presence of a headrest. e h Th eadset contains an electronic component that was de- Online artifact rejection signed to record EEG activity with high fidelity while maintain- ing minimal volume and weight. An MSP430 ultra-low-power e o Th nline artifact detection was initialized at the beginning of microcontroller (Texas Instrument, Dallas, USA) was used to each session in order to interrupt the sequence of stimulations control a dedicated application-specific integrated component in the presence of excessive noise. This technique allows a sim- (ASIC) developed for this purpose: the CIrcuit for NEuronal ple definition of a rejection region for incoming data segments. SIgnal Conversion (CINESIC32). Each input channel is com- e p Th articipant was instructed to stay still for 10 seconds dur - bined with an external capacitor (1.5 nF) in order to suppress ing which a series of n clean overlapping EEG time-windows sqi the risk of leaking current in a first default condition, which X of dimension E  ×  S were extracted, with S the num- i=1..n sqi is essential for medical applications. The analogue channel is ber of samples in the time-window and E = 14 the number of composed of a fully differential low-noise amplifier, followed electrodes. For each time-window X , a covariance matrix Σ of i i by a voltage gain amplifier and a programmable low-pass fil- dimension E  ×  E was computed. Covariance matrices belong ter. Each channel consumes about 34 μA, summing up to an to the Riemannian manifold of symmetric positive-definite averaged consumption of 13 mA at full data streaming includ- matrices wherein a Riemannian metric can be used to define ing baseline consumption at 3.3 V. With these settings 24-hour a distance δ between any two covariance matrices such as [50]: continuous operation can be achieved with one high-ener- 1∕2 gy-density 3.6  V lithium battery. Only 14 of the 32 available −1∕2 −1∕2 Σ , Σ =∥ log Σ Σ Σ ∥= log  , (1) channels were used in the final prototype and each of them i j 1 2 1 e was configured with a (0.5–30 Hz) band-pass filter and a 60 dB −1 −1 voltage gain; analogue signals were digitized through a 12-bit where  , e = 1… E are the E eigenvalues of Σ Σ or of Σ Σ . e 1 2 2 1 analogue-to-digital converter (ADC) with nominal sampling Using this distance, a geometric center of mass M (or Fréchet frequency of 580 Hz per channel. mean) of a set of covariance matrices can be estimated with a Raw EEG were transmitted to a field-trip buffer [62] that gradient descend algorithm solving the following optimization was subsequently read by an OpenViBE acquisition server problem: [42]. Signals were then band-pass-filtered between 1 and 20 Hz with a fourth-order Butterworth with linear phase response argmin  M, Σ (2) and decimated to 145 Hz for further analysis. 212 L. MAYAUD ET AL. Table 4. summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-val- idation procedures used in each phase. abbreviations: minimum distance to mean (mDm) is a r iemannian classifier; support vector machine (sVm) is a classifier; stepwise linear discriminant analysis (swLD a ) is a linear classifier; xD a Wn is a spatial filter for event-related potentials; p1, p2, and V2 refer to patients and volunteers included in clinical studies 1 and 2, respectively. Population Hardware Software Algorithms Cross-validation exp. 1• patients (p2)• roBiK• openViBe• riemannian potato• 5-fold (training) • Volunteers (V2)• matlab 14• mDm• Cross-session exp. 2–3• patients (p1)• tmsi• openViBe• riemannian potato• 5-fold (training) • patients (p2)• roBiK• matlab 14• mDm• Cross-session • Volunteers (V2)• xDaWn+swLDa• Cross-participant In other words, just as the mean in Euclidean space is the val- A special covariance matrix is then estimated using this su- ue minimizing the variance, the Riemannian center of mass per trial such as M of a set of points is the point minimizing their dispersion (variance) in the manifold. For every covariance matrix Σ , we ̃ � � Σ = X X (6) i i i N − 1 can compute a distance δ to the center of mass of the data set as  =  Σ , M . Finally, given all distances we can compute a i i If the data have been previously band-pass filtered in an ap- scalar geometric mean μ and and a scalar geometric standard propriate band-pass region, such a super covariance matrix deviation σ of them using the following formula [49]: contains all the spatial and temporal information needed to achieve the detection of ERPs [70], therefore it can be used � � � � directly as a single feature for the classification algorithm as a ⎛ 2⎞ � � � � 1 1 ⎜ ⎟ point on the Riemannian manifold. For each of the two class- = exp ln  ,  = exp ln , (3) L ⎜ L  ⎟ es, Target (+) and Non-Target (−), a Riemannian center of i i ⎝ ⎠ mass is estimated using the data from the calibration session. which for Riemannian distances will approximate a symmetric The center of mass can be understood simply as the expected distribution [49]. Finally, we can then derive z-scores of the covariance matrix of a trial belonging to the corresponding distances as class, wherein the use of the Riemannian metric ensures that this expectation is a much better representative as compared to the arithmetic mean. In particular, extensive testing pre- ln (4) sented in [70] has established that this expectation is more z  = . ln() robust to noise and outliers. The classification of an unseen trial is achieved by comparing the distance of the trial to the Once μ and σ are estimated on the training data, covariance ma- center of mass of the Target and Non-Target class. A classi- trices computed on new online EEG epochs are discarded when- fication score is given to unseen trials according to the score ever their distance to the center of mass exceeds a z-score of 2.5. function e s Th et of points on the manifold having a distance to a center of mass less than 2.5 standard deviations forms a closed region; in + − ̄ ̄ three dimensions, such a region would look like a potato rather s = 𝜋 𝛿 Σ ,Σ − 𝛿 Σ ,Σ , (7) i i than a sphere, because of the non-linear nature of the Riemannian manifold, which is why this technique was named the ‘Riemanni- where (x)= are probabilities found by fitting a −(+x) 1+exp an Potato’ [48]. During online experiments, visual feedback was logistic regression curve with parameter α and β [66] to the set + − + − provided to patients so that they could relate artifacts to specific ̄ ̄ ̄ ̄ of distances 𝛿 Σ , Σ and 𝛿 Σ , Σ and Σ and Σ are the i i behaviors (blinks, coughing, head movements). This ensures that center of mass of the Target and Non-Target classes, respective- they can associate any artefactual behavior with its impact on the ly. These scores are averaged across repetitions, and the symbol data quality and take corrective actions: move less and blink at is assigned to the symbol at the intersection of the row and appropriate times as much as possible. The Riemannian potato column with highest score. was implemented in OpenViBE for online experiments (Study 2) and in Matlab for offline experiments (Study 1). Appendix 3. Offline analysis Online classifier This offline analysis has the following objectives: Using the Riemannian distance in Equation (1) and the defini- • Experiment 1: comparison of the personalized BCI with tion of center of mass, in Equation (2) the detection of the P300 the traditional assistive technology; evoked potential can be achieved by a deceptively simple clas- • Experiment 2: assessment of the individual contribution sification algorithm named the minimum distance to means of each component (RoBIK headset, artifact-detection (MDM) [50]. Denoting by + the target class of flashes, each tri- algorithm, classifier algorithm) to the overall system al X is concatenated with a prototypical P300 evoked response performance; P (for instance, the ensemble average estimation obtained on • Experiment 3: assessment of the performance of a cali- the training data) to build a ‘super’ trial: bration-free BCI system. X = . (5) i BRAIN-COMPUTER INTERFACES 213 of 10 characters should correctly be identified. The optimal Materials and methods number of repetitions was chosen accordingly, corresponding Elements of the materials and methods are summarized in bit-rates were derived as described in Equation 1, and subse- Table 4. quently compared to that of the scanning device for the same patients. Offline analysis 1: optimal BCI compared to scanning device Offline analysis 2: added value of the prototype design e co Th mparison of performance for communication interfaces A second offline analysis was run to quantify the added value is usually assessed using the bit-rate [67,68]. The advantage of of the proposed BCI prototype design. More precisely, we used bit-rate over accuracy as a performance measure is that it also an offline study to ensure that the hardware design, which was takes into account the speed of the interface and the amount of meant to increase ease of use, came at no cost in performance choices it oer ff s. Bit-rates thereby gives a more representative and we similarly quantified the added value of the Riemannian metric of ‘information flow’ from the participant’s brain to the approach for artifact trial rejection and classification. To do so, of- machine. The bit-rate br is defined as fline analysis 1 was repeated with and without online artifact trial rejection using the SQI index described in Appendix 2. Likewise, 1 − p br = log (M)+ p.log p +(1 − p).log , (1) the performance of the Riemannian MDM classifier was com- 2 2 2 M − 1 pared to a state-of-the-art technique composed of the xDAWN where M is the number of symbols and p the spelling accu- spatial filter combined with a stepwise linear discriminant anal- racy (the percentage of correctly identified symbols). In this ysis (SWLDA) [69]. The comparison was carried out with and study, M equals 36 and 73 for the RoBIK BCI and the scanning without the artifact trial rejection and on both clinical datasets device, respectively. (Studies 1 and 2). Because the RoBIK interface natively inter- The evaluation of the RoBIK prototype was designed to rupts visual stimulations in the presence of noise, it is expected demonstrate the feasibility of a well-designed BCI prototype that most target and non-target epochs from Study 2 are clean. for communication in a clinical context. Therefore, the stim- For this reason, only data from Study 1 were considered to assess ulation parameters (flash duration, average time between two the benefit of artifact rejection, while both datasets were used to stimulations, and number of repetitions) were chosen to pro- compare the classifiers. mote optimal accuracy rather than speed. The system evaluat- ed at the bedside in this study was not optimized for bit-rate, Offline analysis 3: cross-participant-analysis but for optimal data collection and robustness in the context A calibration-free ERP-based BCI has been proposed by of a clinical feasibility investigation. This approach ensured Congedo [70] and Barachant et al. [48]; the center of mass the collection of a large amount of data for the offline analysis. of the available classes is initialized using a database of pre- For the BCI performance, we consider the bit-rate achieved vious users and then continuously updated using the in- with the optimal number of repetitions. In order to assess the coming data from the online sessions of the user. In order optimal number of repetitions, the MDM classifier described to assess the potential of such a approach, offline analysis 2 above was fitted to the training data after artifact identifica- was run a second time replacing the training set by all data tion using the aforementioned SQI index. The test set was a available from other participants (cross-subject learning). bootstrapped dataset generated from the ‘online’ session of While offline analyses 1 and 2 used little data from the same each participant as follows: for each number of repetitions participant, more data are available in this offline analysis; of the stimulation k  =  1…50 (that is the number of times a however, they belong to different participants. To allow for specific letter is flashed before a decision is made), B = 1000 cross-subject comparison, all covariance matrices for the groups of k target and five times k (5000) non-target respons- MDM models were normalized so as to have a unit deter- es were randomly selected (with replacement) and averaged, minant as: so that the target to non-target ratio was preserved. Then, for each group of six responses, the predictions were obtained by −1∕E Σ =Σ ∗ det(Σ) (2) norm assigning 1 to the element featuring the minimum distance to which derives directly from the following property of the de- the center of mass of the Target class and 0 to the five other terminant: det(cΣ) = c det(Σ), where det (Σ) denotes the de- elements. The seed for the random sampling was preserved in terminant of matrix Σ, c is a scalar, and E the size of the matrix. order to evaluate the performance of the different techniques on the same random bootstraps. At the end of this procedure, 6000 predictions were used to derive a performance index Statistical analysis for each participant and for each number of repetitions of The benefit of the proposed BCI design was evaluated the visual stimulation (the flash). In particular, the ‘real ac- by means of a Kruskal-Wallis test, using the technique curacy’ was defined as the square of the raw accuracy, since methods as an independent factor and the bit-rate as the each letter is found at the intersection between two groups of dependent variable. Using the best-performing of each symbols (commonly referred as the ‘lines’ and ‘columns’ in method, optimal bit-rate was identified (at 70% real accu- the traditional P300 speller experiments). If predictions were racy) and compared to that of the scanning system on the to be made at random, the ‘real accuracy’ would be 1 in 36, same patients using a paired Student t-test. For the repeated- meaning that there is a probability of 2.8% that the correct measures framework above, in this manuscript we will refer letter is selected by chance. With such a definition it was ar - to group factor, time factor, and time-group interaction bitrarily decided that an interface should provide a minimum factor and related effects. ‘real accuracy’ of 70%, meaning that on the average 7 out 214 L. MAYAUD ET AL. Figure 6. evolution of ‘real accuracy’ (% of correctly identified characters) in relation to number of flashes of each symbol under different conditions; the three rows of plots indicate the performance in (1) cross-validated training data, (2) test-set (online) data, and (3) cross- subject performance. t he first column of plots shows the effect of artifact removal using the sQi on the data from s tudy 1 only. t he second column shows the impact of the classifier comparing xD aWn + stepwise linear discriminant analysis (sWLD a ) and minimum distance to mean (mDm) r iemannian classifier. t he third column compares the performance of healthy volunteers and patients. t he fourth column compares the data collected with a traditional eeG system (s tudy 1) to that collected with the roBiK headset (s tudy 2). f or all factors considered, data are collapsed across the other factors. Table 5. Bit-rates (bits per minutes) for the clinical sample included in the clinical study of s tudy 2 for scanning device (session 1 only), the roBiK system during the online session, and the performance computed offline using the same data and algorithms but with the use of an optimal number of repetitions (personalized). patients 5, 8, and 9, for whom no BCi signal could be exploited, were removed from this analysis and subsequent statistical testing. P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 s canning 1 24.5 22.6 30.8 20.3 18.9 33.9 20.7 na 25.1 14.5 online roBiK 3.1 1.7 1.7 0.2 0.0 1.4 0.1 na 0.0 0.1 personalized roBiK 27.8 27.8 23.0 27.8 0.0 1.7 2.9 na 0.0 31.5 volunteers and patients and benchmarks the performance of Results the RoBIK EEG headset versus a traditional disc electrode Added value of signal processing and classification system (TMSi amplifier). As detailed in Figure 6 , these com- parisons were carried out at different levels: with a five-fold algorithms cross-validation on the training set, prospectively on the on- Figure 6 shows the influence of several factors with respect line dataset, and using cross-participant fitting. For the latter, to real spelling accuracy: the presence of an artifact- only data from other participants wee used to fit the model. removal technique (the SQI) and the use of a traditional The comparison of optimal (maximum) bit-rates over all algorithm (xDAWN+SWLDA) versus a Riemannian algo- rithm (MDM). It also compares the performance of healthy repetitions for these conditions reveals a significant benefit BRAIN-COMPUTER INTERFACES 215 of the MDM over the xDAWN+SWLDA approach (paired MDM requires only the computation of centers of mass and t-test, t = 32.4, permuted p < .001). This was particularly true distances between two points and has no free parameters to for the cross-participant condition where only the MDM be tuned by cross-validation or heuristics, therefore it reduces model could generalize well (paired t-test, t = 42.5, permut- the risk of over-fitting. In addition to this, Riemannian meth- ed p  <  .001). Other factors were not found significant. This ods generalize well across sessions (see Figure 6, line 2 col- seems to indicate that patients and healthy volunteers have umn 2) because the Riemannian distance function is invari- equivalent performance. Likewise, the performance of the ant by any linear transformation of the data and covariance RoBIK headset did not significantly differ from that of the shifts observed across sessions due to movement in headset traditional EEG system, which strengthened the rationale positions or changes in electrode impedance are of this type. for the improvement in setup time (from 30 minutes to 10 As expected, the performance of the cross-participant model minutes). Finally, these results also indicated that the use of was found to be significantly lower than that of the ‘online’ an artifact-rejection technique does not necessarily improve mode (trained on data from a training session from the same the accuracy, although it results in a reduction of variability, participant), which implies that under these conditions online which can be noticed only with the use of a traditional EEG adaptation may be required to keep the performance of the recording system. calibration-free BCI optimal. e u Th se of the SQI to reject trials contaminated by arti- facts did not show clear benefit in terms of bit-rate. However, Comparison of optimal BCI to scanning device Figure 5 (column 1) shows that monitoring artifacts and stop- As seen in Table 5, seven out of nine patients (77%) who ping the operation in the presence of high noise reduce the completed the Online session during Study 2 could success- variability of the system, which is an important benefit for re- fully use the P300 speller. For those (we excluded two addi- al-word operation. It should be noted that our study has been tional patients P05 and P09 who could not use the system carried out in a realistic environment. Even though the data at all), the comparison of bit-rates did not show statistical were acquired under the strict supervision of an EEG techni- significance between the first scanning session and the cian, they were collected in an environment prone to artifacts. personalized RoBIK performance (T = 0.56, permuted We have made the data de-identified, open, and accessible to p-value = .33). peers in the digital supplement of this article so that the com- munity of researchers can investigate this and other questions further. Discussion No difference in performance was found between the First, the results confirm the superiority of the Riemannian RoBIK and the TMSi headset, even though lower variability MDM approach over traditional classification based on spatial can be observed for the TMSi headset. This supports the well- filtering (for Online mode, line 2 on Figure 6 ). This finding known fact that a gel-based cap collects EEG data more con- replicates previous reports on healthy participants [49,50,70]. sistently. Nonetheless, the RoBIK headset has lower technical In some patients, the Riemannian MDM oer ff s real accuracy specifications (no true DC, 12 bits instead of 24), is there- above 80% with as little as two repetitions, which favorably fore cheaper, and allows a dramatic reduction in setup time compares with state-of-the-art techniques [71]. This supe- (approximately three-fold). While the headset prototype test- riority was even more evident for the cross-subject transfer ed had numerous drawbacks (design, weight, corrosion with learning (Cross-subject, line 3 on Figure 6), which is also in saline water, mechanical electrical contact to name a few) it line with our previous investigations [50,70]. Overall these shows that cheap and easy-to-use EEG systems for bedside results suggest that the Riemannian MDM classifier oer ff s applications are becoming a reality. The trending develop- better generalization properties as compared to spatial filters. ments in dry electrode systems [18,20,71] may be expected Interestingly, the advantages of the MDM algorithm in terms to lower the tolerance threshold further and broaden the use of accuracy and generalization are accompanied by a dramat- of BCIs. ic reduction of the algorithmic complexity; the Riemannian http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Brain-Computer Interfaces Taylor & Francis

Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device

Download PDF

Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device

Abstract

AbstractThis study presents the outcome of the 5-year-long French national project aiming at the development and evaluation of an effective brain-computer interface (BCI) prototype for the communication of patients with acute motor disabilities. It presents results from two clinical studies: a clinical feasibility study carried out partly in the intensive care unit (ICU) and the clinical evaluation of an innovative BCI prototype. In this second study the BCI performance of patients was compared to that of healthy volunteers and benchmarked against a traditional assistive technology (scanning device). Altogether, 15 of 22 patients could control the BCI system with an accuracy significantly above the chance level. The bit-rate of the traditional assistive technology proved superior, even though an equivalent bit-rate could be achieved using personalized parameters for the BCI. Fatigue was found to be the primary limitation factor, which was particularly true for patients and during the use of the BCI. A classifier based on Riemannian geometry was found to contribute significantly to the accuracy of the BCI system. This study demonstrates that the communication of patients with severe motor impairments can be effectively restored using an adequately designed BCI system. All electrophysiological data are freely available at the Physionet.org platform.

Loading next page...
 
/lp/taylor-francis/brain-computer-interface-for-the-communication-of-acute-patients-a-7bbfmRuccd

References (61)

Publisher
Taylor & Francis
Copyright
© 2017 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
ISSN
2326-2621
eISSN
2326-263x
DOI
10.1080/2326263X.2016.1254403
Publisher site
See Article on Publisher Site

Abstract

Brain-Computer interfa Ces, 2016 VoL. 3, no . 4, 197–215 http://dx.doi.org/10.1080/2326263X.2016.1254403 OPEN ACCESS Brain-computer interface for the communication of acute patients: a feasibility study and a randomized controlled trial comparing performance with healthy participants and a traditional assistive device a,b,c,d c a,b,c h Louis Mayaud   , Salvador Cabanilles , Aurélien Van Langhenhove , Marco Congedo   , h a,b,c e f g b,c Alexandre Barachant , Samuel Pouplin , Sabine Filipe , Lucie Pétégnief , Olivier Rochecouste , Eric Azabou , c a a,b,c a,b,c Caroline Hugeron , Michèle Lejaille , David Orlikowski and Djillali Annane a b inserm, Centre d’i nvestigation Clinique et d’innovation technologique (CiC-it ), umr805, Garches, f rance; inserm, equipes t hérapeutiques innovantes et t echnologies appliquées aux troubles neuromoteurs, u . 1179, Garches, f rance; Hôpital r aymond poincaré, apHp, Garches, f rance; d e f mensia technologies sa, paris, f rance; Department DtBs, Cea/Leti, Grenoble, f rance; DiXi microtechniques medical, Besançon, f rance; g h Davidson Consulting, rennes, f rance; Gipsa-Lab, Cnrs, university of Grenoble- alpes, Grenoble i nstitute of t echnology, Grenoble, f rance ABSTRACT ARTICLE HISTORY received 12 m arch 2016 This study presents the outcome of the 5-year-long French national project aiming at the a ccepted 26 o ctober 2016 development and evaluation of an effective brain-computer interface (BCI) prototype for the communication of patients with acute motor disabilities. It presents results from two clinical studies: KEYWORDS a clinical feasibility study carried out partly in the intensive care unit (ICU) and the clinical evaluation assistive technology; of an innovative BCI prototype. In this second study the BCI performance of patients was compared communication to that of healthy volunteers and benchmarked against a traditional assistive technology (scanning aids; event-related device). Altogether, 15 of 22 patients could control the BCI system with an accuracy significantly potentials, p300 speller; electroencephalography; above the chance level. The bit-rate of the traditional assistive technology proved superior, even r iemannian geometry though an equivalent bit-rate could be achieved using personalized parameters for the BCI. Fatigue was found to be the primary limitation factor, which was particularly true for patients and during the use of the BCI. A classifier based on Riemannian geometry was found to contribute significantly to the accuracy of the BCI system. This study demonstrates that the communication of patients with severe motor impairments can be effectively restored using an adequately designed BCI system. All electrophysiological data are freely available at the Physionet.org platform. Introduction facilitate computer access are available [3]. In particular, occupational therapists equip patients with ‘scanning’ Lack of communication in the hospital may be a great systems [4–6] enabling interaction with computers or source of distress for both patients and caregivers, which other devices such as speech synthesizers. The assistive is particularly exacerbated in the acute context of inten- technology (AT) in this case consists of a ‘click interface’ sive care units (ICUs) [1]. Several neurological disorders (or switch) connected to an application where a limited impair communication abilities while leaving cognitive number of options are visually ‘scanned’ (highlighted) for capacities almost untouched. This has long been reported the patient. er Th e are many types of available switches to for chronic conditions such as myopathies, spinal cord control a scanning system [7]: muscular switches, tactile injuries, amyotrophic lateral sclerosis (ALS), and multiple switches, puff switches, and mechanical switches, the most sclerosis (MS), but is also true for acute neuropathies such common type in France, which consist of a press-down as the Guillain-Barré syndrome (GBS). Moreover, quad- button whose activation force, size, location (feet, near the riplegic or locked-in syndrome (LIS) patients undergoing head, thumb), and type of activation (validation on press- invasive mechanical ventilation (via endotracheal tube or down or release) are set according to the patient’s specific tracheotomy) see their communication abilities impaired condition. Whenever a reliable switch command can be by the sudden loss of speech [2]. obtained, the system is connected to a software interface. e u Th se of computers in this context can facili- Typically, this consists of a grid of symbols displayed on a tate communication and many technical solutions to CONTACT Louis mayaud louis.mayaud@gmail.com © 2016 t he a uthor(s). published by informa uK Limited, trading as t aylor & f rancis Group. t his is an open a ccess article distributed under the terms of the Creative Commons a ttribution-nonCommercial-noDerivatives License (http://creativecommons.org/licenses/by-nc- nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 198 L. MAYAUD ET AL. screen, which are flashed at a regular pace (i.e. the ‘scan- path to low-cost EEG technology for the general public, ning’) until the patient selects one. The configuration of although the quality of the signal (hence the overall BCI both the physical and the software interface requires a performance) has proved inferior to research-grade EEG tremendous amount of time and expertise, which is one acquisition systems [25,26]. Thus, to date, a state-of-the-art reason why, as a matter of fact, patients with severe motor EEG-based BCI requires the support of a technical and disabilities in acute contexts such as the ICU are oen ft left clinical team composed of highly qualified professionals unequipped. [27–29]. A major concern for the present study is the e b Th rain-computer interface (BCI) is a longstanding possibility of generating and processing visual ERPs in an technology [8] that translates the brain electrical activ- ICU environment overflowing with uncontrolled sources ity into a command for a device [9,10]. BCIs are usually of noise, both acoustic (alarms, sta, ff mechanical ventila- characterized by: (1) a brain activity recording modality, tion) and electromagnetic (bedside monitors, automated (2) a paradigm that relates sensorial stimulations to some syringes, mechanical ventilation). While EEG applications specific brain activity, and (3) an interface connecting to are known to be particularly sensitive to non-controlled the front-end application by means of pattern-recogni- environments [16,30], to the best of our knowledge no tion techniques. Electroencephalography (EEG)-based report exists about their use in an ICU. BCIs [11,12] have been extensively studied since EEG is P300-based BCI also have specific limitations. First of a non-invasive and portable neuroimaging modality. In all, since several ERPs need to be averaged to obtain a this article, we focus on BCI technology based on the P300 sufficient SNR, the overall bit-rate is low in practice (up event-related potential (ERP). A P300 is a positive ERP to 1 minute per letter in a P300 speller) [31,32]. The signal occurring 300 to 500 ms aer t ft he presentation of a rare processing and machine learning research communities meaningful stimulus presented in a flow of many irrel- actively address this issue by improving ERP detection evant stimuli [13]. e Th repeated presentation of several methods [33–35]. Also, the use of a P300 speller requires stimuli among which there is a possible target for the user constant concentration and there is still a debate on is referred to as the ‘oddball paradigm’ [13–15]. The ‘P300 whether some clinical populations possesses the necessary speller’ is currently the best-known and most widespread attention span to effectively control a P300 speller, espe- ERP-based BCI communication interface. In a typical cially if the session is prolonged [36,37]. This is particu- P300 speller a regular grid of symbols is displayed on the larly true for populations of patients admitted to the ICU screen while lines and columns are flashed at random. The where central nervous system (CNS) depressant drugs participant is asked to concentrate on a target symbol. A are usually administered as they are known to influence P300 is elicited in response to the flash of the line and the vigilance and attention [38]. It is worth noting that while column containing the target symbol. Since the ampli- P300 speller BCI systems have traditionally been clinically tude of the P300 is much smaller as compared to the EEG evaluated in populations of chronic patients such as those background activity and EEG artifacts, the ERP is elicited suffering from ALS [27,28,30,39], its potential benefit in several times and averaged in order to increase the signal- acute nervous conditions, such as GBS, have not yet been to-noise (SNR). The averaging procedure increases the reported. discrimination ability (target ERPs vs. non-target ERPs), To complete the technological transfer from ‘bench to effectively resulting in a trade-off between accuracy and bedside’, BCI must gain ease of use and robustness in terms speed of selection. The P300 speller represents a great of both algorithms and interface (signal-processing and hope for severely disabled patients, especially for those applications). Overcoming the aforementioned technical who are not able to use traditional AT. challenges appears important, particularly in the context Despite the tremendous amount of research over the of a better assessment of pain and cognitive function past 20 years, EEG-based BCIs still sue ff r from signic fi ant (perhaps identification of delirium) of non-communicat - drawbacks such as setup time and sensitivity to noise [16]. ing patients. This would certainly result in an improved For instance, the placement of electrodes connected to the quality of ICU stay. The Robust Brain -computer Interface scalp skin through a conductive gel is a time-consuming for virtual Keyboard (RoBIK) project [40] aimed at the process, certainly prohibitive for everyday use in both development of a BCI system for the communication of hospitals and at the patient’s home. Moreover, caregivers patients that could be used on a daily basis in the hospital are usually not familiar with EEG technology. Attempts with limited external intervention. In order to achieve this to overcome this limitation include the use of caps with goal a multidisciplinary approach was chosen and devel- pre-positioned gel-based [16,17] or dry [18–20] electrodes, opments were carefully framed by clinical specifications possibly interfaced with active recording systems [21,22] and validation, according to a user-centered design. The Recently, the EPOC headset [23,24] has opened the project resulted in two clinical studies: BRAIN-COMPUTER INTERFACES 199 Table 1. summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-val- idation procedures. abbreviations: minimum distance to mean (mDm) is a r iemannian classifier; support vector machine (sVm) is a classifier. xD a Wn is a spatial filter for event-related potentials. Population Hardware Software Algorithms Cross-validation s tudy i• xDaWn• 20-fold (training) • patients (p1)• tmsi• openViBe • sVm• Cross-session s tudy 2• patients (p2)• openViBe• riemannian potato• 5-fold (training) • roBiK • Volunteers (V2)• python 2.7• mDm• Cross-session • Study 1: Identification of the clinical and technical boards (‘Comité de protection des personnes’, CPP, Saint limitations aeff cting the use of BCI applications Germain-en-Laye, 2009/10/15). Quadriplegic patients for the communication of acute patients in a clin- admitted to ICU and rehabilitation unit at a tertiary-care ical setting; a questionnaire was administered to hospital were enrolled aer g ft iving their informed consent. patients and caregivers [41] in order to identify Pregnant women, illiterate patients, patients under judi- appropriate user specifications; cial protection or without social security as well as patients • Study 2: the clinical validation of a BCI system with a history of epilepsy were excluded from the study. developed with specifications derived from Study I, e p Th rotocol consisted of three P300 speller which consisted of sessions [15]. ◦ An EEG headset with portable electronics (1) A training session was given during which designed for convenience of use in the hospital patients were instructed to spell two words (Appendix 1), (‘OCEAN’ and ‘NUAGES’, meaning in French ◦ dedicated sowa ft re comprising a user-friendly ‘Ocean’ and ‘Clouds’, respectively) and for interface and a classification algorithm based on which no feedback was provided. Signals col- Riemannian geometry (Appendix 2); the perfor- lected were then used to train a state-of-the-art mance of this BCI prototype on clinical samples pattern-recognition algorithm based on a spa- was compared with a state-of-the-art assistive tial filter and a classifier, detailed in the next device for the communication of patients (scan- section [33]. ning device), in order to assess the actual clinical (2) e c Th lassifier performance was evaluated dur - value of the two technologies, and withhealthy ing an ‘online’ session during which instruc- volunteers, in order to quantify the drop in per- tions were given for three words (‘OEUFS’, formance induced by the patients’ condition. ‘AVION,’ and ‘OASIS’, meaning in French ‘eggs’, Finally, an offline analysis of the data collected during ‘airplane’, and ‘oasis’, respectively). The letters these two clinical trials was carried out in order to quan- identified by the algorithm aer ft each sequence tify the contribution of each system’s element (hardware of stimulations were displayed on the screen as and software) to the whole and to estimate the perfor - a feedback. mance of a personalized BCI. The details of this analysis (3) Finally, an optional ‘free-spelling’ session was are available in Appendix 3. oer ff ed to the patient, during which no instruc- tion was given and patients could spell what- Materials and methods ever they wished. Table 1 summarizes the populations, hardware, software, At the end of the session patients were requested to algorithms, and cross-validation techniques used in each rate their satisfaction with the BCI system as a commu- clinical trial. nication tool by means of a visual analogue scale (VAS) ranking from 0 (completely unsatisfied) to 10 (completely satisfied). Study 1: clinical feasibility e Th aim of this study was to evaluate the usability of a Data acquisition and processing state-of-the-art P300 speller BCI for communication [15] Twenty-four silver chloride (AgCl) disk-electrodes in the challenging environment of intensive care and reha- were connected by active-shielded coaxial cables to a bilitation units. Porti32 EEG amplifier (TMSi, Twente, Netherlands) sampling at 512  Hz. Signal acquisition, processing, and Study design and protocol storage were performed using the open-source platform e p Th rotocol was registered on ClinicalTrials.gov OpenViBE [42]. Raw EEG signals were stored as GDF (NCT01005524) and received clearance from local ethics 200 L. MAYAUD ET AL. format [43] including the type and location of visual which were implemented as described in this section and stimulations. This data-set is freely available on the used for the second clinical study. Technical details can be Physionet platform . found in Appendices 1 and 2. As pre-processing, all EEG signals were band-pass-fil- tered between 0.01 and 30 Hz by means of a Butterworth The EEG headset fourth-order filter with linear phase response. e Th e R Th oBIK EEG headset, shown in Figure 1 (left), has xDAWN spatial filter [33] was trained on the training been designed to be an effective means of recording EEG session. xDAWN is a spatial filter specifically designed activity usable by people not familiar with EEG technol- for ERP data [44]. The aim of a spatial filter in this context ogy, while oer ff ing performance equivalent to that of a is to enhance the signal of interest (ERPs) while suppressing clinical-grade EEG system. The resulting headset has silver the background noise [45]. It also dramatically reduces chloride (AgCl) electrodes mounted at the following 14 the dimensionality of EEG data, since a few filters suffice standard 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, to summarize the signal of interest (ERPs). e Th spatially P7, Fp2, F3, P8, Po8, and Cz. Connection to the scalp skin filtered signal is fed to an ensemble of support vector is obtained by means of a cotton pad soaked with physi- machine (SVM) classifiers for identification of target and ological saline solution. e Th signal is sampled at 580 Hz non-target stimulation [46]. with 12-bit resolution. Cross-validation scheme The user interface For each participant the accuracy was assessed with a e a Th pplication and generation of visual stimulations were 20-fold cross-validation procedure on the training ses- handled by a dedicated application called ‘Brainmium’ sion. A K-fold validation [47] is a technique oen u ft sed (Figure 1, right). It was designed to be user-friendly so that to estimate the performance of a classification technique a caregiver with basic knowledge of informatics can oper- (and its variability) on ‘unseen’ data. At each fold, 20% ate it aer a s ft hort training. Brainmium implements state- of the data was left aside for model validation (the ‘test’ of-the-art P300 paradigms, using random inter-stimulus set) while the remaining 80% (the ‘training set’) was used interval (ISI) and random group flashing (different from to design the model (the xDAWN spatial filter and the the line-column paradigm), which are meant to reduce ensemble of SVMs). Online letter-recognition accuracy visual fatigue and increase discriminatory power. Details during the test session was simply obtained by applying can be found in Appendix 2. models derived from the training session, live, to the online data. Online artifact rejection In this work we use a signal quality index (SQI) in order to reject trials contaminated by excessive artifacts during Technical developments both the training and the online phase of the experiment. Following this feasibility study and results from a survey e Th SQI is an improvement of the ‘Riemannian potato’ conducted with patients and caregivers [41], we identi- methods [48] based on the suggestion from Congedo [49]. fied optimal hardware and software BCI specifications, Technical details are given in Appendix 2. Figure 1. t he roBiK prototype: the roBiK headset (left) with 14 wet eeG channels and the virtual keyboard interface ‘Brainmium’ (right). BRAIN-COMPUTER INTERFACES 201 Online classifier e o Th nline classification was performed by means of the Riemannian minimum distance to means (MDM) classi- fier [50] as applied to P300 data [49]. The standard method was complemented with a new logistic decision function as detailed in Appendix 2. Study 2: clinical evaluation e p Th rimary objective of this study was to evaluate the performance of the RoBIK BCI prototype and to com- pare its performance to that of a traditional assistive Figure 2.  patients were randomly assigned to try first the roBiK technology. The second objective was to compare the BCi or the scanning device. t he roBiK prototype was evaluated performance of patients using the BCI prototype to that within half a day while the scanning device was evaluated over of healthy volunteers. three sessions in order to account for a possible learning effect. f our different texts of equivalent difficulty were selected so that the roBiK device was evaluated with text 1 or 4. Study design and setting The protocol was registered on ClinicalTrials.gov (NCT01707498) and received clearance from the Bedside procedure local ethics board (CPP de Saint Germain-en-Laye, e R Th oBIK prototype was setup by an occupational ther - 2012/07/05) and the regulatory agency for the use of a apist who had no specific training in electrophysiology. non-CE-marked device (Agence Nationale de Sécurité e p Th rotocol consisted of the following steps: des Médicaments, ANSM, 2012/09/14, 2012-A00613– (1) e Th EEG headset was set up and time was 40). The clinical protocol inclusion criterion for patients counted from the beginning of the installation was the presence of functional quadriplegia. The inclu- to the beginning of step 2; the operator was sion criterion for healthy participants was age greater instructed to reduce the presence of power line than 18 years. Patients were enrolled in the intensive contaminations (50 Hz) by adjusting electrode care unit (ICU) and rehabilitation units at a tertiary care positioning until the presence of eye blinks, hospital after giving their informed consent. Healthy jaw muscular artifacts, and alpha waves could volunteers were enrolled at the Center for Clinical be observed when the patient was instructed to Investigation and Technological Innovation (CIC-IT) blink, clamp his jaw, and close his eyes, respec- after giving their informed consent. Pregnant women, tively. The impedance between each electrode patients with hemodynamic instability, illiterate par- and the reference was measured at the begin- ticipants, patients under judicial protection or without ning and end of each session and maintained social security, epileptic patients, and participants show- below 5 kΩ. ing skin/scalp sensitivity or severe visual impairment or (2) e in Th terface was presented to the patient and aged less than 18 years were excluded from the study. directions for the training sessions were given. e exp Th erimental design is shown in Figure 2; patients When possible, the participant was asked to were randomly assigned to try the RoBIK prototype or re-formulate to make sure the task was cor- the scanning device (described below) first. Because there rectly understood. are no well-documented learning effects in P300-based (3) e Th training session consisted of the spelling of BCIs [51], the RoBiK prototype was evaluated only over 10 consecutive characters. The participant was a single half-a-day session. Instead, the scanning speller instructed to concentrate on the chosen letter performance was estimated over three sessions in order to and count the number of times it flashed. Each estimate a possible learning effect [52]. Each session was letter was flashed 20 times. The target letter was separated by at least a half-day wash-out period in order continuously indicated in the upper section of to avoid a possible bias due to fatigue. the screen and each new letter was notified by At the end of each session we assessed the fatigue and printing in blue the corresponding key on the satisfaction of patients by means of a VAS ranging from 0 virtual keyboard before each new sequence of (very tired / very unsatisfied) to 10 (not tired at all / very random stimulation. satisfied). In both cases participants were asked to fill a (4) Data collected during this training session questionnaire to express their opinion on both techniques were automatically passed down the processing (including comfort and fatigue). 202 L. MAYAUD ET AL. pipeline for training to the MDM classification Statistical analysis algorithm. The data from the training sessions For all tests, samples were tested for normality by means were split into five distinct training and test of the Jarque-Bera test (significance level at 5%) and ana- sets in order to estimate the ‘real accuracy’ lyzed with non-parametric or parametric tests depending (percent of correctly identified characters on whether the hypothesis of normal distribution of the within 36 choices) on unseen data. e p Th atient data was rejected or not, respectively. Tests for comparing was allowed to continue the protocol and pass central location parameters (means) were chosen paired to the ‘online’ session if the estimated accu- or unpaired according to the test at hand. Differences racy was above 70%, meaning that, on average, between the healthy volunteers and patients (unpaired) three spelling errors every 10 characters were for the variables age, setup time, comfort, performance tolerated, otherwise the participant was dis- during the training and online sessions were tested using charged from the study. the Student t-test or the Mann Whitney U-test for nor- (5) e ‘ Th online’ session was equivalent to the pre- mally and non-normally distributed samples, respectively. vious one except that visual feedback was Comparison of fatigue, setup time, and spelling accuracy provided to the patients by the classification in patients (paired) using the RoBIK and the scanning sys- algorithm. The number of repetitions for tem (paired statistics) was assessed with a paired Student the online session was chosen to maximize t-test or a Wilcoxon test depending on whether data were the estimated accuracy on the training data. found to be normally distributed or not. Comparison of e Th online session was timed to last exactly fatigue in patients for each interface was estimated with 10 minutes, during which participants were a repeated-measures ANOVA that uses a multivariate instructed to spell as many letters as possible framework (Hotelling T-square) to account for cor- without corrections. relation between measures [53], which therefore does (6) Finally, patients could use the interface during not need correction for sphericity [54]. Comparison of a ‘free-spelling’ session where no instruction fatigue between healthy volunteers and patients using the was given, but an output was still provided. RoBIK interface was estimated with the same technique es Th e data are freely available on the Physionet considering two independent groups of participants. For platform . all statistical tests the tolerance for type I error was set to 0.05. All analyses were performed using Matlab (Version 8.0.0.783 - R2012b) and toolboxes referenced. The scanning system An occupational therapist, as part of every day’s clinical Results activity, was in charge of equipping patients with a ‘switch interface’ (‘Buddy button’ switch, Ablenet, Roseville, MN, Study 1: clinical feasibility USA). The nominal activation force of the switch varied Patients from 10 to 600 grams. Depending on the specific condi- Twelve quadriplegic patients admitted in adult medical tion of the patient the following parameters were tuned by ICU [7] and rehabilitation units [5] met the inclusion cri- the occupational therapist: activation force, size, location teria and were consecutively included in the study aer ft (feet, near the head, thumb), type of activation (valida- giving their informed consent. Table 1 summarizes the tion on press-down or release), and possible filtering of sample, composed of eight men (67%) and four women, double-clicks. Once a reliable switch command could be aged from 22 to 63 years old. e Th tolerance was good in obtained, the system was connected via the ‘Joycable’ USB all patients. Four patients (33%) did not complete the pro- interface (Sensory software, Malvern, Worcestershire, tocol. Access to the occipital area was complicated by a UK) to the KEYVIT virtual keyboard (Jabbla, Ghent, central catheter in the first patient (#01). In one patient Belgium). On this interface, a regular grid of symbols a technical issue interrupted the inclusion (#02). One flashes lines at a regular pace (i.e. the ‘scanning’) until patient fell asleep during the training session (#10). The the patient selects one line by activating the switch. Once last patient (#11) was suspected to have eyesight problems, a line is selected, each element of the line is then flashed although this could not fully be confirmed by medical files. until a second click selects the desired symbol or letter. Aer in ft stallation of the interface, patients were asked to P300 speller for communication spell as many characters as they could within 10 min. The The results of all patients included in Study 1 are pre- instruction text was printed big enough so that it could sented in Table 2. Out of 10 patients who completed be seen as displayed on an A4 sheet of paper next to the the training session, eight used the system during a test screen. BRAIN-COMPUTER INTERFACES 203 Table 2. Description of the population included in s tudy 1 with results: a C, access to computer; Hm, voluntary head mobility; oe, oral expression; mV, mechanical ventilation, et , endotracheal intubation. ‘t reatments’ details drugs administered to patients during the 48 h preceding the inclusion in the study. t raining session performance is evaluated with a 20-fold cross-validation procedure on the training data using a combination of xDaWn and sVm. i t shows discrimination – area under the receiver operating curve (auroC) – between targets (33% of stimuli, i.e. one line and one column) and non-targets (random classifier is 50%). t esting and online performance shows accuracy in letter selection (1 symbol in 36; chance is 2.7%). t he last lines of the table summarize the distributions: categorical variables are represented by the proportion of each category (noted in parenthesis) and continuous variables are represented by the mean and standard deviation; whenever relevant, the sample size is indicated in parenthesis. f or speller performance, brackets indicate the average number of letters per patient. P-values are computed with a mann-Whiteney U-test (*). Free spelling Training Testing session Session Percent Type of Treatments session AUROC Percent Accuracy Accuracy (%) Satisfac- ID Service Age Gender Origin of handicap HM OE AC MV ET (up to 48 hours) (# letters) (# letters) (# letters) tion EVA Details 1 iCu 21 f emale f unctional post-surgery Y Y n Y Y sufentanyl, m ida- n.a.     n.a. eeG leads could not Quadriplegia (severe zolam, Gabapentine, be setup idiopathic scoliosis) Hydroxyzine 2 iCu 62 f emale post-traumatic quadriplegia Y n n Y Y alprazolam, 80.0% (20) n.a. t echnical issue C5-C6 Hydroxyxine, with acquisition prégabaline, software escitalopram, alimemazine 3 iCu 30 male post traumatic spinal cord n n Y Y Y Baclofene, 73.0% (15) 60.00% (5)   n.a.   hematoma C1-C5 Clonazepam, s odium Valproate, Hydroxyzine 4 iCu 25 male Duchene’s muscular n n Y Y Y t ramadol , alpra- 85.0% (10) 100.00% (15) 100.00% (8) 9 dystrophy zolam, mirtazapine, Hydroxyzine 5 iCu 22 male Duchene’s muscular Y n Y Y Y none 68.0% (10) 73.30% (11) 72.70% (11) 7.6   dystrophy 6 rehab 32 male post-traumatic Quadriple- n n Y n Y Zopiclone , 89.0% (10) 100.00% (15) 100.00% (6) 9 gia C4 escitalopram, alprazolam 7 rehab 58 male Locked-in s yndrome (Lis, Y Y Y n Y Bromazepam, 77.0% (10) 93.30% (14) 100.00% (15) 9.9   s troke) Baclofene 8 rehab 36 male Lis (Head injury) Y Y Y n Y Baclofene 67.0% (10) 86.70% (13) 53.80% (13) 8.5 9 rehab 46 female Lis (s troke) Y Y Y n Y Baclofene 69.0% (10) 40.00% (6)   5.5 problem of concen- tration related to swallowing issues 10 iCu 34 female Guillain-Barré s yndrome n Y n Y n t ramadol, n.a. n.a. painful Guillain Barré (sGB) Hydroxyzine, s yndrome with Gabapentine, sleep deprivation Zopiclone, Clonazepam 11 rehab 59 male Lis (s troke); Y Y Y n Y Zolpidem, 65.0% (10)     3.8 possibly bad eyesight Gabapentine 12 iCu 22 male Duchene’s muscular n n n Y Y none 75.0% (10) 80.00% (4)   4.8 patient was bored dystrophy. and interrupted the protocol   58% (ICU) 37.7±15.3 66% Neuromuscular (33%) 58% 50% 67% 58% 92% Sedated (8%) 74.8±8.0 (n=10) 79.2±20.9 (n=8) 85.3±21.2 (n=5) 7.3 Mean ± std (male) (yes) (yes) (yes) (yes) (yes) [11] [12] [10] Stroke (15%) Use of CNS 76.2±6.5 (n=5) 78.3±16.6 (n=4) 86.4±19.3 (n=2)   iCu depressant (75%) Trauma (42%)   73.4±9.8 (n=5) 80.0±27.2 (n=4) 84.6±26.7 (n=3)   rehabilitation 0.46 0.62 -   p-value* 204 L. MAYAUD ET AL. session with accuracy significantly above chance (2.6%) Study 2: clinical evaluation level (rank sum test, W = 100, p < .001); seven could use Participants the system with more than half the symbols correctly Table 3 summarizes demographic and other variables of identified and an average accuracy of 84.7%. One patient the participants. had an accuracy of 40% and the BCI technique did not As seen in Table 3, the patients recruited in this study work for two patients. Five out of eight patients (62.5%) were on average 46.5 years old (37.0–56.0) and the healthy who co mpleted the test session asked for the optional free volunteers were on average 28.0 years old (21.8–32.2). The spelling session and spelled on average 10 characters with age difference was statically significant (rank sum test, W a median accuracy of 100.0% (72.2,100). Interestingly, = 136, p = .002). One patient was discharged prematurely the spelled words were all the first names of relatives. from the study on his request. Two additional patients did The overall satisfaction for the technique amongst not meet the threshold of performance (cross-validated participants who completed the training sessions was accuracy on training session above 70%) and were dis- high (7.3/10). charged from the study. Table 3. participants enrolled in s tudy 2 (patients and healthy volunteers). abbreviations: aBp , arterial blood pressure – systolic (s) and diastolic (d); bpm, beats per minute; f , female; GBs, Guillain-barré syndrome; Hr, heart rate; Lis, locked-in syndrome, m, male; mmHg, millimetres of mercury; ms, mulitple sclerosis; Q, quadriplegic; s, scanning device. Patients age 46 47 56 60 39 35 64 30 52 37 Gender m m f m f m m f f m pathology Lis GBs ms Lis Lis Q Q Q Q GBs Hr (bpm) 64 76 – 65 101 47 60 58 – 96 aBp s / aBp d (mmHg) 130/80 111/72 – 150/110 111/73 58/78 130/60 86/46 – 85/67 r andomization roBiK s roBiK s roBiK s roBiK roBiK s s Healthy volunteers age 33 50 26 31 32 28 21 21 22 Gender m f m f f m m m f Hr(bpm) 51 75 26 31 – 77 57 61 69 aBp s / aBp d(mmHg) 123/75 111/84 123/69 112/82 – 108/82 122/65 111/66 130/87 Figure 3.  Comparison of setup time (min), self-evaluated comfort, spelling accuracy (%), and bit-rates (bit/min) between the roBiK system (online results) and the three sessions of the scanning device. BRAIN-COMPUTER INTERFACES 205 Figure 4. Comparison of the evolution of fatigue (left) and information rate (right) for patients (dark line) and healthy volunteers (bright line) over three times of the BCi protocol (train, online, free). Comparison of patients and healthy volunteers the scanning device, respectively, which was found stat- populations ically significant (Kruskal-Wallis, H(1) = 4.90, p = .027). For the RoBIK system, the setup time was 11.0 min (8.5– Likewise, the device speed for the RoBIK system was 0.5 15.5) for patients and 13.0 min (10.5–14.2) for healthy characters per minutes (0.5–0.6) against 4.5 characters per volunteers (Figure 3). The difference was not found statisti - minute (3.5–5.0) for the scanning device (Student t-test, cally significant (rank sum test, W = 66, p = .58). Likewise, t(4) = −14.78, p < .001). Naturally these translated into the experience was rated equally pleasant (rank sum test, the significant superiority of the scanning system over W = 85, p = .23), with patients reporting an average rating the RoBIK system, with bit-rates of 28.3 bpm (19.0–31.1) of 2.5 (1.4–3.0) and healthy participants an average rating and 1.7 bpm (1.1–2.1), respectively (Student t-test, t(4) = of 1.5 (1.2–1.9). The comparison of information trans- −12.10, p < .001). fer rate (bit-rate) between patients and healthy controls In terms of self-reported fatigue in the patient group, (Figure 4) did not reveal significant differences (F(1,18) = it was found significantly greater while using the RoBIK 3.90, p = .07) even though a significant time factor (F (2,18) system, 2.7 (2.6–3.0), as compared to using the scan- = 24.46, p < .001) and time-group interaction effect was ning device, 2.0 (1.1–2.9) (F(1,29) = 49.83, p  <  .001). found (F(2,18) = 0.55, p < .001), indicating that the drop Interestingly, the repeated-measures ANOVA did not in EEG discriminatory power between the calibration and identify the time factor (measured before and aer in ft ter - the test session was greater in patients as compared to vention) as significant (F(1,29) = 1.96, p = .172), instead healthy volunteers. This might indicate that the evoked suggesting a strongly significant time-technique interac- potential amplitude (P300) of patients decreases faster tion effect (F(1,29) = 16.33, p < .001), indicating that the over time, which in turn might relate to changes in atten- BCI system was associated with greater fatigability. These tion. This interpretation is corroborated by the results on results, however, did not translate into a decreased self- self-reported fatigue, showing that although no difference reported comfort, which was found equally good for the two between the populations can be found (F(3,15) = 1.00, techniques: 4.3 (2.2–6.8) for BCI and 5.0 (4.3–6.2) for the p = .33) and both samples get tired during sessions (time scanning device (Student t-test; t(7) = −0.60, p = .565). factor, F(3,15) = 9.85, p = .001), the effect is significantly stronger in patients (interaction factor time-population, Discussion F(3,15) = 3.44, p = .049). Study 1: clinical feasibility Feasibility studies and evaluation of performance of Comparison of RoBIK prototype with scanning device P300 speller systems so far have concentrated on rela- e Th RoBIK system was associated in patients with a setup tively small samples (n < 10) of late-stage ALS or acquired time of 11.0 min (8.5–15.5) against only 3.0 min (2.0–4.2) brain injury patients using the system at home or in a for the scanning device, a difference that was found statis - controlled environment [16,27,36,55,56]. A comparison tically significant (Kruskal-Wallis, H (1) = 11.48, p = .001). of performance between normal and severely disabled e p Th erformance of the RoBIK system was signif- participants showed significantly better performance in icantly lower as compared to the performance of the the healthy population [29], which is a strong rationale spelling device in the group of patients (the only group for the evaluation of this technology with a larger variety having evaluated both techniques). The accuracy was 0.8 of patients and preferably in a natural environment. A (0.5–0.8) and 1.0 (1.0–1.0) for the RoBIK system and 206 L. MAYAUD ET AL. recent study on a bigger cohort (n = 27) of ALS patients GBS accompanied by sleep deprivation. There was no at home mentions mechanical ventilation, but it is unclear obvious reason for the low performance found in patient if patients where using the ventilators during the sessions 11 apart from a light visual impairment. This patient may [30]. The Study 1 aimed at the investigation of the use of be a ‘BCI illiterate’ [58]. BCI by acute patients in their real environment of use To the best of our knowledge, this is the first compre- (here, the ICU). hensive attempt to explore the use of a brain-computer As reported in Table 2, we found no statistically interface in an adverse clinical context. Four key factors significant mean difference of performance between were explored: first, a non-controlled clinical environ- mechanically ventilated patients (in the ICU, n = 7) and ment and the particularly adverse setup of the ICU (58%); non-ventilated patients (in the rehabilitation unit, n = 5) second, all ICU patients were evaluated during invasive over the different sessions: training (p = .46) and test mechanical ventilation; third, patients with myopathy or (p = .62). These results have to be handled with care, as the Guillain-Barré syndrome (42%) are populations that have study was certainly not designed to study such an effect. not yet been reported to use BCI; finally, this study pro- However, the individual performance reported in this vides an initial insight into the impact of central nervous study show that neither mechanical ventilation, nor the system (CNS) depressants on concomitant use of a P300 origin of tetraplegia, nor the use of CNS depressants may ERP-based BCI for communication. While this study was be reliably related to failure at controlling the BCI, which clearly not dimensioned to fully explore this phenomenon, certainly is encouraging. we welcome the successful use of the BCI by some patients For three patients who could not achieve adequate under high levels of CNS depressant, which could have performance during the test session, several possible primarily been thought to be prohibitive. Hence, further explanations were identified. A technical problem on the research is required to exactly understand the extent to acquisition module of the software degraded the system which this kind of medication influences the performance performance for patients 2 and 3 (which was subsequently of a BCI. fixed on the open-source platform). This illustrates a well- known limitation of existing BCI systems (usability of Study 2: clinical evaluation hardware and software components) when it comes to their transfer to the patients’ bedside [57]; we believe that The comparison of the BCI performance of healthy the joint efforts of the scientific and industrial commu- volunteers and patients showed no statistical difference nities is gaining momentum to address these limitations. in the primary performance outcome (bit-rate). Our find - Patient 9 reported difficulties in simultaneously swallow- ing adds to the small and contrasted available literature ing and concentrating on the task, finally achieving low [57,59]. However, the progression of bit-rate over time performance. Patient 10 fell asleep during the training within the same day, in particular for patients, was session despite initially showing interest and motivation found significant and corroborated the evolution of self- for the protocol; this was possibly explained by painful reported fatigue, which increased in both groups, but was Figure 5. example of two event-related potentials (erp s) elicited from the interface used in the pilot study (left) and the roBiK prototype (right); the roBiK prototype does not show the typical steady-state visual evoked potential (ssVep) in response to non-target stimulations that can be seen on the non-target response (left). BRAIN-COMPUTER INTERFACES 207 stronger in patients. The inclusion of fatigue and time Put together these innovative steps in EEG data col- in the analysis might in part explain the discrepancies lection and analysis have the potential to dramatically within the literature. We have included self-reported reduce the setup and calibration time. This would limit fatigue in this study because we knew that the typical the accumulation of fatigue prior to the actual use of the population of quadriplegic patients admitted to the ICU communication interface and thereby improve overall sys- is prone to fatigue and because the outcome of Study 1 tem performance, which was shown to be greatly ae ff cted indicated that fatigue was a recurrent complainr about by fatigue. These results are encouraging for the future use the system. As a consequence, we hypothesized it could of P300 spellers for the communication of patients. Apart constitute an important limitation for the use of BCIs in from patients for whom a reliable neural interface could this population. This was despite the fact that some ele- not be found (so called ‘BCI illiterates’), patients’ perfor- ments of the RoBIK prototype were specifically designed mance appeared not to differ significantly from that of to reduce fatigue: for instance, visual fatigue was limited healthy participants. Most importantly, adequately chosen by the use variable ISI, which is illustrated by Figure 5. parameters for the BCI interface lead to an estimated per- Since the patient population was significantly older it is formance that was not found to differ significantly from not possible to disentangle the effect of age from that of that of a traditional assistive technology device (t = 0.56, fatigue. This certainly constitutes a significant limitation permuted p-value = .33). Again, these results should be of our study, even though one can argue that the direction interpreted with extreme care because this study was not of this effect is nonetheless unfavorable to the patient statistically powered to answer these specific questions population since fatigability is expected to increase with and because we excluded three of ten patients (from Study age. Such a hypothesis is, however, in contrast with evi- 2) who could not use the BCI system. dence of a positive correlation between age and perfor- mance in a P300 speller, which was reported in healthy Conclusions volunteers as well as in patients [59,60]. The retrospective Some neurological disorders leave patients with baseline analysis of existing cohorts could help investigate this cognitive status and no or little communication capacity. correlation further. LIS patients – whatever their etiology – are a well-known e co Th mparison of online performance for the use of illustration of such a condition. Quadriplegic patients BCI and the scanning system in the patient population with invasive mechanical ventilation are also temporar- showed that the BCI did not compare advantageously to ily deprived of communication, resulting in additional the traditional assistive technology. All measured indi- stress for all people involved with the management of cators (with the exception of self-reported comfort and their disease. Traditional assistive technologies for chronic satisfaction – non-significant) were found to be in favor patients, such as those suffering from neurodegenerative of the scanning system: setup time was longer for the BCI; disorders, have long been used and most severe cases are accuracy, speed, and bit-rate were all found better for the equipped, whenever possible, with a ‘scanning’ spelling scanning device. Again, fatigue was strongly associated with device interfaced with simple ‘click’ contactors. However, the use of the BCI system, since both the technique and the use of these systems requires the intervention of an the time × technique interaction factors were significant. experienced occupational therapist who must find the optimal set of parameters for each patient. BCIs, on the Offline analysis of clinical data other hand, have long been promised as a potential ‘uni- versal’ assistive device for chronic patients such as those e a Th nalysis of the EEG data collected during the two clin - with ALS. So far BCI technology has consistently shown ical trials described in this paper is reported in Appendix major limitations such as  low information transfer rate 3. e Th results we obtained stress the added value of the (low spelling rate and accuracy) and the need for cum- Riemannian approach over state-of-the art classification bersome and expensive setups for the recording of the techniques, which was found particularly relevant for the EEG and its processing. However, recent advances in EEG development of calibration-free BCIs (also referred to as hardware and software appear to open a new era where ‘cross-learning’). Conversely, the analysis did not identify these limitations will be overcome. the presence of a Riemannian SQI (artifact rejection) as A feasibility study was carried out in the ICU and in a a factor contributing positively to the BCI performance, rehabilitation unit in a population of quadriplegic patients which was surprising. On the hardware side, the offline using a traditional BCI system for communication (the analysis revealed that the gain in setup time oer ff ed by the ‘P300 speller’). Twelve patients were included in the study. hardware prototype developed was not associated with a Among them, eight could successfully control the sys- noticeable drop in system performance. tem with above-chance accuracy. Performance could not 208 L. MAYAUD ET AL. directly be related to the presence of either mechanical To conclude, this study demonstrates that caregivers ventilation or sedation. In contrast, eyesight and fatigue who are unfamiliar with EEG and BCIs in general can were identified as possible limitation factors. restore some form of communication in severely disabled  Based on this pilot phase, technical specifications and patients by means of an adequately designed BCI system. user requirements were drae ft d for the development of However, the population of patients who could immedi- adequate bedside BCI prototypes for communication. e Th ately benefit from it was found to be much smaller than hardware was a 14-channel EEG system that could be set initially expected, since traditional assistive techniques are up in approximately 10 min. The software was developed remarkably effective and compare very favorably to the use with a user-friendly interface and state-of-the art pro- of P300-based spellers. Today, this is probably – more than cessing/classification techniques (based on Riemannian ever – due to the lack of ao ff rdable, convenient, and reli- geometry) in the back-end. able EEG systems, while other technological limitations e p Th rototype’s performance was evaluated in a clinical are finally being overcome. Algorithms, for instance, are trial. Its performance was compared to the performance probably reaching a form of maturity and little further of traditional assistive technology (a scanning spelling improvement can be foreseen for the coming few years; in device) in patients. The comparison was carried out using our opinion, adaptive techniques based on cross-subject bit-rate, accuracy, setup time, comfort, and fatigue, con- and cross-session initialization will play a dominant role. firming the superiority of the traditional AT technique in Fortunately, the ongoing transition of the EEG field from terms of simplicity of use (setup time), cost, performance, cottage industry to a more financially structured sector and – most importantly – patient fatigue. The BCI was also will soon translate into better and cheaper EEG systems. evaluated in a group of healthy volunteers to benchmark In turn, BCIs will cover the needs of an increasing pro- performance, showing that, if both groups performed portion of severely disabled patients and ultimately find equally, a stronger decrease in performance over time was legitimate room next to other assistive technologies on observed in patients. This further strengthens the impor - the occupational therapist’s shelf. tance of fatigue in this population. Finally, three offline analyses were carried out using Notes the data collected during phases 1 and 3: 1. https://physionet.org/works/P300SpellerBCIICU/. • First, the ‘personalized’ estimated performance of 2. https://physionet.org/works/P300SpellerBCIICU/. the BCI rate was derived from the training data, 3. h t t ps://g i t h u b .co m/a lexa n dr e b a rac h a n t/ covariancetoolbox/commit/bcccb4d750b2ad9ae6dc3 pseudo-prospectively applied to the test set (online dd76e7e6633f98bc1f3. session) and the results were compared to those of the scanning device. The analysis confirmed that the bit-rate is greatly improved when an optimal Acknowledgements and patient-specific number of repetitions is cho- We would like to thank the medical staff who have been in- sen. The resulting performance was found compa- volved on this project, and in particular Marjorie Figère, rable to that of the scanning spelling device. Marjorie Dezeaux, and Sandra Potier from the CICIT, • Second, the individual added value of each element Jean-Marie, Gilles, Prof. Lofaso, and Prof. Herault from the Functional Exploration Unit, Justine Bouteille from the New of the design was quantified in an offline analysis. It Technologies Platform (PFNT), as well as many others involved showed that the gain in ease of use generated by the in the project. new headset came at no cost in performance. It also confirmed the superiority of Riemannian methods over state-of-the-art techniques for ERP detection. Funding However, the rejection of artifactual segments of We would like to thank the National Research Agency (ANR) the data did not increase the performance above the and the General Directorate for Armament (DGA) for fund- threshold of statistical significance.  ing this project (Project ANR- 09-TECS-013–01-RoBIK). We • Last but not least, we investigated possible model would like to thank the French Association for Myopathies (AFM) for partly funding this project. initialization using an existing database. The ration- ale was to reduce fatigue by removing the tiring calibration session. The analysis revealed that the ORCID Riemannian methods enjoy superior cross-subject Louis Mayaud   http://orcid.org/0000-0002-3187-8030 generalization, providing a good initialization that Marco Congedo   http://orcid.org/0000-0003-2196-0409 needs to be adapted online [44].  BRAIN-COMPUTER INTERFACES 209 [19] Popescu F, Fazli S, Badower Y, et al. Single trial References classification of motor imagination using 6 dry EEG [1] Vincent JL. Communication in the ICU. Intensive Care electrodes. PLoS ONE. 2007;2(7):e637. Med. 1997;23(10):1093–1098. [20] Sellers E, Turner P, Sarnacki W, et al. A novel dry electrode [2] Haig AJ, Katz R, Sahgal V. Mortality and complications for brain-computer interface. Human-Computer of the locked-in syndrome. Arch Phys Med Rehabil. Interaction Novel Interaction Methods Tech. 2009: 1987;68(1):24–27. 623-631. [3] Frank Lopresti EF, Mihailidis A, Kirsch N. Assistive [21] Hornecker A. Dear customers, dear friends of brain technology for cognitive rehabilitation: state of the art. products. Brain. 2009;33. Neuropsychol Rehabil. 2004;14(1-2):5–39. [22] Renard Y. In the focus. Brain. 2010;37. [4] Simpson R, Koester H, LoPresti E. Evaluation of an [23] Emotiv E. Software Development Kit. 2010. adaptive row/column scanning system. Technol Disabil. [24] Liu Y, Jiang X, Cao T, et al., editors. Implementation 2006;18(3):127–138. of SSVEP based BCI with emotiv EPOC. Virtual [5] Simpson RC, Koester HH. Adaptive one-switch Environments Human-Computer Interfaces and row-column scanning. IEEE Trans Rehabil Eng. Measurement Systems (VECIMS), 2012 IEEE 1999;7(4):464–473. International Conference on; 2012: IEEE. [6] Biswas P, Robinson P. A new screen scanning system [25] Duvinage M, Castermans T, Dutoit T, et al., editors. A based on clustering screen objects. J Assist Technol. P300-based quantitative comparison between the emotiv 2008;2(3):24–31. epoc headset and a medical eeg device. biomedical [7] Cook AM, Polgar JM. Assistive technologies: principles engineering/765: telehealth/766: assistive technologies. and practice. Elsevier Health Sciences; 2014. ACTA Press; 2012. [8] Vidal JJ. Toward direct brain-computer communication. [26] Stytsenko K, Jablonskis E, Prahm C, editors. Evaluation Annu Rev Biophys Bioeng. 1973;2:157-180. Epub of consumer EEG device emotiv epoc. MEi: CogSci 1973/01/01. doi: http://dx.doi.org/10.1146/annurev.bb.02. Conference 2011, Ljubljana; 2011. 060173.001105. PubMed PMID: 4583653. [27] Sellers EW, Donchin E. A P300-based brain–computer [9] Birbaumer N. Breaking the silence: brain?computer interface: initial tests by ALS patients. Clin Neurophysiol. interfaces (bci) for communication and motor control. 2006;117(3):538–548. Psychophysiology. 2006;43(6):517-532. Epub 2006/11/02. [28] Kubler A, Nijboer F, Mellinger J, et al. Patients with doi: http://dx.doi.org/10.1111/j.1469-8986.2006.00456.x. ALS can use sensorimotor rhythms to operate a brain- PubMed PMID: 17076808. computer interface. Neurology. 2005;64(10):1775-1777, [10] Wolpaw JR, Birbaumer N, Heetderks WJ, et al. Brain- 0028-3878. computer interface technology: a review of the first [29] Piccione F, Giorgi F, Tonin P, et al. P300-based brain international meeting. IEEE Trans Rehabil Eng. computer interface: reliability and performance in 2000;8(2):164–173. Epub 2000/07/15 PubMed PMID: healthy and paralysed participants. Clin Neurophysiol. 2006;117(3):531–537. [11] Teplan M. Fundamentals of EEG measurement. Meas Sci [30] McCane L, Vaughan TM, McFarland DJ, et al. Evaluation Rev. 2002;2(2):1–11. of individuals with ALS for in-home use of a P300 brain- [12] Wolpaw JR, McFarland DJ, Neat GW, et al. An computer interface. Soc Neurosci, Submitted. 2009. EEG-based brain-computer interface for cursor [31] Li K, Narayan Raju VN, Sankar R, Arbel Y, Donchin E. control. Electroencephalogr Clin Neurophysiol. Advances and challenges in signal analysis for single 1991;78(3):252–259. Epub 1991/03/01 PubMed PMID: trial P300-BCI. foundations of augmented cognition directing the future of adaptive systems. Springer; 2011. [13] Picton TW. The P300 wave of the human event-related p. 87–94. potential. J Clin Neurophysiol. 1992;9(4):456–479. Epub [32] Khan OI, Kim S-H, Rasheed T, Khan A, Kim T-S, editors. 1992/10/01 PubMed PMID: 1464675. Extraction of P300 using constrained independent [14] Kaper M, Meinicke P, Grossekathoefer U, et al. BCI component analysis. Engineering in Medicine and competition 2003—data set iib: support vector machines Biology Society, 2009 EMBC 2009 Annual International for the p300 Speller Paradigm. IEEE Trans Biomed Eng. Conference of the IEEE. IEEE; 2009. 2004;51(6):1073-1076, 0018-9294. [33] Rivet B, Souloumiac A, Attina V, et al. xDAWN [15] Krusienski DJ, Sellers EW, McFarland DJ, et al. Toward algorithm to enhance evoked potentials: application enhanced P300 speller performance. J Neurosci Methods. to brain–computer interface. IEEE Trans Biomed Eng. 2008;167(1):15. 2009;56(8):2035–2043. [16] Sellers EW, Vaughan TM, Wolpaw JR. A brain-computer [34] Rakotomamonjy A, Guigue V, Mallet G, et al. Ensemble interface for long-term independent home use. of SVMs for improving brain computer interface P300 Amyotroph Lateral Scler. 2010;11(5):449–455. speller performances. Artificial Neural Netw: Biol [17] Müller K-R, Blankertz B. Toward non-invasive brain- Inspirations–ICANN 2005. 2005:45-50. computer interfaces. 2006. [35] Patrone M, Lecumberry F, Martín Á, Ramirez I, Seroussi [18] Grozea C, Voinescu CD, Fazli S. Bristle-sensors–low-cost G. EEG Signal Pre-processing for the p300 speller. flexible passive dry EEG electrodes for neurofeedback progress in pattern recognition, image analysis, computer and BCI applications. J Neural Eng. 2011;8(2):025008. vision, and applications. Springer; 2015. p. 559-66%@ Epub 2011/03/26. doi: http://dx.doi.org/10.1088/1741- 2560/8/2/025008. PubMed PMID: 21436526. 210 L. MAYAUD ET AL. [36] Kübler A, Furdea A, Halder S, et al. A Brain-computer [52] Stewart H, Wilcock A. Improving the communication interface controlled auditory event-related potential rate for symbol based, scanning voice output device (P300) spelling system for locked-in patients. Ann N Y users. Technol Disabil. 2000;13(3):141–150. Acad Sci. 2009;1157(1):90–100. [53] Rencher X. Methods of multivariate data. Analysis. 1995. [37] Guo M, Xu G, Wang L, Wang J, editors. Research on [54] Pernet CR, Chauveau N, Gaspar C, et al. Limo auditory BCI based on wavelet transform. Virtual EEG: a toolbox for hierarchical linear modeling of Environments Human-Computer Interfaces and electroencephalographic data. Comput Intell Neurosci. Measurement Systems (VECIMS), 2012 IEEE 2011;2011:3. International Conference on. IEEE; 2012. [55] Nijboer F, Sellers E, Mellinger J, et al. A P300-based brain– [38] Vienne J, Lecciso G, Constantinescu IO, et al. Differential computer interface for people with amyotrophic lateral effects of sodium oxybate and baclofen on EEG, sleep, sclerosis. Clin Neurophysiol. 2008;119(8):1909–1916. neurobehavioral performance, and memory. Sleep. [56] Riccio A, Leotta F, Tiripicchio S, et al. Can severe Acquired 2012;35(8):1071–1083. 1071-83%@ 0161-8105. Brain Injury users control a communication application [39] Mugler EM, Ruf CA, Halder S, et al. Design and operated through a P300-based brain computer interface? implementation of a P300-based brain-computer interface Age. 2011;41(46):24. for controlling an internet browser. IEEE Trans Neural [57] Cipresso P, Carelli L, Solca F, et al. The use of P300‐based Syst Rehabil Eng 2010;18(6):599-609. Epub 2010/09/02. BCIs in amyotrophic lateral sclerosis: from augmentative doi: http://dx.doi.org/10.1109/TNSRE.2010.2068059. and alternative communication to cognitive assessment. PubMed PMID: 20805058. Brain Behav. 2012; 2:479–498. [40] Mayaud L, Filipe S, Pétégnief L, et al. Robust brain- [58] Guger C, Daban S, Sellers E, et al. How many people are computer interface for virtual keyboard (RoBIK): project able to control a P300-based brain? computer interface results. IRBM; 2013. p. 131–138. (BCI)? Neurosci Lett. 2009;462(1):94–98. [41] Mayaud L, Cabanilles S, Azabou E. Patient needs analysis [59] Silvoni S, Volpato C, Cavinato M, et al. P300-based for brain-computer interfaces. In: Clerc M, Bougrain L, brain–computer interface communication: evaluation Lotte F, editors. Brain computer interfaces. Great Britain: and follow-up in amyotrophic lateral sclerosis. Front ISTE Editions. Croydon, CR0 4YY; 2016. Neurosci 2009;3(60). [42] Renard Y, Lotte F, Gibert G, et al. OpenViBE: an open- [60] Kübler A, Birbaumer N. Brain–computer interfaces source software platform to design, test, and use brain– and com- munication in paralysis: Extinction of goal computer interfaces in real and virtual environments. directed thinking in completely paralysed patients? Clin Presence-Teleop Virt Environ. 2010;19(1):35–53. Neurophysiol. 2008;119:2658–2666. [43] Schlögl A. GDF-A general dataformat for biosignals. [61] Cecotti H, Rivet B, Congedo M, et al. A robust sensor- arXiv preprint cs/0608052. 2006. selection method for P300 brain–computer interfaces. J [44] Congedo M, Korczowski L, Delorme A. Spatio-temporal Neural Eng. 2011;8(1):016001. common pattern: a companion method for ERP analysis [62] Oostenveld R, Fries P, Maris E, et al. FieldTrip: open in the time domain. J Neurosci Methods. 2016;267:74– source software for advanced analysis of MEG, EEG, 88. 74-88%@ 0165-270. and invasive electrophysiological data. Comput Intell [45] Congedo M, Gouy-Pailler C, Jutten C. On the blind Neurosci 2011;2011:156869. Epub 2011/01/22. doi: source separation of human electroencephalogram http://dx.doi.org/10.1155/2011/156869. PubMed PMID: by approximate joint diagonalization of second order 21253357; PubMed Central PMCID: PMC3021840. statistics. Clin Neurophysiol. 2008;119(12):2677–2686. [63] Congedo M, Goyat M, Tarrin N, et al., editors. Brain [46] R akotomamonjy A, Guigue V, Mallet G, et al. Ensemble Invaders’: a prototype of an open-source P300-based of SVMs for improving brain computer interface video game working with the OpenViBE platform. P300 speller performances. Lect Notes Comput Sc. Proceedings of the 5th International Brain-Computer 2005;3696:45–50. PubMed PMID: ISI:000232193800008. Interface Conference 2011; 2011. [47] Lotte F, Congedo M, Lécuyer A, et al. A review of [64] Jin SH, Lin P, Hallett M. Abnormal reorganization classification algorithms for EEG-based brain–computer of functional cortical small-world networks in focal interfaces. J. Neural Eng. 2007;4. hand dystonia. PLoS ONE. 2011;6(12):e28682. Epub [48] Barachant A, Andreev A, Congedo M, editors. The 2011/12/17. doi: http://dx.doi.org/10.1371/journal. Riemannian Potato: an automatic and adaptive artifact pone.0028682. PubMed PMID: 22174867; PubMed detection method for online experiments using Central PMCID: PMC3236757. Riemannian geometry. TOBI Workshop lV. 2013. [65] Townsend G, LaPallo BK, Boulay CB, et al. A novel P300- [49] Congedo M. EEG Source Analysis Université de based brain–computer interface stimulus presentation Grenoble; 2013. paradigm: moving beyond rows and columns Clin [50] Barachant A, Bonnet S, Congedo M, et al. Multiclass Neurophysiol 2010;121(7):1109-1120. Epub 2010/03/30. brain-computer interface classification by Riemannian doi: http://dx.doi.org/10.1016/j.clinph.2010.01.030. PubMed geometry. IEEE Trans Biomed Eng 2012;59(4):920- PMID: 20347387; PubMed Central PMCID: PMC2879474. 928. Epub 2011/10/20. doi: http://dx.doi.org/10.1109/ [66] Congedo M. Introducing the logistic discriminant TBME.2011.2172210. PubMed PMID: 22010143. function in electroencephalography. J Neurother. [51] Nijboer F, Birbaumer N, Kübler A. The influence of 2003;7(2):5–23. psychological state and motivation on brain–computer [67] Serby H, Yom-Tov E, Inbar GF. An improved P300- interface performance in patients with amyotrophic lateral based brain-computer interface. IEEE Trans Neural Syst sclerosis–a longitudinal study. Front Neurosci. 2010;4. Rehabil Eng. 2005;13(1):89–98. BRAIN-COMPUTER INTERFACES 211 [68] Wolpaw JR, Ramoser H, McFarland DJ, et al. EEG- Appendix 2. Details of software development based communication: improved accuracy by response Graphical user interface verification. IEEE Trans Rehabil Eng. 1998;6(3):326–333. [69] Hosmer DW Jr, Wang CY, Lin IC, et al. A computer e Th Brainmium speller interface was designed to automate the program for stepwise logistic regression using calibration of machine learning algorithms implemented in maximum likelihood estimation. Comput Prog Biomed. OpenViBE and to provide a user-friendly interface for patients 1978;8(2):121–134. Epub 1978/06/01 PubMed PMID: and operators. Rather than leaving the application synchroniz- ing the received EEG data with the precise temporal occurrence [70] Congedo M, Barachant A, Andreev A A New generation of visual stimulations, which would be prone to delays and jitter of brain-computer interface based on riemannian induced by the operating system (Windows 7 professional edi- geometry. arXiv preprint arXiv:13108115. 2013. tion, Microsoft Corporation, Redmond, WA, USA), we decided [71] Guger C, Krausz G, Allison BZ, et al. Comparison of to send these stimulations through the USB directly to the EEG dry and gel based electrodes for P300 brain–computer electronic acquisition unit to obtain accurate timestamps for all interfaces. Front Neurosci. 2012;6. stimulation events. Flashes had a duration of 75 ms [55] and ran- dom inter-stimulus intervals (ISI) were drawn from an exponen- tial distribution with mean  = 0.15s. In previous research by Appendix 1. Details of hardware developments our consortium, an exponential ISI was found to reduce the cor- tical fatigue associated with stimulation at fixed frequency while e EEG h Th eadset was designed to minimize the need for EEG enhancing the ERP single-trial estimation [63]. In the original recording preparation: as a primary goal in the design phase, P300 speller paradigm symbols flash by rows and columns. Of- skin preparation with abrasive paste, electrode positioning with ten detection errors arise because of the ‘adjacency-distraction’ gel, and retention with tape had to be avoided. The resulting phenomenon [64,65], according to which non-target symbols in headset is a 3D-printed polyamide structure holding 14 Ag- rows or columns adjacent to the target attract the user’s attention Cl electrodes (Neuroservices, Evry, France) located at standard when they flash, producing evoked activity similar to the P300, 10/20 locations: Po7, Fp1, Oz, Fz, C3, F4, Pz, C4, P7, Fp2, F3, making the detection of the target P300 more difficult. To miti- P8, Po8, and Cz. These locations were selected in order to max- gate this effect we flash the symbols by random groups [63]. Not imize the chance of capturing P300 ERPs using as reference only is the ‘adjacency-distraction’ effect mitigated, we also find and ground the right and left mastoids, respectively, according that the pattern of flashing becomes totally unpredictable, which to previous studies by the RoBIK consortium [61]. Each elec- is expected to sustain the attention of the user and to enhance trode was mounted on a polyethurane wheel connected to a the P300. More importantly, random-group flashing allows arbi- spring in order to control the pressure applied to the scalp with trary positioning of the symbols on the screen (no more need to the electrode. The headset was designed to meet the need of pa- arrange symbols on a grid), which greatly expands the usability tients and safety requirements, as well as to be compliant with of the P300 paradigm. the ICU environment. In particular, the headset could be used in a lying position or in the presence of a headrest. e h Th eadset contains an electronic component that was de- Online artifact rejection signed to record EEG activity with high fidelity while maintain- ing minimal volume and weight. An MSP430 ultra-low-power e o Th nline artifact detection was initialized at the beginning of microcontroller (Texas Instrument, Dallas, USA) was used to each session in order to interrupt the sequence of stimulations control a dedicated application-specific integrated component in the presence of excessive noise. This technique allows a sim- (ASIC) developed for this purpose: the CIrcuit for NEuronal ple definition of a rejection region for incoming data segments. SIgnal Conversion (CINESIC32). Each input channel is com- e p Th articipant was instructed to stay still for 10 seconds dur - bined with an external capacitor (1.5 nF) in order to suppress ing which a series of n clean overlapping EEG time-windows sqi the risk of leaking current in a first default condition, which X of dimension E  ×  S were extracted, with S the num- i=1..n sqi is essential for medical applications. The analogue channel is ber of samples in the time-window and E = 14 the number of composed of a fully differential low-noise amplifier, followed electrodes. For each time-window X , a covariance matrix Σ of i i by a voltage gain amplifier and a programmable low-pass fil- dimension E  ×  E was computed. Covariance matrices belong ter. Each channel consumes about 34 μA, summing up to an to the Riemannian manifold of symmetric positive-definite averaged consumption of 13 mA at full data streaming includ- matrices wherein a Riemannian metric can be used to define ing baseline consumption at 3.3 V. With these settings 24-hour a distance δ between any two covariance matrices such as [50]: continuous operation can be achieved with one high-ener- 1∕2 gy-density 3.6  V lithium battery. Only 14 of the 32 available −1∕2 −1∕2 Σ , Σ =∥ log Σ Σ Σ ∥= log  , (1) channels were used in the final prototype and each of them i j 1 2 1 e was configured with a (0.5–30 Hz) band-pass filter and a 60 dB −1 −1 voltage gain; analogue signals were digitized through a 12-bit where  , e = 1… E are the E eigenvalues of Σ Σ or of Σ Σ . e 1 2 2 1 analogue-to-digital converter (ADC) with nominal sampling Using this distance, a geometric center of mass M (or Fréchet frequency of 580 Hz per channel. mean) of a set of covariance matrices can be estimated with a Raw EEG were transmitted to a field-trip buffer [62] that gradient descend algorithm solving the following optimization was subsequently read by an OpenViBE acquisition server problem: [42]. Signals were then band-pass-filtered between 1 and 20 Hz with a fourth-order Butterworth with linear phase response argmin  M, Σ (2) and decimated to 145 Hz for further analysis. 212 L. MAYAUD ET AL. Table 4. summary of materials and methods for both studies showing the population, hardware, software, algorithms, and cross-val- idation procedures used in each phase. abbreviations: minimum distance to mean (mDm) is a r iemannian classifier; support vector machine (sVm) is a classifier; stepwise linear discriminant analysis (swLD a ) is a linear classifier; xD a Wn is a spatial filter for event-related potentials; p1, p2, and V2 refer to patients and volunteers included in clinical studies 1 and 2, respectively. Population Hardware Software Algorithms Cross-validation exp. 1• patients (p2)• roBiK• openViBe• riemannian potato• 5-fold (training) • Volunteers (V2)• matlab 14• mDm• Cross-session exp. 2–3• patients (p1)• tmsi• openViBe• riemannian potato• 5-fold (training) • patients (p2)• roBiK• matlab 14• mDm• Cross-session • Volunteers (V2)• xDaWn+swLDa• Cross-participant In other words, just as the mean in Euclidean space is the val- A special covariance matrix is then estimated using this su- ue minimizing the variance, the Riemannian center of mass per trial such as M of a set of points is the point minimizing their dispersion (variance) in the manifold. For every covariance matrix Σ , we ̃ � � Σ = X X (6) i i i N − 1 can compute a distance δ to the center of mass of the data set as  =  Σ , M . Finally, given all distances we can compute a i i If the data have been previously band-pass filtered in an ap- scalar geometric mean μ and and a scalar geometric standard propriate band-pass region, such a super covariance matrix deviation σ of them using the following formula [49]: contains all the spatial and temporal information needed to achieve the detection of ERPs [70], therefore it can be used � � � � directly as a single feature for the classification algorithm as a ⎛ 2⎞ � � � � 1 1 ⎜ ⎟ point on the Riemannian manifold. For each of the two class- = exp ln  ,  = exp ln , (3) L ⎜ L  ⎟ es, Target (+) and Non-Target (−), a Riemannian center of i i ⎝ ⎠ mass is estimated using the data from the calibration session. which for Riemannian distances will approximate a symmetric The center of mass can be understood simply as the expected distribution [49]. Finally, we can then derive z-scores of the covariance matrix of a trial belonging to the corresponding distances as class, wherein the use of the Riemannian metric ensures that this expectation is a much better representative as compared to the arithmetic mean. In particular, extensive testing pre- ln (4) sented in [70] has established that this expectation is more z  = . ln() robust to noise and outliers. The classification of an unseen trial is achieved by comparing the distance of the trial to the Once μ and σ are estimated on the training data, covariance ma- center of mass of the Target and Non-Target class. A classi- trices computed on new online EEG epochs are discarded when- fication score is given to unseen trials according to the score ever their distance to the center of mass exceeds a z-score of 2.5. function e s Th et of points on the manifold having a distance to a center of mass less than 2.5 standard deviations forms a closed region; in + − ̄ ̄ three dimensions, such a region would look like a potato rather s = 𝜋 𝛿 Σ ,Σ − 𝛿 Σ ,Σ , (7) i i than a sphere, because of the non-linear nature of the Riemannian manifold, which is why this technique was named the ‘Riemanni- where (x)= are probabilities found by fitting a −(+x) 1+exp an Potato’ [48]. During online experiments, visual feedback was logistic regression curve with parameter α and β [66] to the set + − + − provided to patients so that they could relate artifacts to specific ̄ ̄ ̄ ̄ of distances 𝛿 Σ , Σ and 𝛿 Σ , Σ and Σ and Σ are the i i behaviors (blinks, coughing, head movements). This ensures that center of mass of the Target and Non-Target classes, respective- they can associate any artefactual behavior with its impact on the ly. These scores are averaged across repetitions, and the symbol data quality and take corrective actions: move less and blink at is assigned to the symbol at the intersection of the row and appropriate times as much as possible. The Riemannian potato column with highest score. was implemented in OpenViBE for online experiments (Study 2) and in Matlab for offline experiments (Study 1). Appendix 3. Offline analysis Online classifier This offline analysis has the following objectives: Using the Riemannian distance in Equation (1) and the defini- • Experiment 1: comparison of the personalized BCI with tion of center of mass, in Equation (2) the detection of the P300 the traditional assistive technology; evoked potential can be achieved by a deceptively simple clas- • Experiment 2: assessment of the individual contribution sification algorithm named the minimum distance to means of each component (RoBIK headset, artifact-detection (MDM) [50]. Denoting by + the target class of flashes, each tri- algorithm, classifier algorithm) to the overall system al X is concatenated with a prototypical P300 evoked response performance; P (for instance, the ensemble average estimation obtained on • Experiment 3: assessment of the performance of a cali- the training data) to build a ‘super’ trial: bration-free BCI system. X = . (5) i BRAIN-COMPUTER INTERFACES 213 of 10 characters should correctly be identified. The optimal Materials and methods number of repetitions was chosen accordingly, corresponding Elements of the materials and methods are summarized in bit-rates were derived as described in Equation 1, and subse- Table 4. quently compared to that of the scanning device for the same patients. Offline analysis 1: optimal BCI compared to scanning device Offline analysis 2: added value of the prototype design e co Th mparison of performance for communication interfaces A second offline analysis was run to quantify the added value is usually assessed using the bit-rate [67,68]. The advantage of of the proposed BCI prototype design. More precisely, we used bit-rate over accuracy as a performance measure is that it also an offline study to ensure that the hardware design, which was takes into account the speed of the interface and the amount of meant to increase ease of use, came at no cost in performance choices it oer ff s. Bit-rates thereby gives a more representative and we similarly quantified the added value of the Riemannian metric of ‘information flow’ from the participant’s brain to the approach for artifact trial rejection and classification. To do so, of- machine. The bit-rate br is defined as fline analysis 1 was repeated with and without online artifact trial rejection using the SQI index described in Appendix 2. Likewise, 1 − p br = log (M)+ p.log p +(1 − p).log , (1) the performance of the Riemannian MDM classifier was com- 2 2 2 M − 1 pared to a state-of-the-art technique composed of the xDAWN where M is the number of symbols and p the spelling accu- spatial filter combined with a stepwise linear discriminant anal- racy (the percentage of correctly identified symbols). In this ysis (SWLDA) [69]. The comparison was carried out with and study, M equals 36 and 73 for the RoBIK BCI and the scanning without the artifact trial rejection and on both clinical datasets device, respectively. (Studies 1 and 2). Because the RoBIK interface natively inter- The evaluation of the RoBIK prototype was designed to rupts visual stimulations in the presence of noise, it is expected demonstrate the feasibility of a well-designed BCI prototype that most target and non-target epochs from Study 2 are clean. for communication in a clinical context. Therefore, the stim- For this reason, only data from Study 1 were considered to assess ulation parameters (flash duration, average time between two the benefit of artifact rejection, while both datasets were used to stimulations, and number of repetitions) were chosen to pro- compare the classifiers. mote optimal accuracy rather than speed. The system evaluat- ed at the bedside in this study was not optimized for bit-rate, Offline analysis 3: cross-participant-analysis but for optimal data collection and robustness in the context A calibration-free ERP-based BCI has been proposed by of a clinical feasibility investigation. This approach ensured Congedo [70] and Barachant et al. [48]; the center of mass the collection of a large amount of data for the offline analysis. of the available classes is initialized using a database of pre- For the BCI performance, we consider the bit-rate achieved vious users and then continuously updated using the in- with the optimal number of repetitions. In order to assess the coming data from the online sessions of the user. In order optimal number of repetitions, the MDM classifier described to assess the potential of such a approach, offline analysis 2 above was fitted to the training data after artifact identifica- was run a second time replacing the training set by all data tion using the aforementioned SQI index. The test set was a available from other participants (cross-subject learning). bootstrapped dataset generated from the ‘online’ session of While offline analyses 1 and 2 used little data from the same each participant as follows: for each number of repetitions participant, more data are available in this offline analysis; of the stimulation k  =  1…50 (that is the number of times a however, they belong to different participants. To allow for specific letter is flashed before a decision is made), B = 1000 cross-subject comparison, all covariance matrices for the groups of k target and five times k (5000) non-target respons- MDM models were normalized so as to have a unit deter- es were randomly selected (with replacement) and averaged, minant as: so that the target to non-target ratio was preserved. Then, for each group of six responses, the predictions were obtained by −1∕E Σ =Σ ∗ det(Σ) (2) norm assigning 1 to the element featuring the minimum distance to which derives directly from the following property of the de- the center of mass of the Target class and 0 to the five other terminant: det(cΣ) = c det(Σ), where det (Σ) denotes the de- elements. The seed for the random sampling was preserved in terminant of matrix Σ, c is a scalar, and E the size of the matrix. order to evaluate the performance of the different techniques on the same random bootstraps. At the end of this procedure, 6000 predictions were used to derive a performance index Statistical analysis for each participant and for each number of repetitions of The benefit of the proposed BCI design was evaluated the visual stimulation (the flash). In particular, the ‘real ac- by means of a Kruskal-Wallis test, using the technique curacy’ was defined as the square of the raw accuracy, since methods as an independent factor and the bit-rate as the each letter is found at the intersection between two groups of dependent variable. Using the best-performing of each symbols (commonly referred as the ‘lines’ and ‘columns’ in method, optimal bit-rate was identified (at 70% real accu- the traditional P300 speller experiments). If predictions were racy) and compared to that of the scanning system on the to be made at random, the ‘real accuracy’ would be 1 in 36, same patients using a paired Student t-test. For the repeated- meaning that there is a probability of 2.8% that the correct measures framework above, in this manuscript we will refer letter is selected by chance. With such a definition it was ar - to group factor, time factor, and time-group interaction bitrarily decided that an interface should provide a minimum factor and related effects. ‘real accuracy’ of 70%, meaning that on the average 7 out 214 L. MAYAUD ET AL. Figure 6. evolution of ‘real accuracy’ (% of correctly identified characters) in relation to number of flashes of each symbol under different conditions; the three rows of plots indicate the performance in (1) cross-validated training data, (2) test-set (online) data, and (3) cross- subject performance. t he first column of plots shows the effect of artifact removal using the sQi on the data from s tudy 1 only. t he second column shows the impact of the classifier comparing xD aWn + stepwise linear discriminant analysis (sWLD a ) and minimum distance to mean (mDm) r iemannian classifier. t he third column compares the performance of healthy volunteers and patients. t he fourth column compares the data collected with a traditional eeG system (s tudy 1) to that collected with the roBiK headset (s tudy 2). f or all factors considered, data are collapsed across the other factors. Table 5. Bit-rates (bits per minutes) for the clinical sample included in the clinical study of s tudy 2 for scanning device (session 1 only), the roBiK system during the online session, and the performance computed offline using the same data and algorithms but with the use of an optimal number of repetitions (personalized). patients 5, 8, and 9, for whom no BCi signal could be exploited, were removed from this analysis and subsequent statistical testing. P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 s canning 1 24.5 22.6 30.8 20.3 18.9 33.9 20.7 na 25.1 14.5 online roBiK 3.1 1.7 1.7 0.2 0.0 1.4 0.1 na 0.0 0.1 personalized roBiK 27.8 27.8 23.0 27.8 0.0 1.7 2.9 na 0.0 31.5 volunteers and patients and benchmarks the performance of Results the RoBIK EEG headset versus a traditional disc electrode Added value of signal processing and classification system (TMSi amplifier). As detailed in Figure 6 , these com- parisons were carried out at different levels: with a five-fold algorithms cross-validation on the training set, prospectively on the on- Figure 6 shows the influence of several factors with respect line dataset, and using cross-participant fitting. For the latter, to real spelling accuracy: the presence of an artifact- only data from other participants wee used to fit the model. removal technique (the SQI) and the use of a traditional The comparison of optimal (maximum) bit-rates over all algorithm (xDAWN+SWLDA) versus a Riemannian algo- rithm (MDM). It also compares the performance of healthy repetitions for these conditions reveals a significant benefit BRAIN-COMPUTER INTERFACES 215 of the MDM over the xDAWN+SWLDA approach (paired MDM requires only the computation of centers of mass and t-test, t = 32.4, permuted p < .001). This was particularly true distances between two points and has no free parameters to for the cross-participant condition where only the MDM be tuned by cross-validation or heuristics, therefore it reduces model could generalize well (paired t-test, t = 42.5, permut- the risk of over-fitting. In addition to this, Riemannian meth- ed p  <  .001). Other factors were not found significant. This ods generalize well across sessions (see Figure 6, line 2 col- seems to indicate that patients and healthy volunteers have umn 2) because the Riemannian distance function is invari- equivalent performance. Likewise, the performance of the ant by any linear transformation of the data and covariance RoBIK headset did not significantly differ from that of the shifts observed across sessions due to movement in headset traditional EEG system, which strengthened the rationale positions or changes in electrode impedance are of this type. for the improvement in setup time (from 30 minutes to 10 As expected, the performance of the cross-participant model minutes). Finally, these results also indicated that the use of was found to be significantly lower than that of the ‘online’ an artifact-rejection technique does not necessarily improve mode (trained on data from a training session from the same the accuracy, although it results in a reduction of variability, participant), which implies that under these conditions online which can be noticed only with the use of a traditional EEG adaptation may be required to keep the performance of the recording system. calibration-free BCI optimal. e u Th se of the SQI to reject trials contaminated by arti- facts did not show clear benefit in terms of bit-rate. However, Comparison of optimal BCI to scanning device Figure 5 (column 1) shows that monitoring artifacts and stop- As seen in Table 5, seven out of nine patients (77%) who ping the operation in the presence of high noise reduce the completed the Online session during Study 2 could success- variability of the system, which is an important benefit for re- fully use the P300 speller. For those (we excluded two addi- al-word operation. It should be noted that our study has been tional patients P05 and P09 who could not use the system carried out in a realistic environment. Even though the data at all), the comparison of bit-rates did not show statistical were acquired under the strict supervision of an EEG techni- significance between the first scanning session and the cian, they were collected in an environment prone to artifacts. personalized RoBIK performance (T = 0.56, permuted We have made the data de-identified, open, and accessible to p-value = .33). peers in the digital supplement of this article so that the com- munity of researchers can investigate this and other questions further. Discussion No difference in performance was found between the First, the results confirm the superiority of the Riemannian RoBIK and the TMSi headset, even though lower variability MDM approach over traditional classification based on spatial can be observed for the TMSi headset. This supports the well- filtering (for Online mode, line 2 on Figure 6 ). This finding known fact that a gel-based cap collects EEG data more con- replicates previous reports on healthy participants [49,50,70]. sistently. Nonetheless, the RoBIK headset has lower technical In some patients, the Riemannian MDM oer ff s real accuracy specifications (no true DC, 12 bits instead of 24), is there- above 80% with as little as two repetitions, which favorably fore cheaper, and allows a dramatic reduction in setup time compares with state-of-the-art techniques [71]. This supe- (approximately three-fold). While the headset prototype test- riority was even more evident for the cross-subject transfer ed had numerous drawbacks (design, weight, corrosion with learning (Cross-subject, line 3 on Figure 6), which is also in saline water, mechanical electrical contact to name a few) it line with our previous investigations [50,70]. Overall these shows that cheap and easy-to-use EEG systems for bedside results suggest that the Riemannian MDM classifier oer ff s applications are becoming a reality. The trending develop- better generalization properties as compared to spatial filters. ments in dry electrode systems [18,20,71] may be expected Interestingly, the advantages of the MDM algorithm in terms to lower the tolerance threshold further and broaden the use of accuracy and generalization are accompanied by a dramat- of BCIs. ic reduction of the algorithmic complexity; the Riemannian

Journal

Brain-Computer Interfaces – Taylor & Francis

Published: Oct 1, 2016

Keywords: Assistive technology; communication aids; event-related potentials, P300 speller; electroencephalography; Riemannian geometry

There are no references for this article.