Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Depth electrode neurofeedback with a virtual reality interface

Depth electrode neurofeedback with a virtual reality interface Brain–Computer interfa Ces, 2017 VoL. 4, no . 4, 201–213 https://doi.org/10.1080/2326263X.2017.1338008 OPEN ACCESS a a b a a c Hagar Grazya Yamin , Tomer Gazit , Natalia Tchemodanov , Gal Raz   , Gilan Jackont , Fred Charles , b,d a e Itzhak Fried , Talma Hendler and Marc Cavazza   a b Center for Brain f unctions, t el a viv s ourasky medical Center, t el a viv, israel; Department of neurosurgery, David Geffen s chool of medicine and s emel institute for neuroscience and Human Behavior, university of California Los angeles, Los angeles, C a, usa; Department of Creative technology, f aculty of s cience and technology, Bournemouth university, poole, united Kingdom; f unctional neurosurgery unit, tel-a viv medical Center and s ackler s chool of medicine, t el-a viv university, t el-a viv, israel; s chool of engineering and Digital arts, university of Kent, Canterbury, united Kingdom ABSTRACT ARTICLE HISTORY received 30 July 2016 Invasive brain–computer interfaces (BCI) provide better signal quality in terms of spatial localization, a ccepted 8 may 2017 frequencies and signal/noise ratio, in addition to giving access to deep brain regions that play important roles in cognitive or affective processes. Despite some anecdotal attempts, little work has KEYWORDS explored the possibility of integrating such BCI input into more sophisticated interactive systems Brain–computer interface like those which can be developed with game engines. In this article, we integrated an amygdala (BCi); neurofeedback (nf); depth electrode recorder with a virtual environment controlling a virtual crowd. Subjects were electroencephalogram (eeG); asked to down regulate their amygdala using the level of unrest in the virtual room as feedback on intracranial depth electrodes how successful they were. We report early results which suggest that users adapt very easily to this SUBJECT paradigm and that the timing and fluctuations of amygdala activity during self-regulation can be CLASSIFICATION matched by crowd animation in the virtual room. This suggests that depth electrodes could also CODES serve as high-performance affective interfaces, notwithstanding their strictly limited availability, application development justified on medical grounds only. and evaluation; neurosurgical approaches and methods, affective computing; signal acquisition: eeG (other) Introduction hippocampus, which play important roles in cognitive and ae ff ctive processing. In this article, we explore the With the development of brain–computer interfaces (BCI) development of a depth electrode BCI using a neurofeed- as an interdisciplinary endeavor ranging from neural engi- back (NF) paradigm, in which users are asked to down- neering to user interface technology [1], there has been regulate their amygdala (gamma band) activity to control a growing interest in invasive BCI developed in clinical the behavior of a virtual environment populated by a small settings, to gain knowledge on some fundamental BCI crowd of virtual characters. This system was originally problems, and more speculatively, develop user interfaces developed to improve engagement for patients having to around invasive BCI. Invasive systems are implemented for test the recording ability of their implanted depth elec- strict medical indications, primarily the monitoring and trodes, and we report here early results on its use in two presurgical evaluation of medically intractable epilepsy patients to illustrate the properties of depth electrodes BCI [2]. The two main invasive approaches are electrocorti- and the end-to-end integration in a virtual environment cography (ECoG, [2–4]) where intracranial electrodes are TM based on the Unreal Development Kit (UDK ) engine. placed at the surface of the cerebral cortex, and depth electrodes which are inserted inside inaccessible regions, for instance to monitor and control temporal epilepsy [5]. Previous and related work Compared to EEG electrodes, invasive recordings oer a ff Although the literature on invasive BCI primarily addresses better spatial resolution, the ability to capture high EEG assistive technology for motor impaired patients, several frequencies with fewer artifacts and, in the case of depth electrodes, an even greater spatial resolution, and the authors have reported experiments in which users were ability to access deep regions such as the amygdala and able to control other applications, in particular computer CONTACT Hagar Grazya Yamin yaminhagar@yahoo.com t his paper describes research originally presented at the 6th international BCi meeting, organized by the BCi s ociety and held may 30–June 3rd 2016 in pacific Grove, California, usa. © 2017 t he a uthor(s). published by informa uK Limited, trading as t aylor & f rancis Group. t his is an open a ccess article distributed under the terms of the Creative Commons a ttribution-nonCommercial-noDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 202 H. G. YAMIN ET AL. games, as an illustration of quality and controllability of the strategy and the controlled game, as the user aims at signal. Breshears et al. [6] have used ECoG as an interface simultaneously activating/deactivating the target region to a version of the popular Space Invader game in which and agitating/relaxing the ‘Waiting Room’. This contrasts, a young patient would control one-dimensional lateral for instance, with the use of cognitive tasks such as mental movement of the base spaceship that would fire auton- calculation to control the position of the ball in Brain Ball omously. The same group had previously demonstrated [12]. Like previous work, we report on isolated numbers the possibility of controlling a cursor in two dimensions of patients (here two patients), so the findings are to be with success rates of 53–73% [7]. Previous BCI making interpreted in the context of previous literature and the use of depth electrodes include Krusienski and Shih [8] description of fundamental phenomena associated with who have adapted the P300 spelling paradigm for use with neural activity, and claim no statistical validity. a depth electrode, describing its use by one subject with a hippocampal depth electrode [9]. Lachaux et al. [10] have System overview reported the use of depth electrodes for neurofeedback, using the direct visualization of plotted cortical activity We developed a user interface concept to support NF as a visual feedback channel. The same team was the first called the ‘Waiting Room’ (WR), which is a virtual envi- to use this paradigm with a bespoke game called ‘Brain ronment populated by animated characters (see Figure 1). Ball’ in which subjects would control the position of a ball One frequently observed issue in NF is that the usability on the screen by self-initiated gamma power (triggered by of traditional feedback signals (e.g., gages and thermom- voluntarily engaging in mental calculation). In terms of eters) is oen limi ft ted, for two reasons. The first one is signal mapping, the instantaneous position of the ball was the lack of user engagement in a context which is seen as updated in real-time according to a linear transformation too close to an assessment or an experiment. By contrast, of the instantaneous change in gamma power from the embedding feedback into a task-based application or a parietal depth electrode [11]. game is known to improve engagement [2]. e Th second Overall, this article is the first to combine depth elec- one is that full synchronous coupling between the meas- trodes in a neurofeedback context with a realistic, real- ured BCI signal and the visual signal tends to feedback all time game-like interface developed using the Unreal oscillations and inherent instability related to the effortful TM Development Kit (UDK ) engine. In addition, we depart component of neurofeedback, which can lead to loss of from direct physical control of motion direction or posi- focus, or ‘catastrophic’ reactions when the user cannot tion (spaceship, ball) to consider more abstract concepts sustain their neurofeedback effort. This is the case in par - of mental/aeff ctive state regulation for which the 3-D ticular with high frequency signals derived from EEG: res- interface is primarily a metaphor. Importantly, this met- ampling through moving averages smoothens variations, aphor still establishes continuity between the cognitive but at the cost of delays and impact on variation ranges. Figure 1.  s ystem o verview and experimental s etting. t he user, implanted with Behnke–f ried depth electrode [19], is instructed to appease the situation in the real-time 3-D visualization of the ‘Waiting room’. neurofeedback is based on the input of gamma band amygdala ieeG power. ieeG is recorded using neuroport (Blackrock microsystems). t he user is expected to lower the level of unrest in the ‘Waiting room’ by down-regulating her amygdala activity. t he real-time amygdala gamma band power is plotted during several short nf trials, where the nf epochs (in blue) must show lower values than during the Baseline epochs (in white). BRAIN–COMPUTER INTERFACES 203 Figure 2. Characters standing up from their seats and walking to the reception desk. Figure 3.  arrows represent the deepest macro contact located in the left amygdala of patient D004 (a) and in the right amygdala of patient D005 (b). t he unmarked bulbs represent the location of the rest of the macro contacts on the same electrode (light blue in (a) and purple in (b)) or at a nearby electrode (red bulbs in (a)). A more appropriate feedback interface would both have virtual characters as a metaphor for arousal (see Figure 2). a realistic (semantic) outlook and support some kind of e Th characters can evolve between two states: one resting signal integration that would sustain the user’s neurofeed- state, in which they are sat waiting to be called; and one back cognitive efforts, by canceling transient dips in acti- agitated state, in which they move to the front desk to vation towards the threshold (regardless of the position complain about waiting times. The ratio between the char - of the threshold with respect to the baseline). The WR is acters standing and sitting (N /N ) at the desk defines stand sit designed to meet these requirements, and uses a crowd of a level of ‘unrest’, which is used as the target variable for 204 H. G. YAMIN ET AL. Table 1. t otal number of electrodes and the positioning of the target amygdala macro electrode for the two subjects. Patient ID Total number of electrodes Amygdala macro electrode MNI D004 6 Left: x = –24, y = –4, z = –24 r ight: x = 24, y = –6, z = –29 (located at the border with the hippocampus) D005 10 Left: x = –24, y = –8, z = –26 (located at the border with the hippocampus) r ight: x =24, y = –8, z = –17 BCI signal mapping. The level of activation for the amyg- Table 1. In patient D004 and the first session of D005, the dala is translated into a level of unrest, which is updated feedback was bilateral, the feedback of the second session in the WR through a set of discrete transitions during of D005 was taken from the right amygdala. which additional characters stand up and move towards We localized electrodes by performing computed the front desk. It is this realistic simulation of characters tomography scans aer im ft plant; aligning the computed motion that dampens the BCI signal uc fl tuation, because tomography images with pre-implant MRIs and then labe- the time required for characters to move across the room ling the region of each electrode according to anatomical prevents drastic changes in either populations (waiting landmarks (see Figure 3). or standing). e p Th lacement of electrodes is based solely on the data obtained from scalp EEG, imaging and other tests undergone by the patient. Likewise, the length of time Methods the patient remains implanted is determined only by the requirement to ‘capture’ sufficient seizures on EEG to Apparatus determine the seizure onset zone. We used Behnke–Fried depth electrodes (Ad-Tech medical instruments, USA; see Figure 1) [1,5]. Each electrode has Sampling and signal acquisition eight platinum-coated macro contacts (1.6 mm diameter, separated by 5 mm), and eight platinum-iridium micro e o Th nline interaction between the Neuroport system contacts (seven recording and one reference electrode) and the MatLab file controlling the WR was done using located at the tip of each electrode (38  µm diameter). cbmex platform, a set of MatLab functions provided by e micr Th o contacts are capable of recording single units, Blackrock Microsystem for online communication, with whereas the macrowires record activity of a wider volume the data collected through the neural signal processors. referenced here as iEEG. Because the gamma band activity e sys Th tem amplifier receives signals directly from the collected by the microwires was more prone to noise, we electrodes through a headstage or the patient cable. The used macro contact data for neurofeedback purposes. sampled signal (up to 30 kHz) is amplie fi d, analog filtered (1st-order high-pass at 0.3 Hz and 3rd-order low-pass at 7,500  Hz) and digitized (16-bits at 250 nV resolution) Subjects and then is transmitted to the NSP via a fiber optic link. Two adult patients recruited at the authors’ institution e a Th nalog filter allows for the low-frequency field poten - (D004: male, 23 years old; and D005: male, 40 years old) tials as well as the higher frequency spike signals to be with intractable epilepsy were implanted with depth elec- recorded. A later stage digital filtering will allow these trodes to localize the epileptic focus for possible subsequent two signals to be separated (NeuroPort user’s manual, resection. Patients were stereotactically implanted with six Blackrock Microsystems). Here we used the low fre- and ten depth electrodes from a lateral orthogonal approach quency field potentials, the signals were digitally sampled aiming at targets selected using clinical criteria. Following at 2 kHz, and bandpass filtered between 0.3 and 500 Hz implantation, patients remained between one and two weeks (Neuroport, Blackrock Microsystems, USA). All macro hospitalized for monitoring (subject D004 10 days, subject contacts were referenced to a scalp electrode (Pz). D005 14 days). All patients provided informed consent. All studies conformed to the guidelines of the authors’ institu- Neurofeedback setting: the Waiting Room tion. All patients underwent intracerebral EEG recordings using stereotactically implanted depth electrodes [12,13]. e v Th isual neurofeedback signal, which is the variable to e m Th acro contacts used for neurofeedback are in most be controlled by the user, corresponds to the unrest level cases in the target region (AM); in some cases they are in of the WR. The WR layout makes the unrest state immedi - the boundary with the hippocampus; it is summarized in ately visible to the user through the number of characters BRAIN–COMPUTER INTERFACES 205 Figure 4.  signal damping properties of the Waiting room, plotted with simulated input signals (see f igure 5). t he ‘lowpass’ filtering effect limits rapid fluctuation of unrest levels during neurofeedback (see text for details). Figure 5. Damping properties of BCi signal input by the Wr at various frequencies obtained through input signal simulation. t he spatial layout and updating mode of the Wr damps input signal fluctuations, which is meant to facilitate subjects’ nf effort, by making transient fluctuations less visible. BCi input consists of a target unrest level determined by statistical testing of the ieeG nf signal against a resting epoch baseline. a ctual Wr unrest closely follows variations at low frequencies (f igure 5a) while it fluctuates around an average higher value at higher frequencies (f igure 5b). t hese damping properties are also visible on actual signals on f igure 9. standing at the desk. To preserve the visual realism of with ‘calming down’ the WR, the initial distribution cor- the setting, transitions between levels of unrest are repre- responds to an unrest value of [0.5–1.0] for which a sig- sented through real-time animations showing characters nificant fraction of characters are in their standing state. moving from one location (front desk) to the other (sitting This situation makes it possible to give feedback to the area). The flow of characters, from or towards the desk, user whatever the variation of her BCI input signal, which also gives a clear trend on whether the user succeeds in is important in the early stages of a neurofeedback epoch. sustaining the neurofeedback signal, while damping the e WR unifies a Th n engaging task, which is a good visual BCI input signal fluctuations (see below). metaphor for arousal, with an empirical mechanism to damp It is possible to generate initial configurations at any neurofeedback signal fluctuations, potentially facilitating given unrest state: for instance, when the user is tasked adaptation to the neurofeedback task with minimal training. 206 H. G. YAMIN ET AL. Figure 6.  Block design. each of the sessions included a baseline epoch (BL) of passive viewing of the Waiting room environment for 60 s, for which the unrest level was set to 0.5; an active neurofeedback epoch (nf) for 60 s; and a finger tapping epoch for 3 s, to restore the baseline. In order to characterize these signal damping properties, (60 s each) following an equivalent number of active neu- we have performed an empirical evaluation using gener- rofeedback (NF) epochs (60 s each) (see details of protocol ated square signals simulating BCI input to the WR, which design in Figure 6). Between blocks, a 3 s finger tapping consists of a target unrest level to be visualized in the WR. block was introduced; we have set the finger tapping to We have generated square signals in a frequency range the shortest possible latency, as analysis of preliminary corresponding to expected fluctuations of the BCI input data revealed that the use of finger tapping washout signals, i.e. [0.1–2 Hz], with an amplitude varying across period does not facilitate the execution of the paradigm. the full [0,1] interval of unrest values. At each frequency, e p Th articipants were instructed to ‘appease the waiting we have recorded the WR response in terms of oscilla- room using their brain activity’. No preferred cognitive tion of the actual unrest value and have computed signal strategy was specified for NF, to avoid influencing subject’s attenuation in dB using maximum amplitude for both down-regulation. input (simulated square target unrest) and output (actual e NF p Th robe was the energy in the gamma band unrest) signals. Results from this simulation are plotted (25–35 Hz and 65–85 Hz), the band around 50 Hz was on Figure 4. The simulated input signal varies over the removed to avoid line noise. The probe was calculated amplitude of the normalized input signal (iEEG) rather every 1 s using Welch power spectrum estimation method than ‘unrest values’, which are automatically derived from with eight time windows with 50% overlap (pwelch Matlab it. The attenuation is computed from the initial unrest function). The calculated values were averaged every 3 s. value corresponding to the simulated input rather than the Higher gamma band activity is shown in animal mod- updated one. These results confirm that the WR is acting els of chronic stress [14] and in humans exposed to fearful as a kind of ‘lowpass filter’ where amplitude starts decreas - stimuli [15,16]. The study of Gaona et al. [17] suggested ing for frequencies > 0.6 Hz and with an attenuation level the specificity of different sub-bands across the 60–500 Hz of –8.5 dB/Hz, effectively limiting sudden variations of the range to various cognitive tasks [7]. Gamma oscillations BCI input signal (here, target unrest). have been associated with a wide range of cognitive pro- Figure 5 illustrates more specifically the lowpass filter - cesses including perception, attention and memory [2,18]. ing behavior at two simulated frequencies. On Figure 5a, In addition, Jerbi et al. [12] used high-gamma frequency for low fluctuations, the WR response closely follows the (60–140 Hz) for their depth electrode BCI based on cog- variations of the input signal. However, at higher frequen- nitive tasks. cies (Figure 5b), physical state transitions cannot be com- In the BL, blocks the probe values were used to con- pleted within the time range of fluctuations, because the struct a distribution of gamma power, to which the probe next update cycle is triggered before some characters have values in the NF blocks will be compared. NF probe values reached their previous target state due to the time required were compared to the BL values immediately preceding it. to travel through the environment, even preventing some During BL, outliers were treated as follow: values exceed- characters from changing state altogether. This results in ing median  +  10SD of the 90 first percentiles of the BL the WR unrest fluctuating around an average value. probe was replaced with the median. Then the median and standard deviation (SD) of the BL probe values were recalculated and a normal cumulative distribution func- Neurofeedback tion (normCDF) with this value was estimated. In the NF, Each of the NF sessions included four to eight baseline blocks probe values were compared to the normCDF, if (BL) epochs of passive viewing of the WR environment the probability to receive the NF value or higher in the BRAIN–COMPUTER INTERFACES 207 Figure 7. t he results presented (from patient D004 in (a); from patient D005 in (b) and (c)) show the average ± sem of values of gamma power for the baseline (labeled BL#) and the neurofeedback (labeled nf#) epochs. i n (a), two trials show a significant down-regulation, marked by *. Blue bars (nf) show a trend in down-regulation, although only one trial BL5-nf5 (in (c)) shows a significant down-regulation. BL was in the range 0.25 to 0.75, the WR received 0.5, sampling to 400  Hz was done by using Matlab’s resam- i.e. no change in the state of the room. For probabilities ple function, which applies an antialiasing FIR low pass exceeding those values, the WR received the probability filter to the signal and compensates for the delay intro- itself and the WR was changed accordingly. This was done duced by the filter. Spectrograms were calculated using in order to reinforce major changes in the activity; i.e. the welch method; parameters were selected to receive minor changes around the median activity in the BL did 1  Hz frequency resolution and 1 s temporal resolution. not receive any feedback. Aer c ft alculating the spectrogram, a trace of the power at the different frequency bands – delta (0–4 Hz), theta (4–8  Hz), alpha (8–16  Hz), beta (16–25  Hz), gamma Data analysis (25–35 Hz and 65–85 Hz) and high gamma (85–200 Hz) – All data analysis was performed using manually written was calculated by averaging the power in the specific fre- scripts in Matlab (Mathworks, Natick, MA, USA). Before quency range. Each trace of different frequency band was z calculating the spectrogram macro contact signals were scored, because we were interested in the change in power down sampled to 400  Hz, enabling spectral inference and not its absolute value. From each of these traces a in the range 0–200  Hz (Nyquist frequency). The down vector in length 2*number of blocks, in which each entry 208 H. G. YAMIN ET AL. Figure 8. nf sessions showing the input (in red) and output (in blue) of the Wr application for patient D004 (a) and for the two sessions from patient D005 (b). t he input levels (in red) during nf are normalized to the BL probe probability as describes in the method section. notice that the processed ieeG input levels (in red) are only relevant during nf , as in BL the unrest level of the room is predetermined, therefore the input values in the BL section are steady and does not reflect the gamma power of the macro contact. Figure 9.  t he down-regulation of the higher gamma band (65–85  Hz) is more prominent than the lower band (25–35  Hz). (a–d) the fraction of reduction in gamma power divided to lower and higher ranges used in the paradigm. t he reduction in the higher gamma range is more prominent for both subjects, though it does not reach significance for the left amygdala of subject D005. * represents statistical significance in a paired t-test with p < 0.05. represents the average power in a frequency band in a the ability to down-regulate their amygdala gamma band certain block was created. Each such vector represents the activity obtained by depth electrodes even with little train- course of change in power along the paradigm, similar to ing. A successful relaxation block was defined as a block that plotted in Figure 7. during which amygdala gamma power values were signif- All correlations were Pearson correlations and were icantly lower than those recorded in the baseline blocks calculated using Matlab’s corr function. (Wilcoxon signed rank test; p < 0.05 Bonferroni corrected for the number of blocks in a session). Success was found in 17% of the relaxation blocks. The successful blocks are Results marked with an asterisk in Figure 7. Three sessions from two patients, one from D004 (with A successful relaxation session was defined as a ses- eight BL-NF blocks) and two from D005 (with four and sion in which the gamma power in all NF blocks pulled v fi e BL-NF blocks), were analyzed. Figure 7 presents the together was lower than the BL blocks. Two out of the average gamma power along the BL-NF blocks of the ses- three sessions were found successful: the session of D004 sions, for patient D004 and patient D005. The analysis of and the second session of D005 (Wilcoxon signed rank these preliminary data suggests that participants acquired test; p < 0.05). BRAIN–COMPUTER INTERFACES 209 Figure 10. t he modulation in power is restricted to the gamma band. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) for the right (right panel) and left (left panel) amygdala of subject D004 for the delta (a–b), theta (c–d), alpha (e–f ), beta (g–h), gamma (i–j) and high gamma (k–l) bands. t he reduction in mean power between BL and nf blocks is apparent in the gamma and high gamma bands. 210 H. G. YAMIN ET AL. Figure 11. inspection of the re-referenced signal in the target macro wires showing that the effect in the amygdala is maintained. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) for the right (right panel) and left (left panel) amygdala of subject D004 for the delta (a–b), theta (c–d), alpha (e–f ), beta (g–h), gamma (i–j) and high gamma (k–l) bands. BRAIN–COMPUTER INTERFACES 211 Figure 12. subset of gamma activity from contacts correlated with the activity of the amygdala during the task, from patients D004 and D005. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) from D004: (a) left cerebral white matter; (b) right cerebral white matter; D005 session 1: (c) left hippocampus; (d) right amygdala; (e) right cerebral white matter; (f ) right hippocampus; D005 session 2: (g) right amygdala; (h) right cerebral white matter. In a post-session interview with the subjects regarding and the other extrospective – might influence their success their used strategy to calm the room, D004 reported he rate and the recruiting of other brain regions except the tried to calm himself down, whereas D005 stated he men- amygdala. tally spoke to the crowd in the waiting room telling them Figure 8 shows the variations of the unrest level to relax. The difference in strategies – one introspective (Nstand/Nsit – blue line) in the WR per BL corresponding 212 H. G. YAMIN ET AL. Table 2.  areas (macro contacts) with significant p earson’s r cor- frequency bands. The correlation was calculated between relations to the target (amygdala). Correlations were calculated vectors in length 2*number of blocks, in which each entry after re-referencing. represents the average power in a frequency band in a cer- Subject Macro contact location Pearson’s r Correlation tain block (see Methods). The correlation was significant D004 r ight hippocampus 0.63 only between gamma and high gamma in the right amyg- D004 r ight hippocampus 0.73 dala of subject D004 (R  =  0.96, p < 0.0001) and the left D004 r ight hippocampus 0.73 D004 r ight temporal white matter 0.55 amygdala of subject D005 session 2 (R = 0.86, p = 0.001), D004 r ight temporal white matter 0.61 and between gamma to theta and beta in the right amyg- D004 r ight amygdala 0.79 D004 r ight amygdala 0.74 dala of subject D004 (R  =  0.76, p = 0.001 and R  =  0.60, D004 r ight temporal white matter 0.76 p = 0.01 for theta and beta respectively). To conclude, the D004 r ight temporal white matter 0.77 volitional control by the subjects is restricted to the target D004 Left temporal white matter 0.58 D004 Left middle temporal gyrus 0.55 band – gamma band or to high gamma band. D004 Left amygdala 0.70 To verify that the volitional control is not due to global D004 Left temporal white matter 0.57 D004 Left temporal white matter 0.65 effects, we have re-referenced the macro wire to the global D004 Left temporal white matter 0.56 effect by subtracting from each signal the average signal D004 Left middle temporal gyrus 0.52 D005 – 1 r ight amygdala 0.88 that was collected simultaneously by all macro contacts. D005 – 1 r ight temporal white matter 0.71 Inspection of the re-referenced signal in the target macro D005 – 1 r ight hippocampus 0.81 D005 – 1 Left hippocampus 0.85 wires showed that the ee ff ct in the amygdala is maintained D005 – 2 r ight temporal white matter 0.94 (see Figure 11). Furthermore, we have created for all other D005 – 2 r ight temporal white matter 0.80 macro contact a vector representing its mean value in the gamma band (for a total of 47 in D004 and 71 and 73 macro contacts in D005 session 1 and 2 respectively – the to the input level of relative gamma power (red line). The difference in number is due to noisy contacts that were level of unrest in the BL epoch was set to 0.5 by default removed per session). For each electrode a vector sized and was not influenced by the gamma power. 2*number of blocks was calculated, in which each entry Due to the relatively wide frequency range that we have represents the average power in a frequency band in a used for feedback, we wanted to test whether the observed certain block. We than calculated the correlation between reduction in gamma power is similar at the entire range each macro contact activity and the pattern of activity in or is ae ff cted by the lower (25–35  Hz) or higher range the amygdala. Figure 12 depicts a subset of gamma activity (65–85 Hz). To that aim we have calculated for each sub- from contacts that were correlated with the activity of the ject the fraction of reduction in power between BL and amygdala during the task. Table 2. summarizes the signif- the following NF block ((NF power–BL power)/BL power) icant Pearson correlations (p < 0.05); it is noticeable that in the lower and higher gamma range. Figure 9 depicts only a small fraction of the contacts was correlated with the average fraction of reduction in power in each of the the NF probe (11.5%), and that the structures correlated frequency ranges for both the right and left amygdala of with it are all in the temporal lobe, which has functional both subjects. Asterisks represent a significant difference connectivity to the amygdala. Thus, the volitional control between the bands (paired t-test, p<0.05). It shows that of the amygdala gamma band activity was specific to cor - in most cases the reduction in power between BL and NF tical circuits that are functionally related to the amygdala. is more prominent at the higher frequency range used in Here we have presented preliminary results demon- the paradigm (65–85 Hz). strating that it is possible to train subjects to directly and Figure 10 depicts the trace of the power in the common specifically modulate their amygdala's activity using depth division of the EEG into frequency bands – delta (0–4 Hz), electrodes as a signal acquisition device. Results obtained theta (4–8 Hz), alpha (8–16 Hz), beta (16–25 Hz), (gamma are encouraging, in particular the relatively high success 25–35 Hz and 65–85 Hz) and high gamma (85–200 Hz) – rate considering minimal training received by subjects. in the two target macro contacts: the right and the left amygdala from subject D004. The black line is the instan- taneous power calculated every 1 s (see Methods) and the Conclusions red trace is the average power in a certain block (BL and We have presented results showing the successful use of a NF interchangeably). It demonstrates that the change in depth electrode BCI by two patients. For the foreseeable power we observed in the gamma band is restricted to future, and considering the risks and side effects of depth the target band or to higher frequency bands and, to a electrodes, these systems will be restricted to patients who much lesser extent, to lower frequency bands. To quan- have received a therapeutic indication. Their use as BCI tify it, we have calculated the correlation between the will be mostly a side effect, which distinguishes them from average gamma power and the average power in all other BRAIN–COMPUTER INTERFACES 213 [3] Jacobs J, Kahana MJ. Direct brain recordings fuel BCI use to assist patients with motor impairment. From advances in cognitive electrophysiology. Trends Cogn an interface technology perspective, the type of system Sci. 2010;14(4):162–171. we have presented has the potential to favor engagement [4] Schalk G, Leuthardt EC. Brain-computer interfaces using in therapeutic testing phases of the iEEG, which was the electrocorticographic signals. IEEE Rev Biomed Eng. initial rationale for its development. However, following 2011;4:140–154. previous reports [6], we can speculate that the use of [5] Engel AK, Moll CK, Fried I, et al. Invasive recordings from the human brain: clinical insights and beyond. Nat BCI in a gaming context could have a positive effect on Rev Neurosci.. 2005;6(1):35–47. patients’ morale and self-esteem, and potentially improve [6] Breshears JD, Gaona CM, Roland JL, et al. Decoding their acceptance of the equipment. motor signals from the pediatric cortex: implications Overall, we have shown that it is possible to harness for brain-computer interfaces in children. Pediatrics. the power of depth electrodes to readily integrate them as 2001;128(1):160–168. [7] Schalk G, Miller K, Anderson N, et al. Two-dimensional BCI in a typical interactive application. In addition to the movement control using electrocorticographic signals in potential of VR or gaming interfaces for patient engage- humans. J. Neural Eng.. 2008;5(1):75. ment, we have also reported some emergent properties [8] Krusienski D, Shih J. Control of a brain–computer interface of these interfaces in terms of input signal filtering. The using stereotactic depth electrodes in and adjacent to the dynamic properties of the interface related to the timing hippocampus. J. Neural Eng.. 2011;8(2):025006. of realistic animations result in a low pass filtering that [9] Shih JJ, Krusienski DJ. Signals from intraventricular depth electrodes can control a brain–computer interface. may play a role in facilitating subjects’ adaptation to the J Neurosci Methods. 2012;203(2):311–314. neurofeedback task. Such a feature could prove of a spe- [10] Lachaux J-P, Jerbi K, Bertrand O, et al. BrainTV: a novel cific interest whenever options for neurofeedback training approach for online mapping of human brain functions. sessions are limited, as is the case with depth electrodes. Biol. Res.. 2007;40(4):401–413. Despite the very minimal user sample, the fact that sig- [11] Jerbi K, Ossandón T, Hamame CM, et al. Task- related gamma-band dynamics from an intracerebral nal behavior was consistent with published observations perspective: Review and implications for surface EEG would encourage us to think that this approach can be and MEG. Hum. Brain Mapp.. 2009;30(6):1758–1771. reproduced. [12] Jerbi K, Freyermuth S, Minotti L, et al. Watching brain TV and playing brain ball: Exploring novel BCI strategies using real-time analysis of human intracranial data. Int Disclosure statement Rev Neurobiol. 2009;86:159–168. No potential conflict of interest was reported by the authors. [13] Kahane P, Minotti L, Hoffmann D, et al. Invasive EEG in the definition of the seizure onset zone: depth electrodes. Handb Clin Neurol. 2003;3:109–133. Funding [14] Ghosh S, Laxmi TR, Chattarji S. Functional connectivity from the amygdala to the hippocampus grows stronger This work was supported by the European Union’s Seventh aer s ft tress. J Neurosci. 2013;33(17):7234–7244. Framework Programme for research, technological develop- [15] Oya H, Kawasaki H, Howard MA, et al. ment and demonstration [grant number 602186] and by the Electrophysiological responses in the human amygdala Ministry of Science, Technology and Space [grant number discriminate emotion categories of complex visual 3-11170] and by the Sagol family fund. stimuli. J Neurosci. 2002;22(21):9502–9512. [16] Sato W, Kochiyama T, Uono S, et al. Rapid amygdala gamma oscillations in response to fearful facial ORCID expressions. Neuropsychologia. 2011;49(4):612–617. [17] Gaona CM, Sharma M, Freudenburg ZV, et al. Gal Raz   http://orcid.org/0000-0001-9032-8613 Nonuniform high-gamma (60–500  Hz) power changes Marc Cavazza   http://orcid.org/0000-0001-6113-9696 dissociate cognitive task and anatomy in human cortex. J. Neurosci. 2011;31(6):2091–2100. References [18] Fries P. Neuronal gamma-band synchronization as a fundamental process in cortical computation. Annu Rev [1] McFarland DJ, Wolpaw JR. Brain-computer interfaces Neurosci. 2009;32:209–224. for communication and control. Commun ACM. [19] Fried I, Wilson CL, Maidment NT, et al. Cerebral 2011;54(5):60–66. microdialysis combined with single-neuron and [2] Jayakar P, Gotman J, Harvey AS, et al. Diagnostic utility electroencephalographic recording in neurosurgical of invasive EEG for epilepsy surgery: indications, patients: technical note. J Neurosurg. 1999;91(4):697– modalities, and techniques. Epilepsia. 2016;57(11):1735– http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Brain-Computer Interfaces Taylor & Francis

Depth electrode neurofeedback with a virtual reality interface

Abstract

AbstractInvasive brain–computer interfaces (BCI) provide better signal quality in terms of spatial localization, frequencies and signal/noise ratio, in addition to giving access to deep brain regions that play important roles in cognitive or affective processes. Despite some anecdotal attempts, little work has explored the possibility of integrating such BCI input into more sophisticated interactive systems like those which can be developed with game engines. In this article, we...
Loading next page...
 
/lp/taylor-francis/depth-electrode-neurofeedback-with-a-virtual-reality-interface-TuwD5vZ8WE
Publisher
Taylor & Francis
Copyright
© 2017 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
ISSN
2326-2621
eISSN
2326-263x
DOI
10.1080/2326263X.2017.1338008
Publisher site
See Article on Publisher Site

Abstract

Brain–Computer interfa Ces, 2017 VoL. 4, no . 4, 201–213 https://doi.org/10.1080/2326263X.2017.1338008 OPEN ACCESS a a b a a c Hagar Grazya Yamin , Tomer Gazit , Natalia Tchemodanov , Gal Raz   , Gilan Jackont , Fred Charles , b,d a e Itzhak Fried , Talma Hendler and Marc Cavazza   a b Center for Brain f unctions, t el a viv s ourasky medical Center, t el a viv, israel; Department of neurosurgery, David Geffen s chool of medicine and s emel institute for neuroscience and Human Behavior, university of California Los angeles, Los angeles, C a, usa; Department of Creative technology, f aculty of s cience and technology, Bournemouth university, poole, united Kingdom; f unctional neurosurgery unit, tel-a viv medical Center and s ackler s chool of medicine, t el-a viv university, t el-a viv, israel; s chool of engineering and Digital arts, university of Kent, Canterbury, united Kingdom ABSTRACT ARTICLE HISTORY received 30 July 2016 Invasive brain–computer interfaces (BCI) provide better signal quality in terms of spatial localization, a ccepted 8 may 2017 frequencies and signal/noise ratio, in addition to giving access to deep brain regions that play important roles in cognitive or affective processes. Despite some anecdotal attempts, little work has KEYWORDS explored the possibility of integrating such BCI input into more sophisticated interactive systems Brain–computer interface like those which can be developed with game engines. In this article, we integrated an amygdala (BCi); neurofeedback (nf); depth electrode recorder with a virtual environment controlling a virtual crowd. Subjects were electroencephalogram (eeG); asked to down regulate their amygdala using the level of unrest in the virtual room as feedback on intracranial depth electrodes how successful they were. We report early results which suggest that users adapt very easily to this SUBJECT paradigm and that the timing and fluctuations of amygdala activity during self-regulation can be CLASSIFICATION matched by crowd animation in the virtual room. This suggests that depth electrodes could also CODES serve as high-performance affective interfaces, notwithstanding their strictly limited availability, application development justified on medical grounds only. and evaluation; neurosurgical approaches and methods, affective computing; signal acquisition: eeG (other) Introduction hippocampus, which play important roles in cognitive and ae ff ctive processing. In this article, we explore the With the development of brain–computer interfaces (BCI) development of a depth electrode BCI using a neurofeed- as an interdisciplinary endeavor ranging from neural engi- back (NF) paradigm, in which users are asked to down- neering to user interface technology [1], there has been regulate their amygdala (gamma band) activity to control a growing interest in invasive BCI developed in clinical the behavior of a virtual environment populated by a small settings, to gain knowledge on some fundamental BCI crowd of virtual characters. This system was originally problems, and more speculatively, develop user interfaces developed to improve engagement for patients having to around invasive BCI. Invasive systems are implemented for test the recording ability of their implanted depth elec- strict medical indications, primarily the monitoring and trodes, and we report here early results on its use in two presurgical evaluation of medically intractable epilepsy patients to illustrate the properties of depth electrodes BCI [2]. The two main invasive approaches are electrocorti- and the end-to-end integration in a virtual environment cography (ECoG, [2–4]) where intracranial electrodes are TM based on the Unreal Development Kit (UDK ) engine. placed at the surface of the cerebral cortex, and depth electrodes which are inserted inside inaccessible regions, for instance to monitor and control temporal epilepsy [5]. Previous and related work Compared to EEG electrodes, invasive recordings oer a ff Although the literature on invasive BCI primarily addresses better spatial resolution, the ability to capture high EEG assistive technology for motor impaired patients, several frequencies with fewer artifacts and, in the case of depth electrodes, an even greater spatial resolution, and the authors have reported experiments in which users were ability to access deep regions such as the amygdala and able to control other applications, in particular computer CONTACT Hagar Grazya Yamin yaminhagar@yahoo.com t his paper describes research originally presented at the 6th international BCi meeting, organized by the BCi s ociety and held may 30–June 3rd 2016 in pacific Grove, California, usa. © 2017 t he a uthor(s). published by informa uK Limited, trading as t aylor & f rancis Group. t his is an open a ccess article distributed under the terms of the Creative Commons a ttribution-nonCommercial-noDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 202 H. G. YAMIN ET AL. games, as an illustration of quality and controllability of the strategy and the controlled game, as the user aims at signal. Breshears et al. [6] have used ECoG as an interface simultaneously activating/deactivating the target region to a version of the popular Space Invader game in which and agitating/relaxing the ‘Waiting Room’. This contrasts, a young patient would control one-dimensional lateral for instance, with the use of cognitive tasks such as mental movement of the base spaceship that would fire auton- calculation to control the position of the ball in Brain Ball omously. The same group had previously demonstrated [12]. Like previous work, we report on isolated numbers the possibility of controlling a cursor in two dimensions of patients (here two patients), so the findings are to be with success rates of 53–73% [7]. Previous BCI making interpreted in the context of previous literature and the use of depth electrodes include Krusienski and Shih [8] description of fundamental phenomena associated with who have adapted the P300 spelling paradigm for use with neural activity, and claim no statistical validity. a depth electrode, describing its use by one subject with a hippocampal depth electrode [9]. Lachaux et al. [10] have System overview reported the use of depth electrodes for neurofeedback, using the direct visualization of plotted cortical activity We developed a user interface concept to support NF as a visual feedback channel. The same team was the first called the ‘Waiting Room’ (WR), which is a virtual envi- to use this paradigm with a bespoke game called ‘Brain ronment populated by animated characters (see Figure 1). Ball’ in which subjects would control the position of a ball One frequently observed issue in NF is that the usability on the screen by self-initiated gamma power (triggered by of traditional feedback signals (e.g., gages and thermom- voluntarily engaging in mental calculation). In terms of eters) is oen limi ft ted, for two reasons. The first one is signal mapping, the instantaneous position of the ball was the lack of user engagement in a context which is seen as updated in real-time according to a linear transformation too close to an assessment or an experiment. By contrast, of the instantaneous change in gamma power from the embedding feedback into a task-based application or a parietal depth electrode [11]. game is known to improve engagement [2]. e Th second Overall, this article is the first to combine depth elec- one is that full synchronous coupling between the meas- trodes in a neurofeedback context with a realistic, real- ured BCI signal and the visual signal tends to feedback all time game-like interface developed using the Unreal oscillations and inherent instability related to the effortful TM Development Kit (UDK ) engine. In addition, we depart component of neurofeedback, which can lead to loss of from direct physical control of motion direction or posi- focus, or ‘catastrophic’ reactions when the user cannot tion (spaceship, ball) to consider more abstract concepts sustain their neurofeedback effort. This is the case in par - of mental/aeff ctive state regulation for which the 3-D ticular with high frequency signals derived from EEG: res- interface is primarily a metaphor. Importantly, this met- ampling through moving averages smoothens variations, aphor still establishes continuity between the cognitive but at the cost of delays and impact on variation ranges. Figure 1.  s ystem o verview and experimental s etting. t he user, implanted with Behnke–f ried depth electrode [19], is instructed to appease the situation in the real-time 3-D visualization of the ‘Waiting room’. neurofeedback is based on the input of gamma band amygdala ieeG power. ieeG is recorded using neuroport (Blackrock microsystems). t he user is expected to lower the level of unrest in the ‘Waiting room’ by down-regulating her amygdala activity. t he real-time amygdala gamma band power is plotted during several short nf trials, where the nf epochs (in blue) must show lower values than during the Baseline epochs (in white). BRAIN–COMPUTER INTERFACES 203 Figure 2. Characters standing up from their seats and walking to the reception desk. Figure 3.  arrows represent the deepest macro contact located in the left amygdala of patient D004 (a) and in the right amygdala of patient D005 (b). t he unmarked bulbs represent the location of the rest of the macro contacts on the same electrode (light blue in (a) and purple in (b)) or at a nearby electrode (red bulbs in (a)). A more appropriate feedback interface would both have virtual characters as a metaphor for arousal (see Figure 2). a realistic (semantic) outlook and support some kind of e Th characters can evolve between two states: one resting signal integration that would sustain the user’s neurofeed- state, in which they are sat waiting to be called; and one back cognitive efforts, by canceling transient dips in acti- agitated state, in which they move to the front desk to vation towards the threshold (regardless of the position complain about waiting times. The ratio between the char - of the threshold with respect to the baseline). The WR is acters standing and sitting (N /N ) at the desk defines stand sit designed to meet these requirements, and uses a crowd of a level of ‘unrest’, which is used as the target variable for 204 H. G. YAMIN ET AL. Table 1. t otal number of electrodes and the positioning of the target amygdala macro electrode for the two subjects. Patient ID Total number of electrodes Amygdala macro electrode MNI D004 6 Left: x = –24, y = –4, z = –24 r ight: x = 24, y = –6, z = –29 (located at the border with the hippocampus) D005 10 Left: x = –24, y = –8, z = –26 (located at the border with the hippocampus) r ight: x =24, y = –8, z = –17 BCI signal mapping. The level of activation for the amyg- Table 1. In patient D004 and the first session of D005, the dala is translated into a level of unrest, which is updated feedback was bilateral, the feedback of the second session in the WR through a set of discrete transitions during of D005 was taken from the right amygdala. which additional characters stand up and move towards We localized electrodes by performing computed the front desk. It is this realistic simulation of characters tomography scans aer im ft plant; aligning the computed motion that dampens the BCI signal uc fl tuation, because tomography images with pre-implant MRIs and then labe- the time required for characters to move across the room ling the region of each electrode according to anatomical prevents drastic changes in either populations (waiting landmarks (see Figure 3). or standing). e p Th lacement of electrodes is based solely on the data obtained from scalp EEG, imaging and other tests undergone by the patient. Likewise, the length of time Methods the patient remains implanted is determined only by the requirement to ‘capture’ sufficient seizures on EEG to Apparatus determine the seizure onset zone. We used Behnke–Fried depth electrodes (Ad-Tech medical instruments, USA; see Figure 1) [1,5]. Each electrode has Sampling and signal acquisition eight platinum-coated macro contacts (1.6 mm diameter, separated by 5 mm), and eight platinum-iridium micro e o Th nline interaction between the Neuroport system contacts (seven recording and one reference electrode) and the MatLab file controlling the WR was done using located at the tip of each electrode (38  µm diameter). cbmex platform, a set of MatLab functions provided by e micr Th o contacts are capable of recording single units, Blackrock Microsystem for online communication, with whereas the macrowires record activity of a wider volume the data collected through the neural signal processors. referenced here as iEEG. Because the gamma band activity e sys Th tem amplifier receives signals directly from the collected by the microwires was more prone to noise, we electrodes through a headstage or the patient cable. The used macro contact data for neurofeedback purposes. sampled signal (up to 30 kHz) is amplie fi d, analog filtered (1st-order high-pass at 0.3 Hz and 3rd-order low-pass at 7,500  Hz) and digitized (16-bits at 250 nV resolution) Subjects and then is transmitted to the NSP via a fiber optic link. Two adult patients recruited at the authors’ institution e a Th nalog filter allows for the low-frequency field poten - (D004: male, 23 years old; and D005: male, 40 years old) tials as well as the higher frequency spike signals to be with intractable epilepsy were implanted with depth elec- recorded. A later stage digital filtering will allow these trodes to localize the epileptic focus for possible subsequent two signals to be separated (NeuroPort user’s manual, resection. Patients were stereotactically implanted with six Blackrock Microsystems). Here we used the low fre- and ten depth electrodes from a lateral orthogonal approach quency field potentials, the signals were digitally sampled aiming at targets selected using clinical criteria. Following at 2 kHz, and bandpass filtered between 0.3 and 500 Hz implantation, patients remained between one and two weeks (Neuroport, Blackrock Microsystems, USA). All macro hospitalized for monitoring (subject D004 10 days, subject contacts were referenced to a scalp electrode (Pz). D005 14 days). All patients provided informed consent. All studies conformed to the guidelines of the authors’ institu- Neurofeedback setting: the Waiting Room tion. All patients underwent intracerebral EEG recordings using stereotactically implanted depth electrodes [12,13]. e v Th isual neurofeedback signal, which is the variable to e m Th acro contacts used for neurofeedback are in most be controlled by the user, corresponds to the unrest level cases in the target region (AM); in some cases they are in of the WR. The WR layout makes the unrest state immedi - the boundary with the hippocampus; it is summarized in ately visible to the user through the number of characters BRAIN–COMPUTER INTERFACES 205 Figure 4.  signal damping properties of the Waiting room, plotted with simulated input signals (see f igure 5). t he ‘lowpass’ filtering effect limits rapid fluctuation of unrest levels during neurofeedback (see text for details). Figure 5. Damping properties of BCi signal input by the Wr at various frequencies obtained through input signal simulation. t he spatial layout and updating mode of the Wr damps input signal fluctuations, which is meant to facilitate subjects’ nf effort, by making transient fluctuations less visible. BCi input consists of a target unrest level determined by statistical testing of the ieeG nf signal against a resting epoch baseline. a ctual Wr unrest closely follows variations at low frequencies (f igure 5a) while it fluctuates around an average higher value at higher frequencies (f igure 5b). t hese damping properties are also visible on actual signals on f igure 9. standing at the desk. To preserve the visual realism of with ‘calming down’ the WR, the initial distribution cor- the setting, transitions between levels of unrest are repre- responds to an unrest value of [0.5–1.0] for which a sig- sented through real-time animations showing characters nificant fraction of characters are in their standing state. moving from one location (front desk) to the other (sitting This situation makes it possible to give feedback to the area). The flow of characters, from or towards the desk, user whatever the variation of her BCI input signal, which also gives a clear trend on whether the user succeeds in is important in the early stages of a neurofeedback epoch. sustaining the neurofeedback signal, while damping the e WR unifies a Th n engaging task, which is a good visual BCI input signal fluctuations (see below). metaphor for arousal, with an empirical mechanism to damp It is possible to generate initial configurations at any neurofeedback signal fluctuations, potentially facilitating given unrest state: for instance, when the user is tasked adaptation to the neurofeedback task with minimal training. 206 H. G. YAMIN ET AL. Figure 6.  Block design. each of the sessions included a baseline epoch (BL) of passive viewing of the Waiting room environment for 60 s, for which the unrest level was set to 0.5; an active neurofeedback epoch (nf) for 60 s; and a finger tapping epoch for 3 s, to restore the baseline. In order to characterize these signal damping properties, (60 s each) following an equivalent number of active neu- we have performed an empirical evaluation using gener- rofeedback (NF) epochs (60 s each) (see details of protocol ated square signals simulating BCI input to the WR, which design in Figure 6). Between blocks, a 3 s finger tapping consists of a target unrest level to be visualized in the WR. block was introduced; we have set the finger tapping to We have generated square signals in a frequency range the shortest possible latency, as analysis of preliminary corresponding to expected fluctuations of the BCI input data revealed that the use of finger tapping washout signals, i.e. [0.1–2 Hz], with an amplitude varying across period does not facilitate the execution of the paradigm. the full [0,1] interval of unrest values. At each frequency, e p Th articipants were instructed to ‘appease the waiting we have recorded the WR response in terms of oscilla- room using their brain activity’. No preferred cognitive tion of the actual unrest value and have computed signal strategy was specified for NF, to avoid influencing subject’s attenuation in dB using maximum amplitude for both down-regulation. input (simulated square target unrest) and output (actual e NF p Th robe was the energy in the gamma band unrest) signals. Results from this simulation are plotted (25–35 Hz and 65–85 Hz), the band around 50 Hz was on Figure 4. The simulated input signal varies over the removed to avoid line noise. The probe was calculated amplitude of the normalized input signal (iEEG) rather every 1 s using Welch power spectrum estimation method than ‘unrest values’, which are automatically derived from with eight time windows with 50% overlap (pwelch Matlab it. The attenuation is computed from the initial unrest function). The calculated values were averaged every 3 s. value corresponding to the simulated input rather than the Higher gamma band activity is shown in animal mod- updated one. These results confirm that the WR is acting els of chronic stress [14] and in humans exposed to fearful as a kind of ‘lowpass filter’ where amplitude starts decreas - stimuli [15,16]. The study of Gaona et al. [17] suggested ing for frequencies > 0.6 Hz and with an attenuation level the specificity of different sub-bands across the 60–500 Hz of –8.5 dB/Hz, effectively limiting sudden variations of the range to various cognitive tasks [7]. Gamma oscillations BCI input signal (here, target unrest). have been associated with a wide range of cognitive pro- Figure 5 illustrates more specifically the lowpass filter - cesses including perception, attention and memory [2,18]. ing behavior at two simulated frequencies. On Figure 5a, In addition, Jerbi et al. [12] used high-gamma frequency for low fluctuations, the WR response closely follows the (60–140 Hz) for their depth electrode BCI based on cog- variations of the input signal. However, at higher frequen- nitive tasks. cies (Figure 5b), physical state transitions cannot be com- In the BL, blocks the probe values were used to con- pleted within the time range of fluctuations, because the struct a distribution of gamma power, to which the probe next update cycle is triggered before some characters have values in the NF blocks will be compared. NF probe values reached their previous target state due to the time required were compared to the BL values immediately preceding it. to travel through the environment, even preventing some During BL, outliers were treated as follow: values exceed- characters from changing state altogether. This results in ing median  +  10SD of the 90 first percentiles of the BL the WR unrest fluctuating around an average value. probe was replaced with the median. Then the median and standard deviation (SD) of the BL probe values were recalculated and a normal cumulative distribution func- Neurofeedback tion (normCDF) with this value was estimated. In the NF, Each of the NF sessions included four to eight baseline blocks probe values were compared to the normCDF, if (BL) epochs of passive viewing of the WR environment the probability to receive the NF value or higher in the BRAIN–COMPUTER INTERFACES 207 Figure 7. t he results presented (from patient D004 in (a); from patient D005 in (b) and (c)) show the average ± sem of values of gamma power for the baseline (labeled BL#) and the neurofeedback (labeled nf#) epochs. i n (a), two trials show a significant down-regulation, marked by *. Blue bars (nf) show a trend in down-regulation, although only one trial BL5-nf5 (in (c)) shows a significant down-regulation. BL was in the range 0.25 to 0.75, the WR received 0.5, sampling to 400  Hz was done by using Matlab’s resam- i.e. no change in the state of the room. For probabilities ple function, which applies an antialiasing FIR low pass exceeding those values, the WR received the probability filter to the signal and compensates for the delay intro- itself and the WR was changed accordingly. This was done duced by the filter. Spectrograms were calculated using in order to reinforce major changes in the activity; i.e. the welch method; parameters were selected to receive minor changes around the median activity in the BL did 1  Hz frequency resolution and 1 s temporal resolution. not receive any feedback. Aer c ft alculating the spectrogram, a trace of the power at the different frequency bands – delta (0–4 Hz), theta (4–8  Hz), alpha (8–16  Hz), beta (16–25  Hz), gamma Data analysis (25–35 Hz and 65–85 Hz) and high gamma (85–200 Hz) – All data analysis was performed using manually written was calculated by averaging the power in the specific fre- scripts in Matlab (Mathworks, Natick, MA, USA). Before quency range. Each trace of different frequency band was z calculating the spectrogram macro contact signals were scored, because we were interested in the change in power down sampled to 400  Hz, enabling spectral inference and not its absolute value. From each of these traces a in the range 0–200  Hz (Nyquist frequency). The down vector in length 2*number of blocks, in which each entry 208 H. G. YAMIN ET AL. Figure 8. nf sessions showing the input (in red) and output (in blue) of the Wr application for patient D004 (a) and for the two sessions from patient D005 (b). t he input levels (in red) during nf are normalized to the BL probe probability as describes in the method section. notice that the processed ieeG input levels (in red) are only relevant during nf , as in BL the unrest level of the room is predetermined, therefore the input values in the BL section are steady and does not reflect the gamma power of the macro contact. Figure 9.  t he down-regulation of the higher gamma band (65–85  Hz) is more prominent than the lower band (25–35  Hz). (a–d) the fraction of reduction in gamma power divided to lower and higher ranges used in the paradigm. t he reduction in the higher gamma range is more prominent for both subjects, though it does not reach significance for the left amygdala of subject D005. * represents statistical significance in a paired t-test with p < 0.05. represents the average power in a frequency band in a the ability to down-regulate their amygdala gamma band certain block was created. Each such vector represents the activity obtained by depth electrodes even with little train- course of change in power along the paradigm, similar to ing. A successful relaxation block was defined as a block that plotted in Figure 7. during which amygdala gamma power values were signif- All correlations were Pearson correlations and were icantly lower than those recorded in the baseline blocks calculated using Matlab’s corr function. (Wilcoxon signed rank test; p < 0.05 Bonferroni corrected for the number of blocks in a session). Success was found in 17% of the relaxation blocks. The successful blocks are Results marked with an asterisk in Figure 7. Three sessions from two patients, one from D004 (with A successful relaxation session was defined as a ses- eight BL-NF blocks) and two from D005 (with four and sion in which the gamma power in all NF blocks pulled v fi e BL-NF blocks), were analyzed. Figure 7 presents the together was lower than the BL blocks. Two out of the average gamma power along the BL-NF blocks of the ses- three sessions were found successful: the session of D004 sions, for patient D004 and patient D005. The analysis of and the second session of D005 (Wilcoxon signed rank these preliminary data suggests that participants acquired test; p < 0.05). BRAIN–COMPUTER INTERFACES 209 Figure 10. t he modulation in power is restricted to the gamma band. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) for the right (right panel) and left (left panel) amygdala of subject D004 for the delta (a–b), theta (c–d), alpha (e–f ), beta (g–h), gamma (i–j) and high gamma (k–l) bands. t he reduction in mean power between BL and nf blocks is apparent in the gamma and high gamma bands. 210 H. G. YAMIN ET AL. Figure 11. inspection of the re-referenced signal in the target macro wires showing that the effect in the amygdala is maintained. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) for the right (right panel) and left (left panel) amygdala of subject D004 for the delta (a–b), theta (c–d), alpha (e–f ), beta (g–h), gamma (i–j) and high gamma (k–l) bands. BRAIN–COMPUTER INTERFACES 211 Figure 12. subset of gamma activity from contacts correlated with the activity of the amygdala during the task, from patients D004 and D005. t he power trace (black line) and the average power in BL (white background) and nf (blue background) blocks (red line) from D004: (a) left cerebral white matter; (b) right cerebral white matter; D005 session 1: (c) left hippocampus; (d) right amygdala; (e) right cerebral white matter; (f ) right hippocampus; D005 session 2: (g) right amygdala; (h) right cerebral white matter. In a post-session interview with the subjects regarding and the other extrospective – might influence their success their used strategy to calm the room, D004 reported he rate and the recruiting of other brain regions except the tried to calm himself down, whereas D005 stated he men- amygdala. tally spoke to the crowd in the waiting room telling them Figure 8 shows the variations of the unrest level to relax. The difference in strategies – one introspective (Nstand/Nsit – blue line) in the WR per BL corresponding 212 H. G. YAMIN ET AL. Table 2.  areas (macro contacts) with significant p earson’s r cor- frequency bands. The correlation was calculated between relations to the target (amygdala). Correlations were calculated vectors in length 2*number of blocks, in which each entry after re-referencing. represents the average power in a frequency band in a cer- Subject Macro contact location Pearson’s r Correlation tain block (see Methods). The correlation was significant D004 r ight hippocampus 0.63 only between gamma and high gamma in the right amyg- D004 r ight hippocampus 0.73 dala of subject D004 (R  =  0.96, p < 0.0001) and the left D004 r ight hippocampus 0.73 D004 r ight temporal white matter 0.55 amygdala of subject D005 session 2 (R = 0.86, p = 0.001), D004 r ight temporal white matter 0.61 and between gamma to theta and beta in the right amyg- D004 r ight amygdala 0.79 D004 r ight amygdala 0.74 dala of subject D004 (R  =  0.76, p = 0.001 and R  =  0.60, D004 r ight temporal white matter 0.76 p = 0.01 for theta and beta respectively). To conclude, the D004 r ight temporal white matter 0.77 volitional control by the subjects is restricted to the target D004 Left temporal white matter 0.58 D004 Left middle temporal gyrus 0.55 band – gamma band or to high gamma band. D004 Left amygdala 0.70 To verify that the volitional control is not due to global D004 Left temporal white matter 0.57 D004 Left temporal white matter 0.65 effects, we have re-referenced the macro wire to the global D004 Left temporal white matter 0.56 effect by subtracting from each signal the average signal D004 Left middle temporal gyrus 0.52 D005 – 1 r ight amygdala 0.88 that was collected simultaneously by all macro contacts. D005 – 1 r ight temporal white matter 0.71 Inspection of the re-referenced signal in the target macro D005 – 1 r ight hippocampus 0.81 D005 – 1 Left hippocampus 0.85 wires showed that the ee ff ct in the amygdala is maintained D005 – 2 r ight temporal white matter 0.94 (see Figure 11). Furthermore, we have created for all other D005 – 2 r ight temporal white matter 0.80 macro contact a vector representing its mean value in the gamma band (for a total of 47 in D004 and 71 and 73 macro contacts in D005 session 1 and 2 respectively – the to the input level of relative gamma power (red line). The difference in number is due to noisy contacts that were level of unrest in the BL epoch was set to 0.5 by default removed per session). For each electrode a vector sized and was not influenced by the gamma power. 2*number of blocks was calculated, in which each entry Due to the relatively wide frequency range that we have represents the average power in a frequency band in a used for feedback, we wanted to test whether the observed certain block. We than calculated the correlation between reduction in gamma power is similar at the entire range each macro contact activity and the pattern of activity in or is ae ff cted by the lower (25–35  Hz) or higher range the amygdala. Figure 12 depicts a subset of gamma activity (65–85 Hz). To that aim we have calculated for each sub- from contacts that were correlated with the activity of the ject the fraction of reduction in power between BL and amygdala during the task. Table 2. summarizes the signif- the following NF block ((NF power–BL power)/BL power) icant Pearson correlations (p < 0.05); it is noticeable that in the lower and higher gamma range. Figure 9 depicts only a small fraction of the contacts was correlated with the average fraction of reduction in power in each of the the NF probe (11.5%), and that the structures correlated frequency ranges for both the right and left amygdala of with it are all in the temporal lobe, which has functional both subjects. Asterisks represent a significant difference connectivity to the amygdala. Thus, the volitional control between the bands (paired t-test, p<0.05). It shows that of the amygdala gamma band activity was specific to cor - in most cases the reduction in power between BL and NF tical circuits that are functionally related to the amygdala. is more prominent at the higher frequency range used in Here we have presented preliminary results demon- the paradigm (65–85 Hz). strating that it is possible to train subjects to directly and Figure 10 depicts the trace of the power in the common specifically modulate their amygdala's activity using depth division of the EEG into frequency bands – delta (0–4 Hz), electrodes as a signal acquisition device. Results obtained theta (4–8 Hz), alpha (8–16 Hz), beta (16–25 Hz), (gamma are encouraging, in particular the relatively high success 25–35 Hz and 65–85 Hz) and high gamma (85–200 Hz) – rate considering minimal training received by subjects. in the two target macro contacts: the right and the left amygdala from subject D004. The black line is the instan- taneous power calculated every 1 s (see Methods) and the Conclusions red trace is the average power in a certain block (BL and We have presented results showing the successful use of a NF interchangeably). It demonstrates that the change in depth electrode BCI by two patients. For the foreseeable power we observed in the gamma band is restricted to future, and considering the risks and side effects of depth the target band or to higher frequency bands and, to a electrodes, these systems will be restricted to patients who much lesser extent, to lower frequency bands. To quan- have received a therapeutic indication. Their use as BCI tify it, we have calculated the correlation between the will be mostly a side effect, which distinguishes them from average gamma power and the average power in all other BRAIN–COMPUTER INTERFACES 213 [3] Jacobs J, Kahana MJ. Direct brain recordings fuel BCI use to assist patients with motor impairment. From advances in cognitive electrophysiology. Trends Cogn an interface technology perspective, the type of system Sci. 2010;14(4):162–171. we have presented has the potential to favor engagement [4] Schalk G, Leuthardt EC. Brain-computer interfaces using in therapeutic testing phases of the iEEG, which was the electrocorticographic signals. IEEE Rev Biomed Eng. initial rationale for its development. However, following 2011;4:140–154. previous reports [6], we can speculate that the use of [5] Engel AK, Moll CK, Fried I, et al. Invasive recordings from the human brain: clinical insights and beyond. Nat BCI in a gaming context could have a positive effect on Rev Neurosci.. 2005;6(1):35–47. patients’ morale and self-esteem, and potentially improve [6] Breshears JD, Gaona CM, Roland JL, et al. Decoding their acceptance of the equipment. motor signals from the pediatric cortex: implications Overall, we have shown that it is possible to harness for brain-computer interfaces in children. Pediatrics. the power of depth electrodes to readily integrate them as 2001;128(1):160–168. [7] Schalk G, Miller K, Anderson N, et al. Two-dimensional BCI in a typical interactive application. In addition to the movement control using electrocorticographic signals in potential of VR or gaming interfaces for patient engage- humans. J. Neural Eng.. 2008;5(1):75. ment, we have also reported some emergent properties [8] Krusienski D, Shih J. Control of a brain–computer interface of these interfaces in terms of input signal filtering. The using stereotactic depth electrodes in and adjacent to the dynamic properties of the interface related to the timing hippocampus. J. Neural Eng.. 2011;8(2):025006. of realistic animations result in a low pass filtering that [9] Shih JJ, Krusienski DJ. Signals from intraventricular depth electrodes can control a brain–computer interface. may play a role in facilitating subjects’ adaptation to the J Neurosci Methods. 2012;203(2):311–314. neurofeedback task. Such a feature could prove of a spe- [10] Lachaux J-P, Jerbi K, Bertrand O, et al. BrainTV: a novel cific interest whenever options for neurofeedback training approach for online mapping of human brain functions. sessions are limited, as is the case with depth electrodes. Biol. Res.. 2007;40(4):401–413. Despite the very minimal user sample, the fact that sig- [11] Jerbi K, Ossandón T, Hamame CM, et al. Task- related gamma-band dynamics from an intracerebral nal behavior was consistent with published observations perspective: Review and implications for surface EEG would encourage us to think that this approach can be and MEG. Hum. Brain Mapp.. 2009;30(6):1758–1771. reproduced. [12] Jerbi K, Freyermuth S, Minotti L, et al. Watching brain TV and playing brain ball: Exploring novel BCI strategies using real-time analysis of human intracranial data. Int Disclosure statement Rev Neurobiol. 2009;86:159–168. No potential conflict of interest was reported by the authors. [13] Kahane P, Minotti L, Hoffmann D, et al. Invasive EEG in the definition of the seizure onset zone: depth electrodes. Handb Clin Neurol. 2003;3:109–133. Funding [14] Ghosh S, Laxmi TR, Chattarji S. Functional connectivity from the amygdala to the hippocampus grows stronger This work was supported by the European Union’s Seventh aer s ft tress. J Neurosci. 2013;33(17):7234–7244. Framework Programme for research, technological develop- [15] Oya H, Kawasaki H, Howard MA, et al. ment and demonstration [grant number 602186] and by the Electrophysiological responses in the human amygdala Ministry of Science, Technology and Space [grant number discriminate emotion categories of complex visual 3-11170] and by the Sagol family fund. stimuli. J Neurosci. 2002;22(21):9502–9512. [16] Sato W, Kochiyama T, Uono S, et al. Rapid amygdala gamma oscillations in response to fearful facial ORCID expressions. Neuropsychologia. 2011;49(4):612–617. [17] Gaona CM, Sharma M, Freudenburg ZV, et al. Gal Raz   http://orcid.org/0000-0001-9032-8613 Nonuniform high-gamma (60–500  Hz) power changes Marc Cavazza   http://orcid.org/0000-0001-6113-9696 dissociate cognitive task and anatomy in human cortex. J. Neurosci. 2011;31(6):2091–2100. References [18] Fries P. Neuronal gamma-band synchronization as a fundamental process in cortical computation. Annu Rev [1] McFarland DJ, Wolpaw JR. Brain-computer interfaces Neurosci. 2009;32:209–224. for communication and control. Commun ACM. [19] Fried I, Wilson CL, Maidment NT, et al. Cerebral 2011;54(5):60–66. microdialysis combined with single-neuron and [2] Jayakar P, Gotman J, Harvey AS, et al. Diagnostic utility electroencephalographic recording in neurosurgical of invasive EEG for epilepsy surgery: indications, patients: technical note. J Neurosurg. 1999;91(4):697– modalities, and techniques. Epilepsia. 2016;57(11):1735–

Journal

Brain-Computer InterfacesTaylor & Francis

Published: Oct 2, 2017

Keywords: Brain–computer interface (BCI); neurofeedback (NF); electroencephalogram (EEG); intracranial depth electrodes; Application development and evaluation; neurosurgical approaches and methods, affective computing; signal acquisition: EEG (other)

References