Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Parikshat Sirpal; Rafat Damseh; Ke Peng; Dang Khoa Nguyen; Frédéric Lesage

doi:10.1007/s12021-021-09538-3

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Sirpal, Parikshat; Damseh, Rafat; Peng, Ke; Nguyen, Dang Khoa; Lesage, Frédéric 2022-07-01 00:00:00 In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model’s fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals. Keywords EEG-fNIRS · Functional brain imaging · Deep neural networks · Epilepsy · Resting state · Functional connectivity · Neurovascular coupling Introduction on the surface of the subject’s head. Infrared light emitted from the light source is absorbed or scattered as it enters Functional near infrared spectroscopy (fNIRS) is a non- cerebral tissue. Detected light is used to calculate the blood invasive, mobile, and cost-effective neuroimaging tech - oxygenation changes associated with cerebral hemodynamic nology that uses near infrared light to continually monitor activity using the modified Beer-Lambert law (Kocsis et al., changes in cerebral hemodynamic parameters (i.e., oxygen- 2006; Scholkmann et al., 2014). Concentration changes in ated (HbO) and deoxygenated hemoglobin (HbR), and total the oxygenation of hemoglobin quantifies the absorption of hemoglobin (HbT)) (Jobsis, 1977). The fNIRS method relies infrared light by the brain. on the neurovascular coupling phenomenon which describes The fNIRS method offers several advantages as an alter - the intimate spatial and temporal relationship between neu- native or complement to other functional imaging techniques ral activity and cerebral blood flow to map acute functional (i.e., fMRI) (Strangman et al., 2002). fNIRS offers increased changes in the brain (Girouard & Iadecola, 2006). In a typi- temporal resolution as compared to fMRI, and fNIRS hard- cal fNIRS setup, optodes corresponding to near-infrared ware can be integrated with other modalities such as scalp light sources and their complimentary detectors are placed electroencephalography (EEG) (Fazli et al., 2012; Khan & Hong, 2017; Miller, 2012). fNIRS signals have been recently used in studying brain state decoding as well as proven use- * Parikshat Sirpal ful for brain computer interfacing over the last decade (Hong parikshat.sirpal@polymtl.ca et al., 2015; Khan & Hong, 2015). École Polytechnique de Montréal, Université de Montréal, Scalp EEG technology is the clinical gold standard for stud- C.P. 6079, Succ. Centre-Ville, Montréal H3C 3A7, Canada ying the human brain (Müller-Putz, 2020) and EEG recordings Neurology Division, Centre Hospitalier de L’Université de can be classified into specific frequency bands: alpha, beta, Montréal (CHUM), 1000 Saint-Denis, Montréal H2X 0C1, delta, gamma, and theta (Cho et al., 2014; Freeman et al., 2003; Canada Pedregosa et al., 2011; Zhao et al., 2018). The delta frequency Research Centre, Montréal Heart Institute, Montréal, Canada Vol.:(0123456789) 1 3 538 Neuroinformatics (2022) 20:537–558 range encompasses low frequencies with relatively high ampli- neural activity believed to reflect its functional organization tude and slow waveforms ranging from 0.25–3.0 Hz. Delta (Rojas et al., 2018; Tracy & Doucet, 2015). The interde- frequencies are common in normal sleep and may inciden- pendence of each component (i.e., neural and vascular) is a tally appear with focal lesions, metabolic encephalopathy, or topic of interest to the wider clinical and neuroscience com- hydrocephalus (Amzica & Steriade, 1998; Hofle et al., 1997; munity. fMRI studies have shown that resting state networks Knyazev, 2012). The theta band includes frequencies between in the epileptic brain undergo changes in their functional 4 and 7 Hz. While normal in young individuals, the theta fre- architecture (Luo et al., 2011; Wang et al., 2011). Increas- quency envelope is interpreted as slow activity in awake adults ingly, “task-free” resting state conditions in fMRI studies (Mantini et al., 2007; Pizzo et al., 2016; Sitnikova et al., 2016). have been conducted with the assumption that functionally As with delta waves, theta waves may be seen in focal lesions connected brain networks show similar profiles of activity or in a more generalized distribution in diffuse neurological over time (De Luca et al., 2006; He & Liu, 2008; Niu & He, disorders. Alpha frequencies are between 8 and 13 Hz, rep- 2014; Palva et al., 2010; Richardson, 2012; Shen, 2015). resenting the dominant rhythm in awake adults (Koch et al., In the context of epilepsy, resting state fMRI studies have 2008; Sigala et al., 2014). Beta activity ranges in frequency shown that functional networks are abnormal (Bettus et al., between 14–30 Hz and is usually observed in a bilaterally fron- 2009; Honda et al., 2021; Tracy & Doucet, 2015; Zhang et al., tal symmetrical distribution (Canolty et al., 2006; Freeman 2009, 2010a, b). Pre-clinical studies have proposed that there et al., 2003; Merker, 2016). Higher frequency ranges represent is a correlation between slow fluctuations in the resting state gamma wave oscillations between 30–100 Hz. Gamma activ- BOLD signal (~0.1 Hz) and slow fluctuations in neuronal fir - ity is seen during a wide range of activities, and is enhanced ing rates in gamma band local field potentials (Richardson, in rapid eye movement during sleep (Gross & Gotman, 1999; 2012; Shmuel & Leopold, 2008; Zhang et al., 2020). This Hughes, 2008). suggests that the resting state is related to physiologically Multimodal EEG-fNIRS experimental setups record the active dynamic neuronal processes. Utilizing fNIRS signals spatiotemporal dynamics of brain activity, provide opportu- for resting state functional connectivity has gained attention nities to observe the population dynamics of neural ensem- as a promising imaging tool to study brain function and pro- bles and offer increased benefit in fundamental and clinical vide valuable insight into the intrinsic networks present within analyses (Goldman et al., 2002; Laufs et al., 2003; Martinez- the human epileptic brain (Fishburn et al., 2014; Geng et al., Montes et al., 2004; McKenna et al., 1994; Salek-Haddadi 2017; Niu & He, 2014; Wang et al., 2017). et al., 2003). In such setups, scalp EEG measures the brain’s In this study, we hypothesize that we can predict brain electrical activity, and fNIRS signals encode the brain’s hemodynamics from electrical signals using a deep learn- hemodynamic response (Chiarelli et al., 2017; Ogawa et al., ing architecture from resting state multimodal EEG-fNIRS 1992), with a delay of approximately 3 seconds post neural recordings collected from a cohort of 40 epileptic patients. activity. Data from EEG-fNIRS setups have established causality Following which, we hypothesize that functional connectivity between neuronal firing and changes in HbO, HbR, and HbT, patterns derived from higher EEG frequency envelopes are reflecting electrical and hemodynamic fluctuations dictated increased as compared to lower EEG frequency envelopes. by neurovascular coupling (Hughes, 2008; Logothetis et al., 2001; Mukamel et al., 2005; Singh, 2012). Recent interest has focused on determining spatial hemodynamic correlates from Methods EEG recorded activity, particularly, in the blood oxygen level dependent signal (BOLD) (Czisch et al., 2004; Lemieux et al., Subjects and Protocol 2001; Lövblad et al., 1999). Resting state studies have suc- cessfully demonstrated that low frequency EEG band signals Forty patients (27 males, 13 females; ranging in age of 11 are negatively correlated with modulations in the BOLD sig- to 62 years in age; mean age of 32.42 years, and standard nal, particularly, infra-low gamma EEG band envelopes (Jia deviation of 13.97 years) with refractory focal epilepsy were & Kohn, 2011; Niessing et al., 2005; Sumiyoshi et al., 2012). recruited for prolonged EEG-fNIRS recordings. Epilepsy The characterization of the relationship between electro- diagnosis and epileptic focus localization was based on a physiology and cerebral hemodynamics is clinically relevant comprehensive evaluation which included clinical history, in epilepsy. Seizures are self-terminating paroxysmal rep- video-EEG recording of interictal spikes and seizures, mag- resentations of aberrant brain activity (Moshé et al., 2015). netic resonance imaging (MRI), positron emission tomogra- It is believed that the neurovascular machinery causing phy (PET) and for some patients ictal single photon emission seizures is similarly present in the brain interictally during computed tomography (SPECT) and magnetoencephalog- normal function, suggesting to some extent that epilepsy raphy (MEG) scans. Full details regarding patient profiles is a dynamic disorder (Kobayashi et al., 2006; Richardson, including age, gender, EEG and MRI findings are found in 2012). The resting epileptic brain displays spontaneous Table 1 of (Peng et al., 2014; Sirpal et al., 2019). A subset 1 3 Neuroinformatics (2022) 20:537–558 539 Table 1 Detailed overview of the proposed convolutional neural net- lutions have kernels of size (1,2), and thus their effect is along the work long-short term autoencoder (CNN-LSTM AE) model. The time dimension. Convolutions help in generating embeddings with network receives as input resting state EEG time-series sequences, higher level abstraction of the input EEG sequence. Deconvolutions represented as a single matrix, and is trained to reconstruct the corre- reconstruct the fNIRS sequence at full resolution based on output sponding fNIRS resting state output. Model specifications and hyper - embeddings. The decoder and encoder LSTM units have ReLU (Rec- parameters were heuristically determined. Convolutions and deconvo- tified linear units) activations Layer Description Output size Input EEG sample sequence (EEG sequence length, number of time points is 500, number of EEG channels is 21) EEG Sequence Embedding 2-Dimensional convolution + Average Pooling + 2Dconvolution ∶ stride =(1,2); (EEG sequence length, Dropout kernelsize =(1,7); 125, number of Features Maps) ReLUactivation. Dropout ∶ 20%. AveragePoolingkernel ∶(1,2) 2-Dimensional convolution + Average Pooling + 2Dconvolution ∶ stride =(1,2); (EEG sequence length, Dropout kernelsize =(1,7); 62, number of Features Maps) ReLUactivation. Dropout ∶ 20%. AveragePoolingkernel ∶(1,2) Reshape Reshape into an elongated tensor (EEG sequence length, 62 * number of Features Maps) Encoder LSTM 1 + Dropout An LSTM layer with number of cells equal to the number of (EEG sequence length, 512) EEG sequence length. ReLU activation. Dropout: 20% LSTM 2 + Dropout An LSTM layer with number of cells equal to the number of (1, 256) EEG sequence length. ReLU activation. Dropout: 20% Decoder Repeat Create repeated version of the latent vector (fNIRS sequence length, 256) LSTM 3 + Dropout An LSTM layer with number of cells equal to the number of (fNIRS sequence length, 312) EEG sequence length. ReLU activation. Dropout: 20% LSTM 4 + Dropout An LSTM layer with number of cells equal to the number of (fNIRS sequence length, 695) EEG sequence length. ReLU activation. Dropout: 20% fNIRS Sequence Reconstruction Reshape Reshape into a 2D tensor (fNIRS sequence length, 5, number of Feature Maps 3) Deconvolution1 + Dropout 2Ddeconvolution ∶ stride =(1,2);ReLUactivation; (fNIRS sequence length, 10, kernelsize =(1,2).Dropout ∶ 20%. number of Feature Maps 4) Deconvolution2 + Dropout 2Ddeconvolution ∶ stride =(1,2);ReLUactivation; (fNIRS sequence length, 20, kernelsize =(1,2).Dropout ∶ 20%. number of fNIRS channels) of patients had MRI evidence of encephalomalacia, cortical noise and lights were maintained at a minimum for patient com- dysplasia, and/or hippocampal atrophy, a common finding in fort during data acquisition. Patients were further instructed epilepsy, but this was neither an inclusion nor exclusion cri- to remain calm and placed in comfortable, climate-controlled terion. The presence of such findings (Dhamija et al., 2011; ambulatory suites with curtains drawn to limit ambient light. Woermann & Vollmar, 2009) is a common MRI finding in Patients were continually telemonitored by trained clinical epileptic brains. The institutional review boards of Sainte- staff. fNIRS data was collected using the Imagent Tissue Justine Hospital and Centre Hospitalier de l’Université de Oximeter system (ISS Inc.), a multi-channel frequency Montréal approved the study. domain system recording at 19.5 Hz with wavelengths of 690 nm and 830 nm for sensitivity to HbR and HbO respec- EEG‑fNIRS Data Acquisition and Pre‑Processing tively. EEG data was recorded according to the standard 10–20 system using 21 electrodes (in positions Fp1, Fp2, Continuous EEG-fNIRS recordings were performed at the F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, Optical Imaging Laboratory of Sainte-Justine Hospital in P8, O1, O2) at 500 Hz (Neuroscan Synamps 2TM system). Montréal, Canada. Experimental protocol ensured ambient Custom-made helmets, taking into consideration different 1 3 540 Neuroinformatics (2022) 20:537–558 head sizes and shapes were made to fit comfortably. Plastic signal to noise ratio (SNR) threshold applied in channel and polyvinyl chloride manufacturing materials made them analysis was defined as those channels less than 30% of the rigid and light. The helmets were equipped with a total of mean SNR of all channels. fNIRS channels deemed to have 64 light sources, 16 light detectors and 19 EEG electrodes SNR were eliminated and not included for analysis. This that allowed for stable optical coupling between cortical led to an average of 138 channels per patient. Changes in regions and the scalp. This further prevented inter-optode HbO and HbR were calculated via the HomER and MNE shifting and movement artifacts to a large extent. Sensitiv- software packages (Gramfort et al., 2014; Huppert et al., ity of near-infrared light to cortical tissue was maintained 2009). by positioning the optodes approximately 3–4 cm apart. Multiple consecutive recordings were performed, with Electrodes were placed following the 10–20 EEG instru- each recording approximately 15 min led to a compendium mentation standard, allowing for full head coverage (Barlow of 200 recordings totaling 50 hours of recording time. Data was et al., 1974). Figure 1 shows the EEG-fNIRS helmet placed bandpass filtered in the 0.01 to 0.1 Hz frequency range to on a patient’s head. be in the resting state range (Tong et al., 2012). The resting EEG data was bandpass filtered between 0.1–100 Hz state period (indexed from patients when they were resting to remove instrumental noise and to remove drift related comfortably) ranged between 7 to 10 minutes with a mean of to physiological activity, particularly, higher frequencies. 8.35 minutes (Geng et al., 2017; Li et al., 2015; Zhang et al., The unprocessed raw time series of the HbO and HbR 2010a, b). To correct for motion, we performed dimension signals was bandpass filtered to remove specific frequency reduction via principal component analysis on EEG-fNIRS components attributed to cardiac (approximately 1 Hz) data and removed components with the most variance. Fur- and/or respiratory activity (approximately 0.2–0.3 Hz) ther, artifact rejection with (10% variation from normalized (Gramfort et al., 2014; Lu et al., 2010; Peng et al., 2014). intensity) was applied to remove additional motion artifacts. Signal fidelity was examined prior to analysis by channel- Artifact-free data points were then filtered for the effects of wise verification of signal intensity. Bandpass filtering respiratory and cardiac signal with a cutoff frequency of was applied to EEG data to compute frequency bands of 0.2 Hz. Finally, HbO concentrations were calculated for each interest. We used a FIR bandpass filter and the lowcut and channel using the modified Beer–Lambert law. highcut values (Hz) for the delta, theta, alpha, beta, and Structural MRI registration of optode and electrode posi- gamma frequencies were set as: [1, 4], [4, 8], [8, 12], [12, tion was done using neuro-navigation (Brainsight, Rogue- 30], [30, 100] respectively (Gramfort et al., 2014). The Research Inc.). Channel positions were cross-referenced Fig. 1 EEG and fNIRS channel-configuration and custom-made mul - channel configuration, as well as the EEG (which are in blue dots), timodal EEG-fNIRS helmets used for EEG and fNIRS data acquisi- are superimposed on the patient’s MRI. We used a 3D camera and tion. Helmets of different sizes and shapes to fit patients’ head com - stereotaxic system (Frameless 39 from Rogue research) to determine fortably were made from plastic and polyvinyl chloride making them the 3-D coordinates of the optodes relative to the patient’s anatomical rigid and light. The EEG-fNIRS configuration allows for full head MRI coverage and follows the 10–20 EEG placement system. The fNIRS 1 3 Neuroinformatics (2022) 20:537–558 541 with the patient MRI and adapted to ensure coverage of the receives the feature vector and decodes it into the original epileptic focus, the contralateral homologous region, and as input sequences. LSTM-AEs learn a compressed represen- much area as possible of other brain regions. The MRI was tation of sequential data and have been used in video, text, segmented into six different layers: air, scalp, skull, CSF, audio, and time series sequence data (Lipton et al., 2016; gray matter and white matter. The gray matter layer was Srivastava et al., 2015; Wang et al., 2016). In this work, used to extract six two-dimensional cortical projections. The multiple LSTM layers were incorporated to learn tempo- three-dimensional position of each channel was projected ral representations. Our model also includes convolutional onto these two-dimensional topographic maps, of which the layers to extract high level spatial percepts from channel following views were considered: dorsal, frontal, left and combinations. We input EEG sequential data accounting right views. for hemodynamic delays to perform sequence-to-sequence encoding (Luong et al., 2015; Truong et al., 2018; Vincent Neural Network Architecture et al., 2008; Zhang, 2018). These input EEG sequences are convolved by two convolutional neural networks (CNN) and We built a deep sequence-to-sequence multimodal autoen- subsequently fed into the first two encoding long short-term coder to predict fNIRS signals from input scalp EEG signals. memory (LSTM) modules. EEG data samples are projected Autoencoders are powerful machine learning models trained in the latent space with fixed length vectors that provide in a self-supervised fashion to reconstruct inputs by learning more compressed representations, which are then used to their abstract representations (Kocsis et al., 2006; Lindauer decode and reconstruct the output fNIRS data, by the LSTM et al., 2010; Socher et al., 2011; Vincent et al., 2010). The decoding modules. autoencoder embedded signals in a low dimensional latent After testing multiple architectures with exhaustive space, where both the encoder and decoder are formulated hyper-parameter optimization, we designed our model as as deep neural networks. follows: The encoder is comprised of LSTM layers preceded Recurrent neural networks (RNN) have been widely used by convolutional blocks. Convolutions in each block have a in time series modeling since they account for the tem- kernel size of (1, 7) and stride size of (1, 2). The decoder poral state within data (Baytas et al., 2017; Chung et al., is comprised of LSTM layers which manipulate the vec- 2016; Merity et al., 2018; Mikolov et al., 2010). The output tors in the latent space to provide a final output dimension depends on hidden states and feedback connections present equal to that of an fNIRS sample. We evaluated our model within hidden units. Previous states can be used as inputs, in terms of cross-modal reconstruction error (Zhao et al., thereby allowing RNNs to hold memory. In our model, we 2018), denoted as RE. The objective is to simultaneously used backpropagation through time, a common gradient minimize the distance between fNIRS data samples and descent type training technique (Sutskever et al., 2014). The maximize the distance between each fNIRS and EEG data innate problem of RNN gradient based training is that deriv- points (i.e., minimizing the RE is equivalent to maximizing atives propagated via recurrent connections either become the likelihood function). Once the model was trained, the exceedingly small or large (Goodfellow et al., 2016; Luong corresponding RE was calculated on an independent testing et al., 2015), causing a vanishing or exploding gradient subset (see below) by computing the sum of the Euclidean respectively. Long short-term memory units (LSTM), a vari- distance between x and its corresponding reconstruction, x , ant of the vanilla RNN architecture overcomes the vanishing over all L dimensions, as expressed in Eq. 1 below: gradient problem (Greff et al., 2017; Gregor et al., 2015; Lecun et al., 2015). LSTM units receive external inputs and ∈ = x − x , t T (1) t t,l t,l generate hidden outputs via input, output, and forget gates l=1 and a memory cell. The gates and memory cell are internally Model output is denoted as x . connected with weighted links. The gates are connected with EEG data is processed as follows. First, matching EEG external sources, which are current state sequential inputs and fNIRS data are parsed from our data directory, following and previous hidden states. This prevents the LSTM from which the respective data (EEG or fNIRS) is labeled accord- storing useless or noisy input information (Greff et al., 2017; ing to the resting state periods. Feature scaling is performed Gregor et al., 2015; Lecun et al., 2015). using the MinMaxScaler class (Pedregosa et al., 2011) on The LSTM autoencoder model (LSTM-AE) as proposed EEG input data which sets the range of values between 0 and by Srivastava et al. consists of encoder LSTM units and 1. Input signals are mean centered prior to being fed into the decoder LSTM units (Srivastava et al., 2015). The encoder model. Then, data is fed into the convolutional layers and LSTM receives input sequences and encodes them into a travels to the LSTM and deconvolution modules. A detailed feature vector as the LSTM generates hidden outputs (Lipton schematic view of our model is shown in Fig. 2 below. et al., 2016; Wang et al., 2016). Likewise, the decoder LSTM 1 3 542 Neuroinformatics (2022) 20:537–558 Backpropagation through time was used with a learning Training Details rate of 0.05, batch size of 60 and 50 epochs, all of which were heuristically determined. The model was designed to use patient specific EEG signals • Each fNIRS signal generated corresponds to an EEG as input to decode fNIRS signals. For each patient, the data sequence input. An element in the EEG sequence corre- was randomly split into training, validation, and testing sub- sponds to 1 second of recording with 500 time points (sam- sets, with a proportion of 60% training, 20% testing and 20% pling frequency is 500 Hz) for each EEG channel. Data respectively. We experimented with various model depths and batches were generated for sequence processing by using determined deep LSTMs to outperform shallow LSTMs. This the utility class for batch generation in the Keras frame- is likely due to the larger hidden state which occurs because of work. Briefly, this class uses as input a sequence of data increasing layers. Complete training details are given below. points to produce batches for training and validation. Data points outside of the start and end indices of rest- • We initialized the LSTM’s parameters with the uniform ing state periods (as marked in our ground truth) are not distribution between 0 and 1. This was done to counter- used in the output sequences. The final EEG data used as act the exploding gradients problem intrinsic to LSTMs, input is two dimensional, i.e., [data points, channels]. thereby enforcing a hard constraint on the norm of the gradient by scaling it between 0 and 1. Simultaneously, To summarize, the model was trained as follows: (a) we we specified starting node values for the LSTM computa - designed LSTM layers with corresponding LSTM cells (b) tions by preparing a feed dictionary which has input EEG model parameters were uniformly initialized in the range data and a target label. It is important to note that the between [0,1], (c) dropout was applied with value of 0.2, LSTM can learn how to map input sequences as model and average pooling was applied to reduce the probability training is patient specic fi into a x fi ed dimensional vector of model overfitting, (d) we used backpropagation through representation and can learn temporal dependencies. Fig. 2 Multimodal EEG to fNIRS reconstruction using our patient a 4-D tensor with shape: (samples per batch, sequence length, time specific sequence to sequence LSTM autoencoder model. Given points, and channels). The model has encoder and decoder compart- EEG input data into the encoder, the model decodes and reconstructs ments, each with 2 LSTM layers, determined heuristically. Table 1 fNIRS output. After data collection and resting state segment annota- below provides details of the model tion, data processing and model development, the data is finalized as 1 3 Neuroinformatics (2022) 20:537–558 543 time with a learning rate of 0.05, (e) we used a batch size of level of signal fidelity. That is, signals that were ± 2 stand- 60 and 50 training epochs for each patient. ard deviations of the mean and displayed low SNR (i.e., signal amplitude less than 30% of mean signal amplitude) were removed from analysis. We then computed the Pearson Model Validation product-moment correlation coefficients between the experi - mental fNIRS timeseries of the seed channel and the experi- After training and saving our model’s weights, we validated mental fNIRS timeseries of all other channels. Subsequently, the model’s intra-patient predictive capacity by using indi- the Pearson product-moment correlations (and corresponding vidual EEG recordings as input to predict fNIRS signals. Fisher z-scores) were computed between the experimental This was possible since our dataset contains multiple record- seed channel timeseries and our model’s predicted fNIRS ings from each patient. To diagnose performance, we plotted timeseries for all other channels. The two sets of correla- learning curves to ensure we did not overfit during training. tion coefficients were respectively projected to an MRI head As an illustrative example, Fig. 3 shows the learning curves template based on the three dimensional coordinates of the for patients 1, 4, and 23. corresponding channels using Atlasviewer (Aasted et al., 2015). The connectivity value at each voxel of the cortex Model Predictions was obtained from the correlation coefficients of all channels with a weighted-average method using the reciprocal of the The model predicts signals by appending ‘output state’, and cube of the distance from the voxel to each fNIRS channel. In order to quantitively evaluate and compare the results ‘output prediction’ matrices. LSTM cells are connected recur- rently to each other. Decoder inputs are two-dimensional of our functional connectivity studies, we computed the root mean square error (Eq. 2) i.e., the standard deviation of the matrices which are passed into decoder LSTM layers. fNIRS data is shifted one sequence ahead to hold data in LSTM residuals between functional connectivity values in experi- mental fNIRS and reconstructed fNIRS time courses derived memory and finally decoder outputs are returned due to the data passing through the deconvolution layers. from full spectrum EEG and specific EEG frequency band signals for all patients in our cohort. Functional Connectivity Validation � � � �∑ fc − f i=c i c (2) We chose the seed channel from a region of interest, defined RMSE = FC to be a region which had adequate optode coverage con- firmed by our source/detector montage and an acceptable Fig. 3 Learning curves are generated for the training and validation gap between the two final loss values. We note that the validation loss sets. The training and validation loss decrease to a point of stability decreases to a point of stability and has a small gap with the training with a minimal gap between the two final loss values. We note that loss, mean squared error (MSE) the validation loss decreases to a point of stability with a minimal 1 3 544 Neuroinformatics (2022) 20:537–558 where “C” is the number of channels per functional con- across the brain. Channel locations were chosen if they nectivity analysis, fc is the connectivity value of experimen- offered coverage of most of the brain within the constraints tal fNIRS and f is the connectivity value of model fNIRS of the source/detector montage and had an acceptable level reconstructions. of signal fidelity as indicated in “ EEG-fNIRS Data Acquisi- tion and Pre-Processing” section. As an illustrative example, Fig. 8 shows the model’s spatial predictions for patient 10. Results EEG Frequency Decomposition and Resting State Predictions This section describes the reconstruction results obtained using full spectrum EEG and subsequently EEG frequency After model training and validation, we computed EEG fre- ranges as model input. Intra-patient reconstructions are quency bands, namely: delta [0.5–3 Hz], theta [4–7 Hz], alpha also presented; we explore spatial reconstruction, resting [8–13 Hz], beta [14–30 Hz], and gamma [30–100 Hz]. To state predictions, and functional connectivity. ensure the presence of appropriate power in the frequency ranges, the spectral power of EEG signals was obtained using the Welch’s power spectral density function. Welch's method Full Spectrum EEG Performance and Feature was preferred over other methods (i.e., standard periodogram Analysis spectrum estimation and Bartlett's method) as Welch’s method offsets a reduced frequency resolution with a reduc - Resting state full spectrum EEG signals from all channels tion in signal noise in the estimated power spectra in exchange were input in the model. To decode fNIRS channels from for reducing the frequency resolution (Welch, 1967). The encoded EEG channels, the model’s decoder layers used the Welch method partitions the signal into overlapping segments encoder’s latent state as input as data traveled through LSTM thereby mitigating the loss of edge data. The overlapped data units. Figure 4 below quantifies performance on selected segments are then windowed in the time domain. Subsequent individual patients with full spectrum EEG signals as input. computation includes the discrete Fourier transform, followed Figure 5 provides the group estimate of reconstruction error by averaging the periodograms leading to a final nxm array for all patients given scalp full spectrum EEG recordings. representing power measurements by frequency bins. All computations (including Fourier decomposition, Welch’s power spectral density) were performed using the Intra‑patient Reconstructions on Separate Recording MNE software package (Gramfort et al., 2014). Figure 9 Sessions shows the model’s predictions from EEG frequency ranges input using patient 10 (fNIRS channel 10). Here, we report results on intra-patient fNIRS reconstruc- We calculated decoded fNIRS reconstruction error met- tions provided EEG resting state as input. Specifically, we rics, as shown in Fig. 10, for each EEG frequency range hypothesized that our model when trained with a patient’s and calculated patient wise reconstruction error. The gamma single recording was able to reconstruct fNIRS signals from and beta frequency bands demonstrated the lowest error a subsequent recording. To examine our model’s predictive rates and in the lower EEG frequency ranges, we noticed capacity and to cross-validate our model, we first trained increased fNIRS reconstruction error, possibly owing to the our network on a patient’s single recording. Next, we used fact that our model was possibly not able to learn appropriate our trained network and aimed to reconstruct fNIRS signals features to reconstruct fNIRS signals. from a subsequent recording from the same patient. The data To further determine which EEG frequency band can was partitioned into training, testing, and validation subsets reconstruct fNIRS signals with the lowest reconstruction in a 60/20/20 manner. This was done for all data across all error on average, we calculated band wise reconstruction patients and recordings. Figure 6 displays the group results error for all patients, as shown in Fig. 11. Following which, for intra-patient fNIRS signal reconstructions and Fig. 7 dis- we conducted one-tailed paired t-tests to test whether there plays the fNIRS reconstructions for channel 5 from patient is a statistical difference in reconstruction error between 10 across recordings 1, 3, 4. any two of the five bands when compared to gamma in the following combinations: [delta, gamma], [theta, gamma], Spatial Variability of Reconstructions [alpha, gamma], and [beta, gamma]. Bonferroni correction was then applied to control the family-wise error rate to be We then explored the model’s predictions sensitivity to less than 0.05. The gamma frequency band reconstructs channel location on the head. The topographic robustness of fNIRS signals with increased fidelity on average as com - the model suggests the predictions are reasonably invariant pared to other frequency bands. 1 3 Neuroinformatics (2022) 20:537–558 545 Fig. 4 Decoded predictions of hemodynamic signals from cerebral reconstructed with the lowest reconstruction error, RE, while patient electrical activity. Full spectrum EEG signals from all channels were 10 had the highest. The data has been mean centered and baseline is used as input. fNIRS HbO reconstructions are shown from 3 patients near zero, 250 s is shown here to illustrate seizure free, resting state in channel 10 (Channel 10’s SNR was adequate, located on the left periods. Note that the model accounts for the delay between EEG and temporal lobe). Black and red curves correspond to experimental fNIRS and the model fNIRS predictions are indicative of this delay and reconstructed fNIRS signals respectively. Data from patient 22 1 3 546 Neuroinformatics (2022) 20:537–558 for functional connectivity computations (Fig. 13), followed by analysis using the EEG gamma frequency band as input (Fig. 14). Discussion Deep learning models obviate cumbersome and brittle fea- ture engineering processes replacing them with hierarchical feature learning. In this work, we developed a deep learning CNN-LSTM sequence-to-sequence autoencoder to predict fNIRS signals from resting state EEG signals in the epi- Fig. 5 Full spectrum EEG to fNIRS reconstructions. The group esti- leptic brain. Our model was trained using a 60/20/20 split mate of full spectrum EEG signals from all channels were used as for training, testing, and validation, respectively. The results input in our network architecture, to reconstruct full fNIRS signals here demonstrate that in the context of epileptic resting state from all fNIRS channels recordings, fNIRS signals can be predicted using full spec- trum as well as specific frequency range EEG signals to a Functional Connectivity Results certain extent. We further validated our method by recon- structing the functional connectivity in the brain using the We computed functional connectivity mappings for experi- predicted fNIRS and compared it to the functional connec- tivity using experimental fNIRS. mental fNIRS and our model’s fNIRS reconstructions. We compared experimental fNIRS and fNIRS reconstructions From a neurophysiological standpoint, the resting epi- leptic brain is in a dynamic state and cerebral blood flow derived from both full spectrum EEG and the EEG gamma band for all patients. The root mean square error (stand- is in constant flux (Wang et al., 2011). Recent work has shown the presence of abnormal functional networks in ard deviation of the residuals) was used as an estimator of the error in our connectivity studies. On a group level, we the interictal state (Murta et al., 2015; Richardson, 2012). Thus, even with removal of systemic physiological com- noticed a lower error in functional connectivity analyses between experimental fNIRS and fNIRS reconstructions ponents underlying compensation by molecular and cel- lular mechanisms can possibly help predict components derived from full spectrum EEG as compared with experi- mental fNIRS and fNIRS reconstructions derived from the of systemic physiology in addition to hemodynamic brain activity (Pressl et al., 2019). Our experimental findings gamma band and are shown in Fig. 12. Figures 13, 14 show examples of the functional connec- can be related to known physiological phenomena being generated at the frequency of Mayer waves (~0.1 Hz), as tivity mapping results generated using the correlations of the timeseries from patient 22, with the seed channel chosen as these oscillations reflect fluctuation in cerebral arterial blood pressure (Nikulin et al., 2014; Schwab et al., 2009). 20. First, we used full spectrum EEG predictions as input Fig. 6 Group results for intra- patient fNIRS reconstructions. The network was trained on a patient’s single recording. Next, the network reconstructed subsequent recordings from the same patient. The data was partitioned into training, test- ing, and validation subsets in a 60/20/20 manner. This was done for all data across all patients and recordings 1 3 Neuroinformatics (2022) 20:537–558 547 Fig. 7 Intra-patient predictions of hemodynamic signals from cer- shown here from 3 such recordings. Panels “A”, “B”, and “C” show ebral electrical activity. A full spectrum resting state EEG sin- the respective reconstruction of channel 5 from recordings 1, 3, and –2. gle recording (patient 10 all channels) was used to train the model. 4 (patient 10). The reconstruction error is 2.98 × 10 Black and red After training, we saved the model weights and used as input a sub- curves correspond to experimental and reconstructed fNIRS signals sequent recording from the same patient. fNIRS reconstructions are respectively The presence of these oscillations persisting after filter - and its fNIRS correlate via the neurovascular coupling ing can be partly due to the fact that they share a com- phenomenon. mon spectral range with typical hemodynamic responses These nuanced features within the EEG signal are (Yücel et al., 2016). On the other hand, these oscillations encoded and subsequently decoded by the architectural correspond to cerebral vasomotion (i.e., extra neuronal) components of the model, particularly the convolutional and are possibly related to blood vessel tonal oscillation LSTM parameters (Greff et al., 2017; Sutskever et al., (Aalkjær et al., 2011; Julien, 2006; Quaresima & Ferrari, 2014). The model’s encoder and decoder and parameters 2019; Sassaroli et al., 2012). (e.g., the activation function) may have enhanced feature The exact mechanics of physiological signal presence extraction in resting state EEG data and its corresponding within EEG signals has not been established with cer- correlate in fNIRS signals. In addition, the features com- tainty. However, experimental results from this work sug- puted by using the outputs or hidden states of the recur- gest the following: our model can capture subtle hemody- rent units and the model may extract long-term dependen- namic dependencies within the EEG resting state signal cies (electrical and/or physiological) in resting state EEG 1 3 548 Neuroinformatics (2022) 20:537–558 −3 Fig. 8 fNIRS spatial reconstructions, patient 10. To illustrate our net- (channel 11) to 7.83x10 (channel 62), with the mean RE being −2 work’s fNIRS reconstructions spatially, signals from multiple EEG 6.52x10 for all reconstructions. Black and red curves correspond to channels are used as input, for which the locations are shown on experimental and reconstructed fNIRS signals respectively −1 the brain (blue circles). Reconstruction error ranges from 6.41x10 signals from the LSTM modules via the gating mechanism relationship between CBF and neuronal activity in the (Sutskever et al., 2014). Furthermore, when cerebral blood resting epileptic brain is theorized to be captured by the flow (CBF) varies, changes occur in both the metabolic and model used in this work. Exploring the spatial localiza- electrical activity of cortical neurons with corresponding tion of EEG frequency oscillations can help to determine EEG changes (Sassaroli et al., 2012). if the presence of physiological signals is variable across Events responsible for evoking the fNIRS response can patients and electrodes thereby possibly lending credence be divided into subthreshold synaptic and suprathreshold to the hypothesis that these oscillations are unlikely to be spiking activities (Curtin et al., 2019; Sharbrough et al., generated by a single source. 1973). Excitatory and inhibitory neurons which are often We show spatial decoding is possible using our model. located within close proximity in the brain are simulta- Examination of the LSTM memory units and the latent neously active and may contribute to the hemodynamic space architecture in autoencoders can demonstrate cor- response (Franaszczuk et al., 2003). Slower EEG frequency relation between data that were previously unknown. envelopes (i.e., delta and theta) are generated by the thala- Utilizing the architecture developed here to predict brain mus and cortical cells in layers II-VI. Faster frequencies hemodynamics, a next step would be to understand the (i.e., beta and gamma) arise from cells in layers IV and V structure of the latent variable (multidimensional vec- of the cortex (Foreman & Claassen, 2012; Merker, 2016). tor) to unpack the principal components of the fNIRS or Changes in electrical potential seen in EEG recordings EEG signal. are closely tied to cerebral blood flow (CBF) and when A second point for further investigation is to integrate normal CBF declines to approximately 25– 35 ml/100 g/ an attention mechanism in our model. Since LSTM cells min, the EEG signal first loses faster frequencies, then as can lead to ambiguous memory activations, an attention the CBF decreases to approximately 17–18 ml/100 g/min, mechanism allows for encoding input into a sequence of slower frequencies gradually increase. The interdependent vectors and from this, we can choose a subset adaptively 1 3 Neuroinformatics (2022) 20:537–558 549 Fig. 9 Resting state fNIRS predictions given EEG frequency range tal and reconstructed fNIRS signals respectively. We used a constant input, patient 10, channel 10. We obtained predicted fNIRS recon- experimental fNIRS signal for comparison. The gamma range, which structions given filtered EEG input for the following frequency bands: contains the greatest number of EEG frequencies reconstructs with Delta: 0–3 Hz; Theta: 4–7 Hz; Alpha: 8–13 Hz; Beta: 14–30 Hz; more fidelity compared to ranges with less frequency components Gamma: 30–100 Hz. Black and red curves correspond to experimen- during decoding. In this condition, the model no longer Attention implemented in our model would enable us to needs to utilize fixed length vectors thereby increasing inspect the relationship between encoded and decoded performance metrics at the cost of computational time. sequences by model weight visualization. 1 3 550 Neuroinformatics (2022) 20:537–558 Fig. 10 fNIRS reconstruc- tion error given specific EEG frequency ranges for all patients, all channels. We obtained predicted reconstruc- tions given filtered EEG input for the following frequency ranges: Delta: 0–3 Hz; Theta: 4–7 Hz; Alpha: 8–13 Hz; Beta: 14–30 Hz; Gamma: 30–100 Hz. The gamma range, which contains the greatest number of EEG frequencies, reconstructs with more fidelity and lowest reconstruction error metrics compared to ranges with less frequency components In comparison with lower frequency range EEG signals, selectivity to that of nearby neuronal activity (Jia & Kohn, results here suggest that higher frequency EEG envelopes 2011; Whittingstall & Logothetis, 2009). GABA-ergic reconstruct fNIRS signals with less error. Our results inhibitory interneuron activity is considered to be crucial corroborate that EEG gamma band based fNIRS recon- to generate EEG gamma frequency activity and this may structions show a closer fit between the observed and pre - be increased via interactions with excitatory neurons (Jia dicted hemodynamic responses as opposed to other EEG & Kohn, 2011; Park et al., 2011; Ray & Maunsell, 2010). frequency ranges (Ebisch et al., 2005; Murta et al., 2015; However, to fully interpret the impact of this activity war- Niessing et al., 2005). This is possibly because higher fre- rants an investigation into the cellular mechanisms respon- quencies engage an increased number of neurons, but it is sible for their generation. less apparent if this is attributed to baseline network activ- In the second part of our work, we explored functional ity or part of a pivotal functional role. Gamma rhythms connectivity in the resting state of the epileptic brain. in the brain provide an indication of engaged networks We hypothesized that our network’s predictions can help and have been observed in several cortical and subcorti- reveal functional connections and on a group level, pre- cal structures. These rhythms are typically stronger for dicted fNIRS from full spectrum EEG have higher con- some stimuli as compared to others, thereby displaying nectivity as compared to predictions derived from the Fig. 11 Mean fNIRS recon- struction error given specific EEG frequency ranges for all patients. The gamma range, which contains the greatest number of EEG frequencies, reconstructs with more fidelity and lowest reconstruction error metrics compared to other ranges with less frequency components 1 3 Neuroinformatics (2022) 20:537–558 551 Fig. 12 Estimator error (RMSE) for functional connectivity results signal input. The connectivity derived from the full spectrum EEG for all patients. Error for connectivity analyses between experimen- time series consistently has lower error compared to the connectivity tal fNIRS and predictions using full spectrum and gamma band EEG derived from the gamma band EEG gamma band. Experimental resting state fNIRS data of the hemodynamic response to neuron firing and the fact and predicted fNIRS data was correlated to reveal simi- that some patients are not able to undergo an fMRI scan lar connections near the set seed but metrics decreased easily (i.e., claustrophobia, paroxysmal seizure occurrence generally as distance increased from the seed. This can during scanning) (Pressl et al., 2019; Richardson, 2012). By be due to numerous factors: 1. noise causing a decrease showing the possibility of obtaining brain hemodynamic in reconstruction quality, 2. a decrease in gamma activa- data from neural signals, the results here add an additional tions at the region of interest, and 3. model parameters dimension for understanding the epileptic human brain, aid unable to completely learn the nuances present within in clinical decision making, and provide a complementary the signal. Furthermore, systemic artifacts from the scalp measure to fMRI, particularly in locations where access to and skull behave as dominant noise sources in resting fMRI technology is scarce or not possible. state fNIRS signals, leading to inaccurate reconstruction. Scalp EEG technology remains the clinical gold standard Utilizing an EEG-fNIRS experimental setup with short for the noninvasive assessment of electrical brain activity separation channels, measuring approximately 1–2 cm (Dash et al., 2017). Using EEG signals in conjunction with in spatial separation between source and detector could predicted brain hemodynamics can possibly improve clini- lead to sufficient noise reduction and improved signal cal management and ultimately patient outcomes (Connolly sensitivity (Gagnon et al., 2012; Kohno et al., 2007). We et al., 2015; Helbok & Claassen, 2013). Multimodal EEG- hypothesize that reconstruction metrics and correspond- fNIRS analysis using deep learning frameworks, as the one ing functional connectivity network measures stabilize presented in this work, can improve our understanding of with increased signal quality and resting state duration, cerebral neurovascular coupling and pathophysiology. The thereby decreasing the disparities present between experi- results from this work can be abstracted for applications mental and predicted time series. to other neurological and neuropsychiatric pathologies, The resting state epileptic brain and connectivity between such as stroke, spinal cord injuries, traumatic brain inju- brain networks is dynamic (Deco et al., 2011; McKenna ries, Alzheimer’s disease, attention-deficit hyperactivity et al., 1994). Typically, fMRI has been used for computing disorder, post-traumatic stress disorder, and dementia to functional connectivity but there are inherent limitations name a few (Fair et al., 2013; Phillips et al., 2018; Siegel of fMRI, particularly, slow dynamics, regional variability et al., 2016). Furthermore, hemodynamic predictions from 1 3 552 Neuroinformatics (2022) 20:537–558 Fig. 13 Functional connectivity results between experimental fNIRS (A, C) and predicted resting state fNIRS (B, D) are shown. A and and predicted resting state fNIRS using full frequency spectrum EEG B display the right side of the brain, C and D display the left side as input for patient 22. We employed seed based functional con- of the brain. The connectivity profiles are seen to be similar between nectivity analysis to obtain a surface brain map that describes brain the maps generated using the experimental fNIRS results and the pre- functional connectivity correlation patterns. The seed region of inter- dictions of the model. A RMSE value of 0.07 corresponds to fNIRS est (dark circle) is shown and full spectrum EEG was used as input signal reconstruction from experimental fNIRS and predicted fNIRS into the model. Bilateral brain correlations using experimental fNIRS from full frequency spectrum EEG as model input electrical brain signals can be useful in treatment strategies different treatments (Citerio et al., 2015; Le Roux, 2013). utilizing neurofeedback (i.e., neuroprosthetics, transcranial Currently, therapeutic strategies follow a ‘reactive’ model: direct current stimulation) as well as towards developing corrective actions are triggered by abnormal values in sin- precision medicine strategies (DeBettencourt et al., 2015; gle parameters (i.e., EEG signals) and a stepwise approach Dutta et al., 2015; Kotliar et al., 2017; Nicholson et al., is used with increasing therapeutic intensity. Comprehen- 2016; Ros et al., 2014; Sitaram et al., 2017; Thair et al., sive signals (i.e., EEG and predicted hemodynamics) can 2017). Predicting hemodynamics from EEG increases clin- shift this paradigm towards a ‘goal-directed’ management ical diagnostic specificity, allowing differentiation between strategy (Le Roux, 2013; Maas et al., 2012; Schmidt & De pathological conditions that may appear similar but require Georgia, 2014). 1 3 Neuroinformatics (2022) 20:537–558 553 Fig. 14 Functional connectivity results between experimental fNIRS (A, C) and predicted fNIRS (B, D) are shown. A RMSE value of and predicted fNIRS resting state using EEG gamma band as input 0.15 corresponds to fNIRS signal reconstruction from experimen- for patient 22. Correlations from experimental fNIRS and predicted tal fNIRS and predicted fNIRS from signals derived from the EEG resting state fNIRS using EEG gamma band as input into our model gamma band as model input are displayed. Bilateral brain correlations using experimental fNIRS Conclusion Information Sharing Statement We designed and implemented a deep learning model to Sharing the data used in this study is bound by the ethics predict resting state hemodynamics given specific resting of the institutional review boards of Sainte-Justine Hospital state scalp EEG frequencies from a cohort of epileptic and Centre Hospitalier de l’Université de Montréal which patients. The robust multidimensional dataset used here approved the study. The custom code used in this study is allowed us to investigate the relationship between brain available upon reasonable request. hemodynamics and neural signals via neurovascular cou- Acknowledgements This project was generously supported by The pling. Using a deep learning architecture, we performed Natural Sciences and Engineering Research Council of Canada grant a thorough analysis of each EEG frequency range and its NSERC: NSERC, 239876-and Canadian Institutes of Health Research complementary fNIRS prediction; further we analyzed grant 87183. functional connectivity between brain regions using fre- quency range predictions. We noted that higher EEG fre- Declarations quency bands provided hemodynamic predictions with the highest metrics. Conflict of Interest The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose. Open Access This article is licensed under a Creative Commons Attri- bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long 1 3 554 Neuroinformatics (2022) 20:537–558 as you give appropriate credit to the original author(s) and the source, 113–119). Lippincott Williams and Wilkins. https:// doi. org/ 10. provide a link to the Creative Commons licence, and indicate if changes 1097/ MCC. 00000 00000 000179 were made. The images or other third party material in this article are Connolly, M., Vespa, P., Pouratian, N., Gonzalez, N. R., & Hu, X. included in the article's Creative Commons licence, unless indicated (2015). Characterization of the relationship between intracra- otherwise in a credit line to the material. If material is not included in nial pressure and electroencephalographic monitoring in burst- the article's Creative Commons licence and your intended use is not suppressed patients. Neurocritical Care, 22(2), 212–220. https:// permitted by statutory regulation or exceeds the permitted use, you will doi. org/ 10. 1007/ s12028- 014- 0059-8 need to obtain permission directly from the copyright holder. To view a Curtin, A., Tong, S., Sun, J., Wang, J., Onaral, B., & Ayaz, H. (2019). A copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . systematic review of integrated functional near-infrared spectros- copy (fNIRS) and transcranial magnetic stimulation (TMS) studies. Frontiers in Neuroscience. https://doi. or g/10. 3389/ fnins. 2019. 00084 Czisch, M., Wehrle, R., Kaufmann, C., Wetter, T. C., Holsboer, F., Pollmächer, T., & Auer, D. P. (2004). Functional MRI during References sleep: BOLD signal decreases and their electrophysiological cor- relates. European Journal of Neuroscience. https:// doi. org/ 10. Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what 1111/j. 1460- 9568. 2004. 03518.x is currently thought? In Acta Physiologica. https:// doi. org/ 10. Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien- 1111/j. 1748- 1716. 2011. 02320.x Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Aasted, C. M., Yücel, M. A., Cooper, R. J., Dubb, J., Tsuzuki, D., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Becerra, L., Petkov, M. P., Borsook, D., Dan, I., & Boas, D. A. Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., (2015). Anatomical guidance for functional near-infrared spec- Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on mini- troscopy: AtlasViewer tutorial. Neurophotonics. https:// doi. org/ mal standards for electroencephalography in Canada: A review by 10. 1117/1. nph.2. 2. 020801 the Canadian Society of Clinical Neurophysiologists. In Canadian Amzica, F., & Steriade, M. (1998). Electrophysiological correlates of Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). sleep delta waves. Electroencephalography and Clinical Neuro- Cambridge University Press. https://doi. or g/10. 1017/ cjn. 2017. 217 physiology, 107(2), 69–83. De Luca, M., Beckmann, C. F., De Stefano, N., Matthews, P. M., & Barlow, J. S., Kamp, A., Morton, H. B., Ripoche, A., Shipton, H., Smith, S. M. (2006). fMRI resting state networks define distinct & Tchavdarov, D. B. (1974). EEG instrumentation standards: modes of long-distance interactions in the human brain. Neuro- Report of the Committee on EEG Instrumentation Standards of Image, 29(4), 1359–1367. https://d oi.o rg/1 0.1 016/j.n euroi mage. the International Federation of Societies for Electroencephalog- 2005. 08. 035 raphy and Clinical Neurophysiology. Electroencephalography DeBettencourt, M. T., Cohen, J. D., Lee, R. F., Norman, K. A., & and Clinical Neurophysiology, 37(5), 549–553. https:// doi. org/ Turk-Browne, N. B. (2015). Closed-loop training of attention 10. 1016/ 0013- 4694(74) 90100-X with real-time brain imaging. Nature Neuroscience. https:// doi. Baytas, I. M., Xiao, C., Zhang, X., Wang, F., Jain, A. K., & Zhou, J. (2017). org/ 10. 1038/ nn. 3940 Patient subtyping via time-aware LSTM networks. Proceedings of Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the ACM SIGKDD International Conference on Knowledge Dis- the dynamical organization of resting-state activity in the brain. In covery and Data Mining. https://doi. or g/10. 1145/ 30979 83. 30979 97 Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Bettus, G., Guedj, E., Joyeux, F., Confort-Gouny, S., Soulier, E., Publishing Group. https:// doi. org/ 10. 1038/ nrn29 61 Laguitton, V., Cozzone, P. J., Chauvel, P., Ranjeva, J. P., Dhamija, R., Moseley, B. D., Cascino, G. D., & Wirrell, E. C. (2011). Bartolomei, F., & Guye, M. (2009). Decreased basal fMRI A population-based study of long-term outcome of epilepsy in functional connectivity in epileptogenic networks and con- childhood with a focal or hemispheric lesion on neuroimaging. tralateral compensatory mechanisms. Human Brain Mapping, Epilepsia, 52(8), 1522–1526. https:// doi. org/ 10. 1111/j. 1528- 30(5), 1580–1591. https:// doi. org/ 10. 1002/ hbm. 20625 1167. 2011. 03192.x Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Dutta, A., Jacob, A., Chowdhury, S. R., Das, A., & Nitsche, M. A. Kirsch, H. E., Berger, M. S., Barbare, N. M., & Knight, R. T. (2006). (2015). EEG-NIRS based assessment of neurovascular coupling High gamma power is phase-locked to theta oscillations in human during anodal transcranial direct current stimulation - a stroke neocortex. Science. https:// doi. org/ 10. 1126/ scien ce. 11281 15 case series. Journal of Medical Systems, 39(4), 1–9. https:// doi. Chiarelli, A. M., Zappasodi, F., Di Pompeo, F., & Merla, A. (2017). Simul- org/ 10. 1007/ s10916- 015- 0205-7 taneous functional near-infrared spectroscopy and electroencepha- Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., Galuske, R. lography for monitoring of human brain activity and oxygenation: A. W., & Niessing, J. (2005). Hemodynamic signals correlate A review. Neurophotonics. https://doi. or g/10. 1117/1. n ph.4.4. 041411 tightly with synchronized gamma oscillations. Science. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, Fair, D. A., Nigg, J. T., Iyer, S., Bathula, D., Mills, K. L., Dosenbach, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase represen- N. U. F., Schlaggar, B. L., Mennes, M., Gutman, D., Bangaru, tations using RNN encoder-decoder for statistical machine trans- S., Buitelaar, J. K., Dickstein, D. P., Martino, A. D., Kennedy, D. lation. EMNLP 2014 - 2014 Conference on Empirical Methods N., Kelly, C., Luna, B., Schweitzer, J. B., Velanova, K., Wang, Y. in Natural Language Processing, Proceedings of the Conference. F., & Milham, M. P. (2013). Distinct neural signatures detected https:// doi. org/ 10. 3115/ v1/ d14- 1179 for ADHD subtypes after controlling for micro-movements in Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). resting state functional connectivity MRI data. Frontiers in Sys- Audio Word2Vec: Unsupervised learning of audio segment tems Neuroscience. https:// doi. org/ 10. 3389/ fnsys. 2012. 00080 representations using sequence-to-sequence autoencoder. Pro- Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K. ceedings of the Annual Conference of the International Speech R., & Blankertz, B. (2012). Enhanced performance by a hybrid Communication Association, INTERSPEECH. https://doi. or g/10. NIRS-EEG brain computer interface. NeuroImage, 59(1), 519– 21437/ Inter speech. 2016- 82 529. https:// doi. org/ 10. 1016/j. neuro image. 2011. 07. 084 Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations Fishburn, F. A., Norr, M. E., Medvedev, A. V., & Vaidya, C. J. (2014). for the use of multimodal monitoring in the neurointensive care Sensitivity of fNIRS to cognitive state and load. Frontiers in unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. Human Neuroscience. https:// doi. or g/ 10. 3389/ fnhum. 2014. 00076 1 3 Neuroinformatics (2022) 20:537–558 555 Foreman, B., & Claassen, J. (2012). Quantitative EEG for the detection Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. https:// of brain ischemia. Critical Care. https://doi. or g/10. 1186/ cc112 30 doi. org/ 10. 1016/j. yebeh. 2008. 01. 011 Franaszczuk, P. J., Kudela, P., & Bergey, G. K. (2003). External excita- Huppert, T. J., Diamond, S. G., Franceschini, M. A., & Boas, D. A. tory stimuli can terminate bursting in neural network models. (2009). HomER: A review of time-series analysis methods for Epilepsy Research, 53(1–2), 65–80. https:// doi. or g/ 10. 1016/ near-infrared spectroscopy of the brain. Applied Optics, 48(10), S0920- 1211(02) 00248-6 D280–D298. Freeman, W. J., Burke, B. C., & Holmes, M. D. (2003). Aperiodic Jia, X., & Kohn, A. (2011). Gamma rhythms in the brain. PLoS Biol- phase re-setting in scalp EEG of beta-gamma oscillations by state ogy, 9(4), 2–5. https:// doi. org/ 10. 1371/ journ al. pbio. 10010 45 transitions at alpha-theta rates. Human Brain Mapping. https:// Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and doi. org/ 10. 1002/ hbm. 10120 myocardial oxygen sufficiency and circulatory parameters. Sci- Gagnon, L., Cooper, R. J., Yücel, M. A., Perdue, K. L., Greve, D. N., ence, 198(4323), 1264–1267. & Boas, D. A. (2012). Short separation channel location impacts Julien, C. (2006). The enigma of Mayer waves: Facts and models. Cardio- the performance of short channel regression in NIRS. NeuroIm- vascular Research. https://d oi.o rg/1 0.1 016/j.c ardio res.2 005.1 1.0 08 age. https:// doi. org/ 10. 1016/j. neuro image. 2011. 08. 095 Khan, M. J., & Hong, K. -S. (2015). Passive BCI based on drowsiness Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting- detection: An fNIRS study. Biomedical Optics Express, 6(10), state fNIRS scanning duration on functional brain connectivity 4063. https:// doi. org/ 10. 1364/ boe.6. 004063 and graph theory metrics of brain network. Frontiers in Neurosci- Khan, M. J., & Hong, K. S. (2017). Hybrid EEG-FNIRS-based eight- ence, 11(JUL). https:// doi. org/ 10. 3389/ fnins. 2017. 00392 command decoding for BCI: Application to quadcopter control. Girouard, H., & Iadecola, C. (2006). Neurovascular coupling in the Frontiers in Neurorobotics, 11, 6. https://doi. or g/10. 3389/ fnbo t. normal brain and in hypertension, stroke, and Alzheimer disease. 2017. 00006 Journal of Applied Physiology, 100(1), 328–335. Knyazev, G. G. (2012). EEG delta oscillations as a correlate of basic Goldman, R. I., Stern, J. M., Engel, J., Jr., & Cohen, M. S. (2002). homeostatic and motivational processes. Neuroscience & Biobe- Simultaneous EEG and fMRI of the alpha rhythm. NeuroReport, havioral Reviews, 36(1), 677–695. 13(18), 2487. Kobayashi, E., Bagshaw, A. P., Grova, C., Dubeau, F., & Gotman, J. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT (2006). Negative BOLD responses to epileptic spikes. Human Brain press. Mapping, 27(6), 488–497. https:// doi. org/ 10. 1002/ hbm. 20193 Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, Koch, S. P., Koendgen, S., Bourayou, R., Steinbrink, J., & Obrig, H. D., Brodbeck, C., Parkkonen, L., & Hämäläinen, M. S. (2014). (2008). Individual alpha-frequency correlates with amplitude of MNE software for processing MEG and EEG data. NeuroImage. visual evoked potential and hemodynamic response. NeuroIm- https:// doi. org/ 10. 1016/j. neuro image. 2013. 10. 027 age, 41(2), 233–242. Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE law revisited. Physics in Medicine and Biology, 51(5). https:// Transactions on Neural Networks and Learning Systems. https:// doi. org/ 10. 1088/ 0031- 9155/ 51/5/ N02 doi. org/ 10. 1109/ TNNLS. 2016. 25829 24 Kohno, S., Miyai, I., Seiyama, A., Oda, I., Ishikawa, A., Tsuneishi, S., Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. Amita, T., & Shimizu, K. (2007). Removal of the skin blood flow (2015). DRAW: A recurrent neural network for image generation. artifact in functional near-infrared spectroscopic imaging data 32nd International Conference on Machine Learning, ICML 2015. through independent component analysis. Journal of Biomedical Gross, D. W., & Gotman, J. (1999). Correlation of high-frequency oscil- Optics. https:// doi. org/ 10. 1117/ 12814 249 lations with the sleep-wake cycle and cognitive activity in humans. Kotliar, K., Hauser, C., Ortner, M., Muggenthaler, C., Diehl-Schmid, Neuroscience. https:// doi. org/ 10. 1016/ S0306- 4522(99) 00343-7 J., Angermann, S., Hapfelmeier, A., Schmaderer, C., & Grimmer, He, B., & Liu, Z. (2008). Multimodal functional neuroimaging: inte- T. (2017). Altered neurovascular coupling as measured by optical grating functional MRI and EEG/MEG. IEEE Reviews in Bio- imaging: A biomarker for Alzheimer’s disease. Scientific Reports, medical Engineering, 1(2008), 23–40. https:// doi. org/ 10. 1109/ 7(1), 1–11. https:// doi. org/ 10. 1038/ s41598- 017- 13349-5 RBME. 2008. 20082 33 Laufs, H., Krakow, K., Sterzer, P., Eger, E., Beyerle, A., Salek-Haddadi, Helbok, R., & Claassen, J. (2013). Multimodal invasive monitor- A., & Kleinschmidt, A. (2003). Electroencephalographic signa- ing in status epilepticus: What is the evidence it has a place? tures of attentional and cognitive default modes in spontaneous Epilepsia, 54(SUPPL. 6), 57–60. https:// doi. org/ 10. 1111/ epi. brain activity fluctuations at rest. Proceedings of the National 12279 Academy of Sciences of the United States of America. https://doi. Hofle, N., Paus, T., Reutens, D., Fiset, P., Gotman, J., Evans, A. C., & org/ 10. 1073/ pnas. 18316 38100 Jones, B. E. (1997). Regional cerebral blood flow changes as a Le Roux, P. (2013). Physiological monitoring of the severe traumatic function of delta and spindle activity during slow wave sleep in brain injury patient in the intensive care unit. Current Neurology humans. Journal of Neuroscience. https:// doi. or g/ 10. 1523/ and Neuroscience Reports, 13(3), 331. jneur osci. 17- 12- 04800. 1997 Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature. Honda, Y., Nakamura, S., Ogawa, K., Yoshino, R., Tobler, P. N., https:// doi. org/ 10. 1038/ natur e14539 Nishimura, Y., & Tsutsui, K. I. (2021). Changes in beta and Lemieux, L., Krakow, K., & Fish, D. R. (2001). Comparison of spike- high-gamma power in resting-state electrocorticogram induced triggered functional MRI BOLD activation and EEG dipole model by repetitive transcranial magnetic stimulation of primary motor localization. NeuroImage. https://doi. or g/10. 1006/ nimg. 2001. 0896 cortex in unanesthetized macaque monkeys. Neuroscience Li, Z., Liu, H., Liao, X., Xu, J., Liu, W., Tian, F., He, Y., & Niu, H. Research. https:// doi. org/ 10. 1016/j. neures. 2021. 02. 002 (2015). Dynamic functional connectivity revealed by resting- Hong, K. S., Naseer, N., & Kim, Y. H. (2015). Classification of pre - state functional near-infrared spectroscopy. Biomedical Optics frontal and motor cortex signals for three-class fNIRS-BCI. Neu- Express. https:// doi. org/ 10. 1364/ boe.6. 002337 roscience Letters, 587, 87–92. https:// doi. org/ 10. 1016/j. neulet. Lindauer, U., Dirnagl, U., Füchtemeier, M., Böttiger, C., Offenhauser, 2014. 12. 029 N., Leithner, C., & Royl, G. (2010). Pathophysiological inter- Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: ference with neurovascular coupling–when imaging based on Their relationships with epilepsy and behavior. In Epilepsy and hemoglobin might go blind. Frontiers in Neuroenergetics, 2, 25. 1 3 556 Neuroinformatics (2022) 20:537–558 Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning Murta, T., Leite, M., Carmichael, D. W., Figueiredo, P., & Lemieux, L. to diagnose with LSTM recurrent neural networks. 4th Inter- (2015). Electrophysiological correlates of the BOLD signal for national Conference on Learning Representations, ICLR 2016 EEG-informed fMRI. Human Brain Mapping, 36(1), 391–414. - Conference Track Proceedings.https:// doi. org/ 10. 1002/ hbm. 22623 Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., (2001). Neurophysiological investigation of the basis of the fMRI Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscil- signal. Nature. https:// doi. org/ 10. 1038/ 35084 005 lation neurofeedback modulates amygdala complex connectivity Lövblad, K. O., Thomas, R., Jakob, P. M., Scammell, T., Bassetti, C., and arousal in posttraumatic stress disorder. NeuroImage: Clini- Griswold, M., Ives, J., Matheson, J., Edelman, R. R., & Warach, cal. https:// doi. org/ 10. 1016/j. nicl. 2016. 07. 006 S. (1999). Silent functional magnetic resonance imaging demon- Niessing, J., Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., & strates focal activation in rapid eye movement sleep. Neurology. Galuske, R. A. W. (2005). Neuroscience: Hemodynamic signals https:// doi. org/ 10. 1212/ wnl. 53.9. 2193 correlate tightly with synchronized gamma oscillations. Science, Lu, C. M., Zhang, Y. J., Biswal, B. B., Zang, Y. F., Peng, D. L., & Zhu, 309(5736), 948–951. https:// doi. org/ 10. 1126/ scien ce. 11109 48 C. Z. (2010). Use of fNIRS to assess resting state functional con- Nikulin, V. V., Fedele, T., Mehnert, J., Lipp, A., Noack, C., Steinbrink, nectivity. Journal of Neuroscience Methods, 186(2), 242–249. J., & Curio, G. (2014). Monochromatic Ultra-Slow (~ 0.1 Hz) https:// doi. org/ 10. 1016/j. jneum eth. 2009. 11. 010 Oscillations in the human electroencephalogram and their rela- Luo, C., Li, Q., Lai, Y., Xia, Y., Qin, Y., Liao, W., Li, S., Zhou, D., tion to hemodynamics. NeuroImage, 97, 71–80. https:// doi. org/ Yao, D., & Gong, Q. (2011). Altered functional connectivity in 10. 1016/J. NEURO IMAGE. 2014. 04. 008 default mode network in absence epilepsy: A resting-state fMRI Niu, H., & He, Y. (2014). Resting-state functional brain connectivity: study. Human Brain Mapping, 32(3), 438–449. https:// doi. org/ Lessons from functional near-infrared spectroscopy. The Neuro- 10. 1002/ hbm. 21034 scientist. https:// doi. org/ 10. 1177/ 10738 58413 502707 Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Multi-task Sequence to Sequence Learning. c, 1–10. http://ar xiv. Merkle, H., & Ugurbil, K. (1992). Intrinsic signal changes org/ abs/ 1511. 06114 accompanying sensory stimulation: Functional brain mapping Maas, A. I. R., Menon, D. K., Lingsma, H. F., Pineda, J. A., Sandel, M. with magnetic resonance imaging. Proceedings of the National E., & Manley, G. T. (2012). Re-orientation of clinical research Academy of Sciences of the United States of America. https://doi. in traumatic brain injury: Report of an international workshop org/ 10. 1073/ pnas. 89. 13. 5951 on comparative effectiveness research. Journal of Neurotrauma, Palva, J. M., Monto, S., Kulashekhar, S., & Palva, S. (2010). Neuronal 29(1), 32–46. https:// doi. org/ 10. 1089/ neu. 2010. 1599 synchrony reveals working memory networks and predicts indi- Mantini, D., Perrucci, M. G., Del Gratta, C., Romani, G. L., & Corbetta, vidual memory capacity. Proceedings of the National Academy M. (2007). Electrophysiological signatures of resting state net- of Sciences of the United States of America, 107(16), 7580–7585. works in the human brain. Proceedings of the National Academy https:// doi. org/ 10. 1073/ pnas. 09131 13107 of Sciences of the United States of America. https:// doi. org/ 10. Park, Y., Luo, L., Parhi, K. K., & Netoff, T. (2011). Seizure prediction 1073/ pnas. 07006 68104 with spectral power of EEG using cost-sensitive support vec- Martinez-Montes, E., Valdes-Sosa, P. A., Miwakeichi, F., Goldman, R. tor machines. Epilepsia, 52(10), 1761–1770. https:// doi. org/ 10. I., & Cohen, M. S. (2004). Concurrent EEG/fMRI analysis by 1111/j. 1528- 1167. 2011. 03138.x multiway partial least squares. NeuroImage, 22(3), 1023–1034. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., https:// doi. org/ 10. 1016/j. neuro image. 2004. 03. 038 Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., McKenna, T. M., McMullen, T. A., & Shlesinger, M. F. (1994). The Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, brain as a dynamic physical system. Neuroscience, 60(3), 587– M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in 605. https:// doi. org/ 10. 1016/ 0306- 4522(94) 90489-8 Python. Journal of Machine Learning Research. Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and opti- Peng, K., Nguyen, D. K., Tayah, T., Vannasing, P., Tremblay, J., Sawan, mizing LSTM language models. 6th International Conference M., Lassonde, M., Lesage, F., & Pouliot, P. (2014). FNIRS- on Learning Representations, ICLR 2018 - Conference Track EEG study of focal interictal epileptiform discharges. Epilepsy Proceedings. Research. https:// doi. org/ 10. 1016/j. eplep syres. 2013. 12. 011 Merker, B. H. (2016). Cortical gamma oscillations: Details of their Phillips, A. A., Squair, J. W., Sayenko, D. G., Edgerton, V. R., Gerasimenko, genesis preclude a role in cognition. Frontiers in Computational Y., & Krassioukov, A. V. (2018). An autonomic neuroprosthesis: Neuroscience. https:// doi. org/ 10. 3389/ fncom. 2016. 00078 Noninvasive electrical spinal cord stimulation restores autonomic Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). cardiovascular function in individuals with spinal cord injury. Jour- Recurrent neural network based language model. Proceedings of nal of Neurotrauma, 35(3), 446–451. https:// doi. org/ 10. 1089/ neu. the 11th Annual Conference of the International Speech Com-2017. 5082 munication Association, INTERSPEECH 2010. Pizzo, F., Frauscher, B., Ferrari-Marinho, T., Amiri, M., Dubeau, F., Miller, C. M. (2012). Update on multimodality monitoring. Current & Gotman, J. (2016). Detectability of fast ripples (>250 Hz) Neurology and Neuroscience Reports. https:// doi. org/ 10. 1007/ on the scalp EEG: A proof-of-principle study with subder- s11910- 012- 0274-7 mal electrodes. Brain Topography. https:// doi. or g/ 10. 1007/ Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: s10548- 016- 0481-7 New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884– Pressl, C., Brandner, P., Schaffelhofer, S., Blackmon, K., Dugan, 898). Lancet Publishing Group. https:// doi. org/ 10. 1016/ S0140- P., Holmes, M., Thesen, T., Kuzniecky, R., Devinsky, O., & 6736(14) 60456-6 Freiwald, W. A. (2019). Resting state functional connectivity Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., & Malach, patterns associated with pharmacological treatment resistance R. (2005). Neuroscience: Coupling between neuronal firing, field in temporal lobe epilepsy. Epilepsy Research. https:// doi. org/ potentials, and fMRI in human auditory cortex. Science. https:// 10. 1016/j. eplep syres. 2018. 11. 002 doi. org/ 10. 1126/ scien ce. 11109 13 Quaresima, V., & Ferrari, M. (2019). A mini-review on functional near- Müller-Putz, G. R. (2020). Electroencephalography. Handbook of Clinical infrared spectroscopy (fNIRS): Where do we stand, and where Neurology. https:// doi. org/ 10. 1016/ B978-0- 444- 63934-9. 00018-4 should we go? Photonics, 6(3), 87. h t t p s :/ / d o i . o r g / 1 0 . 3 3 9 0 / photo nics6 030087 1 3 Neuroinformatics (2022) 20:537–558 557 Ray, S., & Maunsell, J. H. R. (2010). Differences in gamma frequencies Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis- across visual cortex restrict their possible use in computation. Peacock, J., Weiskopf, N., Blefari, M. L., Rana, M., Oblak, E., Neuron, 67(5), 885–896. https:// doi. org/ 10. 1016/j. neuron. 2010. Birbaumer, N., & Sulzer, J. (2017). Closed-loop brain training: 08. 004 The science of neurofeedback. Nature Reviews Neuroscience. Richardson, M. P. (2012). Large scale brain models of epilepsy: https:// doi. org/ 10. 1038/ nrn. 2016. 164 Dynamics meets connectomics. Journal of Neurology, Neuro- Sitnikova, E., Hramov, A. E., Grubov, V., & Koronovsky, A. A. (2016). surgery and Psychiatry, 83(12), 1238–1248. https:// doi. org/ 10. Rhythmic activity in EEG and sleep in rats with absence epilepsy. 1136/ jnnp- 2011- 301944 Brain Research Bulletin. https:// doi. org/ 10. 1016/j. brain resbu ll. Rojas, G. M., Alvarez, C., Montoya, C. E., de la Iglesia-Vayá, M., 2015. 11. 012 Cisternas, J. E., & Gálvez, M. (2018). Study of resting-state func- Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. tional connectivity networks using EEG electrodes position as (2011). Semi-supervised recursive autoencoders for predicting seed. Frontiers in Neuroscience. https:// doi. org/ 10. 3389/ fnins. sentiment distributions. EMNLP 2011 - Conference on Empiri- 2018. 00235 cal Methods in Natural Language Processing, Proceedings of Ros, T., Baars, B. J., Lanius, R. A., & Vuilleumier, P. (2014). Tuning the Conference. pathological brain oscillations with neurofeedback: A systems Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsuper- neuroscience framework. Frontiers in Human Neuroscience, 8, vised learning of video representations using LSTMs. 32nd Inter- 1008. https:// doi. org/ 10. 3389/ fnhum. 2014. 01008 national Conference on Machine Learning, ICML 2015. Salek-Haddadi, A., Friston, K. J., Lemieux, L., & Fish, D. R. (2003). Strangman, G., Boas, D. A., & Sutton, J. P. (2002). Non-invasive neuro- Studying spontaneous EEG activity with fMRI. Brain Research imaging using near-infrared light. Biological Psychiatry. https:// Reviews. https:// doi. org/ 10. 1016/ S0165- 0173(03) 00193-0doi. org/ 10. 1016/ S0006- 3223(02) 01550-0 Sassaroli, A., Pierro, M., Bergethon, P. R., & Fantini, S. (2012). Low- Sumiyoshi, A., Suzuki, H., Ogawa, T., Riera, J. J., Shimokawa, H., & frequency spontaneous oscillations of cerebral hemodynamics Kawashima, R. (2012). Coupling between gamma oscillation and investigated with near-infrared spectroscopy: A review. IEEE fMRI signal in the rat somatosensory cortex: Its dependence on Journal on Selected Topics in Quantum Electronics. https:// doi. systemic physiological parameters. NeuroImage. https://doi. or g/ org/ 10. 1109/ JSTQE. 2012. 21835 8110. 1016/j. neuro image. 2011. 12. 082 Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence Informatics, Integration Data Display and Analysis The Partici- learning with neural networks. Advances in Neural Information pants in the International Multidisciplinary Consensus Confer- Processing Systems, 4(January), 3104–3112. ence on Multimodality Monitoring. Neurocritical Care. https:// Thair, H., Holloway, A. L., Newport, R., & Smith, A. D. (2017). Tran- doi. org/ 10. 1007/ s12028- 014- 0037-1 scranial direct current stimulation (tDCS): A Beginner’s guide Scholkmann, F., Kleiser, S., Metz, A. J., Zimmermann, R., Pavia, J. for design and implementation. Frontiers in Neuroscience, 11, M., Wolf, U., & Wolf, M. (2014). A review on continuous wave 641. https:// doi. org/ 10. 3389/ fnins. 2017. 00641 functional near-infrared spectroscopy and imaging instrumenta- Tong, Y., Hocke, L. M., Licata, S. C., & Frederick, B. D. (2012). Low- tion and methodology. NeuroImage, 85, 6–27. frequency oscillations measured in the periphery with near- Schwab, K., Skupin, H., Eiselt, M., Walther, M., Voss, A., & Witte, H. infrared spectroscopy are strongly correlated with blood oxygen (2009). Coordination of the EEG and the heart rate of preterm level-dependent functional magnetic resonance imaging signals. neonates during quiet sleep. Neuroscience Letters. https:// doi. Journal of Biomedical Optics, 17(10), 1060041. https:// doi. org/ org/ 10. 1016/j. neulet. 2009. 09. 03110. 1117/1. jbo. 17. 10. 106004 Sharbrough, F. W., Messick, J. M., & Sundt, T. M. (1973). Correlation Tracy, J. I., & Doucet, G. E. (2015). Resting-state functional connectiv- of continuous electroencephalograms with cerebral blood flow ity in epilepsy: Growing relevance for clinical decision making. measurements during carotid endarterectomy. Stroke. https://doi. Current Opinion in Neurology. https:// doi. org/ 10. 1097/ WCO. org/ 10. 1161/ 01. STR.4. 4. 67400000 00000 000178 Shen, H. H. (2015). Core concept: Resting-state connectivity. Proceed- Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). ings of the National Academy of Sciences of the United States of Semi-supervised Seizure Prediction with Generative Adversarial America. https:// doi. org/ 10. 1073/ pnas. 15187 85112 Networks. 1–6. http:// arxiv. org/ abs/ 1806. 08235 Shmuel, A., & Leopold, D. A. (2008). Neuronal correlates of spon- Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). taneous fluctuations in fMRI signals in monkey visual cortex: Extracting and composing robust features with denoising autoen- Implications for functional connectivity at rest. Human Brain coders. Proceedings of the 25th International Conference on Mapping, 29(7), 751–761. https:// doi. org/ 10. 1002/ hbm. 20580 Machine Learning. https:// doi. org/ 10. 1145/ 13901 56. 13902 94 Siegel, J. S., Snyder, A. Z., Ramsey, L., Shulman, G. L., & Corbetta, Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. M. (2016). The effects of hemodynamic lag on functional con - (2010). Stacked denoising autoencoders: Learning Useful Repre- nectivity and behavior after stroke. Journal of Cerebral Blood sentations in a Deep Network with a Local Denoising Criterion. Flow and Metabolism, 36(12), 2162–2176. https:// doi. org/ 10. Journal of Machine Learning Research. 1177/ 02716 78X15 614846 Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state Sigala, R., Haufe, S., Roy, D., Dinse, H. R., & Ritter, P. (2014). The fNIRS imaging duration for accurate and stable mapping of brain role of alpha-rhythm states in perceptual learning: Insights from connectivity network in children. Scientific Reports , 7(1). https:// experiments and computational models. Frontiers in Compu-doi. org/ 10. 1038/ s41598- 017- 06340-7 tational Neuroscience, 8, 1–19. https:// doi. org/ 10. 3389/ fncom. Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based 2014. 00036 LSTM for aspect-level sentiment classification. EMNLP 2016 - Singh, K. D. (2012). Which “neural activity” do you mean? fMRI, Conference on Empirical Methods in Natural Language Process- MEG, oscillations and neurotransmitters. NeuroImage. https:// ing, Proceedings. https:// doi. org/ 10. 18653/ v1/ d16- 1058 doi. org/ 10. 1016/j. neuro image. 2012. 01. 028 Wang, Z., Lu, G., Zhang, Z., Zhong, Y., Jiao, Q., Zhang, Z., Tan, Q., Sirpal, P., Kassab, A., Pouliot, P., & Nguyen, D. K. (2019). fNIRS Tian, L., Chen, G., Liao, W., Li, K., & Liu, Y. (2011). Altered improves seizure detection in multimodal EEG-fNIRS record- resting state networks in epileptic patients with generalized tonic- clonic seizures. Brain Research, 1374, 134–141. https://d oi.o rg/ ings. Journal of Biomedical Optics, 24(05), 1. https:// doi. org/ 10. 1016/j. brain res. 2010. 12. 034 10. 1117/1. jbo. 24.5. 051408 1 3 558 Neuroinformatics (2022) 20:537–558 Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estima- Zhang, X., Pan, W. J., & Keilholz, S. D. (2020). The relationship between tion of Power Spectra: A Method Based on Time Averaging Over BOLD and neural activity arises from temporally sparse events. Short, Modified Periodograms. IEEE Transactions on Audio and NeuroImage, 207, 116390. https:// doi. org/ 10. 1016/j. neuro image. Electroacoustics. https:// doi. org/ 10. 1109/ TAU. 1967. 11619 012019. 116390 Whittingstall, K., & Logothetis, N. K. (2009). Frequency-band cou- Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Liao, W., Wang, Z., Wang, Z., Li, pling in surface EEG reflects spiking activity in monkey visual K., Chen, H., & Liu, Y. (2010b). Altered spontaneous neuronal cortex. Neuron. https:// doi. org/ 10. 1016/j. neuron. 2009. 08. 016 activity of the default-mode network in mesial temporal lobe epi- Woermann, F. G., & Vollmar, C. (2009). Clinical MRI in children lepsy. Brain Research, 1323, 152–160. https:// doi. org/ 10. 1016/j. and adults with focal epilepsy: A critical review. Epilepsy and brain res. 2010. 01. 042 Behavior, 15(1), 40–49. https:// doi. org/ 10. 1016/j. yebeh. 2009. Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Yang, Z., Liao, W., Chen, Z., 02. 032 Shi, J., & Liu, Y. (2009). Impaired attention network in temporal Yücel, M. A., Selb, J., Aasted, C. M., Lin, P.-Y., Borsook, D., Becerra, lobe epilepsy: A resting FMRI study. Neuroscience Letters, 458(3), L., & Boas, D. A. (2016). Mayer waves reduce the accuracy of esti- 97–101. https:// doi. org/ 10. 1016/j. neulet. 2009. 04. 040 mated hemodynamic response functions in functional near-infrared Zhao, F., Feng, J., Zhao, J., Yang, W., & Yan, S. (2018). Robust LSTM- spectroscopy. Biomedical Optics Express, 7(8), 3078. https:// doi. Autoencoders for Face De-Occlusion in the Wild. IEEE Trans- org/ 10. 1364/ boe.7. 003078 actions on Image Processing. https:// doi. org/ 10. 1109/ TIP. 2017. Zhang, H., Zhang, Y. J., Lu, C. M., Ma, S. Y., Zang, Y. F., & Zhu, C. Z. 27714 08 (2010a). Functional connectivity as revealed by independent com- ponent analysis of resting-state fNIRS measurements. NeuroImage. Publisher's Note Springer Nature remains neutral with regard to https:// doi. org/ 10. 1016/j. neuro image. 2010. 02. 080 jurisdictional claims in published maps and institutional affiliations. Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. https:// doi. org/ 10. 1145/ 32099 78. 32101 84 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Neuroinformatics Springer Journals http://www.deepdyve.com/lp/springer-journals/multimodal-autoencoder-predicts-fnirs-resting-state-from-eeg-signals-Gm32TXcDKz

Loading next page...

References (166)

Peter Roux (2013)
Physiological Monitoring of the Severe Traumatic Brain Injury Patient in the Intensive Care Unit
Current Neurology and Neuroscience Reports, 13
G. Strangman, D. Boas, J. Sutton (2002)
Non-invasive neuroimaging using near-infrared light
Biological Psychiatry, 52
F. Jöbsis (1977)
Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters.
Science, 198 4323
K. Kotliar, C. Hauser, M. Ortner, Claudia Muggenthaler, J. Diehl-Schmid, S. Angermann, A. Hapfelmeier, C. Schmaderer, T. Grimmer (2017)
Altered neurovascular coupling as measured by optical imaging: a biomarker for Alzheimer’s disease
Scientific Reports, 7
Christina Pressl, Philip Brandner, S. Schaffelhofer, K. Blackmon, Patricia Dugan, Manisha Holmes, T. Thesen, R. Kuzniecky, O. Devinsky, W. Freiwald (2018)
Resting state functional connectivity patterns associated with pharmacological treatment resistance in temporal lobe epilepsy
Epilepsy Research, 149
S. Moshé, E. Perucca, P. Ryvlin, T. Tomson (2015)
Epilepsy: new advances
The Lancet, 385
M. deBettencourt, J. Cohen, Ray Lee, K. Norman, N. Turk-Browne (2015)
Closed-loop training of attention with real-time brain imaging
Nature neuroscience, 18
(Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. IEEE Transactions on Audio and Electroacoustics. 10.1109/TAU.1967.1161901)
Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. IEEE Transactions on Audio and Electroacoustics. 10.1109/TAU.1967.1161901
Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. IEEE Transactions on Audio and Electroacoustics. 10.1109/TAU.1967.1161901, Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. IEEE Transactions on Audio and Electroacoustics. 10.1109/TAU.1967.1161901
Walter Freeman, B. Burke (2003)
Aperiodic phase resetting in scalp EEG of beta-gamma oscillations by state transitions at alpha-theta rates
Zhiqiang Zhang, G. Lu, Y. Zhong, Qi-fu Tan, W. Liao, Zhengge Wang, Zhongqiu Wang, Kai Li, Huafu Chen, Yijun Liu (2010)
Altered spontaneous neuronal activity of the default-mode network in mesial temporal lobe epilepsy
Brain Research, 1323
(1999)
Silent functional magnetic resonance imaging demonstrates focal activation in rapid eye movement sleep
(Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. 32nd International Conference on Machine Learning, ICML 2015.)
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. 32nd International Conference on Machine Learning, ICML 2015.
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. 32nd International Conference on Machine Learning, ICML 2015., Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. 32nd International Conference on Machine Learning, ICML 2015.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, Daan Wierstra (2015)
DRAW: A Recurrent Neural Network For Image Generation
ArXiv, abs/1502.04623
(Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Informatics, Integration Data Display and Analysis The Participants in the International Multidisciplinary Consensus Conference on Multimodality Monitoring. Neurocritical Care. 10.1007/s12028-014-0037-1)
Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Informatics, Integration Data Display and Analysis The Participants in the International Multidisciplinary Consensus Conference on Multimodality Monitoring. Neurocritical Care. 10.1007/s12028-014-0037-1
Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Informatics, Integration Data Display and Analysis The Participants in the International Multidisciplinary Consensus Conference on Multimodality Monitoring. Neurocritical Care. 10.1007/s12028-014-0037-1, Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Informatics, Integration Data Display and Analysis The Participants in the International Multidisciplinary Consensus Conference on Multimodality Monitoring. Neurocritical Care. 10.1007/s12028-014-0037-1
U. Lindauer, U. Dirnagl, Martina Füchtemeier, Caroline Böttiger, N. Offenhauser, C. Leithner, G. Royl (2010)
Pathophysiological Interference with Neurovascular Coupling – When Imaging Based on Hemoglobin Might Go Blind
Frontiers in Neuroenergetics, 2
M. Czisch, R. Wehrle, C. Kaufmann, T. Wetter, F. Holsboer, T. Pollmächer, D. Auer (2004)
Functional MRI during sleep: BOLD signal decreases and their electrophysiological correlates
European Journal of Neuroscience, 20
Ke Peng, D. Nguyen, Tania Tayah, P. Vannasing, J. Tremblay, M. Sawan, M. Lassonde, F. Lesage, P. Pouliot (2014)
fNIRS-EEG study of focal interictal epileptiform discharges
Epilepsy Research, 108
A. Sassaroli, Michele Pierro, P. Bergethon, S. Fantini (2012)
Low-Frequency Spontaneous Oscillations of Cerebral Hemodynamics Investigated With Near-Infrared Spectroscopy: A Review
IEEE Journal of Selected Topics in Quantum Electronics, 18
A. Curtin, S. Tong, Junfeng Sun, Jijun Wang, B. Onaral, H. Ayaz (2019)
A Systematic Review of Integrated Functional Near-Infrared Spectroscopy (fNIRS) and Transcranial Magnetic Stimulation (TMS) Studies
Frontiers in Neuroscience, 13
Inci Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil Jain, Jiayu Zhou (2017)
Patient Subtyping via Time-Aware LSTM Networks
Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
E. Martínez-Montes, P. Valdés-Sosa, F. Miwakeichi, R. Goldman, Mark Cohen (2004)
Concurrent EEG/fMRI analysis by multiway Partial Least Squares
NeuroImage, 22
Muhammad Khan, K. Hong (2017)
Hybrid EEG–fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control
Frontiers in Neurorobotics, 11
R. Mukamel, Hagar Gelbard, A. Arieli, U. Hasson, I. Fried, R. Malach (2005)
Coupling Between Neuronal Firing, Field Potentials, and fMRI in Human Auditory Cortex
Science, 309
(Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research.)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research., Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research.
Krishna Singh (2012)
Which “neural activity” do you mean? fMRI, MEG, oscillations and neurotransmitters
NeuroImage, 62
Yann LeCun, Yoshua Bengio, Geoffrey Hinton (2015)
Deep Learning
Nature, 521
G. Citerio, M. Oddo, F. Taccone (2015)
Recommendations for the use of multimodal monitoring in the neurointensive care unit
Current Opinion in Critical Care, 21
V. Quaresima, M. Ferrari (2019)
A Mini-Review on Functional Near-Infrared Spectroscopy (fNIRS): Where Do We Stand, and Where Should We Go?
Photonics
Mark Connolly, Paul Vespa, N. Pouratian, Nestor Gonzalez, Xiao Hu (2015)
Title : Characterization of the Relationship Between Intracranial Pressure and Electroencephalographic Monitoring in Burst-Suppressed Patients
(1967)
Electroacoustics. https:// doi
Teresa Murta, M. Leite, D. Carmichael, P. Figueiredo, L. Lemieux (2014)
Electrophysiological correlates of the BOLD signal for EEG‐informed fMRI
Human Brain Mapping, 36
B. Merker (2016)
Cortical Gamma Oscillations: Details of Their Genesis Preclude a Role in Cognition
Frontiers in Computational Neuroscience, 10
W. Grey, Walter, Denis Hill, Xw Cobb, D. Whitteridge, G. Greville, M. Edited, Geoffrey With, O. Adrian, F. Macdonald, Seymour Philps, Surgeon-In (1950)
OPHTHALMIC OPERATIONS
The Ulster Medical Journal, 19
Andrew Nicholson, T. Ros, P. Frewen, M. Densmore, J. Théberge, R. Kluetsch, R. Jetly, R. Lanius (2016)
Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder
NeuroImage : Clinical, 12
R. Goldman, J. Stern, J. Engel, Mark Cohen (2002)
Simultaneous EEG and fMRI of the alpha rhythm
NeuroReport, 13
H. Girouard, C. Iadecola (2006)
Neurovascular coupling in the normal brain and in hypertension, stroke, and Alzheimer disease.
Journal of applied physiology, 100 1
C. Luo, Qifu Li, Yongxiu Lai, Yang Xia, Yun Qin, W. Liao, Shasha Li, Dong Zhou, D. Yao, Q. Gong (2011)
Altered functional connectivity in default mode network in absence epilepsy: A resting‐state fMRI study
Human Brain Mapping, 32
Zhen Li, Hanli Liu, Xuhong Liao, Jingping Xu, Wenli Liu, Fenghua Tian, Yong He, Haijing Niu (2015)
Dynamic functional connectivity revealed by resting-state functional near-infrared spectroscopy.
Biomedical optics express, 6 7
Shujie Geng, Xiangyu Liu, B. Biswal, Haijing Niu (2017)
Effect of Resting-State fNIRS Scanning Duration on Functional Brain Connectivity and Graph Theory Metrics of Brain Network
Frontiers in Neuroscience, 11
Zhengge Wang, G. Lu, Zhiqiang Zhang, Y. Zhong, Q. Jiao, Zong-jun Zhang, Qi-fu Tan, Lei Tian, Guanghui Chen, W. Liao, Kai Li, Yijun Liu (2011)
Altered resting state networks in epileptic patients with generalized tonic–clonic seizures
Brain Research, 1374
(Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.)
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press., Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
Haijing Niu, Yong He (2014)
Resting-State Functional Brain Connectivity
The Neuroscientist, 20
Weinan Zhang (2018)
Generative Adversarial Nets for Information Retrieval: Fundamentals and Advances
The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
Xing Hao, Guigang Zhang, Shang Ma (2016)
Deep Learning
Int. J. Semantic Comput., 10
(Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). Semi-supervised Seizure Prediction with Generative Adversarial Networks. 1–6. http://arxiv.org/abs/1806.08235)
Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). Semi-supervised Seizure Prediction with Generative Adversarial Networks. 1–6. http://arxiv.org/abs/1806.08235
Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). Semi-supervised Seizure Prediction with Generative Adversarial Networks. 1–6. http://arxiv.org/abs/1806.08235, Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). Semi-supervised Seizure Prediction with Generative Adversarial Networks. 1–6. http://arxiv.org/abs/1806.08235
S. Ogawa, D. Tank, Ravi Menon, J. Ellermann, S. Kim, H. Merkle, K. Uğurbil (1992)
Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging.
Proceedings of the National Academy of Sciences of the United States of America, 89 13
Pascal Vincent, H. Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol (2008)
Extracting and composing robust features with denoising autoencoders
G. Knyazev (2012)
EEG delta oscillations as a correlate of basic homeostatic and motivational processes
Neuroscience & Biobehavioral Reviews, 36
F. Woermann, C. Vollmar (2009)
Clinical MRI in children and adults with focal epilepsy: A critical review
Epilepsy & Behavior, 15
A. Sumiyoshi, Hideaki Suzuki, T. Ogawa, J. Riera, H. Shimokawa, R. Kawashima (2012)
Coupling between gamma oscillation and fMRI signal in the rat somatosensory cortex: Its dependence on systemic physiological parameters
NeuroImage, 60
D. Gross, J. Gotman (1999)
Correlation of high-frequency oscillations with the sleep–wake cycle and cognitive activity in humans
Neuroscience, 94
Chun-Ming Lu, Yu-Jin Zhang, B. Biswal, Y. Zang, D. Peng, Chaozhe Zhu (2010)
Use of fNIRS to assess resting state functional connectivity
Journal of Neuroscience Methods, 186
E. Sitnikova, A. Hramov, V. Grubov, A. Koronovsky (2016)
Rhythmic activity in EEG and sleep in rats with absence epilepsy
Brain Research Bulletin, 120
B. Foreman, J. Claassen (2012)
Quantitative EEG for the detection of brain ischemia
Critical Care, 16
R. Helbok, J. Claassen (2013)
Multimodal invasive monitoring in status epilepticus: What is the evidence it has a place?
Epilepsia, 54
F. Fishburn, M. Norr, A. Medvedev, C. Vaidya (2014)
Sensitivity of fNIRS to cognitive state and load
Frontiers in Human Neuroscience, 8
M. Khan, K. Hong (2015)
Passive BCI based on drowsiness detection: an fNIRS study.
Biomedical optics express, 6 10
J. Palva, S. Monto, Shrikanth Kulashekhar, S. Palva (2010)
Neuronal synchrony reveals working memory networks and predicts individual memory capacity
Proceedings of the National Academy of Sciences, 107
C. Aasted, M. Yücel, R. Cooper, Jay Dubb, D. Tsuzuki, L. Becerra, Mike Petkov, D. Borsook, I. Dan, D. Boas (2015)
Anatomical guidance for functional near-infrared spectroscopy: AtlasViewer tutorial
Neurophotonics, 2
(Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior. In Epilepsy and Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. 10.1016/j.yebeh.2008.01.011)
Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior. In Epilepsy and Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. 10.1016/j.yebeh.2008.01.011
Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior. In Epilepsy and Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. 10.1016/j.yebeh.2008.01.011, Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior. In Epilepsy and Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. 10.1016/j.yebeh.2008.01.011
Xiaoxuan Jia, A. Kohn (2011)
Gamma Rhythms in the Brain
PLoS Biology, 9
Yequan Wang, Minlie Huang, Xiaoyan Zhu, Li Zhao (2016)
Attention-based LSTM for Aspect-level Sentiment Classification
Chad Miller, A. Naidech, C. Robertson (2014)
Multimodality Monitoring: Informatics, Integration Data Display and Analysis
Neurocritical Care, 21
László Kocsis, Peter Herman, Andras Eke (2006)
The modified Beer–Lambert law revisited
Physics in Medicine & Biology, 51
Y. Park, Lan Luo, K. Parhi, T. Netoff (2011)
Seizure prediction with spectral power of EEG using cost‐sensitive support vector machines
Epilepsia, 52
N. Hofle, T. Paus, D. Reutens, P. Fiset, J. Gotman, Alan Evans, B. Jones (1997)
Regional Cerebral Blood Flow Changes as a Function of Delta and Spindle Activity during Slow Wave Sleep in Humans
The Journal of Neuroscience, 17
G. Rojas, C. Alvarez, Carlos Montoya, María Iglesia-Vayá, J. Cisternas, M. Gálvez (2018)
Study of Resting-State Functional Connectivity Networks Using EEG Electrodes Position As Seed
Frontiers in Neuroscience, 12
Jingyu Wang, Qi Dong, Haijing Niu (2017)
The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children
Scientific Reports, 7
L. Lemieux, K. Krakow, D. Fish (2001)
Comparison of Spike-Triggered Functional MRI BOLD Activation and EEG Dipole Model Localization
NeuroImage, 14
K. Hong, Noman Naseer, Y. Kim (2015)
Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI
Neuroscience Letters, 587
T. Huppert, S. Diamond, M. Franceschini, D. Boas (2009)
HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain.
Applied optics, 48 10
(1974)
International Federation of Societies for Electroencephalography and Clinical Neurophysiology.
Electroencephalography and clinical neurophysiology, 37 5
D. Fair, J. Nigg, Swathi Iyer, Deepti Bathula, K. Mills, N. Dosenbach, B. Schlaggar, M. Mennes, David Gutman, S. Bangaru, J. Buitelaar, D. Dickstein, A. Martino, D. Kennedy, C. Kelly, B. Luna, J. Schweitzer, K. Velanova, Yu-Feng Wang, S. Mostofsky, F. Castellanos, M. Milham (2012)
Distinct neural signatures detected for ADHD subtypes after controlling for micro-movements in resting state functional connectivity MRI data
Frontiers in Systems Neuroscience, 6
Chad Miller (2012)
Update on Multimodality Monitoring
Current Neurology and Neuroscience Reports, 12
Minh-Thang Luong, Quoc Le, Ilya Sutskever, O. Vinyals, Lukasz Kaiser (2015)
Multi-task Sequence to Sequence Learning
CoRR, abs/1511.06114
V. Nikulin, T. Fedele, J. Mehnert, A. Lipp, C. Noack, J. Steinbrink, G. Curio (2014)
Monochromatic Ultra-Slow (~0.1Hz) Oscillations in the human electroencephalogram and their relation to hemodynamics
NeuroImage, 97
(Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning to diagnose with LSTM recurrent neural networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings.)
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning to diagnose with LSTM recurrent neural networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings.
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning to diagnose with LSTM recurrent neural networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings., Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning to diagnose with LSTM recurrent neural networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings.
(1974)
EEG instrumentation standards: Report of the Committee on EEG Instrumentation Standards of the International Federation of Societies for Electroencephalography and Clinical Neurophysiology
Electroencephalography and Clinical Neurophysiology, 37
S. Fazli, J. Mehnert, J. Steinbrink, G. Curio, A. Villringer, K. Müller, B. Blankertz (2012)
Enhanced performance by a hybrid NIRS–EEG brain computer interface
NeuroImage, 59
A. Maas, D. Menon, Hester Lingsma, J. Pineda, M. Sandel, G. Manley (2012)
Re-orientation of clinical research in traumatic brain injury: report of an international workshop on comparative effectiveness research.
Journal of neurotrauma, 29 1
Han Zhang, Yu-Jin Zhang, Chun-Ming Lu, Shuang-Ye Ma, Y. Zang, Chaozhe Zhu (2010)
Functional connectivity as revealed by independent component analysis of resting-state fNIRS measurements
NeuroImage, 51
T. McKenna, T. Mcmullen, M. Shlesinger (1994)
The brain as a dynamic physical system
Neuroscience, 60
(Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884–898). Lancet Publishing Group. 10.1016/S0140-6736(14)60456-6)
Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884–898). Lancet Publishing Group. 10.1016/S0140-6736(14)60456-6
Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884–898). Lancet Publishing Group. 10.1016/S0140-6736(14)60456-6, Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884–898). Lancet Publishing Group. 10.1016/S0140-6736(14)60456-6
J. Hughes (2008)
Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior
Epilepsy & Behavior, 13
N. Logothetis, J. Pauls, M. Augath, T. Trinath, A. Oeltermann (2001)
Neurophysiological investigation of the basis of the fMRI signal
Nature, 412
(Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and optimizing LSTM language models. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.)
Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and optimizing LSTM language models. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and optimizing LSTM language models. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings., Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and optimizing LSTM language models. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
L. Gagnon, R. Cooper, M. Yücel, Katherine Perdue, D. Greve, D. Boas (2012)
Short separation channel location impacts the performance of short channel regression in NIRS
NeuroImage, 59
G. Deco, Viktor Jirsa, A. Mcintosh (2010)
Emerging concepts for the dynamical organization of resting-state activity in the brain
Nature Reviews Neuroscience, 12
Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations
F. Scholkmann, S. Kleiser, A. Metz, Raphael Zimmermann, J. Pavia, U. Wolf, M. Wolf (2014)
A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology
NeuroImage, 85
P. Welch (1967)
The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms
IEEE Transactions on Audio and Electroacoustics, 15
D. Mantini, M. Perrucci, C. Gratta, G. Romani, M. Corbetta (2007)
Electrophysiological signatures of resting state networks in the human brain
Proceedings of the National Academy of Sciences, 104
R. Dhamija, B. Moseley, G. Cascino, E. Wirrell (2011)
A population‐based study of long‐term outcome of epilepsy in childhood with a focal or hemispheric lesion on neuroimaging
Epilepsia, 52
C. Julien (2006)
The enigma of Mayer waves: Facts and models.
Cardiovascular research, 70 1
Yasutaka Honda, Shinya Nakamura, K. Ogawa, Rintaro Yoshino, P. Tobler, Y. Nishimura, K. Tsutsui (2021)
Changes in beta and high-gamma power in resting-state electrocorticogram induced by repetitive transcranial magnetic stimulation of primary motor cortex in unanesthetized macaque monkeys
Neuroscience Research, 171
B. He, Zhongming Liu (2008)
Multimodal Functional Neuroimaging: Integrating Functional MRI and EEG/MEG
IEEE Reviews in Biomedical Engineering, 1
A. Chiarelli, F. Zappasodi, F. Pompeo, A. Merla (2017)
Simultaneous functional near-infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: a review
Neurophotonics, 4
F. Pizzo, B. Frauscher, T. Ferrari-Marinho, M. Amiri, F. Dubeau, J. Gotman (2016)
Detectability of Fast Ripples (>250 Hz) on the Scalp EEG: A Proof-of-Principle Study with Subdermal Electrodes
Brain Topography, 29
A. Salek-Haddadi, Karl Friston, L. Lemieux, D. Fish (2003)
Studying spontaneous EEG activity with fMRI
Brain Research Reviews, 43
M. Yücel, J. Selb, C. Aasted, P. Lin, D. Borsook, L. Becerra, D. Boas (2016)
Mayer waves reduce the accuracy of estimated hemodynamic response functions in functional near-infrared spectroscopy.
Biomedical optics express, 7 8
Zhiqiang Zhang, G. Lu, Y. Zhong, Qi-fu Tan, Zhigen Yang, W. Liao, Zhili Chen, Ji-Xin Shi, Yijun Liu (2009)
Impaired attention network in temporal lobe epilepsy: A resting FMRI study
Neuroscience Letters, 458
(Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports, 7(1). 10.1038/s41598-017-06340-7)
Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports, 7(1). 10.1038/s41598-017-06340-7
Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports, 7(1). 10.1038/s41598-017-06340-7, Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports, 7(1). 10.1038/s41598-017-06340-7
Tomas Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur (2010)
Recurrent neural network based language model
(Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning. 10.1145/1390156.1390294)
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning. 10.1145/1390156.1390294
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning. 10.1145/1390156.1390294, Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning. 10.1145/1390156.1390294
(Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised learning of video representations using LSTMs. 32nd International Conference on Machine Learning, ICML 2015.)
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised learning of video representations using LSTMs. 32nd International Conference on Machine Learning, ICML 2015.
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised learning of video representations using LSTMs. 32nd International Conference on Machine Learning, ICML 2015., Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised learning of video representations using LSTMs. 32nd International Conference on Machine Learning, ICML 2015.
F. Zhao, Jiashi Feng, Jian Zhao, Wenhan Yang, Shuicheng Yan (2016)
Robust LSTM-Autoencoders for Face De-Occlusion in the Wild
IEEE Transactions on Image Processing, 27
(Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 10.3115/v1/d14-1179)
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 10.3115/v1/d14-1179
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 10.3115/v1/d14-1179, Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. 10.3115/v1/d14-1179
R. Canolty, E. Edwards, S. Dalal, M. Soltani, S. Nagarajan, H. Kirsch, M. Berger, N. Barbaro, R. Knight (2006)
High Gamma Power Is Phase-Locked to Theta Oscillations in Human Neocortex
Science, 313
(2020)
Electroencephalography
Handbook of Clinical Neurology
N. Truong, Luping Zhou, O. Kavehei (2019)
Semi-supervised Seizure Prediction with Generative Adversarial Networks
2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
K. Whittingstall, N. Logothetis (2009)
Frequency-Band Coupling in Surface EEG Reflects Spiking Activity in Monkey Visual Cortex
Neuron, 64
Anirban Dutta, Athira Jacob, S. Chowdhury, Abhijit Das, M. Nitsche (2015)
EEG-NIRS Based Assessment of Neurovascular Coupling During Anodal Transcranial Direct Current Stimulation - a Stroke Case Series
Journal of Medical Systems, 39
R. Sigala, S. Haufe, Dipanjan Roy, H. Dinse, P. Ritter (2014)
The role of alpha-rhythm states in perceptual learning: insights from experiments and computational models
Frontiers in Computational Neuroscience, 8
E. Kobayashi, A. Bagshaw, C. Grova, F. Dubeau, J. Gotman (2006)
Negative BOLD responses to epileptic spikes
Human Brain Mapping, 27
P. Sirpal, Ali Kassab, P. Pouliot, D. Nguyen, F. Lesage (2019)
fNIRS improves seizure detection in multimodal EEG-fNIRS recordings
Journal of Biomedical Optics, 24
S. Kohno, I. Miyai, A. Seiyama, I. Oda, A. Ishikawa, Shoichi Tsuneishi, T. Amita, K. Shimizu (2007)
Removal of the skin blood flow artifact in functional near-infrared spectroscopic imaging data through independent component analysis.
Journal of biomedical optics, 12 6
S. Koch, Sophie Koendgen, R. Bourayou, J. Steinbrink, H. Obrig (2008)
Individual alpha-frequency correlates with amplitude of visual evoked potential and hemodynamic response
NeuroImage, 41
Ilya Sutskever, O. Vinyals, Quoc Le (2014)
Sequence to Sequence Learning with Neural Networks
ArXiv, abs/1409.3215
(Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the dynamical organization of resting-state activity in the brain. In Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Publishing Group. 10.1038/nrn2961)
Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the dynamical organization of resting-state activity in the brain. In Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Publishing Group. 10.1038/nrn2961
Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the dynamical organization of resting-state activity in the brain. In Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Publishing Group. 10.1038/nrn2961, Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the dynamical organization of resting-state activity in the brain. In Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Publishing Group. 10.1038/nrn2961
(Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). Audio Word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 10.21437/Interspeech.2016-82)
Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). Audio Word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 10.21437/Interspeech.2016-82
Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). Audio Word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 10.21437/Interspeech.2016-82, Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). Audio Word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 10.21437/Interspeech.2016-82
(Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting-state fNIRS scanning duration on functional brain connectivity and graph theory metrics of brain network. Frontiers in Neuroscience, 11(JUL). 10.3389/fnins.2017.00392)
Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting-state fNIRS scanning duration on functional brain connectivity and graph theory metrics of brain network. Frontiers in Neuroscience, 11(JUL). 10.3389/fnins.2017.00392
Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting-state fNIRS scanning duration on functional brain connectivity and graph theory metrics of brain network. Frontiers in Neuroscience, 11(JUL). 10.3389/fnins.2017.00392, Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting-state fNIRS scanning duration on functional brain connectivity and graph theory metrics of brain network. Frontiers in Neuroscience, 11(JUL). 10.3389/fnins.2017.00392
(Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Multi-task Sequence to Sequence Learning. c, 1–10. http://arxiv.org/abs/1511.06114)
Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Multi-task Sequence to Sequence Learning. c, 1–10. http://arxiv.org/abs/1511.06114
Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Multi-task Sequence to Sequence Learning. c, 1–10. http://arxiv.org/abs/1511.06114, Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Multi-task Sequence to Sequence Learning. c, 1–10. http://arxiv.org/abs/1511.06114
J. Tracy, G. Doucet (2015)
Resting-state functional connectivity in epilepsy: growing relevance for clinical decision making.
Current opinion in neurology, 28 2
P. Franaszczuk, P. Kudela, G. Bergey (2003)
External excitatory stimuli can terminate bursting in neural network models
Epilepsy Research, 53
B. Biswal (2015)
Resting-State Functional Connectivity
M. Richardson (2012)
Large scale brain models of epilepsy: dynamics meets connectomics
Journal of Neurology, Neurosurgery & Psychiatry, 83
M. Luca, C. Beckmann, N. Stefano, P. Matthews, Stephen Smith (2006)
fMRI resting state networks define distinct modes of long-distance interactions in the human brain
NeuroImage, 29
Gaelle Bettus, E. Guedj, Florian Joyeux, S. Confort-Gouny, E. Soulier, V. Laguitton, P. Cozzone, P. Chauvel, J. Ranjeva, F. Bartolomei, M. Guye (2009)
Decreased basal fMRI functional connectivity in epileptogenic networks and contralateral compensatory mechanisms
Human Brain Mapping, 30
(Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. NeuroImage: Clinical. 10.1016/j.nicl.2016.07.006)
Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. NeuroImage: Clinical. 10.1016/j.nicl.2016.07.006
Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. NeuroImage: Clinical. 10.1016/j.nicl.2016.07.006, Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. NeuroImage: Clinical. 10.1016/j.nicl.2016.07.006
(Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations for the use of multimodal monitoring in the neurointensive care unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. 113–119). Lippincott Williams and Wilkins. 10.1097/MCC.0000000000000179)
Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations for the use of multimodal monitoring in the neurointensive care unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. 113–119). Lippincott Williams and Wilkins. 10.1097/MCC.0000000000000179
Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations for the use of multimodal monitoring in the neurointensive care unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. 113–119). Lippincott Williams and Wilkins. 10.1097/MCC.0000000000000179, Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations for the use of multimodal monitoring in the neurointensive care unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. 113–119). Lippincott Williams and Wilkins. 10.1097/MCC.0000000000000179
K. Schwab, H. Skupin, M. Eiselt, M. Walther, A. Voß, H. Witte (2009)
Coordination of the EEG and the heart rate of preterm neonates during quiet sleep
Neuroscience Letters, 465
H. Laufs, K. Krakow, P. Sterzer, E. Eger, A. Beyerle, A. Salek-Haddadi, A. Kleinschmidt (2003)
Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest
Proceedings of the National Academy of Sciences of the United States of America, 100
(Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. 10.1145/3209978.3210184)
Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. 10.1145/3209978.3210184
Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. 10.1145/3209978.3210184, Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. 10.1145/3209978.3210184
R. Socher, Jeffrey Pennington, E. Huang, A. Ng, Christopher Manning (2011)
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
Xiaodi Zhang, W. Pan, S. Keilholz (2019)
The relationship between BOLD and neural activity arises from temporally sparse events
bioRxiv
Zachary Lipton, David Kale, C. Elkan, R. Wetzel (2015)
Learning to Diagnose with LSTM Recurrent Neural Networks
CoRR, abs/1511.03677
Klaus Greff, R. Srivastava, J. Koutník, Bas Steunebrink, J. Schmidhuber (2015)
LSTM: A Search Space Odyssey
IEEE Transactions on Neural Networks and Learning Systems, 28
D. Dash, C. Dash, Sara Primrose, L. Hernández-Ronquillo, F. Moien-Afshari, L. Ladino, J. Appendino, Lindsay Mazepa, Christine Elliott, S. Mirsattari, P. Federico, E. Bui, G. Hunter, R. Ramachandrannair, Rohit Sharma, Paula Melendres, J. Nikkel, D. Nguyen, S. Almubarak, Michael Rigby, J. Téllez-Zenteno (2017)
Update on Minimal Standards for Electroencephalography in Canada: A Review by the Canadian Society of Clinical Neurophysiologists
Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques, 44
Kyunghyun Cho, B. Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio (2014)
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Stephen Merity, N. Keskar, R. Socher (2017)
Regularizing and Optimizing LSTM Language Models
ArXiv, abs/1708.02182
R. Sitaram, T. Ros, L. Stoeckel, S. Haller, F. Scharnowski, Jarrod Lewis-Peacock, N. Weiskopf, M. Blefari, Mohit Rana, Ethan Oblak, N. Birbaumer, J. Sulzer (2017)
Closed-loop brain training: the science of neurofeedback
Nature Reviews Neuroscience, 18
Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-yi Lee, Lin-Shan Lee (2016)
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder
Jörn Niessing, B. Ebisch, K. Schmidt, M. Niessing, W. Singer, R. Galuske (2005)
Hemodynamic Signals Correlate Tightly with Synchronized Gamma Oscillations
Science, 309
A. Shmuel, D. Leopold (2008)
Neuronal correlates of spontaneous fluctuations in fMRI signals in monkey visual cortex: Implications for functional connectivity at rest
Human Brain Mapping, 29
Y. Tong, L. Hocke, Stephanie Licata, Blaise Frederick (2012)
Low-frequency oscillations measured in the periphery with near-infrared spectroscopy are strongly correlated with blood oxygen level-dependent functional magnetic resonance imaging signals
Journal of Biomedical Optics, 17
Hayley Thair, Amy Holloway, R. Newport, Alastair Smith (2017)
Transcranial Direct Current Stimulation (tDCS): A Beginner's Guide for Design and Implementation
Frontiers in Neuroscience, 11
(Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research.)
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research., Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research.
Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol (2010)
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
J. Mach. Learn. Res., 11
(Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what is currently thought? In Acta Physiologica. 10.1111/j.1748-1716.2011.02320.x)
Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what is currently thought? In Acta Physiologica. 10.1111/j.1748-1716.2011.02320.x
Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what is currently thought? In Acta Physiologica. 10.1111/j.1748-1716.2011.02320.x, Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what is currently thought? In Acta Physiologica. 10.1111/j.1748-1716.2011.02320.x
Helen Shen (2015)
Core Concept: Resting-state connectivity
Proceedings of the National Academy of Sciences, 112
T. Ros, Bernard Baars, R. Lanius, P. Vuilleumier
Human Neuroscience Hypothesis and Theory Article Tuning Pathological Brain Oscillations with Neurofeedback: a Systems Neuroscience Framework
Fabian Pedregosa, G. Varoquaux, Alexandre Gramfort, V. Michel, B. Thirion, O. Grisel, Mathieu Blondel, Gilles Louppe, P. Prettenhofer, Ron Weiss, Ron Weiss, J. Vanderplas, Alexandre Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay (2011)
Scikit-learn: Machine Learning in Python
ArXiv, abs/1201.0490
F. Amzica, M. Steriade (1998)
Electrophysiological correlates of sleep delta waves.
Electroencephalography and clinical neurophysiology, 107 2
(2014)
Tuning pathological brain oscillations with neurofeedback: A systems neuroscience framework
Frontiers in Human Neuroscience, 8
F. Sharbrough, J. Messick, T. Sundt (1973)
Correlation of Continuous Electroencephalograms With Cerebral Blood Flow Measurements During Carotid Endarterectomy
Stroke, 4
(Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.)
Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.
Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference., Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.
J. Siegel, A. Snyder, L. Ramsey, G. Shulman, M. Corbetta (2016)
The effects of hemodynamic lag on functional connectivity and behavior after stroke
Journal of Cerebral Blood Flow & Metabolism, 36
(Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based LSTM for aspect-level sentiment classification. EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings. 10.18653/v1/d16-1058)
Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based LSTM for aspect-level sentiment classification. EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings. 10.18653/v1/d16-1058
Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based LSTM for aspect-level sentiment classification. EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings. 10.18653/v1/d16-1058, Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based LSTM for aspect-level sentiment classification. EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings. 10.18653/v1/d16-1058
(Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien-Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on minimal standards for electroencephalography in Canada: A review by the Canadian Society of Clinical Neurophysiologists. In Canadian Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). Cambridge University Press. 10.1017/cjn.2017.217)
Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien-Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on minimal standards for electroencephalography in Canada: A review by the Canadian Society of Clinical Neurophysiologists. In Canadian Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). Cambridge University Press. 10.1017/cjn.2017.217
Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien-Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on minimal standards for electroencephalography in Canada: A review by the Canadian Society of Clinical Neurophysiologists. In Canadian Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). Cambridge University Press. 10.1017/cjn.2017.217, Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien-Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on minimal standards for electroencephalography in Canada: A review by the Canadian Society of Clinical Neurophysiologists. In Canadian Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). Cambridge University Press. 10.1017/cjn.2017.217
Supratim Ray, J. Maunsell (2010)
Differences in Gamma Frequencies across Visual Cortex Restrict Their Possible Use in Computation
Neuron, 67
A. Phillips, J. Squair, D. Sayenko, V. Edgerton, Y. Gerasimenko, A. Krassioukov (2017)
An Autonomic Neuroprosthesis: Noninvasive Electrical Spinal Cord Stimulation Restores Autonomic Cardiovascular Function in Individuals with Spinal Cord Injury
Journal of Neurotrauma, 35
(Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010.)
Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010.
Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010., Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010.
(Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert law revisited. Physics in Medicine and Biology, 51(5). 10.1088/0031-9155/51/5/N02)
Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert law revisited. Physics in Medicine and Biology, 51(5). 10.1088/0031-9155/51/5/N02
Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert law revisited. Physics in Medicine and Biology, 51(5). 10.1088/0031-9155/51/5/N02, Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert law revisited. Physics in Medicine and Biology, 51(5). 10.1088/0031-9155/51/5/N02
Alexandre Gramfort, Martin Luessi, E. Larson, D. Engemann, D. Strohmeier, Christian Brodbeck, L. Parkkonen, M. Hämäläinen (2014)
MNE software for processing MEG and EEG data
NeuroImage, 86
C. Aalkjaer, D. Boedtkjer, V. Matchkov (2011)
Vasomotion – what is currently thought?
Acta Physiologica, 202
W. Freeman, B. Burke, M. Holmes (2003)
Aperiodic phase re‐setting in scalp EEG of beta–gamma oscillations by state transitions at alpha–theta rates
Human Brain Mapping, 19

Publisher: Springer Journals
Copyright: Copyright © The Author(s) 2021
ISSN: 1539-2791
eISSN: 1559-0089
DOI: 10.1007/s12021-021-09538-3
Publisher site: See Article on Publisher Site

Abstract

In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model’s fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals. Keywords EEG-fNIRS · Functional brain imaging · Deep neural networks · Epilepsy · Resting state · Functional connectivity · Neurovascular coupling Introduction on the surface of the subject’s head. Infrared light emitted from the light source is absorbed or scattered as it enters Functional near infrared spectroscopy (fNIRS) is a non- cerebral tissue. Detected light is used to calculate the blood invasive, mobile, and cost-effective neuroimaging tech - oxygenation changes associated with cerebral hemodynamic nology that uses near infrared light to continually monitor activity using the modified Beer-Lambert law (Kocsis et al., changes in cerebral hemodynamic parameters (i.e., oxygen- 2006; Scholkmann et al., 2014). Concentration changes in ated (HbO) and deoxygenated hemoglobin (HbR), and total the oxygenation of hemoglobin quantifies the absorption of hemoglobin (HbT)) (Jobsis, 1977). The fNIRS method relies infrared light by the brain. on the neurovascular coupling phenomenon which describes The fNIRS method offers several advantages as an alter - the intimate spatial and temporal relationship between neu- native or complement to other functional imaging techniques ral activity and cerebral blood flow to map acute functional (i.e., fMRI) (Strangman et al., 2002). fNIRS offers increased changes in the brain (Girouard & Iadecola, 2006). In a typi- temporal resolution as compared to fMRI, and fNIRS hard- cal fNIRS setup, optodes corresponding to near-infrared ware can be integrated with other modalities such as scalp light sources and their complimentary detectors are placed electroencephalography (EEG) (Fazli et al., 2012; Khan & Hong, 2017; Miller, 2012). fNIRS signals have been recently used in studying brain state decoding as well as proven use- * Parikshat Sirpal ful for brain computer interfacing over the last decade (Hong parikshat.sirpal@polymtl.ca et al., 2015; Khan & Hong, 2015). École Polytechnique de Montréal, Université de Montréal, Scalp EEG technology is the clinical gold standard for stud- C.P. 6079, Succ. Centre-Ville, Montréal H3C 3A7, Canada ying the human brain (Müller-Putz, 2020) and EEG recordings Neurology Division, Centre Hospitalier de L’Université de can be classified into specific frequency bands: alpha, beta, Montréal (CHUM), 1000 Saint-Denis, Montréal H2X 0C1, delta, gamma, and theta (Cho et al., 2014; Freeman et al., 2003; Canada Pedregosa et al., 2011; Zhao et al., 2018). The delta frequency Research Centre, Montréal Heart Institute, Montréal, Canada Vol.:(0123456789) 1 3 538 Neuroinformatics (2022) 20:537–558 range encompasses low frequencies with relatively high ampli- neural activity believed to reflect its functional organization tude and slow waveforms ranging from 0.25–3.0 Hz. Delta (Rojas et al., 2018; Tracy & Doucet, 2015). The interde- frequencies are common in normal sleep and may inciden- pendence of each component (i.e., neural and vascular) is a tally appear with focal lesions, metabolic encephalopathy, or topic of interest to the wider clinical and neuroscience com- hydrocephalus (Amzica & Steriade, 1998; Hofle et al., 1997; munity. fMRI studies have shown that resting state networks Knyazev, 2012). The theta band includes frequencies between in the epileptic brain undergo changes in their functional 4 and 7 Hz. While normal in young individuals, the theta fre- architecture (Luo et al., 2011; Wang et al., 2011). Increas- quency envelope is interpreted as slow activity in awake adults ingly, “task-free” resting state conditions in fMRI studies (Mantini et al., 2007; Pizzo et al., 2016; Sitnikova et al., 2016). have been conducted with the assumption that functionally As with delta waves, theta waves may be seen in focal lesions connected brain networks show similar profiles of activity or in a more generalized distribution in diffuse neurological over time (De Luca et al., 2006; He & Liu, 2008; Niu & He, disorders. Alpha frequencies are between 8 and 13 Hz, rep- 2014; Palva et al., 2010; Richardson, 2012; Shen, 2015). resenting the dominant rhythm in awake adults (Koch et al., In the context of epilepsy, resting state fMRI studies have 2008; Sigala et al., 2014). Beta activity ranges in frequency shown that functional networks are abnormal (Bettus et al., between 14–30 Hz and is usually observed in a bilaterally fron- 2009; Honda et al., 2021; Tracy & Doucet, 2015; Zhang et al., tal symmetrical distribution (Canolty et al., 2006; Freeman 2009, 2010a, b). Pre-clinical studies have proposed that there et al., 2003; Merker, 2016). Higher frequency ranges represent is a correlation between slow fluctuations in the resting state gamma wave oscillations between 30–100 Hz. Gamma activ- BOLD signal (~0.1 Hz) and slow fluctuations in neuronal fir - ity is seen during a wide range of activities, and is enhanced ing rates in gamma band local field potentials (Richardson, in rapid eye movement during sleep (Gross & Gotman, 1999; 2012; Shmuel & Leopold, 2008; Zhang et al., 2020). This Hughes, 2008). suggests that the resting state is related to physiologically Multimodal EEG-fNIRS experimental setups record the active dynamic neuronal processes. Utilizing fNIRS signals spatiotemporal dynamics of brain activity, provide opportu- for resting state functional connectivity has gained attention nities to observe the population dynamics of neural ensem- as a promising imaging tool to study brain function and pro- bles and offer increased benefit in fundamental and clinical vide valuable insight into the intrinsic networks present within analyses (Goldman et al., 2002; Laufs et al., 2003; Martinez- the human epileptic brain (Fishburn et al., 2014; Geng et al., Montes et al., 2004; McKenna et al., 1994; Salek-Haddadi 2017; Niu & He, 2014; Wang et al., 2017). et al., 2003). In such setups, scalp EEG measures the brain’s In this study, we hypothesize that we can predict brain electrical activity, and fNIRS signals encode the brain’s hemodynamics from electrical signals using a deep learn- hemodynamic response (Chiarelli et al., 2017; Ogawa et al., ing architecture from resting state multimodal EEG-fNIRS 1992), with a delay of approximately 3 seconds post neural recordings collected from a cohort of 40 epileptic patients. activity. Data from EEG-fNIRS setups have established causality Following which, we hypothesize that functional connectivity between neuronal firing and changes in HbO, HbR, and HbT, patterns derived from higher EEG frequency envelopes are reflecting electrical and hemodynamic fluctuations dictated increased as compared to lower EEG frequency envelopes. by neurovascular coupling (Hughes, 2008; Logothetis et al., 2001; Mukamel et al., 2005; Singh, 2012). Recent interest has focused on determining spatial hemodynamic correlates from Methods EEG recorded activity, particularly, in the blood oxygen level dependent signal (BOLD) (Czisch et al., 2004; Lemieux et al., Subjects and Protocol 2001; Lövblad et al., 1999). Resting state studies have suc- cessfully demonstrated that low frequency EEG band signals Forty patients (27 males, 13 females; ranging in age of 11 are negatively correlated with modulations in the BOLD sig- to 62 years in age; mean age of 32.42 years, and standard nal, particularly, infra-low gamma EEG band envelopes (Jia deviation of 13.97 years) with refractory focal epilepsy were & Kohn, 2011; Niessing et al., 2005; Sumiyoshi et al., 2012). recruited for prolonged EEG-fNIRS recordings. Epilepsy The characterization of the relationship between electro- diagnosis and epileptic focus localization was based on a physiology and cerebral hemodynamics is clinically relevant comprehensive evaluation which included clinical history, in epilepsy. Seizures are self-terminating paroxysmal rep- video-EEG recording of interictal spikes and seizures, mag- resentations of aberrant brain activity (Moshé et al., 2015). netic resonance imaging (MRI), positron emission tomogra- It is believed that the neurovascular machinery causing phy (PET) and for some patients ictal single photon emission seizures is similarly present in the brain interictally during computed tomography (SPECT) and magnetoencephalog- normal function, suggesting to some extent that epilepsy raphy (MEG) scans. Full details regarding patient profiles is a dynamic disorder (Kobayashi et al., 2006; Richardson, including age, gender, EEG and MRI findings are found in 2012). The resting epileptic brain displays spontaneous Table 1 of (Peng et al., 2014; Sirpal et al., 2019). A subset 1 3 Neuroinformatics (2022) 20:537–558 539 Table 1 Detailed overview of the proposed convolutional neural net- lutions have kernels of size (1,2), and thus their effect is along the work long-short term autoencoder (CNN-LSTM AE) model. The time dimension. Convolutions help in generating embeddings with network receives as input resting state EEG time-series sequences, higher level abstraction of the input EEG sequence. Deconvolutions represented as a single matrix, and is trained to reconstruct the corre- reconstruct the fNIRS sequence at full resolution based on output sponding fNIRS resting state output. Model specifications and hyper - embeddings. The decoder and encoder LSTM units have ReLU (Rec- parameters were heuristically determined. Convolutions and deconvo- tified linear units) activations Layer Description Output size Input EEG sample sequence (EEG sequence length, number of time points is 500, number of EEG channels is 21) EEG Sequence Embedding 2-Dimensional convolution + Average Pooling + 2Dconvolution ∶ stride =(1,2); (EEG sequence length, Dropout kernelsize =(1,7); 125, number of Features Maps) ReLUactivation. Dropout ∶ 20%. AveragePoolingkernel ∶(1,2) 2-Dimensional convolution + Average Pooling + 2Dconvolution ∶ stride =(1,2); (EEG sequence length, Dropout kernelsize =(1,7); 62, number of Features Maps) ReLUactivation. Dropout ∶ 20%. AveragePoolingkernel ∶(1,2) Reshape Reshape into an elongated tensor (EEG sequence length, 62 * number of Features Maps) Encoder LSTM 1 + Dropout An LSTM layer with number of cells equal to the number of (EEG sequence length, 512) EEG sequence length. ReLU activation. Dropout: 20% LSTM 2 + Dropout An LSTM layer with number of cells equal to the number of (1, 256) EEG sequence length. ReLU activation. Dropout: 20% Decoder Repeat Create repeated version of the latent vector (fNIRS sequence length, 256) LSTM 3 + Dropout An LSTM layer with number of cells equal to the number of (fNIRS sequence length, 312) EEG sequence length. ReLU activation. Dropout: 20% LSTM 4 + Dropout An LSTM layer with number of cells equal to the number of (fNIRS sequence length, 695) EEG sequence length. ReLU activation. Dropout: 20% fNIRS Sequence Reconstruction Reshape Reshape into a 2D tensor (fNIRS sequence length, 5, number of Feature Maps 3) Deconvolution1 + Dropout 2Ddeconvolution ∶ stride =(1,2);ReLUactivation; (fNIRS sequence length, 10, kernelsize =(1,2).Dropout ∶ 20%. number of Feature Maps 4) Deconvolution2 + Dropout 2Ddeconvolution ∶ stride =(1,2);ReLUactivation; (fNIRS sequence length, 20, kernelsize =(1,2).Dropout ∶ 20%. number of fNIRS channels) of patients had MRI evidence of encephalomalacia, cortical noise and lights were maintained at a minimum for patient com- dysplasia, and/or hippocampal atrophy, a common finding in fort during data acquisition. Patients were further instructed epilepsy, but this was neither an inclusion nor exclusion cri- to remain calm and placed in comfortable, climate-controlled terion. The presence of such findings (Dhamija et al., 2011; ambulatory suites with curtains drawn to limit ambient light. Woermann & Vollmar, 2009) is a common MRI finding in Patients were continually telemonitored by trained clinical epileptic brains. The institutional review boards of Sainte- staff. fNIRS data was collected using the Imagent Tissue Justine Hospital and Centre Hospitalier de l’Université de Oximeter system (ISS Inc.), a multi-channel frequency Montréal approved the study. domain system recording at 19.5 Hz with wavelengths of 690 nm and 830 nm for sensitivity to HbR and HbO respec- EEG‑fNIRS Data Acquisition and Pre‑Processing tively. EEG data was recorded according to the standard 10–20 system using 21 electrodes (in positions Fp1, Fp2, Continuous EEG-fNIRS recordings were performed at the F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, Optical Imaging Laboratory of Sainte-Justine Hospital in P8, O1, O2) at 500 Hz (Neuroscan Synamps 2TM system). Montréal, Canada. Experimental protocol ensured ambient Custom-made helmets, taking into consideration different 1 3 540 Neuroinformatics (2022) 20:537–558 head sizes and shapes were made to fit comfortably. Plastic signal to noise ratio (SNR) threshold applied in channel and polyvinyl chloride manufacturing materials made them analysis was defined as those channels less than 30% of the rigid and light. The helmets were equipped with a total of mean SNR of all channels. fNIRS channels deemed to have 64 light sources, 16 light detectors and 19 EEG electrodes SNR were eliminated and not included for analysis. This that allowed for stable optical coupling between cortical led to an average of 138 channels per patient. Changes in regions and the scalp. This further prevented inter-optode HbO and HbR were calculated via the HomER and MNE shifting and movement artifacts to a large extent. Sensitiv- software packages (Gramfort et al., 2014; Huppert et al., ity of near-infrared light to cortical tissue was maintained 2009). by positioning the optodes approximately 3–4 cm apart. Multiple consecutive recordings were performed, with Electrodes were placed following the 10–20 EEG instru- each recording approximately 15 min led to a compendium mentation standard, allowing for full head coverage (Barlow of 200 recordings totaling 50 hours of recording time. Data was et al., 1974). Figure 1 shows the EEG-fNIRS helmet placed bandpass filtered in the 0.01 to 0.1 Hz frequency range to on a patient’s head. be in the resting state range (Tong et al., 2012). The resting EEG data was bandpass filtered between 0.1–100 Hz state period (indexed from patients when they were resting to remove instrumental noise and to remove drift related comfortably) ranged between 7 to 10 minutes with a mean of to physiological activity, particularly, higher frequencies. 8.35 minutes (Geng et al., 2017; Li et al., 2015; Zhang et al., The unprocessed raw time series of the HbO and HbR 2010a, b). To correct for motion, we performed dimension signals was bandpass filtered to remove specific frequency reduction via principal component analysis on EEG-fNIRS components attributed to cardiac (approximately 1 Hz) data and removed components with the most variance. Fur- and/or respiratory activity (approximately 0.2–0.3 Hz) ther, artifact rejection with (10% variation from normalized (Gramfort et al., 2014; Lu et al., 2010; Peng et al., 2014). intensity) was applied to remove additional motion artifacts. Signal fidelity was examined prior to analysis by channel- Artifact-free data points were then filtered for the effects of wise verification of signal intensity. Bandpass filtering respiratory and cardiac signal with a cutoff frequency of was applied to EEG data to compute frequency bands of 0.2 Hz. Finally, HbO concentrations were calculated for each interest. We used a FIR bandpass filter and the lowcut and channel using the modified Beer–Lambert law. highcut values (Hz) for the delta, theta, alpha, beta, and Structural MRI registration of optode and electrode posi- gamma frequencies were set as: [1, 4], [4, 8], [8, 12], [12, tion was done using neuro-navigation (Brainsight, Rogue- 30], [30, 100] respectively (Gramfort et al., 2014). The Research Inc.). Channel positions were cross-referenced Fig. 1 EEG and fNIRS channel-configuration and custom-made mul - channel configuration, as well as the EEG (which are in blue dots), timodal EEG-fNIRS helmets used for EEG and fNIRS data acquisi- are superimposed on the patient’s MRI. We used a 3D camera and tion. Helmets of different sizes and shapes to fit patients’ head com - stereotaxic system (Frameless 39 from Rogue research) to determine fortably were made from plastic and polyvinyl chloride making them the 3-D coordinates of the optodes relative to the patient’s anatomical rigid and light. The EEG-fNIRS configuration allows for full head MRI coverage and follows the 10–20 EEG placement system. The fNIRS 1 3 Neuroinformatics (2022) 20:537–558 541 with the patient MRI and adapted to ensure coverage of the receives the feature vector and decodes it into the original epileptic focus, the contralateral homologous region, and as input sequences. LSTM-AEs learn a compressed represen- much area as possible of other brain regions. The MRI was tation of sequential data and have been used in video, text, segmented into six different layers: air, scalp, skull, CSF, audio, and time series sequence data (Lipton et al., 2016; gray matter and white matter. The gray matter layer was Srivastava et al., 2015; Wang et al., 2016). In this work, used to extract six two-dimensional cortical projections. The multiple LSTM layers were incorporated to learn tempo- three-dimensional position of each channel was projected ral representations. Our model also includes convolutional onto these two-dimensional topographic maps, of which the layers to extract high level spatial percepts from channel following views were considered: dorsal, frontal, left and combinations. We input EEG sequential data accounting right views. for hemodynamic delays to perform sequence-to-sequence encoding (Luong et al., 2015; Truong et al., 2018; Vincent Neural Network Architecture et al., 2008; Zhang, 2018). These input EEG sequences are convolved by two convolutional neural networks (CNN) and We built a deep sequence-to-sequence multimodal autoen- subsequently fed into the first two encoding long short-term coder to predict fNIRS signals from input scalp EEG signals. memory (LSTM) modules. EEG data samples are projected Autoencoders are powerful machine learning models trained in the latent space with fixed length vectors that provide in a self-supervised fashion to reconstruct inputs by learning more compressed representations, which are then used to their abstract representations (Kocsis et al., 2006; Lindauer decode and reconstruct the output fNIRS data, by the LSTM et al., 2010; Socher et al., 2011; Vincent et al., 2010). The decoding modules. autoencoder embedded signals in a low dimensional latent After testing multiple architectures with exhaustive space, where both the encoder and decoder are formulated hyper-parameter optimization, we designed our model as as deep neural networks. follows: The encoder is comprised of LSTM layers preceded Recurrent neural networks (RNN) have been widely used by convolutional blocks. Convolutions in each block have a in time series modeling since they account for the tem- kernel size of (1, 7) and stride size of (1, 2). The decoder poral state within data (Baytas et al., 2017; Chung et al., is comprised of LSTM layers which manipulate the vec- 2016; Merity et al., 2018; Mikolov et al., 2010). The output tors in the latent space to provide a final output dimension depends on hidden states and feedback connections present equal to that of an fNIRS sample. We evaluated our model within hidden units. Previous states can be used as inputs, in terms of cross-modal reconstruction error (Zhao et al., thereby allowing RNNs to hold memory. In our model, we 2018), denoted as RE. The objective is to simultaneously used backpropagation through time, a common gradient minimize the distance between fNIRS data samples and descent type training technique (Sutskever et al., 2014). The maximize the distance between each fNIRS and EEG data innate problem of RNN gradient based training is that deriv- points (i.e., minimizing the RE is equivalent to maximizing atives propagated via recurrent connections either become the likelihood function). Once the model was trained, the exceedingly small or large (Goodfellow et al., 2016; Luong corresponding RE was calculated on an independent testing et al., 2015), causing a vanishing or exploding gradient subset (see below) by computing the sum of the Euclidean respectively. Long short-term memory units (LSTM), a vari- distance between x and its corresponding reconstruction, x , ant of the vanilla RNN architecture overcomes the vanishing over all L dimensions, as expressed in Eq. 1 below: gradient problem (Greff et al., 2017; Gregor et al., 2015; Lecun et al., 2015). LSTM units receive external inputs and ∈ = x − x , t T (1) t t,l t,l generate hidden outputs via input, output, and forget gates l=1 and a memory cell. The gates and memory cell are internally Model output is denoted as x . connected with weighted links. The gates are connected with EEG data is processed as follows. First, matching EEG external sources, which are current state sequential inputs and fNIRS data are parsed from our data directory, following and previous hidden states. This prevents the LSTM from which the respective data (EEG or fNIRS) is labeled accord- storing useless or noisy input information (Greff et al., 2017; ing to the resting state periods. Feature scaling is performed Gregor et al., 2015; Lecun et al., 2015). using the MinMaxScaler class (Pedregosa et al., 2011) on The LSTM autoencoder model (LSTM-AE) as proposed EEG input data which sets the range of values between 0 and by Srivastava et al. consists of encoder LSTM units and 1. Input signals are mean centered prior to being fed into the decoder LSTM units (Srivastava et al., 2015). The encoder model. Then, data is fed into the convolutional layers and LSTM receives input sequences and encodes them into a travels to the LSTM and deconvolution modules. A detailed feature vector as the LSTM generates hidden outputs (Lipton schematic view of our model is shown in Fig. 2 below. et al., 2016; Wang et al., 2016). Likewise, the decoder LSTM 1 3 542 Neuroinformatics (2022) 20:537–558 Backpropagation through time was used with a learning Training Details rate of 0.05, batch size of 60 and 50 epochs, all of which were heuristically determined. The model was designed to use patient specific EEG signals • Each fNIRS signal generated corresponds to an EEG as input to decode fNIRS signals. For each patient, the data sequence input. An element in the EEG sequence corre- was randomly split into training, validation, and testing sub- sponds to 1 second of recording with 500 time points (sam- sets, with a proportion of 60% training, 20% testing and 20% pling frequency is 500 Hz) for each EEG channel. Data respectively. We experimented with various model depths and batches were generated for sequence processing by using determined deep LSTMs to outperform shallow LSTMs. This the utility class for batch generation in the Keras frame- is likely due to the larger hidden state which occurs because of work. Briefly, this class uses as input a sequence of data increasing layers. Complete training details are given below. points to produce batches for training and validation. Data points outside of the start and end indices of rest- • We initialized the LSTM’s parameters with the uniform ing state periods (as marked in our ground truth) are not distribution between 0 and 1. This was done to counter- used in the output sequences. The final EEG data used as act the exploding gradients problem intrinsic to LSTMs, input is two dimensional, i.e., [data points, channels]. thereby enforcing a hard constraint on the norm of the gradient by scaling it between 0 and 1. Simultaneously, To summarize, the model was trained as follows: (a) we we specified starting node values for the LSTM computa - designed LSTM layers with corresponding LSTM cells (b) tions by preparing a feed dictionary which has input EEG model parameters were uniformly initialized in the range data and a target label. It is important to note that the between [0,1], (c) dropout was applied with value of 0.2, LSTM can learn how to map input sequences as model and average pooling was applied to reduce the probability training is patient specic fi into a x fi ed dimensional vector of model overfitting, (d) we used backpropagation through representation and can learn temporal dependencies. Fig. 2 Multimodal EEG to fNIRS reconstruction using our patient a 4-D tensor with shape: (samples per batch, sequence length, time specific sequence to sequence LSTM autoencoder model. Given points, and channels). The model has encoder and decoder compart- EEG input data into the encoder, the model decodes and reconstructs ments, each with 2 LSTM layers, determined heuristically. Table 1 fNIRS output. After data collection and resting state segment annota- below provides details of the model tion, data processing and model development, the data is finalized as 1 3 Neuroinformatics (2022) 20:537–558 543 time with a learning rate of 0.05, (e) we used a batch size of level of signal fidelity. That is, signals that were ± 2 stand- 60 and 50 training epochs for each patient. ard deviations of the mean and displayed low SNR (i.e., signal amplitude less than 30% of mean signal amplitude) were removed from analysis. We then computed the Pearson Model Validation product-moment correlation coefficients between the experi - mental fNIRS timeseries of the seed channel and the experi- After training and saving our model’s weights, we validated mental fNIRS timeseries of all other channels. Subsequently, the model’s intra-patient predictive capacity by using indi- the Pearson product-moment correlations (and corresponding vidual EEG recordings as input to predict fNIRS signals. Fisher z-scores) were computed between the experimental This was possible since our dataset contains multiple record- seed channel timeseries and our model’s predicted fNIRS ings from each patient. To diagnose performance, we plotted timeseries for all other channels. The two sets of correla- learning curves to ensure we did not overfit during training. tion coefficients were respectively projected to an MRI head As an illustrative example, Fig. 3 shows the learning curves template based on the three dimensional coordinates of the for patients 1, 4, and 23. corresponding channels using Atlasviewer (Aasted et al., 2015). The connectivity value at each voxel of the cortex Model Predictions was obtained from the correlation coefficients of all channels with a weighted-average method using the reciprocal of the The model predicts signals by appending ‘output state’, and cube of the distance from the voxel to each fNIRS channel. In order to quantitively evaluate and compare the results ‘output prediction’ matrices. LSTM cells are connected recur- rently to each other. Decoder inputs are two-dimensional of our functional connectivity studies, we computed the root mean square error (Eq. 2) i.e., the standard deviation of the matrices which are passed into decoder LSTM layers. fNIRS data is shifted one sequence ahead to hold data in LSTM residuals between functional connectivity values in experi- mental fNIRS and reconstructed fNIRS time courses derived memory and finally decoder outputs are returned due to the data passing through the deconvolution layers. from full spectrum EEG and specific EEG frequency band signals for all patients in our cohort. Functional Connectivity Validation � � � �∑ fc − f i=c i c (2) We chose the seed channel from a region of interest, defined RMSE = FC to be a region which had adequate optode coverage con- firmed by our source/detector montage and an acceptable Fig. 3 Learning curves are generated for the training and validation gap between the two final loss values. We note that the validation loss sets. The training and validation loss decrease to a point of stability decreases to a point of stability and has a small gap with the training with a minimal gap between the two final loss values. We note that loss, mean squared error (MSE) the validation loss decreases to a point of stability with a minimal 1 3 544 Neuroinformatics (2022) 20:537–558 where “C” is the number of channels per functional con- across the brain. Channel locations were chosen if they nectivity analysis, fc is the connectivity value of experimen- offered coverage of most of the brain within the constraints tal fNIRS and f is the connectivity value of model fNIRS of the source/detector montage and had an acceptable level reconstructions. of signal fidelity as indicated in “ EEG-fNIRS Data Acquisi- tion and Pre-Processing” section. As an illustrative example, Fig. 8 shows the model’s spatial predictions for patient 10. Results EEG Frequency Decomposition and Resting State Predictions This section describes the reconstruction results obtained using full spectrum EEG and subsequently EEG frequency After model training and validation, we computed EEG fre- ranges as model input. Intra-patient reconstructions are quency bands, namely: delta [0.5–3 Hz], theta [4–7 Hz], alpha also presented; we explore spatial reconstruction, resting [8–13 Hz], beta [14–30 Hz], and gamma [30–100 Hz]. To state predictions, and functional connectivity. ensure the presence of appropriate power in the frequency ranges, the spectral power of EEG signals was obtained using the Welch’s power spectral density function. Welch's method Full Spectrum EEG Performance and Feature was preferred over other methods (i.e., standard periodogram Analysis spectrum estimation and Bartlett's method) as Welch’s method offsets a reduced frequency resolution with a reduc - Resting state full spectrum EEG signals from all channels tion in signal noise in the estimated power spectra in exchange were input in the model. To decode fNIRS channels from for reducing the frequency resolution (Welch, 1967). The encoded EEG channels, the model’s decoder layers used the Welch method partitions the signal into overlapping segments encoder’s latent state as input as data traveled through LSTM thereby mitigating the loss of edge data. The overlapped data units. Figure 4 below quantifies performance on selected segments are then windowed in the time domain. Subsequent individual patients with full spectrum EEG signals as input. computation includes the discrete Fourier transform, followed Figure 5 provides the group estimate of reconstruction error by averaging the periodograms leading to a final nxm array for all patients given scalp full spectrum EEG recordings. representing power measurements by frequency bins. All computations (including Fourier decomposition, Welch’s power spectral density) were performed using the Intra‑patient Reconstructions on Separate Recording MNE software package (Gramfort et al., 2014). Figure 9 Sessions shows the model’s predictions from EEG frequency ranges input using patient 10 (fNIRS channel 10). Here, we report results on intra-patient fNIRS reconstruc- We calculated decoded fNIRS reconstruction error met- tions provided EEG resting state as input. Specifically, we rics, as shown in Fig. 10, for each EEG frequency range hypothesized that our model when trained with a patient’s and calculated patient wise reconstruction error. The gamma single recording was able to reconstruct fNIRS signals from and beta frequency bands demonstrated the lowest error a subsequent recording. To examine our model’s predictive rates and in the lower EEG frequency ranges, we noticed capacity and to cross-validate our model, we first trained increased fNIRS reconstruction error, possibly owing to the our network on a patient’s single recording. Next, we used fact that our model was possibly not able to learn appropriate our trained network and aimed to reconstruct fNIRS signals features to reconstruct fNIRS signals. from a subsequent recording from the same patient. The data To further determine which EEG frequency band can was partitioned into training, testing, and validation subsets reconstruct fNIRS signals with the lowest reconstruction in a 60/20/20 manner. This was done for all data across all error on average, we calculated band wise reconstruction patients and recordings. Figure 6 displays the group results error for all patients, as shown in Fig. 11. Following which, for intra-patient fNIRS signal reconstructions and Fig. 7 dis- we conducted one-tailed paired t-tests to test whether there plays the fNIRS reconstructions for channel 5 from patient is a statistical difference in reconstruction error between 10 across recordings 1, 3, 4. any two of the five bands when compared to gamma in the following combinations: [delta, gamma], [theta, gamma], Spatial Variability of Reconstructions [alpha, gamma], and [beta, gamma]. Bonferroni correction was then applied to control the family-wise error rate to be We then explored the model’s predictions sensitivity to less than 0.05. The gamma frequency band reconstructs channel location on the head. The topographic robustness of fNIRS signals with increased fidelity on average as com - the model suggests the predictions are reasonably invariant pared to other frequency bands. 1 3 Neuroinformatics (2022) 20:537–558 545 Fig. 4 Decoded predictions of hemodynamic signals from cerebral reconstructed with the lowest reconstruction error, RE, while patient electrical activity. Full spectrum EEG signals from all channels were 10 had the highest. The data has been mean centered and baseline is used as input. fNIRS HbO reconstructions are shown from 3 patients near zero, 250 s is shown here to illustrate seizure free, resting state in channel 10 (Channel 10’s SNR was adequate, located on the left periods. Note that the model accounts for the delay between EEG and temporal lobe). Black and red curves correspond to experimental fNIRS and the model fNIRS predictions are indicative of this delay and reconstructed fNIRS signals respectively. Data from patient 22 1 3 546 Neuroinformatics (2022) 20:537–558 for functional connectivity computations (Fig. 13), followed by analysis using the EEG gamma frequency band as input (Fig. 14). Discussion Deep learning models obviate cumbersome and brittle fea- ture engineering processes replacing them with hierarchical feature learning. In this work, we developed a deep learning CNN-LSTM sequence-to-sequence autoencoder to predict fNIRS signals from resting state EEG signals in the epi- Fig. 5 Full spectrum EEG to fNIRS reconstructions. The group esti- leptic brain. Our model was trained using a 60/20/20 split mate of full spectrum EEG signals from all channels were used as for training, testing, and validation, respectively. The results input in our network architecture, to reconstruct full fNIRS signals here demonstrate that in the context of epileptic resting state from all fNIRS channels recordings, fNIRS signals can be predicted using full spec- trum as well as specific frequency range EEG signals to a Functional Connectivity Results certain extent. We further validated our method by recon- structing the functional connectivity in the brain using the We computed functional connectivity mappings for experi- predicted fNIRS and compared it to the functional connec- tivity using experimental fNIRS. mental fNIRS and our model’s fNIRS reconstructions. We compared experimental fNIRS and fNIRS reconstructions From a neurophysiological standpoint, the resting epi- leptic brain is in a dynamic state and cerebral blood flow derived from both full spectrum EEG and the EEG gamma band for all patients. The root mean square error (stand- is in constant flux (Wang et al., 2011). Recent work has shown the presence of abnormal functional networks in ard deviation of the residuals) was used as an estimator of the error in our connectivity studies. On a group level, we the interictal state (Murta et al., 2015; Richardson, 2012). Thus, even with removal of systemic physiological com- noticed a lower error in functional connectivity analyses between experimental fNIRS and fNIRS reconstructions ponents underlying compensation by molecular and cel- lular mechanisms can possibly help predict components derived from full spectrum EEG as compared with experi- mental fNIRS and fNIRS reconstructions derived from the of systemic physiology in addition to hemodynamic brain activity (Pressl et al., 2019). Our experimental findings gamma band and are shown in Fig. 12. Figures 13, 14 show examples of the functional connec- can be related to known physiological phenomena being generated at the frequency of Mayer waves (~0.1 Hz), as tivity mapping results generated using the correlations of the timeseries from patient 22, with the seed channel chosen as these oscillations reflect fluctuation in cerebral arterial blood pressure (Nikulin et al., 2014; Schwab et al., 2009). 20. First, we used full spectrum EEG predictions as input Fig. 6 Group results for intra- patient fNIRS reconstructions. The network was trained on a patient’s single recording. Next, the network reconstructed subsequent recordings from the same patient. The data was partitioned into training, test- ing, and validation subsets in a 60/20/20 manner. This was done for all data across all patients and recordings 1 3 Neuroinformatics (2022) 20:537–558 547 Fig. 7 Intra-patient predictions of hemodynamic signals from cer- shown here from 3 such recordings. Panels “A”, “B”, and “C” show ebral electrical activity. A full spectrum resting state EEG sin- the respective reconstruction of channel 5 from recordings 1, 3, and –2. gle recording (patient 10 all channels) was used to train the model. 4 (patient 10). The reconstruction error is 2.98 × 10 Black and red After training, we saved the model weights and used as input a sub- curves correspond to experimental and reconstructed fNIRS signals sequent recording from the same patient. fNIRS reconstructions are respectively The presence of these oscillations persisting after filter - and its fNIRS correlate via the neurovascular coupling ing can be partly due to the fact that they share a com- phenomenon. mon spectral range with typical hemodynamic responses These nuanced features within the EEG signal are (Yücel et al., 2016). On the other hand, these oscillations encoded and subsequently decoded by the architectural correspond to cerebral vasomotion (i.e., extra neuronal) components of the model, particularly the convolutional and are possibly related to blood vessel tonal oscillation LSTM parameters (Greff et al., 2017; Sutskever et al., (Aalkjær et al., 2011; Julien, 2006; Quaresima & Ferrari, 2014). The model’s encoder and decoder and parameters 2019; Sassaroli et al., 2012). (e.g., the activation function) may have enhanced feature The exact mechanics of physiological signal presence extraction in resting state EEG data and its corresponding within EEG signals has not been established with cer- correlate in fNIRS signals. In addition, the features com- tainty. However, experimental results from this work sug- puted by using the outputs or hidden states of the recur- gest the following: our model can capture subtle hemody- rent units and the model may extract long-term dependen- namic dependencies within the EEG resting state signal cies (electrical and/or physiological) in resting state EEG 1 3 548 Neuroinformatics (2022) 20:537–558 −3 Fig. 8 fNIRS spatial reconstructions, patient 10. To illustrate our net- (channel 11) to 7.83x10 (channel 62), with the mean RE being −2 work’s fNIRS reconstructions spatially, signals from multiple EEG 6.52x10 for all reconstructions. Black and red curves correspond to channels are used as input, for which the locations are shown on experimental and reconstructed fNIRS signals respectively −1 the brain (blue circles). Reconstruction error ranges from 6.41x10 signals from the LSTM modules via the gating mechanism relationship between CBF and neuronal activity in the (Sutskever et al., 2014). Furthermore, when cerebral blood resting epileptic brain is theorized to be captured by the flow (CBF) varies, changes occur in both the metabolic and model used in this work. Exploring the spatial localiza- electrical activity of cortical neurons with corresponding tion of EEG frequency oscillations can help to determine EEG changes (Sassaroli et al., 2012). if the presence of physiological signals is variable across Events responsible for evoking the fNIRS response can patients and electrodes thereby possibly lending credence be divided into subthreshold synaptic and suprathreshold to the hypothesis that these oscillations are unlikely to be spiking activities (Curtin et al., 2019; Sharbrough et al., generated by a single source. 1973). Excitatory and inhibitory neurons which are often We show spatial decoding is possible using our model. located within close proximity in the brain are simulta- Examination of the LSTM memory units and the latent neously active and may contribute to the hemodynamic space architecture in autoencoders can demonstrate cor- response (Franaszczuk et al., 2003). Slower EEG frequency relation between data that were previously unknown. envelopes (i.e., delta and theta) are generated by the thala- Utilizing the architecture developed here to predict brain mus and cortical cells in layers II-VI. Faster frequencies hemodynamics, a next step would be to understand the (i.e., beta and gamma) arise from cells in layers IV and V structure of the latent variable (multidimensional vec- of the cortex (Foreman & Claassen, 2012; Merker, 2016). tor) to unpack the principal components of the fNIRS or Changes in electrical potential seen in EEG recordings EEG signal. are closely tied to cerebral blood flow (CBF) and when A second point for further investigation is to integrate normal CBF declines to approximately 25– 35 ml/100 g/ an attention mechanism in our model. Since LSTM cells min, the EEG signal first loses faster frequencies, then as can lead to ambiguous memory activations, an attention the CBF decreases to approximately 17–18 ml/100 g/min, mechanism allows for encoding input into a sequence of slower frequencies gradually increase. The interdependent vectors and from this, we can choose a subset adaptively 1 3 Neuroinformatics (2022) 20:537–558 549 Fig. 9 Resting state fNIRS predictions given EEG frequency range tal and reconstructed fNIRS signals respectively. We used a constant input, patient 10, channel 10. We obtained predicted fNIRS recon- experimental fNIRS signal for comparison. The gamma range, which structions given filtered EEG input for the following frequency bands: contains the greatest number of EEG frequencies reconstructs with Delta: 0–3 Hz; Theta: 4–7 Hz; Alpha: 8–13 Hz; Beta: 14–30 Hz; more fidelity compared to ranges with less frequency components Gamma: 30–100 Hz. Black and red curves correspond to experimen- during decoding. In this condition, the model no longer Attention implemented in our model would enable us to needs to utilize fixed length vectors thereby increasing inspect the relationship between encoded and decoded performance metrics at the cost of computational time. sequences by model weight visualization. 1 3 550 Neuroinformatics (2022) 20:537–558 Fig. 10 fNIRS reconstruc- tion error given specific EEG frequency ranges for all patients, all channels. We obtained predicted reconstruc- tions given filtered EEG input for the following frequency ranges: Delta: 0–3 Hz; Theta: 4–7 Hz; Alpha: 8–13 Hz; Beta: 14–30 Hz; Gamma: 30–100 Hz. The gamma range, which contains the greatest number of EEG frequencies, reconstructs with more fidelity and lowest reconstruction error metrics compared to ranges with less frequency components In comparison with lower frequency range EEG signals, selectivity to that of nearby neuronal activity (Jia & Kohn, results here suggest that higher frequency EEG envelopes 2011; Whittingstall & Logothetis, 2009). GABA-ergic reconstruct fNIRS signals with less error. Our results inhibitory interneuron activity is considered to be crucial corroborate that EEG gamma band based fNIRS recon- to generate EEG gamma frequency activity and this may structions show a closer fit between the observed and pre - be increased via interactions with excitatory neurons (Jia dicted hemodynamic responses as opposed to other EEG & Kohn, 2011; Park et al., 2011; Ray & Maunsell, 2010). frequency ranges (Ebisch et al., 2005; Murta et al., 2015; However, to fully interpret the impact of this activity war- Niessing et al., 2005). This is possibly because higher fre- rants an investigation into the cellular mechanisms respon- quencies engage an increased number of neurons, but it is sible for their generation. less apparent if this is attributed to baseline network activ- In the second part of our work, we explored functional ity or part of a pivotal functional role. Gamma rhythms connectivity in the resting state of the epileptic brain. in the brain provide an indication of engaged networks We hypothesized that our network’s predictions can help and have been observed in several cortical and subcorti- reveal functional connections and on a group level, pre- cal structures. These rhythms are typically stronger for dicted fNIRS from full spectrum EEG have higher con- some stimuli as compared to others, thereby displaying nectivity as compared to predictions derived from the Fig. 11 Mean fNIRS recon- struction error given specific EEG frequency ranges for all patients. The gamma range, which contains the greatest number of EEG frequencies, reconstructs with more fidelity and lowest reconstruction error metrics compared to other ranges with less frequency components 1 3 Neuroinformatics (2022) 20:537–558 551 Fig. 12 Estimator error (RMSE) for functional connectivity results signal input. The connectivity derived from the full spectrum EEG for all patients. Error for connectivity analyses between experimen- time series consistently has lower error compared to the connectivity tal fNIRS and predictions using full spectrum and gamma band EEG derived from the gamma band EEG gamma band. Experimental resting state fNIRS data of the hemodynamic response to neuron firing and the fact and predicted fNIRS data was correlated to reveal simi- that some patients are not able to undergo an fMRI scan lar connections near the set seed but metrics decreased easily (i.e., claustrophobia, paroxysmal seizure occurrence generally as distance increased from the seed. This can during scanning) (Pressl et al., 2019; Richardson, 2012). By be due to numerous factors: 1. noise causing a decrease showing the possibility of obtaining brain hemodynamic in reconstruction quality, 2. a decrease in gamma activa- data from neural signals, the results here add an additional tions at the region of interest, and 3. model parameters dimension for understanding the epileptic human brain, aid unable to completely learn the nuances present within in clinical decision making, and provide a complementary the signal. Furthermore, systemic artifacts from the scalp measure to fMRI, particularly in locations where access to and skull behave as dominant noise sources in resting fMRI technology is scarce or not possible. state fNIRS signals, leading to inaccurate reconstruction. Scalp EEG technology remains the clinical gold standard Utilizing an EEG-fNIRS experimental setup with short for the noninvasive assessment of electrical brain activity separation channels, measuring approximately 1–2 cm (Dash et al., 2017). Using EEG signals in conjunction with in spatial separation between source and detector could predicted brain hemodynamics can possibly improve clini- lead to sufficient noise reduction and improved signal cal management and ultimately patient outcomes (Connolly sensitivity (Gagnon et al., 2012; Kohno et al., 2007). We et al., 2015; Helbok & Claassen, 2013). Multimodal EEG- hypothesize that reconstruction metrics and correspond- fNIRS analysis using deep learning frameworks, as the one ing functional connectivity network measures stabilize presented in this work, can improve our understanding of with increased signal quality and resting state duration, cerebral neurovascular coupling and pathophysiology. The thereby decreasing the disparities present between experi- results from this work can be abstracted for applications mental and predicted time series. to other neurological and neuropsychiatric pathologies, The resting state epileptic brain and connectivity between such as stroke, spinal cord injuries, traumatic brain inju- brain networks is dynamic (Deco et al., 2011; McKenna ries, Alzheimer’s disease, attention-deficit hyperactivity et al., 1994). Typically, fMRI has been used for computing disorder, post-traumatic stress disorder, and dementia to functional connectivity but there are inherent limitations name a few (Fair et al., 2013; Phillips et al., 2018; Siegel of fMRI, particularly, slow dynamics, regional variability et al., 2016). Furthermore, hemodynamic predictions from 1 3 552 Neuroinformatics (2022) 20:537–558 Fig. 13 Functional connectivity results between experimental fNIRS (A, C) and predicted resting state fNIRS (B, D) are shown. A and and predicted resting state fNIRS using full frequency spectrum EEG B display the right side of the brain, C and D display the left side as input for patient 22. We employed seed based functional con- of the brain. The connectivity profiles are seen to be similar between nectivity analysis to obtain a surface brain map that describes brain the maps generated using the experimental fNIRS results and the pre- functional connectivity correlation patterns. The seed region of inter- dictions of the model. A RMSE value of 0.07 corresponds to fNIRS est (dark circle) is shown and full spectrum EEG was used as input signal reconstruction from experimental fNIRS and predicted fNIRS into the model. Bilateral brain correlations using experimental fNIRS from full frequency spectrum EEG as model input electrical brain signals can be useful in treatment strategies different treatments (Citerio et al., 2015; Le Roux, 2013). utilizing neurofeedback (i.e., neuroprosthetics, transcranial Currently, therapeutic strategies follow a ‘reactive’ model: direct current stimulation) as well as towards developing corrective actions are triggered by abnormal values in sin- precision medicine strategies (DeBettencourt et al., 2015; gle parameters (i.e., EEG signals) and a stepwise approach Dutta et al., 2015; Kotliar et al., 2017; Nicholson et al., is used with increasing therapeutic intensity. Comprehen- 2016; Ros et al., 2014; Sitaram et al., 2017; Thair et al., sive signals (i.e., EEG and predicted hemodynamics) can 2017). Predicting hemodynamics from EEG increases clin- shift this paradigm towards a ‘goal-directed’ management ical diagnostic specificity, allowing differentiation between strategy (Le Roux, 2013; Maas et al., 2012; Schmidt & De pathological conditions that may appear similar but require Georgia, 2014). 1 3 Neuroinformatics (2022) 20:537–558 553 Fig. 14 Functional connectivity results between experimental fNIRS (A, C) and predicted fNIRS (B, D) are shown. A RMSE value of and predicted fNIRS resting state using EEG gamma band as input 0.15 corresponds to fNIRS signal reconstruction from experimen- for patient 22. Correlations from experimental fNIRS and predicted tal fNIRS and predicted fNIRS from signals derived from the EEG resting state fNIRS using EEG gamma band as input into our model gamma band as model input are displayed. Bilateral brain correlations using experimental fNIRS Conclusion Information Sharing Statement We designed and implemented a deep learning model to Sharing the data used in this study is bound by the ethics predict resting state hemodynamics given specific resting of the institutional review boards of Sainte-Justine Hospital state scalp EEG frequencies from a cohort of epileptic and Centre Hospitalier de l’Université de Montréal which patients. The robust multidimensional dataset used here approved the study. The custom code used in this study is allowed us to investigate the relationship between brain available upon reasonable request. hemodynamics and neural signals via neurovascular cou- Acknowledgements This project was generously supported by The pling. Using a deep learning architecture, we performed Natural Sciences and Engineering Research Council of Canada grant a thorough analysis of each EEG frequency range and its NSERC: NSERC, 239876-and Canadian Institutes of Health Research complementary fNIRS prediction; further we analyzed grant 87183. functional connectivity between brain regions using fre- quency range predictions. We noted that higher EEG fre- Declarations quency bands provided hemodynamic predictions with the highest metrics. Conflict of Interest The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose. Open Access This article is licensed under a Creative Commons Attri- bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long 1 3 554 Neuroinformatics (2022) 20:537–558 as you give appropriate credit to the original author(s) and the source, 113–119). Lippincott Williams and Wilkins. https:// doi. org/ 10. provide a link to the Creative Commons licence, and indicate if changes 1097/ MCC. 00000 00000 000179 were made. The images or other third party material in this article are Connolly, M., Vespa, P., Pouratian, N., Gonzalez, N. R., & Hu, X. included in the article's Creative Commons licence, unless indicated (2015). Characterization of the relationship between intracra- otherwise in a credit line to the material. If material is not included in nial pressure and electroencephalographic monitoring in burst- the article's Creative Commons licence and your intended use is not suppressed patients. Neurocritical Care, 22(2), 212–220. https:// permitted by statutory regulation or exceeds the permitted use, you will doi. org/ 10. 1007/ s12028- 014- 0059-8 need to obtain permission directly from the copyright holder. To view a Curtin, A., Tong, S., Sun, J., Wang, J., Onaral, B., & Ayaz, H. (2019). A copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . systematic review of integrated functional near-infrared spectros- copy (fNIRS) and transcranial magnetic stimulation (TMS) studies. Frontiers in Neuroscience. https://doi. or g/10. 3389/ fnins. 2019. 00084 Czisch, M., Wehrle, R., Kaufmann, C., Wetter, T. C., Holsboer, F., Pollmächer, T., & Auer, D. P. (2004). Functional MRI during References sleep: BOLD signal decreases and their electrophysiological cor- relates. European Journal of Neuroscience. https:// doi. org/ 10. Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what 1111/j. 1460- 9568. 2004. 03518.x is currently thought? In Acta Physiologica. https:// doi. org/ 10. Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien- 1111/j. 1748- 1716. 2011. 02320.x Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Aasted, C. M., Yücel, M. A., Cooper, R. J., Dubb, J., Tsuzuki, D., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Becerra, L., Petkov, M. P., Borsook, D., Dan, I., & Boas, D. A. Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., (2015). Anatomical guidance for functional near-infrared spec- Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on mini- troscopy: AtlasViewer tutorial. Neurophotonics. https:// doi. org/ mal standards for electroencephalography in Canada: A review by 10. 1117/1. nph.2. 2. 020801 the Canadian Society of Clinical Neurophysiologists. In Canadian Amzica, F., & Steriade, M. (1998). Electrophysiological correlates of Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). sleep delta waves. Electroencephalography and Clinical Neuro- Cambridge University Press. https://doi. or g/10. 1017/ cjn. 2017. 217 physiology, 107(2), 69–83. De Luca, M., Beckmann, C. F., De Stefano, N., Matthews, P. M., & Barlow, J. S., Kamp, A., Morton, H. B., Ripoche, A., Shipton, H., Smith, S. M. (2006). fMRI resting state networks define distinct & Tchavdarov, D. B. (1974). EEG instrumentation standards: modes of long-distance interactions in the human brain. Neuro- Report of the Committee on EEG Instrumentation Standards of Image, 29(4), 1359–1367. https://d oi.o rg/1 0.1 016/j.n euroi mage. the International Federation of Societies for Electroencephalog- 2005. 08. 035 raphy and Clinical Neurophysiology. Electroencephalography DeBettencourt, M. T., Cohen, J. D., Lee, R. F., Norman, K. A., & and Clinical Neurophysiology, 37(5), 549–553. https:// doi. org/ Turk-Browne, N. B. (2015). Closed-loop training of attention 10. 1016/ 0013- 4694(74) 90100-X with real-time brain imaging. Nature Neuroscience. https:// doi. Baytas, I. M., Xiao, C., Zhang, X., Wang, F., Jain, A. K., & Zhou, J. (2017). org/ 10. 1038/ nn. 3940 Patient subtyping via time-aware LSTM networks. Proceedings of Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the ACM SIGKDD International Conference on Knowledge Dis- the dynamical organization of resting-state activity in the brain. In covery and Data Mining. https://doi. or g/10. 1145/ 30979 83. 30979 97 Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Bettus, G., Guedj, E., Joyeux, F., Confort-Gouny, S., Soulier, E., Publishing Group. https:// doi. org/ 10. 1038/ nrn29 61 Laguitton, V., Cozzone, P. J., Chauvel, P., Ranjeva, J. P., Dhamija, R., Moseley, B. D., Cascino, G. D., & Wirrell, E. C. (2011). Bartolomei, F., & Guye, M. (2009). Decreased basal fMRI A population-based study of long-term outcome of epilepsy in functional connectivity in epileptogenic networks and con- childhood with a focal or hemispheric lesion on neuroimaging. tralateral compensatory mechanisms. Human Brain Mapping, Epilepsia, 52(8), 1522–1526. https:// doi. org/ 10. 1111/j. 1528- 30(5), 1580–1591. https:// doi. org/ 10. 1002/ hbm. 20625 1167. 2011. 03192.x Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Dutta, A., Jacob, A., Chowdhury, S. R., Das, A., & Nitsche, M. A. Kirsch, H. E., Berger, M. S., Barbare, N. M., & Knight, R. T. (2006). (2015). EEG-NIRS based assessment of neurovascular coupling High gamma power is phase-locked to theta oscillations in human during anodal transcranial direct current stimulation - a stroke neocortex. Science. https:// doi. org/ 10. 1126/ scien ce. 11281 15 case series. Journal of Medical Systems, 39(4), 1–9. https:// doi. Chiarelli, A. M., Zappasodi, F., Di Pompeo, F., & Merla, A. (2017). Simul- org/ 10. 1007/ s10916- 015- 0205-7 taneous functional near-infrared spectroscopy and electroencepha- Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., Galuske, R. lography for monitoring of human brain activity and oxygenation: A. W., & Niessing, J. (2005). Hemodynamic signals correlate A review. Neurophotonics. https://doi. or g/10. 1117/1. n ph.4.4. 041411 tightly with synchronized gamma oscillations. Science. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, Fair, D. A., Nigg, J. T., Iyer, S., Bathula, D., Mills, K. L., Dosenbach, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase represen- N. U. F., Schlaggar, B. L., Mennes, M., Gutman, D., Bangaru, tations using RNN encoder-decoder for statistical machine trans- S., Buitelaar, J. K., Dickstein, D. P., Martino, A. D., Kennedy, D. lation. EMNLP 2014 - 2014 Conference on Empirical Methods N., Kelly, C., Luna, B., Schweitzer, J. B., Velanova, K., Wang, Y. in Natural Language Processing, Proceedings of the Conference. F., & Milham, M. P. (2013). Distinct neural signatures detected https:// doi. org/ 10. 3115/ v1/ d14- 1179 for ADHD subtypes after controlling for micro-movements in Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). resting state functional connectivity MRI data. Frontiers in Sys- Audio Word2Vec: Unsupervised learning of audio segment tems Neuroscience. https:// doi. org/ 10. 3389/ fnsys. 2012. 00080 representations using sequence-to-sequence autoencoder. Pro- Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K. ceedings of the Annual Conference of the International Speech R., & Blankertz, B. (2012). Enhanced performance by a hybrid Communication Association, INTERSPEECH. https://doi. or g/10. NIRS-EEG brain computer interface. NeuroImage, 59(1), 519– 21437/ Inter speech. 2016- 82 529. https:// doi. org/ 10. 1016/j. neuro image. 2011. 07. 084 Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations Fishburn, F. A., Norr, M. E., Medvedev, A. V., & Vaidya, C. J. (2014). for the use of multimodal monitoring in the neurointensive care Sensitivity of fNIRS to cognitive state and load. Frontiers in unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. Human Neuroscience. https:// doi. or g/ 10. 3389/ fnhum. 2014. 00076 1 3 Neuroinformatics (2022) 20:537–558 555 Foreman, B., & Claassen, J. (2012). Quantitative EEG for the detection Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. https:// of brain ischemia. Critical Care. https://doi. or g/10. 1186/ cc112 30 doi. org/ 10. 1016/j. yebeh. 2008. 01. 011 Franaszczuk, P. J., Kudela, P., & Bergey, G. K. (2003). External excita- Huppert, T. J., Diamond, S. G., Franceschini, M. A., & Boas, D. A. tory stimuli can terminate bursting in neural network models. (2009). HomER: A review of time-series analysis methods for Epilepsy Research, 53(1–2), 65–80. https:// doi. or g/ 10. 1016/ near-infrared spectroscopy of the brain. Applied Optics, 48(10), S0920- 1211(02) 00248-6 D280–D298. Freeman, W. J., Burke, B. C., & Holmes, M. D. (2003). Aperiodic Jia, X., & Kohn, A. (2011). Gamma rhythms in the brain. PLoS Biol- phase re-setting in scalp EEG of beta-gamma oscillations by state ogy, 9(4), 2–5. https:// doi. org/ 10. 1371/ journ al. pbio. 10010 45 transitions at alpha-theta rates. Human Brain Mapping. https:// Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and doi. org/ 10. 1002/ hbm. 10120 myocardial oxygen sufficiency and circulatory parameters. Sci- Gagnon, L., Cooper, R. J., Yücel, M. A., Perdue, K. L., Greve, D. N., ence, 198(4323), 1264–1267. & Boas, D. A. (2012). Short separation channel location impacts Julien, C. (2006). The enigma of Mayer waves: Facts and models. Cardio- the performance of short channel regression in NIRS. NeuroIm- vascular Research. https://d oi.o rg/1 0.1 016/j.c ardio res.2 005.1 1.0 08 age. https:// doi. org/ 10. 1016/j. neuro image. 2011. 08. 095 Khan, M. J., & Hong, K. -S. (2015). Passive BCI based on drowsiness Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting- detection: An fNIRS study. Biomedical Optics Express, 6(10), state fNIRS scanning duration on functional brain connectivity 4063. https:// doi. org/ 10. 1364/ boe.6. 004063 and graph theory metrics of brain network. Frontiers in Neurosci- Khan, M. J., & Hong, K. S. (2017). Hybrid EEG-FNIRS-based eight- ence, 11(JUL). https:// doi. org/ 10. 3389/ fnins. 2017. 00392 command decoding for BCI: Application to quadcopter control. Girouard, H., & Iadecola, C. (2006). Neurovascular coupling in the Frontiers in Neurorobotics, 11, 6. https://doi. or g/10. 3389/ fnbo t. normal brain and in hypertension, stroke, and Alzheimer disease. 2017. 00006 Journal of Applied Physiology, 100(1), 328–335. Knyazev, G. G. (2012). EEG delta oscillations as a correlate of basic Goldman, R. I., Stern, J. M., Engel, J., Jr., & Cohen, M. S. (2002). homeostatic and motivational processes. Neuroscience & Biobe- Simultaneous EEG and fMRI of the alpha rhythm. NeuroReport, havioral Reviews, 36(1), 677–695. 13(18), 2487. Kobayashi, E., Bagshaw, A. P., Grova, C., Dubeau, F., & Gotman, J. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT (2006). Negative BOLD responses to epileptic spikes. Human Brain press. Mapping, 27(6), 488–497. https:// doi. org/ 10. 1002/ hbm. 20193 Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, Koch, S. P., Koendgen, S., Bourayou, R., Steinbrink, J., & Obrig, H. D., Brodbeck, C., Parkkonen, L., & Hämäläinen, M. S. (2014). (2008). Individual alpha-frequency correlates with amplitude of MNE software for processing MEG and EEG data. NeuroImage. visual evoked potential and hemodynamic response. NeuroIm- https:// doi. org/ 10. 1016/j. neuro image. 2013. 10. 027 age, 41(2), 233–242. Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE law revisited. Physics in Medicine and Biology, 51(5). https:// Transactions on Neural Networks and Learning Systems. https:// doi. org/ 10. 1088/ 0031- 9155/ 51/5/ N02 doi. org/ 10. 1109/ TNNLS. 2016. 25829 24 Kohno, S., Miyai, I., Seiyama, A., Oda, I., Ishikawa, A., Tsuneishi, S., Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. Amita, T., & Shimizu, K. (2007). Removal of the skin blood flow (2015). DRAW: A recurrent neural network for image generation. artifact in functional near-infrared spectroscopic imaging data 32nd International Conference on Machine Learning, ICML 2015. through independent component analysis. Journal of Biomedical Gross, D. W., & Gotman, J. (1999). Correlation of high-frequency oscil- Optics. https:// doi. org/ 10. 1117/ 12814 249 lations with the sleep-wake cycle and cognitive activity in humans. Kotliar, K., Hauser, C., Ortner, M., Muggenthaler, C., Diehl-Schmid, Neuroscience. https:// doi. org/ 10. 1016/ S0306- 4522(99) 00343-7 J., Angermann, S., Hapfelmeier, A., Schmaderer, C., & Grimmer, He, B., & Liu, Z. (2008). Multimodal functional neuroimaging: inte- T. (2017). Altered neurovascular coupling as measured by optical grating functional MRI and EEG/MEG. IEEE Reviews in Bio- imaging: A biomarker for Alzheimer’s disease. Scientific Reports, medical Engineering, 1(2008), 23–40. https:// doi. org/ 10. 1109/ 7(1), 1–11. https:// doi. org/ 10. 1038/ s41598- 017- 13349-5 RBME. 2008. 20082 33 Laufs, H., Krakow, K., Sterzer, P., Eger, E., Beyerle, A., Salek-Haddadi, Helbok, R., & Claassen, J. (2013). Multimodal invasive monitor- A., & Kleinschmidt, A. (2003). Electroencephalographic signa- ing in status epilepticus: What is the evidence it has a place? tures of attentional and cognitive default modes in spontaneous Epilepsia, 54(SUPPL. 6), 57–60. https:// doi. org/ 10. 1111/ epi. brain activity fluctuations at rest. Proceedings of the National 12279 Academy of Sciences of the United States of America. https://doi. Hofle, N., Paus, T., Reutens, D., Fiset, P., Gotman, J., Evans, A. C., & org/ 10. 1073/ pnas. 18316 38100 Jones, B. E. (1997). Regional cerebral blood flow changes as a Le Roux, P. (2013). Physiological monitoring of the severe traumatic function of delta and spindle activity during slow wave sleep in brain injury patient in the intensive care unit. Current Neurology humans. Journal of Neuroscience. https:// doi. or g/ 10. 1523/ and Neuroscience Reports, 13(3), 331. jneur osci. 17- 12- 04800. 1997 Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature. Honda, Y., Nakamura, S., Ogawa, K., Yoshino, R., Tobler, P. N., https:// doi. org/ 10. 1038/ natur e14539 Nishimura, Y., & Tsutsui, K. I. (2021). Changes in beta and Lemieux, L., Krakow, K., & Fish, D. R. (2001). Comparison of spike- high-gamma power in resting-state electrocorticogram induced triggered functional MRI BOLD activation and EEG dipole model by repetitive transcranial magnetic stimulation of primary motor localization. NeuroImage. https://doi. or g/10. 1006/ nimg. 2001. 0896 cortex in unanesthetized macaque monkeys. Neuroscience Li, Z., Liu, H., Liao, X., Xu, J., Liu, W., Tian, F., He, Y., & Niu, H. Research. https:// doi. org/ 10. 1016/j. neures. 2021. 02. 002 (2015). Dynamic functional connectivity revealed by resting- Hong, K. S., Naseer, N., & Kim, Y. H. (2015). Classification of pre - state functional near-infrared spectroscopy. Biomedical Optics frontal and motor cortex signals for three-class fNIRS-BCI. Neu- Express. https:// doi. org/ 10. 1364/ boe.6. 002337 roscience Letters, 587, 87–92. https:// doi. org/ 10. 1016/j. neulet. Lindauer, U., Dirnagl, U., Füchtemeier, M., Böttiger, C., Offenhauser, 2014. 12. 029 N., Leithner, C., & Royl, G. (2010). Pathophysiological inter- Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: ference with neurovascular coupling–when imaging based on Their relationships with epilepsy and behavior. In Epilepsy and hemoglobin might go blind. Frontiers in Neuroenergetics, 2, 25. 1 3 556 Neuroinformatics (2022) 20:537–558 Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning Murta, T., Leite, M., Carmichael, D. W., Figueiredo, P., & Lemieux, L. to diagnose with LSTM recurrent neural networks. 4th Inter- (2015). Electrophysiological correlates of the BOLD signal for national Conference on Learning Representations, ICLR 2016 EEG-informed fMRI. Human Brain Mapping, 36(1), 391–414. - Conference Track Proceedings.https:// doi. org/ 10. 1002/ hbm. 22623 Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., (2001). Neurophysiological investigation of the basis of the fMRI Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscil- signal. Nature. https:// doi. org/ 10. 1038/ 35084 005 lation neurofeedback modulates amygdala complex connectivity Lövblad, K. O., Thomas, R., Jakob, P. M., Scammell, T., Bassetti, C., and arousal in posttraumatic stress disorder. NeuroImage: Clini- Griswold, M., Ives, J., Matheson, J., Edelman, R. R., & Warach, cal. https:// doi. org/ 10. 1016/j. nicl. 2016. 07. 006 S. (1999). Silent functional magnetic resonance imaging demon- Niessing, J., Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., & strates focal activation in rapid eye movement sleep. Neurology. Galuske, R. A. W. (2005). Neuroscience: Hemodynamic signals https:// doi. org/ 10. 1212/ wnl. 53.9. 2193 correlate tightly with synchronized gamma oscillations. Science, Lu, C. M., Zhang, Y. J., Biswal, B. B., Zang, Y. F., Peng, D. L., & Zhu, 309(5736), 948–951. https:// doi. org/ 10. 1126/ scien ce. 11109 48 C. Z. (2010). Use of fNIRS to assess resting state functional con- Nikulin, V. V., Fedele, T., Mehnert, J., Lipp, A., Noack, C., Steinbrink, nectivity. Journal of Neuroscience Methods, 186(2), 242–249. J., & Curio, G. (2014). Monochromatic Ultra-Slow (~ 0.1 Hz) https:// doi. org/ 10. 1016/j. jneum eth. 2009. 11. 010 Oscillations in the human electroencephalogram and their rela- Luo, C., Li, Q., Lai, Y., Xia, Y., Qin, Y., Liao, W., Li, S., Zhou, D., tion to hemodynamics. NeuroImage, 97, 71–80. https:// doi. org/ Yao, D., & Gong, Q. (2011). Altered functional connectivity in 10. 1016/J. NEURO IMAGE. 2014. 04. 008 default mode network in absence epilepsy: A resting-state fMRI Niu, H., & He, Y. (2014). Resting-state functional brain connectivity: study. Human Brain Mapping, 32(3), 438–449. https:// doi. org/ Lessons from functional near-infrared spectroscopy. The Neuro- 10. 1002/ hbm. 21034 scientist. https:// doi. org/ 10. 1177/ 10738 58413 502707 Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Multi-task Sequence to Sequence Learning. c, 1–10. http://ar xiv. Merkle, H., & Ugurbil, K. (1992). Intrinsic signal changes org/ abs/ 1511. 06114 accompanying sensory stimulation: Functional brain mapping Maas, A. I. R., Menon, D. K., Lingsma, H. F., Pineda, J. A., Sandel, M. with magnetic resonance imaging. Proceedings of the National E., & Manley, G. T. (2012). Re-orientation of clinical research Academy of Sciences of the United States of America. https://doi. in traumatic brain injury: Report of an international workshop org/ 10. 1073/ pnas. 89. 13. 5951 on comparative effectiveness research. Journal of Neurotrauma, Palva, J. M., Monto, S., Kulashekhar, S., & Palva, S. (2010). Neuronal 29(1), 32–46. https:// doi. org/ 10. 1089/ neu. 2010. 1599 synchrony reveals working memory networks and predicts indi- Mantini, D., Perrucci, M. G., Del Gratta, C., Romani, G. L., & Corbetta, vidual memory capacity. Proceedings of the National Academy M. (2007). Electrophysiological signatures of resting state net- of Sciences of the United States of America, 107(16), 7580–7585. works in the human brain. Proceedings of the National Academy https:// doi. org/ 10. 1073/ pnas. 09131 13107 of Sciences of the United States of America. https:// doi. org/ 10. Park, Y., Luo, L., Parhi, K. K., & Netoff, T. (2011). Seizure prediction 1073/ pnas. 07006 68104 with spectral power of EEG using cost-sensitive support vec- Martinez-Montes, E., Valdes-Sosa, P. A., Miwakeichi, F., Goldman, R. tor machines. Epilepsia, 52(10), 1761–1770. https:// doi. org/ 10. I., & Cohen, M. S. (2004). Concurrent EEG/fMRI analysis by 1111/j. 1528- 1167. 2011. 03138.x multiway partial least squares. NeuroImage, 22(3), 1023–1034. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., https:// doi. org/ 10. 1016/j. neuro image. 2004. 03. 038 Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., McKenna, T. M., McMullen, T. A., & Shlesinger, M. F. (1994). The Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, brain as a dynamic physical system. Neuroscience, 60(3), 587– M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in 605. https:// doi. org/ 10. 1016/ 0306- 4522(94) 90489-8 Python. Journal of Machine Learning Research. Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and opti- Peng, K., Nguyen, D. K., Tayah, T., Vannasing, P., Tremblay, J., Sawan, mizing LSTM language models. 6th International Conference M., Lassonde, M., Lesage, F., & Pouliot, P. (2014). FNIRS- on Learning Representations, ICLR 2018 - Conference Track EEG study of focal interictal epileptiform discharges. Epilepsy Proceedings. Research. https:// doi. org/ 10. 1016/j. eplep syres. 2013. 12. 011 Merker, B. H. (2016). Cortical gamma oscillations: Details of their Phillips, A. A., Squair, J. W., Sayenko, D. G., Edgerton, V. R., Gerasimenko, genesis preclude a role in cognition. Frontiers in Computational Y., & Krassioukov, A. V. (2018). An autonomic neuroprosthesis: Neuroscience. https:// doi. org/ 10. 3389/ fncom. 2016. 00078 Noninvasive electrical spinal cord stimulation restores autonomic Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). cardiovascular function in individuals with spinal cord injury. Jour- Recurrent neural network based language model. Proceedings of nal of Neurotrauma, 35(3), 446–451. https:// doi. org/ 10. 1089/ neu. the 11th Annual Conference of the International Speech Com-2017. 5082 munication Association, INTERSPEECH 2010. Pizzo, F., Frauscher, B., Ferrari-Marinho, T., Amiri, M., Dubeau, F., Miller, C. M. (2012). Update on multimodality monitoring. Current & Gotman, J. (2016). Detectability of fast ripples (>250 Hz) Neurology and Neuroscience Reports. https:// doi. org/ 10. 1007/ on the scalp EEG: A proof-of-principle study with subder- s11910- 012- 0274-7 mal electrodes. Brain Topography. https:// doi. or g/ 10. 1007/ Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: s10548- 016- 0481-7 New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884– Pressl, C., Brandner, P., Schaffelhofer, S., Blackmon, K., Dugan, 898). Lancet Publishing Group. https:// doi. org/ 10. 1016/ S0140- P., Holmes, M., Thesen, T., Kuzniecky, R., Devinsky, O., & 6736(14) 60456-6 Freiwald, W. A. (2019). Resting state functional connectivity Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., & Malach, patterns associated with pharmacological treatment resistance R. (2005). Neuroscience: Coupling between neuronal firing, field in temporal lobe epilepsy. Epilepsy Research. https:// doi. org/ potentials, and fMRI in human auditory cortex. Science. https:// 10. 1016/j. eplep syres. 2018. 11. 002 doi. org/ 10. 1126/ scien ce. 11109 13 Quaresima, V., & Ferrari, M. (2019). A mini-review on functional near- Müller-Putz, G. R. (2020). Electroencephalography. Handbook of Clinical infrared spectroscopy (fNIRS): Where do we stand, and where Neurology. https:// doi. org/ 10. 1016/ B978-0- 444- 63934-9. 00018-4 should we go? Photonics, 6(3), 87. h t t p s :/ / d o i . o r g / 1 0 . 3 3 9 0 / photo nics6 030087 1 3 Neuroinformatics (2022) 20:537–558 557 Ray, S., & Maunsell, J. H. R. (2010). Differences in gamma frequencies Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis- across visual cortex restrict their possible use in computation. Peacock, J., Weiskopf, N., Blefari, M. L., Rana, M., Oblak, E., Neuron, 67(5), 885–896. https:// doi. org/ 10. 1016/j. neuron. 2010. Birbaumer, N., & Sulzer, J. (2017). Closed-loop brain training: 08. 004 The science of neurofeedback. Nature Reviews Neuroscience. Richardson, M. P. (2012). Large scale brain models of epilepsy: https:// doi. org/ 10. 1038/ nrn. 2016. 164 Dynamics meets connectomics. Journal of Neurology, Neuro- Sitnikova, E., Hramov, A. E., Grubov, V., & Koronovsky, A. A. (2016). surgery and Psychiatry, 83(12), 1238–1248. https:// doi. org/ 10. Rhythmic activity in EEG and sleep in rats with absence epilepsy. 1136/ jnnp- 2011- 301944 Brain Research Bulletin. https:// doi. org/ 10. 1016/j. brain resbu ll. Rojas, G. M., Alvarez, C., Montoya, C. E., de la Iglesia-Vayá, M., 2015. 11. 012 Cisternas, J. E., & Gálvez, M. (2018). Study of resting-state func- Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. tional connectivity networks using EEG electrodes position as (2011). Semi-supervised recursive autoencoders for predicting seed. Frontiers in Neuroscience. https:// doi. org/ 10. 3389/ fnins. sentiment distributions. EMNLP 2011 - Conference on Empiri- 2018. 00235 cal Methods in Natural Language Processing, Proceedings of Ros, T., Baars, B. J., Lanius, R. A., & Vuilleumier, P. (2014). Tuning the Conference. pathological brain oscillations with neurofeedback: A systems Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsuper- neuroscience framework. Frontiers in Human Neuroscience, 8, vised learning of video representations using LSTMs. 32nd Inter- 1008. https:// doi. org/ 10. 3389/ fnhum. 2014. 01008 national Conference on Machine Learning, ICML 2015. Salek-Haddadi, A., Friston, K. J., Lemieux, L., & Fish, D. R. (2003). Strangman, G., Boas, D. A., & Sutton, J. P. (2002). Non-invasive neuro- Studying spontaneous EEG activity with fMRI. Brain Research imaging using near-infrared light. Biological Psychiatry. https:// Reviews. https:// doi. org/ 10. 1016/ S0165- 0173(03) 00193-0doi. org/ 10. 1016/ S0006- 3223(02) 01550-0 Sassaroli, A., Pierro, M., Bergethon, P. R., & Fantini, S. (2012). Low- Sumiyoshi, A., Suzuki, H., Ogawa, T., Riera, J. J., Shimokawa, H., & frequency spontaneous oscillations of cerebral hemodynamics Kawashima, R. (2012). Coupling between gamma oscillation and investigated with near-infrared spectroscopy: A review. IEEE fMRI signal in the rat somatosensory cortex: Its dependence on Journal on Selected Topics in Quantum Electronics. https:// doi. systemic physiological parameters. NeuroImage. https://doi. or g/ org/ 10. 1109/ JSTQE. 2012. 21835 8110. 1016/j. neuro image. 2011. 12. 082 Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence Informatics, Integration Data Display and Analysis The Partici- learning with neural networks. Advances in Neural Information pants in the International Multidisciplinary Consensus Confer- Processing Systems, 4(January), 3104–3112. ence on Multimodality Monitoring. Neurocritical Care. https:// Thair, H., Holloway, A. L., Newport, R., & Smith, A. D. (2017). Tran- doi. org/ 10. 1007/ s12028- 014- 0037-1 scranial direct current stimulation (tDCS): A Beginner’s guide Scholkmann, F., Kleiser, S., Metz, A. J., Zimmermann, R., Pavia, J. for design and implementation. Frontiers in Neuroscience, 11, M., Wolf, U., & Wolf, M. (2014). A review on continuous wave 641. https:// doi. org/ 10. 3389/ fnins. 2017. 00641 functional near-infrared spectroscopy and imaging instrumenta- Tong, Y., Hocke, L. M., Licata, S. C., & Frederick, B. D. (2012). Low- tion and methodology. NeuroImage, 85, 6–27. frequency oscillations measured in the periphery with near- Schwab, K., Skupin, H., Eiselt, M., Walther, M., Voss, A., & Witte, H. infrared spectroscopy are strongly correlated with blood oxygen (2009). Coordination of the EEG and the heart rate of preterm level-dependent functional magnetic resonance imaging signals. neonates during quiet sleep. Neuroscience Letters. https:// doi. Journal of Biomedical Optics, 17(10), 1060041. https:// doi. org/ org/ 10. 1016/j. neulet. 2009. 09. 03110. 1117/1. jbo. 17. 10. 106004 Sharbrough, F. W., Messick, J. M., & Sundt, T. M. (1973). Correlation Tracy, J. I., & Doucet, G. E. (2015). Resting-state functional connectiv- of continuous electroencephalograms with cerebral blood flow ity in epilepsy: Growing relevance for clinical decision making. measurements during carotid endarterectomy. Stroke. https://doi. Current Opinion in Neurology. https:// doi. org/ 10. 1097/ WCO. org/ 10. 1161/ 01. STR.4. 4. 67400000 00000 000178 Shen, H. H. (2015). Core concept: Resting-state connectivity. Proceed- Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). ings of the National Academy of Sciences of the United States of Semi-supervised Seizure Prediction with Generative Adversarial America. https:// doi. org/ 10. 1073/ pnas. 15187 85112 Networks. 1–6. http:// arxiv. org/ abs/ 1806. 08235 Shmuel, A., & Leopold, D. A. (2008). Neuronal correlates of spon- Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). taneous fluctuations in fMRI signals in monkey visual cortex: Extracting and composing robust features with denoising autoen- Implications for functional connectivity at rest. Human Brain coders. Proceedings of the 25th International Conference on Mapping, 29(7), 751–761. https:// doi. org/ 10. 1002/ hbm. 20580 Machine Learning. https:// doi. org/ 10. 1145/ 13901 56. 13902 94 Siegel, J. S., Snyder, A. Z., Ramsey, L., Shulman, G. L., & Corbetta, Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. M. (2016). The effects of hemodynamic lag on functional con - (2010). Stacked denoising autoencoders: Learning Useful Repre- nectivity and behavior after stroke. Journal of Cerebral Blood sentations in a Deep Network with a Local Denoising Criterion. Flow and Metabolism, 36(12), 2162–2176. https:// doi. org/ 10. Journal of Machine Learning Research. 1177/ 02716 78X15 614846 Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state Sigala, R., Haufe, S., Roy, D., Dinse, H. R., & Ritter, P. (2014). The fNIRS imaging duration for accurate and stable mapping of brain role of alpha-rhythm states in perceptual learning: Insights from connectivity network in children. Scientific Reports , 7(1). https:// experiments and computational models. Frontiers in Compu-doi. org/ 10. 1038/ s41598- 017- 06340-7 tational Neuroscience, 8, 1–19. https:// doi. org/ 10. 3389/ fncom. Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based 2014. 00036 LSTM for aspect-level sentiment classification. EMNLP 2016 - Singh, K. D. (2012). Which “neural activity” do you mean? fMRI, Conference on Empirical Methods in Natural Language Process- MEG, oscillations and neurotransmitters. NeuroImage. https:// ing, Proceedings. https:// doi. org/ 10. 18653/ v1/ d16- 1058 doi. org/ 10. 1016/j. neuro image. 2012. 01. 028 Wang, Z., Lu, G., Zhang, Z., Zhong, Y., Jiao, Q., Zhang, Z., Tan, Q., Sirpal, P., Kassab, A., Pouliot, P., & Nguyen, D. K. (2019). fNIRS Tian, L., Chen, G., Liao, W., Li, K., & Liu, Y. (2011). Altered improves seizure detection in multimodal EEG-fNIRS record- resting state networks in epileptic patients with generalized tonic- clonic seizures. Brain Research, 1374, 134–141. https://d oi.o rg/ ings. Journal of Biomedical Optics, 24(05), 1. https:// doi. org/ 10. 1016/j. brain res. 2010. 12. 034 10. 1117/1. jbo. 24.5. 051408 1 3 558 Neuroinformatics (2022) 20:537–558 Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estima- Zhang, X., Pan, W. J., & Keilholz, S. D. (2020). The relationship between tion of Power Spectra: A Method Based on Time Averaging Over BOLD and neural activity arises from temporally sparse events. Short, Modified Periodograms. IEEE Transactions on Audio and NeuroImage, 207, 116390. https:// doi. org/ 10. 1016/j. neuro image. Electroacoustics. https:// doi. org/ 10. 1109/ TAU. 1967. 11619 012019. 116390 Whittingstall, K., & Logothetis, N. K. (2009). Frequency-band cou- Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Liao, W., Wang, Z., Wang, Z., Li, pling in surface EEG reflects spiking activity in monkey visual K., Chen, H., & Liu, Y. (2010b). Altered spontaneous neuronal cortex. Neuron. https:// doi. org/ 10. 1016/j. neuron. 2009. 08. 016 activity of the default-mode network in mesial temporal lobe epi- Woermann, F. G., & Vollmar, C. (2009). Clinical MRI in children lepsy. Brain Research, 1323, 152–160. https:// doi. org/ 10. 1016/j. and adults with focal epilepsy: A critical review. Epilepsy and brain res. 2010. 01. 042 Behavior, 15(1), 40–49. https:// doi. org/ 10. 1016/j. yebeh. 2009. Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Yang, Z., Liao, W., Chen, Z., 02. 032 Shi, J., & Liu, Y. (2009). Impaired attention network in temporal Yücel, M. A., Selb, J., Aasted, C. M., Lin, P.-Y., Borsook, D., Becerra, lobe epilepsy: A resting FMRI study. Neuroscience Letters, 458(3), L., & Boas, D. A. (2016). Mayer waves reduce the accuracy of esti- 97–101. https:// doi. org/ 10. 1016/j. neulet. 2009. 04. 040 mated hemodynamic response functions in functional near-infrared Zhao, F., Feng, J., Zhao, J., Yang, W., & Yan, S. (2018). Robust LSTM- spectroscopy. Biomedical Optics Express, 7(8), 3078. https:// doi. Autoencoders for Face De-Occlusion in the Wild. IEEE Trans- org/ 10. 1364/ boe.7. 003078 actions on Image Processing. https:// doi. org/ 10. 1109/ TIP. 2017. Zhang, H., Zhang, Y. J., Lu, C. M., Ma, S. Y., Zang, Y. F., & Zhu, C. Z. 27714 08 (2010a). Functional connectivity as revealed by independent com- ponent analysis of resting-state fNIRS measurements. NeuroImage. Publisher's Note Springer Nature remains neutral with regard to https:// doi. org/ 10. 1016/j. neuro image. 2010. 02. 080 jurisdictional claims in published maps and institutional affiliations. Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. https:// doi. org/ 10. 1145/ 32099 78. 32101 84 1 3

Journal

Neuroinformatics – Springer Journals

Published: Jul 1, 2022

Keywords: EEG-fNIRS; Functional brain imaging; Deep neural networks; Epilepsy; Resting state; Functional connectivity; Neurovascular coupling

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

References (166)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies