Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution

Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a... Convolutional neural network for high-accuracy functional near- infrared spectroscopy in a brain– computer interface: three-class classification of rest, right-, and left- hand motor execution Thanawin Trakoolwilaiwan Bahareh Behboodi Jaeseok Lee Kyungsoo Kim Ji-Woong Choi Thanawin Trakoolwilaiwan, Bahareh Behboodi, Jaeseok Lee, Kyungsoo Kim, Ji-Woong Choi, “Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain– computer interface: three-class classification of rest, right-, and left-hand motor execution,” Neurophoton. 5(1), 011008 (2017), doi: 10.1117/1.NPh.5.1.011008. Neurophotonics 5(1), 011008 (Jan–Mar 2018) Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution † † Thanawin Trakoolwilaiwan, Bahareh Behboodi, Jaeseok Lee, Kyungsoo Kim, and Ji-Woong Choi* Daegu Gyeongbuk Institute of Science and Technology, Department of Information and Communication Engineering, Daegu, Republic of Korea Abstract. The aim of this work is to develop an effective brain–computer interface (BCI) method based on func- tional near-infrared spectroscopy (fNIRS). In order to improve the performance of the BCI system in terms of accuracy, the ability to discriminate features from input signals and proper classification are desired. Previous studies have mainly extracted features from the signal manually, but proper features need to be selected care- fully. To avoid performance degradation caused by manual feature selection, we applied convolutional neural networks (CNNs) as the automatic feature extractor and classifier for fNIRS-based BCI. In this study, the hemo- dynamic responses evoked by performing rest, right-, and left-hand motor execution tasks were measured on eight healthy subjects to compare performances. Our CNN-based method provided improvements in classifi- cation accuracy over conventional methods employing the most commonly used features of mean, peak, slope, variance, kurtosis, and skewness, classified by support vector machine (SVM) and artificial neural network (ANN). Specifically, up to 6.49% and 3.33% improvement in classification accuracy was achieved by CNN com- pared with SVM and ANN, respectively. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. [DOI: 10.1117/1.NPh.5.1 .011008] Keywords: functional near-infrared spectroscopy; brain–computer interface; support vector machine; artificial neural network; con- volutional neural network; feature extraction. Paper 17080SSR received Apr. 6, 2017; accepted for publication Aug. 17, 2017; published online Sep. 14, 2017. 10,24 16 (MEG), electrocorticography (ECoG), functional magnetic 1 Introduction 25–27 resonance imaging (fMRI), and functional near-infrared 12,23,28–33 1.1 Brain–Computer Interface spectroscopy (fNIRS). BCI systems based on MEG, ECoG, fMRI, and EEG generally suffer from bulkiness, high A brain–computer interface (BCI) is a means of communication cost, high sensitivity to head movements, low spatial and tem- between the human brain and external devices. BCIs are typi- poral resolution, and low signal quality. fNIRS-based systems cally designed to translate the neuronal activity of the brain to are known to be more advantageous, in that they can provide 1–9 restore motor function, or to control devices. The major com- moderate temporal and spatial resolution. ponents of an effective BCI system are: (1) acquisition of brain signals using a neuroimaging modality, (2) signal processing 1.2 Functional Near-Infrared Spectroscopy-Based and analysis to obtain features representative of the signal, Brain–Computer Interface System and (3) translation of features into commands to control devices. Well-designed BCI systems have proven to be helpful In the last few decades, fNIRS has been recognized as a prom- for patients with severe motor impairment, and have improved ising noninvasive optical imaging technique for monitoring the their quality of life. For instance, many studies have been suc- hemodynamic response of the brain using neurovascular cou- cessfully conducted with patients who have suffered a pling. Neurovascular coupling in the cerebral cortex captures 10,11 12,13 stroke, have amyotrophic lateral sclerosis, or have spinal the increases in oxygenated hemoglobin (HbO) and reductions 14,15 cord injury (SCI), in which BCI systems have allowed them in deoxygenated hemoglobin (HbR) that occur during brain to control external devices. activity. To accomplish this, fNIRS employs multiple light 16,17 BCI systems have been developed based on invasive as sources and detectors, which emit and receive near-infrared 4,18 well as noninvasive neuroimaging modalities, including light (at wavelengths between 650 and 950 nm), respectively. 18–23 electroencephalography (EEG), magnetoencephalography The emitted light passes through the scalp, tissue, and skull 34–36 to reach the brain. The relationship between light attenua- tion (caused by absorption and scattering) and changes in the *Address all correspondence to: Ji-Woong Choi, E-mail: jwchoi@dgist.ac.kr concentration of HbO and HbR can be expressed by the These authors contributed equally to this work modified Beer–Lambert law (MBLL), since HbO and HbR Neurophotonics 011008-1 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . have different absorption coefficients in the near-infrared and the proposed CNN structures are described in Sec. 3. 31,34,36 wavelengths. Sections 4–6 cover the results, discussion, and conclusion, Several recent studies have focused on building fNIRS-based respectively. BCI systems. In previous research, various types of experiments have been performed to measure the accuracy of systems’ clas- 31,33 28–31 sification, using mental arithmetic, motor imagery, 2 Background 23,30,33,37 motor execution, and other approaches. This type of This section describes the details of the commonly used fea- research is particularly important since the final goal of the tures, machine learning-based classifiers, and how the classifi- BCI is to have a system which is able to interpret subject inten- cation performance is evaluated for BCI systems. tion, and any misclassification in the BCI system can lead to accidents for the user. Accordingly, improving classification accuracy is the most essential feature of the BCI-based commu- 2.1 Features Extracted from the Input Signal 38,39 nication system. To this end, it is important to exploit appro- priate classifiers as well as discriminant features that can While a large body of previous studies have reported various accurately represent the variability in the hemodynamic response features which can be used to extract the hemodynamic signal, signal. the most commonly used features for fNIRS-based BCI are sig- Many of the studies on fNIRS-based BCI primarily focused nal mean (μ ), variance (σ ), kurtosis (K ), skewness (S ), x x x x i i i;j i on different types of feature extraction techniques and machine peak, and slope where such features are computed as learning algorithms. For feature extraction, methods of iden- tifying statistical properties such as mean, slope, skewness, kur- N tosis, etc., from time-domain signals filter coefficients from EQ-TARGET;temp:intralink-;e001;326;578μ ¼ x ; (1) x i;j 41,42 continuous and discrete wavelet transforms (DWTs), and j¼1 measurement based on joint mutual information have been used. In addition, for machine learning-based classification, N 2 ðx − μ Þ 6,23,29,31,33,38 i;j x j¼1 i methods such as linear discriminant analysis, sup- EQ-TARGET;temp:intralink-;e002;326;528σ ¼ ; (2) 12,30,44 30 port vector machine (SVM), hidden Markov model, and artificial neural network (ANN) have received consider- N 4 able attention. ðx − μ Þ ∕N i;j x j¼1 i EQ-TARGET;temp:intralink-;e003;326;490K ¼ ; (3) Among the above-mentioned techniques for feature extrac- i 4 tion, most of the studies have relied on extracting the statistical values of the time-domain signal. However, reaching the highest and classification accuracy depends on different factors, such as 46 N 3 selecting the best set of combined features and the size of ðx − μ Þ ∕N j¼1 i;j x EQ-TARGET;temp:intralink-;e004;326;436S ¼ ; (4) the time window. In addition, classification accuracies vary i 3 based on different mother wavelet functions for decomposi- tion, which affect performance in a heuristic sense. To over- where N is the total number of samples of x , x is the i’th row of i i come the limitations of these conventional methods, therefore, the input x, x is the j’th signal amplitude of the input x , and i;j i an appropriate technique for feature extraction needs to be σ is the variance of x . The signal peak is computed by select- x i determined. ing the maximum value of x , and the slope is computed using linear regression. 1.3 Objective The results of previous studies have demonstrated that convolu- 2.2 Support Vector Machine tional neural networks (CNNs) can successfully achieve high classification accuracy in many applications, including image SVM is a discriminative classifier which optimizes a separating 47,48 49 recognition, artificial intelligence, speech detection, and hyperplane by maximizing the distance between the training 50,51 multiple time-series processing. Considering CNNs’ ability data. The decision boundary is obtained by to extract important features from a signal, CNN may be suitable for fNIRS-based BCI as well. Accordingly, our proposed 2 ðiÞ method utilizes CNN to automatically extract the features EQ-TARGET;temp:intralink-;e005;326;275 min kw k þ C ε from the hemodynamic response signal. To be specific, we (5) i¼1 attempt to answer the following two arguments: (1) does ðiÞ ðiÞ s:t:y ðx · w þ b Þ − 1 þ ε ≥ 0; s s s CNN outperform conventional methods in fNIRS-based BCI? (2) How well does CNN work with the input data of the hemo- where w is the weight vector, C> 0 is the regularization dynamic response signal? s ðiÞ parameter, ε > 0 is the training error, y is the true class To address these questions, we compared the classification label for i’th input x , and b is the bias. Among these, C accuracies of CNN with those of conventional methods when s s plays an important role in reducing the error as well as accom- used as the feature extractor and classifier in fNIRS-based modating the outliers through the training process of the data. In BCI. Then, we analyzed how the trained convolutional filters other words, it controls the trade-off between the data training in CNN optimized the features. error and the norm of the weights. As a matter of fact, determin- The rest of this work is organized as follows. In Sec. 2, the properties of the conventional methods as well as CNN are ing a proper C is a vital step in training the SVM on the input briefly introduced. Subsequently, data acquisition, preprocessing, data. Neurophotonics 011008-2 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . updated by the backward propagation by comparing the com- puted output values from the forward propagation with the desired output values, using a loss function. This iteration is per- formed until the minimum loss function value is achieved. To obtain a proper predictive model with ANN, several hyperparameters, such as learning rate, batch size, and number of epochs, should be considered. The learning rate is the param- eter that controls how fast the weight values in the fully con- nected layers can be updated during the training process. Through the batch learning process, training data are separated into several sets, and this is followed by propagation through the training process, where the batch size is the number of samples in each set. An epoch is defined as the total number of times that the training procedure is completed. 2.4 Convolutional Neural Network Fig. 1 The common structure of ANN. CNN is an effective classifier based on deep network learning. It is highly capable of automatically learning appropriate features from the input data by optimizing the weight parameters of each 2.3 Artificial Neural Network filter, using forward and backward propagation to minimize ANN is a classifier, inspired by a biological brain’s axon, with classification errors. the ability to detect patterns in the training data set, which con- CNN consists of several layers, which are called the input sists of assemblies of interconnected artificial neurons that pro- layer, convolutional layer, fully connected hidden layer, and out- vide nonlinear decision boundaries. Typically, ANN consists of put layer (see Fig. 2). In the convolutional layers, a convolu- multiple layers, respectively called the input layer, fully con- tional filter whose width is equal to the dimension of the nected hidden layer(s), and the output layer, with one or input and kernel size (height) of h is convolved with the more neurons in each layer (see Fig. 1). Through forward propa- input data, where the output of the i’th filter is gation, the output values are computed based on the activation function of the hidden layer(s) by EQ-TARGET;temp:intralink-;e008;326;447o ¼ w · x½i∶i þ h − 1; (8) ð1Þ ð1Þ EQ-TARGET;temp:intralink-;e006;63;412o ¼ a½w · x þ b ; (6) i where w is the weight matrix, x½i∶j is the submatrix of input from row i to j, and o is the result value. ð2Þ ð2Þ Then, in order to build the feature map (the input of the next EQ-TARGET;temp:intralink-;e007;63;381y ¼ a½w · o þ b ; (7) layer), the output of the convolutional layer is converted by an activation function similar to ANN. After each convolutional where o is the output of the first fully connected hidden layer layer, additional subsampling operations such as max-pooling calculated by using an activation function a to transform the ð1Þ and dropout are performed to enhance the performance. summation of bias value b , and the multiplication of the ð1Þ Max-pooling is one of the common methods used to reduce input vector x with the weight vector w . Likewise, y is data size, and it stores only the important data. Dropout, which the output of the second fully connected hidden layer, which helps CNN avoid overfitting during the training process, is a is similarly calculated by using the input vector o of the second ð2Þ ð2Þ regularization step that randomly drops out one or more hidden layer, and the weight vector w and the bias b . nodes. As with ANN, the mentioned hyperparameters such as Through the first iteration of the training procedure, weight values should be initialized. Proper weight initialization is one learning rate, batch size, and number of epochs should be of the important operations for improving the classification per- investigated for CNN in order to improve the classification formance of the networks. Afterward, the weight values are performance. Fig. 2 The common structure of convolutional neural network. Neurophotonics 011008-3 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . 3.2 Data Acquisition For data acquisition, LABNIRS (Shimadzu), an fNIRS device with a multichannel continuous wave with three wavelengths (780, 805, and 830 nm) and a sampling rate of 25.7 Hz, was utilized. A total of 12 sources and 12 detectors, resulting in 34 measurement channels, were placed over the motor areas, C3 and C4, according to the international 10–20 system which corresponds to the motor cortex of the right- and left- hand motor execution (see Fig. 4). The distance between source and detector was 3 cm. 3.3 Experimental Procedure The subjects sat on a comfortable chair in front of a computer Fig. 3 Cross-validation procedure. screen, which displayed the experimental tasks. In the experi- ment, subjects were asked to perform a motor execution task in order to generate a robust signal for better discrimination. To be specific, while a black screen was displayed during the 2.5 Cross Validation rest task, an arrow pointing right or left was shown during k-fold cross validation is used to estimate the classification per- each of the right- or left-hand execution tasks, respectively. 53,59 formance of the predictive model. The first step in this proc- All of the subjects were asked to relax before the experiment ess is to divide the data into k-folds, where each fold contains an in order to stabilize blood flow. For data acquisition, the subjects identical amount of the input data. Then, one fold is used as a were trained to relax during the rest tasks and to perform finger test set, while the remaining folds are used as training sets (see tapping during the motor execution tasks. Each subject per- Fig. 3). Afterward, a classification procedure is applied to the formed 10 experiments of five sessions of right- and left- selected test and training sets. This process is performed for hand motor executions, with two rest blocks per session (see each of the k-folds, and the corresponding accuracies obtained Fig. 5). All the blocks lasted 10 s, and each block became a from each test set are averaged to estimate the performance. sample. The data for all the subjects were collected within three days. We eventually obtained a total of 100 samples of rest, 50 samples of right, and 50 samples of left-hand motor exe- 3 Method cution for each subject. 3.1 Participants 3.4 Acquired Data Preprocessing Eight healthy subjects were recruited for the experiment (ages of 3.4.1 Calculation of hemoglobin concentration changes 25.25  3.81 years, three females, all right-handed). The sub- jects were asked to avoid smoking and drinking alcohol or cof- After signal measurement, we converted the signals of light fee within 3 h prior to the experiment. None of the subjects had intensity into concentration changes of HbO and HbR by been reported for any neurological or brain injuries. Written MBLL, utilizing statistical toolbox NIRS-SPM. The MBLL consent forms were obtained from all subjects. The experiment equation is given by was approved by the Daegu Gyeongbuk Institute of Science and EQ-TARGET;temp:intralink-;e009;326;315 HbO HbR −1 Δ½HbO ε ε ΔOD Technology (DGIST) Institutional Review Board (DGIST- 1 λ1 λ1 λ1 ¼ ; (9) 170414-HR-004-01). HbO HbR d · DPF Δ½HbR ε ε ΔOD λ2 λ2 λ2 Fig. 4 (a) A subject with optodes over motor area C3 and C4 based on the international 10–20 system and (b) the source and detector configuration. Channel numbers 1 to 17 and 18 to 34 were placed over motor areas C4 and C3, respectively. Neurophotonics 011008-4 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . EQ-TARGET;temp:intralink-;e013;326;752S½n¼ a þ d : (13) j j In order to remove the undesired high- and low-frequency Fig. 5 Experimental procedure includes rest and two motor tasks: noises, we exploited a 10-level wavelet decomposition with a right- and left-hand motor execution. 62 Daubechies (db5) mother function. In addition, we used a bandpass frequency between 0.02 and 0.1 Hz, in which the com- bination of low-frequency components d and d from the 10- 8 9 where Δ½HbO and Δ½HbR are the changes in HbO and HbR level decompositions was solely in the same 0.02- to 0.1-Hz fre- concentration, respectively, d is the distance between the light quency range. Therefore, the filtered signal was reconstructed source and detector, DPF is the differential path length factor, ε based on d and d by S ½n¼ d þ d . After filtering, 8 9 denoised 8 9 is the extinction coefficient at wavelength λ, and ΔOD is the the hemodynamic response signals were normalized into optical density change. range (0,1) by subtracting with the signal mean and scaling. 3.5 Feature Extraction and Classification 3.4.2 Filtering After filtering, we trained and tested the classifiers for each indi- The acquired hemodynamic signal contains various physiologi- vidual subject based on the extracted features. Following the cal noises, including the heart rate at 0.8 Hz, respiration at training step, we computed the classification accuracies from 0.2 Hz, Mayer wave at 0.1 Hz, and very-low-frequency oscil- 36,38,40 both the conventional methods (SVM- and ANN-based lations at 0.03 Hz. Among various possible criteria, we 62 fNIRS) and the proposed method (CNN-based fNIRS). In employed wavelet filtering to remove physiological noise. this section, we discuss the details of the conventional methods The wavelet transform is an efficient method of signal analy- and our proposed CNN structure. sis and performs by adjusting its window width in both time and frequency domains. For denoising a signal S½n, first, wavelet coefficients are obtained by shifting and dilating the waveforms 3.5.1 Conventional methods of the so-called mother function ψ½n, and then important coef- As mentioned, features were extracted after the filtering step, ficients are selected to reconstruct the signal by thresholding. followed by normalizing into range (0,1). The obtained input For a more comprehensive analysis, we also exploited multire- data contained 408 feature dimensions (6 features ×2 signal solution analysis (MRA) which decomposes signals into a tree 41,63 of HbO and HbR ×34 channels). Using such features with structure using the DWT. Using MRA based on DWT, S½n the settings above, we evaluated the performance of the conven- can be approximated by expanding both low- and high-fre- tional methods by observing the concentration changes of HbO quency coefficients for M time points as and HbR over all channels using SVM and ANN. Before applying SVM, since such high-dimensional features 1 64 usually suffer from performance degradation in classifiers, a EQ-TARGET;temp:intralink-;e010;63;409S½n¼ pffiffiffiffiffi A ½j ;kΦ ½n ϕ 0 j ;k principle component analysis (PCA) was utilized to decrease the dimensions of the data. This reduces the aforementioned XX effect by maximizing the variance using a smaller number of þ pffiffiffiffiffi D ½j; kΨ ½n; (10) ψ j;k 52 53,65 principle components. Grid search was used to determine j¼j the number of principle components and the C regularization parameters in SVM, and the combination of both parameters 1 n−k pffiffi where Φ ½n and Ψ ½n¼ ψð Þ are scaling and wavelet j ;k j;k which yielded the highest classification accuracy was selected. 0 j j In this study, we report the results for linear SVM and multi- mother functions, respectively, in which the mother function ple structures of ANN (see Table 1). To be specific, structures of ψ½n is dilated with scaling parameter j, translated by k ANN with one hidden layer (ANN1) and two hidden layers which is the number of decomposition levels, and represented (ANN2) were evaluated. For further comprehensive investiga- as Ψ ½n. Since these functions are orthogonal to each other, j;k tion, each structure of ANN was considered with various taking the inner product results in obtaining the approximation pffiffiffiffi coefficients (low frequency) A ½j ;k¼ S½nΦ ½n and ϕ 0 n j ;k Table 1 Structures of ANN. the detailed coefficients (high frequency) D ½j; k¼ pffiffiffiffi S½nΨ ½n. By denoting n j;k Structure Hidden layer Neurons in each hidden layer ANN1-a 1 128 EQ-TARGET;temp:intralink-;e011;63;218a ¼ pffiffiffiffiffi A ½j ;kΦ ½n; (11) j ϕ 0 j ;k 0 0 ANN1-b 1 256 and ANN1-c 1 512 ANN2-a 2 256, 128 EQ-TARGET;temp:intralink-;e012;63;160d ¼ pffiffiffiffiffi D ½j; kΨ ½n; (12) j ψ j;k k ANN2-b 2 512, 256 ANN2-c 2 512, 128 Eq. (10) can be rewritten as Neurophotonics 011008-5 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 2 Hyperparameters of each individual subject for ANN. Subject Parameters ANN1-a ANN1-b ANN1-c ANN2-a ANN2-b ANN2-c 1 Epochs 50 50 100 20 100 100 Batch size 64 64 16 16 16 32 Learning rate 0.001 0.0005 0.001 0.0005 0.001 0.001 2 Epochs 100 100 100 100 50 50 Batch size 16 16 32 16 16 16 Learning rate 0.0005 0.001 0.0005 0.0001 0.0005 0.001 3 Epochs 100 100 50 50 50 100 Batch size 16 16 64 64 32 32 Learning rate 0.0001 0.0001 0.0005 0.0001 0.0001 0.0001 4 Epochs 50 100 50 50 100 100 Batch size 16 32 64 32 32 64 Learning rate 0.0005 0.0005 0.0005 0.0005 0.0001 0.0005 5 Epochs 100 100 100 100 100 100 Batch size 32 64 64 16 64 64 Learning rate 0.0005 0.0005 0.0005 0.0001 0.001 0.001 6 Epochs 100 50 100 100 100 100 Batch size 16 16 32 16 16 16 Learning rate 0.001 0.001 0.0005 0.0005 0.0005 0.0005 7 Epochs 100 100 100 100 50 100 Batch size 16 32 16 16 16 64 Learning rate 0.0005 0.0005 0.001 0.0005 0.0005 0.001 8 Epochs 100 100 50 50 50 100 Batch size 64 64 32 32 32 16 Learning rate 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 numbers of neurons. All of the aforementioned hyperparameters (CNN1) and three convolutional layers (CNN2). Furthermore, were tuned for each subject (see Table 2). each structure of CNN was considered with a distinct number of filters (see Table 3). All of the convolutional filters in the convolutional layers 3.5.2 Proposed structures of convolutional neural network performed one-dimensional convolution with the input data along the vertical axis, as shown in Fig. 6. Each convolutional Instead of using other methods, we employed CNN as the fea- layer consisted of filters with a kernel size of 3, and an ture extractor as well as the classifier in this study. As the input algorithm was used to update the weight values in the training data, the changes in HbO and HbR concentration over all chan- process. After each convolutional layer, max-pooling with a ker- nels were passed through CNN layers using the structures pre- nel size of 2 was applied, followed by dropout with a dropout sented in Table 3. The input data for CNN were an M by N rate of 50%. The first and second fully connected layers con- matrix, where M is the number of points during 10 s that cor- tained 256 and 128 hidden nodes, respectively. The output respond to the sampling rate (M ¼ time × sampling rate ≈ 257) layer had 3 nodes corresponding to the three classes, which and N is the number of channels for both HbO and HbR (34 were classified using softmax. For better understanding of channels each of HbO and HbR). Similar to the process used the structures mentioned here, the input and output sizes of to evaluate the conventional methods, we considered two struc- each layer in our proposed CNN2-a are summarized in Table 4. tures of CNN, that is, CNN with one convolutional layer Neurophotonics 011008-6 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 3 Structures of CNN. Table 4 Input and output size of the CNN2-a. Input Output Structure Convolutional layer Filters in each convolutional layer Layer size size Properties CNN1-a 1 32 Convolutional layer 1 257, 68 257, 32 32 filters with kernel size 3 CNN1-b 1 64 Max-pooling 1 257, 32 128, 32 Kernel size 2 CNN2-a 3 32, 32, 32 Dropout 1 128, 32 128, 32 Dropout rate 50% CNN2-b 3 64, 64, 64 Convolutional layer 2 128, 32 128, 32 32 filters with kernel size 3 Max-pooling 2 128, 32 64, 32 Kernel size 2 Dropout 2 64, 32 64, 32 Dropout rate 50% Convolutional layer 3 64, 32 64, 32 32 filters with kernel size 3 Max-pooling 3 64, 32 32, 32 Kernel size 2 Dropout 3 32, 32 32, 32 Dropout rate 50% Fully connected layer 1 1024 256 256 hidden nodes Fully connected layer 2 256 128 128 hidden nodes Output layer 128 3 3 hidden nodes utilized in this study because of its automatic feature extraction property. To provide better insights into the feature extraction perfor- mance, a visualization of the features extracted by the aforemen- Fig. 6 The input data consisted of the concentration changes of HbO tioned methods is shown and compared. Because high- (red) and HbR (blue) overall channels. A convolutional filter ran dimensional data are difficult to visualize, the PCA was applied through the input data along the vertical axis. to reduce the dimensionality of the data. In this study, we also compared the visualization of the In the proposed structure, the activation functions of all hemodynamic response signals with the features extracted by layers were set to a rectified linear unit (ReLU), which is a non- conventional methods and convolutional filter, by plotting the linear function, as shown in first two principle components of the PCA. The overall pro- cedure to visualize signal features is shown in Fig. 7. 0;x < 0 EQ-TARGET;temp:intralink-;e014;63;317aðxÞ¼ : (14) x; x ≥ 0 3.7 Computational Time in the Classification In our work, various machine learning algorithms were applied Unlike other activation functions, ReLU avoids a vanishing to classify tasks, including rest, right-, and left-hand motor exe- gradient and in practice converges to the optimum point much cutions. For the ANN and CNN, we trained the model using faster. Consequently, it improves the training process of deep GPU GeForce GTX 1070. The data were divided equally neural network architectures on large scale and complex into 10-folds, and then nine folds were used as a training set. data sets. To imitate the environment of a real application, a single sample In addition, the hyperparameters for training all the CNN was fed through the trained model then the computational time structures, including learning rate, number of epochs, and was measured. For training SVM, ANN, and CNN, the hyper- batch size, were chosen for each individual subject using parameters such as C regularization, number of epochs, and Grid search (see Table 5). Adam was applied as a gradient learning rate were set to 1, 1, and 0.01, respectively. descent optimization algorithm, whose parameters β , β , and 1 2 −8 ϵ were set to 0.9, 0.1, and 10 , respectively. 4 Results 4.1 Measured Hemodynamic Responses 3.6 Visualization of Feature Extraction In the experiment, the changes in HbO and HbR concentration Many previous studies of feature extraction in fNIRS-based BCI were measured as the input data for classification. The average have been reported in the past. Since appropriate features and of the hemodynamic response signals was obtained with respect classifiers are desired in order to achieve high classification to the samples from subjects 1 and 2, across full sessions of each accuracy, the proposed method of the CNN exploitation was task for rest, right-, and left-hand motor executions and are Neurophotonics 011008-7 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 5 Hyperparameters of each individual subject for CNN. cerebral cortex during the right-hand motor execution [see Fig. 9(b)], while it is larger in the right cerebral cortex during the left-hand motor execution [see Fig. 9(c)]. The brain behav- Subject Parameters CNN1-a CNN1-b CNN1-c CNN2-a iors observed in Fig. 9 demonstrate that three-class discrimina- tion, for rest, right-, and left-hand motor execution, can be 1 Epochs 100 100 100 100 achieved, since they show different patterns of cortical activation Batch size 16 64 32 16 over the left and right hemispheres. Learning rate 0.001 0.001 0.001 0.0005 4.2 Classification Accuracies 2 Epochs 50 50 100 100 To determine the classification accuracies of the SVM, ANNs, and CNNs, we employed 10-fold cross validation to estimate Batch size 32 16 16 64 performance and to optimize hyperparameters, as we attempted to discriminate the three classes of rest, right-, and left-hand Learning rate 0.0005 0.001 0.0005 0.001 motor execution. In this section, the classification accuracies 3 Epochs 50 100 50 100 of commonly used features classified by SVM and ANNs are compared with those obtained by CNNs. Batch size 16 64 64 32 To be specific, the classification accuracies for all the tested criteria for the individual subjects are presented in Table 6.As Learning rate 0.0001 0.0005 0.001 0.0005 expected, the results of all the individual subjects indicate that 4 Epochs 100 100 100 100 the use of CNN was significantly superior to SVM and ANN. For convenience of analysis, the average of the classification Batch size 32 32 16 16 accuracies of SVM, ANN, and CNN (86.19%, 89.35%, and 92.68%, respectively) are presented in Fig. 10, which confirms Learning rate 0.0001 0.001 0.0001 0.0005 the superior performance of CNN over the conventional meth- 5 Epochs 50 50 100 100 ods. This superior performance is due to CNN’s ability to learn the inherent patterns of the input data, by updating the weight Batch size 64 16 32 64 values of the convolutional filters. The learning performance can be affected by the size of the Learning rate 0.001 0.0005 0.001 0.001 training set, and this is especially true for ANN and CNN, where 6 Epochs 100 100 50 100 a larger-sized training set usually provides higher classification performance. To examine the effect of the size of the data set on Batch size 16 32 16 32 the classification accuracy, the average classification accuracies across all the subjects were obtained, based on different numbers Learning rate 0.0005 0.0001 0.001 0.001 of samples. 7 Epochs 50 100 100 100 To evaluate the classification performance, 10-fold cross val- idation was utilized. For all the classification methods, the clas- Batch size 64 32 64 16 sification performance was found to increase with the number of samples in the data set, and the classification accuracy of CNN Learning rate 0.001 0.001 0.0005 0.0005 outperformed other tested methods for all numbers of samples 8 Epochs 50 100 100 100 (see Fig. 11). Moreover, the CNN was also able to attain higher accuracy with smaller numbers of samples; for instance, CNN Batch size 64 32 64 16 exceeded 90% accuracy with 120 samples, whereas ANN required 200 samples to reach 89% accuracy. Learning rate 0.001 0.001 0.0005 0.0005 4.3 Analysis of Feature Extraction Performance shown in Figs. 8(a)–8(c), respectively. Each row of the input To better understand the feature extraction performance, we data indicates signal amplitudes. These are represented by visualized the three classes of rest, right-, and left-hand red and blue colors, which imply the maximum and minimum motor executions. To be specific, three classes were visualized amplitudes, respectively. The beginning and the end of the tasks using the hemodynamic response signals, features extracted by correspond to 0 and 10 s, respectively. the conventional methods, and the output of the first layer con- As is widely known, neural activity induces typical changes in volutional filter, by plotting the first and second principle com- cerebral blood oxygenation, resulting in increases in HbO con- ponents of PCA (see Fig. 12). The results for subjects 1 and 2 centration and decreases in HbR concentration. In our results, show that the features extracted by the convolutional filters are a similar behavior in the hemodynamic response can be observed, better discriminated compared with commonly used features as shown in Fig. 8. To be specific, the signals obtained from chan- and the hemodynamic response signals. nels over C3 show higher cortical activation of HbO over a period When considering just the binary classification of rest and of 5 to 10 s during the right-hand motor execution [see Fig. 8(b)], motor execution, both the conventional methods and CNN whereas the signals over C4 have higher activation during the left- resulted in well-separable features. However, for the binary clas- hand motor execution [see Fig. 8(c)]. sification of right- and left-hand motor executions, and for mul- Figure 9 shows the averaged signals for the entire experiment ticlass classification, it was clear that features extracted by the over all channels of the left and right hemispheres. It is obvious convolutional filter were better discriminated as compared with that the change in HbO concentration is higher in the left the conventional methods. Neurophotonics 011008-8 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 7 The overall procedure to visualize signal features, including the hemodynamic response signal, commonly used features in fNIRS-based BCI, and output of the convolutional filter (feature map). The first and second principle components of the signal features are illustrated for the visualization. Fig. 8 Average hemodynamic response of each execution task measured from subject 1 and 2: (a) rest, (b) right-, and (c) left-hand motor execution. Each input presents concentration changes of HbO and HbR overall 34 channels. Red and blue colors represent the maximum and minimum amplitude, respectively. 4.4 Convolutional Filters of Convolutional Neural CNN, we examined the first layer of CNN to determine whether Network it is able to identify the distinguishable channels from the input or not. By training the data using forward and backward One might notice that CNN is able to recognize the patterns of propagations, we let CNN learn how to emphasize some chan- three different classes by updating its filters’ weight values. nels containing distinguishable signals by increasing the corre- Therefore, to further investigate the convolutional filters of sponding weight values, since each column of convolutional Neurophotonics 011008-9 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 9 Average signal amplitude of subjects 1 and 2 across left (C3) and right (C4) hemisphere from full sessions of each class: (a) rest, (b) right-, and (c) left-hand motor execution. Red and blue colors imply HbO and HbR, respectively. Solid and dot lines are related to the C3 and C4 motor areas in that order. Table 6 Classification accuracies of the individual subjects (%). S1 S2 S3 S4 S5 S6 S7 S8 Average SVM 88.50 79.00 84.00 84.50 90.50 97.00 99.00 67.00 86.19 ANN1-a 91.67 85.33 84.83 85.00 94.20 96.17 96.50 76.33 88.75 ANN1-b 92.83 83.83 84.67 87.67 94.30 96.00 96.33 75.00 88.83 ANN1-c 92.83 85.67 85.17 87.50 94.50 96.50 96.67 75.83 89.33 ANN2-a 93.67 86.67 84.33 87.50 95.67 96.50 96.83 75.67 89.61 ANN2-b 92.50 86.17 85.67 88.67 94.83 97.00 97.17 76.08 89.76 ANN2-c 92.17 88.00 85.50 87.00 94.83 97.00 97.67 76.17 89.79 CNN1-a 95.00 91.67 84.83 95.67 97.33 99.00 98.67 80.33 92.81 CNN1-b 95.33 92.17 85.83 96.00 96.83 99.00 99.00 80.50 93.08 CNN2-a 94.33 91.83 82.17 95.00 96.67 98.67 98.33 82.17 92.40 CNN2-b 92.83 93.17 83.33 94.33 96.50 99.00 98.17 82.00 92.42 Neurophotonics 011008-10 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 10 Average classification accuracies of the individual subjects. Fig. 11 Average classification accuracies across all the subjects, based on different number of samples. filter interacts with each channel from the input data. To right-hand motor execution, and filter 2 detects left-hand approximate the most distinguishable channel, each column motor execution. of the convolutional filter was averaged after training. Then, the channel of all of the samples of the input data with the high- 4.5 Computational Time est weight value of the averaged convolutional filter was selected for visualization. The computational time for each of the classification algorithms, In order to visualize the essential information, the most dis- i.e., SVM, ANN, and CNN, was averaged across all subjects and tinguishable channels from all the samples were selected. Two structures (see Table 7). For the training process, the computa- examples of the CNN filter weight values from subject 1 are tional time for CNN was ∼2 and 183 times greater than ANN shown in Fig. 13, where each row represents the most distin- and SVM, respectively. For testing time, the computational time guishable signal from a single sample and the red and blue for CNN was ∼6 and 81 times greater than ANN and SVM, colors indicate the maximum and minimum amplitudes, respec- respectively. The computational time for CNN in the training tively. We found that over a period of 5 to 10 s, there were and testing process was longer than ANN and SVM, as its struc- ture is deeper and more complex. However, it provides a better remarkable differences in the signals chosen from both filters performance in terms of classification accuracy. for the three classes of rest, right-, and left-hand motor execution. Subsequently, in Fig. 13(a) which represents the rest task, 5 Discussion both filters have low signal amplitude. Figure 13(b) represents The primary aim of the present study was to evaluate the use the right-hand motor execution, in which filter 1 shows a of CNN versus conventional methods in fNIRS-based higher signal amplitude than filter 2. In the same manner, BCI, particularly in light of the automatic feature extraction Fig. 13(c) shows the left-hand motor execution, in which property of CNN. The proposed and conventional methods filter 2 exhibits a higher signal amplitude compared with filter were investigated to compare their respective classification 1. Therefore, it can be concluded that filter 1 can detect accuracies. Neurophotonics 011008-11 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 12 The visualization of the hemodynamic response signals, commonly used features, and output of the convolutional filter from (a) subject 1 and (b) subject 2. Fig. 13 Each filter trained by subject 1 represents signals from a channel in every samples correspond- ing to the highest weight value. The filters represent three classes in the classification: (a) rest, (b) right-, and (c) left-hand motor execution. Neurophotonics 011008-12 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 7 Computational time(s). On the other hand, in the case of vital applications, systems to control assistive technology devices for a patient with motor impairment require very high accuracy, since any misclassifica- Training time Testing time tion would probably lead to a serious accident. Consequently, in such cases the proposed method is recommended even if it takes SVM 0.00645 0.00059 a longer time, because it achieves higher accuracy with a smaller ANN 0.63299 0.00734 number of samples (see Fig. 11). CNN 1.17945 0.04751 6 Conclusions To enhance the classification accuracy of an fNIRS-based BCI system, we applied CNN for automatic feature extraction and classification, and compared those results with results from con- In the experiment, motor execution tasks performed by ventional methods employing SVM and ANN, with features of healthy subjects were utilized to obtain strong and robust hemo- mean, peak, slope, variance, kurtosis, and skewness. From the dynamic response signals. However, in real applications, motor measurement results for rest, right-, and left-hand motor execu- imagery can produce a greater impact than motor execution tion on eight subjects, the CNN-based scheme provided up to tasks, in both healthy users and in patients with severe motor 6.49% higher accuracy over conventional feature extraction impairment. A previous study reported that the cortical activa- and classification methods, because the convolutional filters tion resulting from motor execution is similar to motor imagery. can automatically extract appropriate features. Hence, it is feasible that a healthy user or a patient without a The results confirmed that there was an improvement in brain injury, such as SCI, will be able to use motor imagery accuracy when using CNN over the conventional methods, for commands instead of motor execution. Further investigation which can lead to the practical development of a BCI system. of the use of motor imagery, and the study of patients with neu- Since classification accuracy is the most essential factor for rological disorders, will be explored in the future. many BCI applications, we will explore further improvements in The results of the classification accuracies in Fig. 10 imply the accuracy of fNIRS-based BCI by implementing various deep that the proposed method using CNN outperforms the conven- learning techniques, as well as combining fNIRS with other neu- tional methods. To be specific, the analysis of signal features by roimaging modalities. To investigate clinical applications, we visualizing the first and second principle components demon- will also undertake experiments with patients. strates that the features extracted by the convolutional filter yield better discriminating features than conventional methods, Disclosures because it is capable of learning appropriate features from the training data. The authors declare that there is no conflict of interest regarding Additionally, the channels corresponding to the highest the publication of this paper. weight value in the trained CNN filter demonstrate that the con- volutional filter emphasizes the discriminating signal from the Acknowledgments training data. It is also worthwhile to note that while the perfor- This work was supported in part by the Basic Science Research mance of feature extraction for the binary classification of rest Program through the National Research Foundation of Korea and motor execution was similar for both the conventional and (NRF) funded by the Ministry of Science and ICT (No. proposed methods, since they showed well-discriminated fea- NRF-2015R1A2A2A01008218), the DGIST R&D Program tures, the proposed method performed better for multiclass of the Ministry of Science and ICT (No. 17-BD-0404) and data. This is because the convolutional filter is able to transform the Robot industry fusion core technology development project mixed data into well-separated data. of the Ministry of Trade, Industry & Energy of Korea Consequently, the proposed method will be appropriate for (No. 10052980). various systems that require multitasks to command. For in- stance, a brain-controlled wheelchair requires multiclass classi- fication to control the wheelchair in several directions. Although References the proposed method requires a longer time for training, it per- 1. T. O. Zander and C. Kothe, “Towards passive brain–computer interfa- forms better in multiclass classification. ces: applying brain–computer interface technology to human-machine The number of samples used to train the classifier directly systems in general,” J. Neural Eng. 8(2), 025005 (2011). affects classification accuracy, especially in the complex classi- 2. J. R. Wolpaw et al., “Brain–computer interface technology: a review of fier, as shown in Fig. 11. For a small number of samples, the the first international meeting,” IEEE Trans. Rehabil. Eng. 8(2), 164– 173 (2000). classification performances of SVM, ANN, and CNN were sim- 3. J. R. Wolpaw et al., “Brain–computer interfaces for communication and ilar. However, as the number of samples increased, the complex control,” Clin. Neurophysiol. 113(6), 767–791 (2002). classifier was able to achieve higher accuracy than the simple 4. G. Dornhege, Toward Brain-Computer Interfacing, MIT Press, classifier, though the computational time to train was much Cambridge, Massachusetts (2007). greater than that of the simple classifier. 5. R. P. Rao, Brain-Computer Interfacing: An Introduction, Cambridge This means there is a trade-off between accuracy and ease of University Press, Cambridge, Massachusetts (2013). 6. V. Kaiser et al., “Cortical effects of user training in a motor imagery use when building the appropriate BCI system. The user must based brain–computer interface measured by fNIRS and EEG,” take a longer time to train the complex classifier to obtain a high- NeuroImage 85, 432–444 (2014). performance classifier. When an application requires ease of 7. J. J. Daly and J. R. Wolpaw, “Brain–computer interfaces in neurological use over safety, the conventional methods might be more appro- rehabilitation,” The Lancet Neurol. 7(11), 1032–1043 (2008). priate, since a shorter time and smaller-sized training set are 8. C. Neuper et al., “Motor imagery and EEG-based control of spelling desired. devices and neuroprostheses,” Prog. Brain Res. 159, 393–409 (2006). Neurophotonics 011008-13 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . 9. K. LaFleur et al., “Quadcopter control in three-dimensional space using 35. F. Matthews et al., “Hemodynamics for brain–computer interfaces,” a noninvasive motor imagery-based brain–computer interface,” J. IEEE Signal Process. Mag. 25(1), 87–94 (2008). Neural Eng. 10(4), 046003 (2013). 36. H. Liang, J. D. Bronzino, and D. R. Peterson, Biosignal Processing: 10. E. Buch et al., “Think to move: a neuromagnetic brain–computer Principles and Practices, CRC Press, Boca Raton, Florida interface (BCI) system for chronic stroke,” Stroke 39(3), 910–917 (2012). (2008). 37. X. Cui, S. Bray, and A. L. Reiss, “Functional near infrared spectroscopy 11. K. Saita et al., “Combined therapy using botulinum toxin A and single- (NIRS) signal improvement based on negative correlation between oxy- joint hybrid assistive limb for upper-limb disability due to spastic hemi- genated and deoxygenated hemoglobin dynamics,” NeuroImage 49(4), plegia,” J. Neurol. Sci. 373, 182–187 (2017). 3039–3046 (2010). 12. U. Chaudhary et al., “Brain–computer interface-based communication 38. S. Weyand et al., “Usability and performance-informed selection of per- in the completely locked-in state,” PLoS Biol. 15(1), e1002593 (2017). sonalized mental tasks for an online near-infrared spectroscopy brain– 13. U. Chaudhary, N. Birbaumer, and M. Curado, “Brain-machine interface computer interface,” Neurophotonics 2(2), 025001 (2015). (BMI) in paralysis,” Ann. Phys. Rehabil. Med. 58(1), 9–13 (2015). 39. D. P.-O. Bos, M. Poel, and A. Nijholt, “A study in user-centered design 14. G. Pfurtscheller et al., “EEG-based asynchronous BCI controls func- and evaluation of mental tasks for BCI,” in Int. Conf. on Multimedia Modeling, pp. 122–134, Springer (2011). tional electrical stimulation in a tetraplegic patient,” EURASIP J. Appl. Signal Process. 2005, 3152–3155 (2005). 40. N. Naseer and K.-S. Hong, “fNIRS-based brain–computer interfaces: a review,” Front. Hum. Neurosci. 9, 3 (2015). 15. K. L. Koenraadt et al., “Preserved foot motor cortex in patients with complete spinal cord injury: a functional near-infrared spectroscopic 41. B. Abibullaev and J. An, “Classification of frontal cortex haemodynamic study,” Neurorehabil. Neural Repair 28(2), 179–187 (2014). responses during cognitive tasks using wavelet transforms and machine 16. E. C. Leuthardt et al., “A brain–computer interface using electrocortico- learning algorithms,” Med. Eng. Phys. 34(10), 1394–1410 (2012). graphic signals in humans,” J. Neural Eng. 1(2), 63–71 (2004). 42. T. Q. D. Khoa and M. Nakagawa, “Functional near infrared spectro- scope for cognition brain tasks by wavelets analysis and neural net- 17. T. N. Lal et al., “Methods towards invasive human brain–computer inter- faces,” in Conf. on Neural Information Processing Systems (NIPS), works,” Int. J. Biol. Med. Sci. 1,28–33 (2008). pp. 737–744 (2004). 43. X. Yin et al., “Classification of hemodynamic responses associated with 18. N. Birbaumer et al., “A spelling device for the paralysed,” Nature force and speed imagery for a brain–computer interface,” J. Med. Syst. 39(5), 53 (2015). 398(6725), 297–298 (1999). 19. L. Parra et al., “Linear spatial integration for single-trial detection in 44. X. Cui, S. Bray, and A. L. Reiss, “Speeded near infrared spectroscopy encephalography,” NeuroImage 17(1), 223–230 (2002). (NIRS) response detection,” PLoS One 5(11), e15474 (2010). 45. B. Abibullaev, J. An, and J.-I. Moon, “Neural network classification 20. M. Cheng et al., “Design and implementation of a brain–computer inter- face with high transfer rates,” IEEE Trans. Biomed. Eng. 49(10), 1181– of brain hemodynamic responses from four mental tasks,” Int. J. 1186 (2002). Optomechatronics 5(4), 340–359 (2011). 21. A. Buttfield, P. W. Ferrez, and J. R. Millan, “Towards a robust BCI: error 46. N. Naseer et al., “Analysis of different classification techniques for two- class functional near-infrared spectroscopy-based brain–computer inter- potentials and online learning,” IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 164–168 (2006). face,” Comput. Intell. Neurosci. 2016, 5480760 (2016). 22. B. Blankertz et al., “The non-invasive Berlin brain–computer interface: 47. Y. Bengio et al., “Scaling learning algorithms towards AI,” Large-Scale fast acquisition of effective performance in untrained subjects,” Kernel Mach. 34(5), 1–41 (2007). NeuroImage 37(2), 539–550 (2007). 48. P. Y. Simard et al., “Best practices for convolutional neural networks 23. S. Fazli et al., “Enhanced performance by a hybrid NIRS–EEG brain– applied to visual document analysis,” in 7th Int. Conf. on Document computer interface,” NeuroImage 59(1), 519–529 (2012). Analysis and Recognition (ICDAR), Vol. 3, pp. 958–962 (2003). 49. D. Silver et al., “Mastering the game of go with deep neural networks 24. J. Mellinger et al., “An MEG-based brain–computer interface (BCI),” NeuroImage 36(3), 581–593 (2007). and tree search,” Nature 529(7587), 484–489 (2016). 25. S. M. LaConte, “Decoding fMRI brain states in real-time,” NeuroImage 50. S. Sukittanon et al., “Convolutional networks for speech detection,” in 56(2), 440–454 (2011). Interspeech (2004). 26. N. Weiskopf et al., “Principles of a brain–computer interface (BCI) 51. Y. Bengio et al., “Learning deep architectures for AI,” Found. Trends based on real-time functional magnetic resonance imaging (fMRI),” Mach. Learn. 2(1), 1–127 (2009). IEEE Trans. Biomed. Eng. 51(6), 966–970 (2004). 52. J. L. Semmlow and B. Griffel, Biosignal and Medical Image Processing, CRC Press, Boca Raton, Florida (2014). 27. B. Sorger et al., “Another kind of BOLD response: answering multiple- choice questions via online decoded single-trial brain signals,” Prog. 53. C.-W. Hsu et al., “A practical guide to support vector classification,” Brain Res. 177, 275–292 (2009). Technical Report, pp. 1–16, Department of Computer Science, 28. S. M. Coyle, T. E. Ward, and C. M. Markham, “Brain–computer inter- National Taiwan University, Taipei (2003). face using a simplified functional near-infrared spectroscopy system,” 54. M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical J. Neural Eng. 4(3), 219–226 (2007). Foundations, Cambridge University Press, Cambridge, Massachusetts 29. N. Naseer and K.-S. Hong, “Classification of functional near-infrared (2009). spectroscopy signals corresponding to the right-and left-wrist motor 55. Y. A. LeCun et al., “Efficient backprop,” in Neural Networks: Tricks of imagery for development of a brain–computer interface,” Neurosci. the Trade, G. Montavon, G. B. Orr, and K.-R. Müller, Eds., pp. 9–48, Lett. 553,84–89 (2013). Springer, Heidelberg (2012). 30. R. Sitaram et al., “Temporal classification of multichannel near-infrared 56. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford spectroscopy signals of motor imagery for developing a brain–computer University Press, Oxford, Mississippi (1995). interface,” NeuroImage 34(4), 1416–1427 (2007). 57. Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners’ 31. K.-S. Hong, N. Naseer, and Y.-H. Kim, “Classification of prefrontal and guide to) convolutional neural networks for sentence classification,” motor cortex signals for three-class fNIRS-BCI,” Neurosci. Lett. 587, CoRR abs/1510.03820 (2015), http://arxiv.org/abs/1510.03820. 87–92 (2015). 58. N. Srivastava et al., “Dropout: a simple way to prevent neural networks 32. J. Shin and J. Jeong, “Multiclass classification of hemodynamic from overfitting,” J. Mach. Learn. Res. 15(1), 1929–1958 (2014). responses for performance improvement of functional near-infrared 59. S. Arlot et al., “A survey of cross-validation procedures for model selec- spectroscopy-based brain–computer interface,” J. Biomed. Opt. 19(6), tion,” Stat. Surv. 4,40–79 (2010). 067009 (2014). 60. R. W. Homan, J. Herman, and P. Purdy, “Cerebral location of 33. M. J. Khan, M. J. Hong, and K.-S. Hong, “Decoding of four movement international 10–20 system electrode placement,” Electroencephalogr. directions using hybrid NIRS-EEG brain–computer interface,” Front. Clin. Neurophysiol. 66(4), 376–382 (1987). Hum. Neurosci. 8, 244 (2014). 61. J. C. Ye et al., “NIRS-SPM: statistical parametric mapping for near- 34. F. Scholkmann et al., “A review on continuous wave functional near- infrared spectroscopy,” NeuroImage 44(2), 428–447 (2009). infrared spectroscopy and imaging instrumentation and methodology,” 62. H. Tsunashima, K. Yanagisawa, and M. Iwadate, “Measurement NeuroImage 85,6–27 (2014). of brain function using near-infrared spectroscopy (NIRS),” in Neurophotonics 011008-14 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . master’s student at Daegu Gyeongbuk Institute of Science and Neuroimageing-Methods, B. Peter, Ed., INTECH Open Access Technology, South Korea. Her research interests include functional Publisher, Rijeka, Croatia (2012). near-infrared spectroscopy and functional magnetic resonance 63. K. He et al., “Deep residual learning for image recognition,” in Proc. of imaging. the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 770– 778 (2016). Jaeseok Lee received his BS degree in radio communications and 64. M. Griebel, “Sparse grids and related approximation schemes for higher PhD in computer and radio communications from Korea University, dimensional problems,” in Proc. of the Conf. on Foundations of South Korea, in 2008 and 2015, respectively. He is a postdoctoral Computational Mathematics (FoCM 2005), pp. 106–161 (2005). researcher at Daegu Gyeongbuk Institute of Science and Technology, 65. J. Bergstra and Y. Bengio, “Random search for hyper-parameter opti- South Korea. His research interests include sparse signal reconstruc- mization,” J. Mach. Learn. Res. 13, 281–305 (2012). tion theory and cyber-physical system security. 66. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proc. of Int. Conf. on Machine Learning Kyungsoo Kim received his BS degree in information and commu- (ICML), pp. 807–814 (2010). nication engineering from Soong-sil University, South Korea, in 2012. 67. D. Kingma and J. Ba, “Adam: a method for stochastic optimization,” He is a PhD candidate at Daegu Gyeongbuk Institute of Science and arXiv preprint arXiv:1412.6980 (2014). Technology, South Korea. His research interests include brain–com- 68. NVIDIA Corporation, “GeForce GTX 1070,” https://www.nvidia.com/ puter interface and brain plasticity and stroke rehabilitation. en-us/geforce/products/10series/geforce-gtx-1070/ (2017). Ji-Woong Choi received his BS, MS, and PhD degrees in electrical engineering from Seoul National University, South Korea, in 1998, Thanawin Trakoolwilaiwan received his BS degree in biomedical 2000, and 2004, respectively. He is an associate professor at Daegu engineering from Mahidol University, Thailand, in 2015. He is a mas- Gyeongbuk Institute of Science and Technology, South Korea. He is ter’s student at Daegu Gyeongbuk Institute of Science and the author of more than 80 journal papers and patents. His current Technology, South Korea. His research interests include brain–com- research interests include advanced communication systems, bio- puter interface and neural engineering. medical communication and signal processing, invasive, and non- invasive brain–computer interface, and magnetic communication Bahareh Behboodi received her BS degree in biomedical engineer- and energy transfer systems. ing from Amirkabir University of Technology, Iran, in 2013. She is a Neurophotonics 011008-15 Jan–Mar 2018 Vol. 5(1) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Neurophotonics SPIE

Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution

Download PDF
16 pages

Loading next page...
 
/lp/spie/convolutional-neural-network-for-high-accuracy-functional-near-hRMvUDFLk9

References (144)

Publisher
SPIE
ISSN
2329-423X
eISSN
2329-4248
DOI
10.1117/1.NPh.5.1.011008
Publisher site
See Article on Publisher Site

Abstract

Convolutional neural network for high-accuracy functional near- infrared spectroscopy in a brain– computer interface: three-class classification of rest, right-, and left- hand motor execution Thanawin Trakoolwilaiwan Bahareh Behboodi Jaeseok Lee Kyungsoo Kim Ji-Woong Choi Thanawin Trakoolwilaiwan, Bahareh Behboodi, Jaeseok Lee, Kyungsoo Kim, Ji-Woong Choi, “Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain– computer interface: three-class classification of rest, right-, and left-hand motor execution,” Neurophoton. 5(1), 011008 (2017), doi: 10.1117/1.NPh.5.1.011008. Neurophotonics 5(1), 011008 (Jan–Mar 2018) Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution † † Thanawin Trakoolwilaiwan, Bahareh Behboodi, Jaeseok Lee, Kyungsoo Kim, and Ji-Woong Choi* Daegu Gyeongbuk Institute of Science and Technology, Department of Information and Communication Engineering, Daegu, Republic of Korea Abstract. The aim of this work is to develop an effective brain–computer interface (BCI) method based on func- tional near-infrared spectroscopy (fNIRS). In order to improve the performance of the BCI system in terms of accuracy, the ability to discriminate features from input signals and proper classification are desired. Previous studies have mainly extracted features from the signal manually, but proper features need to be selected care- fully. To avoid performance degradation caused by manual feature selection, we applied convolutional neural networks (CNNs) as the automatic feature extractor and classifier for fNIRS-based BCI. In this study, the hemo- dynamic responses evoked by performing rest, right-, and left-hand motor execution tasks were measured on eight healthy subjects to compare performances. Our CNN-based method provided improvements in classifi- cation accuracy over conventional methods employing the most commonly used features of mean, peak, slope, variance, kurtosis, and skewness, classified by support vector machine (SVM) and artificial neural network (ANN). Specifically, up to 6.49% and 3.33% improvement in classification accuracy was achieved by CNN com- pared with SVM and ANN, respectively. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. [DOI: 10.1117/1.NPh.5.1 .011008] Keywords: functional near-infrared spectroscopy; brain–computer interface; support vector machine; artificial neural network; con- volutional neural network; feature extraction. Paper 17080SSR received Apr. 6, 2017; accepted for publication Aug. 17, 2017; published online Sep. 14, 2017. 10,24 16 (MEG), electrocorticography (ECoG), functional magnetic 1 Introduction 25–27 resonance imaging (fMRI), and functional near-infrared 12,23,28–33 1.1 Brain–Computer Interface spectroscopy (fNIRS). BCI systems based on MEG, ECoG, fMRI, and EEG generally suffer from bulkiness, high A brain–computer interface (BCI) is a means of communication cost, high sensitivity to head movements, low spatial and tem- between the human brain and external devices. BCIs are typi- poral resolution, and low signal quality. fNIRS-based systems cally designed to translate the neuronal activity of the brain to are known to be more advantageous, in that they can provide 1–9 restore motor function, or to control devices. The major com- moderate temporal and spatial resolution. ponents of an effective BCI system are: (1) acquisition of brain signals using a neuroimaging modality, (2) signal processing 1.2 Functional Near-Infrared Spectroscopy-Based and analysis to obtain features representative of the signal, Brain–Computer Interface System and (3) translation of features into commands to control devices. Well-designed BCI systems have proven to be helpful In the last few decades, fNIRS has been recognized as a prom- for patients with severe motor impairment, and have improved ising noninvasive optical imaging technique for monitoring the their quality of life. For instance, many studies have been suc- hemodynamic response of the brain using neurovascular cou- cessfully conducted with patients who have suffered a pling. Neurovascular coupling in the cerebral cortex captures 10,11 12,13 stroke, have amyotrophic lateral sclerosis, or have spinal the increases in oxygenated hemoglobin (HbO) and reductions 14,15 cord injury (SCI), in which BCI systems have allowed them in deoxygenated hemoglobin (HbR) that occur during brain to control external devices. activity. To accomplish this, fNIRS employs multiple light 16,17 BCI systems have been developed based on invasive as sources and detectors, which emit and receive near-infrared 4,18 well as noninvasive neuroimaging modalities, including light (at wavelengths between 650 and 950 nm), respectively. 18–23 electroencephalography (EEG), magnetoencephalography The emitted light passes through the scalp, tissue, and skull 34–36 to reach the brain. The relationship between light attenua- tion (caused by absorption and scattering) and changes in the *Address all correspondence to: Ji-Woong Choi, E-mail: jwchoi@dgist.ac.kr concentration of HbO and HbR can be expressed by the These authors contributed equally to this work modified Beer–Lambert law (MBLL), since HbO and HbR Neurophotonics 011008-1 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . have different absorption coefficients in the near-infrared and the proposed CNN structures are described in Sec. 3. 31,34,36 wavelengths. Sections 4–6 cover the results, discussion, and conclusion, Several recent studies have focused on building fNIRS-based respectively. BCI systems. In previous research, various types of experiments have been performed to measure the accuracy of systems’ clas- 31,33 28–31 sification, using mental arithmetic, motor imagery, 2 Background 23,30,33,37 motor execution, and other approaches. This type of This section describes the details of the commonly used fea- research is particularly important since the final goal of the tures, machine learning-based classifiers, and how the classifi- BCI is to have a system which is able to interpret subject inten- cation performance is evaluated for BCI systems. tion, and any misclassification in the BCI system can lead to accidents for the user. Accordingly, improving classification accuracy is the most essential feature of the BCI-based commu- 2.1 Features Extracted from the Input Signal 38,39 nication system. To this end, it is important to exploit appro- priate classifiers as well as discriminant features that can While a large body of previous studies have reported various accurately represent the variability in the hemodynamic response features which can be used to extract the hemodynamic signal, signal. the most commonly used features for fNIRS-based BCI are sig- Many of the studies on fNIRS-based BCI primarily focused nal mean (μ ), variance (σ ), kurtosis (K ), skewness (S ), x x x x i i i;j i on different types of feature extraction techniques and machine peak, and slope where such features are computed as learning algorithms. For feature extraction, methods of iden- tifying statistical properties such as mean, slope, skewness, kur- N tosis, etc., from time-domain signals filter coefficients from EQ-TARGET;temp:intralink-;e001;326;578μ ¼ x ; (1) x i;j 41,42 continuous and discrete wavelet transforms (DWTs), and j¼1 measurement based on joint mutual information have been used. In addition, for machine learning-based classification, N 2 ðx − μ Þ 6,23,29,31,33,38 i;j x j¼1 i methods such as linear discriminant analysis, sup- EQ-TARGET;temp:intralink-;e002;326;528σ ¼ ; (2) 12,30,44 30 port vector machine (SVM), hidden Markov model, and artificial neural network (ANN) have received consider- N 4 able attention. ðx − μ Þ ∕N i;j x j¼1 i EQ-TARGET;temp:intralink-;e003;326;490K ¼ ; (3) Among the above-mentioned techniques for feature extrac- i 4 tion, most of the studies have relied on extracting the statistical values of the time-domain signal. However, reaching the highest and classification accuracy depends on different factors, such as 46 N 3 selecting the best set of combined features and the size of ðx − μ Þ ∕N j¼1 i;j x EQ-TARGET;temp:intralink-;e004;326;436S ¼ ; (4) the time window. In addition, classification accuracies vary i 3 based on different mother wavelet functions for decomposi- tion, which affect performance in a heuristic sense. To over- where N is the total number of samples of x , x is the i’th row of i i come the limitations of these conventional methods, therefore, the input x, x is the j’th signal amplitude of the input x , and i;j i an appropriate technique for feature extraction needs to be σ is the variance of x . The signal peak is computed by select- x i determined. ing the maximum value of x , and the slope is computed using linear regression. 1.3 Objective The results of previous studies have demonstrated that convolu- 2.2 Support Vector Machine tional neural networks (CNNs) can successfully achieve high classification accuracy in many applications, including image SVM is a discriminative classifier which optimizes a separating 47,48 49 recognition, artificial intelligence, speech detection, and hyperplane by maximizing the distance between the training 50,51 multiple time-series processing. Considering CNNs’ ability data. The decision boundary is obtained by to extract important features from a signal, CNN may be suitable for fNIRS-based BCI as well. Accordingly, our proposed 2 ðiÞ method utilizes CNN to automatically extract the features EQ-TARGET;temp:intralink-;e005;326;275 min kw k þ C ε from the hemodynamic response signal. To be specific, we (5) i¼1 attempt to answer the following two arguments: (1) does ðiÞ ðiÞ s:t:y ðx · w þ b Þ − 1 þ ε ≥ 0; s s s CNN outperform conventional methods in fNIRS-based BCI? (2) How well does CNN work with the input data of the hemo- where w is the weight vector, C> 0 is the regularization dynamic response signal? s ðiÞ parameter, ε > 0 is the training error, y is the true class To address these questions, we compared the classification label for i’th input x , and b is the bias. Among these, C accuracies of CNN with those of conventional methods when s s plays an important role in reducing the error as well as accom- used as the feature extractor and classifier in fNIRS-based modating the outliers through the training process of the data. In BCI. Then, we analyzed how the trained convolutional filters other words, it controls the trade-off between the data training in CNN optimized the features. error and the norm of the weights. As a matter of fact, determin- The rest of this work is organized as follows. In Sec. 2, the properties of the conventional methods as well as CNN are ing a proper C is a vital step in training the SVM on the input briefly introduced. Subsequently, data acquisition, preprocessing, data. Neurophotonics 011008-2 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . updated by the backward propagation by comparing the com- puted output values from the forward propagation with the desired output values, using a loss function. This iteration is per- formed until the minimum loss function value is achieved. To obtain a proper predictive model with ANN, several hyperparameters, such as learning rate, batch size, and number of epochs, should be considered. The learning rate is the param- eter that controls how fast the weight values in the fully con- nected layers can be updated during the training process. Through the batch learning process, training data are separated into several sets, and this is followed by propagation through the training process, where the batch size is the number of samples in each set. An epoch is defined as the total number of times that the training procedure is completed. 2.4 Convolutional Neural Network Fig. 1 The common structure of ANN. CNN is an effective classifier based on deep network learning. It is highly capable of automatically learning appropriate features from the input data by optimizing the weight parameters of each 2.3 Artificial Neural Network filter, using forward and backward propagation to minimize ANN is a classifier, inspired by a biological brain’s axon, with classification errors. the ability to detect patterns in the training data set, which con- CNN consists of several layers, which are called the input sists of assemblies of interconnected artificial neurons that pro- layer, convolutional layer, fully connected hidden layer, and out- vide nonlinear decision boundaries. Typically, ANN consists of put layer (see Fig. 2). In the convolutional layers, a convolu- multiple layers, respectively called the input layer, fully con- tional filter whose width is equal to the dimension of the nected hidden layer(s), and the output layer, with one or input and kernel size (height) of h is convolved with the more neurons in each layer (see Fig. 1). Through forward propa- input data, where the output of the i’th filter is gation, the output values are computed based on the activation function of the hidden layer(s) by EQ-TARGET;temp:intralink-;e008;326;447o ¼ w · x½i∶i þ h − 1; (8) ð1Þ ð1Þ EQ-TARGET;temp:intralink-;e006;63;412o ¼ a½w · x þ b ; (6) i where w is the weight matrix, x½i∶j is the submatrix of input from row i to j, and o is the result value. ð2Þ ð2Þ Then, in order to build the feature map (the input of the next EQ-TARGET;temp:intralink-;e007;63;381y ¼ a½w · o þ b ; (7) layer), the output of the convolutional layer is converted by an activation function similar to ANN. After each convolutional where o is the output of the first fully connected hidden layer layer, additional subsampling operations such as max-pooling calculated by using an activation function a to transform the ð1Þ and dropout are performed to enhance the performance. summation of bias value b , and the multiplication of the ð1Þ Max-pooling is one of the common methods used to reduce input vector x with the weight vector w . Likewise, y is data size, and it stores only the important data. Dropout, which the output of the second fully connected hidden layer, which helps CNN avoid overfitting during the training process, is a is similarly calculated by using the input vector o of the second ð2Þ ð2Þ regularization step that randomly drops out one or more hidden layer, and the weight vector w and the bias b . nodes. As with ANN, the mentioned hyperparameters such as Through the first iteration of the training procedure, weight values should be initialized. Proper weight initialization is one learning rate, batch size, and number of epochs should be of the important operations for improving the classification per- investigated for CNN in order to improve the classification formance of the networks. Afterward, the weight values are performance. Fig. 2 The common structure of convolutional neural network. Neurophotonics 011008-3 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . 3.2 Data Acquisition For data acquisition, LABNIRS (Shimadzu), an fNIRS device with a multichannel continuous wave with three wavelengths (780, 805, and 830 nm) and a sampling rate of 25.7 Hz, was utilized. A total of 12 sources and 12 detectors, resulting in 34 measurement channels, were placed over the motor areas, C3 and C4, according to the international 10–20 system which corresponds to the motor cortex of the right- and left- hand motor execution (see Fig. 4). The distance between source and detector was 3 cm. 3.3 Experimental Procedure The subjects sat on a comfortable chair in front of a computer Fig. 3 Cross-validation procedure. screen, which displayed the experimental tasks. In the experi- ment, subjects were asked to perform a motor execution task in order to generate a robust signal for better discrimination. To be specific, while a black screen was displayed during the 2.5 Cross Validation rest task, an arrow pointing right or left was shown during k-fold cross validation is used to estimate the classification per- each of the right- or left-hand execution tasks, respectively. 53,59 formance of the predictive model. The first step in this proc- All of the subjects were asked to relax before the experiment ess is to divide the data into k-folds, where each fold contains an in order to stabilize blood flow. For data acquisition, the subjects identical amount of the input data. Then, one fold is used as a were trained to relax during the rest tasks and to perform finger test set, while the remaining folds are used as training sets (see tapping during the motor execution tasks. Each subject per- Fig. 3). Afterward, a classification procedure is applied to the formed 10 experiments of five sessions of right- and left- selected test and training sets. This process is performed for hand motor executions, with two rest blocks per session (see each of the k-folds, and the corresponding accuracies obtained Fig. 5). All the blocks lasted 10 s, and each block became a from each test set are averaged to estimate the performance. sample. The data for all the subjects were collected within three days. We eventually obtained a total of 100 samples of rest, 50 samples of right, and 50 samples of left-hand motor exe- 3 Method cution for each subject. 3.1 Participants 3.4 Acquired Data Preprocessing Eight healthy subjects were recruited for the experiment (ages of 3.4.1 Calculation of hemoglobin concentration changes 25.25  3.81 years, three females, all right-handed). The sub- jects were asked to avoid smoking and drinking alcohol or cof- After signal measurement, we converted the signals of light fee within 3 h prior to the experiment. None of the subjects had intensity into concentration changes of HbO and HbR by been reported for any neurological or brain injuries. Written MBLL, utilizing statistical toolbox NIRS-SPM. The MBLL consent forms were obtained from all subjects. The experiment equation is given by was approved by the Daegu Gyeongbuk Institute of Science and EQ-TARGET;temp:intralink-;e009;326;315 HbO HbR −1 Δ½HbO ε ε ΔOD Technology (DGIST) Institutional Review Board (DGIST- 1 λ1 λ1 λ1 ¼ ; (9) 170414-HR-004-01). HbO HbR d · DPF Δ½HbR ε ε ΔOD λ2 λ2 λ2 Fig. 4 (a) A subject with optodes over motor area C3 and C4 based on the international 10–20 system and (b) the source and detector configuration. Channel numbers 1 to 17 and 18 to 34 were placed over motor areas C4 and C3, respectively. Neurophotonics 011008-4 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . EQ-TARGET;temp:intralink-;e013;326;752S½n¼ a þ d : (13) j j In order to remove the undesired high- and low-frequency Fig. 5 Experimental procedure includes rest and two motor tasks: noises, we exploited a 10-level wavelet decomposition with a right- and left-hand motor execution. 62 Daubechies (db5) mother function. In addition, we used a bandpass frequency between 0.02 and 0.1 Hz, in which the com- bination of low-frequency components d and d from the 10- 8 9 where Δ½HbO and Δ½HbR are the changes in HbO and HbR level decompositions was solely in the same 0.02- to 0.1-Hz fre- concentration, respectively, d is the distance between the light quency range. Therefore, the filtered signal was reconstructed source and detector, DPF is the differential path length factor, ε based on d and d by S ½n¼ d þ d . After filtering, 8 9 denoised 8 9 is the extinction coefficient at wavelength λ, and ΔOD is the the hemodynamic response signals were normalized into optical density change. range (0,1) by subtracting with the signal mean and scaling. 3.5 Feature Extraction and Classification 3.4.2 Filtering After filtering, we trained and tested the classifiers for each indi- The acquired hemodynamic signal contains various physiologi- vidual subject based on the extracted features. Following the cal noises, including the heart rate at 0.8 Hz, respiration at training step, we computed the classification accuracies from 0.2 Hz, Mayer wave at 0.1 Hz, and very-low-frequency oscil- 36,38,40 both the conventional methods (SVM- and ANN-based lations at 0.03 Hz. Among various possible criteria, we 62 fNIRS) and the proposed method (CNN-based fNIRS). In employed wavelet filtering to remove physiological noise. this section, we discuss the details of the conventional methods The wavelet transform is an efficient method of signal analy- and our proposed CNN structure. sis and performs by adjusting its window width in both time and frequency domains. For denoising a signal S½n, first, wavelet coefficients are obtained by shifting and dilating the waveforms 3.5.1 Conventional methods of the so-called mother function ψ½n, and then important coef- As mentioned, features were extracted after the filtering step, ficients are selected to reconstruct the signal by thresholding. followed by normalizing into range (0,1). The obtained input For a more comprehensive analysis, we also exploited multire- data contained 408 feature dimensions (6 features ×2 signal solution analysis (MRA) which decomposes signals into a tree 41,63 of HbO and HbR ×34 channels). Using such features with structure using the DWT. Using MRA based on DWT, S½n the settings above, we evaluated the performance of the conven- can be approximated by expanding both low- and high-fre- tional methods by observing the concentration changes of HbO quency coefficients for M time points as and HbR over all channels using SVM and ANN. Before applying SVM, since such high-dimensional features 1 64 usually suffer from performance degradation in classifiers, a EQ-TARGET;temp:intralink-;e010;63;409S½n¼ pffiffiffiffiffi A ½j ;kΦ ½n ϕ 0 j ;k principle component analysis (PCA) was utilized to decrease the dimensions of the data. This reduces the aforementioned XX effect by maximizing the variance using a smaller number of þ pffiffiffiffiffi D ½j; kΨ ½n; (10) ψ j;k 52 53,65 principle components. Grid search was used to determine j¼j the number of principle components and the C regularization parameters in SVM, and the combination of both parameters 1 n−k pffiffi where Φ ½n and Ψ ½n¼ ψð Þ are scaling and wavelet j ;k j;k which yielded the highest classification accuracy was selected. 0 j j In this study, we report the results for linear SVM and multi- mother functions, respectively, in which the mother function ple structures of ANN (see Table 1). To be specific, structures of ψ½n is dilated with scaling parameter j, translated by k ANN with one hidden layer (ANN1) and two hidden layers which is the number of decomposition levels, and represented (ANN2) were evaluated. For further comprehensive investiga- as Ψ ½n. Since these functions are orthogonal to each other, j;k tion, each structure of ANN was considered with various taking the inner product results in obtaining the approximation pffiffiffiffi coefficients (low frequency) A ½j ;k¼ S½nΦ ½n and ϕ 0 n j ;k Table 1 Structures of ANN. the detailed coefficients (high frequency) D ½j; k¼ pffiffiffiffi S½nΨ ½n. By denoting n j;k Structure Hidden layer Neurons in each hidden layer ANN1-a 1 128 EQ-TARGET;temp:intralink-;e011;63;218a ¼ pffiffiffiffiffi A ½j ;kΦ ½n; (11) j ϕ 0 j ;k 0 0 ANN1-b 1 256 and ANN1-c 1 512 ANN2-a 2 256, 128 EQ-TARGET;temp:intralink-;e012;63;160d ¼ pffiffiffiffiffi D ½j; kΨ ½n; (12) j ψ j;k k ANN2-b 2 512, 256 ANN2-c 2 512, 128 Eq. (10) can be rewritten as Neurophotonics 011008-5 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 2 Hyperparameters of each individual subject for ANN. Subject Parameters ANN1-a ANN1-b ANN1-c ANN2-a ANN2-b ANN2-c 1 Epochs 50 50 100 20 100 100 Batch size 64 64 16 16 16 32 Learning rate 0.001 0.0005 0.001 0.0005 0.001 0.001 2 Epochs 100 100 100 100 50 50 Batch size 16 16 32 16 16 16 Learning rate 0.0005 0.001 0.0005 0.0001 0.0005 0.001 3 Epochs 100 100 50 50 50 100 Batch size 16 16 64 64 32 32 Learning rate 0.0001 0.0001 0.0005 0.0001 0.0001 0.0001 4 Epochs 50 100 50 50 100 100 Batch size 16 32 64 32 32 64 Learning rate 0.0005 0.0005 0.0005 0.0005 0.0001 0.0005 5 Epochs 100 100 100 100 100 100 Batch size 32 64 64 16 64 64 Learning rate 0.0005 0.0005 0.0005 0.0001 0.001 0.001 6 Epochs 100 50 100 100 100 100 Batch size 16 16 32 16 16 16 Learning rate 0.001 0.001 0.0005 0.0005 0.0005 0.0005 7 Epochs 100 100 100 100 50 100 Batch size 16 32 16 16 16 64 Learning rate 0.0005 0.0005 0.001 0.0005 0.0005 0.001 8 Epochs 100 100 50 50 50 100 Batch size 64 64 32 32 32 16 Learning rate 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 numbers of neurons. All of the aforementioned hyperparameters (CNN1) and three convolutional layers (CNN2). Furthermore, were tuned for each subject (see Table 2). each structure of CNN was considered with a distinct number of filters (see Table 3). All of the convolutional filters in the convolutional layers 3.5.2 Proposed structures of convolutional neural network performed one-dimensional convolution with the input data along the vertical axis, as shown in Fig. 6. Each convolutional Instead of using other methods, we employed CNN as the fea- layer consisted of filters with a kernel size of 3, and an ture extractor as well as the classifier in this study. As the input algorithm was used to update the weight values in the training data, the changes in HbO and HbR concentration over all chan- process. After each convolutional layer, max-pooling with a ker- nels were passed through CNN layers using the structures pre- nel size of 2 was applied, followed by dropout with a dropout sented in Table 3. The input data for CNN were an M by N rate of 50%. The first and second fully connected layers con- matrix, where M is the number of points during 10 s that cor- tained 256 and 128 hidden nodes, respectively. The output respond to the sampling rate (M ¼ time × sampling rate ≈ 257) layer had 3 nodes corresponding to the three classes, which and N is the number of channels for both HbO and HbR (34 were classified using softmax. For better understanding of channels each of HbO and HbR). Similar to the process used the structures mentioned here, the input and output sizes of to evaluate the conventional methods, we considered two struc- each layer in our proposed CNN2-a are summarized in Table 4. tures of CNN, that is, CNN with one convolutional layer Neurophotonics 011008-6 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 3 Structures of CNN. Table 4 Input and output size of the CNN2-a. Input Output Structure Convolutional layer Filters in each convolutional layer Layer size size Properties CNN1-a 1 32 Convolutional layer 1 257, 68 257, 32 32 filters with kernel size 3 CNN1-b 1 64 Max-pooling 1 257, 32 128, 32 Kernel size 2 CNN2-a 3 32, 32, 32 Dropout 1 128, 32 128, 32 Dropout rate 50% CNN2-b 3 64, 64, 64 Convolutional layer 2 128, 32 128, 32 32 filters with kernel size 3 Max-pooling 2 128, 32 64, 32 Kernel size 2 Dropout 2 64, 32 64, 32 Dropout rate 50% Convolutional layer 3 64, 32 64, 32 32 filters with kernel size 3 Max-pooling 3 64, 32 32, 32 Kernel size 2 Dropout 3 32, 32 32, 32 Dropout rate 50% Fully connected layer 1 1024 256 256 hidden nodes Fully connected layer 2 256 128 128 hidden nodes Output layer 128 3 3 hidden nodes utilized in this study because of its automatic feature extraction property. To provide better insights into the feature extraction perfor- mance, a visualization of the features extracted by the aforemen- Fig. 6 The input data consisted of the concentration changes of HbO tioned methods is shown and compared. Because high- (red) and HbR (blue) overall channels. A convolutional filter ran dimensional data are difficult to visualize, the PCA was applied through the input data along the vertical axis. to reduce the dimensionality of the data. In this study, we also compared the visualization of the In the proposed structure, the activation functions of all hemodynamic response signals with the features extracted by layers were set to a rectified linear unit (ReLU), which is a non- conventional methods and convolutional filter, by plotting the linear function, as shown in first two principle components of the PCA. The overall pro- cedure to visualize signal features is shown in Fig. 7. 0;x < 0 EQ-TARGET;temp:intralink-;e014;63;317aðxÞ¼ : (14) x; x ≥ 0 3.7 Computational Time in the Classification In our work, various machine learning algorithms were applied Unlike other activation functions, ReLU avoids a vanishing to classify tasks, including rest, right-, and left-hand motor exe- gradient and in practice converges to the optimum point much cutions. For the ANN and CNN, we trained the model using faster. Consequently, it improves the training process of deep GPU GeForce GTX 1070. The data were divided equally neural network architectures on large scale and complex into 10-folds, and then nine folds were used as a training set. data sets. To imitate the environment of a real application, a single sample In addition, the hyperparameters for training all the CNN was fed through the trained model then the computational time structures, including learning rate, number of epochs, and was measured. For training SVM, ANN, and CNN, the hyper- batch size, were chosen for each individual subject using parameters such as C regularization, number of epochs, and Grid search (see Table 5). Adam was applied as a gradient learning rate were set to 1, 1, and 0.01, respectively. descent optimization algorithm, whose parameters β , β , and 1 2 −8 ϵ were set to 0.9, 0.1, and 10 , respectively. 4 Results 4.1 Measured Hemodynamic Responses 3.6 Visualization of Feature Extraction In the experiment, the changes in HbO and HbR concentration Many previous studies of feature extraction in fNIRS-based BCI were measured as the input data for classification. The average have been reported in the past. Since appropriate features and of the hemodynamic response signals was obtained with respect classifiers are desired in order to achieve high classification to the samples from subjects 1 and 2, across full sessions of each accuracy, the proposed method of the CNN exploitation was task for rest, right-, and left-hand motor executions and are Neurophotonics 011008-7 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 5 Hyperparameters of each individual subject for CNN. cerebral cortex during the right-hand motor execution [see Fig. 9(b)], while it is larger in the right cerebral cortex during the left-hand motor execution [see Fig. 9(c)]. The brain behav- Subject Parameters CNN1-a CNN1-b CNN1-c CNN2-a iors observed in Fig. 9 demonstrate that three-class discrimina- tion, for rest, right-, and left-hand motor execution, can be 1 Epochs 100 100 100 100 achieved, since they show different patterns of cortical activation Batch size 16 64 32 16 over the left and right hemispheres. Learning rate 0.001 0.001 0.001 0.0005 4.2 Classification Accuracies 2 Epochs 50 50 100 100 To determine the classification accuracies of the SVM, ANNs, and CNNs, we employed 10-fold cross validation to estimate Batch size 32 16 16 64 performance and to optimize hyperparameters, as we attempted to discriminate the three classes of rest, right-, and left-hand Learning rate 0.0005 0.001 0.0005 0.001 motor execution. In this section, the classification accuracies 3 Epochs 50 100 50 100 of commonly used features classified by SVM and ANNs are compared with those obtained by CNNs. Batch size 16 64 64 32 To be specific, the classification accuracies for all the tested criteria for the individual subjects are presented in Table 6.As Learning rate 0.0001 0.0005 0.001 0.0005 expected, the results of all the individual subjects indicate that 4 Epochs 100 100 100 100 the use of CNN was significantly superior to SVM and ANN. For convenience of analysis, the average of the classification Batch size 32 32 16 16 accuracies of SVM, ANN, and CNN (86.19%, 89.35%, and 92.68%, respectively) are presented in Fig. 10, which confirms Learning rate 0.0001 0.001 0.0001 0.0005 the superior performance of CNN over the conventional meth- 5 Epochs 50 50 100 100 ods. This superior performance is due to CNN’s ability to learn the inherent patterns of the input data, by updating the weight Batch size 64 16 32 64 values of the convolutional filters. The learning performance can be affected by the size of the Learning rate 0.001 0.0005 0.001 0.001 training set, and this is especially true for ANN and CNN, where 6 Epochs 100 100 50 100 a larger-sized training set usually provides higher classification performance. To examine the effect of the size of the data set on Batch size 16 32 16 32 the classification accuracy, the average classification accuracies across all the subjects were obtained, based on different numbers Learning rate 0.0005 0.0001 0.001 0.001 of samples. 7 Epochs 50 100 100 100 To evaluate the classification performance, 10-fold cross val- idation was utilized. For all the classification methods, the clas- Batch size 64 32 64 16 sification performance was found to increase with the number of samples in the data set, and the classification accuracy of CNN Learning rate 0.001 0.001 0.0005 0.0005 outperformed other tested methods for all numbers of samples 8 Epochs 50 100 100 100 (see Fig. 11). Moreover, the CNN was also able to attain higher accuracy with smaller numbers of samples; for instance, CNN Batch size 64 32 64 16 exceeded 90% accuracy with 120 samples, whereas ANN required 200 samples to reach 89% accuracy. Learning rate 0.001 0.001 0.0005 0.0005 4.3 Analysis of Feature Extraction Performance shown in Figs. 8(a)–8(c), respectively. Each row of the input To better understand the feature extraction performance, we data indicates signal amplitudes. These are represented by visualized the three classes of rest, right-, and left-hand red and blue colors, which imply the maximum and minimum motor executions. To be specific, three classes were visualized amplitudes, respectively. The beginning and the end of the tasks using the hemodynamic response signals, features extracted by correspond to 0 and 10 s, respectively. the conventional methods, and the output of the first layer con- As is widely known, neural activity induces typical changes in volutional filter, by plotting the first and second principle com- cerebral blood oxygenation, resulting in increases in HbO con- ponents of PCA (see Fig. 12). The results for subjects 1 and 2 centration and decreases in HbR concentration. In our results, show that the features extracted by the convolutional filters are a similar behavior in the hemodynamic response can be observed, better discriminated compared with commonly used features as shown in Fig. 8. To be specific, the signals obtained from chan- and the hemodynamic response signals. nels over C3 show higher cortical activation of HbO over a period When considering just the binary classification of rest and of 5 to 10 s during the right-hand motor execution [see Fig. 8(b)], motor execution, both the conventional methods and CNN whereas the signals over C4 have higher activation during the left- resulted in well-separable features. However, for the binary clas- hand motor execution [see Fig. 8(c)]. sification of right- and left-hand motor executions, and for mul- Figure 9 shows the averaged signals for the entire experiment ticlass classification, it was clear that features extracted by the over all channels of the left and right hemispheres. It is obvious convolutional filter were better discriminated as compared with that the change in HbO concentration is higher in the left the conventional methods. Neurophotonics 011008-8 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 7 The overall procedure to visualize signal features, including the hemodynamic response signal, commonly used features in fNIRS-based BCI, and output of the convolutional filter (feature map). The first and second principle components of the signal features are illustrated for the visualization. Fig. 8 Average hemodynamic response of each execution task measured from subject 1 and 2: (a) rest, (b) right-, and (c) left-hand motor execution. Each input presents concentration changes of HbO and HbR overall 34 channels. Red and blue colors represent the maximum and minimum amplitude, respectively. 4.4 Convolutional Filters of Convolutional Neural CNN, we examined the first layer of CNN to determine whether Network it is able to identify the distinguishable channels from the input or not. By training the data using forward and backward One might notice that CNN is able to recognize the patterns of propagations, we let CNN learn how to emphasize some chan- three different classes by updating its filters’ weight values. nels containing distinguishable signals by increasing the corre- Therefore, to further investigate the convolutional filters of sponding weight values, since each column of convolutional Neurophotonics 011008-9 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 9 Average signal amplitude of subjects 1 and 2 across left (C3) and right (C4) hemisphere from full sessions of each class: (a) rest, (b) right-, and (c) left-hand motor execution. Red and blue colors imply HbO and HbR, respectively. Solid and dot lines are related to the C3 and C4 motor areas in that order. Table 6 Classification accuracies of the individual subjects (%). S1 S2 S3 S4 S5 S6 S7 S8 Average SVM 88.50 79.00 84.00 84.50 90.50 97.00 99.00 67.00 86.19 ANN1-a 91.67 85.33 84.83 85.00 94.20 96.17 96.50 76.33 88.75 ANN1-b 92.83 83.83 84.67 87.67 94.30 96.00 96.33 75.00 88.83 ANN1-c 92.83 85.67 85.17 87.50 94.50 96.50 96.67 75.83 89.33 ANN2-a 93.67 86.67 84.33 87.50 95.67 96.50 96.83 75.67 89.61 ANN2-b 92.50 86.17 85.67 88.67 94.83 97.00 97.17 76.08 89.76 ANN2-c 92.17 88.00 85.50 87.00 94.83 97.00 97.67 76.17 89.79 CNN1-a 95.00 91.67 84.83 95.67 97.33 99.00 98.67 80.33 92.81 CNN1-b 95.33 92.17 85.83 96.00 96.83 99.00 99.00 80.50 93.08 CNN2-a 94.33 91.83 82.17 95.00 96.67 98.67 98.33 82.17 92.40 CNN2-b 92.83 93.17 83.33 94.33 96.50 99.00 98.17 82.00 92.42 Neurophotonics 011008-10 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 10 Average classification accuracies of the individual subjects. Fig. 11 Average classification accuracies across all the subjects, based on different number of samples. filter interacts with each channel from the input data. To right-hand motor execution, and filter 2 detects left-hand approximate the most distinguishable channel, each column motor execution. of the convolutional filter was averaged after training. Then, the channel of all of the samples of the input data with the high- 4.5 Computational Time est weight value of the averaged convolutional filter was selected for visualization. The computational time for each of the classification algorithms, In order to visualize the essential information, the most dis- i.e., SVM, ANN, and CNN, was averaged across all subjects and tinguishable channels from all the samples were selected. Two structures (see Table 7). For the training process, the computa- examples of the CNN filter weight values from subject 1 are tional time for CNN was ∼2 and 183 times greater than ANN shown in Fig. 13, where each row represents the most distin- and SVM, respectively. For testing time, the computational time guishable signal from a single sample and the red and blue for CNN was ∼6 and 81 times greater than ANN and SVM, colors indicate the maximum and minimum amplitudes, respec- respectively. The computational time for CNN in the training tively. We found that over a period of 5 to 10 s, there were and testing process was longer than ANN and SVM, as its struc- ture is deeper and more complex. However, it provides a better remarkable differences in the signals chosen from both filters performance in terms of classification accuracy. for the three classes of rest, right-, and left-hand motor execution. Subsequently, in Fig. 13(a) which represents the rest task, 5 Discussion both filters have low signal amplitude. Figure 13(b) represents The primary aim of the present study was to evaluate the use the right-hand motor execution, in which filter 1 shows a of CNN versus conventional methods in fNIRS-based higher signal amplitude than filter 2. In the same manner, BCI, particularly in light of the automatic feature extraction Fig. 13(c) shows the left-hand motor execution, in which property of CNN. The proposed and conventional methods filter 2 exhibits a higher signal amplitude compared with filter were investigated to compare their respective classification 1. Therefore, it can be concluded that filter 1 can detect accuracies. Neurophotonics 011008-11 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Fig. 12 The visualization of the hemodynamic response signals, commonly used features, and output of the convolutional filter from (a) subject 1 and (b) subject 2. Fig. 13 Each filter trained by subject 1 represents signals from a channel in every samples correspond- ing to the highest weight value. The filters represent three classes in the classification: (a) rest, (b) right-, and (c) left-hand motor execution. Neurophotonics 011008-12 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . Table 7 Computational time(s). On the other hand, in the case of vital applications, systems to control assistive technology devices for a patient with motor impairment require very high accuracy, since any misclassifica- Training time Testing time tion would probably lead to a serious accident. Consequently, in such cases the proposed method is recommended even if it takes SVM 0.00645 0.00059 a longer time, because it achieves higher accuracy with a smaller ANN 0.63299 0.00734 number of samples (see Fig. 11). CNN 1.17945 0.04751 6 Conclusions To enhance the classification accuracy of an fNIRS-based BCI system, we applied CNN for automatic feature extraction and classification, and compared those results with results from con- In the experiment, motor execution tasks performed by ventional methods employing SVM and ANN, with features of healthy subjects were utilized to obtain strong and robust hemo- mean, peak, slope, variance, kurtosis, and skewness. From the dynamic response signals. However, in real applications, motor measurement results for rest, right-, and left-hand motor execu- imagery can produce a greater impact than motor execution tion on eight subjects, the CNN-based scheme provided up to tasks, in both healthy users and in patients with severe motor 6.49% higher accuracy over conventional feature extraction impairment. A previous study reported that the cortical activa- and classification methods, because the convolutional filters tion resulting from motor execution is similar to motor imagery. can automatically extract appropriate features. Hence, it is feasible that a healthy user or a patient without a The results confirmed that there was an improvement in brain injury, such as SCI, will be able to use motor imagery accuracy when using CNN over the conventional methods, for commands instead of motor execution. Further investigation which can lead to the practical development of a BCI system. of the use of motor imagery, and the study of patients with neu- Since classification accuracy is the most essential factor for rological disorders, will be explored in the future. many BCI applications, we will explore further improvements in The results of the classification accuracies in Fig. 10 imply the accuracy of fNIRS-based BCI by implementing various deep that the proposed method using CNN outperforms the conven- learning techniques, as well as combining fNIRS with other neu- tional methods. To be specific, the analysis of signal features by roimaging modalities. To investigate clinical applications, we visualizing the first and second principle components demon- will also undertake experiments with patients. strates that the features extracted by the convolutional filter yield better discriminating features than conventional methods, Disclosures because it is capable of learning appropriate features from the training data. The authors declare that there is no conflict of interest regarding Additionally, the channels corresponding to the highest the publication of this paper. weight value in the trained CNN filter demonstrate that the con- volutional filter emphasizes the discriminating signal from the Acknowledgments training data. It is also worthwhile to note that while the perfor- This work was supported in part by the Basic Science Research mance of feature extraction for the binary classification of rest Program through the National Research Foundation of Korea and motor execution was similar for both the conventional and (NRF) funded by the Ministry of Science and ICT (No. proposed methods, since they showed well-discriminated fea- NRF-2015R1A2A2A01008218), the DGIST R&D Program tures, the proposed method performed better for multiclass of the Ministry of Science and ICT (No. 17-BD-0404) and data. This is because the convolutional filter is able to transform the Robot industry fusion core technology development project mixed data into well-separated data. of the Ministry of Trade, Industry & Energy of Korea Consequently, the proposed method will be appropriate for (No. 10052980). various systems that require multitasks to command. For in- stance, a brain-controlled wheelchair requires multiclass classi- fication to control the wheelchair in several directions. Although References the proposed method requires a longer time for training, it per- 1. T. O. Zander and C. Kothe, “Towards passive brain–computer interfa- forms better in multiclass classification. ces: applying brain–computer interface technology to human-machine The number of samples used to train the classifier directly systems in general,” J. Neural Eng. 8(2), 025005 (2011). affects classification accuracy, especially in the complex classi- 2. J. R. Wolpaw et al., “Brain–computer interface technology: a review of fier, as shown in Fig. 11. For a small number of samples, the the first international meeting,” IEEE Trans. Rehabil. Eng. 8(2), 164– 173 (2000). classification performances of SVM, ANN, and CNN were sim- 3. J. R. Wolpaw et al., “Brain–computer interfaces for communication and ilar. However, as the number of samples increased, the complex control,” Clin. Neurophysiol. 113(6), 767–791 (2002). classifier was able to achieve higher accuracy than the simple 4. G. Dornhege, Toward Brain-Computer Interfacing, MIT Press, classifier, though the computational time to train was much Cambridge, Massachusetts (2007). greater than that of the simple classifier. 5. R. P. Rao, Brain-Computer Interfacing: An Introduction, Cambridge This means there is a trade-off between accuracy and ease of University Press, Cambridge, Massachusetts (2013). 6. V. Kaiser et al., “Cortical effects of user training in a motor imagery use when building the appropriate BCI system. The user must based brain–computer interface measured by fNIRS and EEG,” take a longer time to train the complex classifier to obtain a high- NeuroImage 85, 432–444 (2014). performance classifier. When an application requires ease of 7. J. J. Daly and J. R. Wolpaw, “Brain–computer interfaces in neurological use over safety, the conventional methods might be more appro- rehabilitation,” The Lancet Neurol. 7(11), 1032–1043 (2008). priate, since a shorter time and smaller-sized training set are 8. C. Neuper et al., “Motor imagery and EEG-based control of spelling desired. devices and neuroprostheses,” Prog. Brain Res. 159, 393–409 (2006). Neurophotonics 011008-13 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . 9. K. LaFleur et al., “Quadcopter control in three-dimensional space using 35. F. Matthews et al., “Hemodynamics for brain–computer interfaces,” a noninvasive motor imagery-based brain–computer interface,” J. IEEE Signal Process. Mag. 25(1), 87–94 (2008). Neural Eng. 10(4), 046003 (2013). 36. H. Liang, J. D. Bronzino, and D. R. Peterson, Biosignal Processing: 10. E. Buch et al., “Think to move: a neuromagnetic brain–computer Principles and Practices, CRC Press, Boca Raton, Florida interface (BCI) system for chronic stroke,” Stroke 39(3), 910–917 (2012). (2008). 37. X. Cui, S. Bray, and A. L. Reiss, “Functional near infrared spectroscopy 11. K. Saita et al., “Combined therapy using botulinum toxin A and single- (NIRS) signal improvement based on negative correlation between oxy- joint hybrid assistive limb for upper-limb disability due to spastic hemi- genated and deoxygenated hemoglobin dynamics,” NeuroImage 49(4), plegia,” J. Neurol. Sci. 373, 182–187 (2017). 3039–3046 (2010). 12. U. Chaudhary et al., “Brain–computer interface-based communication 38. S. Weyand et al., “Usability and performance-informed selection of per- in the completely locked-in state,” PLoS Biol. 15(1), e1002593 (2017). sonalized mental tasks for an online near-infrared spectroscopy brain– 13. U. Chaudhary, N. Birbaumer, and M. Curado, “Brain-machine interface computer interface,” Neurophotonics 2(2), 025001 (2015). (BMI) in paralysis,” Ann. Phys. Rehabil. Med. 58(1), 9–13 (2015). 39. D. P.-O. Bos, M. Poel, and A. Nijholt, “A study in user-centered design 14. G. Pfurtscheller et al., “EEG-based asynchronous BCI controls func- and evaluation of mental tasks for BCI,” in Int. Conf. on Multimedia Modeling, pp. 122–134, Springer (2011). tional electrical stimulation in a tetraplegic patient,” EURASIP J. Appl. Signal Process. 2005, 3152–3155 (2005). 40. N. Naseer and K.-S. Hong, “fNIRS-based brain–computer interfaces: a review,” Front. Hum. Neurosci. 9, 3 (2015). 15. K. L. Koenraadt et al., “Preserved foot motor cortex in patients with complete spinal cord injury: a functional near-infrared spectroscopic 41. B. Abibullaev and J. An, “Classification of frontal cortex haemodynamic study,” Neurorehabil. Neural Repair 28(2), 179–187 (2014). responses during cognitive tasks using wavelet transforms and machine 16. E. C. Leuthardt et al., “A brain–computer interface using electrocortico- learning algorithms,” Med. Eng. Phys. 34(10), 1394–1410 (2012). graphic signals in humans,” J. Neural Eng. 1(2), 63–71 (2004). 42. T. Q. D. Khoa and M. Nakagawa, “Functional near infrared spectro- scope for cognition brain tasks by wavelets analysis and neural net- 17. T. N. Lal et al., “Methods towards invasive human brain–computer inter- faces,” in Conf. on Neural Information Processing Systems (NIPS), works,” Int. J. Biol. Med. Sci. 1,28–33 (2008). pp. 737–744 (2004). 43. X. Yin et al., “Classification of hemodynamic responses associated with 18. N. Birbaumer et al., “A spelling device for the paralysed,” Nature force and speed imagery for a brain–computer interface,” J. Med. Syst. 39(5), 53 (2015). 398(6725), 297–298 (1999). 19. L. Parra et al., “Linear spatial integration for single-trial detection in 44. X. Cui, S. Bray, and A. L. Reiss, “Speeded near infrared spectroscopy encephalography,” NeuroImage 17(1), 223–230 (2002). (NIRS) response detection,” PLoS One 5(11), e15474 (2010). 45. B. Abibullaev, J. An, and J.-I. Moon, “Neural network classification 20. M. Cheng et al., “Design and implementation of a brain–computer inter- face with high transfer rates,” IEEE Trans. Biomed. Eng. 49(10), 1181– of brain hemodynamic responses from four mental tasks,” Int. J. 1186 (2002). Optomechatronics 5(4), 340–359 (2011). 21. A. Buttfield, P. W. Ferrez, and J. R. Millan, “Towards a robust BCI: error 46. N. Naseer et al., “Analysis of different classification techniques for two- class functional near-infrared spectroscopy-based brain–computer inter- potentials and online learning,” IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 164–168 (2006). face,” Comput. Intell. Neurosci. 2016, 5480760 (2016). 22. B. Blankertz et al., “The non-invasive Berlin brain–computer interface: 47. Y. Bengio et al., “Scaling learning algorithms towards AI,” Large-Scale fast acquisition of effective performance in untrained subjects,” Kernel Mach. 34(5), 1–41 (2007). NeuroImage 37(2), 539–550 (2007). 48. P. Y. Simard et al., “Best practices for convolutional neural networks 23. S. Fazli et al., “Enhanced performance by a hybrid NIRS–EEG brain– applied to visual document analysis,” in 7th Int. Conf. on Document computer interface,” NeuroImage 59(1), 519–529 (2012). Analysis and Recognition (ICDAR), Vol. 3, pp. 958–962 (2003). 49. D. Silver et al., “Mastering the game of go with deep neural networks 24. J. Mellinger et al., “An MEG-based brain–computer interface (BCI),” NeuroImage 36(3), 581–593 (2007). and tree search,” Nature 529(7587), 484–489 (2016). 25. S. M. LaConte, “Decoding fMRI brain states in real-time,” NeuroImage 50. S. Sukittanon et al., “Convolutional networks for speech detection,” in 56(2), 440–454 (2011). Interspeech (2004). 26. N. Weiskopf et al., “Principles of a brain–computer interface (BCI) 51. Y. Bengio et al., “Learning deep architectures for AI,” Found. Trends based on real-time functional magnetic resonance imaging (fMRI),” Mach. Learn. 2(1), 1–127 (2009). IEEE Trans. Biomed. Eng. 51(6), 966–970 (2004). 52. J. L. Semmlow and B. Griffel, Biosignal and Medical Image Processing, CRC Press, Boca Raton, Florida (2014). 27. B. Sorger et al., “Another kind of BOLD response: answering multiple- choice questions via online decoded single-trial brain signals,” Prog. 53. C.-W. Hsu et al., “A practical guide to support vector classification,” Brain Res. 177, 275–292 (2009). Technical Report, pp. 1–16, Department of Computer Science, 28. S. M. Coyle, T. E. Ward, and C. M. Markham, “Brain–computer inter- National Taiwan University, Taipei (2003). face using a simplified functional near-infrared spectroscopy system,” 54. M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical J. Neural Eng. 4(3), 219–226 (2007). Foundations, Cambridge University Press, Cambridge, Massachusetts 29. N. Naseer and K.-S. Hong, “Classification of functional near-infrared (2009). spectroscopy signals corresponding to the right-and left-wrist motor 55. Y. A. LeCun et al., “Efficient backprop,” in Neural Networks: Tricks of imagery for development of a brain–computer interface,” Neurosci. the Trade, G. Montavon, G. B. Orr, and K.-R. Müller, Eds., pp. 9–48, Lett. 553,84–89 (2013). Springer, Heidelberg (2012). 30. R. Sitaram et al., “Temporal classification of multichannel near-infrared 56. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford spectroscopy signals of motor imagery for developing a brain–computer University Press, Oxford, Mississippi (1995). interface,” NeuroImage 34(4), 1416–1427 (2007). 57. Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners’ 31. K.-S. Hong, N. Naseer, and Y.-H. Kim, “Classification of prefrontal and guide to) convolutional neural networks for sentence classification,” motor cortex signals for three-class fNIRS-BCI,” Neurosci. Lett. 587, CoRR abs/1510.03820 (2015), http://arxiv.org/abs/1510.03820. 87–92 (2015). 58. N. Srivastava et al., “Dropout: a simple way to prevent neural networks 32. J. Shin and J. Jeong, “Multiclass classification of hemodynamic from overfitting,” J. Mach. Learn. Res. 15(1), 1929–1958 (2014). responses for performance improvement of functional near-infrared 59. S. Arlot et al., “A survey of cross-validation procedures for model selec- spectroscopy-based brain–computer interface,” J. Biomed. Opt. 19(6), tion,” Stat. Surv. 4,40–79 (2010). 067009 (2014). 60. R. W. Homan, J. Herman, and P. Purdy, “Cerebral location of 33. M. J. Khan, M. J. Hong, and K.-S. Hong, “Decoding of four movement international 10–20 system electrode placement,” Electroencephalogr. directions using hybrid NIRS-EEG brain–computer interface,” Front. Clin. Neurophysiol. 66(4), 376–382 (1987). Hum. Neurosci. 8, 244 (2014). 61. J. C. Ye et al., “NIRS-SPM: statistical parametric mapping for near- 34. F. Scholkmann et al., “A review on continuous wave functional near- infrared spectroscopy,” NeuroImage 44(2), 428–447 (2009). infrared spectroscopy and imaging instrumentation and methodology,” 62. H. Tsunashima, K. Yanagisawa, and M. Iwadate, “Measurement NeuroImage 85,6–27 (2014). of brain function using near-infrared spectroscopy (NIRS),” in Neurophotonics 011008-14 Jan–Mar 2018 Vol. 5(1) Trakoolwilaiwan et al.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy. . . master’s student at Daegu Gyeongbuk Institute of Science and Neuroimageing-Methods, B. Peter, Ed., INTECH Open Access Technology, South Korea. Her research interests include functional Publisher, Rijeka, Croatia (2012). near-infrared spectroscopy and functional magnetic resonance 63. K. He et al., “Deep residual learning for image recognition,” in Proc. of imaging. the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 770– 778 (2016). Jaeseok Lee received his BS degree in radio communications and 64. M. Griebel, “Sparse grids and related approximation schemes for higher PhD in computer and radio communications from Korea University, dimensional problems,” in Proc. of the Conf. on Foundations of South Korea, in 2008 and 2015, respectively. He is a postdoctoral Computational Mathematics (FoCM 2005), pp. 106–161 (2005). researcher at Daegu Gyeongbuk Institute of Science and Technology, 65. J. Bergstra and Y. Bengio, “Random search for hyper-parameter opti- South Korea. His research interests include sparse signal reconstruc- mization,” J. Mach. Learn. Res. 13, 281–305 (2012). tion theory and cyber-physical system security. 66. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proc. of Int. Conf. on Machine Learning Kyungsoo Kim received his BS degree in information and commu- (ICML), pp. 807–814 (2010). nication engineering from Soong-sil University, South Korea, in 2012. 67. D. Kingma and J. Ba, “Adam: a method for stochastic optimization,” He is a PhD candidate at Daegu Gyeongbuk Institute of Science and arXiv preprint arXiv:1412.6980 (2014). Technology, South Korea. His research interests include brain–com- 68. NVIDIA Corporation, “GeForce GTX 1070,” https://www.nvidia.com/ puter interface and brain plasticity and stroke rehabilitation. en-us/geforce/products/10series/geforce-gtx-1070/ (2017). Ji-Woong Choi received his BS, MS, and PhD degrees in electrical engineering from Seoul National University, South Korea, in 1998, Thanawin Trakoolwilaiwan received his BS degree in biomedical 2000, and 2004, respectively. He is an associate professor at Daegu engineering from Mahidol University, Thailand, in 2015. He is a mas- Gyeongbuk Institute of Science and Technology, South Korea. He is ter’s student at Daegu Gyeongbuk Institute of Science and the author of more than 80 journal papers and patents. His current Technology, South Korea. His research interests include brain–com- research interests include advanced communication systems, bio- puter interface and neural engineering. medical communication and signal processing, invasive, and non- invasive brain–computer interface, and magnetic communication Bahareh Behboodi received her BS degree in biomedical engineer- and energy transfer systems. ing from Amirkabir University of Technology, Iran, in 2013. She is a Neurophotonics 011008-15 Jan–Mar 2018 Vol. 5(1)

Journal

Neurophotonics – SPIE

Published: Jan 1, 2018

There are no references for this article.