Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies

Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies entropy Article Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies Aditya Nalwaya *, Kritiprasanna Das and Ram Bilas Pachori Department of Electrical Engineering, Indian Institute of Technology Indore, Indore 453552, India * Correspondence: adityanalwaya@iiti.ac.in Abstract: Human dependence on computers is increasing day by day; thus, human interaction with computers must be more dynamic and contextual rather than static or generalized. The development of such devices requires knowledge of the emotional state of the user interacting with it; for this purpose, an emotion recognition system is required. Physiological signals, specifically, electrocar- diogram (ECG) and electroencephalogram (EEG), were studied here for the purpose of emotion recognition. This paper proposes novel entropy-based features in the Fourier–Bessel domain instead of the Fourier domain, where frequency resolution is twice that of the latter. Further, to represent such non-stationary signals, the Fourier–Bessel series expansion (FBSE) is used, which has non-stationary basis functions, making it more suitable than the Fourier representation. EEG and ECG signals are decomposed into narrow-band modes using FBSE-based empirical wavelet transform (FBSE-EWT). The proposed entropies of each mode are computed to form the feature vector, which are further used to develop machine learning models. The proposed emotion detection algorithm is evaluated using publicly available DREAMER dataset. K-nearest neighbors (KNN) classifier provides accuracies of 97.84%, 97.91%, and 97.86% for arousal, valence, and dominance classes, respectively. Finally, this paper concludes that the obtained entropy features are suitable for emotion recognition from given physiological signals. Keywords: Fourier–Bessel series expansion; spectral entropy; ECG; EEG; FBSE-EWT Citation: Nalwaya, A.; Das, K.; Pachori, R.B. Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies. Entropy 2022, 24, 1322. https://doi.org/ 1. Introduction 10.3390/e24101322 Nowadays, humans are becoming more and more dependent on computers for their various day-to-day tasks. Thus, the need for making computers more user-friendly and Academic Editor: Daniel Abasolo engaging is becoming more and more common rather than just being logically or com- Received: 7 August 2022 putationally efficient [1]. To make computers more user-friendly, it must recognize the Accepted: 16 September 2022 emotion of the user interacting with it, as emotion is the most basic component that is Published: 20 September 2022 responsible for human attention, action, or behavior in a particular situation [2]; therefore, Publisher’s Note: MDPI stays neutral for the applications related to human–computer interaction (HCI), an emotion recognition with regard to jurisdictional claims in system is very useful [3]. Further, such HCI system can be very useful in areas related to published maps and institutional affil- mental health care [4], smart entertainment [5], and an assistive system for a physically iations. disabled person [6]. Emotion recognition systems generally have two different approaches for recognizing emotion, i.e., explicit approach, which includes expression on the face, speech, etc., that are visible and can be easily captured for analysis; however, the problem with such an approach Copyright: © 2022 by the authors. is that the signals obtained in this approach are not very reliable because the subject under Licensee MDPI, Basel, Switzerland. consideration can hide his/her emotion, so these might not show the actual emotional This article is an open access article state of the subject. An alternative way of recognizing emotion is an implicit approach distributed under the terms and where physiological signals, such as electroencephalogram (EEG), electrocardiogram (ECG), conditions of the Creative Commons galvanic skin response (GSR), etc., are being captured and analyzed. Such signals are not Attribution (CC BY) license (https:// visible and are generated inside the body by the autonomous nervous system (ANS). creativecommons.org/licenses/by/ Any change in the emotional state of a person is reflected in the respiration rates, body 4.0/). Entropy 2022, 24, 1322. https://doi.org/10.3390/e24101322 https://www.mdpi.com/journal/entropy Entropy 2022, 24, 1322 2 of 22 temperature, heart rate, and other physiological signals, which are not under the control of the subject [7]. The brain is a central command center. It reacts to any external stimuli by firing neurons inside the brain, causing changes in ANS activity, due to which, the routine activity of the heart and other peripheral organs varies. In addition, change in the emotional state affects heart activity [8]. EEG and ECG signals are the biopotential that reflect the activities of the brain and heart, respectively. Thus, in this study, to recognize the emotion of the subject, physiological signals namely, EEG and ECG are considered. Although in the literature, some other physiological signals are also used to determine the emotional state of a person, such as changes in the conductivity of the skin, which is measured using GSR [9–11]. In [12], the authors have extracted time–frequency domain features using fractional Fourier transform (FrFT); the most relevant features are selected using the Wilcoxon method, which is then given as input to a support vector machine (SVM) classifier for determining the emotional state of the person. Similarly, the respiration rate also helps in determining the emotional state of the person. In [13], authors have used deep learning (DL)-based feature extraction method and logistic regression, determining different emotional states from the obtained features. In recent years, several emotion recognition frameworks have been established using different physiological signals. The authors in [14] have proposed a methodology that uses ECG signal features derived from different intrinsic mode functions (IMFs) for emotion classification. Using bivariate empirical mode decomposition (BEMD), IMFs are obtained. The instantaneous frequency and local oscillation feature are computed, which are given as input to the linear discriminant classifier to discriminate different emotional states. A method is proposed to determine negative emotion through a single channel ECG signal [15]. Different features, extracted from the ECG signals, are linear features, i.e., mean of RR interval. Nonlinear features consist of bispectral analysis, power content in different frequency bands, time–domain features contains different statistical parameters of ECG signal, and time–frequency domain features were obtained using wavelet-based signals decomposition. Based on these features, different emotions were detected using several machine learning classifiers. The authors of [16] have extracted four different features from the ECG signal, i.e., heart rate variability (HRV), with-in beat (WIB), frequency spectrum, and signal decomposition-based features. To evaluate the performance of the obtained features, ensemble classifiers are used. In [17], the authors have extracted ECG signal features at different time scale using wavelet scattering, and features obtained are given as an input to various classifiers for evaluating their performance. Similarly, different emotion recognition techniques have been proposed based on EEG signals. Anuragi et al. [18] have proposed EEG based emotion detection framework. Using the Fourier–Bessel series expansion (FBSE)-based empirical wavelet transform (EWT) (FBSE-EWT) method, the EEG signals are decomposed into four sub-band signals from which features such as energy and entropy are extracted. The features are selected using neighborhood component analysis (NCA), using different classifiers, such as k-nearest neighborhood (KNN), ensemble bagged tree, and artificial neural network (ANN), emotion class is identified. Sharma et al. [19] have used discrete wavelet transform (DWT) for EEG signal decomposition. Third-order cumulant nonlinear features are extracted from each sub-band, and using swarm optimization features, dimensionality is reduced. To classify the emotional states, a long short-term memory-based technique is used. Bajaj et al. [20] have used multiwavelet transform to decompose EEG signals into narrow sub-signals. Using Euclidean distance, features are extracted that are computed from a 3-D phase space diagram. Multiclass least squares support vector machines (MC-LS-SVM) classifier is used for identifying class of emotion; however, the authors in [21] have computed features, namely entropy and ratio of the norms-based measure, are computed from the sub- signals that are obtained after decomposing EEG signal using multiwavelet decomposition. The feature matrix is given as input to MC-LS-SVM for determining the emotional state. In [22], features related to changes in the spectral power of EEG signal are extracted Entropy 2022, 24, 1322 3 of 22 using a short-time Fourier transform (STFT). The best 55 features among 60 features are selected using F-score. A feature vector is formed from the selected features which are given as an input to the SVM classifier for determining the emotional state of the subject. Liu et al. [23] have extracted fractal dimension and higher-order crossing (HOC) as features via the sliding window approach from the given EEG signal; based on these features, an emotion class is determined using an SVM classifier. In [24], the authors have decomposed EEG signals in different sub-bands using DWT. From the obtained wavelet coefficient features, entropy and energy of the coefficients are calculated. Finally, using SVM and KNN classifiers, emotional states are obtained.The authors in [25] have used a multi-channel signal processing technique called multivariate synchrosqueezing transform (MSST) for representing EEG signals in time–frequency domain. Independent component analysis (ICA) is used to reduce dimensionality of the obtained high-dimensional feature matrix. In addition, its performance is compared with non-negative matrix factorization (NMF), which is an alternative feature selection method. The reduced feature matrix is passed to different classifiers such as SVM, KNN, and ANN to discriminate different emotions. Gupta et al. used flexible analytic wavelet transform for decomposing EEG signals into narrow-band sub-bands [26]. From each sub-band, information potential (IP) is computed using Reyni’s quadratic entropy, and using moving average filter the extracted features are smoothed. To determine the different emotion classes, a random forest classifier is used. Ullah et al. segmented the large duration EEG signal into smaller epoch [27] from each segment features; specifically, Fisher information ratio, entropy, statistical parameters, and Petrosian fractal dimension, are computed. The above feature vector is passed to a sparse discriminative ensemble learning (SDEL) classifier to identify the emotion class. Bhattacharyya et al. decomposed EEG signals into different modes using FBSE-EWT [28]. From each mode of KNN entropy, multiscale multivariate Hilbert marginal spectrum (MHMS), and spectral Shannon entropy, features are calculated. Features are smoothed and fed to a classifier called sparse autoencoder-based random forest (ARF) for determining the class of human emotion. Nilima et al. used the empirical mode decomposition (EMD) method for decomposing EEG signal [29]. The second-order difference plot (SODP) features are calculated from each IMF. A two-hidden layer multilayer perceptron is used for multi class classification and SVM is used for binary classification. Nalwaya et al. [30], have used tunable Q-factor wavelet transform (TQWT) to separate various rhythms of EEG. Different statistical and information potential features are computed for each rhythm, which are then fed to SVM cubic classifier for emotion identification. Further, some of the studies use both ECG and EEG signals. In [31], the authors have recorded EEG and ECG signals while the subject is exposed to immersive virtual environments to elicit emotion. From ECG signal, time–domain features, frequency domain features, and non-linear features are calculated. Whereas from EEG signal band power and mean phase connectivity features are calculated. Both EEG and ECG features are combined to form a single feature matrix whose dimensionality is then reduced using principal component analysis (PCA). Then, the reduced matrix is passed to an SVM classifier to determine the emotional state of the subject. The literature review of previous studies on human emotion identification shows emerging research trends in finding appropriate signal decomposition techniques, dis- tinctive features, and machine learning algorithms to classify emotional states. Most of the studies carried out previously used a single modality, i.e., either EEG, ECG, or other peripheral physiological signals, and very few studies have been conducted using multiple modalities. Despite this, there is still scope for improvement in the classification accuracy of the earlier proposed methods. In this article, an automated emotion identification framework using EEG and ECG signals is developed. Furthermore, the Fourier–Bessel (FB) domain has been explored instead of working in traditional Fourier domain, due to various advantages associated with latter one. FBSE uses Bessel functions as basis functions whose amplitude decay with time and are damped in nature, which make them more suitable for non-stationary signal Entropy 2022, 24, 1322 4 of 22 analysis. FBSE spectrum has twice the resolution as compared to Fourier domain spectral representation. Thus, looking at such advantages, FBSE-EWT is used instead of EWT for extracting different modes from the given signal. New FB-based spectral entropy, such as Shannon spectral entropy (SSE), log energy entropy (LEE), and Wiener entropy (WE), have been proposed, which are used as features from the obtained modes. Then, smoothing of the feature values is performed by applying moving average over obtained features values, which is then given as input to SVM and KNN classifiers for emotional class identification. The block diagram of the proposed methodology is shown in Figure 1. EEG/ ECG FBSE-EWT based signal decomposition Feature extraction Feature smoothening Classifier Emotional state Figure 1. Block diagram for the proposed emotion detection methodology. The rest of the article is organized as follows: In Section 2, the material and method- ology are discussed in detail. In Section 3, results obtained after applying the proposed methodology are presented. Section 4 presents the performance of the proposed method with the help of the results obtained, compares it with other exiting methodologies, and highlights some of its limitations. Finally, Section 5 concludes the article. 2. Materials and Methods The emotion recognition framework presented in this article consists of preprocessing, signal processing, feature extraction, feature smoothing, and classification steps. 2.1. Dataset Description A publicly available DREAMER dataset is used to evaluate the proposed methodology. It contains raw EEG and ECG signals, which are recorded from 23 healthy participants while the subject is watching audio and video clips. Emotions are quantified in terms of three different scales: arousal, valence, and dominance [32]. Each participant was shown 18 different clips of different durations with a mean of 199 s. The EPOC system by Emotive was used, which contains 16 gold-plated dry electrodes placed in accordance with the international 10–20 system EEG, and were recorded from, i.e., AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, P7, P8, O1, and O2; two reference electrodes, M1 and M2 were placed over mastoid, as described in [32]. To obtain the ECG signals, electrodes were placed in two vector directions, i.e., right arm, left leg (RA-LL) vector and left arm, left leg (LA-LL) vector. ECG was recorded using the SHIMMER ECG sensor. Both EEG and ECG signals were recorded at different sampling rates, i.e., 128 Hz, and 256 Hz, respectively. The sample EEG and ECG signals obtained from the dataset are shown in Figure 2 for different classes of emotion (high arousal (HA) or low arousal (LA), high dominance (HD) or low dominance (LD), high valence (HV) or low valence (LV)). The dataset also contains information relating Entropy 2022, 24, 1322 5 of 22 to the rating of each video on a scale of 1 to 5. Each participant rated the video as per his/her level of emotion elicited in three different dimensions, i.e., valence (pleasantness level), arousal (excitement level), and dominance (level of control). Rating between 1–5 was labeled as ‘0’ (low) or ‘1’ (high) for each dimension, with 3 as the threshold, i.e., if a participant rates a video between 1 to 3, it will be considered as low or ‘0’ and a value above 3 is considered as high or ‘1’ [33,34]. EEG signal of LA class EEG signal of HA class 3800 3838 EEG signal of LD class EEG signal of HD class 4480 4500 EEG signal of LV class EEG signal of HV class 20 40 60 80 100 120 20 40 60 80 100 120 ECG signal of LA class ECG signal of HA class ECG signal of of LD class ECG signal of of HD class ECG signal of of LV class ECG signal of HV class 50 100 150 200 250 50 100 150 200 250 Sample number Sample number Figure 2. Epochs of raw EEG and ECG signals obtained from the DREAMER dataset. 2.2. Preprocessing At this stage, typically, noise and artifacts from the raw signal are removed with the help of filters and denoising algorithms. As in [32], the analysis of only the last 60 s of signal was performed. Further, signals were segmented into small epochs of 1 s, which were then used for further processing. The mean value was subtracted from each epoch. 2.3. FBSE-EWT Analyzing non-stationary signals such as EEG and ECG is difficult as the signal is time-varying in nature and its properties also change continuously. In order to understand the properties of such signals, decomposing them into narrow-band simpler components can help to make it easier to understand; therefore, the preprocessed signal was given to a signal decomposition algorithm, which will decompose the input EEG and ECG signal into various modes. FBSE-EWT is an improved version of the EWT, which has adaptive basis functions derived from the signals. FBSE-EWT has been used for many biomedical signal processing applications, such as epileptic seizure detection [35,36], valvular heart disease diagno- sis [37], and posterior myocardial infarction detection [38]. FBSE-EWT technique flow is shown in Figure 3 and its step by step working is as follows: Amplitude Entropy 2022, 24, 1322 6 of 22 Mode 1 Mode 2 FBSE Scale-space Boundary EWT based Input signal spectrum representation detection filter-bank Mode M Figure 3. Flow diagram of FBSE-EWT based signal decomposition. 1. FBSE spectrum: FBSE has twice the frequency resolution as compared to the Fourier domain. FBSE spectrum of a signal s(n) of length S samples can be obtained us- ing zero-order Bessel functions. The magnitude of the FB coefficients K(i) can be computed mathematically as follows [39–42]: S1 2 b n K(i) = ns(n) J (1) å 0 2 2 S [ J (b )] S 1 i n=0 where J () is the zero-order and J () is the first-order Bessel functions. Positive roots 0 1 of zeroth order Bessel function are denoted by b , which are arranged in ascending order, where i = 1, 2, 3, , S. A one to one relation between order i and continuous frequency is given by [39,42], 2p f S b  (2) where b  b + p  ip, f denotes the sampling frequency. Here, f is the i i1 i th continuous frequency, corresponding to the i order and is expressed as [39], 2 f S i  (3) For covering whole bandwidth of s(n), i must vary from 1 to S. The FBSE spectrum is the plot of magnitude of the FB coefficient K(i) versus frequency f . 2. Scale-space based boundary detection [39,42,43]: For FBSE spectrum, scale-space representation can be obtained by convolving the signal with a kernel of Gaussian type, which is expressed as follows: J(i, q) = K(i n)g(n; q) (4) n=N 2q where g(n; q) = e . Here, N = W q + 1 with 3  W  6 and q is the scale 2pq parameter. As the scale-space parameter, i.e., # = , # = 1, 2, , # , increases, max the number of minima decreases and no new minima will appear in the scale space plan. The FBSE spectrum is segmented using the boundary detection technique. The FBSE spectrum ranges from 0 to p and the FBSE spectrum segments are denoted as [0, w ], [w , w ], , and [w , p], where w , w , , w are boundaries. Typically 1 1 2 i1 1 2 i1 boundaries are defined between two local minima, which are obtained by the two curves in the scale–space plane, whose length is greater than the threshold, which is obtained by using Otsu method [44]. 3. EWT filter bank design: After obtaining the filter boundaries, based on these param- eters, empirical scaling and wavelet functions are adjusted and different band-pass filters are designed. Wavelets scaling (F (w)) and wavelet functions (n (w)) were j j constructed, and the mathematical expressions are given by [45], 1 if jwj  (1 t)w > j h i pg(t,w ) F (w) = cos if (1 t)w  jwj  (1 + t)w (5) j j j 0 otherwise Entropy 2022, 24, 1322 7 of 22 1 if (1 + t)w  jwj  (1 t)w > j j+1 h i pg(t,w ) j+1 cos if (1 t)w  jwj  (1 + t)w j+1 j+1 h i n (w) = (6) pg(t,w ) sin if (1 t)w  jwj  (1 + t)w > j j 0 otherwise h i jwj(1t)w where, g(t, w ) = d . To obtain the tight frames parameter, d and t are 2tw defined as >0 if x  0 d(x) = (7) and d(x) + d(1 x) = 1 8 x 2 [0, 1] 1 if x  1 " # w w j+1 j t < min (8) w + w j+1 j 4. Filtering: Expression for detailed coefficients from the EWT filter bank for the analyzed signal a(m) is given by [45], D (j, n) = a(m)n (m n)dm (9) a,n The approximate coefficients from the EWT filter bank are computed by [45], A (0, n) = a(m)F (m n)dm (10) a,F th The j empirical mode is obtained by convolving the wavelet function with the detail coefficients, where j = 1, 2, , M are different empirical modes. Original signal can be reconstructed by adding all M reconstructed modes and one low-frequency component; mathematically, both are expressed as below [45] r (n) = D (j, n) ? n (n) (11) j a,n j r (n) = A (0, n) ? F (n) (12) 0 a,F 1 where ? denotes the convolution operation. 2.4. Feature Extraction In order to understand and quantify information associated with any dynamically changing phenomenon, entropy is the most widely used parameter [46]. Thus, in order to understand the nonstationary behavior of physiological signals considered in this study, entropies, as a feature set, are considered. Some of the advantages of FBSE representation are that Bessel functions decay with time, and so provide more suitable representation of the nonstationary signal, and that FBSE has a frequency resolution twice that of the Fourier spectrum. In addition, there are many previous studies on emotion recognition where entropies have been used [26,28,42,47]. SSE, WE, and LEE are defined as follows: 2.4.1. SSE Uniformity in the distribution of signal energy can be measured using spectral entropy. Entropy measures the uncertainty, which has been derived from the Shannon’s expression. The SSE is defined based on the energy spectrum E obtained using FBSE. The energy th corresponding to the i order FB coefficient K(i) is defined mathematically as [48] 2 2 2 K(i) S [ J (b )] 1 i E = (13) 2 Entropy 2022, 24, 1322 8 of 22 The SSE is expressed mathematically as [49] H = P(i) log (P(i)) (14) SSE å i=1 where S is the total number of FBSE coefficients. P(i) is the normalized energy distribution th over i order is mathematically defined as, P(i) = (15) å i i=1 2.4.2. WE It is another measure of flatness in the distribution of signal spectral power. The WE is also called the measure of spectral flatness. It is calculated by taking the ratio of geometric mean to arithmetic mean of the energy spectrum (E ) of the FB coefficient for order i and is expressed as [50], Õ i i=1 H = S (16) WE å i i=1 WE is a unitless quantity; its output is purely a numerical value ranging from 0 to 1, where 1 represents a uniform (flat) spectrum and 0 indicates a pure tone. 2.4.3. LEE Another variant of information measurement using LEE is defined as logarithm of P(i) in the FB domain and it is given by [51] H = log (P(i)) (17) LE i=1 2.5. Feature Smoothing The brain is the control center of all human activities and internal functioning. The typical EEG signal obtained is thus a combination of various brain activity and other noise, either related to the environment or body [52]. Rapid fluctuations in feature value may arise from these noises. Human emotion changes gradually [26]; in order to reduce the effect of such variation on the emotional state-related feature values, a moving average filter with a window size of 3 samples is utilized. The effect of the moving average filter on the raw feature value of the first channel’s first epoch can be seen in Figure 4. 2.6. Classifiers Feature vectors extracted from the EEG signals are used for the classification of dif- ferent emotions. Based on the feature vector, the classifier will discriminate the data into high and low dimensions of emotion, i.e., either HV or LV, HA or LA, and HD or LD. In this study, SVM with the cubic kernel and KNN are used independently for the purpose of classification. SVM is a supervised machine learning algorithm that classifies data by first learning from labeled data belonging to different classes. The main principle behind the classifier working is finding decision boundaries that are formed by a hyperplane, and it helps in separating data into two separate classes. The hyperplane is constructed by training the classifier with sample data. The optimum location of the hyperplane or decision boundary depends on the support vectors, which are the points nearest to the decision boundary [53]. Entropy 2022, 24, 1322 9 of 22 SVM iteratively optimizes the location of hyperplane in order to maximize the margin. A margin is a total separation between two classes. SVM can be linear or nonlinear classifier depending on kernels used. Generally, in the case of a linear SVM straight line, flat plane, or an N-dimensional hyperplane are the simplest way of separating data into two groups, but there are certain situations where nonlinear boundary separates the data more efficiently. In case of a nonlinear classifier, different kernels can be polynomial, hyperbolic tangent function, or Gaussian radial basis function. In this work, an SVM classifier with a cubic kernel is used. Generalized mathematical expression for defining the hyperplane of the SVM classifier can be expressed as follows [54]: " # f (x) = sign b f K(x, x ) + c (18) å i i i i=1 where b is a positive real constant, R is total number of observations, c is a real con- stant, K(x, x ) is a kernel or feature space, x input vector, and output vector is denoted i i by f . For a linear feature space K(x, x ) = x x, for polynomial SVM of any order p, i i T p K(x, x ) = (x x + 1) defines the feature space, i.e., ( p = 2) for quadratic polynomial and ( p = 3) for cubic polynomial. SSE feature Smoothed SSE feature 3.5 2.5 10 20 30 40 50 60 10 20 30 40 50 60 WE feature Smoothed WE feature 0.02 0.03 0.015 0.02 0.01 0.01 0.005 10 20 30 40 50 60 10 20 30 40 50 60 LEE feature Smoothed LEE feature 2600 2100 10 20 30 40 50 60 10 20 30 40 50 60 Epochs Epochs Figure 4. Feature values obtained from signal epochs is shown on the left and its smoothed version is shown on the right. The KNN is a non-parametric, supervised machine learning algorithm used for data classification and regression [55]. It categorizes data points into different groups based on the distance from some of the nearest neighbors. For classifying any particular training data, the KNN algorithm follows these steps: Step 1: Compute distance between sample data and other sample using anyone of the distance metrics such as Euclidean, Mahalanobis, or Minkowski distance. Step 2: Rearrange the distant metric obtained from the first step in ascending order and top k values are considered with distance from current sample is minimum. Feature values Entropy 2022, 24, 1322 10 of 22 Step 3: Class is assigned to the sample data depending on the maximum number of nearest neighbors class. 3. Results In order to evaluate the performance of our proposed methodology, a publicly avail- able DREAMER dataset is used [32]. In this section, the results obtained after applying the proposed emotion identification method over the DREAMER dataset is discussed in detail. The raw EEG data are segmented into an epoch of one second. An EEG epoch length is 128 samples and the ECG length is 256 samples. These epochs of the EEG signal are de- composed into four modes using FBSE-EWT, as shown in Figure 5, and the corresponding filter bank is shown in Figure 6. The decomposition of EEG and ECG signal epochs gives a minimum of four number of modes. So, we set the required number of modes to four to obtain a uniform number of features across different observations. From each mode, three features are extracted. The EEG data consist of 14 channels; therefore, the dimension of the feature vector is 168 (= 14 3 4). Similarly, from the two channels of the ECG, we obtain a 24-element feature vector. The number of trials for each participant is 18 and there are a total of 23 participants, which gives us a total of 24,840 (= 60 18 23) observations or epochs. The final dimensions of the feature matrices are 24, 840 168 and 24, 840 24 for EEG and ECG, respectively. For the multimodal case, both the feature from EEG and ECG are combined, where we obtain a feature matrix with the dimension of 24, 840 192. This feature matrix is classified using two classifiers: SVM and KNN. For KNN (k = 1), Euclidean distance is used as a distance metric; for SVM, cubic kernels are used. EEG epoch ECG epoch -100 -150 Mode 1 Mode 1 -100 -50 Mode 2 Mode 2 0 0 -20 -50 Mode 3 Mode 3 -50 -50 Mode 4 Mode 4 0 0 -20 -20 20 40 60 80 100 120 50 100 150 200 250 Sample number Sample number (a) (b) Figure 5. FBSEEWTbased signal decomposition of (a) EEG and (b) ECG epochs into different modes. Performance was evaluated using the classifier learner application present in MATLAB 2022a. The system used for computing has an Intel i7 CPU with 8 GB of RAM. For three different dimensions of emotion, i.e., arousal, dominance, and valence, three different binary classifications are performed. Thus, each classifier groups data into either high or low arousal, high or low dominance, and high or low valence classes. Low and high class is Amplitude Entropy 2022, 24, 1322 11 of 22 decided based on the rating given by the participant on a scale of 1 to 5, with 3 as threshold, i.e., rating 6 3 is consider low and all above it is considered as high [33,34], due to which, unbalanced classes are obtained for some participants [32]. Further, the performance of the proposed method is evaluated using the first features obtained from EEG signals, then ECG signals, and then using the combined features. FBSE-EWT filter-bank FBSE-EWT filter-bank 3 3 2 2 1 1 0 0 0 20 40 60 0 50 100 Frequency (Hz) Frequency (Hz) Figure 6. FBSEEWTbased filter bank used for decomposing (a) EEG and (b) ECG epochs into different modes (filter corresponding to different modes are shown in different colors). Features obtained from EEG and ECG signals of different subjects are combined together and the classifiers are trained and tested subject independently. Both SVM and KNN are evaluated independently using five-fold cross-validation, where different signal feature observations are placed randomly independent to the subject. For the EEG feature and arousal, the dimension accuracy obtained for SVM is 81.53% and for KNN it is 97.50%. Similarly, performance was evaluated for the case of dominance and valence, which are summarized in Tables 1–3. The tables consist of classification results obtained by using different modalities, i.e., EEG, ECG, and the combined multimodal approach, and the best results obtained are highlighted using bold fonts. The confusion matrices for three different emotion dimensions classifications based on EEG, ECG, and multimodal signals are shown in Figures 7–9. Table 1. Performance with different modalities for arousal emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.50 97.89 97.01 97.67 97.78 (k = 1) EEG SVM 81.53 82.04 80.81 86.02 83.98 (Cubic) KNN 69.94 73.17 65.73 73.56 73.37 (k = 1) ECG SVM 63.84 65.09 61.35 77.11 70.59 (Cubic) KNN 97.84 98.10 97.51 98.06 98.08 (k = 1) Multimodal SVM 84.54 84.75 84.24 88.45 86.56 (Cubic) * in percent. Accuracy parameters, such as sensitivity, specificity, precision, and F1 score, are shown in Tables 1–3. From these tables, it may be observed that the accuracy obtained from EEG and multimodal are almost similar; however, among the two, multimodal is still the winner as it can be seen that all other parameters of classifier reliability check are having sightly better result than the unimodal EEG signal. Following mathematical expressions have been used for calculating parameters namely sensitivity, specificity, precision, and F1 score [53]: TP Sensitivity = (19) TP + FN Magnitude Magnitude Entropy 2022, 24, 1322 12 of 22 TN Specificity = (20) FP + TN TP Precision = (21) TP + FP 2TP F1 Score = (22) 2TP + FP + FN where TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative. Table 2. Performance with different modalities for dominance emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.68 97.89 97.44 97.63 97.76 (k = 1) EEG SVM 81.53 82.04 80.81 86.02 83.98 (Cubic) KNN 69.25 70.33 68.08 70.57 70.45 (k = 1) ECG SVM 62.30 62.90 61.56 66.81 64.80 (Cubic) KNN 97.91 97.95 97.86 98.02 97.99 (k = 1) Multimodal SVM 84.12 83.74 84.55 86.15 84.93 (Cubic) * in percent. Table 3. Performance with different modalities for valence emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.55 97.97 96.89 97.98 97.98 (k = 1) EEG SVM 81.49 82.03 80.47 88.97 85.36 (Cubic) KNN 68.82 74.59 60.22 73.69 74.13 (k = 1) ECG SVM 63.31 66.03 55.28 81.31 72.88 (Cubic) KNN 98.17 98.23 97.86 98.29 97.19 (k = 1) Multimodal SVM 84.10 84.77 82.90 89.93 87.28 (Cubic) * in percent. Entropy 2022, 24, 1322 13 of 22 EEG (KNN) EEG (SVM) 0 13,654 326 0 12,026 1954 10,566 1 294 1 2633 8227 0 1 0 1 Predicted class Predicted class ECG (KNN) ECG (SVM) 0 10,284 3696 0 10,780 3200 1 3770 7090 1 5781 5079 0 1 0 1 Predicted class Predicted class Multimodal (SVM) Multimodal (KNN) 0 12,365 1615 0 13,709 271 1 2225 8635 10,594 1 266 0 1 0 1 Predicted class Predicted class Figure 7. Confusion matrix for arousal emotion dimension. True class True class True class True class True class True class Entropy 2022, 24, 1322 14 of 22 EEG (KNN) EEG (SVM) 0 12,594 306 0 12,026 1954 1 271 11,669 1 2633 8227 0 1 0 1 Predicted class Predicted class ECG (KNN) ECG (SVM) 0 9103 3797 0 8619 4281 1 3841 8099 1 5083 6857 0 1 0 1 Predicted class Predicted class Multimodal (KNN) Multimodal (SVM) 0 12,645 255 0 11,113 1787 11,676 1 264 1 2158 9782 0 1 0 1 Predicted class Predicted class Figure 8. Confusion matrix for dominance emotion dimension. True class True class True class True class True class True class Entropy 2022, 24, 1322 15 of 22 EEG (KNN) EEG (SVM) 0 14,756 304 0 13,399 1661 1 305 9475 1 2936 6844 0 1 0 1 Predicted class Predicted class ECG (SVM) ECG (KNN) 0 11,097 3963 0 12,246 2814 1 3781 5999 1 6301 3479 0 1 0 1 Predicted class Predicted class Multimodal (KNN) Multimodal (SVM) 0 14,785 275 0 13,544 1516 1 257 9523 1 2433 7347 0 1 0 1 Predicted class Predicted class Figure 9. Confusion matrix for valence emotion dimension. 4. Discussion Although, in recent years, significant research has been carried out on emotion recogni- tion related topics, still, it is challenging due to the fuzziness in distinction among different emotions. The study presented in this article demonstrated an approach for developing an automated human emotion identification that is able to categorize the three dimensions of emotion, i.e., arousal, valence, and dominance, into high or low classes of a particular dimension. In order to retrieve emotion-related information from the physiological signals, the FBSE-EWT signal decomposition method is used. Then, various FB-based entropies, such as SSE, WE, and LEE features, are computed from the modes obtained after signal decomposition. Figures 10 and 11 show different boxplots of mentioned entropy features obtained from EEG and ECG, respectively. The plots give information related to feature value distribution among different classes. Due to difference in the interquartile distance among different feature values good classification accuracy is obtained. Then, the extracted True class True class True class True class True class True class Entropy 2022, 24, 1322 16 of 22 features are fed to two different classifiers: SVM and KNN. The above results show that the KNN classifier is found to have more accuracy than the SVM classifier. Further, the multi-model scenario is found to be more beneficial to reliably identify the emotional states of the subject. A statistical significance test was performed to show that the accu- racy improvement in the multimodal emotion recognition model is significant [56]. An analysis of variance (ANOVA) test was performed on the 5-fold accuracies of different modalities, such as EEG, ECG, and multimodal. Figure 12 shows the accuracies of the high and low states of arousal, dominance, and valance using boxplots. The p-values 18 21 for the statistical test for arousal, dominance, and valance are 1.33 10 , 2.82 10 , and 1.50 10 , respectively, which indicates a significant improvement in accuracies for the multimodal emotion detection model. The proposed methodology is compared with the various existing approaches to identify the emotional states. The proposed method found to have superior performance as highlighted in Table 4 using bold fonts. In [57], various features related to EEG and ECG signals, such as power spectral density (PSD), HRV, entropy, EEG-topographical image-based, and ECG spectrogram image-based DL features, are calculated. Topic et al. [58] used EEG signals for emotion detection and has derived holographic features and for maximizing model accuracy, only relevant channels are selected. In [59], using data preprocessing, frame level features are derived from the EEG signal, from which, effective features are extracted using a teacher–student frame- work. In [60], the deep canonical correlation analysis (DCCA) method is proposed for emotion recognition. From the table, it may be noted that currently, various studies are being performed for emotion detection based on a DL-related framework [34,58,60–62] as it automatically finds the most significant features; however, it is difficult to understand or to find the reason behind the results obtained. Moreover, DL models are computation- ally complex and require a large amount of data to train [28]. Feature extraction in other existing state-of-the-art methods is a complex process and less accurate than the study presented here, which differentiates itself from the other existing studies as the results are encouraging. Thus, the proposed method has several advantages over the various listed DL-based feature extraction methods, such as being easy to comprehend, less complex, and more reliable. The disadvantage of the proposed multimodal approach is that its time complexity has increased compared to a single EEG-only modality. Still, the increased reliability makes it more suitable for feature extraction. As the results obtained using the proposed methodology have only been tested on a small size dataset, in the future, they can be tested on a larger database in order to verify their validity. Furthermore, in this study, a fixed 60 s duration physiological signals was used; this can also be made adaptive in order to select the time portion, which can make the methodology more efficient and adaptive. Entropy 2022, 24, 1322 17 of 22 Table 4. Comparison of existing methodologies applied on DREAMER dataset with the proposed emotion recognition method. Accuracy (%) Authors (Year) Methodology Modality LA-HA LD-HD LV-HV EEG & ECG 62.32 61.84 62.49 Katsigiannis et al. [32] (2018) PSD features and SVM classifier Song et al. [62] (2018) DGCNN EEG 84.54 85.02 86.23 Zhang et al. [34] (2019) PSD features and GCB-net classifier EEG 89.32 89.20 86.99 Bhattacharyya et al. [28] (2020) MFBSE-EWT based entropy features and ARF classifier EEG 85.4 86.2 84.5 Cui et al. [61] (2020) RACNN EEG 97.01 - 95.55 Kamble et al. [63] (2021) DWT, EMD-based features, and CML and EML-based classifier EEG 93.79 - 94.5 Li et al. [64] (2021) 3DFR-DFCN EEG 75.97 85.14 82.68 Siddharth et al. [57] (2022) PSD, HRV, entropy, DL based feature, and LSTM based classifier EEG & ECG 79.95 - 79.95 Topic et al. [58] (2022) Holographic features and CNN EEG 92.92 92.97 90.76 Gu et al. [59] (2022) FLTSDP EEG 90.61 91.00 91.54 Liu et al. [60] (2022) DCCA EEG & ECG 89.00 90.7 90.6 Proposed work FBSE-EWT-based entropy features and KNN classifier EEG & ECG 97.84 97.91 97.86 PSD = power spectral density, SVM = support-vector machines, DGCNN = dynamic graph convolutional neural network, GCB-net = graph convolutional broad network, MFBSE-EWT = multivariate FBSE-EWT, ARF = adaptive random forest, RACNN = region-based convolutional neural network, DWT = discrete wavelet transform, EMD = empirical mode decom- position, CML = conventional machine learning, 3DFR-DFCN = 3-D feature representation and dilated fully convolutional network, HRV = heart rate variability, DL = deep learning, LSTM = long short term memory, CNN = convolutional neural network, FLTSDP = frame level teacher-student framework with data privacy, DCCA = deep canonical correlation analysis. 0.1 5.5 5.5 2200 2200 5 5 2100 2100 0.08 4.5 4.5 4 4 1900 1900 0.06 3.5 3.5 1800 1800 3 1800 3 0.04 1700 1700 2.5 2.5 0.02 2 2 1.5 1.5 1400 1400 HA LA HA LA HA LA HD LD HD LD HD LD Emotion class Emotion class Emotion class Emotion class Emotion class Emotion class 0.1 4.5 0.08 3.5 0.06 0.04 2.5 0.02 1.5 HV LV HV LV HV LV Emotion class Emotion class Emotion class Figure 10. Box plots for the features calculated using EEG signals. SSE LEE SSE WE LEE SSE WE LEE WE Entropy 2022, 24, 1322 18 of 22 0.18 0.18 0.16 0.16 0.14 0.14 3800 0.12 0.12 0.1 0.1 3400 3400 0.08 0.08 3200 0.06 3200 0.06 0.04 3000 0.04 0.02 0.02 0 0 HA LA HA LA HA LA HD LD HD LD HD LD Emotion class Emotion class Emotion class Emotion class Emotion class Emotion class 6 4.5 3.5 2.5 1.5 0.5 HV LV HV LV HV LV Emotion class Emotion class Emotion class Figure 11. Box plots for the features calculated using ECG signals. 90 90 80 80 75 75 70 70 (c) (a) (b) Figure 12. Box plots showing statistical significance of accuracies for different modalities (a) arousal, (b) dominance, and (c) valence. 5. Conclusions This study presents an automated emotion identification by a multimodal approach using FB domain-based entropies. The publically available DREAMER data set was used to evaluate the performance of the proposed algorithm. Physiological signals obtained from the dataset, namely EEG and ECG, are decomposed using the FBSE-EWT into four modes. EEG ECG Multimodal EEG ECG Multimodal EEG ECG Multimodal SSE Accuracy (in %) LEE SSE WE Accuracy (in %) LEE SSE WE Accuracy (in %) LEE WE Entropy 2022, 24, 1322 19 of 22 From these modes, several new FB-based entropy features, such as FB spectral-based SSE, LEE, and WE were computed. The dataset consisted of three-dimensional emotional space, i.e., arousal, valence, and dominance. Each of the dimensions are categorized into high and low classes based on the rating provided along with the dataset. The proposed method using the KNN classifier provides the highest accuracies of 97.84%, 97.91%, and 97.86% for arousal, valence, and dominance emotion classes, respectively. After comparing it with results obtained from current methods, significant improvement in the results is obtained. Moreover, the multimodal approach is found to be the most accurate in terms of identifying human emotional states. Author Contributions: A.N., K.D. and R.B.P. contributed equally to this work. All authors have read and agreed to the published version of the manuscript. Funding: This study is supported by the Council of Scientific & Industrial Research (CSIR) funded Research Project, Government of India, Grant number: 22(0851)/20/EMR-II. Institutional Review Board Statement: Not applicable. Data Availability Statement: The EEG and ECG data are provided by the Stamos Katsigiannis collected at University of the West of Scotland. Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: ANN Artificial neural network ANOVA Analysis of variance ANS Autonomous nervous system ARF Autoencoder-based random forest BEMD Bivariate empirical mode decomposition DCCA Deep canonical correlation analysis DL Deep learning DWT Discrete wavelet transform ECG Electrocardiogram EEG Electroencephalogram EMD Empirical mode decomposition EWT Empirical wavelet transform FB Fourier–Bessel FBSE Fourier–Bessel series expansion FBSE-EWT FBSE-based EWT FN False negative FP False positive FrFT Fractional Fourier transform GSR Galvanic skin response HA High arousal HCI Human–computer interaction HD High dominance HOC Higher-order crossing HRV Heart rate variability HV High valence ICA Independent component analysis IMF Intrinsic mode functions Entropy 2022, 24, 1322 20 of 22 IP Information potential KNN K-nearest neighbors LA Low arousal LA-LL Left arm and left leg LD Low dominance LEE Log energy entropy LV Low valence MC-LS-SVM Multiclass least squares support vector machines MHMS Multivariate Hilbert marginal spectrum MSST Multivariate synchrosqueezing transform NCA Neighborhood component analysis NMF Non-negative matrix factorization PCA Principal component analysis PSD Power spectral density RA-LL Right arm and Left leg SDEL Sparse discriminative ensemble learning SODP Second order difference plot SSE Shannon spectral entropy STFT Short time Fourier transform SVM Support vector machines TN True negative TP True positive TQWT Tunable Q-factor wavelet transform WE Wiener entropy WIB With-in beat References 1. Ptaszynski, M.; Dybala, P.; Shi, W.; Rzepka, R.; Araki, K. Towards context aware emotional intelligence in machines: Computing contextual appropriateness of affective states. In Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, CA, USA, 11–17 July 2009; AAAI: Menlo Park, CA, USA; pp. 1469–1474. 2. Vingerhoets, A.; Nyklícek, I.; Denollet, J. Emotion Regulation; Springer: Berlin/Heidelberg, Germany, 2008. 3. Kroupi, E.; Yazdani, A.; Ebrahimi, T. EEG correlates of different emotional states elicited during watching music videos. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA, 9–12 October 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 457–466. 4. Jin, L.; Kim, E.Y. Interpretable cross-subject EEG-based emotion recognition using channel-wise features. Sensors 2020, 20, 6719. 5. Kołakowska, A.; Landowska, A.; Szwoch, M.; Szwoch, W.; Wrobel, M.R. Emotion recognition and its applications. In Human- Computer Systems Interaction: Backgrounds and Applications 3; Springer: Berlin/Heidelberg, Germany, 2014; pp. 51–62. 6. Šimic, ´ G.; Tkalci ˇ c, ´ M.; Vukic, ´ V.; Mulc, D.; Španic, ´ E.; Šagud, M.; Olucha-Bordonau, F.E.; Vukšic, ´ M.; R. Hof, P. Understanding emotions: Origins and roles of the amygdala. Biomolecules 2021, 11, 823. 7. Doukas, C.; Maglogiannis, I. Intelligent pervasive healthcare systems. In Advanced Computational Intelligence Paradigms in Healthcare-3; Springer: Berlin/Heidelberg, Germany, 2008; pp. 95–115. 8. McCraty, R. Science of the Heart: Exploring the Role of the Heart in Human Performance; HeartMath Research Center, Institute of HeartMath: Boulder Creek, CA, USA, 2015. 9. Filippini, C.; Di Crosta, A.; Palumbo, R.; Perpetuini, D.; Cardone, D.; Ceccato, I.; Di Domenico, A.; Merla, A. Automated affective computing based on bio-signals analysis and deep learning approach. Sensors 2022, 22, 1789. 10. Kipli, K.; Latip, A.A.A.; Lias, K.; Bateni, N.; Yusoff, S.M.; Tajudin, N.M.A.; Jalil, M.; Ray, K.; Shamim Kaiser, M.; Mahmud, M. GSR signals features extraction for emotion recognition. In Proceedings of Trends in Electronics and Health Informatics, Singapore, 22 March 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 329–338. 11. Dutta, S.; Mishra, B.K.; Mitra, A.; Chakraborty, A. An analysis of emotion recognition based on GSR signal. ECS Trans. 2022, 107, 12535. 12. Panahi, F.; Rashidi, S.; Sheikhani, A. Application of Fractional Fourier Transform in Feature Extraction from Electrocardiogram and Galvanic Skin Response for Emotion Recognition. Biomed. Signal Process. Control. 2021, 69, 102863. 13. Zhang, Q.; Chen, X.; Zhan, Q.; Yang, T.; Xia, S. Respiration-based emotion recognition with deep learning. Comput. Ind. 2017, 92, 84–90. 14. Agrafioti, F.; Hatzinakos, D.; Anderson, A.K. ECG pattern analysis for emotion detection. IEEE Trans. Affect. Comput. 2011, 3, 102–115. 15. Cheng, Z.; Shu, L.; Xie, J.; Chen, C.P. A novel ECG-based real-time detection method of negative emotions in wearable applications. In Proceedings of the 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), Shenzhen, China, 15–17 December 2017; IEEE: Washington, DC, USA, 2017; pp. 296–301. Entropy 2022, 24, 1322 21 of 22 16. Dissanayake, T.; Rajapaksha, Y.; Ragel, R.; Nawinne, I. An ensemble learning approach for electrocardiogram sensor based human emotion recognition. Sensors 2019, 19, 4495. 17. Sepúlveda, A.; Castillo, F.; Palma, C.; Rodriguez-Fernandez, M. Emotion recognition from ECG signals using wavelet scattering and machine learning. Appl. Sci. 2021, 11, 4945. 18. Anuragi, A.; Sisodia, D.S.; Pachori, R.B. EEG-based cross-subject emotion recognition using Fourier-Bessel series expansion based empirical wavelet transform and NCA feature selection method. Inf. Sci. 2022, 610, 508–524. 19. Sharma, R.; Pachori, R.B.; Sircar, P. Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed. Signal Process. Control. 2020, 58, 101867. 20. Bajaj, V.; Pachori, R.B. Human emotion classification from EEG signals using multiwavelet transform. In Proceedings of the 2014 International Conference on Medical Biometrics, Shenzhen, China, 30 May–1 June 2014; IEEE: Washington, DC, USA, 2014; pp. 125–130. 21. Bajaj, V.; Pachori, R.B. Detection of human emotions using features based on the multiwavelet transform of EEG signals. In Brain-Computer Interfaces; Springer: Berlin/Heidelberg, Germany, 2015; pp. 215–240. 22. Lin, Y.P.; Wang, C.H.; Jung, T.P.; Wu, T.L.; Jeng, S.K.; Duann, J.R.; Chen, J.H. EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 2010, 57, 1798–1806. 23. Liu, Y.; Sourina, O. Real-time subject-dependent EEG-based emotion recognition algorithm. In Transactions on Computational Science XXIII; Springer: Berlin/Heidelberg, Germany, 2014; pp. 199–223. 24. Mohammadi, Z.; Frounchi, J.; Amiri, M. Wavelet-based emotion recognition system using EEG signal. Neural Comput. Appl. 2017, 28, 1985–1990. 25. Mert, A.; Akan, A. Emotion recognition based on time-frequency distribution of EEG signals using multivariate synchrosqueezing transform. Digit. Signal Process. 2018, 81, 106–115. 26. Gupta, V.; Chopda, M.D.; Pachori, R.B. Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sens. J. 2018, 19, 2266–2274. 27. Ullah, H.; Uzair, M.; Mahmood, A.; Ullah, M.; Khan, S.D.; Cheikh, F.A. Internal emotion classification using EEG signal with sparse discriminative ensemble. IEEE Access 2019, 7, 40144–40153. 28. Bhattacharyya, A.; Tripathy, R.K.; Garg, L.; Pachori, R.B. A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sens. J. 2020, 21, 3579–3591. 29. Salankar, N.; Mishra, P.; Garg, L. Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomed. Signal Process. Control. 2021, 65, 102389. 30. Nalwaya, A.; Das, K.; Pachori, R.B. Emotion identification from TQWT-Based EEG rhythms. In AI-Enabled Smart Healthcare Using Biomedical Signals; IGI Global: Hershey, PA, USA, 2022; pp. 195–216. 31. Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 2018, 8, 13657. 32. Katsigiannis, S.; Ramzan, N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 2017, 22, 98–107. 33. Cheng, J.; Chen, M.; Li, C.; Liu, Y.; Song, R.; Liu, A.; Chen, X. Emotion recognition from multi-channel eeg via deep forest. IEEE J. Biomed. Health Inform. 2020, 25, 453–464. 34. Zhang, T.; Wang, X.; Xu, X.; Chen, C.P. GCB-Net: Graph convolutional broad network and its application in emotion recognition. IEEE Trans. Affect. Comput. 2019, 13, 379–388. 35. Gupta, V.; Bhattacharyya, A.; Pachori, R.B. Automated identification of epileptic seizures from EEG signals using FBSE-EWT method. In Biomedical Signal Processing; Springer: Berlin/Heidelberg, Germany, 2020; pp. 157–179. 36. Anuragi, A.; Sisodia, D.S.; Pachori, R.B. Epileptic-seizure classification using phase-space representation of FBSE-EWT based EEG sub-band signals and ensemble learners. Biomed. Signal Process. Control. 2022, 71, 103138. 37. Khan, S.I.; Qaisar, S.M.; Pachori, R.B. Automated classification of valvular heart diseases using FBSE-EWT and PSR based geometrical features. Biomed. Signal Process. Control. 2022, 73, 103445. 38. Khan, S.I.; Pachori, R.B. Derived vectorcardiogram based automated detection of posterior myocardial infarction using FBSE-EWT technique. Biomed. Signal Process. Control. 2021, 70, 103051. 39. Bhattacharyya, A.; Singh, L.; Pachori, R.B. Fourier-Bessel series expansion based empirical wavelet transform for analysis of non-stationary signals. Digit. Signal Process. 2018, 78, 185–196. 40. Schroeder, J. Signal Processing via Fourier-Bessel series expansion. Digit. Signal Process. 1993, 3, 112–124. 41. Das, K.; Verma, P.; Pachori, R.B. Assessment of chanting effects using EEG signals. In Proceedings of the 2022 24th International Conference on Digital Signal Processing and Its Applications (DSPA), 30 March–1 April 2022, Moscow, Russia; IEEE: Washington, DC, USA, 2022, pp. 1–5. 42. Pachori, R.B.; Sircar, P. EEG signal analysis using FB expansion and second-order linear TVAR process. Signal Process. 2008, 88, 415–420. 43. Gilles, J.; Heal, K. A parameterless scale-space approach to find meaningful modes in histograms—Application to image and spectrum segmentation. Int. J. Wavelets Multiresolut. Inf. Process. 2014, 12, 1450044. 44. Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 1979, 9, 62–66. 45. Gilles, J. Empirical wavelet transform. IEEE Trans. Signal Process. 2013, 61, 3999–4010. Entropy 2022, 24, 1322 22 of 22 46. Kowalski, A.; Martín, M.; Plastino, A.; Rosso, O. Bandt-Pompe approach to the classical-quantum transition. Phys. D Nonlinear Phenom. 2007, 233, 21–31. 47. Zheng, W.L.; Lu, B.L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 2015, 7, 162–175. 48. Pachori, R.B.; Hewson, D.; Snoussi, H.; Duchêne, J. Analysis of center of pressure signals using empirical mode decomposition and Fourier-Bessel expansion. In Proceedings of the TENCON 2008—2008 IEEE Region 10 Conference, Hyderabad, India, 19–21 November 2008; IEEE: Washington, DC, USA, 2008; pp. 1–6. 49. Shannon, C.E. A mathematical theory of communication. ACM SIGMOB. Mob. Comput. Commun. Rev. 2001, 5, 3–55. 50. Dubnov, S. Generalization of spectral flatness measure for non-Gaussian linear processes. IEEE Signal Process. Lett. 2004, 11, 698–701. 51. Aydın, S.; Saraoglu, ˘ H.M.; Kara, S. Log energy entropy-based EEG classification with multilayer neural networks in seizure. Ann. Biomed. Eng. 2009, 37, 2626–2630. 52. Li, X.; Zhang, Y.; Tiwari, P.; Song, D.; Hu, B.; Yang, M.; Zhao, Z.; Kumar, N.; Marttinen, P. EEG based emotion recognition: A tutorial and review. ACM Comput. Surv. (CSUR) 2022. https://doi.org/10.1145/3524499. 53. Sebastiani, F. Machine learning in automated text categorization. ACM Comput. Surv. (CSUR) 2002, 34, 1–47. 54. Cortes, C.; Vapnik, V. Support-vector networks. Machine learning 1995, 20, 273–297. 55. Fukunaga, K. Introduction to Statistical Pattern Recognition, 2nd ed.; Academic Press: Cambridge, USA, 1990; pp. 300–361. 56. Muralidharan, N.; Gupta, S.; Prusty, M.R.; Tripathy, R.K. Detection of COVID19 from X-ray Images Using Multiscale Deep Convolutional Neural Network. Appl. Soft Comput. 2022, 119, 108610. 57. Siddharth, S.; Jung, T.P.; Sejnowski, T.J. Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Trans. Affect. Comput. 2019, 13, 96–107. 58. Topic, A.; Russo, M.; Stella, M.; Saric, M. Emotion recognition using a reduced set of EEG channels based on holographic feature maps. Sensors 2022, 22, 3248. 59. Gu, T.; Wang, Z.; Xu, X.; Li, D.; Yang, H.; Du, W. Frame-Level Teacher-Student Learning with Data Privacy for EEG emotion recognition. IEEE Trans. Neural Netw. Learn. Syst. 2022. https://doi.org/10.1109/TNNLS.2022.3168935. 60. Liu, W.; Qiu, J.L.; Zheng, W.L.; Lu, B.L. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Trans. Cogn. Dev. Syst. 2021, 14, 715–729. 61. Cui, H.; Liu, A.; Zhang, X.; Chen, X.; Wang, K.; Chen, X. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl.-Based Syst. 2020, 205, 106243. 62. Song, T.; Zheng, W.; Song, P.; Cui, Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 2018, 11, 532–541. 63. Kamble, K.S.; Sengupta, J. Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals. IEEE Sens. J. 2021, 22, 2496–2507. 64. Li, D.; Chai, B.; Wang, Z.; Yang, H.; Du, W. EEG emotion recognition based on 3-D feature representation and dilated fully convolutional networks. IEEE Trans. Cogn. Dev. Syst. 2021, 13, 885–897. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Entropy Multidisciplinary Digital Publishing Institute

Automated Emotion Identification Using Fourier&ndash;Bessel Domain-Based Entropies

Nalwaya, Aditya; Das, Kritiprasanna; Pachori, Ram Bilas
Entropy , Volume 24 (10) – Sep 20, 2022
Download PDF
22 pages

Loading next page...
 
/lp/multidisciplinary-digital-publishing-institute/automated-emotion-identification-using-fourier-ndash-bessel-domain-Bv9iQeQ6J4

References (67)

Publisher
Multidisciplinary Digital Publishing Institute
Copyright
© 1996-2022 MDPI (Basel, Switzerland) unless otherwise stated Disclaimer The statements, opinions and data contained in the journals are solely those of the individual authors and contributors and not of the publisher and the editor(s). MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Terms and Conditions Privacy Policy
ISSN
1099-4300
DOI
10.3390/e24101322
Publisher site
See Article on Publisher Site

Abstract

entropy Article Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies Aditya Nalwaya *, Kritiprasanna Das and Ram Bilas Pachori Department of Electrical Engineering, Indian Institute of Technology Indore, Indore 453552, India * Correspondence: adityanalwaya@iiti.ac.in Abstract: Human dependence on computers is increasing day by day; thus, human interaction with computers must be more dynamic and contextual rather than static or generalized. The development of such devices requires knowledge of the emotional state of the user interacting with it; for this purpose, an emotion recognition system is required. Physiological signals, specifically, electrocar- diogram (ECG) and electroencephalogram (EEG), were studied here for the purpose of emotion recognition. This paper proposes novel entropy-based features in the Fourier–Bessel domain instead of the Fourier domain, where frequency resolution is twice that of the latter. Further, to represent such non-stationary signals, the Fourier–Bessel series expansion (FBSE) is used, which has non-stationary basis functions, making it more suitable than the Fourier representation. EEG and ECG signals are decomposed into narrow-band modes using FBSE-based empirical wavelet transform (FBSE-EWT). The proposed entropies of each mode are computed to form the feature vector, which are further used to develop machine learning models. The proposed emotion detection algorithm is evaluated using publicly available DREAMER dataset. K-nearest neighbors (KNN) classifier provides accuracies of 97.84%, 97.91%, and 97.86% for arousal, valence, and dominance classes, respectively. Finally, this paper concludes that the obtained entropy features are suitable for emotion recognition from given physiological signals. Keywords: Fourier–Bessel series expansion; spectral entropy; ECG; EEG; FBSE-EWT Citation: Nalwaya, A.; Das, K.; Pachori, R.B. Automated Emotion Identification Using Fourier–Bessel Domain-Based Entropies. Entropy 2022, 24, 1322. https://doi.org/ 1. Introduction 10.3390/e24101322 Nowadays, humans are becoming more and more dependent on computers for their various day-to-day tasks. Thus, the need for making computers more user-friendly and Academic Editor: Daniel Abasolo engaging is becoming more and more common rather than just being logically or com- Received: 7 August 2022 putationally efficient [1]. To make computers more user-friendly, it must recognize the Accepted: 16 September 2022 emotion of the user interacting with it, as emotion is the most basic component that is Published: 20 September 2022 responsible for human attention, action, or behavior in a particular situation [2]; therefore, Publisher’s Note: MDPI stays neutral for the applications related to human–computer interaction (HCI), an emotion recognition with regard to jurisdictional claims in system is very useful [3]. Further, such HCI system can be very useful in areas related to published maps and institutional affil- mental health care [4], smart entertainment [5], and an assistive system for a physically iations. disabled person [6]. Emotion recognition systems generally have two different approaches for recognizing emotion, i.e., explicit approach, which includes expression on the face, speech, etc., that are visible and can be easily captured for analysis; however, the problem with such an approach Copyright: © 2022 by the authors. is that the signals obtained in this approach are not very reliable because the subject under Licensee MDPI, Basel, Switzerland. consideration can hide his/her emotion, so these might not show the actual emotional This article is an open access article state of the subject. An alternative way of recognizing emotion is an implicit approach distributed under the terms and where physiological signals, such as electroencephalogram (EEG), electrocardiogram (ECG), conditions of the Creative Commons galvanic skin response (GSR), etc., are being captured and analyzed. Such signals are not Attribution (CC BY) license (https:// visible and are generated inside the body by the autonomous nervous system (ANS). creativecommons.org/licenses/by/ Any change in the emotional state of a person is reflected in the respiration rates, body 4.0/). Entropy 2022, 24, 1322. https://doi.org/10.3390/e24101322 https://www.mdpi.com/journal/entropy Entropy 2022, 24, 1322 2 of 22 temperature, heart rate, and other physiological signals, which are not under the control of the subject [7]. The brain is a central command center. It reacts to any external stimuli by firing neurons inside the brain, causing changes in ANS activity, due to which, the routine activity of the heart and other peripheral organs varies. In addition, change in the emotional state affects heart activity [8]. EEG and ECG signals are the biopotential that reflect the activities of the brain and heart, respectively. Thus, in this study, to recognize the emotion of the subject, physiological signals namely, EEG and ECG are considered. Although in the literature, some other physiological signals are also used to determine the emotional state of a person, such as changes in the conductivity of the skin, which is measured using GSR [9–11]. In [12], the authors have extracted time–frequency domain features using fractional Fourier transform (FrFT); the most relevant features are selected using the Wilcoxon method, which is then given as input to a support vector machine (SVM) classifier for determining the emotional state of the person. Similarly, the respiration rate also helps in determining the emotional state of the person. In [13], authors have used deep learning (DL)-based feature extraction method and logistic regression, determining different emotional states from the obtained features. In recent years, several emotion recognition frameworks have been established using different physiological signals. The authors in [14] have proposed a methodology that uses ECG signal features derived from different intrinsic mode functions (IMFs) for emotion classification. Using bivariate empirical mode decomposition (BEMD), IMFs are obtained. The instantaneous frequency and local oscillation feature are computed, which are given as input to the linear discriminant classifier to discriminate different emotional states. A method is proposed to determine negative emotion through a single channel ECG signal [15]. Different features, extracted from the ECG signals, are linear features, i.e., mean of RR interval. Nonlinear features consist of bispectral analysis, power content in different frequency bands, time–domain features contains different statistical parameters of ECG signal, and time–frequency domain features were obtained using wavelet-based signals decomposition. Based on these features, different emotions were detected using several machine learning classifiers. The authors of [16] have extracted four different features from the ECG signal, i.e., heart rate variability (HRV), with-in beat (WIB), frequency spectrum, and signal decomposition-based features. To evaluate the performance of the obtained features, ensemble classifiers are used. In [17], the authors have extracted ECG signal features at different time scale using wavelet scattering, and features obtained are given as an input to various classifiers for evaluating their performance. Similarly, different emotion recognition techniques have been proposed based on EEG signals. Anuragi et al. [18] have proposed EEG based emotion detection framework. Using the Fourier–Bessel series expansion (FBSE)-based empirical wavelet transform (EWT) (FBSE-EWT) method, the EEG signals are decomposed into four sub-band signals from which features such as energy and entropy are extracted. The features are selected using neighborhood component analysis (NCA), using different classifiers, such as k-nearest neighborhood (KNN), ensemble bagged tree, and artificial neural network (ANN), emotion class is identified. Sharma et al. [19] have used discrete wavelet transform (DWT) for EEG signal decomposition. Third-order cumulant nonlinear features are extracted from each sub-band, and using swarm optimization features, dimensionality is reduced. To classify the emotional states, a long short-term memory-based technique is used. Bajaj et al. [20] have used multiwavelet transform to decompose EEG signals into narrow sub-signals. Using Euclidean distance, features are extracted that are computed from a 3-D phase space diagram. Multiclass least squares support vector machines (MC-LS-SVM) classifier is used for identifying class of emotion; however, the authors in [21] have computed features, namely entropy and ratio of the norms-based measure, are computed from the sub- signals that are obtained after decomposing EEG signal using multiwavelet decomposition. The feature matrix is given as input to MC-LS-SVM for determining the emotional state. In [22], features related to changes in the spectral power of EEG signal are extracted Entropy 2022, 24, 1322 3 of 22 using a short-time Fourier transform (STFT). The best 55 features among 60 features are selected using F-score. A feature vector is formed from the selected features which are given as an input to the SVM classifier for determining the emotional state of the subject. Liu et al. [23] have extracted fractal dimension and higher-order crossing (HOC) as features via the sliding window approach from the given EEG signal; based on these features, an emotion class is determined using an SVM classifier. In [24], the authors have decomposed EEG signals in different sub-bands using DWT. From the obtained wavelet coefficient features, entropy and energy of the coefficients are calculated. Finally, using SVM and KNN classifiers, emotional states are obtained.The authors in [25] have used a multi-channel signal processing technique called multivariate synchrosqueezing transform (MSST) for representing EEG signals in time–frequency domain. Independent component analysis (ICA) is used to reduce dimensionality of the obtained high-dimensional feature matrix. In addition, its performance is compared with non-negative matrix factorization (NMF), which is an alternative feature selection method. The reduced feature matrix is passed to different classifiers such as SVM, KNN, and ANN to discriminate different emotions. Gupta et al. used flexible analytic wavelet transform for decomposing EEG signals into narrow-band sub-bands [26]. From each sub-band, information potential (IP) is computed using Reyni’s quadratic entropy, and using moving average filter the extracted features are smoothed. To determine the different emotion classes, a random forest classifier is used. Ullah et al. segmented the large duration EEG signal into smaller epoch [27] from each segment features; specifically, Fisher information ratio, entropy, statistical parameters, and Petrosian fractal dimension, are computed. The above feature vector is passed to a sparse discriminative ensemble learning (SDEL) classifier to identify the emotion class. Bhattacharyya et al. decomposed EEG signals into different modes using FBSE-EWT [28]. From each mode of KNN entropy, multiscale multivariate Hilbert marginal spectrum (MHMS), and spectral Shannon entropy, features are calculated. Features are smoothed and fed to a classifier called sparse autoencoder-based random forest (ARF) for determining the class of human emotion. Nilima et al. used the empirical mode decomposition (EMD) method for decomposing EEG signal [29]. The second-order difference plot (SODP) features are calculated from each IMF. A two-hidden layer multilayer perceptron is used for multi class classification and SVM is used for binary classification. Nalwaya et al. [30], have used tunable Q-factor wavelet transform (TQWT) to separate various rhythms of EEG. Different statistical and information potential features are computed for each rhythm, which are then fed to SVM cubic classifier for emotion identification. Further, some of the studies use both ECG and EEG signals. In [31], the authors have recorded EEG and ECG signals while the subject is exposed to immersive virtual environments to elicit emotion. From ECG signal, time–domain features, frequency domain features, and non-linear features are calculated. Whereas from EEG signal band power and mean phase connectivity features are calculated. Both EEG and ECG features are combined to form a single feature matrix whose dimensionality is then reduced using principal component analysis (PCA). Then, the reduced matrix is passed to an SVM classifier to determine the emotional state of the subject. The literature review of previous studies on human emotion identification shows emerging research trends in finding appropriate signal decomposition techniques, dis- tinctive features, and machine learning algorithms to classify emotional states. Most of the studies carried out previously used a single modality, i.e., either EEG, ECG, or other peripheral physiological signals, and very few studies have been conducted using multiple modalities. Despite this, there is still scope for improvement in the classification accuracy of the earlier proposed methods. In this article, an automated emotion identification framework using EEG and ECG signals is developed. Furthermore, the Fourier–Bessel (FB) domain has been explored instead of working in traditional Fourier domain, due to various advantages associated with latter one. FBSE uses Bessel functions as basis functions whose amplitude decay with time and are damped in nature, which make them more suitable for non-stationary signal Entropy 2022, 24, 1322 4 of 22 analysis. FBSE spectrum has twice the resolution as compared to Fourier domain spectral representation. Thus, looking at such advantages, FBSE-EWT is used instead of EWT for extracting different modes from the given signal. New FB-based spectral entropy, such as Shannon spectral entropy (SSE), log energy entropy (LEE), and Wiener entropy (WE), have been proposed, which are used as features from the obtained modes. Then, smoothing of the feature values is performed by applying moving average over obtained features values, which is then given as input to SVM and KNN classifiers for emotional class identification. The block diagram of the proposed methodology is shown in Figure 1. EEG/ ECG FBSE-EWT based signal decomposition Feature extraction Feature smoothening Classifier Emotional state Figure 1. Block diagram for the proposed emotion detection methodology. The rest of the article is organized as follows: In Section 2, the material and method- ology are discussed in detail. In Section 3, results obtained after applying the proposed methodology are presented. Section 4 presents the performance of the proposed method with the help of the results obtained, compares it with other exiting methodologies, and highlights some of its limitations. Finally, Section 5 concludes the article. 2. Materials and Methods The emotion recognition framework presented in this article consists of preprocessing, signal processing, feature extraction, feature smoothing, and classification steps. 2.1. Dataset Description A publicly available DREAMER dataset is used to evaluate the proposed methodology. It contains raw EEG and ECG signals, which are recorded from 23 healthy participants while the subject is watching audio and video clips. Emotions are quantified in terms of three different scales: arousal, valence, and dominance [32]. Each participant was shown 18 different clips of different durations with a mean of 199 s. The EPOC system by Emotive was used, which contains 16 gold-plated dry electrodes placed in accordance with the international 10–20 system EEG, and were recorded from, i.e., AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, P7, P8, O1, and O2; two reference electrodes, M1 and M2 were placed over mastoid, as described in [32]. To obtain the ECG signals, electrodes were placed in two vector directions, i.e., right arm, left leg (RA-LL) vector and left arm, left leg (LA-LL) vector. ECG was recorded using the SHIMMER ECG sensor. Both EEG and ECG signals were recorded at different sampling rates, i.e., 128 Hz, and 256 Hz, respectively. The sample EEG and ECG signals obtained from the dataset are shown in Figure 2 for different classes of emotion (high arousal (HA) or low arousal (LA), high dominance (HD) or low dominance (LD), high valence (HV) or low valence (LV)). The dataset also contains information relating Entropy 2022, 24, 1322 5 of 22 to the rating of each video on a scale of 1 to 5. Each participant rated the video as per his/her level of emotion elicited in three different dimensions, i.e., valence (pleasantness level), arousal (excitement level), and dominance (level of control). Rating between 1–5 was labeled as ‘0’ (low) or ‘1’ (high) for each dimension, with 3 as the threshold, i.e., if a participant rates a video between 1 to 3, it will be considered as low or ‘0’ and a value above 3 is considered as high or ‘1’ [33,34]. EEG signal of LA class EEG signal of HA class 3800 3838 EEG signal of LD class EEG signal of HD class 4480 4500 EEG signal of LV class EEG signal of HV class 20 40 60 80 100 120 20 40 60 80 100 120 ECG signal of LA class ECG signal of HA class ECG signal of of LD class ECG signal of of HD class ECG signal of of LV class ECG signal of HV class 50 100 150 200 250 50 100 150 200 250 Sample number Sample number Figure 2. Epochs of raw EEG and ECG signals obtained from the DREAMER dataset. 2.2. Preprocessing At this stage, typically, noise and artifacts from the raw signal are removed with the help of filters and denoising algorithms. As in [32], the analysis of only the last 60 s of signal was performed. Further, signals were segmented into small epochs of 1 s, which were then used for further processing. The mean value was subtracted from each epoch. 2.3. FBSE-EWT Analyzing non-stationary signals such as EEG and ECG is difficult as the signal is time-varying in nature and its properties also change continuously. In order to understand the properties of such signals, decomposing them into narrow-band simpler components can help to make it easier to understand; therefore, the preprocessed signal was given to a signal decomposition algorithm, which will decompose the input EEG and ECG signal into various modes. FBSE-EWT is an improved version of the EWT, which has adaptive basis functions derived from the signals. FBSE-EWT has been used for many biomedical signal processing applications, such as epileptic seizure detection [35,36], valvular heart disease diagno- sis [37], and posterior myocardial infarction detection [38]. FBSE-EWT technique flow is shown in Figure 3 and its step by step working is as follows: Amplitude Entropy 2022, 24, 1322 6 of 22 Mode 1 Mode 2 FBSE Scale-space Boundary EWT based Input signal spectrum representation detection filter-bank Mode M Figure 3. Flow diagram of FBSE-EWT based signal decomposition. 1. FBSE spectrum: FBSE has twice the frequency resolution as compared to the Fourier domain. FBSE spectrum of a signal s(n) of length S samples can be obtained us- ing zero-order Bessel functions. The magnitude of the FB coefficients K(i) can be computed mathematically as follows [39–42]: S1 2 b n K(i) = ns(n) J (1) å 0 2 2 S [ J (b )] S 1 i n=0 where J () is the zero-order and J () is the first-order Bessel functions. Positive roots 0 1 of zeroth order Bessel function are denoted by b , which are arranged in ascending order, where i = 1, 2, 3, , S. A one to one relation between order i and continuous frequency is given by [39,42], 2p f S b  (2) where b  b + p  ip, f denotes the sampling frequency. Here, f is the i i1 i th continuous frequency, corresponding to the i order and is expressed as [39], 2 f S i  (3) For covering whole bandwidth of s(n), i must vary from 1 to S. The FBSE spectrum is the plot of magnitude of the FB coefficient K(i) versus frequency f . 2. Scale-space based boundary detection [39,42,43]: For FBSE spectrum, scale-space representation can be obtained by convolving the signal with a kernel of Gaussian type, which is expressed as follows: J(i, q) = K(i n)g(n; q) (4) n=N 2q where g(n; q) = e . Here, N = W q + 1 with 3  W  6 and q is the scale 2pq parameter. As the scale-space parameter, i.e., # = , # = 1, 2, , # , increases, max the number of minima decreases and no new minima will appear in the scale space plan. The FBSE spectrum is segmented using the boundary detection technique. The FBSE spectrum ranges from 0 to p and the FBSE spectrum segments are denoted as [0, w ], [w , w ], , and [w , p], where w , w , , w are boundaries. Typically 1 1 2 i1 1 2 i1 boundaries are defined between two local minima, which are obtained by the two curves in the scale–space plane, whose length is greater than the threshold, which is obtained by using Otsu method [44]. 3. EWT filter bank design: After obtaining the filter boundaries, based on these param- eters, empirical scaling and wavelet functions are adjusted and different band-pass filters are designed. Wavelets scaling (F (w)) and wavelet functions (n (w)) were j j constructed, and the mathematical expressions are given by [45], 1 if jwj  (1 t)w > j h i pg(t,w ) F (w) = cos if (1 t)w  jwj  (1 + t)w (5) j j j 0 otherwise Entropy 2022, 24, 1322 7 of 22 1 if (1 + t)w  jwj  (1 t)w > j j+1 h i pg(t,w ) j+1 cos if (1 t)w  jwj  (1 + t)w j+1 j+1 h i n (w) = (6) pg(t,w ) sin if (1 t)w  jwj  (1 + t)w > j j 0 otherwise h i jwj(1t)w where, g(t, w ) = d . To obtain the tight frames parameter, d and t are 2tw defined as >0 if x  0 d(x) = (7) and d(x) + d(1 x) = 1 8 x 2 [0, 1] 1 if x  1 " # w w j+1 j t < min (8) w + w j+1 j 4. Filtering: Expression for detailed coefficients from the EWT filter bank for the analyzed signal a(m) is given by [45], D (j, n) = a(m)n (m n)dm (9) a,n The approximate coefficients from the EWT filter bank are computed by [45], A (0, n) = a(m)F (m n)dm (10) a,F th The j empirical mode is obtained by convolving the wavelet function with the detail coefficients, where j = 1, 2, , M are different empirical modes. Original signal can be reconstructed by adding all M reconstructed modes and one low-frequency component; mathematically, both are expressed as below [45] r (n) = D (j, n) ? n (n) (11) j a,n j r (n) = A (0, n) ? F (n) (12) 0 a,F 1 where ? denotes the convolution operation. 2.4. Feature Extraction In order to understand and quantify information associated with any dynamically changing phenomenon, entropy is the most widely used parameter [46]. Thus, in order to understand the nonstationary behavior of physiological signals considered in this study, entropies, as a feature set, are considered. Some of the advantages of FBSE representation are that Bessel functions decay with time, and so provide more suitable representation of the nonstationary signal, and that FBSE has a frequency resolution twice that of the Fourier spectrum. In addition, there are many previous studies on emotion recognition where entropies have been used [26,28,42,47]. SSE, WE, and LEE are defined as follows: 2.4.1. SSE Uniformity in the distribution of signal energy can be measured using spectral entropy. Entropy measures the uncertainty, which has been derived from the Shannon’s expression. The SSE is defined based on the energy spectrum E obtained using FBSE. The energy th corresponding to the i order FB coefficient K(i) is defined mathematically as [48] 2 2 2 K(i) S [ J (b )] 1 i E = (13) 2 Entropy 2022, 24, 1322 8 of 22 The SSE is expressed mathematically as [49] H = P(i) log (P(i)) (14) SSE å i=1 where S is the total number of FBSE coefficients. P(i) is the normalized energy distribution th over i order is mathematically defined as, P(i) = (15) å i i=1 2.4.2. WE It is another measure of flatness in the distribution of signal spectral power. The WE is also called the measure of spectral flatness. It is calculated by taking the ratio of geometric mean to arithmetic mean of the energy spectrum (E ) of the FB coefficient for order i and is expressed as [50], Õ i i=1 H = S (16) WE å i i=1 WE is a unitless quantity; its output is purely a numerical value ranging from 0 to 1, where 1 represents a uniform (flat) spectrum and 0 indicates a pure tone. 2.4.3. LEE Another variant of information measurement using LEE is defined as logarithm of P(i) in the FB domain and it is given by [51] H = log (P(i)) (17) LE i=1 2.5. Feature Smoothing The brain is the control center of all human activities and internal functioning. The typical EEG signal obtained is thus a combination of various brain activity and other noise, either related to the environment or body [52]. Rapid fluctuations in feature value may arise from these noises. Human emotion changes gradually [26]; in order to reduce the effect of such variation on the emotional state-related feature values, a moving average filter with a window size of 3 samples is utilized. The effect of the moving average filter on the raw feature value of the first channel’s first epoch can be seen in Figure 4. 2.6. Classifiers Feature vectors extracted from the EEG signals are used for the classification of dif- ferent emotions. Based on the feature vector, the classifier will discriminate the data into high and low dimensions of emotion, i.e., either HV or LV, HA or LA, and HD or LD. In this study, SVM with the cubic kernel and KNN are used independently for the purpose of classification. SVM is a supervised machine learning algorithm that classifies data by first learning from labeled data belonging to different classes. The main principle behind the classifier working is finding decision boundaries that are formed by a hyperplane, and it helps in separating data into two separate classes. The hyperplane is constructed by training the classifier with sample data. The optimum location of the hyperplane or decision boundary depends on the support vectors, which are the points nearest to the decision boundary [53]. Entropy 2022, 24, 1322 9 of 22 SVM iteratively optimizes the location of hyperplane in order to maximize the margin. A margin is a total separation between two classes. SVM can be linear or nonlinear classifier depending on kernels used. Generally, in the case of a linear SVM straight line, flat plane, or an N-dimensional hyperplane are the simplest way of separating data into two groups, but there are certain situations where nonlinear boundary separates the data more efficiently. In case of a nonlinear classifier, different kernels can be polynomial, hyperbolic tangent function, or Gaussian radial basis function. In this work, an SVM classifier with a cubic kernel is used. Generalized mathematical expression for defining the hyperplane of the SVM classifier can be expressed as follows [54]: " # f (x) = sign b f K(x, x ) + c (18) å i i i i=1 where b is a positive real constant, R is total number of observations, c is a real con- stant, K(x, x ) is a kernel or feature space, x input vector, and output vector is denoted i i by f . For a linear feature space K(x, x ) = x x, for polynomial SVM of any order p, i i T p K(x, x ) = (x x + 1) defines the feature space, i.e., ( p = 2) for quadratic polynomial and ( p = 3) for cubic polynomial. SSE feature Smoothed SSE feature 3.5 2.5 10 20 30 40 50 60 10 20 30 40 50 60 WE feature Smoothed WE feature 0.02 0.03 0.015 0.02 0.01 0.01 0.005 10 20 30 40 50 60 10 20 30 40 50 60 LEE feature Smoothed LEE feature 2600 2100 10 20 30 40 50 60 10 20 30 40 50 60 Epochs Epochs Figure 4. Feature values obtained from signal epochs is shown on the left and its smoothed version is shown on the right. The KNN is a non-parametric, supervised machine learning algorithm used for data classification and regression [55]. It categorizes data points into different groups based on the distance from some of the nearest neighbors. For classifying any particular training data, the KNN algorithm follows these steps: Step 1: Compute distance between sample data and other sample using anyone of the distance metrics such as Euclidean, Mahalanobis, or Minkowski distance. Step 2: Rearrange the distant metric obtained from the first step in ascending order and top k values are considered with distance from current sample is minimum. Feature values Entropy 2022, 24, 1322 10 of 22 Step 3: Class is assigned to the sample data depending on the maximum number of nearest neighbors class. 3. Results In order to evaluate the performance of our proposed methodology, a publicly avail- able DREAMER dataset is used [32]. In this section, the results obtained after applying the proposed emotion identification method over the DREAMER dataset is discussed in detail. The raw EEG data are segmented into an epoch of one second. An EEG epoch length is 128 samples and the ECG length is 256 samples. These epochs of the EEG signal are de- composed into four modes using FBSE-EWT, as shown in Figure 5, and the corresponding filter bank is shown in Figure 6. The decomposition of EEG and ECG signal epochs gives a minimum of four number of modes. So, we set the required number of modes to four to obtain a uniform number of features across different observations. From each mode, three features are extracted. The EEG data consist of 14 channels; therefore, the dimension of the feature vector is 168 (= 14 3 4). Similarly, from the two channels of the ECG, we obtain a 24-element feature vector. The number of trials for each participant is 18 and there are a total of 23 participants, which gives us a total of 24,840 (= 60 18 23) observations or epochs. The final dimensions of the feature matrices are 24, 840 168 and 24, 840 24 for EEG and ECG, respectively. For the multimodal case, both the feature from EEG and ECG are combined, where we obtain a feature matrix with the dimension of 24, 840 192. This feature matrix is classified using two classifiers: SVM and KNN. For KNN (k = 1), Euclidean distance is used as a distance metric; for SVM, cubic kernels are used. EEG epoch ECG epoch -100 -150 Mode 1 Mode 1 -100 -50 Mode 2 Mode 2 0 0 -20 -50 Mode 3 Mode 3 -50 -50 Mode 4 Mode 4 0 0 -20 -20 20 40 60 80 100 120 50 100 150 200 250 Sample number Sample number (a) (b) Figure 5. FBSEEWTbased signal decomposition of (a) EEG and (b) ECG epochs into different modes. Performance was evaluated using the classifier learner application present in MATLAB 2022a. The system used for computing has an Intel i7 CPU with 8 GB of RAM. For three different dimensions of emotion, i.e., arousal, dominance, and valence, three different binary classifications are performed. Thus, each classifier groups data into either high or low arousal, high or low dominance, and high or low valence classes. Low and high class is Amplitude Entropy 2022, 24, 1322 11 of 22 decided based on the rating given by the participant on a scale of 1 to 5, with 3 as threshold, i.e., rating 6 3 is consider low and all above it is considered as high [33,34], due to which, unbalanced classes are obtained for some participants [32]. Further, the performance of the proposed method is evaluated using the first features obtained from EEG signals, then ECG signals, and then using the combined features. FBSE-EWT filter-bank FBSE-EWT filter-bank 3 3 2 2 1 1 0 0 0 20 40 60 0 50 100 Frequency (Hz) Frequency (Hz) Figure 6. FBSEEWTbased filter bank used for decomposing (a) EEG and (b) ECG epochs into different modes (filter corresponding to different modes are shown in different colors). Features obtained from EEG and ECG signals of different subjects are combined together and the classifiers are trained and tested subject independently. Both SVM and KNN are evaluated independently using five-fold cross-validation, where different signal feature observations are placed randomly independent to the subject. For the EEG feature and arousal, the dimension accuracy obtained for SVM is 81.53% and for KNN it is 97.50%. Similarly, performance was evaluated for the case of dominance and valence, which are summarized in Tables 1–3. The tables consist of classification results obtained by using different modalities, i.e., EEG, ECG, and the combined multimodal approach, and the best results obtained are highlighted using bold fonts. The confusion matrices for three different emotion dimensions classifications based on EEG, ECG, and multimodal signals are shown in Figures 7–9. Table 1. Performance with different modalities for arousal emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.50 97.89 97.01 97.67 97.78 (k = 1) EEG SVM 81.53 82.04 80.81 86.02 83.98 (Cubic) KNN 69.94 73.17 65.73 73.56 73.37 (k = 1) ECG SVM 63.84 65.09 61.35 77.11 70.59 (Cubic) KNN 97.84 98.10 97.51 98.06 98.08 (k = 1) Multimodal SVM 84.54 84.75 84.24 88.45 86.56 (Cubic) * in percent. Accuracy parameters, such as sensitivity, specificity, precision, and F1 score, are shown in Tables 1–3. From these tables, it may be observed that the accuracy obtained from EEG and multimodal are almost similar; however, among the two, multimodal is still the winner as it can be seen that all other parameters of classifier reliability check are having sightly better result than the unimodal EEG signal. Following mathematical expressions have been used for calculating parameters namely sensitivity, specificity, precision, and F1 score [53]: TP Sensitivity = (19) TP + FN Magnitude Magnitude Entropy 2022, 24, 1322 12 of 22 TN Specificity = (20) FP + TN TP Precision = (21) TP + FP 2TP F1 Score = (22) 2TP + FP + FN where TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative. Table 2. Performance with different modalities for dominance emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.68 97.89 97.44 97.63 97.76 (k = 1) EEG SVM 81.53 82.04 80.81 86.02 83.98 (Cubic) KNN 69.25 70.33 68.08 70.57 70.45 (k = 1) ECG SVM 62.30 62.90 61.56 66.81 64.80 (Cubic) KNN 97.91 97.95 97.86 98.02 97.99 (k = 1) Multimodal SVM 84.12 83.74 84.55 86.15 84.93 (Cubic) * in percent. Table 3. Performance with different modalities for valence emotion dimension. Modality Model Accuracy * Sensitivity * Specificity * Precision * F1 Score * KNN 97.55 97.97 96.89 97.98 97.98 (k = 1) EEG SVM 81.49 82.03 80.47 88.97 85.36 (Cubic) KNN 68.82 74.59 60.22 73.69 74.13 (k = 1) ECG SVM 63.31 66.03 55.28 81.31 72.88 (Cubic) KNN 98.17 98.23 97.86 98.29 97.19 (k = 1) Multimodal SVM 84.10 84.77 82.90 89.93 87.28 (Cubic) * in percent. Entropy 2022, 24, 1322 13 of 22 EEG (KNN) EEG (SVM) 0 13,654 326 0 12,026 1954 10,566 1 294 1 2633 8227 0 1 0 1 Predicted class Predicted class ECG (KNN) ECG (SVM) 0 10,284 3696 0 10,780 3200 1 3770 7090 1 5781 5079 0 1 0 1 Predicted class Predicted class Multimodal (SVM) Multimodal (KNN) 0 12,365 1615 0 13,709 271 1 2225 8635 10,594 1 266 0 1 0 1 Predicted class Predicted class Figure 7. Confusion matrix for arousal emotion dimension. True class True class True class True class True class True class Entropy 2022, 24, 1322 14 of 22 EEG (KNN) EEG (SVM) 0 12,594 306 0 12,026 1954 1 271 11,669 1 2633 8227 0 1 0 1 Predicted class Predicted class ECG (KNN) ECG (SVM) 0 9103 3797 0 8619 4281 1 3841 8099 1 5083 6857 0 1 0 1 Predicted class Predicted class Multimodal (KNN) Multimodal (SVM) 0 12,645 255 0 11,113 1787 11,676 1 264 1 2158 9782 0 1 0 1 Predicted class Predicted class Figure 8. Confusion matrix for dominance emotion dimension. True class True class True class True class True class True class Entropy 2022, 24, 1322 15 of 22 EEG (KNN) EEG (SVM) 0 14,756 304 0 13,399 1661 1 305 9475 1 2936 6844 0 1 0 1 Predicted class Predicted class ECG (SVM) ECG (KNN) 0 11,097 3963 0 12,246 2814 1 3781 5999 1 6301 3479 0 1 0 1 Predicted class Predicted class Multimodal (KNN) Multimodal (SVM) 0 14,785 275 0 13,544 1516 1 257 9523 1 2433 7347 0 1 0 1 Predicted class Predicted class Figure 9. Confusion matrix for valence emotion dimension. 4. Discussion Although, in recent years, significant research has been carried out on emotion recogni- tion related topics, still, it is challenging due to the fuzziness in distinction among different emotions. The study presented in this article demonstrated an approach for developing an automated human emotion identification that is able to categorize the three dimensions of emotion, i.e., arousal, valence, and dominance, into high or low classes of a particular dimension. In order to retrieve emotion-related information from the physiological signals, the FBSE-EWT signal decomposition method is used. Then, various FB-based entropies, such as SSE, WE, and LEE features, are computed from the modes obtained after signal decomposition. Figures 10 and 11 show different boxplots of mentioned entropy features obtained from EEG and ECG, respectively. The plots give information related to feature value distribution among different classes. Due to difference in the interquartile distance among different feature values good classification accuracy is obtained. Then, the extracted True class True class True class True class True class True class Entropy 2022, 24, 1322 16 of 22 features are fed to two different classifiers: SVM and KNN. The above results show that the KNN classifier is found to have more accuracy than the SVM classifier. Further, the multi-model scenario is found to be more beneficial to reliably identify the emotional states of the subject. A statistical significance test was performed to show that the accu- racy improvement in the multimodal emotion recognition model is significant [56]. An analysis of variance (ANOVA) test was performed on the 5-fold accuracies of different modalities, such as EEG, ECG, and multimodal. Figure 12 shows the accuracies of the high and low states of arousal, dominance, and valance using boxplots. The p-values 18 21 for the statistical test for arousal, dominance, and valance are 1.33 10 , 2.82 10 , and 1.50 10 , respectively, which indicates a significant improvement in accuracies for the multimodal emotion detection model. The proposed methodology is compared with the various existing approaches to identify the emotional states. The proposed method found to have superior performance as highlighted in Table 4 using bold fonts. In [57], various features related to EEG and ECG signals, such as power spectral density (PSD), HRV, entropy, EEG-topographical image-based, and ECG spectrogram image-based DL features, are calculated. Topic et al. [58] used EEG signals for emotion detection and has derived holographic features and for maximizing model accuracy, only relevant channels are selected. In [59], using data preprocessing, frame level features are derived from the EEG signal, from which, effective features are extracted using a teacher–student frame- work. In [60], the deep canonical correlation analysis (DCCA) method is proposed for emotion recognition. From the table, it may be noted that currently, various studies are being performed for emotion detection based on a DL-related framework [34,58,60–62] as it automatically finds the most significant features; however, it is difficult to understand or to find the reason behind the results obtained. Moreover, DL models are computation- ally complex and require a large amount of data to train [28]. Feature extraction in other existing state-of-the-art methods is a complex process and less accurate than the study presented here, which differentiates itself from the other existing studies as the results are encouraging. Thus, the proposed method has several advantages over the various listed DL-based feature extraction methods, such as being easy to comprehend, less complex, and more reliable. The disadvantage of the proposed multimodal approach is that its time complexity has increased compared to a single EEG-only modality. Still, the increased reliability makes it more suitable for feature extraction. As the results obtained using the proposed methodology have only been tested on a small size dataset, in the future, they can be tested on a larger database in order to verify their validity. Furthermore, in this study, a fixed 60 s duration physiological signals was used; this can also be made adaptive in order to select the time portion, which can make the methodology more efficient and adaptive. Entropy 2022, 24, 1322 17 of 22 Table 4. Comparison of existing methodologies applied on DREAMER dataset with the proposed emotion recognition method. Accuracy (%) Authors (Year) Methodology Modality LA-HA LD-HD LV-HV EEG & ECG 62.32 61.84 62.49 Katsigiannis et al. [32] (2018) PSD features and SVM classifier Song et al. [62] (2018) DGCNN EEG 84.54 85.02 86.23 Zhang et al. [34] (2019) PSD features and GCB-net classifier EEG 89.32 89.20 86.99 Bhattacharyya et al. [28] (2020) MFBSE-EWT based entropy features and ARF classifier EEG 85.4 86.2 84.5 Cui et al. [61] (2020) RACNN EEG 97.01 - 95.55 Kamble et al. [63] (2021) DWT, EMD-based features, and CML and EML-based classifier EEG 93.79 - 94.5 Li et al. [64] (2021) 3DFR-DFCN EEG 75.97 85.14 82.68 Siddharth et al. [57] (2022) PSD, HRV, entropy, DL based feature, and LSTM based classifier EEG & ECG 79.95 - 79.95 Topic et al. [58] (2022) Holographic features and CNN EEG 92.92 92.97 90.76 Gu et al. [59] (2022) FLTSDP EEG 90.61 91.00 91.54 Liu et al. [60] (2022) DCCA EEG & ECG 89.00 90.7 90.6 Proposed work FBSE-EWT-based entropy features and KNN classifier EEG & ECG 97.84 97.91 97.86 PSD = power spectral density, SVM = support-vector machines, DGCNN = dynamic graph convolutional neural network, GCB-net = graph convolutional broad network, MFBSE-EWT = multivariate FBSE-EWT, ARF = adaptive random forest, RACNN = region-based convolutional neural network, DWT = discrete wavelet transform, EMD = empirical mode decom- position, CML = conventional machine learning, 3DFR-DFCN = 3-D feature representation and dilated fully convolutional network, HRV = heart rate variability, DL = deep learning, LSTM = long short term memory, CNN = convolutional neural network, FLTSDP = frame level teacher-student framework with data privacy, DCCA = deep canonical correlation analysis. 0.1 5.5 5.5 2200 2200 5 5 2100 2100 0.08 4.5 4.5 4 4 1900 1900 0.06 3.5 3.5 1800 1800 3 1800 3 0.04 1700 1700 2.5 2.5 0.02 2 2 1.5 1.5 1400 1400 HA LA HA LA HA LA HD LD HD LD HD LD Emotion class Emotion class Emotion class Emotion class Emotion class Emotion class 0.1 4.5 0.08 3.5 0.06 0.04 2.5 0.02 1.5 HV LV HV LV HV LV Emotion class Emotion class Emotion class Figure 10. Box plots for the features calculated using EEG signals. SSE LEE SSE WE LEE SSE WE LEE WE Entropy 2022, 24, 1322 18 of 22 0.18 0.18 0.16 0.16 0.14 0.14 3800 0.12 0.12 0.1 0.1 3400 3400 0.08 0.08 3200 0.06 3200 0.06 0.04 3000 0.04 0.02 0.02 0 0 HA LA HA LA HA LA HD LD HD LD HD LD Emotion class Emotion class Emotion class Emotion class Emotion class Emotion class 6 4.5 3.5 2.5 1.5 0.5 HV LV HV LV HV LV Emotion class Emotion class Emotion class Figure 11. Box plots for the features calculated using ECG signals. 90 90 80 80 75 75 70 70 (c) (a) (b) Figure 12. Box plots showing statistical significance of accuracies for different modalities (a) arousal, (b) dominance, and (c) valence. 5. Conclusions This study presents an automated emotion identification by a multimodal approach using FB domain-based entropies. The publically available DREAMER data set was used to evaluate the performance of the proposed algorithm. Physiological signals obtained from the dataset, namely EEG and ECG, are decomposed using the FBSE-EWT into four modes. EEG ECG Multimodal EEG ECG Multimodal EEG ECG Multimodal SSE Accuracy (in %) LEE SSE WE Accuracy (in %) LEE SSE WE Accuracy (in %) LEE WE Entropy 2022, 24, 1322 19 of 22 From these modes, several new FB-based entropy features, such as FB spectral-based SSE, LEE, and WE were computed. The dataset consisted of three-dimensional emotional space, i.e., arousal, valence, and dominance. Each of the dimensions are categorized into high and low classes based on the rating provided along with the dataset. The proposed method using the KNN classifier provides the highest accuracies of 97.84%, 97.91%, and 97.86% for arousal, valence, and dominance emotion classes, respectively. After comparing it with results obtained from current methods, significant improvement in the results is obtained. Moreover, the multimodal approach is found to be the most accurate in terms of identifying human emotional states. Author Contributions: A.N., K.D. and R.B.P. contributed equally to this work. All authors have read and agreed to the published version of the manuscript. Funding: This study is supported by the Council of Scientific & Industrial Research (CSIR) funded Research Project, Government of India, Grant number: 22(0851)/20/EMR-II. Institutional Review Board Statement: Not applicable. Data Availability Statement: The EEG and ECG data are provided by the Stamos Katsigiannis collected at University of the West of Scotland. Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: ANN Artificial neural network ANOVA Analysis of variance ANS Autonomous nervous system ARF Autoencoder-based random forest BEMD Bivariate empirical mode decomposition DCCA Deep canonical correlation analysis DL Deep learning DWT Discrete wavelet transform ECG Electrocardiogram EEG Electroencephalogram EMD Empirical mode decomposition EWT Empirical wavelet transform FB Fourier–Bessel FBSE Fourier–Bessel series expansion FBSE-EWT FBSE-based EWT FN False negative FP False positive FrFT Fractional Fourier transform GSR Galvanic skin response HA High arousal HCI Human–computer interaction HD High dominance HOC Higher-order crossing HRV Heart rate variability HV High valence ICA Independent component analysis IMF Intrinsic mode functions Entropy 2022, 24, 1322 20 of 22 IP Information potential KNN K-nearest neighbors LA Low arousal LA-LL Left arm and left leg LD Low dominance LEE Log energy entropy LV Low valence MC-LS-SVM Multiclass least squares support vector machines MHMS Multivariate Hilbert marginal spectrum MSST Multivariate synchrosqueezing transform NCA Neighborhood component analysis NMF Non-negative matrix factorization PCA Principal component analysis PSD Power spectral density RA-LL Right arm and Left leg SDEL Sparse discriminative ensemble learning SODP Second order difference plot SSE Shannon spectral entropy STFT Short time Fourier transform SVM Support vector machines TN True negative TP True positive TQWT Tunable Q-factor wavelet transform WE Wiener entropy WIB With-in beat References 1. Ptaszynski, M.; Dybala, P.; Shi, W.; Rzepka, R.; Araki, K. Towards context aware emotional intelligence in machines: Computing contextual appropriateness of affective states. In Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, CA, USA, 11–17 July 2009; AAAI: Menlo Park, CA, USA; pp. 1469–1474. 2. Vingerhoets, A.; Nyklícek, I.; Denollet, J. Emotion Regulation; Springer: Berlin/Heidelberg, Germany, 2008. 3. Kroupi, E.; Yazdani, A.; Ebrahimi, T. EEG correlates of different emotional states elicited during watching music videos. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA, 9–12 October 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 457–466. 4. Jin, L.; Kim, E.Y. Interpretable cross-subject EEG-based emotion recognition using channel-wise features. Sensors 2020, 20, 6719. 5. Kołakowska, A.; Landowska, A.; Szwoch, M.; Szwoch, W.; Wrobel, M.R. Emotion recognition and its applications. In Human- Computer Systems Interaction: Backgrounds and Applications 3; Springer: Berlin/Heidelberg, Germany, 2014; pp. 51–62. 6. Šimic, ´ G.; Tkalci ˇ c, ´ M.; Vukic, ´ V.; Mulc, D.; Španic, ´ E.; Šagud, M.; Olucha-Bordonau, F.E.; Vukšic, ´ M.; R. Hof, P. Understanding emotions: Origins and roles of the amygdala. Biomolecules 2021, 11, 823. 7. Doukas, C.; Maglogiannis, I. Intelligent pervasive healthcare systems. In Advanced Computational Intelligence Paradigms in Healthcare-3; Springer: Berlin/Heidelberg, Germany, 2008; pp. 95–115. 8. McCraty, R. Science of the Heart: Exploring the Role of the Heart in Human Performance; HeartMath Research Center, Institute of HeartMath: Boulder Creek, CA, USA, 2015. 9. Filippini, C.; Di Crosta, A.; Palumbo, R.; Perpetuini, D.; Cardone, D.; Ceccato, I.; Di Domenico, A.; Merla, A. Automated affective computing based on bio-signals analysis and deep learning approach. Sensors 2022, 22, 1789. 10. Kipli, K.; Latip, A.A.A.; Lias, K.; Bateni, N.; Yusoff, S.M.; Tajudin, N.M.A.; Jalil, M.; Ray, K.; Shamim Kaiser, M.; Mahmud, M. GSR signals features extraction for emotion recognition. In Proceedings of Trends in Electronics and Health Informatics, Singapore, 22 March 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 329–338. 11. Dutta, S.; Mishra, B.K.; Mitra, A.; Chakraborty, A. An analysis of emotion recognition based on GSR signal. ECS Trans. 2022, 107, 12535. 12. Panahi, F.; Rashidi, S.; Sheikhani, A. Application of Fractional Fourier Transform in Feature Extraction from Electrocardiogram and Galvanic Skin Response for Emotion Recognition. Biomed. Signal Process. Control. 2021, 69, 102863. 13. Zhang, Q.; Chen, X.; Zhan, Q.; Yang, T.; Xia, S. Respiration-based emotion recognition with deep learning. Comput. Ind. 2017, 92, 84–90. 14. Agrafioti, F.; Hatzinakos, D.; Anderson, A.K. ECG pattern analysis for emotion detection. IEEE Trans. Affect. Comput. 2011, 3, 102–115. 15. Cheng, Z.; Shu, L.; Xie, J.; Chen, C.P. A novel ECG-based real-time detection method of negative emotions in wearable applications. In Proceedings of the 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), Shenzhen, China, 15–17 December 2017; IEEE: Washington, DC, USA, 2017; pp. 296–301. Entropy 2022, 24, 1322 21 of 22 16. Dissanayake, T.; Rajapaksha, Y.; Ragel, R.; Nawinne, I. An ensemble learning approach for electrocardiogram sensor based human emotion recognition. Sensors 2019, 19, 4495. 17. Sepúlveda, A.; Castillo, F.; Palma, C.; Rodriguez-Fernandez, M. Emotion recognition from ECG signals using wavelet scattering and machine learning. Appl. Sci. 2021, 11, 4945. 18. Anuragi, A.; Sisodia, D.S.; Pachori, R.B. EEG-based cross-subject emotion recognition using Fourier-Bessel series expansion based empirical wavelet transform and NCA feature selection method. Inf. Sci. 2022, 610, 508–524. 19. Sharma, R.; Pachori, R.B.; Sircar, P. Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed. Signal Process. Control. 2020, 58, 101867. 20. Bajaj, V.; Pachori, R.B. Human emotion classification from EEG signals using multiwavelet transform. In Proceedings of the 2014 International Conference on Medical Biometrics, Shenzhen, China, 30 May–1 June 2014; IEEE: Washington, DC, USA, 2014; pp. 125–130. 21. Bajaj, V.; Pachori, R.B. Detection of human emotions using features based on the multiwavelet transform of EEG signals. In Brain-Computer Interfaces; Springer: Berlin/Heidelberg, Germany, 2015; pp. 215–240. 22. Lin, Y.P.; Wang, C.H.; Jung, T.P.; Wu, T.L.; Jeng, S.K.; Duann, J.R.; Chen, J.H. EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 2010, 57, 1798–1806. 23. Liu, Y.; Sourina, O. Real-time subject-dependent EEG-based emotion recognition algorithm. In Transactions on Computational Science XXIII; Springer: Berlin/Heidelberg, Germany, 2014; pp. 199–223. 24. Mohammadi, Z.; Frounchi, J.; Amiri, M. Wavelet-based emotion recognition system using EEG signal. Neural Comput. Appl. 2017, 28, 1985–1990. 25. Mert, A.; Akan, A. Emotion recognition based on time-frequency distribution of EEG signals using multivariate synchrosqueezing transform. Digit. Signal Process. 2018, 81, 106–115. 26. Gupta, V.; Chopda, M.D.; Pachori, R.B. Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sens. J. 2018, 19, 2266–2274. 27. Ullah, H.; Uzair, M.; Mahmood, A.; Ullah, M.; Khan, S.D.; Cheikh, F.A. Internal emotion classification using EEG signal with sparse discriminative ensemble. IEEE Access 2019, 7, 40144–40153. 28. Bhattacharyya, A.; Tripathy, R.K.; Garg, L.; Pachori, R.B. A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sens. J. 2020, 21, 3579–3591. 29. Salankar, N.; Mishra, P.; Garg, L. Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomed. Signal Process. Control. 2021, 65, 102389. 30. Nalwaya, A.; Das, K.; Pachori, R.B. Emotion identification from TQWT-Based EEG rhythms. In AI-Enabled Smart Healthcare Using Biomedical Signals; IGI Global: Hershey, PA, USA, 2022; pp. 195–216. 31. Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 2018, 8, 13657. 32. Katsigiannis, S.; Ramzan, N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J. Biomed. Health Inform. 2017, 22, 98–107. 33. Cheng, J.; Chen, M.; Li, C.; Liu, Y.; Song, R.; Liu, A.; Chen, X. Emotion recognition from multi-channel eeg via deep forest. IEEE J. Biomed. Health Inform. 2020, 25, 453–464. 34. Zhang, T.; Wang, X.; Xu, X.; Chen, C.P. GCB-Net: Graph convolutional broad network and its application in emotion recognition. IEEE Trans. Affect. Comput. 2019, 13, 379–388. 35. Gupta, V.; Bhattacharyya, A.; Pachori, R.B. Automated identification of epileptic seizures from EEG signals using FBSE-EWT method. In Biomedical Signal Processing; Springer: Berlin/Heidelberg, Germany, 2020; pp. 157–179. 36. Anuragi, A.; Sisodia, D.S.; Pachori, R.B. Epileptic-seizure classification using phase-space representation of FBSE-EWT based EEG sub-band signals and ensemble learners. Biomed. Signal Process. Control. 2022, 71, 103138. 37. Khan, S.I.; Qaisar, S.M.; Pachori, R.B. Automated classification of valvular heart diseases using FBSE-EWT and PSR based geometrical features. Biomed. Signal Process. Control. 2022, 73, 103445. 38. Khan, S.I.; Pachori, R.B. Derived vectorcardiogram based automated detection of posterior myocardial infarction using FBSE-EWT technique. Biomed. Signal Process. Control. 2021, 70, 103051. 39. Bhattacharyya, A.; Singh, L.; Pachori, R.B. Fourier-Bessel series expansion based empirical wavelet transform for analysis of non-stationary signals. Digit. Signal Process. 2018, 78, 185–196. 40. Schroeder, J. Signal Processing via Fourier-Bessel series expansion. Digit. Signal Process. 1993, 3, 112–124. 41. Das, K.; Verma, P.; Pachori, R.B. Assessment of chanting effects using EEG signals. In Proceedings of the 2022 24th International Conference on Digital Signal Processing and Its Applications (DSPA), 30 March–1 April 2022, Moscow, Russia; IEEE: Washington, DC, USA, 2022, pp. 1–5. 42. Pachori, R.B.; Sircar, P. EEG signal analysis using FB expansion and second-order linear TVAR process. Signal Process. 2008, 88, 415–420. 43. Gilles, J.; Heal, K. A parameterless scale-space approach to find meaningful modes in histograms—Application to image and spectrum segmentation. Int. J. Wavelets Multiresolut. Inf. Process. 2014, 12, 1450044. 44. Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 1979, 9, 62–66. 45. Gilles, J. Empirical wavelet transform. IEEE Trans. Signal Process. 2013, 61, 3999–4010. Entropy 2022, 24, 1322 22 of 22 46. Kowalski, A.; Martín, M.; Plastino, A.; Rosso, O. Bandt-Pompe approach to the classical-quantum transition. Phys. D Nonlinear Phenom. 2007, 233, 21–31. 47. Zheng, W.L.; Lu, B.L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 2015, 7, 162–175. 48. Pachori, R.B.; Hewson, D.; Snoussi, H.; Duchêne, J. Analysis of center of pressure signals using empirical mode decomposition and Fourier-Bessel expansion. In Proceedings of the TENCON 2008—2008 IEEE Region 10 Conference, Hyderabad, India, 19–21 November 2008; IEEE: Washington, DC, USA, 2008; pp. 1–6. 49. Shannon, C.E. A mathematical theory of communication. ACM SIGMOB. Mob. Comput. Commun. Rev. 2001, 5, 3–55. 50. Dubnov, S. Generalization of spectral flatness measure for non-Gaussian linear processes. IEEE Signal Process. Lett. 2004, 11, 698–701. 51. Aydın, S.; Saraoglu, ˘ H.M.; Kara, S. Log energy entropy-based EEG classification with multilayer neural networks in seizure. Ann. Biomed. Eng. 2009, 37, 2626–2630. 52. Li, X.; Zhang, Y.; Tiwari, P.; Song, D.; Hu, B.; Yang, M.; Zhao, Z.; Kumar, N.; Marttinen, P. EEG based emotion recognition: A tutorial and review. ACM Comput. Surv. (CSUR) 2022. https://doi.org/10.1145/3524499. 53. Sebastiani, F. Machine learning in automated text categorization. ACM Comput. Surv. (CSUR) 2002, 34, 1–47. 54. Cortes, C.; Vapnik, V. Support-vector networks. Machine learning 1995, 20, 273–297. 55. Fukunaga, K. Introduction to Statistical Pattern Recognition, 2nd ed.; Academic Press: Cambridge, USA, 1990; pp. 300–361. 56. Muralidharan, N.; Gupta, S.; Prusty, M.R.; Tripathy, R.K. Detection of COVID19 from X-ray Images Using Multiscale Deep Convolutional Neural Network. Appl. Soft Comput. 2022, 119, 108610. 57. Siddharth, S.; Jung, T.P.; Sejnowski, T.J. Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Trans. Affect. Comput. 2019, 13, 96–107. 58. Topic, A.; Russo, M.; Stella, M.; Saric, M. Emotion recognition using a reduced set of EEG channels based on holographic feature maps. Sensors 2022, 22, 3248. 59. Gu, T.; Wang, Z.; Xu, X.; Li, D.; Yang, H.; Du, W. Frame-Level Teacher-Student Learning with Data Privacy for EEG emotion recognition. IEEE Trans. Neural Netw. Learn. Syst. 2022. https://doi.org/10.1109/TNNLS.2022.3168935. 60. Liu, W.; Qiu, J.L.; Zheng, W.L.; Lu, B.L. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Trans. Cogn. Dev. Syst. 2021, 14, 715–729. 61. Cui, H.; Liu, A.; Zhang, X.; Chen, X.; Wang, K.; Chen, X. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl.-Based Syst. 2020, 205, 106243. 62. Song, T.; Zheng, W.; Song, P.; Cui, Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 2018, 11, 532–541. 63. Kamble, K.S.; Sengupta, J. Ensemble machine learning-based affective computing for emotion recognition using dual-decomposed EEG signals. IEEE Sens. J. 2021, 22, 2496–2507. 64. Li, D.; Chai, B.; Wang, Z.; Yang, H.; Du, W. EEG emotion recognition based on 3-D feature representation and dilated fully convolutional networks. IEEE Trans. Cogn. Dev. Syst. 2021, 13, 885–897.

Journal

Entropy – Multidisciplinary Digital Publishing Institute

Published: Sep 20, 2022

Keywords: Fourier–Bessel series expansion; spectral entropy; ECG; EEG; FBSE-EWT

There are no references for this article.