{'fraction_non_alphanumeric': 0.0308536675813988, 'fraction_numerical': 0.014944745234740042, 'mean_word_length': 5.197885543553206, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unified framework for Identity and Imagined Action Recognition from EEG patterns

We present a unified deep learning framework for user identity recognition and imagined action recognition, based on electroencephalography (EEG) signals. Our solution exploits a novel phased subsampling preprocessing step as a form of data augmentation, and a mesh-to-image representation to encode the inherent local spatial relationships of multi-electrode EEG signals. The resulting image-like data is then fed to a convolutional neural network, to process the local spatial dependencies, and eventually analyzed through a Bidirectional LSTM module, to focus on temporal relationships. Our solution is compared against several methods in the state of the art, showing comparable or superior performance on different tasks. Preliminary experiments are also conducted in order to direct future works towards everyday applications relying on a reduced set of EEG electrodes.


I. INTRODUCTION
Brain-Computer Interface (BCI) is a term that refers to a wide variety of techniques and technologies, designed to establish a direct communication link between the brain (or, more generally, functional components of the central nervous system) and an external computational device. BCI, sometimes also referred to as Brain-Machine Interface (BMI), can be decisive in improving the life quality of people affected by motor-impairing disorders, but it can also be employed for entertainment purposes.
Among the various technologies that enable BCI, here we focus on the usage of Electroencephalography (EEG) signals as collected by a non-invasive mesh of electrodes to be worn on the subject's head. Brain-Computer Interfaces can be applied as a tool to transfer an imagined action (as opposed to a physical action [1], [2]) to a physical actuator, such as assistive robotic arms [3] and spherical exploration robots [4], [5]. BCI has also found application in authentication systems, where the user's brain signature can be exploited as a behavioral biometric key for identification and verification. This has the advantage of allowing authentication for motor-impaired people, or to improve the robustness as "inherent" component in multi-factor authentication systems.
In this paper we present a unique solution, based on artificial neural networks, that simultaneously addresses multiple tasks: action recognition, user identification, and user verification. Conceptually, we can envision a scenario where the same application technology, such as the aforementioned robotic arms M. Buzzelli  or exploration robots, can be both accessed and controlled with a unique Brain-Computer Interface. Our solution, illustrated in Figure 1, is based on the novel joint use of the following elements: • Phased subsampling preprocessing for data augmentation.
• Mesh-to-image representation for the exploitation of local spatial relationships. • Two-stage neural architecture composed of Convolutional for local spatial processing, followed by Bidirectional Long-Short Term Memory (Bi-LSTM) for exploitation of temporal relationships. The document is structured as follows. Section II explores the significant works from the scientific literature in terms of EEG representation and interpretation, as well as the existing datasets of EEG data. Section III describes in detail our proposed solution for action recognition and user identification/verification. Section IV defines the adopted experimental protocol and provides comments on the experimental results. Section V offers speculative considerations about future directions of this work, such as the use of a subset of electrode terminals for everyday use of BCI systems.

A. Methods for EEG representation and interpretation
The scientific literature offers several works related to the analysis of Electroencephalography (EEG) signals, starting from those specifically focused on developing features and representations that can be used for a variety of downstream applications. Bashivan et al. [6] specifically showed the importance of crafting features that are invariant to inter-and arXiv:2205.04357v1 [cs.HC] 9 May 2022 intra-subject differences, as well as to inherent noise associated with the collection of EEG data. They transformed the signals into a sequence of topology-preserving multi-spectral images. They then trained a deep recurrent convolutional network, and focused on a task of cognitive load classification for representation learning and assessment. Inspired by the described data representation, here we introduce a mesh-to-image transformation that enables a 2D convolutional processing, while reducing the potential loss of information introduced by the interpolation used in [6]. Kasabov et al. [7] argued that Spiking Neural Networks (SNN) are particularly suited to the representation of EEG and other types of spatio/spectro temporal brain data, as they are based on the same computational principle, i.e. spiking formation processing. Based on this observation, the authors defined a neural architecture (Neu-Cube) based on a three-dimensional evolving SNN, aiming at approximating the structural and functional areas of interest of the brain that are related to modeling temporal brain data. The proposed representation was presented as potentially useful for the discovery of functional pathways from data, as well as the prediction of system of future brain activities. Monsy and Vinod [8] devised a novel feature called Frequency-Weighted Power (FWP), which represents the power of a specific frequency band multiplied and integrated by its corresponding power density value. The authors evaluated this representation in the context of user identification, and hypothesized that the improvement with respect to traditional Power Spectral Density is due to FWP's dependency on a specific frequency point, as such carrying more discriminative information.
Parallel to the research within application-agnostic representations, a branch of scientific literature focused on the development of application-specific representations, such as those aimed at biometric identification. Fraschini et al. [9] presented a phase-synchronization approach aimed at recognizing and exploiting personal distinctive brain network organization. The authors band-pass-filtered the subjects EEG signals, and estimated functional connectivity using the Phase Lag Index. They then reconstructed a weighted network based on the resulting connectivity matrix, and subsequently computed the nodal eigenvector centrality, which was used to assign a level of importance of different neural regions. According to their findings, resting-state functional brain network topology appears to provide better discriminative power than using only a measure of functional connectivity. Thomas and Vinos [10] proposed a simple yet effective authentication technique based on cross-correlation values of Power Spectral Density (PSD) features of gamma band , showing its superior performance when compared to delta, theta, alpha and beta bands of the signals. La Rocca et al. [11] argued that traditional features, such as power spectrum estimation, which characterize the activity of single brain regions, ignore possible temporal dependencies between the generated signals. Based on the observation that richer features reside in the functional coupling of different brain regions, the authors proposed the fusion of spectral coherence-based connectivity between such regions. Subsequently, Yang et al. [12] identified a potential problem in using the EEG signals captured during the socalled "resting phase", due to the ambiguity of interpreta-tion by the users, leading to uncontrollable heterogeneity in the data. Additionally, the authors proposed a novel feature based on wavelet transformation to extract identity-specific information. They adopted an experimental protocol aimed at establishing the difference in performance when training and testing using data collected from different tasks. In general, it has been proven that proper data representation and encoding is crucial for successful interpretation. To this extent, we will describe our data preprocessing and representation pipeline in Section III-A, which includes encoding the spatial relationship of the sensor mesh into a three-dimensional matrix.
In more recent years, many researchers migrated to the design and development of application-specific uses of EEG data, leaving the task of feature definition to the training process of neural networks, showing remarkable results. With respect to action recognition, also referred to as motion imagery, Chen et al. [13] proposed a method that applies Multi-task Recurrent Neural Networks to learn distinctive features from EEG signals, and exploits the temporal correlation between different frequency channels to improve the recognition of imagined actions. Dose et al. [14] devised a convolutional neural network for imagined action recognition, and extended it for subject-specific adaptation. They conducted an analysis of the learned filters, to provide a level of interpretability on the predicted class. Zhang, Davoodnia et al. [15] addressed the classification of left/right hand movement by extracting preliminary features from the time and frequency domain, and subsequently processing them through a Long Short-Term Memory (LSTM) network that exploits an attention mechanism. Zhang, Yao et al. [16] developed two levels of neural processing: a convolutional (CNN) and a recurrent (RNN) neural network. These were used jointly in either a cascade or parallel setup, to explore the space of feature representations. In terms of user identification and verification, Sun et al. [17] also combined a traditional convolutional neural network (CNN) with a LSTM module to extract data-defined spatio-temporal features of the EEG signals, and showed the empirical improvement in user identification accuracy. The authors also experimented with reducing the number of electrodes to potentially lower the cost of a practical implementation of a user recognition system. Recently, Lu et al. [18] produced a comprehensive comparison study, which confirmed the effectiveness of preceding an RNN-based temporal analysis by a CNN-based preprocessing of the input signals. We leverage on this finding by designing an original neural architecture structured according to the aforementioned paradigm: jointly exploiting a convolutional backbone for data-driven preprocessing, and a Bi-LSTM head for handling temporal relationships. When combined with our phased subsampling data augmentation, we are able to fully exploit the task separation and achieve excellent recognition performance for both action recognition and user identification.

B. Datasets of EEG data
The domain of EEG signal processing is associated to a number of datasets released through the years.
In 1990, Keirn and Aunon [19] published the eponymous dataset. For data collection, the electrodes were connected through band-pass analog amplifier filters set at 0.1-100 Hz, and sampled at 250 Hz. All the data was converted with an IBM-AT controlling a Lab Master analog-to-digital converter at 12 bits of accuracy. In 1999, Beigleiter [20] released the UCI KDD (University of California Irvine, Knowledge Discovery in Databases) dataset. The data was collected within the framework of a larger study the correlation between EEG signals and the genetic predisposition to alcoholism. It contains measurements from 64 electrodes placed on the scalp of the subjects, sampled at 256 Hz for 1 second. Between 2001 and 2008, four versions of the BCI Competition (Brain-Computer Interface) have been organized [21]- [24]. With the final iteration of the challenge, several problems have been included, such as classifying signals affected by eye movement artifacts, or the direction of wrist movements based on magneto-encephalography, and the associated dataset are characterized by a wide variety of bands and sampling rates. In 2005 Hunter et al. [25] published the Australian EEG Database: a web-based dataset of 18,500 EEG records from a regional public hospital throughout a time span of 11 years. A notable feature of this dataset is the variety of users, ranging in age from a premature infant born at 24 weeks gestation, up to 90 year old people. In 2012 Koelstra et al. [26] released DEAP: a Database for Emotion Analysis using Physiological signals. Both the EEG and peripheral physiological signals of 32 participants were recorded while watching 40 one-minute long excerpts of music videos. In addition to the recorded signals, user ratings and frontal video recording were also collected.
The data used in this research comes from the PhysioNet dataset [27], [28] for motor movement/imagery (MMI), collected and processed using the BCI2000 software system. The EEG data are recorded using 64 electrodes, with a sampling rate of 160Hz, and positioned according to the international 10-10 system (excluding electrodes Nz, F9, F10, FT9, FT10, A1, A2, TP9, TP10, P9, and P10). The dataset was collected by having 109 human subjects perform a number of guided operations, which involve both the real and imagined movement of body parts in response to visual stimuli. The EEG activity recorded during these sessions is then assembled in different ways, to define the various recognition tasks addressed in this paper: action recognition, user identification, and user verification. The data collection sessions were structured as follows. Each of the 109 subjects performed 14 so-called "runs": a one-minute baseline run with open eyes, a oneminute baseline run with closed eyes, and three run cycles of the following four tasks:

RECOGNITION
Based on EEG data relative to motor imagery, we propose a unique solution capable of performing three tasks: action recognition, user identification, and user verification. The first two tasks are pure classification problems, while the third task can be described as a metric learning problem and it is derived from the second classification model (user identification). Additional details on the experimental protocol are provided in Section IV-A.

A. Data preprocessing and representation
The dataset is first preprocessed with a 1Hz high-pass filter, in order to reduce noise, the negative effect of signal artifacts, and power-line interference.
The filtered signals are segmented with a non-overlapping sliding window that spans 80 samples. For example, 1 second of data, corresponding to 160 samples, is split into two 0.5s windows, composed of 80 samples each.
Phased subsampling preprocessing is then applied on segmented data as a form of data augmentation, by sampling each segment multiple times at fixed interval with varying phase. Let: be a segment S composed of M samples. The corresponding data augmentation produces N sub-segments S (n) : Preliminary experiments showed that the joint use of segmentation and subsampling-based data augmentation with N = 4 allows for improved performance in all tasks. As an example, the described procedure leads to 20-samples-long data sub-segments extracted from 0.5 seconds of raw signal, as visualized in Figure 2.
Z-score standardization is then applied to the information from each electrode considered individually, by rescaling the data distribution so that the mean of observed values is 0 and the standard deviation is 1.
Finally, the relative position of the electrodes is encoded in a matrix representation that preserves the three-dimensional nature of the underlying data, composed of 10 rows, 11 columns, and 20 channels. The rows and column directly encode the spatial distribution of the electrodes on the user's scalp, following the mapping visualized in Figure 3, whereas the channels encode the temporal information after the segmentation and subsampling procedure previously described. It can be noted that our mesh-to-image mapping involves a discrete relocation of the exact input values [16], [18]. Differently, the approach originally introduced by Bashivan et al. [6] consists in a continuous projection from a 3D surface to a 2D image, relying on values interpolation and thus possibly causing loss of information.

B. Neural architecture
We define a hybrid neural model composed of a Convolutional Neural Network (CNN) that processes the input 3D mesh with a pyramid representation, followed by a Recurrent Neural Network (RNN) to specifically exploit the temporal component of the input signal, using a Bidirectional Long-Short Term Memory (B-LSTM) module. The overall structure is illustrated in Table I.
The data dimensionality is represented as (temporal index, mesh row, mesh column, channel), and the input size (20, 10, 11, 1) is obtained by reordering the 3D matrix representation described in Section III-A. A series of four convolutional blocks is then defined to sequentially process the data, by enlarging the channel dimension without restricting the spatial dimensions. Each block is composed of: 1) A time-distributed convolutional layer, which applies convolution independently to each element defined in the first dimension. 2) A batch normalization layer, which stabilizes the learning procedure by standardizing the input activations. 3) An Exponential Linear Unit (ELU) activation layer, which applies a smooth transition to handle negative inputs, compared to the traditional Rectified Linear Unit. The resulting activation is flattened, while still preserving the temporal dimension, into a single high-dimensionality vector (28160 elements per temporal index). This is then processed with a sequence of fully-connected layer, dropout, batch normalization, ELU activation, thus producing a 1024long vector for each temporal index.
The temporal component is finally exploited and processed through a Bi-directional LSTM, whose hidden layer is internally duplicated and used to handle two versions of the input data: in forward and reverse temporal order. The resulting activation, a 256-long vector, is eventually mapped to the problem size (5 output classes for action recognition, or 109 classes for user identification) by means of one last fullyconnected layer. The learning process is guided by a softmax cross entropy loss, comparing the output vector with the index class provided as ground truth annotation. The network is trained for a total of 80 epochs, with 0.0001 learning rate, and batch size equal to 32.

A. Experimental protocol
Three macro tasks have been defined on the basis of the PhysioNet EEG data recorded during the actions described in Section II-B: 1) Action recognition. The five action classes of interest are: • Resting state with closed eyes • Imagined motion of the left fist • Imagined motion of the right fist • Imagined motion of both fists • Imagined motion of both feet Experiments related to action recognition are divided into two sub categories: a) Inter-subject action recognition. 103 out of the 109 available users are selected, excluding 6 subjects with corrupted data [15], [29], [30]. A 10-fold cross validation is applied by randomly sampling the entire dataset, thus selecting at each fold 90% of the data for training, and 10% for test. Temporal windows spanning 1 second of signals have been used. b) Intra-subject action recognition. 20 users are selected. Each user is evaluated independently, splitting their data as 90% for training and 10% for test. Temporal windows spanning 0.25 seconds of signals have been used, in order to virtually increase the available data for each user.
The 109 human subjects are interpreted as classes in a user classification task. The data is randomly split as 75% for training, 25% for validation, and 10% for test, following the experimental setup from existing works in the state of the art [17]. Temporal windows spanning 1 second of signals have been used.

3) User verification.
We optimize a threshold to determine whether any two "bio-signature" signals belong to the same human subject. The signals are described in terms of 256dimensional neural features using the model trained for user identification, and each neural-encoded signal is compared against all other signals using the Euclidean distance. The resulting distance matrix is used to compute a Receiver Operating Characteristic curve (considering a valid match when the most similar feature corresponds to the same user) and to consequently evaluate the Equal Error Rate as the threshold selection criterion. We experimentally verify how the system performance depends on the gesture being imagined, as well as the impact of having an unknown subject at test time, for a total of four experimental combinations: a) Gesture-independent, known users (GI-KU): The user classification model used for feature extraction is trained on data from all 109 users, using all imagined gestures indiscriminately. b) Gesture-independent, unknown users (GI-UU): The model is trained on data from 99 users, using all imagined gestures indiscriminately. The distance-based verification is performed on the remaining 10 users. c) Gesture-dependent, known users (GD-KU): The model is trained on data from all 109 users, considering the five gestures one at a time. The   TABLE II  QUANTITATIVE EVALUATION FOR THE TASK OF 5-CLASS ACTION   RECOGNITION, EVALUATED IN THE INTER-SUBJECT SCENARIO, WHERE  MULTIPLE USERS ARE CONSIDERED SIMULTANEOUSLY AT TRAINING AND  TEST TIME [6] 67.31% --5 108 Kasabov 2014 [7] 80.64% --5 108 results from the five gestures are eventually averaged. d) Gesture-dependent, unknown users (GD-UU): The model is trained on data from 99 users, considering the five gestures one at a time. The distancebased verification is performed on the remaining 10 users. The results from the five gestures are eventually averaged. In all cases, temporal windows spanning 1 second of signal have been used.

B. Results
The results for the action recognition task are reported in Table II for inter-subject evaluation and in Table III for intra-subject evaluation. The accuracy for intra-subject action recognition is expectedly higher, confirming the importance of subject-specific features. Nonetheless, the more challenging task of inter-subject action recognition produces excellent performance, reaching 93.89% accuracy, 95.90% precision and 91.80% recall. The performance of several other methods are also reported, although it should be noted that the extreme heterogeneity in experimental conditions, despite all methods being evaluated on the same dataset, does not allow for a direct comparison. To this extent, the number of classes of interest for action recognition, as well as the number of involved used are reported in table for reference. For example, the best configurations analyzed in the study by Lu et al. [18] return a higher accuracy for inter-subject action recognition, which is however evaluated on a limited subset of 20 users.
The results for user identification and for user verification are presented in Table V and Table IV respectively, with high accuracy and low Equal Error Rates (EER). As expected,  [35] 98.55% 100 Yang et al. 2018 [12] 99.00% 108 Singh et al. 2015 [36] 100.00% 109 La Rocca et al. 2014 [11] 100.00% 108 the EER is higher when testing on unknown users (UU as compared to KU), and it is also higher when assessing a gesture-independent setup (GI as compared to GD), corroborating the findings of the experiments in action recognition.
In terms of comparison with existing methods, note that there is a small intersection of methods that were tested for both user verification and user identification, namely the solutions by Yang et al. [12], Sun et al. [17], and Monsy et al. [8]. The considerations about heterogeneity of experimental conditions, however, also apply in this context, where different methods applied a different selection and organization of the data coming from the PhysioNet dataset. In terms of user verification, specifically, we classified the existing methods based on the combination described in Section IV-A, related to gesture independent/dependent (GI/GD) assessment over known/unknown users (KU/UU). Nonetheless, the underlying set of actions highlight how a direct comparison is not possible.

DEVELOPMENTS
In this section we investigate the possibility of reducing the available number of electrodes for action and user classification, strategically positioned in locations that can be potentially accessed by wearable devices, such as smart headbands, smart hats, or smart glasses. We considered the following configurations, illustrated in Figure 4: All these experiments were run with a temporal window spanning 0.5 seconds. Figure 5 presents a visualization of the decline in performance that results from reducing the number of electrodes, starting from the total 64 in the "Full" configuration. A good compromise between number of electrodes and accuracy of both tasks, appears to be the EEGlass 4 configuration, achievable with a smart headband or hat. When tested for user verification, this particular configuration yields an Equal Error Rate of 21.89% for the Gesture-Dependent/Known-User experiment, and 27.63% for the Gesture-Independent/Unknown-User experiment, thus one to two orders of magnitude higher than the 64-electrode configuration. This decline in performance suggests that the   electrodes-subset configuration is not ideal for user identification, but further investigation is left for future works.

VI. CONCLUSION
We have presented a unified neural framework for action recognition, user identification and user verification, based on data collected from electroencephalography signals. Our solution exploits a phased subsampling preprocessing to synthetically increase the available training data, and relies on a mesh-to-image representation to encode the electrodes pattern into an image-like data structure, which is first processed by a convolutional neural architecture, and finally analyzed by a bidirectional LSTM. The proposed method appears to be effective both in action recognition and user identification, achieving accuracy always well above 90%. A comparison with methods from the state of the art confirms the validity of these results, although a direct comparison is not applicable due to the spread heterogeneity of testing configurations. User verification has been evaluated in four different setups, in order to provide a full-spectrum baseline for future comparisons, ranging from 0.39% Equal Error Rate, up to 6.16%.
To provide a direction for future developments, we have presented a preliminary investigative speculation using a sub- set of the available electrodes. The results provide a first broad picture of the decline in performance that stems from manually-selected electrodes configuration. In the future, we will consider genetic algorithms and other meta heuristics [43] for the automated selection of electrodes subsets.