{'fraction_non_alphanumeric': 0.028728933690344936, 'fraction_numerical': 0.00809576311230115, 'mean_word_length': 5.160683097224918, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bridging between Cognitive Processing Signals and Linguistic Features via a Unified Attentional Network

Cognitive processing signals can be used to improve natural language processing (NLP) tasks. However, it is not clear how these signals correlate with linguistic information. Bridging between human language processing and linguistic features has been widely studied in neurolinguistics, usually via single-variable controlled experiments with highly-controlled stimuli. Such methods not only compromises the authenticity of natural reading, but also are time-consuming and expensive. In this paper, we propose a data-driven method to investigate the relationship between cognitive processing signals and linguistic features. Specifically, we present a unified attentional framework that is composed of embedding, attention, encoding and predicting layers to selectively map cognitive processing signals to linguistic features. We define the mapping procedure as a bridging task and develop 12 bridging tasks for lexical, syntactic and semantic features. The proposed framework only requires cognitive processing signals recorded under natural reading as inputs, and can be used to detect a wide range of linguistic features with a single cognitive dataset. Observations from experiment results resonate with previous neuroscience findings. In addition to this, our experiments also reveal a number of interesting findings, such as the correlation between contextual eye-tracking features and tense of sentence.


Introduction
Cognitively-inspired NLP which uses cognitive processing signals to enhance NLP models in a wide range of tasks, such as sentiment analysis (Barrett et al. 2018), dependency parsing (Strzyz, Vilares, and Gómez-Rodríguez 2019), named entity recognition (Hollenstein and Zhang 2019), part of speech tagging (Barrett et al. 2016), etc. From a computational perspective, cognitive processing signals can introduce additional information that underlies the ways that human brains comprehend texts (Mathias et al. 2020;Mishra, Kanojia, and Bhattacharyya 2016;Ren and Xiong 2021). Such information could be exploited to teach machines to process texts. However, how these cognitive processing signals resonate with linguistic information that is usually used by NLP models is not clear yet. Bridging be-tween cognitive processing signals and linguistic features is certainly desirable for cognitively-inspired NLP.
In neurolinguistics, since the early discovery that the brain is directly involved in sentence comprehension in patients with brain damage, the process of language comprehension in the brain has been studied for hundreds of years (Wood 1996). Current studies have investigated the relationship between brain activities and a number of linguistic features, including basic features (e.g., Word Length, Word Frequency) (Rayner 1998) and complex features (e.g., Semantic Surprisal, Syntactic or Semantic Ambiguities) (Rogalsky and Hickok 2009). Most studies have conducted specially designed controlled experiments for specific linguistic features, and draw conclusions by analyzing differences in brain activities (Constable et al. 2004;Friederici 2011). For instance, in order to locate the region of the brain with a specific function of syntactic processing in sentence comprehension, researchers have compared and analyzed fMRI images that are recorded when subjects read sentences vs. meaningless word lists (Friederici 2011). In such controlled experiments, reading materials for testing are usually unnatural, lacking lexical and syntactic richness exhibited in real-world texts. Hence, the recorded cognitive processing signals are also unnatural. Moreover, the study of each type of linguistic features requires multiple records of cognitive processing signals, which is very labor-intensive and timeconsuming. Thus, a unified and effective method to bridge between cognitive processing signals and linguistic features is also beneficial to neurolinguistics.
In this paper, we propose an efficient data-driven framework based on attention mechanism to study the relationship between linguistic features and cognitive processing signals. Different from neurolinguistic methods, the cognitive processing signals used in our experiments are all from natural reading materials, rather than highly-controlled stimuli.
The results of our model are hence closer to reflect the actual text comprehension process. In order to bridge between cognitive processing signals and linguistic features, we propose an attentional framework to learn the importance of cognitive processing signals to linguistic features by feeding signals as inputs to the model to predict linguistic features. Compared with traditional feature selection methods (e.g., Mutual Information, Random Forests), the proposed attention mechanism can capture dependency between words. To model the relationship between cognitive processing signals and linguistic features, we adapt standard word-level attention into feature-level attention.
For a broad coverage of linguistic features and cognitive processing signals, we design a total of 12 bridging tasks that predict linguistic features from cognitive processing signal inputs. This basic methodology behind our model, which feeds representations into a classifier for linguistic prediction, has been widely used to study the interpretability of pre-training models in NLP (Conneau et al. 2018;Hewitt and Manning 2019). According to the nature of linguistic features, we divide the bridging tasks into three categories: lexical, syntactic and semantic. For the generality of our bridging tasks, linguistic features used in our experiments can be easily obtained via off-the-shelf NLP tools. Yet another advantage of our model is that we can study the connections of cognitive processing signals to arbitrary linguistic features with only one dataset annotated with cognitive processing signals, without requiring to develop a large number of controlled experiments.
In a nutshell, our main contributions include: • We propose a unified attentional framework to bridge between cognitive processing signals and linguistic features, which adapts traditional word-level attention to feature-level attention. • Twelve bridging tasks are developed to investigate the weighted alignments of eye-tracking and EEG signals to a wide range of linguistic features, including lexical, syntactic and semantic features. • Experiments provide new evidences for previous neurolinguistic findings. Additionally, our experiments exhibit new interesting findings, which are not present previously, on the underlying relations between cognitive processing signals and linguistic features. We demonstrate the effectiveness of our proposed method by conducting a number of comparison experiments from the computational perspective.

Relations between Linguistic Features and Cognitive Processing Signals
In neurolinguistics, experiments have proven that cognitive processing signals can effectively reflect important textual information in language comprehension (Kiliańska-Przybyło and Grotek 2021; van Heuven and Dijkstra 2010). Rayner et al. (2004) have studied word recognition with eye-tracking signals. They ask participants to read highlycontrolled stimuli, such as: (1) used a knife to chop the large carrots.
(2) used a pump to inflate the large carrots. The target word is carrots in all sentences. Obviously, the combination of verb and instrument is anomalous in sentence (2). The eye-tracking results show that the gaze duration of the target word in sentence (2) is significantly longer than that in sentence (1). It suggests that gaze duration is sensitive to semantic anomaly. Other methods that probe linguistic features based on Electroencephalogram (EEG) or functional magnetic resonance imaging (fMRI) signals in brain are similar to this research. These neurolinguistic studies usually develop variable-controlled experiments that only change the parameters of interest while other variables are kept intact. Hence, any changes in cognitive processing signals can be attributed to the variation (Weiss and Mueller 2003;Constable et al. 2004). Mitchell et al. (2008) map a word into a semantic feature space and analyze the distribution of each semantic feature in the brain by predicting the corresponding fMRI image of the word.

Feature Selection
The way that we aggregate information from cognitive signals using the attention mechanism and feed cognitive information to a classifier is related to feature selection. The general practice of feature selection is assigning an "importance" score to each feature. Based on the estimated scores, we can improve the performance of model by enhancing relatively important features or provide explanations for black-box model. This technique has been widely used in many different areas (Liu and Motoda 1998;Liu and Yu 2005). The feature selection methods for classification can be roughly grouped into three categories: filtering methods, wrapper methods, and embedded methods (Saeys, Inza, and Larrañaga 2007). The filtering methods usually obtain feature importance scores by calculating the correlation between features and target tags, such as using mutual information (Liu et al. 2021), pearson correlation coefficient (Coelho, de Pádua Braga, and Verleysen 2010). These approaches have excellent scalability, but ignore the dependency between features. The wrapper methods generate and evaluate various feature subsets by training and testing a specific classification model, such as recursive feature elimination (Zeng et al. 2009). Although these methods consider the dependency between features, they are usually computationally intensive. In the embedded methods, the process of feature scoring is built into classifiers, such as random forest (Díaz-Uriarte and de Andrés 2006), SVM (Khan et al. 2018). Advantages of these methods include the interaction between features and classifiers as well as less computational cost than the wrapper methods. The attention mechanism used in this paper can be grouped into the embedded methods, which learn the degree of importance of features during neural model training.

The Unified Attentional Bridging Network
Our model is an unified framework for bridging between different cognitive processing signals and a wide range of linguistic features. The bridging task is defined as a classification task that predicts linguistic features from cognitive processing signals. Since the attention mechanism has outstanding feature selection ability, we utilize the attention mechanism (Lin et al. 2017b) to capture the relationship between cognitive processing signals and linguistic features. The framework of our model is visualized in Figure 1, which is composed of four layers: input layer, attention layer, encoding layer and predicting layer.
Cognitive Processing Signals. The inputs of our model are word-level cognitive processing signals that are recorded  Figure 1: Neural framework of the proposed attentional network for cognitive-linguistic bridging tasks. α represents the degree of importance of cognitive processing signals to the final prediction. y is the final prediction of the model. under a natural reading configuration. For a given sequence X, its distributed representation can be denoted as H ∈ R n×d , where n and d are the length of sentence and the dimension of cognitive processing signal embeddings, respectively. In this paper, we use two types of cognitive processing signals: eye-tracking and EEG. More details of these two types of signals will be presented in Experiments Section. Attention Layer. It is widely acknowledged that attention mechanism can capture semantic dependencies between words in sentences. In this paper, we aim to capture the sensitivity of each cognitive processing signal to linguistic features, so we change the traditional word-level attention into feature-level attention. The updated calculation process of attention is as follows: rameters in the model. v is used to learn importance vectors where the dimension size is equal to the dimension size of cognitive processing signal embeddings. α∈ R d obtained after softmax normalization are real values that represent the degree of the importance of each cognitive processing signal to a target linguistic feature. The weighted sentence representation can be formulated as: H att = H⊗α (2) where ⊗ denotes the Hadamard product that multiplies elements at the same position.
Encoding Layer. We stack a Bi-LSTM layer over the attentional layer to encode the new sentence representation to capture contextual information. The Bi-LSTM encoder is a concatenation of the forward and backward hidden states.
Predicting Layer. If the target output is a sequence of labels, we use conditional random field (CRF) (Lafferty, Mc-Callum, and Pereira 2001) as the predictor. First, we map the output of Bi-LSTM H into another semantic space, the dimension of which is equal to the number of output tags. For a given time step i, it can be formulated as follows: (3) where W s and b s are trainable parameters. Then, we calculate the score of the entire predicted tag sequence y as follows: where T is a transition score matrix that refers to the transition probability of two successive labels. Finally, the model will output the tag sequence with the highest score estimated in Eq. 4. For single-output classification tasks, we use a maxpooling to reduce the dimension of the output of Bi-LSTM and obtain a vector representation h ∈ R d for each sentence. The probability of a predicted label is obtained by a softmax function as follows: where W p and b p are trainable parameters.

Linguistic Features and Bridging Tasks
To select specific linguistic features for our bridging tasks, we mainly follow two criteria: (1) Interpretability. The selected linguistic features should be interpretable and independent of each other.
(2) Extensibility. The selected linguistic features can be automatically generated by off-theshelf NLP tools so that these features can be easily annotated to the training data and bridging tasks can be easily extended to available cognitive datasets. In this way, we construct four bridging tasks for each group of linguistic features (i.e., lexical, syntactic, semantic). Lexical Features. Normally, the occurrence of complex words will disturb the comprehension of text. The first task is hence to predict the length of a word (WordLen). We use the average length of all words in a sentence as the prediction label. The second task in this group is to predict lexical density (LD) that refers to the ratio of the number of content words (e.g., Noun, Verb) to the total number of words in a sentence. This task allows us to analyze the response of cognitive processing signals to content words and function words. The third bridging task is for predicting degree of polysemy (DP), which is the total number of senses contained by each word in the sentence. Intuitively, the more meanings of words in a sentence, the harder the comprehension of the text is. The senses of word can be automatically induced from WordNet 1 . The final out-of-vocabulary (OOV) task in this group is to sum the number of words in a sentence, which do not occur in the list of common words: the combination of the General Word Service List 2 EARLY first fixation duration (FFD) the duration of word w that is first fixated first pass duration (FPD) the sum of the fixations before eyes leave the word w LATE number of fixations (NFIX) the number of times word w that is fixated fixation probability (FP) the probability that word w is fixated mean fixation duration (MFD) the average fixation durations for word w total fixation duration (TFD) the total duration of word w that is fixated n re-fixations (NR) the number of times word w that is fixated after the first fixation re-read probability (RRP) the probability of word w that is fixated more than once CONTEXT total regression-from duration (TRD) the total duration of regressions from word w w-2 fixation probability (w-2 FP) the fixation probability of the word w-2 w-1 fixation probability (w-1 FP) the fixation probability of the word w-1 w+1 fixation probability (w+1 FP) the fixation probability of the word w+1 w+2 fixation probability (w+2 FP) the fixation probability of the word w+2 w-2 fixation duration (w-2 FD) the fixation duration of the word w-2 w-1 fixation duration (w-1 FD) the fixation duration of the word w-1 w+1 fixation duration (w+1 FD) the fixation duration of the word w+1 w+2 fixation duration (w+2 FD) the fixation duration of the word w+2 To avoid the problem of data imbalance, the outputs of each task are three discrete labels from three different intervals segmented according to the values of these features (e.g., DP, OOV). Each label has the equal number of training instances. We regard these tasks as a three-class classification problem. Syntactic Features. Syntactic features focus on how varied and sophisticated sentence elements and their structures are (Lu 2010). The first task in this group is to predict complex nominals per clause (CNC), which is defined as the ratio of the number of complex nominals to the number of clauses. Complex nominals, e.g., nominal groups or nominal clauses, are frequently used to measure the complexity of a sentence in English texts (Nakov 2013). We use L2 Syntactic Complexity Analyzer tool 4 to obtain the annotation of this feature. The second task is predicting the length of a sentence (SenLen), aiming to investigate the sensitivity of different cognitive processing signals to sentence length. The above two tasks are also recast as a three-class classification problem. The third task is predicting part-of-speech (POS) tags, designed to check the differences in human cognition processing of different parts of speech in sentences. POS tags are obtained by using Stanford Parser 5 . The bigram shift (BShift) task tests the extent to which different cognitive processing signals are sensitive to grammatical errors related to word order. This task is a binary classification problem, where we randomly swap two adjacent words in a sentence as a negative sample while the original sentence is regarded as a positive sample.
Semantic Features. We define semantic features of a sentence as features containing meaningful information ob-3 http://www.victoria.ac.nz/lals/resources/academicwordlist/. 4 http://www.personal.psu.edu/xxl13/downloads/l2sca.html. 5 https://nlp.stanford.edu/software/lex-parser.shtml. tained from context. In this paper, semantic features for bridging tasks are mainly inspired by Conneau et al. (2018). The first task in this group is to detect the tense of a sentence, which is usually dependent on the part-of-speech of the verb in the main clause: VBP/VBZ/VBG labeled as the present tense, VBD/VBN as the past tense, and VBC/VBF as the future tense. The subject number (SubjNum) task and object number (ObjNum) task focus on the number of subjects and objects of a sentence, respectively. These two tasks are to identify whether the target is singular or plural. The Tense, SubjNum and ObjNum are considered as semantic tasks rather than syntactic tasks, since these tasks essentially require understanding the meaning of a given sentence (e.g., whether the event described in the sentence occurred in the past). The changes of the labels of these tasks do not result in any changes in sentence structures. Discourse Connector Count (DCC) task is to predict the number of discourse connectors (e.g., however, because) that logically connects different discourse units. These connectors usually have causal or inferential implications. We aim to explore the sensitivity of cognitive processing signals to discourse coherence by this task. The Discourse Connector List 6 is used to annotate this task. Similar to the previous lexical tasks, DCC is also set as a three-class classification task.

Experiments Dataset and Cognitive Processing Signals
We used a cognitive dataset curated under natural reading: Zurich Cognitive Language Processing Corpus (ZuCo) . It is a publicly available dataset 7 of simultaneous eye-tracking and EEG signals from subjects reading natural sentences. This corpus includes recordings of 12 adult and native speakers reading approximately 1100 English sentences.
The dataset includes two reading paradigms: normal reading and task-specific reading. In the latter, subjects are asked  to exercise some comprehension task before recording. The goal of this work is to analyze real language comprehension in natural reading. Therefore, we only used the data of normal reading. The reading materials in this paradigm consist of 700 sentences.
Eye-tracking. The eye-tracking data of ZuCo are collected by an infrared video-based eye tracker EyeLink 1000 Plus. Since we want to cover all eye movement features, in addition to the measured fixation duration features, we also added the fixation probability which plays an important role in syntactic understanding (Demberg and Keller 2008). Moreover, the fixation features of precursors and postcursors (e.g., w-1 fixation duration) are considered in this work. In total, we used 17 eye-tracking features that cover all stages of gaze behaviors to probe the linguistic features in eyetracking signals. These features are divided into three groups based on reading stage: EARLY, the gaze behaviors when a word is first fixated, which reflect early word acquisition and syntactic processing; LATE, the gaze behaviors over a word after sentence reading, which generally reflect late syntactic processing and semantic understanding of sentences; CON-TEXT, the eye-tracking features over neighboring words of the current word. More details of these 17 features are shown in Table 1.

Experiment Settings
The dimension of hidden states in the Bi-LSTM encoder was set to 20. Due to the data imbalance issue in Tense, SubjNum and ObjNum bridging tasks, we used focal loss (Lin et al. 2017a) to reduce the proportion of easily classified samples in the training. The cross-entropy loss function was used for other bridging tasks. To obtain robust experimental results, we performed 5-fold cross validation for all bridging tasks.

Results
The experiment results are shown in Table 2. From the results of eye-tracking signals, we have the following observations. First, FPD and FFD have relatively high attention scores on WordLen and OOV features, indicating that the perception of word length and word proficiency appears in the brain when the word if first fixated. Second, NR and TFD are highly correlated with DP and OOV. This observation is in accord with previous neurolinguistic finding that the difficulty of word understanding strongly influences the number of regressions to the word (Duffy 1992). Third, not surprisingly, the late eye movement behaviors (e.g., NFIX, TFD) are aligned to CNC feature that detects complex nominals. Fourth, it is worth noting that the probability of contextual words being fixated plays an important role in multiple bridging tasks (i.e., SenLen, Bshift, Tense, DCC). This is consistent with the finding that the constraints of contextual words affect the fixation duration and skipping frequency of the current word (Rayner and Well 1996). We therefore suggest that eye movement behaviors over contextual words could be used to assess the syntactic and semantic complexity of sentences. Finally, we find that late eye movement behaviors (e.g., MFD, TFD) over words are more important for the judgment of subject and object singularity or plurality in sentences than early eye movement behaviors. In summary, early eye movement behaviors focus on lexical features while eye movement behaviors over contextual words are sensitive to syntactic and semantic informa-tion. On the other hand, the eye-tracking features of late eye movement behaviors seem to correlate with all three types of linguistic features tested in our experiments, lexical, syntactic and semantic features.
EEG signals that record neuronal activity at a millisecond time scale can provide dynamic temporal information for brain language processing. Studying the oscillatory changes in electrical signals in the brain can help us understand various reading comprehension processes. Results of EEG are also consistent with previous neuroscience findings. (1) As Table 2 shows, compared with other frequency range, the attention scores of t1 and t2 are generally higher than other EEG features in the lexical bridging tasks, especially in LD task. This conforms with early insights that both open-class (OC) words (e.g., nouns, verbs) and closed-class (CC) words (e.g., articles, determiners) elicit a power increase in the theta frequency range, but OC words (related to the LD feature) have stronger power changes than CC words (Bastiaansen et al. 2005). (2) The theta and beta frequency range   are good at CNC bridging task, resonating with the previous finding that there is a large gap in beta frequency range between sentences with complex syntax and sentences with simple syntax (Weiss et al. 2005). (3) The high frequency band (gamma) is acknowledged to be related to semantic information (Weiss and Mueller 2003). The results on DCC, which demonstrate the significance of gamma frequency band in DCC bridging task, clearly provide yet another evidence for this. More interestingly, the gamma frequency band is sensitive to word order (BShift task), and alpha frequency band is related to the number of subjects and objects in sentences.
Overall, these results on eye-tracking and EEG signals provide new evidences for previous neuroscience findings from a data-driven and computational perspective. These results also suggest that cognitive processing signals are related to rich linguistic information, which can be explored in various NLP tasks. Our model provides a unified and efficient way to probe linguistic features behind cognitive processing signals.

Ablation Study
To verify the robustness of the attention mechanism in our bridging framework, we further carried out bridging experiments without using the Bi-LSTM encoder. The results for eye-tracking signals are shown in Figure 2, and the results for EEG signals can be found in Appendix. Comparing the complete model to the variant model without the Bi-LSTM encoder, we find that the curves of attention scores obtained by the two models in the same bridging task are in general accord with each other. Although cognitive processing signals with the highest attention scores in some bridging tasks under the two models are different, the overall trends are the same. This suggests that the feature-level attention in our model can well capture alignments between linguistic features and cognitive processing signals.

Effectiveness Validation by Signal Masking
To further validate the effectiveness of our model, we conducted masking experiments on all bridging tasks. First, we sort cognitive processing signals in a descending order according to their attention scores. Then, on the test set, one cognitive processing signal is masked at a time to take a deep look into the role of the attention layer. The results of masking experiments with the two types of cognitive processing signals (eye-tracking and EEG) are shown in Figure 3. All results are obtained by 5-fold cross validation.
Partially masking some cognitive processing signals results in a drop in performance. Meanwhile, the higher attention score of the cognitive processing signals being masked, the more significant the performance drop, which indicates that the proposed attention layer can measure the degree of (a) LD task (Eye) (b) CNC task (Eye) (c) Tense task (Eye) (d) LD task (EEG) (e) CNC task (EEG) (f) Tense task (EEG) Figure 4: Results of our model and other feature selection methods on bridging tasks with eye-tracking features and EEG features. 'MI', 'RFE' and 'RF' denote Mutual Information, Recursive Feature Elimination and Random Forest, respectively. The evaluation metric is F 1 (shown in y-coordinate). X-coordinate denotes the number of features selected (i.e., k).
importance of features to neural network classification. Additionally, we find that the performance of masking cognitive processing signals with low attention scores is better than the original model that uses all cognitive processing signals. This suggests that some cognitive processing signals may bring negative noises to the bridging task.

Comparison to Other Feature Selection Methods
The attention layer in our framework can be considered as a feature selector: selecting cognitive processing signals that are highly correlated with target linguistic features to feed into the predicting layer. To evaluate classification performance of our attentional feature selection against traditional feature selection methods, we conducted comparative experiments on eye-tracking and EEG signals. We adopted various feature selection methods, including our attention method, to estimate an "importance" score for each feature. Then, we selected top k features as inputs to a classifier, and evaluated the trained classifier on the test set to obtain the differences on classification performance. We used two classification methods to evaluate feature selection: logistic regression classifier (LR) and LSTM model. We chose three different traditional feature selection methods, one from each type of feature selection methods as described in the section of related work: (1) Mutual Information, a filtering method measuring the dependence between two random variables by the joint probability distribution of the two variables and their marginal probability distributions; (2) Recursive Feature Elimination, a wrapper method recursively eliminating a small number of features that have dependencies and collinearity in the classification model; (3) Random Forest, an embedded method that combines a number of decision tree classifiers and returns average prediction results of these classifiers. We conducted experiments on all bridging tasks with 5-fold cross-validation.
The feature importance scores estimated by these methods and our model on bridging tasks (LD, CNC and Tense) are shown in Table 3.
Due to the space limit, we only show partial results in Figure 4 while the full results can be found in Appendix. Obviously, either with LR or LSTM, the classification performance with features selected by our method is significantly better than that with features selected by the three traditional methods on almost all bridging tasks. It suggests that our proposed method is able to effectively select useful features. In addition, the performance of the LSTM model is better than LR model in most cases. However, LR outperforms LSTM on some tasks when the number of selected features is small. We conjecture that this may be due to the overfitting problem of LSTM on small training sets with less feature information.

Conclusions
In this paper, we have presented a unified model to investigate the relationship between cognitive processing signals and linguistic features by using a feature-level attention mechanism to select relevant cognitive processing signals in linguistic bridging tasks. This method can test a wide variety of linguistic features on a single dataset with cognitive processing signals recorded under natural reading. Such a data-driven method may act as a surrogate to controlled experiments in neurolinguistics. Experiment results corroborate and extend previous findings, demonstrating that our method can effectively detect linguistic information in cognitive processing signals.