{'fraction_non_alphanumeric': 0.03194397812844741, 'fraction_numerical': 0.011415415607463187, 'mean_word_length': 5.119753448781919, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Spectral Probing

Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages-highlighting their potential as robust, lightweight task descriptors.


Introduction
Analyzing the contextualized embedding representations of pre-trained language models (LMs) using lightweight probes (Hewitt and Liang, 2019;Voita and Titov, 2020) has identified latent features in the untuned encoders which are highly relevant to downstream NLP tasks at various layer depths (Tenney et al., 2019). Orthogonally, linguistic phenomena are also encoded at different timescales: i.e., rapidly changing (sub-)word-level information versus slower changing sentence or paragraph-level information. Decomposing contextualized embeddings into frequencies with different rates of change has yielded initial insights into the timescales at which these task-specific latent phenomena occur (Tamkin et al., 2020). These findings currently rely on handcrafted spectral filters and are limited to English. To enable more efficient analyses of finer-grained, continuous frequency spectra in contextualized representations covering more tasks and languages, this work contributes: • A fully differentiable spectral probing framework for learning which frequencies are relevant for specific NLP tasks (Section 2). 1 • A multilingual probing study examining timescale characteristics of seven diverse NLP tasks across six languages (Section 3).
• An analysis of the relationships between the spectral profiles of different tasks and their consistency across languages (Section 4).

Probing for Spectral Profiles
Spectral Probing (Figure 1) builds on established signal processing methods (Ahmed et al., 1974) and recent findings on the manual frequency filtering of contextual embeddings (Tamkin et al., 2020). The method automatically learns spectral profiles which measure the relevance of specific frequencies to a given task by amplifying or reducing contextual information with different rates of change.
Discrete Cosine Transform (Ahmed et al., 1974;DCT) is an invertible method for decomposing any sequence of real values {x 0 , . . . , x N −1 } (e.g., all values of an embedding dimension) into a weighted sum over cosine waves with different frequencies. The number of frequencies equals the sequence length N , as the lowest frequency wave is a constant (k = 0) and the highest frequency wave completes one cycle every timestep (k = N − 1).
The coefficient X (k) n for a wave at DCT index k at timestep n is calculated as: Inverting the DCT (IDCT) using all X (k) n will return the original sequence. However, weighting coefficients for some k by 0 will return a filtered version. Zeroing out all k above a threshold will only retain lower frequencies and make values oscillate with a slow rate of change. Vice-versa, zeroing out all k below a threshold will only retain higher frequencies-amplifying short-term changes.
Fixed-band Filters Applying frequency filters to a sequence of contextualized embeddings extracts linguistic information at different timescales. Within this formulation, the values across each embedding dimension are gathered into a real-valued sequence to which transformations such as the DCT can be applied. In seminal work, Tamkin et al.

Learnable Filters
To capture the complete picture, we propose spectral probing which learns a continuous weighting of frequencies relevant to a task. In effect, the spectral filter is a vector γ ∈ R M for which each entry corresponds to the weight assigned to a particular frequency. Before inverting the DCT, each X (k) n is multiplied by the sigmoidscaled weight γ (k) ∈ [0, 1] which will then retain or filter out frequencies at index k. As M depends on the sequence length N , which changes across inputs, the spectral probe dynamically scales γ to the length at hand using adaptive mean pooling. In practice, we set M to the maximum input length for our given encoder (e.g., 512 for BERT) and shrink γ appropriately, as a wave cannot cycle more often than there are values. It would however be equally possible to set M smaller than N and interpolate the filter up to the length required. Overall, γ is a lightweight parameter which can be easily incorporated between the frozen encoder and probing head, and uses the existing training objective to jointly learn which frequencies to amplify or filter out.

Monolingual
Setup Initially, we compare spectral probing to previous fixed-band filters by reproducing the highest and lowest frequency experiments by Tamkin et al. (2020). These are the tasks of tagging partsof-speech (POS) in the Penn Treebank (Marcus et al., 1993; PTB) as well as classifying TOPICS in the 20 Newsgroups corpus (Lang, 1995;20News).
On the modeling side, we follow Tamkin et al. (2020) and train a linear probe (Alain and Bengio, 2017) on top of the frozen LM encoder to classify each manually/automatically filtered contextual embedding in an input sequence. This corresponds to probing and evaluating for the amount of task-relevant information in each sub-word across a sequence (e.g., underlying topic contextualization). The bands for the five manual filters follow the original definitions (see Section 2), and we compare them to unfiltered (ORIG) as well as automatically filtered (AUTO) embeddings from our spectral probe (details in Appendix B). Results Figure 2 shows the accuracy (ACC) of the six prior filtering strategies in addition to the learned frequency weightings of the spectral probe. The unfiltered and manually filtered embeddings corroborate previous findings (Tamkin et al., 2020), with high frequencies performing best on POS, and the lowest frequencies performing best on TOPIC.
The spectral probe achieves 95.9% ACC for POS, outperforming ORIG by a 0.1% margin and the best manual filter by 5.2%. The spectral profile in Figure 2a (right) sheds light on why this may be the case: While it also prioritizes high (sub-)word-level frequencies, the learned filter additionally includes surprising amounts of mid-high and lower frequencies, emphasizing the need for both local and global context to achieve high performance.
For TOPIC, the spectral probe achieves 72.1% ACC, outperforming both ORIG (41.3%) and the fixed low-band filter (71.2%). The learned filter (see Figure 2b, right) mirrors the fixed-band results: Only the lowest bands are active, while all higher ones are not. As mid-low frequencies still appear to contain weaker amounts of topic information, the soft inclusion of this region by the spectral probe could account for its performance boost. Overall, spectral probing confirms and refines frequency ranges from prior work while surfacing more detail and requiring no manual probe engineering, with only a single probing run instead of five.

Multilingual
Leveraging spectral probing, we extend timescale analyses beyond English and investigate spectral profiles across more diverse tasks and languages. For each language-task combination we train a linear probe on the unfiltered embeddings of multilingual BERT (Devlin et al., 2019; mBERT) and on the automatically filtered representations from our spectral probe. The remaining settings are identical to the monolingual setup (details in Appendix B).
Results Table 1 shows equivalent or higher ACC for the spectral filter compared to the unfiltered embeddings for all tasks and languages. This increase is less pronounced for token-level tasks, but much larger for tasks where sequence-level information is critical. Figure 3 visualizes how POS, DEP and NER retain large parts of the original spectrum, while QA, SENTI, TOPIC and NLI appear to benefit from filtering out higher frequencies. This shows how tasks exhibit structures at different timescales and that spectral probing is able to identify these communicative levels consistently not only in English, but also across languages-an effect which we analyze more extensively next.

Spectral Profiling Analysis
Each task's distinct spectral profile ( Figure 3) allows us to analyze their relation to the timescale hierarchy of linguistic structures, and quantify crosstask similarities within and across languages. For this we use the percentage-normalized L1 distance (i.e., 0%-100% overlap) between filters ( Figure 4).

Cross-task Overlap
Overall, we observe a dichotomy between broad-frequency, token-level tasks and low-frequency, sequence-level tasks (Figures 3 and 4a). In addition, there appears to be a hierarchy which depends on the timescales of the linguistic structures involved. Notably, compared to prior fixed-band filters, none of the learned filters fully excludes low frequencies. For instance, highfrequency information is most important to retrieve POS, but reaching the performance of the original embeddings also requires some lower-frequency information-most likely to disambiguate difficult cases based on sentence-level context. DEP appears to benefit the least from both lower and higher-frequency information. Instead, the strong weight on mid-high frequencies matches the fact that dependency relations span multiple words and benefit from information at the phrase-level. NER sees a further decrease in high-frequency information, coupled with an uptick in lower frequencies. We hypothesize that phrase and sentencelevel information become more important for disambiguating certain entity types (e.g., ORG and LOC). Across the token-level tasks this shift from higher to lower frequencies is also reflected in filter overlap which decreases from syntactic to semantic token-level tasks, while their overlap with sentencelevel tasks increases ( Figure 4a).
The sequence-level tasks share low-frequency spectral profiles which overlap more with each other than do the token-level tasks. In fact, SENTI and TOPIC overlap almost perfectly (although the latter involves less mid-range frequencies). This similarity is unlikely to be the result of the shared underlying dataset as both tasks also overlap with the unrelated XNLI and JSNLI datasets. At the same time, the POS and DEP tasks, which also share datasets, have a lower overlap despite being based on the exact same inputs. Overall, SENTI, TOPIC and NLI all appear to rely on information which is consistent across a sequence-explaining why simple methods such as mean-pooled sentence embeddings can be effective in these scenarios.
QA provides an intermediate case: While it is reliant on low frequencies it also includes more mid-low and a small amount of higher frequency information. This is reflected in Figure 4a, where it shares more overlap with the token-level tasks than all other sequence-level tasks. Since probing for the correctness of a question-answer pair is dependent on finer-grained information than the general sentiment, topic or semantic coherence of a sequence, this inclusion of higher frequency information matches linguistic intuitions.
Cross-lingual Consistency Finally, we investigate the similarity of learned spectral profiles across languages. While Figure 3 shows that there is some variance between the filters of different languages within a task, Figure 4b shows that actual quantitative overlap between languages is high, ranging from 94%-98%. This holds even across distinctive pairs such as JA-EN which differ substantially in factors such as sub-word length and distance between syntactic dependents. This strong consistency highlights the potential for spectral profiles to provide language-agnostic features for task characterization and comparison.

Conclusion
Linguistic information at different timescales is an, as of yet, underexplored dimension in contextualized embeddings. We propose a fully differentiable spectral probe which automatically learns to weigh frequencies that are relevant to a specific task and improves over prior fixed-band filters by capturing continuous mixtures over frequencies (Section 2). This enables us to not only outperform the manual filters while using one probe instead of five, but to also identify that high-frequency tasks still benefit from low-frequency information (Section 3.1).
Extending spectral probing to seven tasks in six languages, we trained task-specific filters which outperformed the original, unfiltered embeddings. The resulting spectral profiles furthermore shed light onto how linguistic information at different timescales relates to different task types (Section 3.2). They not only match the linguistic intuitions underlying each task, but also enable quantitative comparisons between them. The analysis of the filters' overlap surfaced a clear dichotomy between token and sequence-level tasks, but also highlighted intersecting frequency ranges which contain information relevant across task types. Finally, the language-agnostic nature of these spectral profiles highlights future avenues towards more robust task descriptors (Section 4).

Limitations
Our experiments cover a diverse, but nonexhaustive set of NLP tasks and languages. While more extensive than prior related work (Tenney et al., 2019;Tamkin et al., 2020), we elaborate in the following regarding the motivation of the final setup: As the aim of our study was to investigate the cross-lingual properties of the underexplored timescale dimension of contextualized representations, the set of languages and tasks used in our experiments emphasizes consistency across languages. This limits us to high-resource languages for which datasets covering every task are available. However, with cross-lingual stability confirmed in our experiments, the study of lower-resourced languages is a clear avenue for future research.
Despite using a set of well-established datasets, it is important to keep data quality in mind when interpreting the results-even for these high-resource languages. In our initial exploratory data analyses, we identified and confirmed limitations known to the original dataset authors in that many include silver, or weakly filtered annotations driven by automatic matching and translation (e.g., WikiANN, XNLI, JSNLI). As we are less interested in benchmarking performance and rather focus on the feasibility and analysis of our spectral profiles, individual data instances of lesser quality should however be of limited concern. Appendix A details how each dataset was constructed originally, and also how it was pre-processed by us, such that results can be interpreted in the appropriate context.
In terms of modeling, we hope that future work will investigate spectral probes and their resulting task profiles across more encoder models with different architectures and pre-training strategies. Finally, while we have demonstrated spectral profiles to be suitable for characterizing different tasks consistently across languages, future research could supplement them with other descriptors such as embedding layer depth in order to identify even more distinctive profiles.

Ethics Statement
Given the theoretical nature and wide applicability of this work-both in terms of data domains and model architectures-it is difficult to anticipate broader impacts and future ethical implications. In general, benefits and harms in the field of probing originate from the information being investigated: While we are interested in linguistic timescale characteristics, probe-like methods have also been applied to protected attributes of data subjects in order to, for example, de-bias LMs (Ravfogel et al., 2020). Since this process involves personal information, any experiments extracting such characteristics should be sufficiently vetted for ethical acceptability. With spectral profiles being a relatively broad descriptor however, we do not expect them to identify frequencies exclusive to personal information or to replace existing, domainspecific probing methods.  After removing unanswerable questions, we use each correct QA pair to generate an additional incorrect pair for the same question, yielding a total set of 13,516 instances used in our experiments. To generate an incorrect answer, we sample an alternative answer of the same type (e.g., time, number) which does not equal the correct answer. Finally, we randomly split the data 80/20 into training and validation portions for which the instances are aligned across languages (i.e., the same questions and answers). The final task is a binary classification task for whether a QA pair is true or false, with a random baseline of 50%.  Manual, fixed-band filters as well as the automatically learned filters are applied to the contextualized embeddings produced by the last layer of either model. As visualized in Figure 1, we decompose the sequence of values from each em-bedding dimension (768 in both LMs) using the DCT (Ahmed et al., 1974;DCT-II), weight the appropriate k by a fixed amount or by the learned weight in γ, before applying the IDCT to reconstruct a sequence of real values. These make up each dimension of the filtered embeddings.
Following Tamkin et al. (2020), the original/filtered embeddings are passed to a linear probe (Alain and Bengio, 2017) consisting of two parameters: a transformation W ∈ R E×C and a bias b ∈ R C , where E is the embedding dimension and C is the number of classes specific to each task.
Training As we run probing experiments, neither the 108M-parameter BERT, nor the 178Mparameter mBERT are fine-tuned. We only train the linear probe which has 1,538-36,912 parameters depending on the task, plus the 512 parameters of the learned spectral filter γ. As in Tamkin et al.
(2020), we use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 10 − 3 which decays by 0.5 every time the loss plateaus. Updates are applied in batches of size 32 across a maximum of 30 epochs, with an early stopping patience of 1. Each setup is run with the five random seeds: 1932, 2771, 7308, 8119, 9095. On our hardware consisting of an NVIDIA A100 GPU with 40GBs of VRAM and an AMD Epyc 7662 CPU, training a probe takes approximately 10 minutes.
Evaluation In order to probe a sequence of contextualized embeddings for information at different timescales, it is necessary to apply each filter at the sub-word level. To measure the effect of different frequencies, we follow Tamkin et al. (2020) and evaluate all tasks using accuracy (ACC) on the sub-word level. Note that for token-level tasks each token label is therefore repeated across all of its sub-words, while for sequence-level tasks, each sub-word is classified with the label of its sequence.

C Detailed Results
The following supplements the results presented in Section 3 with more detailed scores. Table 3 lists the exact scores for the monolingual English experiments on POS and TOPIC using the ORIG embeddings, the fixed-band filters and the learned AUTO filter. Table 4 lists the detailed scores for the ORIG and AUTO-filtered embeddings per language, in addition to the cross-lingual mean and standard deviation, across our seven tasks.
While the scores across random initializations never exceed a standard deviation of 1.0, it is important to note that scores may have higher variance across languages. This is to be expected due to different data across languages as well as pre-training availability. However we note that overall performance patterns (i.e., higher AUTO and relative task performance) are consistent across languages.     Table 4: Detailed Multilingual Results (ACC) for unfiltered (ORIG) and automatically learned filters (AUTO) on the tasks of POS-tagging, dependency relation classification (DEP), named entity recognition (NER), question answering (QA), sentiment analysis (SENTI), TOPIC classification, and natural language inference (NLI). Each task covers the languages German (DE), English (EN), Spanish (ES), French (FR), Japanese (JA) and Chinese (ZH). Reported are the mean over five random initializations ± standard deviations as well as the mean over languages (AVG) ± the standard deviation across languages. The latter results are reported in Table 1, in addition to the spectral profiles (frequency weightings) learned by AUTO in Figure 3.