{'fraction_non_alphanumeric': 0.03998007637390005, 'fraction_numerical': 0.02301178814544247, 'mean_word_length': 4.633370744481856, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Decoding speech from non-invasive brain recordings

Decoding language from brain activity is a long-awaited goal in both healthcare and neuroscience. Major milestones have recently been reached thanks to intracranial devices: subject-specific pipelines trained on invasive brain responses to basic language tasks now start to efficiently decode interpretable features (e.g. letters, words, spectrograms). However, scaling this approach to natural speech and non-invasive brain recordings remains a major challenge. Here, we propose a single end-to-end architecture trained with contrastive learning across a large cohort of individuals to predict self-supervised representations of natural speech. We evaluate our model on four public datasets, encompassing 169 volunteers recorded with magneto- or electro-encephalography (M/EEG), while they listened to natural speech. The results show that our model can identify, from 3s of MEG signals, the corresponding speech segment with up to 72.5% top-10 accuracy out of 1,594 distinct segments (and 44% top-1 accuracy), and up to 19.1% out of 2,604 segments for EEG recordings -- hence allowing the decoding of phrases absent from the training set. Model comparison and ablation analyses show that these performances directly benefit from our original design choices, namely the use of (i) a contrastive objective, (ii) pretrained representations of speech and (iii) a common convolutional architecture simultaneously trained across several participants. Together, these results delineate a promising path to decode natural language processing in real time from non-invasive recordings of brain activity.

Over the past decades, BCI has made significant progress in decoding language from the brain using intracranial recordings. In particular, it is now possible to decode phonemes, speech sounds [Pei et al., 2011, Akbari et al., 2019, hand gestures [Stavisky et al., 2018, Willett et al., 2021 and articulatory movements [Anumanchipalli et al., 2019, Moses et al., 2021 from electrodes implanted in the cortex or over its surface. For instance, Willett et al. [2021] decoded 90 characters per minute (with a 94% accuracy, i.e. roughly ≈15-18 words per minute) from a spinal-cord injury patient recorded in the motor cortex during 10 hours of writing sessions. Similarly, Moses et al. [2021] decoded 15.2 words per minute (with a top-1 accuracy of 74.4 with a vocabulary of 50 words) in an anarthria patient implanted in the sensorimotor cortex and recorded over 48 sessions spanning over 22 hours.
Despite their high signal-to-noise ratio, invasive recordings face major practical challenges: they require brain surgery and are difficult to maintain over long time periods. To address this limitation, several laboratories have instead focused on decoding brain activity from non-invasive recordings. In particular, magneto-and electro-encephalography (M/EEG) can record macroscopic brain signals in real time with a safe and potentially wearable setup [Boto et al., 2018]. However, these devices produce notoriously noisy signals that can vary greatly across sessions and across individuals [Schirrmeister et al., 2017, King et al., 2018, Hämäläinen et al., 1993. To address this issue, previous studies typically engineered pipelines that output hand-crafted features, which, in turn, can be learned by a decoder trained on one subject at a time to predict a limited set of interpretable linguistic categories, such as part-of-speech categories or words from a small vocabulary [Lopopolo and van den Bosch, 2020, Chan et al., 2011, Nguyen et al., 2017. In such cases, the transformations of both brain and speech signals is thus set by the researchers.
Instead, we here propose to decode natural speech processing from non-invasive M/EEG recordings by using a single architecture and a data-driven approach. To this aim, we introduce a convolutional neural network stacked onto a "Subject Layer" and trained with a contrastive objective to predict the deep representations of the audio waveform learnt by a dedicated module pretrained on 56k hours of speech [Baevski et al., 2020] (Figure 1). We validate our approach on four public M/EEG datasets by decoding 3 s audio segments from the brain activity of 169 participants passively listening to speech. With a sample of 3 seconds of M/EEG signals, our model identifies the matching audio segment (i.e. zero-shot decoding) with up to 72.5% top-10 accuracy (out of 1,594 segments) for MEG and up to 19.1% top-10 accuracy (out of 2,604 segments) for EEG. Figure 1: Method We aim to decode speech from the brain activity of healthy participants recorded with magnetoencephalography (MEG) or electroencephalography (EEG) while they listen to stories and/or sentences. For this, our model extracts the deep contextual representations of 3 s speech signals (Y ) from a pretrained self-supervised model (wav2vec 2.0: Baevski et al. [2020]) and learns the representations Z of the brain activity on the corresponding 3 s window (X) that maximally align with these speech representations with a contrastive loss (CLIP: Radford et al. [2021]). The representation Z is given by a deep convolutional network. At evaluation, we input the model with left-out sentences and compute the probability of each 3 s speech segment given each brain representation. The resulting decoding can thus be "zero-shot" in that the audio snippets predicted by the model need not be present in the training set. This approach is thus more general than standard classification approaches where the decoder can only predict the categories learnt during training.

Method
We first formalize the general task of neural decoding and motivate the use of a contrastive loss for training. We introduce the rich speech representation given by the pre-trained self-supervised module wav2vec 2.0 [Baevski et al., 2020], before introducing the deep learning architecture we use for brain decoding.

Neural decoding
We aim to decode speech from a time series of high-dimensional brain signals recorded with non-invasive magneto-encephalography (MEG) or electro-encephalography (EEG) while healthy volunteers passively listened to spoken sentences in their native language. How spoken words are represented in the brain is largely unknown [Hickok and Poeppel, 2007]. Thus, it is common to train decoders in a supervised manner to predict a latent representation of speech known to be relevant to the brain [Akbari et al., 2019, Angrick et al., 2019b,a, Krishna et al., 2020, Komeiji et al., 2022. For example, the Mel spectrogram is often targeted for neural decoding because it representats sounds similarly to the cochlea [Mermelstein, 1976]. Let X ∈ R C×T be a segment of a brain recording of a given subject while she listens to a speech segment of the same duration, with C the number of M/EEG sensors and T the number of time steps. Let Y ∈ R F ×T be the latent representation of speech, using the same sample rate as X for simplicity, here the Mel spectrogram with F frequency bands. Thus, supervised decoding consists of finding a decoding function: f reg : R C×T → R F ×T such that f reg predicts Y given X. We denote byŶ = f reg (X) the representation of speech decoded from the brain. When f reg belongs to a parameterized family of models like deep neural networks, it can be trained with a regression loss L reg (Y,Ŷ ) (e.g. the Mean Square Error), Empirically, we observed that this direct regression approach faces several challenges: decoding predictions appear to be dominated by a non-distinguishable broadband component when speech is present (Figure 2.A-B). This challenge motivates our three main contributions: the introduction of a contrastive loss, a pre-trained deep speech representation, and a dedicated brain decoder.

Contrastive loss
First, we reasoned that regression may be an ineffective loss because it departs from our objective: decoding speech from brain activity. Consequently, we replaced it with a contrastive loss, namely, the "CLIP" loss (originally for Contrastive Language-Image Pre-Training) by Radford et al. [2021], which was originally designed to match latent representations in two modalities, text and images. We implement the CLIP loss as follows: Let X be a brain recording segment and Y ∈ R F ×T the latent representation of its corresponding sound (a.k.a "positive sample"). We sample N − 1 negative samplesȲ j∈{1,...,N −1} over our dataset and we add the positive sample asȲ N = Y . We want our model to predict the probabilities ∀j ∈ {1, . . . , N }, p j = P Ȳ j = Y . We thus train a model f clip mapping the brain activity X to a latent representation Z = f clip (X) ∈ R F ×T . The estimated probability can then be approximated by the dot product of Z and the candidate speech latent representations Y j , followed by a softmax: with ·,· the inner product over both dimensions of Z andŶ . We then train f clip with a cross-entropy between p j andp j . Note that for a large enough dataset, we can neglect the probability of sampling twice the same segment, so that we have p j = 1 j=N , and the cross-entropy simplifies to Following [Radford et al., 2021], we use the other elements of the batch as negative samples at train time. At test time, the negative samples correspond to all of the segments of the test but the positive one. Mel-spectrogram predicted with a direct regression loss L reg of a brain decoder (orange). C. Replacing the regression loss with a CLIP loss [Radford et al., 2021] improves reconstruction in the same subject, still using the mel-spectrogram as the speech representation. D. Now replacing the mel-spectrogram with wav2vec 2.0 [Baevski et al., 2020]. The probabilities given by (2) are used to rebuild a mel-spectrogram. E. Architecture of the brain module. Architecture used to process the brain recordings. For each layer, we note first the number of output channels, while the number of time steps is constant throughout the layers. The model is composed of a spatial attention layer, then a 1x1 convolution without activation. A "Subject Layer" is selected based on the subject index s, which consists in a 1x1 convolution learnt only for that subject with no activation. Then, we apply five convolutional blocks made of three convolutions. The first two use residual skip connection and increasing dilation, followed by a BatchNorm layer and a GELU activation. The third convolution is not residual, and uses a GLU activation (which halves the number of channels) and no normalization. Finally, we apply two 1x1 convolutions with a GELU in between.

Speech module
Second, the Mel spectrogram is a low-level representation of speech and is thus unlikely to match the rich variety of cortical representations [Hickok and Poeppel, 2007]. Consequently, we replaced the Mel spectrograms Y with latent representations of speech, that are either learned end-to-end ("Deep Mel" model) or learned with an independent self-supervised speech model ("wav2vec 2.0", Baevski et al. [2020]) As detailed in the result section, the "Deep Mel" model uses an architecture similar to the brain module, but proved less efficient than its pretrained counterpart. We will thus focus the decoding results obtained with wav2vec 2.0.
Wav2vec 2.0 is trained to transform the raw waveform with convolutional and transformer blocks to predict masked parts of its own latent representations. Baevski et al. [2020] showed that the resulting model can be efficiently fine-tuned to achieve state-of-the-art performance in speech recognition. Besides, this model effectively encodes a wide variety of linguistic features Dunbar, 2022, Adolfi et al., 2022]. Finally, recent work shows the existence of linear correspondence between the activations of the brain and those of wav2vec 2.0 , Vaidya et al., 2022. Consequently, we here test whether this model effectively helps the present decoding task. In practice, we use the wav2vec2-large-xlsr-53 1 , which has been pre-trained on 56k hours of speech from 53 different languages.

Brain module
Finally, for the brain module, we use a deep neural network f clip , input with raw M/EEG times series X and a one-hot-encoding of the corresponding subject s, and outputs the latent brain representation Z, with the same sample rate as X. This architecture consists of (1) a spatial attention layer over the M/EEG sensors followed (2) by a subject-specific 1x1 convolution designed to leverage inter-subject variability, which input to (3) a stack of convolutional blocks. An overview of the model is given in Figure 2. In the following, given a tensor U , we will note U (i,...) access to specific entries in the tensor.
Spatial attention and subject layer. The brain data is first remapped onto D 1 = 270 channels with a spatial attention layer based on the location of the sensors. The 3D sensor locations are first projected on a 2D plane obtained with the MNE-Python function find_layout , which uses a device-dependent surface designed to preserve the channel distances. Their 2D positions are finally normalized to [0, 1]. For each output channel, a function over [0, 1] 2 is learnt, parameterized in the Fourier space. The weights over the input sensors is then given by the softmax of the function evaluated at the sensor locations. Formally, each input channel i has a location (x i , y i ) and each output channel j is attached a function a j over [0, 1] 2 parameterized in the Fourier space as z j ∈ C K×K with K=32 harmonics along each axis, i.e.
The output is given by a softmax attention based on the evaluation of a j at each input position (x i , y i ): with SA the spatial attention. In practice, as a j is periodic, we scale down (x, y) to keep a margin of 0.1 on each side. We then apply a spatial dropout by sampling a location (x drop , y drop ) and removing from the softmax each sensor that is within a distance of d drop of the sampled location. We then add a 1x1 convolution (i.e. with a kernel size of 1) without activation and with the same number D 1 of output channels. Finally, to leverage inter-subject variability, we learn a matrix M s ∈ R D1,D1 for each subject s ∈ [S] and apply it after the spatial attention layer along the channel dimension. This is similar but more expressive than the subject embedding used by Chehab et al. [2021] for MEG encoding, and follows decade of research on subject alignment [Xu et al., 2012, Haxby et al., 2020.
Residual dilated convolutions. We then apply a stack of five blocks of three convolutional layers. For the k-th block, the first two convolutions are applied with residual skip connections (except for the very first one where the number of dimension potentially doesn't match), outputs D 2 = 320 channels and are followed by batch normalization [Ioffe and Szegedy, 2015] and a GELU activation [Hendrycks and Gimpel, 2016]. The two convolutions are also dilated to increase their receptive field, respectively by 2 2k mod 5 and 2 2k+1 mod 5 (with k zero indexed). The third layer in a block outputs 2D 2 channels and uses a GLU activation [Dauphin et al., 2017] which halves the number of channels. All convolutions use a kernel size of 3 over the time axis, a stride of 1, and sufficient padding to keep the number of time steps constant across layers. The output of the model is obtained by applying two final 1x1 convolutions: first with 2D 2 outputs, followed by a GELU, and finally with F channels as output, thus matching the dimensionality of speech representations. Given the expected delay between a stimulus and its corresponding brain responses, we further shift the input brain signal by 150 ms into the future to facilitate the alignment between Y and Z.

Sample definition and preprocessing
M/EEG is generally considered to capture neural signals from relatively low frequency ranges [Hämäläinen et al., 1993]. Consequently, we first resampled all brain recordings down to 120 Hz with Torchaudio [Yang et al., 2021] and then split the data into training, validation, and testing splits with a size roughly proportional to 70%, 20%, and 10%. We define a "sample" as a 3 s window of brain recording with its associated speech representation. A "segment" is a unique 3 s window of speech sound. As the same segment can be presented to multiple subjects (or even within the same subject in Gwilliams et al. [2020]), the splits are defined so that one segment is always assigned to the same split across repetitions. We ensure that there is no identical sentences across splits, and checked that each sentence was pronounced by a unique speaker. Furthermore, we exclude all segments overlapping over different splits. For clarity, we restrict the test segments to those that contain a word at a fixed location (here 500 ms into the sample). M/EEG data can suffer from large artifacts, e.g. eye movements, or variations in the electro-magnetic environment [Hämäläinen et al., 1993]. To limit their impact, we apply a "baseline correction" (i.e. we subtract to each input channel its average over the first 0.5 s) and a robust scaler with scikit-learn [Pedregosa et al., 2011]. We clamp values greater than 20 after normalization to minimize the impact of large outlier samples. For the Mel spectrogram, we use 120 Mel bands (see Section A.2 in the Appendix) [Young et al., 2002], with a normalized STFT with a frame size of 512 samples and hop length of 128 samples, using audio sampled at 16kHz. We apply log-compression, i.e. log( + mel), with =10 −5 . When using wav2vec 2.0, we average the activations of the last four layers of its transformer. We use standard normalization for both representations. To further assess the gains from using a self supervised representation, we also test a "Deep Mel" variant, where we train a deep transformation of the Mel, with the same architecture as the one applied to the brain recording, without the spatial attention and subject layer, and matching the output dimension of wav2vec 2.0. This transformation is trained along with the brain decoder using the contrastive objective (3). By definition, the Deep Mel model only sees the audio from the each of the studied datasets (unlike wav2vec 2.0).

Datasets
We test our approach on four public datasets, two based on MEG recordings and two on EEG. We provide an overview of the main characteristics of the datasets on Table 1, including the number of train and test segments and vocabulary size over both splits. For all datasets, healthy adult volunteers passively listened to speech sounds (accompanied with some memory or comprehension questions to ensure participants were attentive), while their brain activity was recorded with MEG or EEG. In Schoffelen et al. [2019], Dutch-speaking participants listened to decontextualized Dutch sentences and word lists (Dutch sentences for which the words are randomly shuffled). In Gwilliams et al. [2020], English-speaking participants listened to four fictional stories from the Masc corpus [Ide et al., 2010] in two identical sessions of one hour. In Broderick et al. [2018], English-speaking participants listened to extracts of "The old man and the see". In Brennan and Hale [2019], English-speaking participants listened to a chapter of "Alice in Wonderlands". See Section A.1 in the Appendix for more details.

Training
One training epoch is defined as 1,200 updates using Adam [Kingma and Ba, 2014] with a learning rate of 3·10 −4 and a batch size of 128. We stop training when no improvement is observed on the valid set for 10 epochs and keep the best model based on the valid loss. For the direct regression of the Mel spectrogram, we use the MSE loss. We use two V100 GPUs with 16GB of memory.

Evaluation
Segment-level evaluation. In Figure 2, we estimate the Mel spectrogram from the model output. Given a segment and its matching audio (here the sentence "Thank you for coming Ed"), we retrieve the predicted distribution over the 1, 594 segments given by (2). We use this distribution to average the Mel spectrogram of each candidate segment. Similarly, the top-10 segment accuracy indicates whether the true segment is in the top-10 most likely segments according to the same probabilities.
Word-level evaluation. We also evaluate the model at the word level (Figure 4). For each word of the test set, we select a 3 s segment starting with this word. We input the model with the corresponding brain recordings, and output the probability distribution over all test segments including the true segment. To obtain the distribution over the vocabulary, we group the candidate segments by their first word and sum the probabilities within each group.

Accurately decoding speech from M/EEG recordings
Our model predicts the proper segment, out of more than 1,000 possible ones, with a top-10 accuracy of 72% and 67% for MEG datasets (top-1 accuracy of 44% and 36%) ( Table 2). For more than half of samples, the true audio segment is ranked first or second in the decoders' predictions. For comparison, a model that predicts a uniform distribution over the vocabulary ('random model') only achieves a 2% top-10 accuracy on the same MEG datasets. Decoding performance for EEG datasets is lower: our model reaches 19% and 31% top-10 accuracy. While modest, these scores are four times higher than the random baseline.

Effect of contrastive loss, deep speech representations, and number of participants
Our ablation highlights the importance of: (1) the contrastive loss, (2) the use of deep speech representations [Baevski et al., 2020] and (3) the combination of a large number of participants. First, a model trained to predict the Mel spectrogram with a regression objective ('base model' in Table  2) achieves 10% top-10 accuracy on average across datasets -i.e. nearly five times lower than our model, when using the model output to rank the candidate segments by cosine similarity.
Second, predicting the Mel spectrogram with a contrastive loss leads to a 3X improvement over the base model, and gains another 16 points by using wav2vec 2.0 as the speech representation. We verified that the wav2vec 2.0's latent representations provide higher decoding performances than those learnt end-to-end with contrastive learning, as shown by the results of the Deep Mel model on Table 2.
Third, to test whether our model effectively leverage the inter-individual variability, we trained it on a variable number of subjects and computed its accuracy on the first 10% of subjects. As shown in Figure 3B, decoding performance increases as the model is trained with more subjects on the two MEG datasets. This ability to learn from multiple subjects is strengthened by another ablation experiment: training on all participants, but without the subject-specific layer, leads to a drop of 17% accuracy on average across the four datasets (Table 3) Finally, other design choices modestly but significantly impact the performance of our model. Performance systematically decreases when removing skip connections, the spatial attention module, the initial or final convolutional layers (Table 3). We also show how essential clamping is to train the model, except for the [Gwilliams et al., 2020] dataset, which led to similar performances, although with a doubling of the training time. See Section A.2 in the Appendix for more ablations analyses.

Discussion
Here, we aim to decode natural speech from non-invasive brain recordings of healthy participants. Our results, based on the largest decoding study of M/EEG responses to speech to date, show that combining (1) a contrastive objective, (2) a convolutional architecture enhanced by a "Subject Layer", and (3) pretrained speech representations allows a zero-shot decoding of 3 s speech sounds up to 73% top-10 accuracy.  Speech decoding at scale. These results complement previous methods designed to decode speech from non-invasive brain recordings. For example, Sun and Qin [2016], Sree and Kavitha [2017], and Moinnereau et al. [2018] all developed a decoder to classify 11, 5 and 2 distinct imagined phonemes, respectively, from EEG signals. Similarly, Lopopolo and van den Bosch [2020], Chan et al. [2011], Nguyen et al. [2017 respectively developed a decoder to classify 6 distinct part-of-speech (48% accuracy), 10 words (83% accuracy) and 3 words (70% accuracy), from MEG signals. Finally, both Dash et al. [2020] and Wang et al. [2017] trained a classifier to decode 5 distinct sentences from MEG activity (both around 94% accuracy). Other approaches based on functional Magnetic Resonance Imaging (fMRI) have also been explored [Gauthier and Levy, 2019, Affolter et al., 2020, Pascual et al., 2022, Fernandino et al., 2022, but the low temporal resolution of fMRI signals appears to drastically limit the possibility of real-time decoding. Critically, all of these models were trained on individual subjects to categorize a very small number of highly-repeated categories and/or hand-crafted features [Ali et al., 2022, Jayaram and -an approach which is necessarily limiting given the combinatorics of language. By contrast, our model achieves zero-shot decoding by matching a large number of brain recordings to the deep representations of their corresponding speech sounds.
In the footsteps of intracranial studies. The present non-invasive study is limited to speech perception. It thus differs from the recent achievements obtained in a small set of heavily-trained patients implanted for clinical purposes and tasked to produce language [Herff et al., 2015, Martin et al., 2016, Angrick et al., 2019b, Willett et al., 2021, Moses et al., 2021, Angrick et al., 2021, Kohler et al., 2021. In particular, Willett et al. [2021] showed that a 1 s time window of neuronal activity in the motor cortex suffices to decode one of 26 characters with 94.1% top-1 accuracy during a spelling task. Similarly, Moses et al. [2021] showed that 4 s of neuronal activity recorded in the sensory-motor cortices is sufficient to decode the intention to communicate one of 50 words with a median word error rate of 25.6%. While our approach needs to be generalized to similar production tasks, the Figure 4: A. Single-word prediction for the first three subjects of Gwilliams et al. [2020] listening to the sentence "Thank you for coming, Ed". Text color indicates whether the decoded word is accurate. Text size is proportional to the log-probability output by our model. B. Top-1 accuracy at the word level (as explained in Section 3.4) as a function of the number of negatives during inference. C. The R 2 summarize how word frequency, part-of-speech tag, word embedding, and contextual embedding respectively predict the accuracy of single-word and single-segment decoding (Appendix A.4). Error bars are the SEM across participants.
possibility of leveraging data from multiple subjects and large natural language datasets, together with the multiplication of public neuroimaging datasets, makes us hopeful about the possibility of decoding word production from non-invasive brain activity. This possibility may benefit from hardware developments: although the MEG systems used here are not portable, high-temperature MEG sensors are now available and increasingly used [Boto et al., 2018]. Combined with A.I. systems, these wearable devices delineate a safe and operational path to diagnose, prognose and restore language processing in non-or poorly-communicating patients without the risks of brain surgery.
Societal impact Although these results hold great promise for the development of a safe and scalable system to help patients with communication deficits, the scientific community should remain vigilant that it will not be adapted to decode brain signals without the consent of the participants. This possibility appears unlikely at this stage: unlike other biomarkers, such as fingerprints, DNA and facial features, EEG and MEG signals cannot be acquired unbeknownst to the participants. Furthermore, teeth clenching, eye blinks and other muscle movements are known to massively corrupt these signals, and thus presumably provide a simple way to counter downstream analyses. In any case, we believe that open science remains the best way to responsibly assess risks and benefits in this domain.

A.4 Analyses of single-trial predictions
Does our model predict all words similarly? To address this question, we evaluate whether our model's ability to decode individual words depends on their properties, namely their zipf frequency as provided by Wordfreq 6 , as well as their part-of-speech tag and their word embedding as provided by spaCy 7 . Similarly, we evaluate whether the decoding of the entire 3 s speech segment varies with its linguistic properties, as assessed by its average word embedding as well as its sentence embedding, as computed with Laser 8 . For this, we trained a regularized ridge regression with scikit-learn 9 's default parameters to predict the softmax probability of the true word output by the decoder, given a feature. We then estimate the R 2 with a 5-split cross-validation: i.e. how well the feature predicts the probability of being selected by the decoder. The results, displayed in Figure 4-C, show that the word and segment embedding effectively explain the single-trial decoding accuracy. These results thus suggest that our decoder uses semantic and contextual information to make its predictions.

A.5 Decoding of isolated words
To what extent can our approach be used to decode words presented in isolation? To explore this issue, we evaluated our model using a subset from Schoffelen et al. [2019], where subjects are presented with random word lists. We use a segment ranging from -300 ms to +500 ms relative to word onset.
The results, displayed in Supplementary Figure A.1, show that our model reaches a top-1 accuracy of 25.0% with a vocabulary size of 50. While this performance is low, it is interesting to compare it to Moses et al. [2021] who report a top-1 accuracy of 39.5% with a model trained to decode the production of individual words, without the use of a language model, i.e. independently of context.