{'fraction_non_alphanumeric': 0.030302150756102517, 'fraction_numerical': 0.012916146634545612, 'mean_word_length': 4.76250209065061, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Inducing Meaningful Units from Character Sequences with Slot Attention

Characters do not convey meaning, but sequences of characters do. We propose an unsupervised distributional method to learn the abstract meaning-bearing units in a sequence of characters. Rather than segmenting the sequence, this model discovers continuous representations of the"objects"in the sequence, using a recently proposed architecture for object discovery in images called Slot Attention. We train our model on different languages and evaluate the quality of the obtained representations with probing classifiers. Our experiments show promising results in the ability of our units to capture meaning at a higher level of abstraction.


Introduction
When we look at a complex high-dimensional scene, we perceive its constituent objects, and their properties such as shape and material. Similarly, what we perceive when we read a piece of text builds on the word-like units it is composed of. Linguists call these units morphemes, the smallest meaningful units in a language.
In recent years, there has been an emerging interest in unsupervised object discovery in vision (Eslami et al. 2016;Greff et al. 2019;Engelcke et al. 2020). The goal is to segment the scene into its objects without supervision and ideally obtain an object-centric representation of the scene. These representations will lead to better generalization to unknown scenes, and additionally will facilitate abstract reasoning over the image. Most of this work achieves its goal via modeling the scene as a composition of objects and learning the latent object representations through an autoencoding objective. Eslami et al. (2016) use a recurrent neural network for predicting the latent properties of an object at every step and decode those properties to reconstruct the objects in the scene. Greff et al. (2019) use spatial Gaussian mixture model with amortized iterative refinement under the variational framework for learning the latent object representations. Recently, Locatello et al. (2020) propose an iterative attention-based algorithm called Slot Attention for discovering objects. They define a set of feature vectors (i.e., slots) which can bind to any object in the image by applying their proposed algorithm. Their method is simple and efficient to use, in comparison to the previous work in the field.
Inspired by this line of work in the image domain, we aim at finding meaning bearing units from a sequence of characters without supervision. More specifically, our goal is to learn a set of abstract continuous representations of objects in the text. We adapt the Slot Attention module (Locatello et al. 2020) for our purpose. To the best of our knowledge, we are the first to employ an unsupervised object discovery method in text for inducing meaningful units.
Our work is closely related to unsupervised morphology learning (Creutz 2003;Narasimhan, Barzilay, and Jaakkola 2015;Eskander et al. 2020) from the linguistic point of view, where the the smallest meaning bearing units in text correspond to morphemes. However, there are fundamental differences between our work and morphology learning. First, we learn a set of vector representations of text which are not explicitly tied to the text segments. Second, our model learns its set of vector representations by considering the entire input sentence, rather than individual space-delimited words.
We propose a model to encode the character sequence into slots, where each slot represents one meaningful unit in the sequence. As our unsupervised objective, a decoder conditions on the set of slots to reconstruct the original input. We use the Transformer architecture (Vaswani et al. 2017) as our starting point for this sequence to sequence model, adding a Slot Attention module for learning the hidden representation in between the Transformer encoder and decoder, as depicted in Figure 1.
Slot Attention has been tested on synthetic data with a limited number of objects (Locatello et al. 2020). To adapt Slot Attention to the domain of real text data, we make the following modifications. We train different parameters for each slot, and add a fixed amount of noise to their initialization. This gives our model the capacity needed to distinguish a large number of textual units, but prevents the model from using this capacity to learn arbitrary encodings of the text. We also design our model so that it can handle textual sequences with an unknown number of meaningful units. To this end, we postulate an adequate number of slots to support the longest sequence, and then add an L 0 regularizing layer on top of the Slot Attention module to prune out extra slots and retain only the necessary ones.
We evaluate what the slots have learned both by visualising the attention patterns of the trained models, and by employing probing classification tasks. In the probing tasks, we predict the corresponding Byte-Pair-Encoding (Sennrich, Haddow, and Birch 2016) tokens of the sequence from its slots. Additionally, we examine the slots' ability to predict the Morfessor (Virpioja et al. 2013) outputs, which are more linguistically motivated. Our experiments show promising results in the ability of slots to discover units which capture meaningful information at a higher level of abstraction than characters.
To summarize, our contributions are as follows: • We propose a novel model for learning meaning-bearing units from a sequence of characters by effectively adapting the Slot Attention method and integrating an L 0 regularizing layer to determine the required units (Section 2) • We analyze the resulting units by visualizing the attention maps of the decoder over the slots and observe the desired sparse and contiguous patterns among them (Section 4.2).
• We show that slots are able to discover units which capture meaningful information by probing their ability to predict predefined meaningful units (Section 4.3).

Problem Formulation
Given a sequence of characters X = x 1 x 2 . . . x N , we want to find a set of meaning-bearing units (slots) M = {m 1 , . . . , m K }, which could best represent X in a higher level of abstraction. As an example, consider the sequence "she played basketball". We expect our slots to represent the set of morphemes of the sequence, namely {she, play, -ed, basket, -ball}. This hypothesis comes from the linguistics perspective, where morphemes are considered the smallest meaningful units in a language.

Overview
We learn our representations through encoding the input sequence into slots and then reconstructing the original sequence from them. Particularly, we use an auto-encoder structure where slots act as the bottleneck between the encoder and decoder. Figure 1 shows a sketch of our proposed model. First, we encode the input character sequence by a Transformer encoder (Vaswani et al. 2017), which gives us one vector per character. Then, we apply a version of a Slot Attention module (Locatello et al. 2020) over the encoded sequence to learn the slots. Intuitively, Slot Attention will learn a soft clustering over the input where each cluster corresponds to a meaningful unit in the sequence. We integrate an L 0 regularizing layer, i.e., L 0 Drop layer (Zhang, Titov, and Sennrich 2020), on top of the slots to prune out the unnecessary ones. Since the number of slots is fixed during the course of training, this encourages the model to only use as many slots as necessary, and thereby stops the model from converging to trivial solutions, such as passing every character in a short sequence through a separate slot. Finally, the Transformer decoder reconstructs the input sequence autoregressively using attention over the set of slots.

Model
Encoder. We use Transformer encoder architecture for encoding our sequence (Vaswani et al. 2017). It consists of subsequent layers of self-attention and non-linearity for building a new representation of the input sequence. Finally, we obtain representation X = x 1 x 2 . . . x N from our input sequence X. Slot Attention for Text. After encoding the character sequence, we use our modified version of Slot Attention for discovering meaningful units of the input character sequence. Slot Attention is a recent method for unsupervised object representation learning (Locatello et al. 2020). It learns a set of feature vectors (slots) from the input representations by using an iterative attention based algorithm. Algorithm 1 shows the pseudo code of this method. Abstractly, in every iteration, the following steps are taken. First, an attention map is computed by slots acting as queries and inputs as keys. Then, this map is normalized over slots, which makes the slots compete for representing each token of the input. Afterwards, the slots' updates are computed as the (normalized) weighted mean over the attention weights and the input values. Finally, the slots get updated through a Gated Recurrent Unit (GRU) (Cho et al. 2014) followed by a residual MLP. This process iterates a fixed number of times.
In (Locatello et al. 2020), the slots are initialized randomly from a Normal distribution with shared learnable parameters µ and σ, i.e., slot i ∼ N (µ shared , σ shared ). (1) We found in our initial experiments that this initialization method does not lead to very good results in the text domain. In contrast with the experimental setting of (Locatello et al. 2020), which used artificially generated data with a Algorithm 1: Slot Attention module (Locatello et al. 2020). q, k, v map the slots and inputs to a common dimension D.
Require: inputs ∈ R N ×Dinput , slots ∼ N (µ, diag(σ)) ∈ R K×D slots inputs = LayerNorm(inputs) for i = 1 to T do slots prev = slots slots = LayerNorm(slots) attn = Softmax( 1 √ D k(inputs).q(slots) T , axis = 'slots') updates = WeightedMean(weights = attn + δ , values = v(inputs)) slots = GRU(states = slots prev, input = updates ) slots += MLP(LayerNorm(slots)) end for return slots small number and range of objects, real language requires learning about a very large vocabulary of morphemes, and each example can have a large number of morphemes. This suggests that a model for language needs a more informative initialization with more trained parameters, in order to have the capacity to learn about this large vocabulary and distinguish all the objects in a long sentence. To investigate this issue, we propose two other initializations for adapting Slot Attention to text.
To increase the capacity of the model to learn an appropriate initialization, we consider a separate µ per slot and we fix the σ to a predefined value for all the slots. Namely, the slots are initialized as By assigning a separate µ for each slot, the initialization has many more trained parameters. This allows the model to learn about different kinds of units, such as ones that occur at different positions, or ones that have different types of forms, but we do not make any assumptions about what those differences might be. However, since these µ parameters are learnable, we need to fix the number of slots before training them and use the same number of slots at test time.
In addition, the intuition behind fixing the σ is to force the slots to compress the information in a meaningful way. Since the number of possible n-grams in text is finite but the slots can have any continuous value in the space of R D slots , the slots tend to learn an arbitrary mapping from n-grams in the input to the slots, while turning σ to zero. Thus, there is no need for the slots to learn the underlying meaningbearing units. Therefore, by imposing a constant noise on slots through the constant σ, we limit the information which can be passed through each slot, from the information theoretic point of view.
As an alternative to having trained parameters for each slot, we could still distinguish all the slots at initialization simply using position embeddings. This would hand-code the assumption that meaningful units are distributed across the string and are local in the string. Thus, we try an alternative initialization of the following form.
where PE(i) is the positional encoding of the ith slot, using the same positional encoding function as inputs and distributing the slots evenly across the input sequence. With this formulation, we explicitly inject positional information into the slots. Moreover, we let the model to learn a shared σ, instead of fixing it, to increase the capacity of our model, since µ is also shared among all the slots. With either initialization method, we then obtain the set of slots M as Neural Sparsification Layer: L 0 Drop. The number of slots needed to represent a sequence varies among different sequences in the data. Thus, we consider an upper-bound over the number of required slots and prune the extra ones per input sequence. We accomplish this by using a neural sparsification layer called L 0 Drop (Zhang, Titov, and Sennrich 2020). This will allow our model to dynamically decide on the number of required slots for every input sequence. This layer consists of binary-like gates g = g 1 . . . g K that for every input m i works as Each gate is a continuous random variable in the [0, 1] interval, sampled from a hard-concrete distribution (Louizos, Welling, and Kingma 2018). This distribution assigns most of its probability mass over its endpoints (i.e., 0 and 1) in favour of the sparsification goal. A sample g i from this distributaion is obtained from stretching and rectifying a sample from the BinaryConcrete distribution (Maddison, Mnih, and Teh 2017;Jang, Gu, and Poole 2017): where β and are hyperparameters and α i is predicted as a function of the encoder output m i : where w is a learnable vector. This will allow the model to dynamically decide which inputs to pass and which ones to prune. Finally, the L 0 penalty, which yields the expected number of open gates, is computed as where the probability of g i being exactly 0 is provided in closed form in Louizos, Welling, and Kingma. We follow the same approach as Louizos, Welling, and Kingma at evaluation time and consider the expectation of each gate as its value.
We refer to the pruned slots after applying the L 0 Drop layer as M = m 1 . . . m K . In contrast to Zhang, Titov, and Sennrich (2020), we do not aggregate the pruned inputs at decoding time since the efficiency gain in short sequences is negligible.
Decoder. Lastly, we regenerate the input sequence from the set of slots by using a simple, shallow decoder. To this end, we use a one-layer Transformer decoder (Vaswani et al. 2017) with a single attention head over the slots. A simple decoder forces the slots to learn representations with a straightforward relationship to the input, which we expect to be more meaningful. In other words, we do not use a powerful decoder because it would be able to decode even low quality representations of the input, which are less meaningful.

Training Objective
All components of our model are fully differentiable and hence, we can train it end-to-end. We use the Gumble trick for sampling HardConcrete variables (Maddison, Mnih, and Teh 2017;Jang, Gu, and Poole 2017). We train our model with the following objective: which consists of the reconstruction loss from the decoder (L rec ) and the L 0 penalty for the open gates. Hyperparameter λ, the sparsification rate, controls the ratio between the two losses. In practice, we find that in order to impose enough sparsity in the slots, we should slightly increase λ during the course of training using scheduling techniques.

Related Work
Unsupervised Object Discovery. There is a recent line of research in the image domain for discovering objects in a scene without explicit supervision, and building an objectcentric representation of them. Most of this work is built around the idea of compositionality of the scenes (Greff et al. 2016;Greff, Van Steenkiste, and Schmidhuber 2017;Van Steenkiste et al. 2018;Greff et al. 2019;Burgess et al. 2019;Engelcke et al. 2020;Locatello et al. 2020;Emami et al. 2021). MONet ) uses a recurrent attention network for providing the location masks of the objects in the image. It then reconstructs every object independently using a Variational AutoEncoder (Kingma and Welling 2013) conditioned on the image and its location mask. GENESIS (Engelcke et al. 2020) proposes a more advanced architecture which models the masks as latent variables (in addition to objects) and considers an autoregressive prior over them to capture the dependencies between the components. Greff, Van Steenkiste, and Schmidhuber per object and explicitly learn the position and appearance of the objects in the scene. Lately, Locatello et al. (2020) propose to learn object representations (slots) through an iterative attention-based algorithm (namely Slot Attention) over the input features.
In contrast to this line of work in vision, our approach is specifically designed for text. We use additional components in our architecture to resolve the requirements of modeling textual data. Furthermore, our model is trained and evaluated on real text datasets, in contrast to these previous models which have only been shown to be effective on synthetic scenes. Unsupervised Morphology Learning. Unsupervised morphology induction is the task of identifying the constituent morphemes of a word given only the distributions of character sequences in the language, and has been studied for many years (Elman 1990;Creutz and Lagus 2002;Baroni, Matiasek, and Trost 2002). Morphemes have strong linguistic motivations, and are practically important in many downstream tasks because they are the smallest meaning-bearing units in a language (Can and Manandhar 2014).
Many approaches have been proposed for discovering the underlying morphemes or morpheme segmentations. Morfessor variants are based on probabilistic machine learning methods (MDL, ML, MAP) for morphological segmentation (Creutz and Lagus 2002;Creutz 2003;Creutz andLagus 2005, 2007;Virpioja et al. 2013). Some researchers take a Bayesian approach for modeling word formation (Poon, Cherry, and Toutanova 2009; Narasimhan, Barzilay, and Jaakkola 2015; Bergmanis and Goldwater 2017; Luo, Narasimhan, and Barzilay 2017). Adaptor Grammars are another approach for modeling morphological inflections (Sirts and Goldwater 2013;Eskander, Rambow, and Yang 2016;Eskander, Klavans, and Muresan 2019;Eskander et al. 2020). In addition, Xu et al. (2018Xu et al. ( , 2020 built their models upon the notion of paradigms, set of morphological categories that can be applied to a set of words. Moreover, Soricut and Och (2015);Üstün and Can (2016) extract morphemes by considering the semantic relations between words in the continuous embedding space. Cao and Rei (2016) propose to learn word embeddings by applying a bi-directional RNN with attention over the character sequence. They hypothesize that morpheme boundaries will attract most of the attention weights. Furthermore, Ataman, Aziz, and Birch (2020) model word formation as latent variables which mimic morphological inflections in the task of machine translation.
Our work differs from the previous work in classical morphology learning in two ways. First, instead of explicitly discovering morphemes, we learn a set of continuous vector representations of the input, which can then be processed to extract the morphemes. The model itself has no explicit relation between these unit representations and segments of the input. Second, our model learns representations of an entire input sentence, rather than individual space-delimited words. This makes fewer assumptions about morphemes, and considers the context of the words in a sentence. Our work is similar to Ataman, Aziz, and Birch (2020) in modeling morphology implicitly in the latent space. However, we employ a self-supervised objective for our purpose which is more general compared to their supervised loss, as we do not need labeled data to train our model. Unsupervised Character Segmentation. Learning to segment a character sequence in unsupervised fashion is another relevant area to our work. Chung, Ahn, and Bengio (2017) propose Hierarchical Multi-scale RNNs for modeling different levels of abstractions in the input sequence. Specifically, at each time-step, the transition to the upper layer is done via a segmentation decision in the current layer. In the language modeling task, they observe that the first layer is roughly segmenting the sequence into words, namely at space boundaries. Sun and Deng (2018) propose Segmental Language Models for Chinese word segmentation. Moreover, in (Kawakami, Dyer, and Blunsom 2019), the authors design a model to learn the latent word segments in an unsegmented character sequence with a language modeling objective. As we mentioned earlier, we learn continuous vector representations of text which is different from explicitly detecting discrete character segments. Subword Discovery Algorithms. This set of algorithms have become a standard component of NLP models in recent years. Byte-Pair-Encoding (BPE) (Sennrich, Haddow, and Birch 2016) is an iterative algorithm which merges the two consecutive tokens with the highest frequency in every step until it reaches the desired vocabulary size. Word-piece (Schuster and Nakajima 2012), sentence-piece (Kudo and Richardson 2018) and unigram LM (Kudo 2018) are other similar subword tokenization algorithms. In contrast to these methods, which mostly use local statistical information of the data, our model is trained over complete sentences to learn a sophisticated representation. Moreover, as we stated previously, we learn abstract continuous units which are not limited to segmentations of the string and are not explicitly mapped to subwords.

Experiments
In this section we will explain our experimental setup and results. We evaluate our unsupervised model both qualitatively and quantitatively. First, we visualize some of the attention maps to show what the slots are corresponding to (4.2). Afterwards, we probe the slots' vectors ability to capture meaningful information by two classification tasks (4.3). Finally, we analyze different slot initialization distributions (4.4).

Experimental Setup
Languages and Data. We apply our model to languages from different morphological typologies. We select English (EN), German (DE), Spanish (ES) and Czech (CS) from the fusional family and Finnish (FI) from the agglutinative typology. For English we use the raw Wikitext2 dataset provided by Merity et al. (2017). For the rest of languages we use Multilingual Wikipedia Corpus (MWC) (Kawakami, Dyer, and Blunsom 2017). We lowercased the text and retained the characters which occur more than 25 times in the corpus, following Kawakami, Dyer, and Blunsom (2017). We replace the low-frequent characters with an unknown placeholder.
Training Settings. We use a standard Transformer architecture (Vaswani et al. 2017) with model dimension 256. The encoder consists of 2 layers with 4 self-attention heads and the decoder consists of 1 layer with 1 self-attention head and 1 attention head over the slots. We use the same positional encodings as in (Vaswani et al. 2017). We feed in the sentences with less than 128 characters to our model and consider the number of slots as 64 (half of the maximum input length). In addition, we take the dimension of slots as 128, and run the algorithm for T =1 iterations. 1 We initialized the slots according to equation (2) in Sections 4.3 and 4.2.
We scheduled the λ parameter in the training loss to start with a low value of 2 × 10 −5 and exponentially increase it every 10 epochs until it reaches a certain limit. We control this parameter in a way that the final number of open gates roughly equals the average number of BPE tokens in a sequence. We used Adam optimizer (Kingma and Welling 2013) for training our models with learning rate 10 −4 and train our models for 200 epochs. More details of the settings are available in the Appendix.

Visualization
In order to show some qualitative results of our model, we visualize the attention maps for generating every output, shown in Figure 2. In particular, we show the attention of the decoder over slots when generating every output character. Interestingly, although we do not impose any sparsity in the decoder's attention weights, the attention maps are quite sparse. Namely, at each generation step only a few slots are attended, and each slot is attended while generating only a few characters. In addition, although we do not impose any bias towards discovering segments of the input, the characters which are generated while attending to a given slot are contiguous in the string (the vertical bands in Figure 2). We believe that the emergence of contiguous spans is a result of the bottleneck we create with the slots and L 0 Drop layer. This means that the model is trying to put correlated information about the input in the same vector, so that it can represent the string more efficiently. The strongest correlations in the character string are local in the input, so each slot tends to represent characters which are next to each other in the input.
In the early steps of training, when the sparsity ratio (λ) is small, each slot tends to represent a bigram of characters (2a) and later on, trigrams (2b). These observations confirm the necessity of the L 0 Drop layer for converging to better units. As the ratio increases, the number of active slots reduces and they become more specialized in representing contiguous meaning-bearing segments of input. For instance, the word cooking in 2c is represented by two slots cook and ing. That these segments roughly correspond to the morphemes which we want the model to discover, is verified quantitatively in the probing experiments in Section 4.3.  Figure 2: Attention of the decoder over slots (x-axis) for generating every output (y-axis) during the course of training. The vertical axis shows the target character output and the horizontal axis shows the slots. The target output sequence is "the red colour associated with lobsters only appears after cooking.".
Learned Positional Information. We observed similar attention patterns for the slots across different input samples and thereby, we conjecture that there must be a correlation between what the slots have learned and the corresponding positions in the input sequence. To evaluate this, we averaged the attention maps between all the samples from the test set and visualize them in Figure 3. The averaged attention map verifies our hypothesis that the slots are highly correlated with the position of characters in the sequence. However, they still have the flexibility to move the boundaries of which segment they represent in accordance with the input.   Probing Tasks. We define two probing tasks. First, we measure how well the slots match to the corresponding BPE tokens in the sequence (Sennrich, Haddow, and Birch 2016) as frequency-based units. In addition, we consider Morfessor (Virpioja et al. 2013) as a linguistically inspired method for discovering the morpheme segments, and measure how well the slots correspond to the outputs of the Morfessor tool.

Probing Analysis
Probing Classifier. We train a probing classifier for mapping a slot's vector to the output vocabulary, namely, f (m i ) : R D slots → R S , where S is the number of BPE or Morfessor tokens.We apply the classifier with shared parameters over each of our slots and obtain a set of predictions, i.e., {f (m 1 ), f (m 2 ), . . . , f (m K )}. As we are dealing with a set, during training we need to find a one-to-one matching between the classifier's predictions and the target tokens. Therefore, we use the Hungarian matching algorithm (Kuhn 1955) for finding the best match in terms of minimizing the classification loss. Consider the best matching as i j → j, which matches the i j th slot with the jth output (i.e., y j ). We then compute the loss as L = K j=1 l(f (m ij ), y j ), where l is the cross-entropy between the two terms.
We consider the complete set of slots after applying the L 0 Drop layer as the inputs to our classifier. Slots whose L 0 Drop gate is closed are simply input as zero vectors. This gives us a fixed number of vectors. Because the two sides of matching should have the same size to obtain a one-to-one match, we add an extra target label (i.e., empty) for representing the pruned slots. Due to the fact that many slots are pruned out, considering a measure like accuracy could be misleading, since a classifier which outputs empty label will achieve very high accuracy. Therefore, we build a confusion matrix as follows. We consider all non-empty labels as positive and the empty ones as negative, and we report precision (P), recall (R) and F1 measure, to better reflect what the slots have learned.  Our probing classifier consists of two fully connected layers with ReLU activation function in between the two layers. The hidden dimension of the classifier is the same as the slots' dimension, which is 128. We use the same datasets as our main model and train BPE with vocabulary size of 5000 for all languages. For Morfessor, we use the pretrained model and consider the set of its outputs on the training data as our target labels.
To provide a baseline for evaluating our trained slot representations, we train the probing classifier to interpret the vectors produced by a random model. We initialize our model randomly and use it to generate untrained slot vectors, which are then used to train the probing classifier.
Probing Results. Table 1 shows the results of the probing tasks over the baseline (untrained) and the unsupervised learned slots (slot-attn) on different languages. As the results show, the trained slots achieve much higher performance in both tasks in comparison to the random baselines. Our model achieves high precision in predicting the nonempty labels. However, its performance is weaker on the recall side, which is probably due to the imbalance between empty and non-empty labels in the training set. Particularly, the empty labels comprise around 66% of the data which highly biases the model towards predicting them. This imbalance shows its effect mainly on the random baselines and results in predicting the empty label for almost all of the samples. This behavior leads to high precision but very low recall. Therefore, the gap between the learned slots and the random ones is especially obvious on the recall side, where the learned slots guide the classifier to predict non-empty labels. Since the trained model outputs on average the same number of non-zero slots as the number of BPE tokens (due to closed gates in the L 0 Drop layer), the probing classifier usually only has one slot from which it can predict any given target token, which also reduces recall. There is no clear difference in our model's performance on BPE or on Morfessor, and it performs equally well on both the agglutinative language (Finnish) and the fusional languages. We further  show some examples from the predictions of the classifiers in Table 3 in the Appendix. These results verify that our proposed model is effectively finding meaning-bearing units (i.e., frequency-based from BPE and linguistically inspired from Morfessor) without any supervision. When the probing classifier predicts a specific target unit from a slot, it is almost always exactly correct (high precision). And for the majority of target units (about two-thirds), there is a specific slot from which the probing classifier succeeds in predicting that exact unit (recall). Since previous work has shown that these target representations are effective ways to capture meaning for downstream tasks, and since a large percentage of these units can be predicted from our induced units, there is every reason to believe that the units discovered by the proposed model will also be effective meaning-bearing units for downstream tasks. We leave the empirical verification of this effectiveness to future work. Figure 3 shows that many individual slots have a strong preference for representing segments in specific positions in the input. Perhaps it would be sufficient to initialize slots with position embeddings, and thereby avoid learning specific parameters for each slot, as in equation (3) above. We analyse the effect of this more constrained form of slot initialization in this section. We also compare to the original definition of Slot Attention, where slots are randomly initialized from a single shared distribution (Locatello et al. 2020). Table 2 shows the probing results of our model under different slot initializations. Having a separate µ for each slot (Eq 2) increases the capacity of the model and thus yields better results in comparison to sharing µ between the slots. The model especially achieves higher recall in this case, which implies better recovery of the BPE and Morfessor tokens. However, we achieve similar results with fewer parameters when initializing slots with positional information (Eq 3). This positional information leads to large improvements compared to the model trained without it (Eq 1). But there is still a substantial performance gain from using our full model, with different µ per slot, compared to just using positional information.

Slot Initialization Analysis
In this paper, we propose a model for discovering meaningful units in text in an unsupervised fashion. We use an autoencoder architecture for encoding a character sequence into a set of continuous slot vectors and decoding them to reconstruct the original sequence. We enforced the set of slots to act as a bottleneck in transferring information between the encoder and the decoder by adding a constant noise to their vectors and adding an L 0 regularizing layer on top of the slots which only retains the necessary vectors. We evaluate our model by probing the final slot vectors to predict predefined BPE and Morfessor tokens. In comparison to a random baseline, our representations effectively capture meaningful information in both experiments. Further analysis confirm that previously proposed and plausible alternative models do not perform as well.
In future, we plan to employ our representations in possible downstream tasks such as dependency parsing. We anticipate that these meaningful units could be beneficial in improving the performance of these tasks.