{'fraction_non_alphanumeric': 0.03696059004843681, 'fraction_numerical': 0.01343020695728754, 'mean_word_length': 4.392046297670277, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Mechanism for Producing Aligned Latent Spaces with Autoencoders

Aligned latent spaces, where meaningful semantic shifts in the input space correspond to a translation in the embedding space, play an important role in the success of downstream tasks such as unsupervised clustering and data imputation. In this work, we prove that linear and nonlinear autoencoders produce aligned latent spaces by stretching along the left singular vectors of the data. We fully characterize the amount of stretching in linear autoencoders and provide an initialization scheme to arbitrarily stretch along the top directions using these networks. We also quantify the amount of stretching in nonlinear autoencoders in a simplified setting. We use our theoretical results to align drug signatures across cell types in gene expression space and semantic shifts in word embedding spaces.


Introduction
Modern deep networks are capable of learning rich latent spaces, i.e., vector spaces that capture similarity between data points (Ch. 15 of [11]). These learned structured latent spaces play an important role in the success of several downstream inference tasks such as unsupervised clustering [18], topic modeling [8], and classification in low-data regimes [26].
For many downstream learning tasks such as those in Figure 1, a structured latent space in which meaningful semantic directions are aligned is particularly useful. For example, alignment of semantic directions (as in Figure 1a) can be useful for identifying word analogies or performing word arithmetic with commonly used word embeddings such as GloVE [21] and word2vec [25].
Recent work [5,15] empirically demonstrated that autoencoders are capable of automatically producing aligned spaces when embedding gene expression data, but the mechanism through which neural networks produce aligned latent spaces has been unclear.
The main finding of this work is that autoencoders can produce aligned latent spaces by stretching along the top left singular vectors of the data. The following are our contributions. 1. We prove that stretching along the top singular vectors of the data leads to alignment in the latent space. We analyze the case of a mixture of Gaussians with unaligned major axes and prove that stretching along the top left singular vectors aligns these axes.
2. We prove that asymmetrically initialized linear autoencoders stretch along the top left singular vectors of the data, thereby producing aligned latent spaces. We derive the stretching factor through a gradient flow analysis. In a simplified setting, we prove that, under low rank initialization, non-linear autoencoders also stretch along the top left singular vectors of the data.
3. We apply our theory to produce autoencoders on GloVe embeddings [21] that strengthen alignment of meaningful semantic directions. The resulting aligned latent spaces provide an improvement on word analogy tasks. We lastly apply our theory to produce autoencoders that align drug signatures across cell types on gene expression data from CMap [29].

Related Work
Prior work [15] demonstrated empirically that variational autoencoders [13] can align the effect of perturbations across multiple cell types in single-cell gene expression data. In particular, the work demonstrated that under-parameterized, non-linear autoencoders automatically align the effect of stimulating cells by a single perturbation (IFN-β) in the latent space. Follow-up work [5] demonstrated that over-parameterized 1-hidden layer autoencoders with leaky ReLU [31] activation align the effect of multiple perturbations between cell types from the connectivity map (CMap) [29]. In this work, we prove that over-parameterized, 1-hidden layer autoencoders produce aligned latent spaces by stretching along the top left singular vectors of the data. Several recent works [3,4,19] study the behavior of over-parameterized neural networks in supervised learning. While there are infinitely many interpolating solutions in the over-parameterized regime, these works demonstrated that an implicit bias through training encourages solutions that generalize. Further works [33,24,23] characterized the implicit bias of over-parameterized autoencoders in unsupervised settings. In particular, [23,24] demonstrated that over-parameterized autoencoders learn solutions that are contractive around the training examples. These works, however, did not analyze the latent space produced by overparameterized autoencoders.
It is well-known that linear autoencoders under a balanced spectral initialization [1,27,28,10] learn the subspace spanned by the top principal components of the data [2,22]. In this work, we analyze overparameterized linear autoencoders under an asymmetric spectral initialization, i.e. one layer is initialized at 0 and the other is initialized using a matrix polynomial in the data. Importantly, unlike prior spectral initialization analyses, our asymmetric initialization is easy to compute in practice. In order to derive the solution at the end of training in linear autoencoders under spectral initialization, we use a gradient flow analysis, i.e. gradient descent with infinitesimal step size [10,23,9].  singular value and right singular vector of B. Let q 1 , q 2 ∈ R d be unit vectors. Then, Lemma 1 demonstrates that the spectrum of B governs whether two directions become more aligned. For example, if σ 1 is much larger than σ i for i > 1 and the top eigenvector of B has roughly similar cosine similarity between the two directions (i.e cos(q 1 , v 1 ) ≈ cos(q 2 , v 1 )), then q 1 and q 2 will become more aligned in the latent space. In Figure 2, we give an example of a matrix B that aligns the major axes of multivariate Gaussians by stretching along the direction [1,1] In this work, we identify a class of matrices B that provably boost alignment for directions of interest. We begin with a characterization of such B for Gaussian mixture models in the next section.

Stretching Along Left Singular Vectors Leads to Alignment
In this section, we will show that stretching along the top left singular vectors of the data can boost alignment along directions of interest. We begin by using the power method [30] to provide intuition as to why stretching along singular vectors of the data leads to alignment.
Intuition from the Power Method. The power method [30] provides a mechanism for producing trivial alignment of directions in data. Namely, repeated multiplication of data X ∈ R d×n by a polynomial in XX T will project the data onto the top eigenvector of XX T . In this setting, the embedded points will lie on a line and thus, the cosine similarity between the difference of pairs of unique points will be either 1 (perfectly aligned) or −1 (perfectly anti-aligned). 2 Fully projecting the data onto this top eigenvector is undesirable since this projection is not invertible and thus, we cannot decode the embedding back to the original input space. However, a partial iteration of the power method using XX T will stretch along the top eigenvector of XX T while yielding an invertible map. We will demonstrate that using the power method to partially stretch along this top eigenvector is beneficial for aligning meaningful directions. We begin by proving that stretching along this top eigenvector aligns the major axes of Gaussians in a mixture model.

Identifying the Top Singular Vector of a Gaussian Mixture
We will first characterize the top eigenvector and eigenvalue of XX T drawn from a mixture of Gaussians, and then use this result to quantify the increase in alignment after stretching along this top eigenvector. To aid our analysis, we choose a simple setting where we can explicitly compute the eigenvectors and eigenvalues of 1 n XX T as n → ∞ (Proof in Appendix B).
By Lemma 2, multiplication by 1 n XX T will stretch along the direction of the average of the means. We next verify that stretching along this direction leads to an alignment of major axes in a mixture of Gaussians.

Quantifying Alignment in Gaussian Mixture Models
We now consider a setting with two Gaussians in R 2 and use Lemma 2 to prove that stretching this mixture of Gaussians along the average of the means will align the major axes of the individual Gaussians (proof in Appendix C).
Consider the embedding BX, where B ∈ R d×d has eigenvectors u 1 and u 2 and corresponding eigenvalues α and β where α 2 > β 2 . Let q Σ1 and q Σ2 be the top eigenvectors of Σ 1 and Σ 2 . Then the cosine similarity of q Σ1 and q Σ2 before and after embedding with B is: Thus, if B stretches along the top left singular vectors of X, then the major axes of individual Gaussians in the mixture become more aligned. As is shown by the following corollary, setting B = ( 1 n XX T ) k , for some degree k ≥ 1 leads to alignment (Proof in Appendix D).
Remarks. We empirically verify this result in Appendix D. The above corollaries imply that stretching along the top singular vector of the data aligns the major axes of the individual Gaussians. In general, however, the degree of alignment depends on how aligned each direction of interest is to the overall stretching direction. As a simple example, in the above setting, stretching instead along the major axis of one of the Gaussians would not significantly boost alignment. Indeed, the ideal direction, v, to stretch along is one where semantically meaningful directions are roughly equally aligned with v. Consistent with the result of this section, we show in Section 6 that stretching along the top eigenvector of XX T leads to alignment of semantically meaningful directions on large datasets.

Alignment with Over-parameterized Autoencoders
In this section, we prove that over-parameterized, 1-hidden layer linear autoencoders trained using gradient flow from an asymmetric spectral initialization produce embeddings that are stretched along the top left singular vectors of the data. We then prove that a similar phenomenon occurs for non-linear 1-hidden layer ReLU autoencoders trained on two data points. We lastly connect this stretching to alignment of directions of interest in the latent space by demonstrating that asymmetrically initialized autoencoders align the major axes of a Gaussians in the mixture model from Section 4. We begin with a review of autoencoder preliminaries.

Autoencoder Preliminaries
We begin with the training setup for vanilla autoencoders.
Definition 1 (Autoencoding). Given a dataset X ∈ R d×n with n training examples and neural network where x (i) denotes the i th column of X.
In this work, we will reference the following spectral initialization scheme used in [10,27,28].
Definition 2 (Spectral Initialization). Let f (x) = ABx denote a linear autoencoder with x ∈ R d , A ∈ R d×k , and B ∈ R k×d . We assume gradient descent is used to solve Eq. (1) on dataset X and let A (0) , B (0) denote the parameters at initialization. The spectral initialization corresponds to B are positive, semi-definite diagonal matrices.
The spectral initialization scheme is commonly used to understand training dynamics in neural networks. Assuming symmetric spectral initialization (i.e. Σ

Stretching in Over-parameterized Linear Autoencoders
We begin by deriving the gradient flow solution for 1-hidden layer autoencoders from arbitrary spectral initialization [10,27,28]. By exploring how initialization impacts the singular values of the gradient flow solution, we derive a simple, computationally tractable asymmetric spectral initialization scheme that yields stretching along the top left singular vectors of the training data.
Theorem 1. Let f (x) = ABx denote a neural network with A, B ∈ R d×d and x ∈ R d , and let X ∈ R d×n denote the training data. If A, B are initialized according to the spectral initialization (Definition 2), then gradient flow on A, B converges to: The full proof is provided in Appendix E; it is inspired by the proof strategy used in [23]. We now provide an initialization scheme that leads to alignment in the latent space. Asymmetric Spectral Initialization. In order to generate aligned embeddings, we now turn to the following asymmetric spectral initialization. Provided that B (0) is a polynomial in 1 n XX T + I and A (0) = 0, Theorem 1 guarantees that B stretches inputs along the top directions given by the top left singular vectors of the training data, X. This is formalized in the following corollary.
Corollary 3. Let f (x) = ABx denote a neural network with A, B ∈ R d×d and x ∈ R d , and let U X Σ X V T X denote the SVD of X. Let A (0) = 0 and B (0) be a polynomial in 1 n XX T + I. Let σ i (B (t) ) denote the i th singular value of B (t) . Then B (∞) given by gradient flow satisfies: Remarks. From Corollary 3, we find that B (∞) after gradient descent from an asymmetric spectral initialization will stretch along the top singular vectors even more than B (0) . We note that this property does not necessarily hold for symmetric spectral initialization. Moreover, we can achieve an arbitrary amount of stretching along the top singular vectors of the data by scaling the degree of the matrix polynomial used to initialize B (0) . Finally, unlike general spectral initialization, our asymmetric initialization above does not require computing any singular vectors, and hence is readily usable in practice.

Aligning Latent Spaces with Linear Autoencoders
Returning to the example of a Gaussian mixture from Section 4, we now use Theorem 1 to prove that asymmetrically initialized linear autoencoders can promote alignment. Specifically, we revisit the setting of Corollary 1, which consisted of two Gaussians whose major axes are nearly orthogonal. By generating B from our linear autoencoder, we find that autoencoding will align the major axes of these Gaussians (Proof in Appendix F).

Corollary 4.
In the setting of Corollary 1, set B to be the encoding matrix of an autoencoder f (x) = ABx after gradient flow where A is initialized to 0 and B is initialized to ( 1 n XX T ) k . Then, as n tends to infinity, Thus the alignment of q 1 and q 2 will increase, and this alignment can be arbitrarily boosted by encoding with higher degree k.
Since autoencoders with asymmetrical spectral initialization stretch along the top directions of the dataset, we can use them to boost alignment while allowing for perfect reconstruction via decoding. In Section 6, we explore how linear autoencoders can be used to align real-world datasets.

Stretching in Over-parameterized Nonlinear ReLU Autoencoders
In the previous section, we proved that asymmetrically initialized linear over-parameterized autoencoders stretch along the top left singular vectors of the data. We now provide a proof that the same is true for asymmetrically initialized over-parameterized ReLU autoencoders in a simplified setting with two opposing datapoints. We use gradient flow to compute the closed forms of A (∞) , B (∞) . Since the dataset [−x, x] has exactly one singular vector with non-zero singular value (i.e x), we compute the stretching factor as the magnitude of the new embedding of x (i.e ||Bx|| 2 ).
Theorem 2. Let f (x) = Aφ(Bx) be a neural network with A ∈ R d×2d , B ∈ R 2d×d , and φ as a ReLU activation. For some x ∈ R d with η = ||x|| 2 2 , let X = {x, −x} denote the training set. Let A and B be initialized as Then after using gradient flow, we have that A (∞) = α(B (0) ) T and B (∞) = βB (0) , where Moreover, gradient flow will stretch the embedding of x by a quadratic factor of η: The full proof can be found in Appendix G, but proceeds through gradient flow analysis similar to Theorem 1. As in the linear setting, we find that by asymmetrically initializing A and B, we can stretch the embedding of the dataset along the top left singular vector.

Applications
Thus far, we have demonstrated that autoencoders can align directions of interest in the latent space by stretching along the left singular vectors of the data. While we primarily focused on proving this for the case of a Gaussian mixture model, we now apply our theoretical results to produce autoencoders that align semantic/syntactic directions in word embedding space and drug signatures in gene expression space.

Alignment of GloVe Embeddings
In this section, we apply our theory to align semantic/syntactic directions in GloVe word embeddings [21]. We use 300 dimensional GloVe vectors that were trained on Common Crawl with 840 billion tokens. We consider the embeddings of the 500,000 most frequent words, and encode these embeddings with linear and non-linear autoencoders. Details about our experimental setup and training procedures can be found in Appendix H.

Alignment of Meaningful Directions
We first consider how autoencoders can align meaningful directions in the GloVe embedding data. We consider word relationships from the Google analogy dataset [16], which consists of 19,544 question pairs We study how stretching the top singular vectors of the data through autoencoders can align these directions. Prior work suggests that the top singular vectors of GloVe data are closely related to the frequency of a word [17]. We confirm this finding by plotting, for word embedding a, the frequency ranking of that word against its cosine similarity with the singular vector u 1 (Appendix I.1). Recall from Lemma 1 that, to boost alignment, we would like the the stretching axis to be roughly equally aligned with the directions of interest. Since we consider relationships between words which are independent of word frequency, we suspect that stretching along the top singular vectors of the GloVe will allow us to align word relationships in the data.
To evaluate alignment in these directions, we randomly draw two word pairs (a 1 , b 1 ) and (a 2 , b 2 ) which capture the same relationship. We then plot the cosine distances of b 1 − a 1 and b 2 − a 2 before and after autoencoding (Appendix I.2). We find that autoencoding with either a linear or nonlinear autoencoder boosts alignment of these semantic directions in the latent space.
We can further visualize this alignment by visualizing the latent space before and after autoencoding with multidimensional scaling (MDS). In Figure 3, we visualize the alignment of a subset of the pairs in the relationship "negation" (e.g aware → unaware). We find that autoencoding with both linear and non-linear autoencoders is able to align the direction of negation. We depict the full MDS visualization for negation and demonstrate alignment of other relationships in Appendix I.3.

Word Analogy Metrics
To evaluate alignment on a more quantitative level, we study the effect of autoencoding on word analogy tasks. We consider a list of 4-tuples of the form (a 1 , b 1 , a 2 , b 2 ) which have the analogy relationship a 1 :b 1 ::a 2 :b 2 . One such example might be the embeddings of (men, women, king, queen).
We evaluate two metrics for word analogy alignment. In the first, analogy, we simply calculate the average cosine similarity cos(a 2 − a 1 , b 2 − b 1 ). In the second, analogy-accuracy, we evaluate whether b 2 = argmax b * cos(b * , a 2 − a 1 + b 1 ) following [14]. We form the 4-tuples by combining every possible pair within each relationship in the Google Analogy dataset [16].
In Table 1, we find that autoencoding substantially boosts the analogy metric, as semantic directions become more aligned. Intriguingly, we can boost alignment beyond a nonlinear autoencoder by simply using a linear autoencoder with a higher degree k. Moreover, we find that encoding the GloVe dataset with a smaller degree linear autoencoder also boosts accuracy on the analogy task.

Word Similarities
In the previous subsections, we showed that autoencoding word embeddings can align meaningful linguistic directions. Word locality is another important property of word embedding spaces: similar words should have similar representations. In this section, we show that aligning the space through autoencoding does not significantly degrade the locality of words in the latent space.
Word similarity datasets are comprised of word pairs along with human annotated ratings of the words' relatedness. We compute the similarity of two words as the cosine similarity between the words' embeddings. Following [7], we evaluate the metric wsim as the Spearman's rank correlation between the rankings computed  Figure 4: Alignment of drug signatures across 2 cell types (MCF7 and A549) using (a) LeakyReLU autoencoders from [5] and (b,c) linear autoencoders with asymmetric spectral initialization. Using linear autoencoders can lead to greater alignment than using a LeakyReLU autoencoder.
via cosine similarity and the rankings from the human annotation. We use two word-similarity datasets: Mturk-771 [12], which rates 771 word pairs on a scale of 0 to 5, and MEN [6], which rates 3000 word pairs on a scale of 0 to 50.
Reference Point Correction Word similarity metrics typically compute the cosine similarity of individual embeddings with respect to the origin o = 0. However, since autoencoding stretches along the top singular vector, this origin is a poor reference point since it does not stretch along with the rest of the dataset. Instead, we choose a reference point that will also stretch alongside the top principal component: for the original GloVe dataset X, we set the reference point to be o = λ 1 u 1 , where λ 1 , u 1 are the top left singular value and vector of X. Further considerations on this correction can be found in Appendix I. 4.
In Table 1, we find that the word similarity metrics remain largely stable after autoencoding, indicating that alignment does not significantly warp the latent space. Thus, autoencoding can align GloVe data without losing important properties such as word locality.

Drug Signature Alignment in CMap
Finally, we apply our theoretical results to gene expression data from CMap [29] to boost the alignment of drug signatures, i.e the difference in gene expression for a cell type at control and after applying the drug. Prior work [5] demonstrated that 1-hidden layer over-parameterized autoencoders with Leaky ReLU [31] activation produced pre-activation latent spaces which aligned drug signatures across cell types. The boost in alignment was evaluated by plotting the cosine similarity of drug signatures across cell types in the original and the autoencoded latent space.
Following the setting of [5], we measure the alignment of drug signatures between MCF7 and A549 cell types using linear and nonlinear autoencoders (Figure 4). We find that linear autoencoders with asymmetric spectral initialization can boost alignment even beyond the alignment given by the Leaky ReLU autoencoder in [5]. Moreover, using a high degree polynomial for asymmetric initialization (i.e., degree 30 in Figure 4c) leads to all drug signatures being nearly perfectly aligned.

Conclusion
In this work, we analyzed a mechanism through which autoencoders with asymmetric spectral initialization produce aligned latent spaces by stretching along the top singular vectors of the dataset. After fully characterizing the degree of stretching in linear autoencoders, we showed that we can arbitrarily stretch along the top directions by adjusting the initialization. We further replicated this behavior for nonlinear autoencoders in a simplified setting. We lastly applied our theoretical results to develop autoencoders that align linguistic shifts in word embedding spaces and drug signatures in gene expression data.
While we analyzed autoencoders initialized using asymmetric spectral initialization, [5] observed that even the default initialization scheme from PyTorch [20] can empirically lead to alignment. An interesting direction of future work would be to provide theoretical guarantees around what initialization regimes in linear and non-linear settings guarantee alignment of semantically meaningful directions after gradient descent. Another interesting avenue would be to characterize alignment of autoencoders in infinite width settings to facilitate analysis of more complex autoencoders. While neural tangent kernel (NTK) analyses are not readily applicable for understanding latent space analysis, other parameterizations that admit feature learning (such as the µP parameterization [32]) may be more helpful for understanding alignment.

Acknowledgements
This work was partially supported by NSF (DMS-1651995), ONR (N00014-17-1-2147 and N00014-18-1-2765), the MIT-IBM Watson AI Lab, and a Simons Investigator Award to C. Uhler. The Titan Xp used for this research was donated by the NVIDIA Corporation.

A Proof of Lemma 1: Alignment of linearly embedded datasets
Proof. We note that: We thus have:

B Proof of Lemma 2: Stretching Lemmas
Proof. Let x (j) i be the the ith example from X 1 and x (2) i be the ith example from X 2 . Then we note that Then by the strong law of large numbers, as n tends to infinity, we have that: We can then plug in the relevant values for Σ i :

C Proof of Corollary 1: Alignment of 2D Gaussians
Proof. For our particular setting, from Lemma 2, we know that: Simplifying our expression for B gives us the following form: We want to see how the angle between the principal directions of the original (centered) datasets changes after embedding. We note that the top unnormalized eigenvectors of the two centered datasets are: The cosine distance between these two directions is: The cosine similarity of the expected embeddings of these two directions is (after a fair amount of arithmetic): Moreover note that since γ > 0, we have that cos(Bq Σ1 , Bq Σ2 ) > cos(q Σ1 , q Σ2 ).  . Then B = E[ 1 n XX T ] k has the exact same eigenvectors, but has eigenvalues λ 1 = (c + 7) k and λ 2 = (c + 3) k . From Corollary 1, we know that the cosine distance of q Σ1 and q Σ2 after embedding withB will be for α = (c + 7) k , β = (c + 3) k . From Lemma 2, we know that 1 n XX T a.s − − → E[ 1 n XX T ]. Then, by the continuous mapping theorem, we have that B = ( 1 n XX T ) k a.s − − → (E[ 1 n XX T ]) k . Therefore B converges a.s toB. Since computing the cosine similarity between Bq Σ1 and Bq Σ2 given B is a continuous mapping, we have that Moreover, as k increases, γ also increases, which means cos(Bq Σ1 , Bq Σ2 ) will go up and the two directions will become more aligned. We empirically verify this claim with several values of c, and find that our theoretical result holds up to the third decimal place.
An empirical verifiction of Corollary 2 is presented in Figure 5. For generating these plots, we chose c = −.9 and k = 35 3 . 3 We used a scale factor of 1 2.25 to control the norm of the latent dimensions for visualization.

E Proof of Theorem 1: Gradient Flow Analysis for Linear Autoencoders
Proof. Gradient descent using learning rate γ proceeds as follows: We first show by induction that under the given initialization, A (t) , B (t) are always polynomials in XX T , and hence, only the the singular values of A (t) , B (t) need be considered in the update. The base case holds by definition, and so we assume that the same is true at time t. If B are diagonal matrices that are polynomials in Sigma X . Then at time t + 1: is a polynomial in Σ X . Hence, A (t+1) is a polynomial in XX T . We can proceed analogously for B (t+1) and induction is complete. The result above implies that we need only consider gradient descent updates on Σ B , which proceed as follows: By re-arranging terms and taking the limit as γ → 0 (i.e. gradient flow), we get: where Σ A , Σ B are functions of time t (i.e. Σ A = Σ A (t)).
By the balancedness equation (i.e. multiplying dΣ A by Σ A and multiplying dΣ B by Σ B ), the above yields: Now by integration coordinate-wise in time, we get: ( Lastly, we use the condition that at the end of training, our neural network must interpolate the data. where I r is a diagonal matrix of r 1's (with r denoting the rank of X). Now for each diagonal entry of Σ A , Σ B (denoted Σ Ai,i , Σ B i,i ), Equations (2) and (3) yield a system of 2 equations with 2 unknowns, which can be solved exactly. For ease of notation let Σ Ai,i = a i , Σ B i,i = b i , Σ xi,i = x i . Summarizing below for a single diagonal entry, we have: Let c 2 = b 2 0 − a 2 0 . Reducing the system of equations above, we have: We take the positive square root since this yields the solution closest to the initialization.

F Proof of Corollary 4: Alignment of 2D Gaussians after autoencoding
Proof. Let λ i be the ith eigenvalue of XX T . By Theorem 1, if we initialize A (0) = 0 and B (0) = ( 1 n XX T ) = U X Σ B U T X , then after gradient descent we have: For our particular setting, from Corollary D, we know that B (0) converges as n tends towards infinity: Moreover, we note that the mapping from B (0) → B is continuous since it is a composition of applying the matrix square root and matrix polynomials, which are both continuous. Then using the continuous mapping theorem, we have that B converges almost surely toB, whereB can be expressed as We thus have that that: for the above α and β. Moreover, as k increases, the γ goes up, and Bq Σ1 and Bq Σ2 become more aligned.

G Proof of Theorem 2: Stretching of ReLU Autoencoders
We begin by first proving the following lemma about properties of gradient descent in the nonlinear setting.
Lemma 3. Let f (x) = Aφ(Bx) be a one hidden layer ReLU neural network, with A ∈ R d×h , B ∈ R h×d , and φ as a ReLU activation. Let X = {x, −x} with x ∈ R d denote our training set, and let η = ||x|| 2 2 . Then, the following hold: 2. Gradients: If A = xa T and B = bx T for a, b ∈ R h , then the gradients of the autoencoder loss, L in (1) with respect to A and B are: Proof. We first note that: Moreover, if we can express A = xa T and B = bx T , our gradients simplify to: using the fact that φ(z) − φ(−z) = z. From these expressions, we find that we can express ∇ A L = xa T and ∇ B L = b x T , proving our invariance property. Now suppose that A = xa T and B = bx T . Then we have: which proves our second claim. We empirically verify this result up to the third decimal place by training a network with SGD with learning rate 0.1 with x chosen as a uniformly random vector with dimension 1001 and norm 2.
The form of the gradients in Lemma 3 allows us apply a similar gradient flow analysis as in Theorem 1 to get a closed form for A (∞) , B (∞) . In the following lemma, we characterize the trajectories of a and b for A = xa T and B = bx T and establish equilibrium conditions on a and b.
Lemma 4. In the setting of Lemma 3, let A (0) = x(a (0) ) T and B (t) = b (0) x T . After t steps of gradient descent on A and B, A (t) = x(a (t) ) T and B (t) = b (t) x T such that: Moreover, at equilibrium time step t , the following hold: Proof. Using our closed form expressions for the gradient in Theorem 3, we notice that: We can integrate of both sides of the equation to get the values of A and B at time t: By our invariance property, we know that A (t) can be expressed as x(a (t) ) T and B (t) can be expressed as b (t) x T . We can thus rewrite this form as: Moreover, we note that at equilibrium time step t our gradients must be 0, which gives us Lemma 4 allows us to provide closed forms for A (∞) , B (∞) . We are now ready to prove the overarching Theorem 2.
Proof. Define w T = x T −x T . Suppose that we initialize B = bx T and A = xa T such that b = βw and a = αw. Then we note that a T φ(b) = −a T φ(−b) = αβη. We can easily see from Lemma 3 that such a parameterization is invariant across gradient steps, since the gradients take the form: for C α,β = 2η(η 2 αβ − 1). Then from Lemma 3, we know that for equilibrium time step t , a (t ) can be expressed as α (t ) w and b (t ) can be expressed as β (t ) w. Then we can use our findings from point 3 of the above Lemma: Solving this system of equations, for α (0) = 0 and β (0) = 1 gives us Then we have that

I.1 Interpretation of the Top Frequency Directions of GloVe
Let X ∈ R 300×500,000 be the GloVe embeddings of the 500,000 most frequent words. We compute the top singular vector u 1 ∈ R 300 of X. In Figure 6, for each word with embedding w, we plot cos(w, u 1 ) against the frequency ranking for that word. We find that frequent words are more aligned with the top singular vector.

I.2 Alignment of meaningful directions before and after autoencoding
Let (a 1 , a 2 ) and (b 1 , b 2 ) be two pairs of word embeddings such that the corresponding words have the same relationship. An example might be the embeddings of (man, woman) and (king, queen). Then for encoding matrix B, we plot the cosine similarity of the translation vector before encoding (i.e cos(a 2 − a 1 , b 2 − b 1 )) and after encoding (i.e cos(Ba 2 − Ba 1 , Bb 2 − Bb 1 )).
We take every combination of pairs with the same relationship from the Google Analogy test set (this results in 19430 combinations), and plot using the linear autoencoder with asymmetric spectral initialization (k = 2), as well as a LeakyReLU autoencoder (Figure 7). We find that both autoencoders increase the alignment of the word pairs, but using the linear autoencoder results in more alignment. For a linear autoencoder with k = 2, 76% of the autoencoded pairs have cosine similarity above 0.5.

I.3 Visualizing alignment before and after autoencoding using MDS
We visualize the latent space before and after autoencoding using multidimensional scaling (MDS). For a given relationship from the Google Analogy test set, we take the set of all words that appear in a pair for that relationship. We perform MDS with two components on the word embeddings of that set before and after autoencoding. Using the coordinates derived from MDS, we plot the pairs to visualize alignment. In Figures 8 and 9, we visualize the alignment of two relationships: negation (i.e aware → unaware) and superlative (i.e cold → coldest). We find that embeddings derived from a linear autoencoder with asymmetric spectral initialization align the translation vectors for word pairs with the same relationship.

I.4 Reference Point Correction
Word similarity metrics capture the locality of a latent space by computing whether similar words have similar embeddings [7]. To capture the similarity of two embeddings a and b, these metrics typically compute their cosine similarity against a fixed reference point at the origin o = 0. Specifically, the similarity of the embeddings in the original space would be cos(a − o, b − o) = cos(a, b) and the similarity after encoding with a matrix B would be cos (B(a − o), B(b − o)) = cos(Ba, Bb). However, from our theory, we know that autoencoding will stretch along the top singular vectors. The point o = 0 is thus a poor reference point since it will not stretch along the rest of the dataset. We instead pick an reference point o that will stretch alongside the top principal component: for the original GloVe dataset X ∈ R d×N with top left singular value and vector (λ 1 , u 1 ), we set our reference point as o = λ 1 u 1 . This is equivalent to simply translating our latent space before computing word similarity to a new origin point (note that translation will not change alignment).
In Figure 10 we show the effect of setting the reference point to λ 1 u 1 . When choosing the default origin at 0, the angle between embeddings w 1 and w 2 becomes much larger as we stretch in the red direction. However, by choosing the corrected o , the reference point stretches alongside the rest of the dataset, so the angle between w 1 − o and w 2 − o does not significantly change after autoencoding. Before Autoencoding After Autoencoding w 2 Figure 10: A schematic depicting reference point correction. When using a default reference point of o = 0 (the grey dot) the cosine similarity between embeddings w 1 and w 2 become distorted after stretching. However, if we choose a reference point along the stretching direction (the black dot), then the cosine similarity remains stable even after autoencoding.
In Table 2, we display the word similarity metric wsim with and without reference point correction. As expected, we find that stretching using autoencoders degrades the word similarity metric, if we pick the default reference point o = 0. However, if we choose the corrected reference point o = λ 1 u 1 , the word similarity metrics remain largely stable, especially for the linear autoencoders.