{'fraction_non_alphanumeric': 0.03868244412151408, 'fraction_numerical': 0.010552902913293197, 'mean_word_length': 4.63411306042885, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Manifold Computing and Visualization

The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear dimensionality reduction (NLDR) and visualization. This paper proposes a novel method, called elastic locally isometric smoothness (ELIS), to empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions. Extensive experiments, comparisons, and ablation study demonstrate that ELIS can deliver results not only superior to UMAP and t-SNE for and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation.


Introduction
Manifold learningaims to find from a set of higher dimensional data its embedding or representation in a low dimensional latent space. Nonlinear dimensionality reduction (NLDR) aims to construct a transformation that is generalizable to unseen data. It is hoped that the lower dimensional representation can be used in conjunction with a simple metric such as the Euclidean distance for downstream tasks such as classification and visualization. Manifold data generation performs the inverse transformation to generate data from samples in the latent space. We call this collection of manifold related problems manifold computing. The basis for manifold computing is the manifold assumption [16,8].
Great advances have been made in the past two decades in manifold computing and visualization. ISOMAP [22] and LLE (locally linear embedding) [19] are classic methods for manifold learning. More recent developments include local geometry-based method [9,25,5,15], graph spectral analysis [6] and latent variable models [20]. The most popular high dimensional data visualization methods to date are t-SNE [13] and UMAP [14], with wide applications such as bio-science and technology [1,7]. While the aforementioned are traditional machine learning, deep learning-based methods include autoencoders [10,17]. The problem can be considered from the viewpoints of geometry deep learning [4]) and topology data analysis [23,17].
The ability to preserve geometric structure of nonlinear manifolds and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold-based computing and visualization. Recently, Markov-Lipschitz deep learning (MLDL) [12] is proposed as a general framework for manifold learning, NLDR, visualization and manifold data generation. The idea is to impose the constraint of geometric isometry across neural network layers to preserve the local geometric structure of manifold data. This effectively transforms a vector-based transformation of conventional neural networks into a local distance-preserving homeomorphism. Such local homeomorphisms avoid the arXiv:2010.14831v1 [cs.LG] 28 Oct 2020 transformation from collapse, twisting, or crossing, so as to improve generalization, stability, and robustness. Locally isometric smoothness (LIS) [12], which imposes straight distance-preserving, is proposed as a method in the MLDL framework. LIS has demonstrated significant advantages in manifold learning and NLDR.
This paper proposes a more advanced method in the MLDL framework, called elastic locally isometric smoothness (ELIS), aimed to empower deep neural networks with ability to tackle the high nonlinearity and non-Euclideanity challenges arising from complicated manifolds in high dimension spaces that LIS is unable to cope with. Whereas LIS preserves the straight distances between neighboring points, ELIS is based on a similarity metric that is nonlinear in distance and a two-way divergence loss (of nearby neighbors and far-away pairs, respectively); this renders more flexibility and capacity in tackling the challenges yet under the control of the ELIS regularization. As the result, ELIS bridges gaps between non-Euclidean manifolds in the input space and resulting Euclidean hyperplanes in the learned lower dimensional latent space, with geometric structure of the manifolds preserved. Both ELIS and LIS can be considered as a form of graph neural networks (GNN) [21] but without the aggregation generally present in GNNs. They are more like what is called "manifold learning 2.0" [3]. The distinctive features of ELIS (and LIS) in comparison with related methods are summarized in Table 1. ELIS-based neural networks can accomplish all the functionalities in the general MLDL framework, for which none of the methods can achieve. Extensive experiments, comparisons, and ablation study demonstrate that ELIS-based neural networks produce results not only superior to the SOTA t-SNE and UMAP for NLDR and visualization but also better than other algorithms of manifold and autoencoder learning, including LIS, for NLDR and manifold data generation. The main contributions of this paper are summarized below: (1) Proposing the ELIS constraint in the MLDL framework, based on a similarity metric which is nonlinear in distance. It inherits the metric-preserving property of LIS so that the resulting layer-wise transformation is geometrically smooth, hence topologically homeomorphic, yet possesses more flexibility than LIS in handling highly nonlinear manifolds in high dimensional spaces.
(2) Proposing conditions for a class of nonlinear similarity functions for converting from distance to similarity, in conjunction with a two-way divergence loss. This ensures the metric-preserving and neighbor-confining properties.
(3) Proposing two instances of ELIS-based neural networks: an ELIS encoder for manifold learning and visualization and an ELIS autoencoder for manifold reconstruction and data generation.
(4) Providing several SOTA results that surpass UMAP and other leading algorithms.
In the following, Section 2 introduces LIS and presents ELIS formulations, and the Section 3 presents extensive experiments. The code is provided in the Supplementary Material.

Elastic Locally Isometric Smoothness
Both ELIS and LIS are formulated in the MLDL framework (illustrated in Fig.A1 in Appendix) which is aimed to regularize neural transformations through imposing the ELIS constraint between layers to achieve certain well-behaving properties. However, the ELIS formulation tackles challenges of highly nonlinear manifold data in high dimensional spaces using a more flexible and effective way, much inspired by t-SNE [13] and UMAP [14]. Let X = {x 1 , . . . , x M } be a set of M samples in the input space R N with the index set S = {1, . . . , M }. These samples may come from one or several lower dimensional manifolds M X ⊂ R N . When M X is Riemannian, its tangent subspace T x (M X ) at any x ∈ M X is locally isomorphic to an Euclidean space of dimensionality dim(M X ) < N . Therefore, we can use a cascade of nonlinear neural transformations to "unfold" nonlinear manifolds in a high dimensional input space into hyper-planar regions in a lower dimensional latent space.
Both ELIS and LIS aim to accomplish the following 4 tasks, of which few neural networks can do all: (1) Manifold learning: to learn an embedding in a latent space M Z ⊂ R n , where n < N , based on the local structure of X. (2) Representation Learning: to learn the underlying mapping Φ : M X =⇒ M Z for the embedding that is generalizable to unseen data x ∈ X, x ∈ M X . (3) Visualization: to visualize the embedding in 2D or 3D space. (4) Manifold generation: to find the inverse mapping Φ −1 : M Z =⇒ M X and generate new data on M X from samples in M Z . ELIS is aimed to surpass LIS.

The LIS constraint and neural networks
The LIS constraint is aimed to best preserve the local distances of the data between two metric spaces, encouraging a vector-based neural transformation Φ(X | W ), where W is the transformation matrix of the neural network, to become a well-behaved local distance-preserving homeomorphism. This can be achieved by adding the following LIS loss [12], imposed between two layers (metric spaces) l and l L (l,l ) where d : is the result of the effective transformation Φ from layer l to l , and N i is the set of neighbors of i. Without prior knowledge, d ij is usually computed as the Euclidean distance, albeit it may not well reflect the reality. It is hoped that after a series of proper nonlinear transformations, the input data is transformed into an embedding in the latent space such that the Euclidean distance make more sense in describing mutual relationships between points. In this work, we aim to find such transformations.
The LIS loss effectively minimizes the bi-Lipschitz constant of Φ. It is through the neighborhood system, N = {N i | i ∈ S}, that the influence of a point on the others is propagated to afar. For this reason, the collection of random variable x (l) constitutes a Markov random field. Equ. (1) is defined w.r.t. N i (Markovianity) and aimed to minimizing the bi-Lipschitz constant, hence the name Markov-Lipschitz [12].
The basic LIS loss is augmented by an auxiliary "push-way" term [12] L (l,l ) in which π[·] ∈ {0, 1} is the indicator function and B is a bound. This term is aimed to help "unfold" nonlinear manifolds, by exerting a spring force to push away from each other those pairs (i, j) which are non-neighbors at layer l but nearby (distance smaller than B) at layer l .
These two losses are combined to form a LIS-based encoder loss for manifold learning and dimension reduction where µ is a weight and (l, l ) is summed over a set of designated layer pairs (currently designed manually). A LIS-based autoencoder can be formulated by applying the LIS constraint between layers within the decoder and between the encoder and decoder layers. LIS-based neural networks have significant advantages [12].

The ELIS constraint
The proposed ELIS constraint is aimed to tackle difficulties in "flattening" highly nonlinear manifolds in a high dimensional space into hyperplanes in a lower dimensional space. It imposes a more flexible nonlinear similaritypreserving constraint as opposed to the distance-preserving (isometry) constraint of Vanila LIS. More specifically, ELIS transforms a distance into a similarity metric using a nonlinear function and defines a KL loss based on similarities between nearby pairs and far-away pairs. This makes the metric-preserving constraint of ELIS more flexible than the straight distance-preserving of LIS to accomplish the challenging task.
Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions.
Converting distance to similarity. Following UMAP, we assume that X (l) is fixed (e.g., the input layer) and X (l ) at subsequent layers l are computed as a result of manifold learning. The nonlinear similarities between x i and x j at each layer is computed as follows. First, define an nearest neighbor (NN)-normalized distance We require that g(η) satisfy the following necessary conditions ∀η = d i|j ≥ 0: is monotonically decreasing, that is, g (η) < 0 for η > 0; Condition (2) -its first derivative diminishes in the limit, lim η→∞ |g (η)| = 0.
The first condition above ensures that the similarity is monotonically decreasing as the distance increases. The second condition restricts the influence from "far-away" points x j on x i , between which the distance η ij = x i −x j is relatively large, to diminish; this effectively leads to a soft neighborhood system, as opposed to the "hard" bounded neighborhoods in the LIS, to provides proper adaptive control on the contributions of neighboring points to back-propagation.
We further require that the g(η) to be a function of η 2 -for convenience not a necessity, such that its first derivative takes the form g (η) = 2ηh(η) where h(η) is also a function of η 2 . h(η) can be called an adaptive influence function because it adaptively controls to what extent x j 's influence x i .
Specifically for ELIS, we define the following σ i -data-adaptive, ν-parameterized nonlinear similarity where ν ∈ R + is similar to the degree of freedom (DoF) parameter in the t-distribution, is a function of ν which sets the limit lim ν→+∞ g(0 | σ i , ν) = 1 (∀σ i > 0)), and the data-adaptive parameter σ i > 0, playing a calibration role, is estimated from the data by best fitting the equation for the perplexity-like hyperparameter Q given. The adaptive influence function corresponding to Equ. (5) is While other choices satisfying the aforementioned necessary conditions, including the normalized Gaussian and Cauchy functions used in t-SNE [13] and the fitted polynomial function used in UMAP [14], can also work for ELIS, we find Equ. (5) a better choice not only because it produces better results but also because we can use the ν parameter as a continuation tool for preventing the training from converging to bad local minima and for controlling separation margin between different manifolds, as will be shown in the ablation study.
Computing similarities u ij and u ij . Because the symmetry u i|j = u j|i does not hold due to differences in σ i for the input layer (l), the following symmetrization is performed On the other hand, for the subsequent latent layers, the computation of σ i and ρ i for each i would bring about huge computational costs. To overcome this problem, we directly set σ i = 1 and ρ i = 0 (this also ensures the symmetry u i|j = u j|i ). While σ i and ρ i are needed to deal with unevenness and outliers of the data for the input layer, the necessity becomes not so demanding as the layer goes deeper after layers of nonlinear manifold unfolding. From u (l) ij of layer l can be constructed a weighted graph G(S, X (l) , U (l) ) consisting of a set S of nodes with node attributes X (l) and edge attributes (weights) The global structure of a manifold is discovered from local geometry of data through the graph G.
Formulating the ELIS losse. ELIS transforms the distance metric d ij into a similarity metric using a nonlinear function u ij = g(d ij ) and defines the ELIS loss between layers l and l , in terms of similarities u (l) ij | i, j ∈ S, i = j} at layers l and its counterpart U (l ) = {u (l ) ij | i, j ∈ S, i = j} at layers l . The ELIS loss is defined by what we call the two-way divergence (a.k.a. the fuzzy information for discrimination [2] and the fuzzy set cross entropy in UMAP [14]) The first term is the directed divergence of the two fuzzy sets of similarities, in lieu of the LIS' distance-preserving term of Equ. (1); the second term can be considered as the directed divergence of the two corresponding complement fuzzy sets, replacing the push-way term of Equ. (2).
Equ. (10) is called the "two-way divergence" because the first term on the right side of the equation imposes similaritybased attraction forces between nearby (intra-manifold) pairs whereas the second term exerts dissimilarity-based repulsion forces between far-away (inter-manifold) pairs. In other words, intra-manifold points are transformed to a cluster in the latent space, mainly as the result of the first term whereas inter-manifold point pairs push away from each other to different clusters, mainly due to the second term.
Note also that ELIS applies the two terms in a soft, adaptive way via its weighted neighborhood graph G where the edges are effectively restricted by pairs of corresponding nodes (data points) between which the absolute gradients ∇ W L (l,l ) ELIS (W | X (l) , X (l ) ) ≥ > 0 are nonzero, in contrast to the "hard" neighborhood system in LIS.
The ELIS loss can be rearranged as follows When ij ) fixed, the optimization only needs to minimize second part involving u (l ) ij .

ELIS encoder and autoencoder
The ELIS encoder consists of a cascade of nonlinear forward neural transformations constrained by the ELIS loss, aimed for manifold learning and NLDR. An ELIS (and LIS) encoder can learn an NLDR transformation without the need for a decoder (as required by autoencoders), and this encoder can generalize to unseen data (that ISOMAP, LLE, t-SNE and UMAP cannot). The total loss for the ELIS encoder is the sum of all the ELIS losses over a prescribed set of layer pairs (l, l ) where α (l,l ) weight the relative importance of L (l,l ) ELIS . The ELIS autoencoder has two purposes: (1) to further regularize or optimize the ELIS encoder-based manifold learning by using an ELIS decoder, and (2) to enable generation of new data of the learned manifolds by the trained ELIS decoder. The ELIS autoencoder structure consists of the ELIS encoder and decoder in cascade. The ELIS decoder is aimed to approximate the inverse transformations of the ELIS encoder and is made to be entirely symmetric to the ELIS encoder in its network structure. The overall weight matrices becomes W = [W Enc , W Dec ]. The loss function is composed of three terms: where L Enc (W Enc ) is the same as Equ. (12), L Dec (W Dec ) is defined in the same way following the symmetry, and the reconstruction loss L Rec (W ) is the summed over all the corresponding layers wherex (l) i are the data points at the corresponding layer of the decoder and γ l are the weights. The constraints due to L Rec (W ) and L T ie (W ) are illustrated by the dashed lines in Fig. A1 in Appendix.

Experiments
The following experiments are aimed to evaluate ELIS in comparison with other five algorithms: UMAP [14], t-SNE [13] (for visualization), MLLE [25], TopoAE [17] and LIS [12] in terms of visual inspection and numerical metrics for manifold computing, visualization and data generation. Nine datasets are used, including five toy datasets:  [17] for the description) and (5) SpheresB (101-D, a modified composition from SpheresA); and four real-world datasets: (6) Coil20 (16384-D) and (7) Coil100 (49152-D) [18], (8) MNIST (784-D) [11], and (9) Fashion-MNIST (784-D) [24]. The toy datasets are used because their geometric and topological structures are clear for the evaluation. The SpheresA dataset [17]is composed of 1 large sphere enclosing 10 small ones in 101-D space. SpheresB differs from SpheresA in that its large sphere consists of only 500 samples (whereas that in SpheresA has 5000) -the data is so sparse that the smallest within-sphere distance on the larger sphere can be greater than that between the larger sphere and some small ones. 5 performance metrics are used for the evaluation, whose exact definitions are given in Appendix A.2.
The pseudo-codes of the ELIS encoder and autoencoder and hyperparameter settings are described in Appendix A.1. The implementation uses the PyTorch 1.6.1 library running on Ubuntu 18.04 on NVIDIA v100 GPU. The time is spent mainly in the computation of neighbors. At present, the ELIS algorithms computes the neighborhood for every point pair, hence have the complexity of O(M 2 ) for each cross-layer pair (l, l ). The complexity can be reduced to O(M 1.14 ) if using the nearest-neighbor-descent algorithm of UMAP [14].

Manifold Learning and Generation
Manifold Learning. Table 2 compares performances of the five NLDR methods where bold numbers are the best results, and underline the second best. The ELIS encoder has overall the best performance. Fig. 1 visualizes some representative results, and more results are given in Table A2, Fig. A2 -Fig. A4 in Appendix A.3.

Ablation Study
Cross-layer ELIS constraint. The cross-layer ELIS constraint is weighted by α(l, l ) as in Equ. (12). Four weight schemes (1) Head-Tail, (2) Head-Mids, (3) Mids-Tail and (4) Head-Mids + Mids-Tail are designed for an L-layer encoder, as described in details in Appendix A.5. The results are compared in Table A3 and Fig. A7. Overall, the "Head-Mids + Mids-Tail" scheme, which imposes the most extensive cross-layer ELIS constraints and also needs more computation, achieves the best results. This justifies the use of the proposed ELIS method for performance improvements.
Effect of final ν value. The final ν value has significant influence on the within-manifold and between-manifold scatters. Fig. A8 and Fig. A9 in Appendix A.5 demonstrate the effect of varying ν in the input space and varying ν in the latent space on the results, respectively.
Continuation in ν. Continuation in hyperparameter ν in latent space is used during the ELIS encoder learning process. The algorithm starts with a small ν in latent space to include more global information. Then it gradually increases the value to focus more locally. The continuation strategy results in significant better solutions as shown in Fig. A10 in Appendix A.5.
The major merits of ELIS. Finally, we summarize the comparative results of the experiments in the following

Conclusion
The proposed ELIS method preserves the nonlinear similarity metric locally across layers of deep neural networks by optimizing two-way divergence loss. It effectively tackles difficulties in deep manifold computing and visualization with the local geometry-preserving property. Empirical results, comparisons, and ablation study demonstrate that ELIS is not only superior to UMAP and t-SNE for NLDR and visualization but also better than other leading manifold and autoencoder learning algorithms for NLDR and manifold data reconstruction and generation. Future work includes the following: (1) extending the unsupervised version of MLDL to self-supervise, semi-supervised and supervised tasks; (2) further formulating MLDL so that cross-layer link hyperparameters α become part of learnable hyperparameters.   (5) Calculate l layer's decoder embedding X (l) ←− Φ  A.2 Definitions of performance metrics 1. Cont (Continuity) is asymmetric to Trust (from space X (l ) to space X (l) ): 2. Trust (Trustworthiness) measures how well the k nearest neighbors of a point are preserved when going from space X (l) to space X (l ) :

ACC (svm)
The ACC (svm) is calculated as follows.
(3) For each fold, train the linear kernel SVM classifier using the training set and test it in the test set. (4) Calculate the mean value of the classification accuracy.

ACC (NN) is defined as follows:
i,1 is the nearest neighbor point of node i in layer L.

AUC is defined as follows:
AU ij is the distance in layer L. P 0 is the set of positive sample pair, P 1 is the set of negative sample pair.

A.3 Manifold Learning and NLDR
Manifold Learning Results. This subsection shows more results of manifold learning and NLDR obtained by using the ELIS-Enc and the ELIS-AE, in comparison with the other methods, on training and testing datasets. Some typical embedding results of manifold learning using the three autoencoder methods are visualized in Fig. A2, where t-SNE and UMAP are not included because these non-transformational methods are unable to generalize to test datasets. LIS-Enc and TopoAE learned poor results on the training set, so it did not work well on the test set either. ELIS-AE, as Figure A2: Comparison of visualization results of autoencoders on training and testing sets a autoencoder-based mehtod, have a huge advantage in terms of generalization performance, because it can handle the test data. And it is very easy to apply to specific tasks such as classification, regression and clustering.   Symmetry of the objects and ELIS-Enc's embedding in Coil20. For Coil20, information about the symmetry of the objects in the picture can be obtained by analyzing the embedding generated by ELIS-Enc. Details of the ELIS-Enc embedding of the Coil20 are shown in Fig. A5.
We divided the Coil20's manifolds into four patterns based on the symmetry of the objects in the image and the shape of the manifold.
(1) Objects that are single plane mirror symmetric have elongated ellipse embedding shapes; For objects with single plane mirror symmetry, an angle can be found from which an image taken by rotating to the left is approximately equal to an image taken by rotating to the right. The corresponding two-dimensional manifolds are therefore elongated ellipse (The endpoints of the two long axes of the ellipse correspond to the two images obtained by taking pictures along the plane of symmetry.). (2) Objects that are rotational symmetric have round embedding shapes; For rotational symmetric objects, the resulting pictures are always very similar no matter what angle they are taken from, so that the resulting two-dimensional manifold is squeezed inward into a circle.  The learned decoders of ELIS, LIS or TopoAE (all with the same network structure) are used to generate images of unseen views by interpolating between two embedding points and then reconstructing by the decoders. More specifically, it is done as follows: (1) Train an autoencoder and use the encoder to produce the embedding in 2D latent space.
(2) Uniform linear interpolation. 41 consecutive points (The case of Fig. A6) are sampled along the embedding of a manifold; 10 pairs of neighboring points between the first and last of the 41 points are obtained equi-index apart; their mid-points are computed. (3) Data generation. An image reconstructed from each of the 10 mid-points using an decoder. Fig. A6 demonstrate that ELIS-AE produces higher quality results than LIS-AE and TopoAE in terms of boundary clarity and image sharpness.
In this ablation study, a 10-layer neural network is used and the width of the network is determined depending on the dataset. The evaluation metrics for four different cross-layer schemes are presented in Table A3. The results of different cross-layer schemes are shown in Fig. A7. Figure A7: Comparison in visualisation of four different cross-layer schemes.
The visualization results and metrics show that the cross-layer scheme (4) has better results in a 10-layer network. The network is very difficult to train if the ELIS loss acts only on the first and last layers (cross-layer scheme (1) ). The network will be easier to train if ELIS losses acts from the first layer and all intermediate layers (cross-layer scheme (2)). The ELIS losses in the middle and last layers (cross-layer scheme (3)) does not improve the performance of the embedding if used alone. However, if used in conjunction with the cross-layer scheme (4), it will improve the metric of the resulting latent space.  Effect of ν value. Fig. A8 and Fig. A9 show the effect of ν value for the input layer and ν ν for the latent layer on resulting embeddings. Figure A8: Embedding results with varying ν in input space.
ν in input space controls the range of sensations in data space. if input space's ν is small, the derivative of the probabilistic mapping function of the input layer will be small. The probability will be insensitive to distance in data space. In other words, ELIS-Enc will degradation into LIS-Enc. If more attention is paid to the global information of the input data, raise ν. If more attention is paid to local information about the input data, lower input space's ν. By default, ELIS-Enc does not change this hyperparameter, but defaults to input space's ν = 100.
ν in latent space controls degree of display in latent space (show detailed information or show global information). if latent space's ν is small, ELIS will tend to display global information. If latent space's ν is large, ELIS-Enc will tend to display the local information. Fig. A9 shows, from left to right, a process that goes from showing global information to showing excessive local information. Continuation strategy. Continuation in hyperparameter latent space's ν is used during the ELIS encoder learning process. The algorithm starts with a small latent space's ν value to include more global information. Then it gradually increases the value to focus more locally. The necessity of parameter continuation is shown in Fig. A10. Figure A10: Ablation study of with and without parameter continuation in latent space ν. (The upper row shows results obtained via parameter continuation ν = 0.001 → ν = 100 in latent space, the lower row shows results with a fixed ν = 100) Experiments prove that the effect of parameter continuation (ν = 0.001 → ν = 100 in latent space) is very obvious.
There is a big improvement in Swiss Roll, Coil20 and MNIST.