{'fraction_non_alphanumeric': 0.03157710075434185, 'fraction_numerical': 0.014735980352026197, 'mean_word_length': 4.599869024230517, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning a Reduced Basis of Dynamical Systems using an Autoencoder

Machine learning models have emerged as powerful tools in physics and engineering. Although flexible, a fundamental challenge remains on how to connect new machine learning models with known physics. In this work, we present an autoencoder with latent space penalization, which discovers finite dimensional manifolds underlying the partial differential equations of physics. We test this method on the Kuramoto-Sivashinsky (K-S), Korteweg-de Vries (KdV), and damped KdV equations. We show that the resulting optimal latent space of the K-S equation is consistent with the dimension of the inertial manifold. The results for the KdV equation imply that there is no reduced latent space, which is consistent with the truly infinite dimensional dynamics of the KdV equation. In the case of the damped KdV equation, we find that the number of active dimensions decreases with increasing damping coefficient. We then uncover a nonlinear basis representing the manifold of the latent space for the K-S equation.


I. INTRODUCTION
Evolution of physical and engineering systems is generally expressed as nonlinear partial differential equations (PDEs). These PDEs are able to capture a wide range of complex phenomena and are therefore indispensable for making predictions of scientific interest. However, most PDEs of practical interest are not analytically tractable. Highly efficient numerical methods can obtain solutions to these PDEs, but are severely limited by the strong multi-scale nature of the underlying dynamics and limitations of hardware resources. There is therefore considerable interest in the development of reduced models that capture only the most important dynamics of the physical phenomenon of interest [1][2][3][4]. In recent years, machine learning algorithms have been borrowed from the computer vision community and adapted for physical applications [5][6][7][8][9][10][11][12][13], offering an enticing approach for blending data with physical principles. In fact, a key goal in merging machine learning with physics problems is to embed known physical laws into machine learning algorithms [8]. The current work presents a general approach to finding a dynamically-relevant manifold of canonical PDEs using autoencoders. Other recent work has applied autoencoders to learning inertial manifolds of PDEs in a physically-meaningful manner [14][15][16][17][18]. In particular, [17] introduces the Hybrid Neural Network (HNN), in which an autoencoder is used to learn the difference between the data and a linear projection onto the principle component analysis (PCA) basis. The HNN embeds translation invariance and energy conservation and learns the dynamics on the inertial manifold of the Kuramoto-Sivashinsky equation. In the present work, the traditional mean-squared error loss function between the input and reconstruction of the autoencoder is augmented with the sparsity-promoting mean absolute error loss function, which is applied to the latent space. In this way, the latent space of the trained autoencoder only contains the minimal dimensions needed to reconstruct the solution. This approach is tested on two equations that are known to have an inertial manifold (the Kuramoto-Sivashinsky equation and the damped Korteweg-de Vries equation) and one equation whose dynamics are truly infinite dimensional (the undamped KdV equation). In the case of the Kuramoto-Sivashinsky equation, the trained autoencoder and latent space is used to find a non-linear reduced solution basis whose dimension is consistent with that of the inertial manifold.

II. GOVERNING EQUATIONS AND DATASETS
The Kuramoto-Sivashinsky (K-S) equation is, where u = u (x, t) is the solution field, x ∈ [0, L], and t ∈ R + . The final integration time is denoted by T . Equation (1) is subject to periodic boundary conditions u (0, t) = u (L, t) and initial condition u 0 = cos 2π L x 1 + sin 2π L x . The K-S equation has its roots in physics [19][20][21][22], and is a frequently-studied equation in mathematics [23][24][25][26][27][28][29][30]. Of particular relevance to the present work is the result that the dynamics of the K-S equation are confined to an inertial manifold [24]. That is, despite the K-S equation being a non-integrable equation whose solutions exhibit spatio-temporal chaos, the underlying dynamics are exponentially attracted to a finite-dimensional manifold. The dimension of the inertial manifold increases with the bifurcation parameter L.
The damped Korteweg-de Vries (KdV) equation, also considered in this work, is, where x ∈ [−π, π] and η ≥ 0 is a damping coefficient. The KdV equation is subject to periodic boundary conditions u (−π, t) = u (π, t) and initial condition u 0 = 3A 2 sech 2 (A (x + 2) /2) + 3B 2 sech 2 (B (x + 1) /2) with A = 16 and B = 25. The classical KdV equation is recovered for η = 0. Similarly to the K-S equation, the KdV equation is a paradigmatic equation in mathematical physics [31,32], leading to the discovery of solitons, which are a direct bridge between observed The datasets for this work were generated by solving (1) and (2) using an exponential time-differencing fourth-order Runge-Kutta method [33] and a pseudospectral Fourier method in space. Spatial snapshots of these solution fields are used as input to the autoencoder. Figure 1 shows examples of the K-S dataset ( Figure 1a) and the KdV dataset ( Figure 1b). The numerical parameters used for the cases in this work are presented in Table I in Appendix A.

III. METHODOLOGY
Autoencoders are a self-supervised neural network architecture that can be used to find a low-dimensional manifold that represents the data [34,35]. The input to the encoder is mapped to a lower dimensional space called the latent space. The latent space is then expanded through the decoder to reproduce the input to the encoder. An autoencoder with linear activation functions can be shown to be equivalent to the singular value decomposition [36]. In the present work, the input is a snapshot of the solution field obtained from a highfidelity numerical simulation that used N points in space and N t points in time. A snapshot in space is denoted by u n = u (t n ) for n = 1, . . . , N t and u ∈ R N is a vector representing the solution at N discrete points in space. This snapshot is mapped to a latent space of dimension N z with z j (t n ) the j th component of the latent space corresponding to snapshot n. The reconstructed output of the autoencoder is denoted by u n . The weights and biases associated with each node of the autoencoder are tuned to minimize the total loss, is the mean squared error (MSE) reconstruction loss and is the mean absolute error (MAE) penalization loss on the latent dimensions. The sparsity of the latent space is controlled by the regularization parameter λ ≥ 0. A classical autoencoder corresponds to λ = 0. The dimension of the latent space, N z , is not known a priori, but the MAE penalization promotes sparsity in the latent dimensions while the MSE loss boosts the reconstruction performance. The appropriate value of λ will therefore restrict the latent space to the dimensions necessary for a good reconstruction. Figure 2 depicts the autoencoder and loss functions used in this work. All networks used in the current work used fullyconnected networks with sinusoidal activation functions. For conciseness, we denote the encoder architecture by N → H 1 → H 2 · · · → H D → N z where H k represents the number of nodes in layer k and D is the number of hidden layers. The decoder uses the reverse form of the encoder portion. The autoencoders were trained used gradient descent with gradient clipping [37] to limit the maximum gradient to 10. The Adamax optimizer [38] was used in all experiments. The transient portion of the dataset was excluded from both the training and validation sets. In general, 80% of the remaining dataset was retained for training and 20% was used for validation. Table II in Appendix B contains specific details on all autoencoder architectures used in this work including their hyperparameters.

IV. RESULTS
Autoencoders were trained on datasets generated from the K-S equation (1) and the KdV equation (2). The cases considered in this work are summarized in Tables I  and II in Appendix A and B, respectively. Each case was run for a range of regularization parameter values. For each value of λ, the dataset was split into a training and validation set and the MSE (4) was monitored on the validation set during training. Two models were saved for each experiment. One model was saved at the minimum of the training loss curve while the other model was saved at the minimum of the MSE loss on the validation set. The model with the lowest MSE loss on the validation set at each value of λ, L * u (u valid , u valid ), was taken to represent the optimal regularization parameter for that case and was used for analysis.

A. Kuramoto-Sivashinsky Equation
Using the K-S dataset with L = 22, the autoencoder was trained to find u (x, t) with the total loss (3) for λ ∈ 10 −4 , 2 . The architecture of the encoder was 512 → 256 → 128 → 64 → 32, corresponding to N z = 32. Figure 3 (a) presents L * u (u valid , u valid ) for different values of λ and shows a minimum at λ = 0.398. Figure 3 (b) shows the reconstruction by the trained autoencoder using λ = 0.398 on a snapshot from the validation set. More insight can be obtained by passing each snapshot through the trained network and extracting the latent dimensions corresponding to each snapshot. This process results in N t vectors (one for each snapshot), each of size N z . Figure 4 shows that of the 32 latent dimensions, only 10 are consistently active. This is consistent with, but slightly larger than, the known dimension of the inertial manifold for the K-S equation [30,39]. The interquartile range (IQR) can be used as an indicator of the variability of the latent dimensions. When normalized by the largest IQR (the most active latent dimension) a very clear separation between "active" and "non-active" dimensions emerges as shown in the bottom of Figure 4. The largest IQR of the remaining 22 dimensions is 0.238 of the most active dimension.
The trained autoencoder model also provides a way to develop a nonlinear basis for the learned manifold using the technique of activation maximization. The idea behind this technique is to determine the input that maximizes the output of a specific node in the neural network. In the present work, we were interested in the inputs that would maximize each latent dimension. The input that maximized a given latent dimension was interpreted as a component of the basis of the low-dimensional manifold. The input that maximizes latent dimension z j is determined from, and is a regularization used to smooth the resulting field with the regularization parameter β ≥ 0. The function evaluation z j (v) corresponds to an evaluation of the encoder portion of the trained autoencoder with input v. Note that v is not a temporal snapshot of the dataset, but instead represents an arbitrary input to the autoencoder. Gradient ascent was used to solve (6), The step size γ was set to 1 and the regularization strength β was set to 3. The maximization of each latent dimension was initialized from a random distribution in space where each point was drawn uniformly in [0, 1]. The result is a basis for the reduced manifold. Figures 5a  and 5c show the components of the basis and their power spectra, respectively. The power spectra clearly show that the components of the discovered basis consist of a handful of distinct modes. In contrast to the active dimensions, Figures 5b and 5d show that the inputs that maximize the non-active dimensions are constants near zero. Finally, we compute the optimal number of latent dimensions for L = [22,26,30,35,43,45, 50] and repeat the experiment ten times for each value of L. Figure 6 depicts the scaling of the number of latent dimensions with L, along with uncertainty bounds, and shows a nearly linear scaling, consistent with the scaling of the number of active modes [40].  4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

B. KdV Equation
The latent space penalization enabled the discovery of a reduced basis for an equation that has an inertial manifold. In contrast to the K-S equation, the undamped KdV equation does not possess an inertial manifold and, moreover, the dynamics are truly infinite dimensional [32]. The undamped KdV equation therefore provides a test for the method in which the optimal latent space penalization is expected to be zero. Before training the autoencoder, the input was normalized by its maximum value. Following the same procedure as for the K-S equation, the optimal latent space regularization was determined to be λ = 0 implying that the system does not possess an underlying finite-dimensional manifold. Figure 3 (c) presents a snapshot of a solution from the validation set and the corresponding prediction from the autoencoder at λ = 0. The lack of a finitedimensional manifold is further supported by Figure 7, in which the IQR for each snapshot of the dataset is visualized for each dimension in the latent space. There is no separation of the latent space dimensions into "active" and "nonactive" components.

C. Damped KdV Equation
As a final test case, the damped KdV equation was studied over a range of damping coefficients, η (see Table I in Appendix A). The damped KdV equation possesses an inertial manifold and it therefore provides a fertile test ground for understanding the behavior of the latent space as the damping coefficient is varied. For each damping coefficient, the autoencoder with latent space penalization was trained across a number of regularization parameters. The number of active dimensions for each damped KdV equation had to be determined. However, the minima in the MSE loss, L u vs. regularization parameter, λ, on the validation sets were shallow. Figure 8a presents a representative example for one particular model. Rather than extract a single minimum as was done in the K-S case, a different procedure was employed to provide a more robust result. As before, the model with the minimum validation loss was extracted at each λ and used for the analysis. Instead of retaining a single model at the minimum of the L u (λ) curve, all models were retained in the shallow minimum. The latent dimensions for every snapshot of the dataset were computed for each of these models and the IQR was again used to assess the activity of each latent dimension. The number of active dimensions was computed by counting the number of latent dimensions with IQR above a certain threshold, IQR thresh , for each model. To guard against sensitivity in the choice of IQR thresh , a range of thresholds was tested between 0.1 and 0.8. This process results in an array representing the number of active dimensions for each value of η. The array contains the number of active dimensions for each model in the shallow minimum and each threshold value. The median number of dimensions over each model in the shallow minimum for each case and each threshold value was computed, which resulted in an array for each case that represents the number of active dimensions for each choice of threshold. As expected, the threshold value only shifts the number of active dimensions up and down, but the behavior of the number of active dimensions over η for each threshold value remains the same. The final result is obtained by taking the median over the threshold values. Figure 8b shows the variation of active dimensions as the threshold is increased. The number of active latent dimensions   decreases nearly monotonically with increasing damping coefficient as shown in Figure 9.

V. CONCLUSIONS
We introduced an autoencoder with latent space penalization applied to datasets generated from nonlinear partial differential equations in physics. The latent space penalization is used to restrict the dimensionality of the latent space to the minimal number of dimensions, thereby ensuring that the dynamics embodied in the dataset are captured by a low-dimensional manifold. In the case of the K-S equation, the optimal latent space dimension was consistent with the known dimensionality of the inertial manifold for bifurcation parameter L = 22. We then determined a nonlinear basis for this manifold, which could in principle be used in a reduced order model. The KdV equation does not posses an inertial manifold and  has truly infinite dimensional dynamics. In this case, we found optimal results without any latent space penalization, which is consistent with the known properties of the KdV equation. Finally, when applying this technique to the damped KdV equation, which once again has an iner-tial manifold, we found the beginnings of power law scaling for the number of active dimensions for sufficiently large damping coefficient. This technique opens interesting directions for determining reduced order models.