{'fraction_non_alphanumeric': 0.035136724492796235, 'fraction_numerical': 0.010401352543369597, 'mean_word_length': 4.645020746887967, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Symmetrical Graph Neural Network for Quantum Chemistry, with Dual R/K Space

Most of current neural network models in quantum chemistry (QC) exclude the molecular symmetry, separate the well-correlated real space (R space), and momenta space (K space) into two individuals, which lack the essential physics in molecular chemistry. In this work, by endorsing the molecular symmetry and elementals of group theory, we propose a comprehensive method to apply symmetry in the graph neural network (SY-GNN), which extends the property-predicting coverage to all the orbital symmetry for both ground and excited states. SY-GNN shows excellent performance in predicting both the absolute and relative of R and K spaces quantities. Besides the numerical properties, SY-GNN also can predict the orbitals distributions in real space, providing the active regions of chemical reactions. We believe the symmetry endorsed deep learning scheme covers the significant physics inside and is essential for the application of neural networks in QC and many other research fields in the future.


Introduction
The design of molecules with on-demand properties is of significant importance to developing functional materials and drugs, and one of the leading methodologies is density functional theory (DFT) 1 ; however, DFT is extremely time-consuming for the large molecules. To accelerate this process, recently, various machine learning (ML) methods have been applied to QC calculations to approximate molecular energies and other properties. [2][3][4][5][6][7][8][9][10][11][12][13][14] The main advantage of the ML techniques is that, once the model is well-trained, the predictions can be rapid. Meanwhile, deep neural networks (DNNs) have been applied to achieve fast QC calculations. Compared to the kernel methods in ML, the training time of the DNNs scales linearly with the number of data samples. While some approaches make use of painstakingly designed descriptors 15 , message passing algorithms 3 , a type of graph neural network, has been applied to predict properties from molecular structures. Very recently, some neural networks that learn a representation directly from atom types and positions are widely applied. Deep Tensor Neural Networks (DTNNs) 12 is one that enables spatially and chemically resolved insights into the energy of molecules. SchNet 5 , as a variant of DTNNs, uses continuous filter convolutional layers to learn representations for molecules and materials to achieve high precision property prediction in a wide range of quantum-mechanical properties.
Though the success of ML and DNNs has led to wide-spread applications in other scientific fields, the interpretability, and dataset-independent transferability are essential prerequisites to guarantee the value of the ML/DNNs application outcome, especially in QC. However, the efficiency of some current DNNs are dataset dependent and are not working for predicting large molecules. 16 Both the prediction error, memory cost, and training time will dramatically increase while the size of the molecule increasing, indicating the pure numerical ML/DNNs architecture is well-challenged for the large scale demand from QC. Moreover, till now most of the ML/DNNs architectures are introduced from the areas like computer-science or data-science, in which research areas there is little physicsconstrained data correlation, while in QC research, all of the data correlates with the K and R spaces, and a severe defect is that the K and R spaces quantities are treated as separated ones in the previous ML/DNNs' application in QC, which maps the scientific physical chemistry data into pure engineering numerical ones, the solid-state physics or molecular orbital theory actually disappears. The physicsinspired ML models can provide more accuracy and a secondary understanding from the raw data 17 . The critical loss for the K/R correlation is the ignorance of the symmetry, which directly constrains the K/R quantities.
In traditional QC DFT calculation, the symmetry is widely applied to reduce the computation cost of the DFT to that for an N/4 point fast-Fourier-transformation (FFT) with pre-processing and postprocessing. 18 The symmetry operation provides an efficient solution for the structure refine and property prediction for some large molecules in the DFT scheme. Here in order to solve the dataset dependent issue in the DNNs scheme, we extend the symmetry operation to the DNNs scheme and present a novel deep learning model SY-GNN, which includes the symmetric atomic interactions and is capable for covering both the R space and K space features. The introduction of symmetry operation in SY-GNN not only just improved the numerical quantity, more importantly, since the symmetry itself dedicates the atomic interaction, but it also endorses the solid-state physics, which makes the SY-GNN acts more robust, especially in the energy degeneracy and excited states selection than other now available DNNs. Fig. 1A shows the mainframe and overall architecture of SY-GNN. First, we define a molecule ℳ as a set of n atoms, i.e. ℳ = {( 1 , 1 ), ( 2 , 2 ), ⋯ , ( , )} = {( , )} =1 , where Zi is the atomic number, i.e. atom type of the ith atom, and ri is the position of the atom in the 3-dimensional Cartesian coordinate of the ith atom. Though the layers of SY-GNN, the atoms are represented by a tuple of features = � 1 , ⋯ , �, where ∈ ℝ with the lth layer, the number of atoms n, the number of feature map F. The feature of site i is initialized using an embedding layer dependent on the atom type Zi

Method
These embeddings ∈ ℝ are vectors to represent information of atoms disregarding the environment they are in, where d is the dimensionality of vectors. Furthermore, they are initialized randomly and optimized via backpropagation during the training process.
The graph theory layer (GT) is a layer use the related rotation matrix of different symmetry group to find out the primitive unit and the equivalent atoms in a symmetrical molecule. Section S2 shows the details of the GT layer.
Since the molecules are symmetric, the same type of atoms in the symmetric position should have identical features, and we can share tensor values among them but not do the redundant calculation. At the beginning of interaction modules, we pick-up only atoms in a minimal part ⊂ ℳ of a symmetrical molecule. For each molecule ℳ with symmetry, there is the symmetrical operation ()that can be used to perform on the minimal part , and ℳ = ( ) ∪ � ( )� ∪ ⋯.
In the interaction modules, we share their features with other symmetric atoms after all calculations in the interaction module. It means that all the atoms belonging to the same symmetry operation share the same features in the output.
For a molecule with D6h symmetry like benzene in Fig. 1B, during each turn of interaction calculation, we only need to calculate the interaction between one carbon or hydrogen atom and the other atoms. After updating the features for these two atoms, keep the features of other symmetric atoms and the new feature of the carbon and hydrogen atoms in synchronization. Theoretically, we can reduce the amount of calculation by five-sixths.
In the previous models for predicting quantum chemical properties like DTNNs and SchNet, residual connections 19 are applied to connect each layer. SY-GNN applies the dense-connections scheme 20 , and symmetries in molecules are applied to enhance the learning ability of the neural network. Since densely connectivity is used in our architecture, the number of output features of the interaction module is F, while the number of input features is mF, which is related to the index of interaction module m. We concatenate all output from all previous modules to get the input for this module.
The first layer in the interaction module is an atom-wise dense layer that used to make dimensionality reduction of the feature map, which means : ℝ → ℝ . Atom-wise dense layers are fully-connected layers applied to the features ( −1) of each atom i in layer l-1 to learn atom interactions and return the features ( ) as the input of the next layer.
where weights W (l) and biases b (l) shared across all atoms, so SY-GNN is scalable with the number of atoms. This layer is used to reduce the calculation in the cfconv layers while refining the features via interatomic learning.SY-GNN applies the continuous filter convolutional (cfconv) layer 21 , which is a generalization of discrete convolutional layers, and they make sense physically since atoms are in arbitrary positions instead of a gird like pixels in images. The performed cfconv layers can be represented by where "⊙" represents the element-wise multiplication. For computational efficiency, we apply the feature-wise convolutions here. Here we use a filter-generating network ( ) : ℝ 3 → ℝ that maps the position of atoms to the corresponding filter values.
In SY-GNN, shifted soft-plus activate function 5,21 ssp(x)=ln((e x +1)/2) are used as non-linearity after the atom-wise dense layers. The shifted soft-plus function is smooth and ensures ssp(0)=0. It can improve the convergence of the neural network. The overall formula for interaction module is here − 1 is the inverse operation of , sharing the feature according to the symmetry.
After several interatomic interaction modules, the features of each atom updates through the interactions with other atoms. Several dense atom-wise layers continued with a pooling layer are applied to obtain the output properties. Section S3 shows more detail about the pooling layer. Fig. 1B shows how symmetry is applied to share features among atoms in the whole process. The group theory (GT) layer first identifies the primitive unit and equivalent atoms. Compared with the full atomic sets in the conventional cell, the primitive unit contains the least number of atoms without losing any information; it can restore to the conventional molecules through the symmetry operation. Different from the traditional GNN scheme, in SY-GNN, only the atoms in the primitive unit are needed to calculate the interaction with all other atoms. Any atom shares the features with its equivalent ones found out in the GT layer. The three example molecules have the same primitive unit "CH" with different symmetry groups, which are Cyclobutadiene, Benzene, and Cyclooctatetraene, with point groups D4h, D6h, and D8h respectively. Here in SY-GNN, the three molecules are all recognized as the primitive unit "CH" but have different numbers of interactions with other primitive parts inside the molecule. Fig. S1 shows the difference between with and without symmetry when to predict the same molecule.

Results
In this work, we apply the QM9 22 and QM-sym database 23 for the training and predicting procedure. During the training process to predict free energy, without any memory saving tricks, the maximum GPU memory occupied for SchNet is 11759MB, while for SY-GNN is only 4785MB. For all the properties predicted, SY-GNN can significantly save the memory occupied compared to SchNet, which allows us to train on more extensive and more complex molecules on a single GPU without using distributed communication technology. As shown in Fig  Next, we turn for numerical precision. Table. 1 shows the prediction errors of some molecular QC properties from several literature neural network models, including SchNet 5 and FMAPP 2 . The "Type" column includes the classification of the properties from R space quantity or K space quantity, the K space includes the property related to the momenta space, while the R space includes the properties more dependent on the real space character. Besides the K, R catalog, the properties also can belong to the "extensive" (ex) or "intensive" (in) division, like the enthalpy, it directly scales with the number of atoms in the molecules; while for the band gap, it seldom correlates with the molecular size directly. As Table 1 shows, from the criterial of mean-absolute-errors (MAE) and root-mean-square-errors (RMSE), the SY-GNN produced the lowest error and more accurate precision compared with the robust SchNet and FMAPP scheme in the literature. Fig. 3 shows the distance between the DFT result and SY-GNN predicted the result of chemical properties, including the ex (R) quantity isotropic polarizability ( ), heat capacity at 298.15 K (Cv), Free energy at 298.15 K (G), also includes the in (K) properties HOMO energy (ϵHOMO), LUMO energy (ϵLUMO), and band gap. We can see from Fig. 3 that each of the properties can perfectly benchmark the DFT data with the coefficient of determination (R 2 ) approaching 1.00. The prediction result is shown in Table. 1.   In QC, since the R and K spaces are automorphic through the symmetry operation 24 , the quantities from R and K spaces are well-correlated rather than isolated. However, it is difficult for DNNs to learn the symmetry in an unsupervised way 25 ; once the K and R spaces treated individually, there is always a trade-off for the precision of R space quantities once the K space quantities are well trained in DNNs, vice versa. Fig. 4A summarizes K/R space error data from some popular ML models, including Random Forest (RF) with Bond-Angle Machine Learning (BAML) 26 , Kernel Ridge Regression (KRR) with Bag of Bonds (BoB) 27 , KRR with Projected Histograms (HDAD), gated graph network (GG) 7 , MPNN (edge neural network with set-to-set) 3 , DTNNs, SchNet, FMAPP and Multilevel Graph Convolutional Neural Network (MGCN) 4 . The algorithm of the weighted R and K space errors is in section S4. We can see that most of the DNNs models favor more on the R space quantities, with the error ER ranges from 10 −3~1 0 −4 , the K space error EK ranges around 10 -1 , the EK/ER ratio is around 10 3 , such anisotropic distribution of the K/R space errors origin from the simple separation for the correlated K and R spaces. However, in solid-state physics, a coarse R space data quality never reveals the reliability of K space, a 10 3 relative EK/ER ratio indicates the DTNNs actually fails in learning physics inside, a previous research 28 proved that a perfect-fit model even could not handle a simple pendulum problem without getting the right physics shape. We can see from Fig. 4A, the SY-GNN achieves the best error performance in both K and R spaces among all the DNNs scheme, and the EK/ER quantity arrives near 1 because the K and R spaces in SY-GNN are tightly constrained and correlated by the same symmetry operations.
Besides the K/R space catalog, the extensive/intensive data types also matter for the precision. For some extensive data type, like enthalpy (H) and free energy (G), the relative errors will significantly reduce when the size of molecule increase, because of the larger denominator. Fig compares 2 models adopted cfconv layers, SY-GNN, and SchNet. We can see when the size of the molecules is small (Natoms<25), SchNet has low related errors in the intensive quantity prediction, but extremely high related errors of the extensive quantity prediction. While SY-GNN can achieve low related errors for all sizes of molecules, the introduction of symmetry operations only counts on the reduced primitive unit, for the symmetrical QM-sym database, the extensive quantities never scale with the number of molecules and only dependent on the smaller primitive unit.
Since the symmetry here refers to the atomic symmetry, it is about the atomic arrangement, especially the bonding types. Naively, the bond order can directly reveal the or type of bonding in the carbon-based system, and the chain or ring motif also can directly refer to the aliphatic or aromatic hydrocarbon. Fig. 5A shows the local motifs separation after the training in SY-GNN based on the heat capacity (Cv) and electronic spatial extent (⟨ 2 ⟩), both of these two features are extensive in real space. Cv is almost directly proportional to the number of atoms. While ⟨ 2 ⟩ can refer to the effective volume of a molecule in Gaussian package 29 , for the same number of atoms, the chain-shape motif occupies less volume and relatively small ⟨ 2 ⟩ compared with the ring structure, thus SY-GNN can determine the existence of ring and chain-motifs, which is essential for obtaining the orbital symmetry from group theory. Fig. 5B illustrates the detailed procedure of how SY-GNN can map the energy levels in the momenta space to the orbital distribution in real space, taking the molecule shown (named 1,2,3,4tetrapropylidene-cyclobutane, with C4h symmetry) as an example. First, SY-GNN gets whether there is ring-motif in the molecule and confirms the π orbital bonding types. Since SY-GNN has learned the molecular symmetry, once the π orbitals are sets as the basis, the orbital symmetry can be well derived for the LUMO+1, LUMO, HOMO, and HOMO-1 orbitals, respectively. Meanwhile, the orientation of the pz orbitals can also be derived from the symmetry (Details are in section S5). Combined with the location of the atoms in the lattice, the distribution of the given orbital can directly be predicted in real space. As in Fig. 5B, the predicted orbital distribution is compared with the Gaussian09 results. Most of the previous DTNNs provides the band gap predictions alone, which reveals limited information within the K space only. While the combination of the orbital energy in K space and orbital distribution in R space, especially for the frontier orbitals like HOMO, LUMO, can be crucial for the practical application of materials design and properties modifications technologies 30,31 .
Moreover, SY-GNN can also provide more scopes in the excited states. By defining symmetry operations of the initial and targeting orbitals ψi and ψj, and the transition moment operator µ, the intensity of the transition formulates as (Detailed explanation in S6):

∝ ∫ ⊗ ⊗ �7�
For the molecule 1,2,3,4-tetrapropylidene-cyclobutane in Fig. 5B with C4h symmetry, SY-GNN can directly predict that the transition via light from HOMO to LUMO is dark, and the first bright transition is HOMO to LUMO+1 with the in-plane (x,y) excitation, as shown in the inserted table in Fig. 5B. In Fig. 5B, the energy levels are also shown correlated to the hopping parameter β. (Detailed in section S5) By comparing the orbital energies from the Gaussian output, SY-GNN can get the value of directly. In the example of 1,2,3,4-tetrapropylidene-cyclobutane, the average value of -β is 23.35 kcal/mol, which is close to the theoretic 18.8 kcal/mol 32 , the difference could originate from the details 33 in DFT during the training procedure. Thanks to the enclosed symmetry, SY-GNN can predict more physical quantities than the traditional DNNs in the literature, including the wavefunction distribution and symmetry of spectral transition of electrons.

Fig. 5. Prediction of SY-GNN on orbital-associated features. (A)
Feature vectors for the rings and chains motifs with ⟨ 2 ⟩ and Cv, the green and pink colors illustrate the ring and chain motifs, respectively, we pick up 1/200 structures in the database for clarity. (B) Scheme for the orbital distribution prediction in SY-GNN. First, the motif information with symmetry is derived, then the orbital symmetry and orbital energy are calculated by group theory and benchmarks with the orbital distribution predicted by Hückel theory. The blue and yellow color represent the + and -sign of pz orbital, both in the predicted and Gaussian09 output. The inserted table shows the product table of the related orbitals.

Conclusions
To conclude, in this work, we developed the symmetry endorsed graph neural network SY-GNN, for the application of accurate and accelerated QC properties prediction. Besides the significant acceleration in computing efficiency, it recovers the correlation between the automorphic K and R space through the molecular symmetry, learns the K/R space quantities in a synergetic way, resulting in the lower errors in both of the absolute and relative domains between the K/R space. It also includes the symmetrical Hückel molecular orbital theory and can predict the orbital distribution in K and R spaces, including the selection rules for the excited states. The SY-GNN provides more theoretic and physics asides for the neural network's application in quantum chemistry.

Section S1. The flow of the SY-GNN model
The SY-GNN model uses atomic numbers Z and atomic coordinates r as inputs. The main algorithm can be summarized as follows: 1) Use group theory to find out the primitive unit and equivalent atoms of the symmetrical molecule.
2) Encode each atom in the molecule into the feature space.
3) Based on equivalent atoms information, the interaction module updates the feature of atoms in the primitive unit. 4) Concatenate the input and output of the interaction module and share the feature among equivalent atoms. 5) Repeat steps 3 and 4 according to the number of interactive modules. 6) Sum up the features of all atoms in the molecules. 7) Use different output layer to output the predicted result.

Section S2. Graph Theory Layer
Algorithm 1 shows an example of the process of graph theory layer to determine the equivalent atoms of a molecule with C3h symmetry.

Section S3. Details for SY-GNN architecture
For predicting intensive properties like dipole moment, highest occupied molecular orbital (ϵHOMO), lowest unoccupied molecular orbital (ϵLUMO), zero-point vibrational energy (ZPVE), the symmetry of orbital and band gap, average pooling is applied. For predicting extensive properties, including isotropic polarizability (α), electronic spatial extent (⟨ 2 ⟩), energy (U, U0), heat capacity (Cv), free energy (G), enthalpy (H), we calculate the sum of all the atomic contributions as the pooling layer.

Section S4. Calculation of K/R space error
The data used to calculate K/R space error for all models except SY-GNN are from Lu et al.'s paper. The prediction error of U, G, H for models RF+BAML, KRR+BOB, KRR+HDAD, and GG vibration-related lack. Based on our experience, the prediction errors of U, G, H are very closed to the prediction error of U0. So, for these four models, we assume the prediction error of U, G, H are the same as the prediction error of U0.
There are eight properties belongs to R space, α, G, H, U, U0, ⟨ 2 ⟩, Cv, and ZPVE. Since both U0 and u represent the internal energy of the molecule but only at different temperatures, we only include U0 in the calculation of R space error and exclude U. All 4 properties, ϵHOMO, ϵLUMO, ϵgap, and μ are used to calculate the K space error.
We use ̄a vg to represent the average size of molecules in the database. For QM9, it should be about 17.98, and about 49.76 for QM-sym. We can calculate K/R space error by following formulas: Almost all properties in R space are extensive, so we add a term ̄a vg 2.5 in the denominator to eliminate the effect of the size of molecules. We choose the power of 2.5 by considering the dimension of molecules in QM-Sym. Since the molecules increase the size in the 2D plane, and each molecule has 3D real structure, we take a 2.5 as the dimension of QM-Sym. Almost all properties of the K space quantities are intensive, so we do not all this term.

Section S5. Shapes of the molecular orbitals
The location and the shape of the orbitals could be determined given the irreducible representation of the corresponding orbitals. In the database utilized, a molecule whose symmetry group is C4h was picked as an example to show the procedure.
The example is called 1,2,3,4-tetrapropylidene-cyclobutane, as shown in Fig. 5B. The atoms on the central cyclobutene were numbered 1, 2, 3, 4, and the atoms linked to them via the double bonds were numbered 5, 6, 7, 8, respectively. In such a molecule, since the double bonds occur only on the atoms numbered 1,2,3,4,5,7,8, in determining the π orbitals, only these 8 atoms require attention. The character table and the representation of the π bonds are as in Table. S1.
Thus, according to the decomposition formula = ( ) ( )/ℎ, the result could be written in the form: indicating that there are two Au orbitals, two Bu orbitals and four Eg orbitals. Based on the transformation properties and the character table of the given molecule, the orbitals could be determined as: : � 1 + 2 + 3 + 4 5 + 6 + 7 + 8 : Beware that the above wave equations require normalization. By putting orbitals with the same symmetry in one secular equation to solve for the eigenvalues and plotting them for the energy, with the assumption that the on-site energy α=0, and the hopping parameter β<0, the energy level diagram could be drawn, as in Fig. 5B. In constructing the secular equations, the entries are calculated with the Hückel approximation: Based on the energy of the energy level, i.e. the eigenvalue of the corresponding secular equation, one could solve for the linear combination of the wave equations involved and thus determine the shape of the molecular orbitals based on the absolute value and the sign of the coefficients. Take Bu as an example, the secular equation from the wave equations of Bu is indicating that the wave equations of Bu are along the same direction. Thus, as can be seen in the figure, for Bu orbital, the atomic orbitals ψ1, ψ3, ψ5, ψ7 are along the same direction, while the others are opposite.

Section S6. Spectral transition probability
The transition probability of electrons between the molecular orbitals via light could be determined by the tensor product of the relative irreducible representations of the initial and final energy levels.
By picking the irreducible representations of the involved orbitals as ψi, ψj, and defining a transition moment operator µ for the incident light, the intensity of the transition is given by the equation (7).
The characters of the representation of the tensor product are equal to the products of the characters for the same symmetry operation of the representations involved. Only when the symmetric operation is present in the result of ⊗ ⊗ , will this integral be nonzero, i.e. the transition of electrons from ψi orbital to ψj orbital via operator μ is possible. In the case of the spectral transition of electrons, the transition moment operator is the irreducible representation with the same linear symmetry as the incident light. For example, if the molecule is of C4h symmetry as described by the character table above, the incident light along the z-direction is described by μ=Au.
As an example, the transition property of the molecule above, from HOMO to LUMO, can be calculated as in Table. S2.
As shown in the decomposition, since Ag occurs in only two of them decompositions, so only when the light is incident along the xy plane, will the spectral transition for HOMO -1 → MOand HOMO → MO +1 be possible. The others will not be able to transit via light along x, y, z directions, but might be able to be triggered by other mechanisms.

Figures and Tables for Supplementary Materials
Fig. S1. The difference for the model to process with and without symmetry. The figure shows that for trans-1,2-Dichloroethylene with C2h symmetry, symmetry can help to reduce half redundant interaction calculation.