{'fraction_non_alphanumeric': 0.031727430555555554, 'fraction_numerical': 0.011197916666666667, 'mean_word_length': 5.247559652928416, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Dipole Moments and Polarizabilities

Machine learning of scalar molecular properties such as potential energy has enabled widespread applications. However, there are relatively few machine learning models targeting directional properties, including permanent and transition dipole (multipole) moments, as well as polarizability. These properties are essential to determine intermolecular forces and molecular spectra. In this chapter, we review machine learning models for these tensorial properties, with special focus on how to encode the rotational equivariance into these models by taking a similar form as the physical definition of these properties. You will then learn how to use an embedded atom neural network model to train dipole moments and polarizabilities of a representative molecule. The methodology discussed in this chapter can be extended to learn similar or higher-rank tensorial properties, such as magnetic dipole moments, non-adiabatic coupling vectors, and hyperpolarizabilities.


[H1 Introduction]
Modern quantum chemistry theories offer various ways of solving electronic Schrödinger equations. These solutions yield not only the total potential energy, but also the corresponding wavefunction and electron density, from which a diversity of physicochemical properties of molecules and materials can be derived. With the aid of machine learning (ML), accurate and efficient high-dimensional representations of potential energy [1] and electron density [2] have emerged in recent years, which have accelerated first-principles molecular simulations in a wide range of systems by several orders of magnitude [3][4]. Such advances have been reviewed in several chapters of this book. While in this chapter, we focus on ML of (permanent or transition) dipole moments and related properties, such as multipoles and polarizabilities, all can be obtained from quantum chemical calculations. These properties are essentially responsible for the interaction between light and matter at different levels [5], which can be used for computing infrared spectra (dipole moment), Raman spectra (polarizability), and electronic spectra (transition dipole moment, as will be seen in the next chapter), etc. On the other hand, they are often useful for evaluating the long-range electrostatic, induction, and dispersion contributions to intermolecular forces [6]. As a result, learning these properties is of great importance both in spectroscopic simulations and in the construction of force fields with inclusion of long-range interactions.
It is important to note that the potential energy, dipole/multipole moment, and polarizability tensor have different behaviors under symmetry operations. As a scalar property, the potential energy is invariant with respect to overall translation, rotation, 4 and permutation of identical atoms of the system. In comparison, the permanent dipole moment is a measure of charge separation in a system, which is a vector classically expressed by the separated charge (q) multiplied by the vector between the positive and negative charges ( r ), i.e. =q μ r . In a more general description, electric multipole moments are correlated with the multipole expansion of charge density, where dipole moment ( μ ) is the first-order term and there are second-order (quadrupole, Θ ), thirdorder (octupole, Ω ) moments, and so on and so forth. For example, μ and Θ are first-and second-rank tensors with the following components [7], where i q is the i th point charge, i r is its position relative to a certain origin (typically the center of mass or the central atom in the atomistic ML framework) in the molecule, with the Greek subscripts denoting x, y, z components, 1    if α≠β, otherwise 0    . These multiple moments determine the weak interaction of a molecule with an external field. In addition to these ground state response properties, it should be noted that the transition dipole moment associates with the transition between two different electronic states, and is computed from the integral over electronic coordinates of the dipole moment operator with respective wavefunctions, where i  and f  correspond to the initial and final state wavefunctions, respectively. Since the transition dipole moment reflects the change of charge 5 distribution upon electronic transition, its vectorial direction depends on the transition type (e.g. parallel or perpendicular to the molecular plane) and its sign depends on the relative phase of the initial and final state wavefunctions, resulting in additional complexity for ML.
On the other hand, the polarizability (α) is a second-rank tensor as a result of the second-order response of potential energy (V) to an electric field ( E ), reflecting the ability of the distortion of the charge distribution by the field, While the polarizability can depend on the field strength, we are often interested in the static polarizability at 0  E . Apparently, these tensorial properties contain multiple coordinate-dependent components that are translationally and permutationally invariant, yet equivariant with respect to rotation, which means that rotating the molecule will lead to an equivalent rotation to the whole tensor. Next, we will review various approaches to deal with these issues and emphasize the essence of mimicking the physical definition of each property in developing relevant ML models.
This strategy requires the molecule to be placed in a fixed body-fixed frame when performing ab initio calculations and fits individual dipole moment components independently, which is very system-specific and does not guarantee symmetry rigorously. Likewise, it remains a common way for bigger systems to manually align all configurations to a local reference frame so as to learn the one-to-one correspondence between molecular structures and individual components of a tensorial property [17][18][19][20][21][22][23][24]. Specifically, Popelier and coworkers proposed to replace the empirical functions with machine learned atomic multipole moments for specific molecules in terms of internal coordinates [17,22], with the aim to better model the electrostatic long-range interactions in polarizable force fields [23]. This idea was later extended to learn a transferable model of atomic multipole moments for a wide range of molecules with the Coulomb-matrix descriptor [18]. Marquetand and coworkers [21,24] and Jiang and coworkers [20,[25][26] instead focused on learning permanent and transition dipole moments with various structural descriptors and applying machinelearned properties to boost the simulations of ultraviolet and infrared spectra. Despite these successful applications, this strategy is less well-defined in heavily-distorted or dissociable systems and could cause discontinuity at the boundary of symmetry operation.
Taking the rotational equivariance of a tensorial property fully into account is necessary to obtain a continuous and accurate representation. The first properly 7 symmetry-adapted representation of permanent dipole moment can be perhaps traced back to the pioneering work of Bowman and coworkers [27] using the so-called permutationally invariant polynomials (PIPs) [28][29] approach. These PIPs are symmetrized products of functions of relevant interatomic distances according to the invariant theory [28], which are invariant with respect to permutation of any identical atoms and thus suitable for PES construction. To fit dipole moments, the authors expand the dipole moment as the sum of atomic contributions with the assumption that electronic contributions in Eq. (1) have been integrated out and concentrated at individual atomic sites. In such so-called "atomic charge" or "partial charge" representation, one only needs to fit the scalar atomic charge and the vectorial property is guaranteed by the atomic position vector. However, it is important to realize that the charges of two identical atoms should be exchanged under the permutation operation, while the charge of any unique atom should remain unchanged. In practice, therefore, effective atomic charges of unique atoms will be expressed by invariant polynomials with full permutation symmetry, while those of identical atoms have to take a lower symmetry that distinguish relevant atoms making the resultant PIPs interchangeable with respect to permutation [30]. This adaption successfully encodes the correct equivariant property and permutation symmetry of the dipole moment and has been widely applied in learning permanent dipoles of small molecules [31][32][33]. The same trick was adopted in the PIP-NN approach developed by Guo and coworkers [34][35] to develop several symmetry-preserving permanent dipole surfaces [36][37]. This approach was further extended by Medders and Paesani to derive formally exact many-body 8 expansions of the dipole moment and polarizability for liquid water [38].
The drawback of using a molecule-wise descriptor like PIPs is that its complexity increases dramatically with the size of the molecule. Indeed, on the basis of the atomic charge model, it is more natural to develop an atom-wise ML representation for the dipole moment. The first successful atomistic ML model was proposed by Behler and Parrinello in 2007 for representing high-dimensional PESs [39]. The core idea later adopted in many ML potentials (see the first three Chapters of Part 2) is to decompose the total energy into atomic energies. For a system with N atoms, each of which depends on the respective atom-centered local environment only. Each atomic energy is a scalar output of an atomic NNs whose parameters are elementdependent, making the total energy permutationally invariant. While this Behler and Parrinello type of NN (BPNN) model was proposed to learn total energy since atomic energies are not meaningful outcomes of quantum chemistry, it was soon extended to predict atomic environment-dependent charges [40] computed by the Hirshfeld charge partitioning scheme. [41] Note that atomic charges of identical atoms, like atomic energies, are naturally interchangeable in this BPNN model, thus avoiding the more intricate symmetrization in PIPs. This idea was later taken by Rai and Bakken [42], who instead used random forest to fit atomic charges to reproduce the quantum chemical electrostatic potential and then retrieved dipole moments using Eq. (1). 9 These studies have laid the foundation for later ML models of tensorial properties, with the wide acceptance of the atomistic concept in the community. To avoid the arbitrariness in any charge partitioning scheme, Marquetand and coworkers [43] first proposed to directly train BPNNs to reproduce dipole moment (like the total energy), in which the environment-dependent atomic charges are inferred as latent variables (like atomic energies), by minimizing the following cost function, Here Ndata is the number of data points in the training set, NN and Ref in superscripts refer to the values of NN output and the target. Note that Q is the total charge of the system (e.g. Q=0 for neutral species) and Qq   , which is actually not a necessity in Eq. (6) if the dipole moment is the only quantity of interest. In such a scheme, the only difference between training dipole moment and potential energy is that now the output of atomic NNs is first multiplied with the atomic position vector and then summed over to generate the total dipole moment. This strategy can be easily extended to include higher order multipole moments [43]. This atomic charge model has been adapted with other atomistic ML models to learn permanent dipole moments [44][45][46][47]. This is made by default in some NN packages, such as TensorMol [48] and PhysNet [49], so as to conveniently take long-range electrostatic interactions into account. It has also been recently extended to learn transition dipole moments for a number of excited states of several molecules, with which the UV spectra of similar molecules yet not included in the training set were 10 reasonably reproduced [50]. However, being an excited-state property, learning transition dipole moment is more involved than learning the permanent counterpart (see next Chapter Learning excited-state properties for more details). First, since the phase of quantum wavefunction is always arbitrary, so is the sign of transition dipole moment, a proper phase correction needs to be done before using any kind of ML model. While this is not a trivial task, Marquetand et al. have proposed a pragmatic procedure for phase correction by computing wave function overlaps between new geometries and a reference one (e.g., equilibrium geometry), with which the complete training set can be phase corrected [21]. The same group further developed an automatic phase correction program by enabling the sign flip during training and chose the proper one with lower fitting error [51]. Second, it is less recognized that the direction of transition dipole moment also depends on the molecular orbitals involved in the transition. For example, the transition dipole moment vector of a planar molecule will be in plane for parallel transition, while out of plane for perpendicular transition. However, since all atoms lie in the plane, none of atomic dipoles defined in Eq. (1) will have out-of-plane components. As a consequence, the simple atomic charge model would completely fail to predict a perpendicular transition dipole moment. It should be noted that this influence is not limited to pure planar geometries. Indeed, such deficiency will certainly introduce a distortion in the feature space, which hinders the ability to perform the overall regression of transition dipole moments. To solve the problem, we introduce two analogous vectors to the permanent dipole [45], namely, μ will define a specific plane (e.g., the molecular plane for a planar geometry) and their cross product will give rise to a third vector perpendicular to this plane, 3 3 [H2 Learning polarizabilities] Compared to the large amount of work for learning dipole moment as discussed above, ML models that fully account for the symmetry of higher-rank tensorial properties are scarce. Based on Gaussian process regression, Ceriotti and coworkers have first made use of a tensorial kernel function in spherical coordinates to describe the structural similarity between molecular configurations and meanwhile account for the rotational equivariance of the tensorial properties up to arbitrary order [52][53][54]. This so-called symmetry-adapted Gaussian process regression (SA-GPR) model for polarizability tensor has been applied to simulate the vibrational Raman spectra of Paracetamol in various crystal forms [55]. By effectively learning an atom-centered model of the polarizability, the SA-GPR scheme has been used to predict the polarizability in chemical space that are transferable between different molecules. [53] Taking tensorial kernel functions is sufficient to preserve the desired symmetry of the kernel-based tensorial representation, because its output is essentially a liner product of tensorial kernels. However, using the same strategy to construct the rotationallyequivariant input would not work for conventional NNs, because feeding such an input into an arbitrary nonlinear function like NNs will generally lead to predictions that do not satisfy the desired symmetry properties. [54] Following the definition in Eq. (4), we recently modified the atomic charge NN framework [45] to learn the polarizability tensor. To this end, we design an auxiliary matrix (D) given by the partial derivatives of atomic charge with respect to atomic coordinates (analogous to the atomic force matrix, in 3×N dimension), 1 ( 1, 2, ... , ) Here, D J is the jth column of D matrix. Multiplying the D matrix by its transpose gives us a symmetric 3×3 matrix that has the same transformation as α, However, α NN1 is a semidefinite matrix by construction, while molecular polarizability itself is not necessarily semidefinite. We thus create another symmetric matrix NN2 α in the following way, 13 where r is the 3×N Cartesian coordinate matrix. Furthermore, it is important to note that both NN1 α and NN2 α become a rank-deficient matrix when the molecular geometry is planar, while the molecular polarizability tensor is not. The simplest way to correct this is to incorporate a scalar matrix NN3 α , whose diagonal elements are identical which can be thus represented by a scalar EANN (just like the potential).
Combining these three terms yields the full representation of the NN-based polarizability tensor, that fulfills the symmetry of α. This approach is in principle applicable to any atomistic ML model with symmetry-invariant descriptors.
A similar approach was independently developed by Sommers et al. [56] based on Diagonal elements of F correspond to M scalar outputs of regular DPMD. More details on the DPMD method may be found in the original publication [56] and Chapter 13: Dynamics with neural network potentials. In practice, this expression has more or less a combinative effect of Eqs. (10) and (11). Sommers et al. [56] have applied this model to liquid water and simulated the Raman spectra by machine-learned polarizabilities.
However, this approach will in principle suffer from the aforementioned rank deficiency issue for a planar molecule, since each row of T resembles a vector in the same plane in this case.
[H2 Other relevant approaches] Significant efforts have also been made in developing equivariant message-passing type of neural networks (MPNNs), which can preserve directional information by making the neurons not only scalars but also vectors (tensors) that are internally equivariant to rotation [57][58]. Very recently, equivariant MPNNs have been adapted to predict dipole moments and polarizability tensors enabling accurate infrared and Raman spectra from MD simulations [59], as well as multipole moments to describe electrostatic interaction [44]. Interestingly, a similar expression as Eq. (12) was used by Schütt et al. in their MPNN model for learning polarizability, who have rationalized that the first two terms are related with the anisotropic components of the polarizability tensor, while the last term corresponds to the isotropic component [59]. 15 We note that there are some alternative approaches for learning tensorial properties differing from those discussed above. Within kernel-based regression, for example, Christensen et al. applied the response operator to kernel functions to mimic the computation of the response properties of potential energy to an electric field, which explicitly incorporates electric field gradients into the training [60]. This is perhaps the only ML model that yields field-dependent quantities. On the other hand, one may directly learn the equivariant ground state electronic wavefunctions (see Chapter Learning wavefunction), e.g. using the SchOrb model [61][62], from which the dipole moment, polarizability and related quantities can be all obtained.
[H2 Learning tensorial properties with embedded atom neural networks] Before we move on to case studies, we describe the details of learning tensorial properties with our recently developed embedded atom neural network (EANN) model [45,63]. This model is inspired from the empirical embedded atom method (EAM) force field which was originally applied to metals [64], also based on the atomic energy where ˆˆî j i j  r r r , withˆ( , , )  r is the Gaussian-type orbital centered at atom j parameterized by its center (rs), width (α), and angular momenta (L= lx+ly+lz), has enabled accurate and efficient learning of potential energies and forces [65][66][67][68][69][70], as well as Hessians [71]. It has been found to be more efficient than many other descriptorbased ML models [67] and can be further improved to have a message-passing feature [72].
[H1 Case studies] In this section, you will see how to use machine learning for learning permanent and transition dipole moments, as well as polarizability tensors. We use the Nmethylacetamide (NMA) molecule as a toy system, which is a representative residual in protein backbones [20]. Permanent dipole moments and polarizabilities were calculated by density functional theory (DFT). The perpendicular nπ* and parallel ππ* transition dipole moments, relative to the molecular surface as illustrated in Figure 1, were calculated by time-dependent density functional theory (TDDFT) [73]. All these 17 calculations were performed using the Gaussian 09 package at the B3LYP/cc-pVDZ level [74]. We extracted 10000 NMA configurations from 1000 different protein backbones at room temperature from the Protein Data Bank (PDB) [75], among which 80% randomly selected structures were used for training and the rest were used for testing. Phase corrections have been carefully done by evaluating the wavefunction overlaps between these structures and a reference configuration. These data can be found at https://github.com/zylustc/Learning-Dipole-Moments-and-Polarizabilities.  This chapter reviews some recent advances on learning dipole moments and related tensorial properties, with special focus on how to reconstruct the physical form of each property to achieve symmetry-preserving and rotational equivariant ML models. In such ways, the modern ML models of learning these tensorial properties can be as accurate and efficient as their scalar counterparts of learning potential energies. There is also rapid progress on the development of unified rotationally-equivariant messagepassing neural networks, in which the scalar and tensorial properties can be learned in the same way and different outputs are adapted to corresponding targets [57][58][59]. This type of model is likely to lead to an increased accuracy in the prediction of tensorial properties (yet also an increase of the computational cost). On the other hand, Ceriotti et al. have recently pointed out that it is essential to move beyond the local atomic charge model and include nonlocal effects when describing dipole moments in large 21 molecules whose dipole is almost entirely generated by charge separation [77]. It is worthwhile to note that learning (transition) dipole moments bare much similarity as learning other vectorial properties involving excited states, such as non-adiabatic coupling vectors [51] and spin-orbit coupling vectors [78] (see next Chapter Learning excited-state properties). It is also straightforward to apply the model discussed to learn magnetic dipole moments which are useful for predictions of electronic circular dichroism spectra [79]. Some similar ideas and treatments are also invoked in learning electronic friction tensors of adsorbed species on metal surfaces, which have even more complex permutational and rotational equivariance properties [80][81][82]. Altogether, we look forward to seeing more advances in machine learning of dipole moments and relevant tensorial properties in the near future.