{'fraction_non_alphanumeric': 0.02512072808320951, 'fraction_numerical': 0.008636701337295691, 'mean_word_length': 5.123684958771681, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A photonic chip-based machine learning approach for the prediction of molecular properties

Machine learning methods have revolutionized the discovery process of new molecules and materials. However, the intensive training process of neural networks for molecules with ever-increasing complexity has resulted in exponential growth in computation cost, leading to long simulation time and high energy consumption. Photonic chip technology offers an alternative platform for implementing neural networks with faster data processing and lower energy usage compared to digital computers. Photonics technology is naturally capable of implementing complex-valued neural networks at no additional hardware cost. Here, we demonstrate the capability of photonic neural networks for predicting the quantum mechanical properties of molecules. To the best of our knowledge, this work is the first to harness photonic technology for machine learning applications in computational chemistry and molecular sciences, such as drug discovery and materials design. We further show that multiple properties can be learned simultaneously in a photonic chip via a multi-task regression learning algorithm, which is also the first of its kind as well, as most previous works focus on implementing a network in the classification task.


INTRODUCTION
Data-driven approaches, in particular machine learning methods, have in the past decade become an indispensable tool for material design and molecular discovery [1,2]. The availability of large datasets, both experimental and computational, and the advancements in algorithms and computer capability significantly accelerate the screening and identification of molecules and materials with desired properties in vast chemical space. Among the machine learning methods, deep learning methods involving large and sophisticated neural networks have emerged as the leading candidates for molecular property prediction. However, despite their promising performance, training of such deep neural networks inadvertently leads to substantial computational cost associated with expensive hardware and high power consumption. For example, the highly successful recent work, Alphafold [3], for protein folding typically requires training times in the order of weeks using hundreds of costly graphic processing units (GPUs) [4]. Furthermore, while the computational costs of machine learning models continue to double every few months [5], the efficiency of digital computational chips has not managed to improve at a similar pace [6]. Thus, training of ever more sophisticated machine learning models leads to exponential growth of energy consumption [7], which comes with a huge environmental cost [8].
Optical computing offers an alternative paradigm to conventional digital computing for computation-intensive tasks, such as implementing deep neural networks. Optical computing implementations of machine learning methods have been realized in neuromorphic photonics [9,10], all-optical neural networks [11,12] and photonic reservoir computing [13,14].
In fact, optical implementation of neural networks has recently been demonstrated to surpass cutting-edge GPUs in terms of speed and energy consumption. Optical computing offers several advantages over conventional digital computers, such as low power usage [15][16][17], ultrafast optoelectronics with high noise robustness [18], inherent parallelism [19], and large information storage [20]. Furthermore, optical neural networks implementing complexvalued neural networks have been developed recently [21]. Despite the promises of optical neural network, its potential in molecular sciences such as chemistry and biology has not been explored. Additionally, most current works on optical neural networks focus on single-task learning (STL) method. Yet, humans typically acquire knowledge and arrive at generalization through multi-task learning (MTL). In the study of molecules, we often encounter the need to predict the values of many properties (e.g. free energy and enthalpy) as the most desirable molecules or materials typically optimize multiple properties simultaneously. This sort of multi-task learning differs fundamentally from single-task classification and requires modfied algorithms. Instead of producing a category code to categorize a set of input data into a specific class, we seek to produce and optimize a continuous real-valued output [22].
It also requires us to be able to learn multiple properties (tasks) simultaneously in hope that we can leverage the knowledge contained in one task to generalize and improve performance of other tasks [23,24]. With this motivation in mind, we hope to implement multi-task regression neural network within the optical scheme.
Furthermore, molecular properties are inherently quantum mechanical and its calculations involve complex-valued arithmetics, it would therefore be appealing to attempt molecular property prediction with complex-valued neural networks, a function that can be accomplished on an optical chip with no additional hardware cost.
In this work, we apply an optical neural chip to the task of implementing a complexvalued neural network to predict multiple quantum mechanical properties of molecules.
Specifically, the contributions of our work are of three folds. Firstly, it constitutes the first application of optical machine learning methods to molecular sciences. Secondly, previous works of optical neural networks only consider single-task classification learning, here we extend the optical neural networks implementation to a different class of machine learning tasks, namely multi-task regression learning, and obtain satisfactory accuracy. Lastly, we demonstrate that complex-value neural networks offer superior performance for molecular property prediction as compared to the conventional real-value neural networks.

Molecular Database and Machine Learning Model
In this work, we use a database consisting of computed molecular properties, the QM9 database [25], to train a photonic neural network. The database contains approximately 134k molecules that consist of up to 9 non-hydrogen heavy atoms (carbon, oxygen, nitrogen and fluorine). Some notable entries in this database are small amino acids, such as GLY and ALA, and pharmaceutically relevant organic materials, such as pyruvic acid, piperazine and hydroxy urea. The molecular geometries and chemical properties in the database are calculated by density functional theory methods.
For this dataset, we encoded the chemical structure of the molecule in a Coulomb matrix (see Methods for details). Since the largest molecules in the database contain up to 29 atoms (including hydrogen), the Coulomb matrix for each molecule is of size 29 × 29, and zero padding is applied to the molecules with fewer than 29 atoms. As we will elaborate later, for our current optical chip we are limited to 16 real feature inputs, which are then subsequently encoded in 8 complex inputs on the chip. To reduce the dimensionality of the Coulomb matrix to 16, we adopted the eigenspectrum representation of the Coulomb matrix. For each Coulomb matrix, we diagonalized it to find its 29 eigenvalues, and only took the largest 16 eigenvalues as our feature set for each molecule. Our simulations show that even with a restricted subset of the eigenvalues, the dataset is still rich enough to support accurate learning of the chosen chemical properties (See Figure 4 in our Methods).
We focus on the predictions of three molecular properties: enthalpy, internal energy, and free energy. More details about our model, such as label scaling and model validation, can be found in Methods.

Details of Optical Chip
Diagrams of the optical chip are shown in Figure 1. Similar to a typical neural network implemented on a classical computer, it comprises of an input layer, multiple hidden layers, and an output layer. In an optical neural chip (ONC), light signals are encoded and manipulated by controlling their magnitude and phase. The Maxwell equations that describe such systems of interacting wave guides are inherently complex in nature, naturally giving rise to a complex arithmetic. Usually an arbitrarily complex transformation matrix W can be decomposed into W = U DV via Singular value decomposition, where U and V anre unitary and D is a diagonal matrix. For this work, we consider the transformations that can be expressed as W = U D. This is not universal, but has a large searching space compared to just a single unitary matrix, and experimentally meets our requirements for the prediction of molecular properties.
The ONC chip implements this transformation by acting as a multiport interferometer.
Multiport interferometers implement linear transformation between several optical channels by arranging many Mach-Zehnder interferometers (MZIs) in a specific pattern (a triangular mesh) [26,27], and it can be shown that any discrete unitary operator can be decomposed and implemented in such a way [28]. Each MZI consists of two beam splitter (BS) -phase shifter (PS) pairs. By setting the transmissivity of both BS to be fixed at 50 : 50, one MZI implements the complex unitary transformation between the 2 neighbouring wave guides indexed by p and q: which is essentially a rotation matrix. The PS can be thermally modulated to tune their phase shifts, allowing us to reconfigure the chip parameters without requiring complicated The optical neural is thus capable of implementing a complex-valued neural network, which is similar to a conventional real-valued neural network that is implemented on classical computers, with the only difference being that it relies on complex arithmetic. For a system with N wave guides and an arrangement of MZIs described in [26][27][28], this has the effect of implementing a hidden layer of N complex-valued neurons. The output of such a neuron can be expressed as: where the weights ω i and bias b are complex numbers and are determined by the specific choices of θs and φs of the MZIs in the ONC chip.
The last (right most) layer of MZIs can be used to implement the output layer by enabling coherent detection. The light signals at the end of the neural network contain both magnitude and phase information, and we would optimally want to access both, rather than just the magnitude information. By additional two columns of MZIs that interfere any two adjacent output ports, we are able to determine the phase of the output signals by varying The Supplementary Information also contains comparisons of a similar complex-valued and real-valued neural network on the data set, and suggests that a complex-valued network is superior at learning than a similar real-valued network.

Experimental result
First, we discuss the neural networks trained independently for enthalpy, free energy, and internal energy. The neural network training process with the genetic algorithm (GA) is illustrated in Fig.2, using free energy as the example. The entire training process is visualised in Fig.2a, and show that the optical neural network is capable of generating good successive generations by crossover (see Methods for more details). The evolution of the best and mean cost function values with the GA iteration are shown in Fig. 2b and Figure 3 show the results when we learned the labels concurrently (multitask learning). We note that our results for the multitask learning are similar to the case where we learned the labels separately. Furthermore, we emphasize that no additional resources were required to perform multitask learning, as the chip inherently implements parallel learning of labels, as long as we have enough photon detectors.

DISCUSSIONS AND CONCLUSIONS
The use of machine learning models, in particular deep neural networks, has been demonstrated to be effective in learning the hidden relationships between complex, high dimensional data sets [22,30]. These techniques provide us with a more efficient and economical way to predict molecular properties and hold enormous promise to accelerate material design and drug discovery. [31]. Despite their excellent performance, the complicated structures necessary for such networks, especially for performing more complex learning tasks will require increasingly more computing power and higher energy consumption.
Furthermore, the vast majority of such models rely on a real-valued neural network. Recent studies suggest that using complex-valued neural networks could significantly improve the performance of similar models [32] by offering richer representational capacity [33], faster convergence [34], strong generalization [35] and noise-robust memory mechanisms [36]. It has also been shown that complex valued neural networks have potential in domains where the data is naturally represented with complex numbers, or the problem is complex by design [37]. Thus, using such complex networks could also potentially be better in dealing with quantum-mechanical problems, as quantum mechanics is inherently a complex-valued theory. However, it is typically not efficient to implement a complex-valued network on a classical digital computer as complex numbers have to be represented by two real numbers on the digital computer [38,39], which increases the computationally expensive components of the neural network algorithms [40,41].
To overcome this, optical computing has been proposed as an alternate computing platform. The applications of optical computing to run neural networks provides various advantages over classical digital computers, ranging from low electrical power usage, being more energy efficient and robust to noise, and its inherent parallelism allowing us to break down the computation into small steps that are performed in parallel. Most interestingly, it is capable of truly complex-valued arithmetic, allowing us to implement complex-valued neural networks with no additional cost. Such complex-valued networks have been shown to also hold advantages over similar real-valued networks. However, most studies on applying optical computing to neural networks have focused on single-task classification learning.
This is a different class of machine learning tasks as compared to what is usually required for applications in chemistry, which is multi-task regression learning, and it was not known previously if such promising prior results in single-task classification would carry over.
Our work is the first known application of optical neural networks to chemistry. This is the first known application of an optical neural network on chip to optimizing a continuous valued regression task. It is also the first work to show that such a setup is capable of learning multiple properties at once (multi-task learning) at no additional cost. This also suggests that they would fare well in similar learning paradigms in machine learning, such as transfer learning and multi-output regression. It also demonstrates that complex-valued optical neural networks can achieve performance that is similar to classical real-valued neural networks implemented on digital computers, for the specific task of learning chemical properties of molecules. Although at a small scale right now, there are no physical reasons preventing such devices from being scaled up. It is also reasonable to assume that similar models will be effective in regressions tasks for other types of data sets.
By utilizing such optical chips in a manner where we offload the majority of the computational tasks of a neural network off a classical digital computer to them, we also show the potential of creating future hybrid computing systems, blending the advantages of optical and neuromorphic computing with those of classical digital computers. It is also possible to extend our ONC into a fully fledged multi-layer neural network with multiple hidden layers by cascading the optical circuits. Such cascaded photonic neural networks have also been demonstrated to be effective in performing various complicated machine learning tasks [42].
Much work has been done on figuring out how to efficiently and effectively encode the physical properties of a molecule in a form that can be fed as an input to a machine learning model. One method is to encode it in a form called the Coulomb matrix, and such methods have been routinely applied to the QM7 [43,44] and QM9 [25,45] data sets. While the Coulomb matrix is a low-level molecular descriptor, it is extremely simple to calculate and generate, does not require any domain knowledge, and is already able to obtain promising results [46][47][48].
The elements of the Coulomb matrix are defined by Equations 3 and 4: where Z i is the atomic number of atom i, and R i is its position in atomic units. Note that the Coulomb matrix is symmetric and is invariant to translation and rotation. The labeling of the atom indexes is also not unique, thus for any given molecule, many different

Scaling of data
It is known that many machine learning and data-driven analytical methods perform better when the numerical features and labels are scaled to a standard range [50,51]. In this work, we are dealing with eigenvalue features that numerically range from a few hundred to single digit numbers, and property labels that are of the order 10 5 . While the inputs features do not pose too much of a problem due to the way we construct our optical chip, the output labels do need to be scaled to a smaller range. We also want to avoid our labels having negative values, due to the method we are detecting outputs on our optical chip.
Thus, we used a simple Min-Max normalization scaling technique on each label separately, so that all the labels are scaled into the range [0, 1]. This was done with this map: where for each label, y min (y max ) is the minimum (maximum) value for that label in the entire data set.

Model validation
For our model we divided up the data set randomly into a train group consisting of 4 5 ths of the data, and a test group consisting of the remaining 1 5 th of the data. While a more systematic manner of distributing the data could be used (for example, stratified 5-fold cross validation with identical cross validation folds, used in [44]), from our simulations over many random divisions of data, this did not play a major part in affecting the accuracy of our model. We used a simple mean square error objective function for our model, and relied on easily available commercial optimization software to train the parameters in our model [52].

Experimental set up
The chip is fabricated at Advanced Micro Foundry, Singapore. The chip is wire-bonded to a printed circuit board, providing independent control of each phase shifter by an electronic current driver with 1-kHz frequency and 12-bit resolution (Qontrol Devices, Inc.). Laser pulses are generated by an Ultrafast Optical Clock device (PriTel Inc.) with a repetition rate of 500 MHz, central wavelength of 1550.12 nm and bandwidth of 2 nm. The input single pump light is coupled into the chip by a one-dimensional subwavelength grating coupler, and detected off chip by 8-mode grating coupler arrays. A polarization controller is utilized to maximize the coupling efficiency of the fibre to chip. A Peltier controlled by Thorlabs TED200C is used to stabilize the temperature of the chip and reduce the heat fluctuations caused by the ambient temperature and the heat crosstalk within the chip.

Genetic algorithm
While our previous simulations on a digital computer utilized variations of gradient descent methods, for the experiment we utilized a genetic algorithm to optimize the parameters for the ONC, to cut down on the time needed to conduct the experiment.
A genetic algorithm is a global optimization algorithm that can be summarized as so: 1. The genetic algorithm first starts out by creating a random initial population.
2. The algorithm then iteratively creates a sequence of new populations, known as generations. It does so with the following steps: (a) Calculates the scores of the current population with the cost function. These scores are know as the fitness scores.
(b) Chooses a certain number of the current population with the best fitness scores.
These are known as the elite and are passed on to the next generation.
(c) Produces children from the current generation (parents), with crossovers (combining the entries of a pair of parents) and mutations (randomly making changes to parents). These children are then also passed on to the next generation.
3. The current generation is replaced with the next generation.
4. This is repeated until the stopping criteria is met. In our case, the stopping criteria is a set number of iterations/generations. For our genetic algorithm, we conducted it over 50 training iterations (otherwise known as generations), with each generation having a population of 50. Between each generation, we selected the top 3 to survive into the next generation (the elite), then generated 47 additional samples, 39 by crossover, and 8 by mutation. Also due to time constraints, instead of utilizing the whole data set, we only utilized 100 randomly chosen data points in the data set to serve as our training data, and another 30 randomly chosen data points to serve as our test data. This implies that our results will be less accurate than what we expect. However, we believe that even with this small data set, our results demonstrate that our ONC possesses a strong ability to implement a complex valued neural network model.