{'fraction_non_alphanumeric': 0.023735395534130094, 'fraction_numerical': 0.011616618124602457, 'mean_word_length': 5.168077637827793, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning meets nanophotonics: A generalized accurate predictor for near fields and far fields of arbitrary 3D nanostructures

Deep artificial neural networks are powerful tools with many possible applications in nanophotonics. Here, we demonstrate how a deep neural network can be used as a fast, general purpose predictor of the full near-field and far-field response of plasmonic and dielectric nanostructures. A trained neural network is shown to infer the internal fields of arbitrary three-dimensional nanostructures many orders of magnitude faster compared to conventional numerical simulations. Secondary physical quantities are derived from the deep learning predictions and faithfully reproduce a wide variety of physical effects without requiring specific training. We discuss the strengths and limitations of the neural network approach using a number of model studies of single particles and their near-field interactions. Our approach paves the way for fast, yet universal methods for design and analysis of nanophotonic systems.

Dielectric and metallic nanostructures of sub-wavelength size can be designed such that their interaction with light differs significantly from bulk materials. Nanophotonics aims to exploit optical resonances and strong localized fields that can be designed by nanoparticle geometry and material choices. [1][2][3][4] The unique nanoscale properties can be used in applications such as plasmonic nanoantennas for near-field energy concentration and meta-surfaces for directing and controlling light. More complex optical behavior can be designed such as polarization conversion, 5 chirality, 6 localized heat generation 7 or nonlinear optical effects [8][9][10] .
The interaction of light with many types of nanostructures can be modelled accurately by solving the classical Maxwell's equations, 11 for which many commercial and open source methods are available. However, numerical nanophotonic simulations are often time-consuming and can take hours or even days for complex systems. 12 Many subsequent simulations are needed for iterative optimization methods such as the rational design of nanosystems by topology optimization and inverse design. [13][14][15][16] Fast evaluation of optical properties is highly desirable also for example in real-time sensors based on 3D molecular nanorulers, 17,18 or in the active control of smart and reconfigurable nano-materials 19,20 .
Methods of artificial intelligence, and in particular deep learning, 21,22 are powerful tools with potentially groundbreaking relevance for nano-optics. First captivating applications have been reported in experimental photonics and nanooptics. Examples are the possibility of phase recovery in conventional microscopy, 23 stabilization of lasers 24 and a large variety of applications in data analysis and interpretation [25][26][27][28][29][30][31][32][33] . For forward modelling, early work has shown that artificial neural networks (ANNs) can be used as approximate predictors for light-matter interaction phenomena or optical scattering at nanostructures. Examples are strong-field ionization of potassium atoms, 34 SHG of a specific fluorescent molecule, 35 the optical transmittance of "H"-shaped particles 36 or the scattering cross sections of multilayer spheres 37 . All of those reported ANN techniques apply to single, very specific problems and for a particular nanostructure geometry.
Furthermore, deep artificial neural networks are highly promising to approach notoriously difficult inverse problems in nano-optics, like the design of meta-surfaces or the tailoring of optical properties of individual nano-structures. Even though no generalized and fast inverse design method has as yet been reported, rapid progress has been made in this direction over the last two years. 13,[36][37][38][39][40][41][42][43] The overall trend in these studies so far is, that for every specific inverse problem using a particular geometric model, a neural network needs to be designed in a time demanding and computationally very expensive process, involving hyper-parameter optimization, training data generation, training and extensive testing.
Here, we present a general approach to nano-optical modeling which is distinct from all previous works by its capability for fast and accurate modeling of generalized nano-optical effects in a variety of nanostructures. Requiring only a single training procedure, our concept fully generalizes ANNs for nano-photonic simulations and allows to tremendously accelerate predictions for countless problems in nano-optics. We demonstrate that the generalized network captures a range of complex nano-optical near-and far-field effects in nanostructures, such as higher order antenna resonances, electric and magnetic dipole modes, non-radiating anapole states or Kerker-type directional scattering, without the need of any specific training for these effects. Our approach is based on a three-dimensional, fully convolutional neural network (CNN), trained on predicting a coupled dipole representation of the fields inside nanostructures of arbitrary shape. These predictions can be used subsequently to reconstruct many secondary physical quantities with uncertainties as low as few percent.
An overview of the model description is shown in Figure 1. The aim of our work is to develop an ANN capable of predicting the time-harmonic electric polarization density inside nanostructures of arbitrary shape. For our demonstration we use a geometric model consisting of a rectangular grid of positions, as illustrated by the example of Figure 1a, left panel. Nanostructures are mapped onto the 3D grid using a standard volume discretization approach. The resulting geometry can be directly evaluated with numerical simulations using the coupled dipole approximation (CDA). Upon normal incidence plane wave illumination (k along −Z) and linear polarization  The three-dimensional convolutional network follows an encoder-decoder architecture and is organized in a sequence of residual blocks. The principal layout of these blocks, the number of kernels as well as the layer dimensions of the silicon predictor are shown in the sketch of the network. The six output channels of the network contain the real and imaginary parts of the x, y and z components of the complex electric field inside the nanostructure. For more details see Section III of the Supporting Information. Following the calculation of the self-consistent electric polarization inside the structure, the latter can be interpreted as dipole moments p(r i ) of the single mesh cells at r i . (b) Various physical quantities in the near-field and far-field, as illustrated can be derived from p(r i ). This includes the electric or magnetic near-field (c), the Poynting vector (d) or far-field scattering patterns (e) among many others (f). The model includes a dielectric substrate (n subst = 1.45). The structure is illuminated by a linearly polarized plane wave from the top with λ 0 = 700 nm. along X, we numerically calculate the field at every mesh cell inside the nanostructure via the Green Dyadic Method (GDM), 44 for which we use a home-built python implementation "pyGDM". 45 A short summary of the GDM formalism can be found in Section I of the Supporting Information. The model includes a dielectric substrate (n subst = 1.45). While in principle any simulation method can be used to generate the network training data, we use the GDM for its simplicity and its good convergence in the case of small nanostructures as well as for the fact that the results are by design available on the required cubic discretization grid.
The conceptual beauty of the CDA method lies in the initial assumption that every meshpoint can be approximated by a dipolar polarizability, which allows to treat every cell inside the nanostructure as an oscillating dipole moment. In consequence, various physical quantities in the near-field as well as in the far-field region can be derived from the inter-nal fields of a CDA simulation. Figure 1b illustrates a few of the possible observables that can be calculated from the dipole discretization. Using according Green's dyads, the optical electric and magnetic fields (and in consequence the Poynting vector) can be obtained at any location outside the nanostructure (see Figure 1(c-d)). 45 Extinction, scattering or absorption cross sections can be calculated almost effortless, 1 as well as the polarization state and spatial patterns of the scattering (Figure 1e), 5,46 the dissipated heat or local temperature gradients, 7,47 nonlinear effects like second or third harmonic generation and multi-photon luminescence 10,48,49 or the multipole decomposition of the optical response. 2 In consequence, an ANN capable to accurately predict the internal electric fields of a photonic nanostructure represents a generalized, phenomenological model of light-matter interaction, with tremendous potentials for rapid nano-optical simulations.
A 3D grid of positions is used as input layer to the network which is illustrated in Figure 1a, middle panel. The network used in this study is a three-dimensional symmetric, fully convolutional network with "U-Net"-type shortcut connections between corresponding convolutional and up-sampling units. This type of ANN is known to be particularly strong at the reconstruction of spatial information. 50 Shortcuts were found to be essential for our purpose to obtain good approximations of the internal fields. In addition to the U-Net design, we organize the network in residual blocks 51,52 which allows us to maintain good learning performance on a very deep architecture with as much as 91 layers including 33 three-dimensional convolutional operations. All network details and hyperparameters are given in Section III of the Supporting Information.
The output of the network is composed of 6 layers of the same size as the input grid, which correspond to the real and imaginary parts of E x , E y and E z . We use complex amplitudes for the fields, so phase information and retardation effects are included in the neural network predictions. An illustration of real part field-vectors is shown in Figure 1a, right panel.
For the network training, we simulate the internal fields of 30000 random nanostructures. We test the network on two datasets, the first contains planar gold nanostructures (15 nm height) of random polygonal shapes. The second set is composed of silicon pillar structures (200 nm height), consisting of one or more arranged cuboidal blocks. Details about the geometric models are given in Section II of the Supporting Information. For convenience we use fixed height structures throughout this demonstration, but we want to emphasize that this is not an inherent limitation of the approach. On the contrary, variable height structures can be modeled without any further modifications and without significant loss of accuracy, as shown in the supporting information figure S9. We fix the illumination conditions to normal incidence plane wave excitation at a wavelength of λ 0 = 700 nm with linear polarization along X. In consequence, the neural network is limited to predictions under the conditions chosen for the training data generation. Hence, in our approach a separate ANN needs to be trained if the structure model or material is modified and also for every illumination configuration, e.g. for every angle of an oblique incidence illumination. We note that under normal incidence, any linear polarization angle can be achieved by a rotation of the nanostructure. Fully arbitrary polarization states of the illumination can furthermore be obtained via superposition of perpendicular linear polarizations, as demonstrated in the supporting information Fig. S7 for left circular polarization.
In both cases of plasmonic and dielectric structures we use 28000 samples for training and the remaining 2000 structures for validation and benchmarking. On an NVIDIA P6000 GPU, the training with the gold (silicon) dataset takes around 2 (10) minutes per epoch. We stop training after 100 epochs, longer training leads to no further improvement in validation accuracy (see Figure S10 in the Supporting Information). On the same GPU, the trained network delivers its prediction in around 3 ms for a planar gold structure and 6 ms for a silicon structure. On a third generation Intel i7 quad-core CPU (i7-3770) the predictions for the gold, respectively silicon structures take around 53 ms and 250 ms. Depending on the chosen hardware platform, this is 3 to 5 orders of magnitude faster than the conventional simulations, which take seconds to minutes on the Intel i7 CPU. The simulation time for the full datasets, running in parallel on two workstations, was approximately 10 days. We note that since the simulations are entirely independent, the data generation is a socalled "embarrassingly parallel" task that can be parallelized to any extent. The GDM simulations can potentially be accelerated by GPUs, but according to literature, less than one order of magnitude speed-up is expected for GPU-based LU decomposition. 53 While a more formal benchmarking of the ANN is discussed further below, we start our results by demonstrating the ability of the network to capture some of the well-known physical effects in metallic and dielectric nanostructures. The ANN's capability to generalize to these arbitrary situations is tested by constructing a number of specific cases at which optical effects occur and compare the predictions of the neural network to numerical simulations. Figure 2a shows the ANN predicted real and imaginary part of the amplitude of the internal field components E x and E y upon X polarized plane wave illumination of the planar gold nano-polygon. 54,55 Clearly, the neural network correctly predicts the form and distribution of fields as given by the CDA simulation. In addition to arbitrary objects, the ANN shows a good systematic scaling of antenna behaviour, as illustrated in Figure 2b where we plot the magnitude of the electric field intensity for gold nanorod antennas of 75 nm width and lengths ranging from a point dipole, to half-wave and multi-order antenna resonances. In figure 2c, the electric field intensity on top of the center of the nanorod is shown as function of the rod length. The neural network is able to make an accurate prediction of the mode structure and effective scaling of plasmonic nanoantennas within the domain under study in a generalized, phenomenological model of light-matter interaction. 56 We note that the peak field enhancement at resonance is consistently underestimated by the ANN. We believe this is due to the random generation process of the training data, which therefore includes only very few structures that exactly hit a resonance condition.
In Figure 2d we finally show the scattering cross section (SCS) of a gold rod antenna as function of its length for perpendicular incident polarizations TE (incident field polarized perpendicular to the rod long axis) and TM (field along the rod axis). Again the neural network reliably predicts the resonance positions. Interestingly, apart from the dipole antenna resonance where the underestimated nearfield is reflected also in the SCS, prediction and scattering quantitatively matches better for longer rod antennas. This is a result of the weak coupling to the far-field of higher order modes, hence the underestimation of the nearfield strength has no great impact on the far-field scattering in these cases.
Next to plasmonic nanoparticles and antennas, recently dielectric nanostructures have received tremendous interest for use in meta-surfaces and low-absorption antennas. 57,58 Here, we evaluate the performance of the ANN in inferring the response of silicon nanorods with fixed height of 200 nm and varying lengths and widths. Figure 3a and b shows the internal electric field energy (a) and scattering cross section (b) for a silicon nanorod with a square cross-section of 200 × 200 nm 2 , plotted against rod length. The incident plane wave is linearly polarized along the width of the rod (transverse electric, TE). In this dataset we observe the typical signature of an "anapole" mode for a length indicated by a dashed vertical line, at which a superposition of toroidal and electric dipole mode is excited. 59 The anapole mode is characterized by a minimum in the scattering cross section accompanied by a maximum of the electric field energy inside the structure. 60 Both signatures are reproduced by the ANN (solid lines) in excellent agreement with the numerical simulations (dashed lines). The corresponding field profile predicted by the ANN is shown in Figure 3c and reproduces the characteristic internal field distribution of the anapole.
The response of the silicon nanorods for polarizations along the width (TE) and along the length (TM) of the nanorod is further investigated in a parameter study where we independently tuned the length and width of the structure. Figure 3d and 3e show the electric and magnetic dipole contributions to the nanorod extinctions. The two separate contributions were extracted from the total solution through a Taylor-like expansion of the electric polarization. 2 The neural network predictions (left subplots) agree very well with the numerical simulations (right subplots) in both the positions and amplitudes of the different magneto-electric modes in the structure. Also shown are 3D quiver plots of the internal field distributions at selected parameters, which show that the underlying field distributions inferred by the network match very well the simulated ones.
Precise tuning of the nanostructure geometry allows to obtain specific conditions of uni-directional "Kerker-type" light scattering due to the simultaneous excitation of electric dipole (ED) and magnetic dipole (MD) modes of similar magnitude. 46,61,62 For the fixed structure height of 200 nm and illumination wavelength used in our predictor network (λ 0 = 700 nm), we find that ED and MD modes are simultaneously excited for a square cuboid with side length L = 100 nm. Figure 3f shows a 3D quiver plot (i) where the MD can be observed in the upper part of the 3D electric field distribution (vortex formed by the field vectors), whereas the ED is situated at the bottom of the silicon block (field vectors parallel to the substrate plane). The superposition of both contributions leads to a strongly directional scattering pattern (see center plot). Panel (ii) shows the effect when the block size is increased to L = 160 nm. Here the MD becomes the predominantly excited mode, leading to a bi-directional scattering, as expected for a dipolar source of radiation. Both internal 3D field plots as well as the far-field scattering patterns show very good agreement between ANN and simulation. Generally the ratio of forward to backward total scattering is of interest in dielectric structures as it shows large variations with particle geometry. Figure 3g shows the forward / backward ratio for a silicon nanorod of 100 nm width and with increasing length for (i) TE and (ii) TM polarized illumination. The scattering from the nanorods was integrated over the respective full half hemisphere to obtain the total scattering intensity. Overall, the ANN predicts accurately the directional character of the scattering in all cases. The position of the quadrupole magnetic mode is indicated by the vertical line, even the quite complex electric field distributions at this position are correctly predicted by the network. For longer rods ( 400 nm), the accuracy of the ANN deteriorates slightly, the network tends to overestimate scattering and extinction cross sections (see Figure 3(d-e) and 3g). We assume that this is due to the maximum length and width of 300 nm used for the silicon blocks in the training data. Considering this constraint on the training process, the network manages to generalize impressively well to larger individual nanostructures. In summary, our generalized predictor neural network manages to accurately predict various nano-optical phenomena in the near-field as well as in the far-field, without having been specifically trained on those effects.
Next to the response of individual structures, it is critically important that the ANN can capture the mutual interactions between isolated nanostructures. Figure 4 explores the nearfield coupling for a silicon dimer structure. Results for gold nanodimers are shown in the Supporting Information Figure  S6. The scaling of near-field coupling between the structures was studied by varying the inter-particle gap as illustrated in Figure 4a. The effect of near-field coupling is clearly visible in the quiver plots in Figure 4b. Visually the network predictions can barely be distinguished from the numerical simulations. Quantitative agreement is also obtained for the scaling of the near-field in the center of the gap (indicated by a red cross in Figure 4a). The field enhancement derived via ANN is compared in Figure 4c (solid lines) to numerical simulations (dashed lines), yielding indeed an excellent agreement, apart from a slight overestimation of the field intensity.
In order to assess the impact of optical coupling between the two silicon blocks, we artificially turn off near-field interactions in the simulations (orange lines). This is done by calculating the optical response of both blocks separately, and using the internal dipole moments of those isolated simulations (respectively isolated ANN predictions) to derive the field intensity in the gap. We find that near-field interactions become non-negligible only for small distances 100 nm, which is correctly described also by the predictor network. This observation is in agreement with experimental results of cathodoluminescence imaging of hybridized mode field profiles in silicon dimers. 63 In figure 4d, we furthermore study the optical chirality C of the near-field in the center of the gap, 10 nm above the nanostructure's top surface. C is a measure of the selectivity at which opposite handed chiral molecules interact with the electromagnetic field. 64,65 We plot C normalized to the chirality of a left circular polarized plane wave C LCP . As expected, in the symmetric configuration the field has no chirality. It is however possible to break the symmetry of the dimer by shifting one of the blocks vertically by a distance ∆Y , as illustrated in Figure 4e. The internal fields are shown in top-view quiver plots in Figure 4f and provide evidence that the neural network correctly predicts near-field coupling effects in the asymmetric dimer. The electric field vectors inside the silicon strongly vary dependent on the relative positions of the two blocks. The optical chirality as function of the vertical shift is shown in Figure 4g for three different gap sizes. Chirality is induced through asymmetry in the dimer geometry, which is again correctly reflected in the ANN predictions. Interestingly, the electromagnetic field above the gap features almost no chirality if the near-field interactions are deactivated in the simulations. Hence, the electromagnetic chirality is driven here by optical coupling between the two silicon blocks. This is confirmed by the decreasing magnitude of C for increasing gap widths (from top to bottom in Figure 4g), which is also in agreement with recent experimental results. 66 In contrast, for plasmonic dimers, it is found that near-field coupling suppresses the chirality (Supporting Information Figure S6). 65 The above results reveal the capacity of the neural network in inferring many of the important physical effects in the interaction of light with nanostructures. While the simple model examples provide insight, a more formal assessment of the ANN performance is needed. Figure 5 shows results of the statistical data analysis of the entire validation set for the plasmonic (a-c) and silicon (d-f) models, each consisting of 2000 random structures which were not used for training. In Figure 5a and d the statistics for the normalized cross correlation between simulated and predicted field intensity distributions are shown for the far-field, the near-field in the vicinity, and the internal field inside the structure. Considering that a cross correlation of 0.8 indicates already a good qualitative agreement, the network predictions are mostly excellent, however for a non-negligible number of "outliers", the ANN results are significantly worse.
The same trends can be found in a quantitative analysis of the far-field cross-sections, the peak internal field intensity (Figure 5b,e) and the polarization state (Figure 5c,f). So while the network predictions are mostly very good, being a data-driven approach, an inevitable error of a few percent is inherent to the method. Also, the ANN bears some risk of getting a result with significant error. The risk of a failed prediction is of the order of around 5%, as expressed by the amount of outliers in figure 5. Examples of outliers in the validation data are shown in Figure S4 and S5 of the Supporting Information. In our simple examples we have also seen some reduction in ANN performance around resonant poles in the polarizability and for larger sized structures in Figure 2. The statistical evaluation indicates that the plasmonic predictor network performs generally worse than the ANN trained on silicon cuboidal structures. The induced plasmonic currents cause strong depolarization effects on short lengthscales, which are strongly dependent on the structure geometry. We therefore speculate that this complex and featurerich plasmonic optical response is more difficult to predict, compared to dielectric nanostructures. This lack of perfection is the general property of neural networks and new methods gauging the reliability of solutions have been proposed using for example multiple, independently trained ANNs. 67 Additionally, while ANNs are good at interpolating complex functions from few data, their performance is reduced when extrapolating outside of the known parameter space. 68 Despite these limitations, our results clearly show that the ANN's capabilities go far beyond that of a simple fit function, as it is able to apply trained behavior to new highly nontrivial configurations (see also Supporting Information Figure S8 on the prediction of curved structures). The ANN yields a generalized and powerful capability for generating field distributions from structural arrangements.
Improved performance of the ANN in specific ranges may be achieved by further extending the training dataset. In our current demonstration we chose to limit ourselves to planar configurations and single materials. However the method could be further generalized to arbitrary hybrid-material structures using the three-dimensional distribution of the dielectric constant as input. Periodic structures can be treated using training data describing one unit-cell of the periodic structure. 69,70 In addition, the approach can be extended to spectrally resolved predictions using multiple output layers. Spectral training might be accelerated by transfer learning. Generalized predictor networks can be potentially used together with evolutionary optimization schemes for fast, universal nano-photonic inverse design, overcoming the need of designing a specific inverse network for every target problem and nanostructure model. 71 In conclusion, we presented an approach combining the coupled dipole approximation with deep artificial neural networks, which is capable to accelerate universal electrodynamical simulations of arbitrary 3D nanostructures by many orders of magnitude. Our generalized predictor network needs to be trained only once using a volume discretization of the three-dimensional electric polarization inside nanostructures of arbitrary shape. Being a data-driven approach, the method comes necessarily with a loss of accuracy. Despite this inherent shortcoming, we could demonstrate that the network predictions imply only a small error in the order of around five percent. On the other hand, the prediction of the internal fields allows to derive manifold quantities and effects in the near-field as well as in the far-field region. We have shown, that both regions are covered by the predictor with very good accuracy. We demonstrated furthermore that the network developed a generalized intuition about Maxwell's equations, being capable to reproduce complex nano-optical effects occurring in plasmonic and dielectric nanostructures. The ANN faithfully reproduces phenomena like higher order localized plasmon modes, magnetic and electric dipole modes, non-radiating anapole states, Kerker-type directional scattering or optical chirality. We demonstrated that the network also developed an understanding of near-field interactions between separated nanostructures. We foresee that ultrarapid generalized predictor networks bare tremendous potential for applications in nanophotonics.

ACKNOWLEDGMENTS
We gratefully thank Arnaud Arbouet and Christian Girard for fruitful discussions. We thank the NVIDIA Corporation for the donation of a Quadro P6000 GPU used for this research. This work was supported by the German Research Foundation (DFG) through a research fellowship (WI 5261/1-1). OM acknowledges support through EP-SRC grant EP/M009122/1. All data supporting this study are openly available from the University of Southampton repository (DOI: 10.5258/SOTON/D1088).

SUPPORTING INFORMATIONS
A description of methods and further data with Figures S1-S10. A video of the oscillating fields inside the silicon nanodimer for varying vertical distances comparing ANN and simulation. This material is available free of charge via the Internet at http://pubs.acs.org.