{'fraction_non_alphanumeric': 0.027597192922030396, 'fraction_numerical': 0.009945322254273045, 'mean_word_length': 4.8435220125786165, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fast and accurate simulation of particle detectors using generative adversarial networks

Deep generative models parametrised by neural networks have recently started to provide accurate results in modelling natural images. In particular, generative adversarial networks provide an unsupervised solution to this problem. In this work we apply this kind of technique to the simulation of particle-detector response to hadronic jets. We show that deep neural networks can achieve high-fidelity in this task, while attaining a speed increase of several orders of magnitude with respect to traditional algorithms.


Introduction
The extraction of results from high energy physics data crucially relies on accurate models of particle detectors, and on complex algorithms that infer the properties of incoming particles from signals recorded in electronic sensors. Numerical models, based on Monte Carlo methods, are used to simulate the interaction between elementary particles and matter. In particular, the Geant4 toolkit [1] features stateof-the art models and is employed to simulate particle detectors at the CERN LHC. Reconstruction algorithms routinely used at collider experiments (see e.g. [2] and [3]) are based on estimators of particle trajectories and energy deposits. This information is subsequently aggregated in order to reconstruct energy, type and direction of final state particles produced by the collision of the primary beams.
The CERN LHC complex will undergo a series of upgrades [4] over the next ten years that will allow collecting a dataset roughly 30 times larger than the one currently available. The number of simultaneous interactions per bunch crossing in such a future dataset will increase by a factor of about 4, compared to the present levels. It is estimated [5] that a 4-fold improvement in the algorithm performance, at constant funding, will be needed for the LHC computing infrastructure to keep up with the larger volume and complexity of the data. This provides a very strong motivation for the development of fast algorithms for particle detector simulation and data reconstruction.
In this work, we develop a generative model parametrised by a deep neural network, that is capable of predicting the combined effect of particle detector simulation models and reconstuction algorithms to hadronic jets. The results are based on samples of simulated hadronic jets produced in proton-proton collisions at √ s = 7 TeV that was published by the CMS collaboration on the CERN open data portal. The dataset [6,7,8,9,10,11,12,13,14,15,16,17,18] is part of the "level 3" category in the the High Energy Physics (HEP) data preservation classification [19] and it contains the result of the Geant4 simulation of the CMS detector and of the subsequent data reconstruction algorithms used by the CMS collaboration.
Generative adversarial neural networks (GANs) [20] are pairs of neural networks, a generative model and a discriminative one, that are trained concurrently as players of a minimax game. The task of the generative network is to produce, starting from a latent space with a fixed distribution, samples that the discriminative model tries to separate from samples drawn from a target dataset. It can be shown [20] that with this kind of setup the generator is able to learn the distribution of the target dataset.
Since they were first proposed, GANs have been applied to an increasingly large number of problems in machine learning, mostly dealing with natural-image data, but not only. Applications of adversarial networks were also proposed in the context of HEP, mainly with two purposes: training of robust discriminators that are insensitive to systematic effects, or uncorrelated from observables used for signal extraction [21,22,23], and for event generation and detector simulation [24,25,26]. Similar applications were proposed in the context of cosmic ray experiments [27].
We use GANs to train a generative model that generates the reconstruction-level detector response to a hadronic jet, conditionally on its particle level content. We represent hadronic jets as "gray-scale" images of fixed size centred on the jet axis, where the pixel intensity reflects the fraction of jet energy deposited in the corresponding geometrical cell. The architecture of the networks and the problem formulation are based on the image-to-image translation work described in [28]. We introduce a few differences to tailor the approach to the generation of jet images: we explicitly model the set of non-empty pixels in the generated images, which are much sparser than in natural images; we enforce a good modelling of the total pixel intensity; and we condition the generator on a number of auxiliary features.
Related work has recently been presented in [25] and [27]. The authors of [25] proposed to use a deep convolutional neural network to simulate calorimeter showers, thus aiming at modelling the particle interaction with the detector medium. The solution that we explore here allows the largest reduction in computation time, by predicting directly the objects used at analysis level, and thus replacing both detector simulation and reconstruction algorithms. This philosophy is similar to that of the parametrised detectors simulations that are often used in HEP for phenomenological studies, and that are very limited in accuracy. We show that using a deep neural network model allows attaining accuracies that are comparable to that of the full simulation and reconstruction chain. The approach of [27], that studied the application of GANs to the generation of air-showers, is more similar to ours, as it aims at predicting the patterns reconstructed by the detectors, conditionally on the energy and type of the primary particles.

Inputs and problem formulation
For this study we use simulated samples of hadronic jets produced by the CMS collaboration and published on the CERN open data portal. In particular we take hadronic jets produced in proton-proton collisions at √ s = 7 TeV. These events feature state-of-the-art characteristics in terms of simulation and reconstruction algorithms in HEP. The events were generated with the PYTHIA6 event generator [29], the CMS detector response was simulated using Geant4 [1]. Concurrent proton-proton interactions ("pile-up") were simulated, roughly reproducing the LHC running conditions of 2011. The samples contain the results of the full CMS reconstruction chain [3].
In the input dataset, hadronic jets were clustered with the anti-kt algorithm [30], using the FastJet library [31] and a distance parameter of 0.5. We used two sets of hadronic jets: those clustered from the list of stable particles produced by PYTHIA6, and those clustered from the list of reconstructed particle candidates. In the following we term "particle-level jets" the former, and "reconstructed jets" the latter.
Following standard practices in collider experiments, we employ a cylindrical system of coordinates. The origin of the coordinate system is set to the centre of the CMS detector, the z axis is chosen to be parallel to the beam line, and the x axis is chosen to point towards the centre of the LHC ring. We indicate as φ and θ the azimuthal and polar angles, respectively, and we define the pseudorapidity η as log(tan(θ/2)).
In each event, we select particle-level jets with a transverse component of the momentum above 20 GeV and with an absolute value of the pseudorapidity below 2.5. A search is then performed in the reconstructed jet collection to find reconstructed jets satisfying ∆η 2 + ∆φ 2 < 0.3, where ∆η and ∆φ are, respectively, the difference between pseudo-rapidity and azimuth of the particle-level and reconstructed jets. Pairs of reconstructed and particle-level jets satisfying these conditions are considered in this study.
Jet images are constructed by opening a window of size ∆η × ∆φ = 0.3 × 0.3 around the particle-level jet axis. The window is split into 32 × 32 identical square pixels and the intensity associated to each pixel is proportional to the total transverse momentum of the jet components contained in it, divided by the transverse momentum of the particle-level jet. Roughly 80-90% of the jet energy is contained in the jet window that we considered, and the jet components not contained in the window are attached to the borders of the image.
Each pair of jet images is further associated to four auxiliary features: the transverse momentum (p gen T ), pseudo- rapidity (η gen ) and azimuthal angle (φ gen ) of the particlelevel jet, and the number of pile-up interactions (n P U ) that were simulated in the event under consideration.
No further preprocessing or standardisation was performed on the input data.
The distribution of p gen T for the selected hadronic jets is shown by the blue histogram in figure 1. The shape of the distributions is multi-modal because the original dataset was split in several bins in the scale of the parton-parton interaction. A set of weighting factors, as a function of p gen T was applied to obtain a monotonically falling distribution. The distribution obtained after applying such weighting factors is shown by the red histogram.

Notation and learning setup
We adopt the following notation: we denote by x and y the jet image at particle and reconstruction level, respectively, we indicate with c the set of auxiliary features, and we use z to denote a latent space of uniformly distributed noise. With this notation, our problem can be formulated as follows. -Given: where p c , p x and p y are the input-data distributions, and U indicates the uniform distribution.
-We want to construct a function G(.), such that y = G(z, x, c) ∼ p y (y |x, c).
The function G(.) is a generative model that approximates the combined response of particle detector simulation and reconstruction algorithms to hadronic jets. Following the GAN paradigm, we look for a solution to this problem by introducing a discriminative model D(.) and setting-up a minimax game between the two models, with value function V (G, D) defined as: While the problem could in principle be solved using the GAN setup alone, we inject additional information in order to stabilise and speed-up the convergence. In particular, we take into account two facts: 1. G(.) and y should match on average; 2. y is very sparse.
The authors of [28] show that the first requirement can be efficiently satisfied by adding an L 2 -or L 1 -norm term to the loss function. We adopt this approach, using in particular an L 2 -norm term.
To explicitly take into account the second requirement, we modify the structure of the generator, by increasing the depth of its output: one channel is used to model the pixel intensity, while a second channel, to which we refer as a "soft-mask", models the probability of a pixel to be non-zero. We denote the two channels as G 0 and G 1 , and we modify the generative model loss function by adding a term of this form: where λ is the associated hyperparameter.
To generate images, we sample the soft-mask probabilities to create a "hard-mask" binary stochastic layer G 1 (z) = 1 z<G1 , where 1 is the so-called indicator function. The GAN value function in eq. 1 becomes V (G 2 , D) and the differentiability is preserved by replacing G 1 with G 1 during back-propagation [32].
Finally, we enforce a good modelling of the total image intensity, which is proportional to the reconstructed jet energy, with two additions: we add an extra term to the generative model loss function, proportional to the mean squared error of the total image intensities; we introduce a second discriminative model D T that receives as input the total reconstructed jet image intensities, and the auxiliary features c, and we setup an additional minimax game with value function V T (G 2 , D T ).
Two additional hyperparameters, τ and µ, are introduced to tune the strength of the two contributions. The generative model and the discriminative model D are implemented as convolutional neural networks [33]. For the generator we adopt the so called "U-net" architecture [34], that consists of an encoding section followed by a decoding one, with additional skip connections linking encoding and decoding layers with the same spatial dimension. The input images are first fed into a batch normalisation layer [35], which allows running the network on non-standardised inputs. Afterwards, we use 5 encoding layers and 5 decoding ones. Each encoding layer consists of a convolutional unit, followed by a batch-normalisation one and by a leaky ReLU activation [36], with a slope of 0.2 in the negative domain. The encoding filters size is chosen to be of 3×3, with a stride of 2 in order to reduce the representation width. The number of filters is set to 16 for the first layer and it is doubled at each step. The decoding layers comprise a concatenation unit to implement the skip connections, followed by up-convolutional units, batch normalisation and leaky ReLU activation ones. Dropout units are also employed in the first two layers of the decoding section. At each step in the decoding section, the depth of the representation is halved, while its width is doubled. This is achieved by decreasing the number of convolutional filters, while appropriately choosing the stride and padding parameters. Auxiliary conditional features are injected in the architecture as follows: they are first passed through a batch normalisation unit and then into a 1×1 convolution unit whose output matches the depth of the last encoding layer; the 1×1 convolutional unit output is subsequently concatenated with that of the last encoding convolution. Noise can be injected into the architecture at the same level. However, we obtained better results by feeding noise only in the form of a stochastic sampling of the output soft-mask. Figure 2 shows a graphical summary of the generator architecture.
The discriminative model D uses 4 layers, comprising 3×3 convolutional filter units, batch normalisation units and leaky ReLU activation ones. Stride and padding are tuned in such a way that the largest field of view of the convolution layers is of 13×13 pixels. The model acts as a "patch-GAN" [28], i.e. it is only sensitive to the local structure of the jet images. The convolutional layers are followed by a fully-connected layer with a sigmoid activation function. The auxiliary variables are treated similarly to what is done in the generative model and are injected at the input of the fully connected layer.
The model D T takes as input the total intensities for y (or G 2 ) as well as c, and is parametrised as a feedforward fully connected neural network with 4 layers, using dense units, batch normalisation and leaky ReLU activations. The fully connected layers have widths of 64-64-32-16 and are followed by an output layer with a sigmoid activation function. Fig. 3 Joint distribution of the particle-level ("gen") and reconstruction level pixel intensities. The left plot shows the distribution of the input data, while the right plot is obtained using the generative model.
We did not perform a formal optimisation of the neural networks architecture, but we picked a particu-

Results
The models were trained on two million jet images extracted from the dataset described in section 2. The TensorFlow [37] framework, and the Keras [38] high level interface were used to implement and train the models. Computing resources from the Piz Daint Cray supercomputer located at the Swiss Centre for Supercomputing were used to obtain the results that we present here. Two independent Adam [39] optimisers were employed for parameter sets of the generative and discriminative models. NVIDIA Pascal P100 GPUs were used to accelerate the computations and the models were trained for 10-20 epochs, which were sufficient to achieve convergence. The training time for these models was around 1 hour per epoch. Inference ran at roughly 100Hz on Intel Xeon CPUs and at roughly 10kHz on NVIDIA Pascal P100 GPUs, which are to be compared to the typical time scale for event simulation and reconstruction, i.e. 10 −1 -10 −2 Hz. Figure 3 shows the correlation between the particles and detector-level pixel intensities for the input dataset and for the predicted images, obtained from a test sam- well measured jet components populate the diagonal; errors in the position reconstruction of the jet components lead to energy migration between close-by pixels and thus contribute to the sub-populations located close to the horizontal and vertical axes; non-reconstructed jet components manifest as empty reconstruction-level pixels that correspond to nonempty ones at particle level and therefore to the sub-population located along the horizontal axis pile-up effects lead to non-empty pixels in the reconstruction level image in correspondence to empty particle-level pixels and so contribute to the subpopulation close to the vertical axis.
As can be seen from the figure, our set-up (right panel) achieves a good modelling of the relative weight of the three sub-populations, which result from a non-trivial set of effects.
In figure 4 we show the distributions of the total pixel intensities obtained integrating over rings in ∆R centred on the particle-level jets axis. Blue histograms are obtained from the input dataset, while red ones show the results of the generative model. Figure 5 shows the evolution of distribution of the aggregated pixel intensities as a function of the particle-level jet transverse momentum. These results show that our set-up allows good modelling of hadronic jet structure over more than two orders of magnitude in jet transverse momentum. We further investigate the goodness of the learned model by evaluating its ability to reproduce high level jet features that are typically used in physics analyses. We concentrate, in particular on two sets of variables: 1. variables used in the context of quark/gluon discrimination; 2. jet substructure variables used in the context of merged jets discrimination.
From the first set, we choose the so-called major and minor axes, i.e. the eigenvalues of the η-φ covariance matrix of the jet image, and the p T D variable, i.e. the ratio between the square root of the second and first non-central moment of the pixel intensities [40]. From the second set, we choose the ratio between the 2-and 1-subjettines [41] and that between the 2-and 3subjettiness. The subjettiness variables were computed using the FastJet package [42,43], approximating each jet as a set of mass-less particles with energies and directions obtained from the pixel intensities and positions. Figure 6 shows the distribution of the quark/gluon and merged jets discrimination variables that we considered aggregated over the test dataset, while figure 7  Fig. 7 Evolution of the quark/gluon (first two rows) discrimination and merged jet tagging (last row) variables as a function of the particle level jet transverse momentum. Solid lines represent the median of the distribution, filled regions show the inter-quartile range, while dashed lines mark the 10% and 90% quantiles. Blue lines are obtained from the input data, while red ones are obtained using the generative model.
shows the evolution of the distributions as a function of the transverse momentum of the jet at particle level. The level at which these variables are predicted by our set-up is good, even though some mismodelling can be observed for the quark-gluon discriminating variables.
In particular, the p T D distribution suggests that the correlation between the number of non-empty pixels and their energy sharing is not perfectly modelled. In addition, the size of the major and minor axes is overestimated for jets with transverse momentum below roughly 300GeV and it is underestimated above.

Discussion
The results that we discussed above represent a step forward in terms of accuracy of fast simulation systems proposed in the context of collider detector physics. We believe that three main aspects which were not exploited in similar works contributed to this: the use a generative model that is designed to handle well space correlations, and the use of a conditioning space (i.e. that of particle-level images) that encodes large amounts of spatial information; the explicit handling of the sparsity through the soft-mask layer; the use of physics-driven constraints on the total intensity of the jet images.
This hybrid approach that combines the adversarial setup with a parametric loss function allowed stabilising the training, and speeding up the convergence. However, this aspect of the work is not yet completely satisfactory. The performance was found to be very sensitive to the choice of the loss function hyper-parameters, and the training did not reach, in general, a stationary state. A review of the loss function structure, possibly incorporating the use of alternative formulations of the GAN game [44,45,46], and of the model training strategy in general, will be important to allow streamlining the method, and will the subject of future work.
Furthermore, the current approach introduces noise only through the stochastic generation of the set of active pixels. Our attempts at injecting noise at a more fundamental level in the generative model structure have been unsuccessful so far. Similar problems have been reported by researchers working on natural image generation (see e.g. [28]). A more extensive investigation of the handling of noise will also be performed in a subsequent work.

Conclusions
We have reported on a method that uses deep neural networks to learn the response of particle detectors simulation and reconstruction algorithms. The method is based on generative adversarial networks and it was applied to the generation of hadronic jet images at the CERN LHC.
We trained a generative model to reproduce the combined response of state-of-the-art simulation and reconstruction algorithms. This was possible thanks to the exploitation of the open datasets published by the CMS collaboration under the HEP data preservation initiative.
Starting from proposals made for natural image processing, we devised a hybrid set-up based on the combined use of generative adversarial networks and analytic loss functions that is able to take into account the conditioning on auxiliary variables and physics-driven constraints on the generation process.
Our method allows reducing the computation time required to obtain reconstruction-level hadronic jets from particle-level jets by several orders of magnitude, while achieving a very good accuracy in reproducing the simulation and reconstruction algorithms response. The model is in particular capable of reproducing the evolution of the reconstructed jet shapes as a function of several conditional variables. Physics-driven high level features commonly used for merged jets tagging and quark/gluon discrimination, and their evolution, are also generally well predicted.
The results obtained with this work represent a promising step forward towards the development of fast and accurate simulation systems that will be crucial for the future of collider experiments in high energy physics.