{'fraction_non_alphanumeric': 0.02072041166380789, 'fraction_numerical': 0.007066895368782161, 'mean_word_length': 5.152807091599831, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Inversion of 1D frequency- and time-domain electromagnetic data with convolutional neural networks

Inversion of electromagnetic data finds applications in many areas of geophysics. The inverse problem is commonly solved with either deterministic optimization methods (such as the nonlinear conjugate gradient or Gauss-Newton) which are prone to getting trapped in a local minimum, or probabilistic methods which are very computationally demanding. A recently emerging alternative is to employ deep neural networks for predicting subsurface model properties from measured data. This approach is entirely data-driven, does not employ traditional gradient-based techniques and provides a guess to the model instantaneously. In this study, we apply deep convolutional neural networks for 1D inversion of marine frequency-domain controlled-source electromagnetic (CSEM) data as well as onshore time-domain electromagnetic (TEM) data. Our approach yields accurate results both on synthetic and real data and provides them instantaneously. Using several networks and combining their outputs from various training epochs can also provide insights into the uncertainty distribution, which is found to be higher in the regions where resistivity anomalies are present. The proposed method opens up possibilities to estimate the subsurface resistivity distribution in exploration scenarios in real time.


Introduction
The meteoric rise of deep learning (DL) models in many fields of science and technology has attracted considerable attention from the geophysical community and made artificial intelligence one ) and many other geological problems. The main advantages of DL methods are that they allow the detection and exploitation of nonlinear dependencies in the data without specifying a particular model in advance. In order to achieve maximum performance, DL methods require massive amounts of data, which makes them well-suited for handling large-scale datasets. They also scale better with problem size compared to other machine learning techniques, and in many cases are able to produce better results. Features automatically learned by DL may potentially be more valuable and better suitable for analysis than hand-engineered features. By exploiting different layers of abstraction, deep neural networks are able to discover both the low-level and high-level features in data.
Inversion of electromagnetic (EM) data is an important problem with applications in different areas of geophysics. Numerical methods for 1D EM inversion have been extensively studied in the past decades, e.g., Farquharson et al. (2003), Auken et al. (2005), Constable  of the EM field in two independent modes, namely a transverse electric (TE) mode and a transverse magnetic (TM) mode, and the resulting field is simply the sum of these two modes (Ward and Hohmann, 1988).
The inverse problem in non-unique, i.e. many models fit the data equally well. Local optimization methods for inversion such as the nonlinear conjugate gradient or Gauss-Newton are highly prone to getting trapped in a local minimum, especially when the starting model is chosen far away from the optimal model. Inversion of large-scale geophysical data often requires the use of high performance computing systems, especially when measurements at multiple source-receiver positions are considered (Puzyrev et al., 2018). Probabilistic inversion methods have been applied to various In this study, we explore the potential of deep CNN for 1D EM inversion. This approach to inversion is entirely data-driven, does not employ traditional gradient-based techniques and provides a guess to the model instantaneously. Our research provides two contributions to the field: (a) development of 1D CNN-based models for EM inversion and (b) application of these models to both synthetic and real datasets. In particular, we demonstrate the CNN approach on two types of electromagnetic survey measurements: marine frequency-domain controlled-source electromagnetics (CSEM) and time-domain electromagnetics (TEM). The former method is typically used for offshore hydrocarbon exploration while the latter is more commonly employed in mineral exploration, environmental engineering problems and hydrogeological studies. This paper is organized as follows. First, we formulate the EM problems of interest in Section 2.
In Section 3, we describe the architecture of the deep neural networks employed and the training process. The performance of the method is investigated on two examples including synthetic and real EM data in Section 4. Finally, the last section summarizes the outcomes and points out future research directions.

Background
The marine frequency-domain CSEM method measures the electric and magnetic fields on the seafloor which are produced by a grounded electric dipole antenna carrying alternating current. This type of transmitter generates vertical current flow in the seafloor and the method is therefore used to detect and image resistive targets like hydrocarbon reservoirs. In a typical survey, multiple EM receivers measuring up to three electric and three magnetic field components are deployed on the seafloor and the electric dipole antenna -transmitting multiple frequencies -is towed over the array; measurements are therefore made at transmitter-receiver offsets ranging from 100s of meters to 10s of kilometers. While 3D anisotropic inverse modelling is now the standard interpretation tool in such multi-offset, multi-component, multi-frequency CSEM surveys, we demonstrate our CNN methodology using synthetic data generated from 1D isotropic layered models; extension to 3D will be the subject of a future paper. A large amount of training data is one of the key requirements for DL methods. In order to accurately establish a relationship between the data and model parameters, the networks are required to see as many as possible different examples during training. To create such large database of synthetic CSEM responses, we use a parallelized version of the algorithm described in Swidinsky et al. (2018) to calculate the EM response of a CSEM system above a multi-layered seafloor.
In the TEM method, steady current in an ungrounded transmitting loop of wire is abruptly switched off, inducing secondary currents to flow in the subsurface by Faraday's Law. These secondary currents are primarily sensitive to conductive targets (such as massive sulfide mineralization and freshwater aquifers) and through Ampere's Law produce a secondary magnetic field which decays with time. The time-derivative of this magnetic field is typically measured as a decaying voltage using a vertical induction coil receiver placed at the center of the transmitting loop.
In this configuration, it is still very common to use 1D inversion methods to interpret survey data, because the measurements are generally sensitive to the subsurface structure directly beneath the transmitter. Layered models of conductivity versus depth from multiple TEM soundings are stitched together to produce a pseudo-2D or pseudo-3D image of the subsurface. Our results using neural networks trained on 1D models are therefore directly applicable to present-day field practice and we demonstrate the CNN methodology on a real ABEM WalkTEM dataset from Denmark, provided to us by Guideline Geo. We use a modified version of the algorithm described in Swidinsky et al. (2012) to calculate the EM response of a TEM system above a multi-layered earth.

Inversion with neural networks
The goal of the DL inversion is to identify the subsurface model directly from the given data in a single step without constructing the gradients. The workflow consists of three stages: (1) generation of data, (2) training of the network and (3) prediction of model parameters. While the first stage can be challenging in 2D and 3D conditions, the generation of 1D data typically does not pose significant problems due to the low number of model parameters and the availability of fast numerical simulators.
Once a sufficiently large and representative set of measurements and corresponding resistivity models is generated, it can be used in network training. After training, the network is expected to correctly predict the resistivity distribution both for the data used in training (training set) as well as for new, previously unseen data (test set).
CNN are arguably the most common type of deep neural networks and one of the most powerful developments in artificial intelligence in recent decades. They can be viewed as a natural extension of neural networks for processing images and data with a grid-like topology. CNN have an ability to learn filters that represent repeating patterns. These networks have shown good performance in data processing tasks where picking the most important features from segments of data regardless of the specific location of these features within the data is essential. Deep CNNs include multiple convolutional layers, thus forming a hierarchy of feature detection. As the features extracted at the previous level become the input at the next level, the model is able to capture small-and large-scale features. Besides their natural applications in computer vision, social media, robotics, and autonomous vehicles, CNN are actively extended to other types of problems such as modelling and simulation of physical systems or predictive analytics. While the method allowed for 2D pixel-based inversion with 10,000 unknowns, the monitoring setup is such that all the 20,000 sample models has one (or several located close to each other) resistive targets. The exploration setup is more challenging as there is limited information about the resistivity distribution and thus many more situations should be considered.
During training, the parameters (weights) of the network are progressively updated by minimizing the error on the training dataset (training error). We are interested though in how well the method generalizes to the data it has not seen before (test error) since this determines its practical performance. In order to choose optimal values for the hyperparameters of the network and to stop training before the network starts overfitting to the training data, we monitor the error on the validation dataset (validation error) during the training and choose the network that achieves the lowest validation error. This early stopping criterion allows avoiding overfitting to the training data.

Data generation
One of the key requirements of DL methods is a large amount of data required for training. In Both the CSEM and TEM input data are standardized using the mean and standard deviation values on the corresponding training sets: and natural logarithm of the resistivity = ln( ) is used as the unknown model parameter.

Frequency-domain CSEM inversion
In the first model scenario, we consider a marine CSEM setup where low-frequency EM energy is generated by a towed electric dipole. Following the well-studied model described by Key Table 1. In Figure 3, we show the inversion results using only two frequencies of 0.1 and 1 Hz. Using less data and lack of high frequency content impacts our ability to determine accurately the resistivity in the cases where several thin targets are clustered together, such as in example (h). The RMSE and R 2 metrics for most of the cases are inferior compared to the 5-frequency scenario, although the main features of the models are still determined correctly. To get a better understanding of the potential accuracy of the CNN-based inversion, we investigate the performance of the neural networks trained on datasets of different sizes ranging from 10 thousand to half a million examples. Table 2 shows the training and test errors for the CNN used above and a fully-connected network with 7 hidden layers each having 512 neurons. Using small training datasets results in a large gap between the training and test error (the networks overfit the data but generalize poorly). Several hundreds of thousands of training examples seem to be an optimal number needed for accurate and reliable prediction of the resistivity parameters in this scenario. The convolutional architecture achieves lower test set errors compared to the fully-connected network.
CNN inversion can also provide some insights into uncertainty distribution. Figure  hyperparameters and comparing their results from several training epochs can be used as a rather simple way to quantify uncertainty in the inverted models. Figure 5 shows the inversion results for a 2D model comprised of 100 1D resistivity profiles.
The data is obtained by 100 independent 1D simulations. These 1D models are parametrized in a slightly different way than the one described in Table 1 (lower baseline resistivity and smaller random fluctuations); nevertheless, the inversion results are accurate and the RMSE and R 2 metrics are even superior compared to the models from the test dataset. Both shallow thin resistive targets are restored well; the position of the deeper target is also accurate although its resistivity is underestimated by ~30-45 Ohm-m. The uncertainty panel on Figure 5 shows the maximum difference in the predictions of the networks from 10 different epochs (scaled by the average resistivity value in the corresponding cell; the average is taken over the predictions from 10 epochs and the true model).

Time-domain inversion
In this section, we examine the performance of the method on an ABEM WalkTEM dataset deterministic inversion) which is expected since the "real" models come from another distribution (5 layered-models versus 50-layered models used in training); nevertheless, the quality of inversion is quite good as shown in Figures 6 and 7.

Discussion and conclusions
Traditional EM inversion methods are commonly based on gradient-based algorithms, which suffer from non-linearity and non-uniqueness of the inverse problem. On the other hand, the use of probabilistic inversion methods is often limited due to their high computational cost. A possible alternative lies in inversion based on deep neural networks, which are capable of learning deep representations and identifying complex patterns in data. This is of particular importance for large real datasets where complex data interactions are difficult or even impossible to specify within existing models. In this study, we explored the potential of DL based inversion with two 1D EM problems and showed that deep CNN can accurately reconstruct the resistivity distribution in the subsurface from measured data.