{'fraction_non_alphanumeric': 0.027309877404564937, 'fraction_numerical': 0.007525166949068076, 'mean_word_length': 5.031559963931469, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated model generation for electronic devices using neural network approaches

This paper describes artificial neural network approaches to convert the conventionally expensive process of electronic modelling into an automated model generation process. Artificial neural networks are trained using machine learning algorithm to learn the electronic behaviour, and the trained neural network becomes a model to help predict the electronic device behaviour. The automated model generation algorithm performs adaptive data sampling to determine the amount and the distribution of training data needed to train neural networks. The algorithm also determines the number of hidden neurons needed to achieve a compact and accurate model. Also incorporated into the automated model generation method is an efficient interpolation approach to make the process much faster. The objective of the described method is to generate a compact neural-network based model with better accuracy and in less time than conventional approach. Examples of automated modeling of radio-frequency and microwave filters used in wireless electronic systems are described showing the advantage of this technique.


Introduction
Artificial neural networks are important vehicles in data processing and machine learning. In addition to popular applications such as image processing, speech recognition and control, artificial neural networks have also found interesting applications in engineering fields such as electronic modeling and design. Artificial neural networks are trained to learn the behavior of high-frequency electronic devices through machine learning algorithms. The trained neural networks can then represent the model of electronic devices [1,2]. Because neural networks can extract underline relationships from electronics data, the neural network is a much more efficient way to evaluate electronic design solutions than direct look-up table approach using measurement data. Various approaches to apply neural networks to electronic modeling and data processing have been presented in the past, such as radio-frequency and microwave modeling and design [1,2], high-power high-frequency transistor modeling [3], electromagnetic modeling/optimization [4,5] and waveguide filter inverse modeling [6]. The combination of electronic knowledge and neural networks led to knowledge-based neural networks for electronic modeling [7] and space mapping concepts [8] has been applied to map empirical models to accurate device behavior through neural networks. Another important direction is automation of neural network model development which was introduced to integrate all subtasks like data generation, neural-network selection, training and test into one unified framework [9]. In addition to making neural network training more efficient, the automated model generation also addresses two critical problems of neural network training, i.e., determining the suitable amount of training data and the suitable number of hidden neurons. The neural network training is performed stage by stage, where in each stage the amount of training data or the neural network size is adjusted, and neural network is re-trained. In this way, the suitable amount of training data and the suitable number of hidden neurons are automatically determined as part of the training process. However, process in [9] involves repetitive training of neural networks while detecting if the training data is sufficient in different regions of the nonlinear electronic parameter space. This reduces the model development efficiency.
This paper describes an advance on automated model generation [10]. We develop systematic algorithm to incorporate data sampling, data generation, neural network training and testing in a more efficient manner. We also incorporate interpolation techniques to avoid training the intermediate neural networks during the automated model generation process. Training is performed only after data sampling process has finished, making the automated modeling generation process faster than the previously published approaches. In addition, we add a procedure to reduce the number of hidden neurons so that the guess of initial neural-network structure can be more flexible and the final size of the neural-network model is more compact. Demonstration of automated model generation for a radiofrequency and microwave filters are also described.

Training data and data generation for high-frequency electronics modeling
Let x represent a n-vector containing physical/geometrical parameters of an electronic device, e.g., gate length and gate width of a high-frequency transistor, or width and spacing of a transmission line. Let y represent a m-vector containing the responses of the device/circuit under consideration, e.g., drain current of a transistor, or scattering parameters of a transmission line. Let the physics/ electromagnetics relationship between x and y be represented as . This relation can be highly nonlinear and multi-dimensional. The theoretical model for this relationship may not be available (e.g., a new semiconductor device), or theory may be too complicated to implement, or the theoretical model may be computationally too intensive for online microwave design and repetitive optimization (e.g., 3D full-wave electromagnetic analysis inside a Monte Carlo statistical design loop).
To overcome these problems, we aim to develop a fast and accurate artificial neural network model for the relation f by teaching/training the artificial neural network to learn the electronic behaviour from a set of data. Let the training data be denoted as input-output sample pairs, {(x (i) , d (i) ), i=1,2, …, Nd}, where d (i) represents the data for output y when the inputs are x (i) , and Nd represents the number of training data. The measured/simulated data of the outputs and the inputs are related by, The data generation can be done either by measurement of the device, or by simulation using electromagnetic simulators. Typically, data for nonlinear active devices such as Gallium Nitride transistors are obtained by measurement. The scheme to gather measurement data is that we first apply different voltages to the transistor gate and drain, and then measure the drain current to obtain current voltage data. Secondly we generate small-signal data by measuring the scattering parameter data (called the S-parameter data) over a frequency range, and re-perform such measurement again and again for different operating bias voltages. Thirdly, we perform large-signal measurement by exciting the device with large signal at the gate, and measure the distorted large-signal at the drain. In this way, we obtain the measurement data in the form of harmonic data including signal at fundamental frequency, second-harmonic frequency, and all the high-order harmonic frequencies. This harmonic data measurement is repeated for different operating bias points, different values of fundamental frequencies and different levels of input signal power. Consequently, a large amount of data is collected, which contain valuable information of the device internal dynamic relationships, for which no explicit formula is available.
Data generation for passive electronic components are typically achieved through simulation. This data can be supplemented by measurement data. The reason is that the typical design variables for passive components are geometrical variables such as height of an iris window in a waveguide resonant cavity, or length and width of a microstrip in a filter. It is expensive to manufacture a large number of passive components corresponding to different geometries just for purpose of data collection. Therefore, electromagnetic simulation become typical choice for collecting data for passive high-frequency electronic components. The amount of data becomes large if more geometrical variables (e.g., length 1, length 2, …, width 1, width 2, … for multiple coupled microstrip lines) are needed. In this case, we need to obtain data for various combinations of values from many geometrical parameters.

Training of neural networks with automated model generation algorithm
To efficiently use data to train the neural networks, our automated model generation process includes initial data generation, neural network training, neural network size adjustment, additional data generation, and neural network testing. Incremental data generation is performed whenever and wherever data are needed as decided by the model generation algorithm during neural network training process.
The overall automated model generation process proceeds in stage-wise process [10]. Each stage includes data generation, neural-network structure adaptation, training and test. Different stages may have different data and different neural-network size. Our process uses three sets of data, namely training, interpolation and test data. Training data are used to adjust neural-network weights during training. Interpolation data are used to create local models in subregions of the model input space R 0 to assess the adequacy of training data during training process. Additional training data are added upon detecting data inadequacy by the interpolation criteria. Test data are used to check the quality of the final neural model.
Let (x (i) , y (i) ) represent the ith data. Let L k , I k , and T k denote training, interpolation and test data respectively in the kth stage. Let H k denote the number of hidden neurons of neural network S k in the kth stage. The algorithm regards the original input space R 0 as one region at first. If the test error in the region is large, the region would be divided subregions and new data are generated. In each stage, the algorithm compares the test error in each subregion. The subregion with the maximum test error is chosen as the worst region R * . Then R * would be further divided into 2 n new subregions in the next stage. In order to avoid training intermediate neural networks in each stage, we evaluate the test error of various subregions using interpolation approaches during each stage of the automated model generation process. Different subregions have different interpolations. When calculating the interpolation in one region, the region is called the interpolation region. Even-though the electronic behavior over the entire region of design space is complex and nonlinear therefore requiring a neural networks to learn, the interpolation space is usually in a small local subregion. Therefore, a simpler interpolation function g(x) is sufficient for interpolation space. In this way we speedup the overall model generation process. In our work, we use polynomial interpolation. Let g(x) be the polynomial interpolation function with N terms. Suppose p is a vector of coefficients in the interpolation function, and h(x) is a vector of polynomial terms in the multi-dimensional space of x. For example, for a second order polynomial in two variables, n 2, N 6, p = [ p1, p2,_, p6] and h = [1, x1, x2, x1x2, x1, x2].
In general for neural networks with multiple outputs, say m outputs, p would be an N x m matrix. To compute the test error for an interpolation region, we select M training data in and around the interpolation region to form matrices A and B, such that where A has N columns and M rows, and B has m columns and M rows, where N, M and m are the number of elements in h, the number of training data selected, and the number of model outputs, respectively. The elements in B are output values y (i) from M training data. To determine A, the locations of these M training points x (i) are related to x * , which is at the center of the worst region R * . x (i) is obtained as where r is the number of times the original region was divided to reach the present worst subregion R * , xmin and xmax are minimum and maximum values of the variables in the original x space, x * is the test point in R * , and Qi is a n n diagonal matrix with the values 0, +1, or -1 on its diagonal. Different combinations of 0, +1, and -1 in the diagonal of Qi allow us to obtain different training points. After normalizing the x and y values, the A matrix can be derived as a constant matrix, containing 0, +1, and -1. Additional scaling factors can be applied to particular rows in A in order to emphasis the importance of fitting particular data points. The least squares solution of normalized version of (2) is Subsequently the interpolation function is obtained for the subregion R * . Then, the test error EIn of the interpolation regions is calculated by comparing the g(x) and test data. The subregion with the largest test error will be designated as the new worst region R * . Automated model generation process will then enter the next stage where the new worst region is further divided, forming a new set of subregions, and the process described in (2)-(4) is repeated. In this way, we take advantage of availability of training data to produce localized interpolation function, avoiding the training of neural network during intermediate stages of automated model generation process, and speeding up model development. Neural network training is performed only during the last few stages of automated model generation process. The final model is a neural network valid in the entire region. We also add a procedure reducing the number of hidden neurons after good-learning is detected, as shown in figure 1. In the figure, E d represents user-defined error threshold for training the model. S k denotes the neural network in the kth stage of automated model generation process, and H k represents the number of hidden neurons of S k . Symbols EIn and El denote interpolation error w.r.t. test data, and training error w.r.t. training data, respectively.
The algorithm will recursively reduce the number of neurons until the error of the reduced neural network starts to increase. This procedure will allow the guess of the initial number of hidden neurons to be more flexible, and make the final neural model more compact. The overall algorithm can start from an initial set of data and initial neural network, and ultimately find the most compact neural network to learn the given electronic problem using data collected at the most critical regions of the input parameter space.  In this section, we describe results of applying the automated model generation method to an example of neural network modeling of a bandpass filter in wireless communication systems [10]. More specifically, the application of the method is demonstrated by the development of a parametric model of a High-Temperature Superconductor (HTS) quarter-wave parallel coupled-line microstrip filter [10,11] used in radio-frequency and microwave wireless electronic systems [8]. The HTS filter structure is illustrated in figure 2. In order to generate training data for neural networks to learn, the filter geometrical formulation is setup in a 3-dimension electromagnetic simulator. The electromagnetic simulator is executed upon demand to generate training data at the worst sub-regions of the modeling space in each stage as determined by the automated model generation algorithm. The response behavior of the radio-frequency/microwave filter is best represented by the transfer function and the return loss of the filter. Since this filter operates in 4 GHz frequency range, conventional resistor/capacitor/inductor representation is not applicable. Electromagnetic oriented models are required to capture the subtle relationship between geometrical parameters and the filter responses. However, standard electromagnetic simulation is time-consuming costing the designer's expensive online design time. We use neural networks to learn from electromagnetic data of the filter. Using the automated model generation process, neural network models can be developed offline, including automated data generation offline and automated neural network training offline. The trained neural network is a fast and accurate model for designer's online use, providing fast estimation of electromagnetic behavior of the filter during filter design. In this way, filter design can be substantially faster.

Experimental / Theoretical Results and Discussions
The input parameters for the neural network model are x = [L1, L2, L3, S1, S2, S3] T , where L1, L2 and L3 are the lengths of the parallel coupled-line sections and S1 , S2 and S3 are the gaps between the sections. The outputs for the neural network are y=[S11, S21] T where S11 represent the return loss of the filter, and S21 represent the transfer function of the filter in terms of the input-output wave signals.
Training data generation is done by an electromagnetic simulator, called CST Microwave Studio. In this example, there are six geometrical variables, i.e., n 6. Matrix A has 70 columns meaning that the interpolation function has 70 coefficients or 70 polynomial terms (i.e., N=70). The number of rows of matrix A is 729, meaning that the number of training samples around the neighborhood of an interpolation space is 729 (i.e., M=729). Matrix B has 2 column and 729 rows. The worst region R * is divided into 64 subregions, so there are 64 A matrices used during interpolation process. The automated modeling methods used 1394 electromagnetic training geometries. After 6 stages of the process in Figure 1, the final neural network training is finished. The test error of the final neural network is 1.59% representing a good accuracy for engineering use. Figure 3 shows the neural network response compared with original electromagnetic response, confirming that the neural network has good accuracy for predicting the electromagnetic behavior of the filter. Notice that the  For comparison purpose, we also attempted conventional manual data generation and training of neural networks. To achieve similar test accuracy (1.59%), we endup with much more training data (4096 electromagnetic training geometries), and more training time (85 hours including data generation and neural network training). Using the automated model generation algorithm, the training data is judicially selected such that highly nonlinear sub-regions of the modeling space have more training samples, and smooth subregions have less training samples. In this way, only 1394 electromagnetic training geometries are sufficient to achieve same accuracy as manual training with 4096 training geometries. Subsequently the automated model generation process is much faster than manual data generation and training.
Further comparison is performed by comparing the method in this paper with the automated training algorithm of [9]. The method described in this paper achieved same accuracy of trained neural network as that of [9]. However, the model trained by the method in this paper used less stages (6 stages by the method in this paper versus 9 stages by the existing automated model generation of [9]). The total amount of time for developing the model including electromagnetic data generation and neural network training by the method in this paper is less than the existing automated model generation of [9] (22 hours by automated algorithm in this paper versus 26 hours by previous automated algorithm), and produced a more compact neural network. Other aspects for the comparison are equal between the automated model generation algorithm of this paper and the automated model generation algorithm of [9], including same optimization method for training, same electromagnetic simulator, the same geometrical configurations for the bandpass filters. In training the neural networks, the quasi-Newton algorithm, which is a second-order gradient based optimization algorithm, was used for training. The models trained by the method in this paper and that in [9] are both validated by comparing with original electromagnetic data simulated at 10001 frequency points across the entire frequency range of interests. The reason for a net speedup of model development by the method