{'fraction_non_alphanumeric': 0.0236152855302705, 'fraction_numerical': 0.011040912715451144, 'mean_word_length': 4.7166900420757365, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting County Level Corn Yields Using Deep Long Short Term Memory Models

Corn yield prediction is beneficial as it provides valuable information about production and prices prior the harvest. Publicly available high-quality corn yield prediction can help address emergent information asymmetry problems and in doing so improve price efficiency in futures markets. This paper is the first to employ Long Short-Term Memory (LSTM), a special form of Recurrent Neural Network (RNN) method to predict corn yields. A cross sectional time series of county-level corn yield and hourly weather data made the sample space large enough to use deep learning technics. LSTM is efficient in time series prediction with complex inner relations, which makes it suitable for this task. The empirical results from county level data in Iowa show promising predictive power relative to existing survey based methods.


Introduction
In the 2001 Nobel Prize winning paper "The Market for Lemons" 1 , George Akerlof shows that in a second-hand car market where asymmetric information exists, sellers know the quality of their cars but the buyers do not. Buyers offer a price based on the expected quality of the car. Sellers of the high-quality cars which are worth more than the average price will exit the market. This drives up the proportion for low value cars and in turn drives down the offer price. Eventually, only "lemons" are left in the market and the market collapses. The key to this collapse is that sellers have more information than buyers. This is called information asymmetry.
The solution to information asymmetry is to provide public information to all participants in the market at the same time. The United States Department of Agriculture (USDA) has been predicting national corn yield and production every year since 1964. The main method they use is survey based. They also use enumerators who make field visits in important corn production areas. The results from this traditional statistical method is subjective since it is the farmer's estimation at the point the survey was taken. USDA has tried new sources of data collection such as satellite imagery from MODIS (moderate resolution imaging spectroradiometer) from NASA. However, as of fall 2017, USDA continues to rely on the survey-based data.
Several private companies are possibly in a position to improve on the USDA survey. Examples include Lanworth 2 , Tellus Labs 3 and Climate Corp 4 . These companies set up plant growth models based on weather information and expert knowledge and they monitor satellite imagery and weather patterns. They incorporate as many independent lines of evidence as possible into their estimates and produce daily yield estimates in contrast with the monthly state-level prediction from the USDA. However, customers need to pay these companies for their estimates, possibly resulting in information asymmetry in the market. Traders in the CME corn futures who have preferential access to this information may be in a position to make profitable trades to the detriment of traders 4 who do not have access. The motivation for this work is to utilize modern data science to provide public daily corn yield prediction at a county-level and to aggregate this information to a national level. The long run intent is to eliminate information asymmetry in corn futures markets. Our work indicates that modern machine learning method has the potential to improve predictions relative to the USDA.

Background Knowledge
Corn is mainly planted in the Midwestern part of the United State (the green area in Figure 1), in an area called the Corn Belt. The region is characterized by level land, deep fertile soils, and a high organic soil concentration 5 . USDA usually reports the nationwide corn yield in late February of the year following harvest. Corn yield growth increased rapidly after 1950 with genetic improvement in seed and in farm management ( Figure 2).
Past research using machine learning predicted yield with discrete weather variables 6,7 . To the best of our knowledge, no research has been done to predict yield at any point during the growing season using only data accumulated up to that point. To accomplish this we use Long Short-Term Memory (LSTM) 8 , a special form of Recurrent Neural Network (RNN) 9,10,11 .Its efficiency in capturing long-term dependencies and predicting time series with complex inner relations makes it a perfect choice for our work. Though LSTM is one of the most popular methods in deep learning, it has never been used in any other field except natural language processing. This work is the first to apply LSTM in crop yield prediction, and demonstrates its potential to address other prediction problems. Machine learning has been considered to predict crop yields 18,19 . Monisha and Robert (2004) use an artificial neural network (ANN) model with rainfall data to predict corn and soybean yield 6 .
The Maryland Water Quality Improvement Act of 1998 requires mandatory nutrient management planning on all agricultural land in Maryland. In order to predict yields under typical climatic condition, they choose a machine learning method and find ANN models consistently produced more accurate yield predictions than regression models. Newlands and Townley-Smith (2010) were the first to apply Bayesian Network (BN) into crop yield prediction 20 . They attempt to predict energy crop yield and provide predicted probability distributions. However, they do not attempt to provide point estimates since BN is designed for categorical variables.

Yield Data
County level data was collected for each of Iowa's 99 counties from 1980 to 2016. The first 33 years were selected as training data, while the most recent four years were used for out of sample

Weather & Soil Data
Three types of input variables associated closely with corn yield are available. They are hourly weather data, soil quality data and soil moisture data. The hourly weather data was purchased from a professional weather data company -Weather Underground 23 . The data is a representative of an entire 19 Figure 3). We match the counties with each district and assign the value to those counties (i.e. counties in the same CRD has the same PDSI value).
Soil quality 26 data was collected from the SSURGO database 27 (database for storing gridded soil survey results) and aggregated to the county level using only areas classified as cropland according to the NLCD 28 (National Land Cover Database). For continuous variables, we aggregated to the county level by taking the average. The data covers the whole Corn Belt region at county level.
There are over one hundred variables in this dataset. Each variable is a constant number for each county, since soil quality typically does not change over time. We pick fourteen variables from the data, which we think are most related to corn yield ( Figure 1). Root zone for water storage and soil droughty vulnerability are considered the most two significant soil variables.

Variable selection and data preprocessing
The time series of input variables can be expressed in hourly or daily format. Each county for each year is a sample record with output valueyield, and the corresponding input time series falling into the growth period April to October, so the length of the input time series {} t x would be t=5,136 for hourly input vectors or t=214 for daily input vectors. However, for hourly inputs vectors, there are too many parameters needed to estimate with only 3,267 training samples.
Therefore we use a daily input sequence {} t x with t=214.
Variable Selection. There are totally twenty-eight candidate input variables. In addition to the fourteen soil quality variables and PDSI, we also include max/min/mean temperature for each day, total daily rainfall, daily average wind speed, max rainfall during the day, accumulated rainfall and GDD accumulated up to that date. Since July is the most important month for corn growth 29 , rainfall and max temperature in July are also included. The ratio of acres planted for corn divided by the total acres planted may also influence the corn yield since farmers in corn intensive counties will specialize in corn management techniques and management. Two interaction terms (max temperature*soil droughty and max*PDSI) are also included. The idea behind this is that at high temperatures soil moisture will be more important than at low temperatures.
First, we trained the model with all 28 variables included. Our county level yield predictions turned out to be almost a horizontal line at the county average yield level. We believe this was due to the use of too many constant terms which is irrelevant to the model. We use MRMR 30 (minimum redundancy maximum relevance), a feature selection method introduced in Peng (2005) to rank the soil quality data and eliminated those variables that had low rank, we also calculated the 10 correlation between weather variables and eliminated the highly correlated ones. After trial and error with different combination of the input variables and with expert knowledge, we arrived at the "best" ten input variables for corn yield prediction. These are max/min/mean temperature, total daily rainfall, wind speed, soil root space for holding water (rootznaws), soil droughty vulnerability (droughty), PDSI, accumulative rainfall and GDD. CRD level and all other data are also average numbers for county area (the most precise data point should be each farmland, which is not available).
Ten input variable sequences are stored in the format of 3D tensor cube. This is a key step to make our data fit into the model. All input data should be converted into tensor format for LSTM training in computer. Figure 1c is an example of 3D tensor cube. X-axis indicates the number of the input variables, Y-axis is the length of the time series and Z-axis shows the number of samples.
Hence the dimension of our 3D tensor is10 214 70026  . PDSI is monthly data, so it repeats the times of number of days in each month. While rootznaws and droughty repeat 214 times since they are constant.

Methodology
A supervised methodrecurrent neural network (RNN) 9,10,11is used first. RNN is a family of neural networks for processing sequential data. It is typically used in text prediction and speech recognition. RNN is very useful for cases where there are nonlinear and unknown interactions but it does not provide causal relationships. The challenge is to fit our problem into RNN format. Even though RNN is specially created for time series data, previous applications have focused the prediction of the following points in the same time series. This paper develops a novel way adapt RNN to predict crop yields a problem where crop yield responses from prior decades may have predictive power.

Recurrent Neural Network
For a regular RNN, the network consists of three layers: an input layer, a hidden layer and an output layer. () t x is the input sequence, () t y is the output sequence and () t h is a series of hidden states. The number of the hidden layers is not constrained to one. In the deep learning recurrent neural networks, the number of the hidden layers can reach eight or more. Adding hidden layers can help to study the more complex structure of the model, but also requires more data. ,, U V W are shared weights that we need to learn. And there is an activation function f that Wh  , which should be chosen before learning the networks. The corn yield prediction problem could not fit into a regular RNN, therefore we use the many-to-one RNN model here ( Figure 5). Many-to-one RNN model is suitable when there is sequence input with one output, thus it is perfectly match with our data format described in the data section.

Long short-term memory (LSTM)
The mathematical challenge of learning long-term dependencies in recurrent networks is called the vanishing gradient problem. As the length of input sequence increases, it becomes harder to capture the influence of the earliest stages. The gradients to the first several input points vanish and become equal to zero. Therefore a special RNN model called Long short-term memory (LSTM) 8 is developed. In the recurrency of the LSTM the activation function is the identity function with a derivative of 1.0. So, the backpropagated gradient neither vanishes or explodes but remains constant. Figure 5 shows the difference in the framework between regular RNN and LSTM, where tanh (hyperbolic tangent function) is a commonly used activation function. It is clear that LSTM has a more complex structure to capture the recursive relation between the input and hidden layer.
LSTM adds a new sequence {} t c , called cell state, upon the regular RNN. Cell state is a space that is specifically designed for storing past information, i.e. the memory space. It mimics the way the human brain operates when making decisions. Operation is executed to update old cell state Our LSTM model has been learned using a Python package called 'keras' 33 on top of Theano 34 backend. Hyperparameter searching is an important process before the commencement of the learning process. We assign a set of numbers for hyperparameters such as learning rate, number of hidden layers, number of hidden nodes in each layer, dropout rate and let the machine randomly pick one value in the set for each hyperparameter. Usually after searching for over 300 models with different combination of hyperparameter settings, we can find the 'best' model and the corresponding 'best' hyperparameters.

Results and Discussion
Our choses model has two hidden layers, daily input vectors and two combination samples. Figure 6 and 7 shows the "best" prediction with percentage and constant adjustment respectively. 16 The black line is our prediction while the red is the true yield. There is also a chart showing the absolute error between the prediction result and the true yield. We include the absolute error as a standard to judge the performance of the model. part, we conclude that whichever year the yield is adjusted to, there will be hardly any influence on prediction results. Therefore, we uniformly adjust the yield to 2013 base for all models.

Our initial model starts with hourly input vector {}
We train the initial model with daily input vectors and try to improve the model from three changes in the settings: 1. use two hidden layers instead of one; 2. adding more training samples created with combination method (two+three counties average); 3. including more input variables (variable selection from the 28 variables introduced in section 3.3). Table 1 shows that 10 "best" input variables with 2 hidden layers and combination samples has the smallest mean squared error ( Figure 9). Nevertheless, the fluctuation of prediction is still less than the true yield. Does this average trend exist because of including too much combination samples? Therefore, we trained the "best" model again with only two counties combination samples added, which totals 19734 samples. We also use the constant genetic gain adjustment with yield data.      USDA and constant adjustment has almost the same performance. However, for 2016, our prediction are much lower than the actual yield, which indicating that the weather event in 2016 were not represented in the historical data.

Early Prediction
In order to make comprehensive comparison between USDA and our model, we also trained other three models as early prediction for data available to August, September and October respectively (i.e. y=122, 153 and 183 for the 3D tensor cube in section 3.3). Figure

Summary and Future Work
In this paper, we demonstrate a LSTM model for county-level corn yield prediction in Iowa.
The model performs reasonably well based on expert's opinion, which indicates the potential of LSTM in yield prediction rather than in language processing. The good performance of early monthly prediction through LSTM has shown the possibility of high-quality daily prediction which is publicly available. While the main future goal of this research is to provide accurate daily county-level prediction for the whole Corn Belt, we also consider to design specific models for prediction under extreme cases such as flood or drought in the future.