{'fraction_non_alphanumeric': 0.03411046111493462, 'fraction_numerical': 0.01303337921541638, 'mean_word_length': 5.098898216159497, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Utilizing Post-Hurricane Satellite Imagery to Identify Flooding Damage with Convolutional Neural Networks

Post-hurricane damage assessment is crucial towards managing resource allocations and executing an effective response. Traditionally, this evaluation is performed through field reconnaissance, which is slow, hazardous, and arduous. Instead, in this paper we furthered the idea of implementing deep learning through convolutional neural networks in order to classify post-hurricane satellite imagery of buildings as Flooded/Damaged or Undamaged. The experimentation was conducted employing a dataset containing post-hurricane satellite imagery from the Greater Houston area after Hurricane Harvey in 2017. This paper implemented three convolutional neural network model architectures paired with additional model considerations in order to achieve high accuracies (over 99%), reinforcing the effective use of machine learning in post-hurricane disaster assessment.


Introduction
Natural disasters, such as hurricanes, are extreme meteorological, hydrological or geological events that cause tremendous damage to affected communities. There are three crucial 'test_another' which is unbalanced and contains 8,000/1,000 images of damaged/undamaged classes [22].
In creating this dataset, Cao and Choe hoped to produce a model that efficiently assesses post-hurricane damage. Cao and Choe have shared their dataset with the intention of motivating others to build on their research, allowing for a more efficient damage assessment process [6].
Hence, this paper will explore how recent popular convolutional neural network architectures perform on this dataset, and compare the findings to Cao and Choe's model.

Convolutional Neural Network
Before trying to grasp convolutional neural networks, one should first have a basic understanding of neural networks. Neural networks (NNs) were first introduced by McCulloch and Pitts in 1943 [26], and are viewed as simplified models of the neural processing that occurs in human brains. More specifically, they are composed of artificial neurons linked together in networks, and thus are also known as artificial neural networks (ANN) (see Figure 3). Neural networks are utilized to make data-driven predictions/decisions [29]. Over the years, additional improvements have been introduced into the field: backpropagation [28]; splitting the data into training, validation, and test sets [30]; cross-validation [31]; regularization [32]; stochastic gradient descent (SGD) [33]; batch normalization [34]; data augmentation [35]. The neural networks which contain multiple processing layers give name to a new field within artificial intelligence: deep learning [36]. The structure consists of the first layer (input layer), the last layer (output layer), and the layer(s) in between (hidden layer(s)) [32]. Most neural networks are also known as feedforward neural networks because information only travels forwards through the network [27]. Neural networks which contain feedback connections are called recurrent neural networks. The neural networks (including convolutional neural networks) described in this paper are feedforward neural networks. Each neuron within the neural network contains a set of learnable parameters: a bias value and a collection of weights. Each input is then multiplied by a weight, and the weighted sum of the inputs is added to the bias value. Finally, this is fed into a chosen activation function, giving the final output of the neuron. Neural networks apply gradient descent in order to determine the set of parameter values that minimize a chosen loss function.
Through this method, neural networks seek to obtain a minimum loss [32].
Convolutional neural networks (CNNs) are a subset of artificial neural networks, and are most commonly applied in analyzing visual imagery [37,38] (see Figure 4). Convolutional neural networks contain three different components: convolutional layer(s), pooling layer(s), and fully connected layer(s). Convolutional layers are the center pieces of CNNs. These layers contain a set of learnable filters, each of which have small widths and heights, but extend through the whole depth of the input volume. The filters repeatedly take dot products of weighted parameters. For example, a 3x3 convolutional filter will take the dot product of 9 weighted parameters in a 3x3 block. This is then pushed through an activation function that later acts as an input to the next layer [21,32,46]. Common activation functions include sigmoid, rectified linear unit (ReLU), and leaky ReLU. For this paper, we will utilize the ReLU activation function as it has the potential to speed up the model training while maintaining consistent performance [45].
The pooling layer, also known as a subsampling layer, is utilized to reduce the dimensions of the inputs. Therefore, it is useful in reducing the number of parameters and computation required. It utilizes a small net and replaces the output value in each net by a new value (see Figure 5). The most popular, and the one implemented in this paper, is max pooling. This replaces the output value by the maximum within the net [46,47]. Finally, the network is flattened into a sequence of fully connected layers. The neurons in a fully connected layer (also known as the dense layer) are linked to all activations in the previous layer, like in a regular neural network. The output is determined in the same way as a normal neural network (inputs multiplied by weights, added onto a bias, and transformed through an activation function) [21]. For classification tasks, the softmax activation function is commonly employed in the last layer. This ensures that the values of each neuron corresponds to the probabilities of a single class for each image [32].

Data Preprocessing
Since the dataset contains folders with jpeg satellite images, we utilized python to load the data into multidimensional numpy arrays. Since each image is 128x128 pixels, and each pixel has RGB values, the final multidimensional numpy array will have size Nx128x128x3, where N is the number of images.
To preprocess the data, we normalize the data by scaling the pixels. Since each RGB value is between 0 and 255, we divide each pixel by 255, which causes each RGB value to be between 0 and 1. We normalized the data, because it is necessary for obtaining adequate results and quickening the runtime of the neural network, according to Sola and Sevilla [21].

Model Architecture
We opted to train pre-existing model architectures which have performed exceptionally well on other image datasets rather than create our own. The three model architectures implemented in this paper are based on AlexNet [23] and VGGNet [39]. AlexNet is a convolutional neural network architecture that was designed by Krizhevsky et al., and was utilized in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 (see  There exists key differences between the architectures of the two models chosen. VGGNet employs much smaller receptive fields and stride (3x3 with a stride of 1) compared to AlexNet (11x11 with a stride of 4). Furthermore, VGGNet contains groups of 2 or 3 convolutional layers before every max pooling layer. Meanwhile, the first two convolutional layers in AlexNet are followed directly by a max pooling layer. Most crucially, VGGNet has much greater depth than AlexNet, and thus achieved a higher accuracy in ILSVRC [23,39].
Given the limited access to graphic processing units (GPUs), training time became one of the most important factors in determining which models to implement. We decided to employ the AlexNet architecture directly as presented by Krizhevsky et al. because of its relatively fast training time, its ease of implementation, and also its overall high accuracy in image classification. Similar to Krizhevsky et al., we also implemented dropout layers after the first two fully-connected layers, because otherwise Krizhevsky et al. warns that the model substantially overfits [23]. For VGGNet, we implemented and trained a VGG16 network, same as Simonyan and Zisserman's model D architecture in Figure 7. However, since this architecture takes an extensive amount of time to train without access to a GPU, we also implemented a 3-block VGG-style architecture. The 3-block VGGNet maintains the general structure of a VGGNet architecture (two convolutional layers followed by a max pooling layer), but instead of containing 16 or 19 weight layers, the 3-block VGGNet only has 9 weight layers (see Table 1)

Model Training
In order to train our model with data, we employed the Keras library with TensorFlow [56] backend. We set the default mini batch size to 64. For all three model architectures we set the default learning rate to 0.001. Furthermore, the weights for the CNN were initialized using the He Uniform initializer [55]. The activation function utilized for the layers excluding the output layer is the rectified linear unit (ReLU) [45]. As aforementioned, the output layer employs the softmax activation function. The loss function employed is the cross-entropy loss. In fact, Weber et al. discovered that the most successful and reliable strategy for image classification is to implement a cross-entropy loss on top of a softmax classification function for the output layer [9]. We implemented the stochastic gradient descent (SGD) optimizer [33], with the default momentum set to 0.9. Because of the expensive computational cost of convolutional neural networks, we will tune the hyperparameters 4 greedily rather than in a full grid search. For example, if we determine that the best learning rate for a model is 0.001, we will utilize that learning rate when tuning other hyperparameters as well. Further, we will also choose which models to tune greedily, based on the few that performed the best with default parameters.

Model Evaluation
After the model completes training and tuning, the models will be compared to each other, and to Cao and Choe's model [6]. In addition to comparing accuracies, other measures will also be examined. More specifically we will focus on the confusion matrix or error matrix, which allows us to visualize the performance of the model true negative rate (TNR), also known as specificity, which equals the true negatives over all actual negatives; positive predictive value (PPV), also known as precision, which equals the true positives over all predicted positives; negative predictive value (NPV) which equals the true negatives over all predicted negatives; and F-score, also known as F-measure or F1 score, which is the harmonic mean of recall (TPR) and precision (PPV) [49]. In essence, these values are different types of accuracy measurements , and selecting an appropriate one depends on the objective of the experimentation [48]. For this paper, we will compare accuracies as a base level of measurement. Another adequate unit of measurement to utilize would be one which allows us to minimize the false negatives obtained (e.g. recall, F1 score, etc.), since incorrectly classifying an image as undamaged could be detrimental. In section 4, we will calculate these values for each model.

Validation
The validation set is obtained from dividing the training set into two parts: a training portion and a validation portion [30]. However, Cao and Choe had already separated the images into training and validation sets [22], and therefore we will simply utilize those. After training the model at every epoch, we are able to predict the responses for the validation set which provides us with an estimate of the true generalization error [30]. More importantly, the validation set is employed to tune hyperparameters-variables which we can change in order to control the model's behavior. The model cannot automatically adapt these variables and thus needs to be altered manually. Examples include the model's architecture (e.g. depth), the learning rate, the number of epochs, etc. [32].
In addition to simply employing a validation set, we utilized a method named cross-validation, or more specifically the k-fold cross-validation where k=5. After combining the training and validation sets, we split the data into 5 non-overlapping subsets. For the duration of 5 trials, we would set one of the subsets (a different one each time) to be the validation set while the other 4 become the training set. At the end, the accuracy would be the average accuracy of all 5 trials [32]. Cross-validation allows for low bias, since it is averaging the accuracies of multiple trials, thereby increasing the credibility of our model's results [50].

Model Considerations
One of the most difficult challenges of machine learning is improving its ability to generalize to external data that it has never seen before [35]. Specifically, one significant problem is overfitting the training dataset. Overfitting results when the trained model becomes more flexible than required, leading to unnecessary levels of complexity which results in a worse performance [51] (see Figure 9). While the model may seem to perform better on the training set, in reality it becomes too adapted towards the training set and stops generalizing towards previously unseen data [52]. We implemented multiple methods in order to reduce this, such as data augmentation, early stopping, batch normalization, dropout regularization, and weight decay.

Data Augmentation
Data augmentation is a method to increase the amount of data while also improving the quality of these datasets. It has been discovered that larger datasets result in better models [54], which is one advantage of data augmentation. Ultimately, data augmentation allows researchers to resolve the issue of overfitting the training set. Data augmentation includes two sections: data warping and oversampling. Data warping means altering existing data while preserving their labels (e.g. geometric transformations, random erasing, etc.) while oversampling means creating synthetic data and adding them to the training set (e.g. mixing images, etc.). The two data augmentation methods could even be combined [35]. For this paper, we utilized data warping through geometric transformations.

Early Stopping
Early stopping will terminate the model's training when the model has consecutively underperformed for a fixed number of epochs. More specifically, early stopping checks the validation set for overfitting errors (i.e. when the training error is larger than the validation error by a pre-specified threshold). This prevents the model from getting worse at generalizing to previously unseen data, thereby reducing overfitting [52]. We utilized early stopping for all our models. Furthermore, we set the default patience of early stopping (the number of epochs to pass without improvements on the validation set) to 5 epochs.

Batch Normalization
Training deep learning models with stochastic gradient descent (SGD) [33] has been extremely effective, but requires the careful tuning of hyperparameters (i.e. the learning rate and the initial parameter values). In fact, each layer's inputs are affected by the parameters in previous layers, creating cascading effects after small changes. This is problematic as the layers need to constantly adapt to the new distribution, which becomes more difficult as the depth increases. These problems can usually be addressed by utilizing the ReLU activation function, careful initialization, and low learning rates. However, to avoid this problem altogether, we can implement batch normalization. Batch normalization is a method which fixes each layer's input's mean and variance [34]. It does this by normalizing each batch, subtracting the batch's mean and dividing by the batch's standard deviation [35]. Ultimately, this decreases training time, allows us to use higher learning rates without divergence, and also regularizes the model [34].

Dropout Regularization
Dropout is a method which randomly drops neurons and all of their connections from the neural network during training (see Figure 10). This prevents the neural network from adapting too much towards its neighboring neurons. Although these co-adaptations may work well for the trained dataset, they also decrease generalization. Dropout will break these co-adaptations by randomly dropping neurons, preventing any neuron from becoming excessively crucial. This in turn reduces overfitting and increases the model's generalization. One side effect, however, is that models that implement the dropout method will typically take longer to train [57].

Weight Decay
Another method to increase the model's generalization pertains to weight decay, which prevents the parameter weights for each neuron from growing too large unless it is absolutely necessary. It does this by penalizing the cost function for excessively large weights. In our model, we utilized the L2 regularization, which adds a penalty term to the cost function: lambda times the sum of the weights squared. Lambda is a hyperparameter which controls how firmly we want to penalize large weights. We defaulted lambda to 0.001 as it performed the best on our models. Cortes et al. discovered that the L2 regularization was much more effective than the L1 regularization without decreasing performance which was why we implemented the L2 regularization [58, 59].

Results
As described in section 3.3.2, the three model architectures presented are AlexNet, VGG16, and a 3-block VGG-style architecture. Since the models utilize binary cross-entropy with a softmax activation function on the output layer, we set the model to decide either 'Flooded/Damaged' or 'Undamaged' based on whichever the function decides has a greater chance (higher than 50%). Furthermore, as described in section 3.3.3, we opted to tune the hyperparameters greedily. In this case, we trained each model based on the same default hyperparameters (few exceptions) and then chose the best to fine-tune the hyperparameters. Table 2 confirmed that the models were performing well regardless of the training and validation set utilized. Table 3, Table 4, and Table 5 display the different model architectures along with several modifications, and the accompanying accuracies.
In this case, the best model trained (VGG16 with data augmentation, batch normalization, and weight decay) achieved an accuracy of 99.15% (see Table 5 and Figure 11). This means that for every 100 satellite images received, the model will be able to predict about 99 of them correctly. Table 5 also indicates that the model achieves high results on the other values displayed, such as the F1-score. Even on the unbalanced data set, the model achieved an accuracy of 98.81% (see Figure 12).

Discussion
Interestingly, for AlexNet, implementing batch normalization made the model perform worse on average. In fact, implementing any of the additional modifications (i.e. batch normalization, data augmentation, and weight decay) made the model perform worse on average. This could be due to the inherent dropout layers we already added within the AlexNet model, making the other methods implemented relatively useless. Ultimately, although the AlexNet model performed much worse than the VGGNet architectures, it was much faster to train.
The 3-block VGGNet has the opposite results of AlexNet, where implementing batch normalization made the model perform better on average. Even with additional modifications (i.e. weight decay, dropout regularization, etc.), adding on batch normalization to the model increased its accuracy by 2.85% on average (e.g. adding batch normalization to a model with dropout regularization increased its accuracy by 5.45%). On the other hand, employing data augmentation seemed to worsen the model. Implementing data augmentation, batch normalization, and weight decay simultaneously, however, achieved on average 3.22% higher accuracy than the other runs solely employing data augmentation as a modification. Overall, the VGGNet model achieved 2.93% higher accuracy than AlexNet after the modifications.
The VGG16 architecture had similar results as the 3-block VGGNet, where adding on batch normalization increased the model's accuracy by 0.15% on average, and with data augmentation, batch normalization, and weight decay, the model achieved the highest accuracy of 99.15%. This is a 0.3% increase in accuracy from the 3-block VGGNet model albeit much slower to train.
Compared to Cao and Choe, our approaches were similar due to the large runtimes of the models. However, unlike Cao and Choe's models, we utilized early stopping, weight decay, and batch normalization, which made a difference in our results. Although Cao and Choe did implement dropout and data augmentation, the increased amount of methods utilized could have increased our model's regularization towards overfitting, especially with early stopping. Further, Cao and Choe only tested a few modifications (i.e. CNN, CNN with leaky ReLU activation function, and CNN with dropout and data augmentation). The limited amount of architectures Cao and Choe tested could have hindered their ability to achieve a better performance.
Additionally, we utilized different optimizers. While we implemented stochastic gradient descent, Cao and Choe employed RMSprop and Adam optimizers, which could have also led to differences in accuracy. Overall, our best model performance (VGG16 with data augmentation, batch normalization, and weight decay) achieved 1.86% higher accuracy than Cao and Choe's model for the balanced test set and 1.73% higher accuracy on the unbalanced test set [6].
Recently, more innovative models have been introduced (e.g. ResNet) that we would like to further explore in the future. Furthermore, due to our limited access to a GPU, we restricted the depth of our network. In the future, another network with greater depth could be trained (e.g. VGG19) to compare their performance. Moreover, the models are only trained on data specific to the dataset (the Greater Houston area in the aftermath of Hurricane Harvey [6]). In the future this could be generalized towards other hurricane events, and more data would be extremely useful in hyperparameter tuning and increasing generalization of the model(s). Additionally, due to the gravity of the aftermath, most of these images will not have time to be preprocessed. Thus, it is also crucial for the model(s) to effectively label low quality images (e.g. images partially covered by clouds), further reducing response times [6].

Conclusion
We conclude that convolutional neural networks possess the ability to automatically and very accurately label buildings as 'Flooded/Damaged' or 'Undamaged'. This paper explored the use of three model architectures (AlexNet, VGG16, and a 3-block VGG-style architecture), with modifications including data augmentation, batch normalization, dropout regularization, weight decay, etc. In the end, our model achieved a top accuracy of 99.15%. However, implementing the most innovative recent models could lead to even better results. Ultimately, CNNs can effectively be employed for flood detection, reducing the necessity for field reconnaissance while also reducing the response time.