{'fraction_non_alphanumeric': 0.034562465602641715, 'fraction_numerical': 0.007154650522839846, 'mean_word_length': 4.867290926703261, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EarthMapper: A Tool Box for the Semantic Segmentation of Remote Sensing Imagery

Deep learning continues to push state-of-the-art performance for the semantic segmentation of color (i.e., RGB) imagery; however, the lack of annotated data for many remote sensing sensors (i.e. hyperspectral imagery (HSI)) prevents researchers from taking advantage of this recent success. Since generating sensor specific datasets is time intensive and cost prohibitive, remote sensing researchers have embraced deep unsupervised feature extraction. Although these methods have pushed state-of-the-art performance on current HSI benchmarks, many of these tools are not readily accessible to many researchers. In this letter, we introduce a software pipeline, which we call EarthMapper, for the semantic segmentation of non-RGB remote sensing imagery. It includes self-taught spatial-spectral feature extraction, various standard and deep learning classifiers, and undirected graphical models for post-processing. We evaluated EarthMapper on the Indian Pines and Pavia University datasets and have released this code for public use.


I. INTRODUCTION
S EMANTIC segmentation is automatically labeling every pixel in an image with a semantic category, which is used by the remote sensing community for land-cover classification [17], vegetation quality estimation [22], among others. In computer vision, results on semantic segmentation benchmarks [8], [20] have rapidly increased thanks to deep convolutional neural networks (DCNNs) that are pre-trained on millions of annotated RGB images (e.g., from ImageNet). While there is a large amount of annotated, publicly available RGB imagery for training large DCNNs, annotated data for specialized imaging systems, such as multispectral and hyperspectral imagery (HSI), is scarce because generating large, sensor specific datasets is expensive and man-power intensive. To compensate, the remote sensing community has embraced unsupervised [13] and semi-supervised methods [14], [27] for the semantic segmentation of overhead HSI.
Deep unsupervised methods have shown some success at HSI semantic segmentation [18], [23]. These methods extract spatial-spectral features from an HSI using a deep network that is typically trained to reconstruct its input (subject to some constraints), which results in it learning how to extract good features for this reconstruction task. The most common approach is to use a stacked convolutional autoencoder. After training, the encoder portion of the network is used to extract features that are then passed to some sort of classifier (e.g., a support vector machine (SVM)). The majority of past work has trained their deep feature extraction networks directly on the image that will be classified. This is computationally expensive and the features learned may not generalize to other datasets. Instead, in self-taught feature learning, spatial-spectral features are directly learned from a large corpus of unlabeled HSI [13]. Self-taught learning methods acquire a more generalized set of spatial-spectral features, enabling state-of-the-art segmentation performance across multiple datasets without retraining the feature extraction model per dataset.
Current software tool boxes for remote sensing classification do not include deep learning methods for feature extraction or classification [3], [12]. In this letter, we introduce EarthMapper, a semantic segmentation pipeline for non-RGB remote sensing data that uses self-taught learning. EarthMapper includes undirected graphical models (UGMs), which are used as a post-processing technique to improve classification performance. One of the included UGMs is a fully-connected conditional random field, which has not been previously evaluated for HSI segmentation.
We demonstrate EarthMapper's utility on two standard HSI benchmarks: Indian Pines and Pavia University. EarthMapper has been publicly released on GitHub. It is modular, so it can be easily modified to incorporate future advancements.
II. RELATED WORK Unsupervised feature extraction has been remarkably successful for sensors that have little annotated data available and currently represents the state-of-the-art solution for classifying HSI. Researchers have developed a number of spatial-spectral feature extraction methods that learn the features directly from the data [18], [19], [21], [28]. Because they are learned from a single HSI, these features may perform poorly with other datasets. Learning features per image is computationally expensive for processing large quantities of HSI [23]. Self-taught feature learning was recently introduced as a method to build spatial-spectral feature extracting frameworks that generalize well across a variety of different scenes and sensors [13]. These frameworks 1) learn their feature representation from a large quantity of unlabeled HSI, 2) are used to extract discriminative features from the smaller labeled datasets, and 3) the features are fed to a standard classifier (e.g., SVM).
UGMs have been widely used in HSI classification to produce smooth, coherent segmentation maps by enforcing spatial contexts [11]. Their main advantage is that they can be easily combined with other classification approaches as a post-processing method that improves accuracy. UGMs have been used with classifiers trained directly on image pixels [24] and on image derived features [26]. Although researchers have started to combine deep learning with UGMs for HSI [1], [6], earlier works focused on supervised feature learning rather than unsupervised. Most UGM methods for HSI have employed a simple grid-structured pairwise model. While more advanced higher-order models, e.g., robust P n , have been investigated for HSI [29], these models require a large number of training pixels. A better approach is to use a fully-connected pairwise model [16], which uses a graph with edges between every node rather than only 4-connected nodes as in gridstructured models; however, until this work, they had not been tested for the semantic segmentation of HSI.
Popular hyperspectral remote sensing libraries, such as Spectral Python and Orfeo, and applications, such as ENVI and eCognition, contain only traditional methods for HSI segmentation, and do not carry deep feature learning or graphical models. To make these methods more easily accessible to the broad remote sensing community, we have created the EarthMapper Tool Box.

III. EARTHMAPPER TOOL BOX
EarthMapper is a modular framework for HSI semantic segmentation (see Fig. 1), which we have released for public use 1 . EarthMapper has a variety of pre-trained self-taught feature learning frameworks (Sec. III-A), which can be used to extract spatial-spectral features from annotated HSI data. These features are passed to a classifier that outputs a preliminary classification map, which can then be cleaned-up with one of two UGM post-processing methods (Sec. III-B).
MICA is a shallow feature extractor that convolves a bank of learned filters with an input HSI. It uses ICA to learn bar/edge, gradient, and blob detecting filters from image data. The learned filters resemble receptive field properties of neurons in primary visual cortex. In [13], MICA worked reasonably well at classifying benchmark HSI datasets, but its optimal receptive field size was dependent on the GSD of the dataset to be classified. SCAE [13] is a deep feature extracting framework. While autoencoders had been used in prior work with remote sensing data, SCAE was shown to work better on multiple datasets because it was trained on large quantities of publicly-available remote sensing imagery rather than on a single image. This allowed the learned features to generalize and transfer well between different datasets. The issue with SCAE is that backpropagation during training tends to do a better job at correcting errors closer to the input layers and neglects the errors in deeper layers [25]. SMCAE is an improved self-taught learning framework that uses a weighted sum of the reconstruction losses between each symmetrical encoderdecoder layer in the autoencoder, which allows backpropagation to correct errors in both shallow and deeper layers [14].

B. Undirected Graphical Models for Post-Processing
UGMs factorize the joint probability for a set of variables over the cliques of a undirected graph, such that each variable is conditionally independent of other variables given its neighbors [15]. They can be used to model the spatial dependencies between the labels of neighboring pixels in HSI. When applied to images, the label associated with every pixel in the image is represented by a graph node and edges represent the relationship between the labels. The expressiveness of the UGMs is controlled by the structure of the graph and energy functions defined over the graph's cliques. The most common type of UGM used for HSI classification is a pairwise model [11]. It defines the joint distribution of the pixel labels of an image as where y = [y 1 , ..., y N ] T is a vector containing labels of all N pixels in the image, E (y) is the total energy and Z = y exp (−E (y)) is the partition function. The total energy is equal to the sum of unary energies and pairwise energies defined over all nodes and edges of the graph, such that where V is the set of all nodes, D is the set of all edges, E i (y i ) is the unary energy function at i th pixel, and E ij (y i , y j ) is the pairwise energy function defined at the edge between i th and j th pixels. The unary energy captures the spectral information at the pixel location while the pairwise energy captures the spatial relationships between a pair of pixels. The UGMs whose energy functions are not dependent on input features are called the Markov random fields (MRF) and the ones whose energy functions are dependent on the input features are called the conditional random fields (CRF). The inference about the pixel labels, y, is generally performed by maximum a posteriori estimation, which is equivalent to minimizing the total energy, E (y), by an optimization algorithm such as GraphCuts [4]. There are usually parameters associated with the energy functions. These can be optimized using maximum likelihood estimation; however, when there are only a few parameters, they are usually tuned using gridsearch over the validation set.
EarthMapper includes grid-structured and fully-connected pairwise UGMs for post-processing. The grid-structured model is the most common type used in HSI semantic segmentation, and a detailed description can be found in [11]. The fullyconnected model, as described in this letter, has not been previously explored for HSI segmentation. A fully-connected pairwise model has an edge between every pair of nodes in the graph. It is more expressive than the grid-structured model as each node is not just connected to its 4-neighbors.
The unary energy function is given by the negative logarithm of the class probability predicted by the classifier, E i (y i ) = − log (P (y i |x i )). The pairwise energy function used depends on the spatial location of the pixels and is given by where, p i = [r i , c i ] and p j = [r j , c j ] are the spatial coordinates of i th and j th pixels with r i and c i being the row and the column numbers of the i th pixel and r j and c j being the row and the column numbers of the j th pixel. w 1 and θ γ are scalar parameters that are tuned using the validation set. This pairwise energy function promotes the labeling of pixels which are spatially close to one another in the same class. Since, the number of edges in a fully connected graph grows at O(n 2 ) with the number of nodes, it is inefficient to perform inference in such models using standard algorithms, especially when the image size is large. In this case, the mean-field approximate inference can be used to efficiently make predictions [16].

C. Existing HSI Analysis Tool Boxes
There are multiple remote sensing clustering and semantic segmentation software packages available, which are compared in Table I. The Geospatial Data Abstraction Library (GDAL) [10] is a popular open-source C/C++ library used to read, write, and process remote sensing imagery. ENVI is the leading remote sensing software package, and has several basic processing and semantic segmentation algorithms built-in, but it is not open-source. Spectral Python is an open-source Python library that has several image processing, clustering, and classification algorithms built-in [3]. The Orfeo Toolbox [12] and RSGISLib [5] are both open-source libraries that specialize in Geographic Object Base Image Analysis (GEOBIA) based classification. They both have some advanced feature extraction methods that can be used to improve performance. eCognition [9] and InterImage [2] are GUI based frameworks that also specialize in GEOBIA classification. InterImage is open-source; whereas, eCognition is proprietary. All of these packages do not include deep learning support, and many of them do not have any advanced spatial-spectral feature extraction methods built-in.

A. Stacked Multi-Loss Convolutional Autoencoder (SMCAE)
We use SMCAE to extract spatial-spectral features from the source image that will be classified. SMCAE is pre-trained on open-source imagery acquired by AVIRIS, GLiHT, and Hyperion sensors. These images are of a diverse set of scenes, and for AVIRIS, at multiple scales. Prior to passing the data to SMCAE, we re-sample the spectral bands of the input HSI to match that of the target image (i.e., HSI used to train feature extractor). We pass the re-sampled data through a feature extractor and then scale the output to zero-mean/unit-variance prior to passing it to the classifier. Specific implementation details can be found in [14].

B. Semi-Supervised Multilayer Perceptron (SS-MLP)
We use a SS-MLP neural network for classification. It takes a feature vector extracted from either the raw HSI cube or SMCAE and assigns each pixel a set of probabilities that it belongs to a given class. We can either 1) take the index (i.e., argmax) of the maximum probability to give us our class label or 2) pass those probabilities to the CRF to postprocess the classification map, which reduces salt-and-pepper classification errors. We cross-validate (see experiments for more details) to determine the optimal number of hidden layers (2-10), units per hidden layer (64-3000), and weight decay parameter for regularization. It is trained with a mini-batch size of 8, and we used Nadam [7] to optimize SS-MLP. We start with an initial learning rate of 0.002 and drop the learning rate by a factor of 10 when the validation loss plateaus. We stop training when the validation loss does not decrease over a period of 50 epochs. More details can be found in [14].

C. Fully-Connected CRF
The parameters of the post-processing CRF were tuned using grid-search over a range of [10 -3 , 10 3 ] and 30 iterations of mean field approximation were used for inference.

D. Data Description
We experiment on two widely used public datasets-Indian Indian and Pavia University. They are described in Table II.

E. Low-Shot Learning Experiment
In this experiment, we probe how each component of EarthMapper's pipeline affects performance with only a small quantity of training data (i.e., low-shot learning). Thirty independent random trials are performed to calculate the mean and the standard deviation of the performance metrics-overall accuracy (OA), average accuracy (AA), and kappa coefficient. In each experimental trial, a training set and a validation set consisting of 15 and 35 randomly selected pixels belonging to each class are created. The pipeline is trained on these pixels and is evaluated on the remaining pixels in the image. We also measure the performance of the pipeline with and without SMCAE feature extraction and CRF-based post-processing. Table III shows the classification results for our Earth-Mapper pipeline on the Indian Pines and Pavia University datasets. In every case, the additional post-processing with the CRF improves classification performance, but the performance increase is less pronounced when we have a better (i.e., deeper) feature representation of the data.
The combination of SMCAE feature extraction and CRF post-processing yields the best performance. The performance increase from using SMCAE features or a CRF is statistically significant (paired t-test, P < 0.01) compared to when neither is used. The performance increase from using SMCAE features and a CRF is statistically significant (paired t-test, P < 0.01) compared to using only one of the two.

F. GRSS Data and Evaluation Server Results
We also evaluated EarthMapper on the versions of the Indian Pines and Pavia University datasets provided by the IEEE GRSS Data and Algorithm Standard Evaluation (DASE) website. Although these versions have more training samples, they are harder because the training samples are spatially colocated rather than randomly sampled throughout the scene, so the model sees less within-class variability. We cross-validate SS-MLP using a 90%/10% training/validation (per-class) split. Table IV compares EarthMapper's results against the prior state-of-the art SuSA framework, which used a mean-pooled SMCAE feature response and the SS-MLP classifier [14]. The current EarthMapper configuration does not use a meanpooling operation and uses a CRF for post-processing, since mean-pooling can cross-contaminate different object classes for a given pixel. The CRF worked very well with highresolution HSI (i.e., Pavia University), but there was still a small performance increase for Indian Pines as well. V. CONCLUSION We have presented EarthMapper, a modular framework for the semantic segmentation of remote sensing imagery. We demonstrate that incorporating deep, spatial-spectral feature extraction and UGM post-processing in the pipleline improves classification performance. EarthMapper yielded state-of-theart performance for the two HSI benchmarks hosted on the GRSS DASE website. EarthMapper is publicly available and can easily be adapted to incorporate new components.