{'fraction_non_alphanumeric': 0.03319862826230458, 'fraction_numerical': 0.010062629807924522, 'mean_word_length': 4.982086102282577, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.

traditionally used as priors for the estimates. This particularity suggests that the so-far dogma of expert-free fully automated deep learning should be questioned for remote sensing and physical models should be re-introduced into the concept, as, for example, in the concept of emulators [12].
Remote sensing scientists have exploited the power of deep learning to tackle these different challenges and started a new wave of promising research. In this paper, we review these advances.
After the introductory Section II detailing deep learning models (with emphasis put on convolutional neural networks), we enter sections dedicated to advances in hyperspectral image analysis (Section III-A), synthetic aperture radar (Section III-B), very high resolution (Section III-C, data fusion (Section III-D), and 3D reconstruction (Section III-E). Section IV then provides the tools of the trade for scientists willing to explore deep learning in their research, including open codes and data repositories. Section V concludes the paper by giving an overview of the challenges ahead.

II. FROM PERCEPTRON TO DEEP LEARNING
Perceptron is the basic of the earliest NNs [13]. It is a bio-inspired model for binary classification that aims to mathematically formalize how a biological neuron works. In contrast, deep learning has provided more sophisticated methodologies to train deep NN architectures. In this section, we recall the classic deep learning architectures used in visual data processing.
A. Autoencoder models 1) Autoencoder and Stacked Autoencoder (SAE): An autoencoder [14] takes an input x ∈ R D and, first, maps it to a latent representation h ∈ R M via a nonlinear mapping: where Θ is a weight matrix to be estimated during training, β is a bias vector, and f stands for a nonlinear function, such as the logistic sigmoid function or a hyperbolic tangent function. The encoded feature representation h is then used to reconstruct the input x by a reverse mapping leading to the reconstructed input y: where Θ is usually constrained to be the form of Θ = Θ T , i.e., the same weight is used for encoding the input and decoding the latent representation. The reconstruction error is defined as the Euclidian distance between x and y that is constrained to approximate the input data x (i.e., making x − y 2 2 → 0). The parameters of the autoencoder are generally optimized by stochastic gradient descent (SGD).
An SAE is a neural network consisting of multiple layers of autoencoders in which the outputs of each layer are wired to the inputs of the following one.
2) Sparse Autoencoder: The conventional autoencoder relies on the dimension of the latent representation h being smaller than that of input x, i.e., M < D, which means that it tends to learn a low-dimensional, compressed representation. However, when M > D, one can still discover interesting structures by enforcing a sparsity constraint on the hidden units. Formally, given a set of unlabeled data X = {x 1 , x 2 , · · · , x N }, training a sparse autoencoder [15] boils down to finding the optimal parameters by minimizing the following loss function: where J(x i , y i ; Θ, β) is an average sum-of-squares error term, which represents the reconstruction error between the input x i and its reconstruction y i . KL(ρ ρ j ) is the Kullback-Leibler (KL) divergence between a Bernoulli random variable with mean ρ and a Bernoulli random variable with meanρ j . KL-divergence is a standard function for measuring how similar two distributions are: it has symmetric connections between these two layers. No connecting exists within the hidden layer or the input layer. The energy function of an RBM can be defined as follows: where W , c, and b are learnable weights. Here, the input x is also named as the visible random variable, which is denoted as v in [16]. The joint probability distribution of the RBM is defined as: where Z is a normalization constant. The form of the RBM makes the conditional probability distribution computationally feasible, when x or h are fixed.
The feature representation ability of a single RBM is limited. However, its real power emerges when a couple of RBMs are stacked, forming a DBN [16]. Hinton et al. [16] proposed a greedy approach that trains RBM in each layer to efficiently train the whole DBN.

B. Convolutional neural networks (CNNs).
Unsupervised deep neural networks have been under the spot in the recent year. The leading model is the convolutional neural network (CNN), which learns the filters performing convolutions in the image domain. Here, we briefly review some successful CNN architectures proposed in computer vision in the recent years. For a comprehensive introduction on CNNs, we invite the reader to consider the excellent book by Goodfellow and colleagues [17]. AlexNet (cf. Fig. 2) scaled the insights of LeNet [18] into a deeper and much larger network that could be used to learn the appearance of more numerous and more complicated objects.
The contributions of AlexNet are as follows: -Using rectified linear units (ReLU) as nonlinearity functions that are capable of decreasing training time, as ReLU is several times faster than the conventional hyperbolic tangent function.
-Implementing dropout layers in order to avoid the problem of overfitting.
-Using data augmentation techniques to artificially increase the size of the training set (and see a more diverse set of situations). From this, the training patches are translated and reflected on the horizontal and vertical axes.
One of the keys of the success of AlexNet is that the model was trained on GPUs. Since GPUs can offer a much larger number of cores than CPUs, it allows much faster training, which in turn allows one to use larger datasets and bigger images.
2) VGG Net: The design philosophy of the VGG Nets [3] is simplicity and depth. In 2014, Simonyan and Zisserman created VGG Nets that strictly makes use of 3 × 3 filters with stride and padding of 1, along with 2 × 2 max-pooling layers with stride 2. The main points of VGG Nets are that they: -Use filters with small receptive field of 3 × 3, rather than using larger ones (5 × 5 or 7 × 7, as in Alexnet).
-Have the same feature map size and number of filters in each convolutional layer of the same block.
-Increase the size of the features in the deeper layers, roughly doubling after each maxpooling layer.
-Use scale jittering as one data augmentation technique during training.
VGG is one of the most influential CNN models, as it reinforces the notion that CNNs with deeper architectures can promote hierarchical feature representations of visual data, which in turn improves the classification accuracy. A drawback is that, to train such a model from scratch, one would need large computational power and a very large labeled training set.

3) ResNet:
He et al. [4] pushed the idea of very deep networks even further by proposing the 152-layers ResNet -which won ILSVRC 2015 with an error rate of 3.6% and set new records in classification, detection, and localization through a single network architecture. In [4], authors provide an in-depth analysis about the degradation problem, i.e., simply increasing the number of layers in plain networks results in higher training and test errors, and claim that it is easier to optimize the residual mapping in the ResNet than to optimize the original, unreferenced mapping in the conventional CNNs. The core idea of ResNet is to add shortcut connections that by-pass two or more stacked convolutional layers by performing identity mapping, which are then added together with the output of stacked convolutions.

III. REMOTE SENSING MEETS DEEP LEARNING
Deep learning is taking off in remote sensing, as shown in Fig. 4, which summarizes the number of papers on the topic since 2014. Their exponential increase confirms the rapid surge of interest in deep learning for remote sensing. In this section, we focus on a variety of remote sensing applications that are achieved by deep learning and provide an in-depth investigation from the perspectives of hyperspectral image analysis, interpretation of SAR images, interpretation of high-resolution satellite images, multimodal data fusion, and 3D reconstruction.

A. Hyperspectral Image Analysis
Hyperspectral sensors are characterized by hundreds of narrow spectral bands. This very high spectral resolution enables us to identify the materials contained in the pixel via spectroscopic analysis. Analysis of hyperspectral data is of high importance in many practical applications, such as land cover/use classification or change and object detection. Also, because high quality hyperspectral satellite data is becoming available, e.g., via the launch of EnMAP, planned in Inspired by the success of deep learning in computer vision, preliminary studies have been carried out on deep learning in hyperspectral data analysis, which brings new momentum into this field. In this section, we would like to review two application cases, namely, land cover/use classification (III-A1) and anomaly detection (III-A2).  [19] for spectral-spatial hyperspectral image classification.
1) Hyperspectral Image Classification: Supervised classification is probably the most active research area in hyperspectral data analysis. There is a vast literature on this topic using the conventional supervised machine learning models, such as decision trees, random forests, and support vector machines (SVMs) [20]. With the investigation of hyperspectral image classification [21], a major finding was that various atmospheric scattering conditions, complicated light scattering mechanisms, inter-class similarity, and intra-class variability result in the hyperspectral imaging procedure being inherently nonlinear. It is believed that, in comparison to the aforementioned "shallow" models, deep learning architectures are able to extract high-level, hierarchical, and abstract features, which are generally more robust to the nonlinear input data.
i) Autoencoders for hyperspectral data classification: A first attempt in this direction can be found in [22], where authors make use of a stacked autoencoder to extract hierarchical features in the spectral domain. Subsequently, in [23], authors employ DBM. Similarly, Tao et al. [24] use sparse stacked autoencoder to learn an effective feature representation from unlabeled data, and then the learned features are fed into a linear SVM for hyperspectral data classification.
ii) Supervised CNNs: In [25], authors train a simple 1D CNN that contains five layers, namely, an input layer, a convolutional layer, a max pooling layer, a fully connected layer, and an output layer -and directly classify the hyperspectral images in spectral domain.
Makantasis et al. [26] exploited a 2D CNN to encode spectral and spatial information, followed by a multi-layer perceptron performing the actual classification. In [27], authors attempt to carry out classification of crop types using 1D CNN and 2D CNN. They concluded that the 2D CNNs can outperform the 1D CNNs, but some small objects in the final classification map provided by 2D CNN are smoothed and misclassified. To avoid overfitting, Zhao and Du [28] propose a spectral-spatial feature-based classification framework, which jointly makes use of a local discriminant embedding-based dimension reduction algorithm and a 2D CNN. In [21], authors propose a self-improving CNN model, which combines a 2D CNN with a fractional order Darwinian particle swarm optimization algorithm to iteratively select the most informative bands that are suitable for training the designed CNN. Santara et al. [29] propose an end-to-end band-adaptive spectral-spatial feature learning network to address the problems of the curse of dimensionality. In [30], to allow CNN appropriately trained using limited labeled data, authors present a novel pixel-pair CNN to significantly augment the number of training samples.
Following recent vision developments in 3D CNNs [31], in which the third dimension usually refers to the time axis, such architecture has also been employed in hyperspectral classification.
In other words, in 3D CNN, convolution operations are performed spatial-spectrally while in 2D CNNs they are done only spatially. Compared to 1D and 2D CNNs, 3D CNNs can model spectral information better owing to 3D convolution operations. Authors in [19] introduced a supervised, 2 regularized 3D CNN-based model (see Fig. 5). While authors of [32] followed a similar idea for spatial-spectral classification. iii) Unsupervised Deep Learning: To be less dependent on the existence of large annotated collections of labeled data, unsupervised feature extraction remains of great interest. Authors of [35] propose an unsupervised convolutional network for learning spectral-spatial features using sparse learning to estimate the network weights in a greedy layer-wise fashion instead of end-to-end learning. Mou et al. [33,34] propose a network architecture called fully Residual Conv-Deconv network for unsupervised spectral-spatial feature learning of hyperspectral images.
They report an extensive study of the filters learned (cf. Fig. 6). iv) RNN for Hyperspectral Image Classification: In [36], authors propose a RNN model with a new activation function and modified gated recurrent unit for hyperspectral image classification, which can effectively analyze hyperspectral pixels as sequential data and then determine information categories via network reasoning (cf. Fig. 7). 2) Anomaly Detection: In a hyperspectral image, the pixels whose spectral signatures are significantly different from the global background pixels are considered anomalies. Since the prior knowledge of the anomalous spectrum is difficult to obtain in practice, anomaly detection is usually solved by background modeling or statistical characterization for hyperspectral data.
So far, the only mark addressing this problem via deep learning can be found in [37]. Li et al. [37] propose an anomaly detection framework, in which a multi-layer CNN is trained by using the differences in gray values between neighboring pixel pairs in the reference image as input data. Then, in the test phase, anomalies are detected by evaluating differences between neighboring pixel pairs using the trained CNN.
In summary, deep learning has been widely applied to the multi/hyper-spectral image classification, and some promising results have been achieved. In contrast, for other hyperspectral data analysis tasks, such as change and anomaly detections, deep learning has just made its mark [37,38]. Some potential problems to be further explored include nonlinear spectral unmixing, hyperspectral image enhancement, hyperspectral time-series analysis, etc.

B. Interpretation of SAR Images
Over the past few years, there have been many publications in deep learning-related studies for SAR image analysis. Among these studies, deep learning techniques have been mostly applied in typical applications, including automatic target recognition (ATR), terrain surface classification, and parameter inversion. This section reviews some of the relevant studies in this area.

1) Automatic Target Recognition: SAR ATR is an important application, in particular for
military surveillance [39]. A standard architecture for efficient ATR consists of three stages: detection, discrimination, and classification. Each stage tends to perform a much complicated and refined processing than its predecessor, and selects the candidate objects for the next stage processing. However, all three stages can be treated as a classification problem and, for this reason, deep learning has found its marks.
Chen et al. [40] introduce CNN into SAR ATR and tested on the standard ATR dataset MSTAR [41]. The major issue is found to be the lack of sufficient training samples as compared to optical images. It might cause severe overfitting and, therefore, greatly limits the capability of generalizing the model. Data augmentation is employed to counteract overfitting. Chen et al. [42] propose to further remove all fully-connected layers from conventional CNNs which are accountable for most trainable parameters. The final performance is demonstrated superior on the MSTAR dataset (i.e., a state-of-the-art accuracy of 99.1% on standard operating condition (SOC)). Extensive experiments are conducted to test the generalization capability of the socalled AConvNets and they are found to be quite robust in several extended operating conditions (EOC). The removal of the fully-connected layers, which is originally designed to be a trainable classifier, might be justifiable in this case, because the limited number of target types can be seen as the feature templates that the AConvNets is extracting.
Among these studies, the one common finding is that data augmentation is necessary and the most critical step for SAR ATR using CNNs. Various augmentation strategies are proposed, including translation, rotation, interpolation, etc.
Cui et al. [47] introduce DBN to SAR ATR, where stacked RBM are used to extract features and then fed to trainable classifier.
Wagner [48] proposes to use CNN to first extract feature vectors and then fed them to a SVM for classification. The CNN is trained with a fully-connected layer but only the previous activations are used. A systematic data augmentation approach is employed, which includes elastic distortions and affine transformations. It is intended to mimic typical imaging errors, such as a changing range (which is scale dependent on the depression angle) or an incorrectly estimated aspect angle.
More studies applying CNNs to the ART problem are found. Bentes et al. [49] apply CNN to ship-iceberg discrimination and tested on TerraSAR-X StripMap images. Schwegmann et al. [50] apply a specific type of deep neural networks, the highway networks, to ship discrimination in SAR imagery and achieved promising results. Ødegaard et al. [51] apply CNN to detect ships in harbor background in SAR images. To address the issue of lack of training samples, they employed a simulation software to generate simulated data for training. Song et al. [52] follow this idea and introduce a deep generative neural network for SAR ATR. A generative deconvolutional NN is first trained to generate simulated SAR image from a given target label, during which a feature space is constructed in the intermediate layer. A CNN is then trained to map an input SAR image to the feature space. The goal is to develop an extended ATR system which is capable of interpreting a previously unseen target in the context of all known targets.  2) Terrain surface classification: When terrain surface classification uses SAR, in particular polarimetric SAR (PolSAR), data is another important application in radar remote sensing. This is very similar to the task of image segmentation in computer vision. Conventional approaches are mostly based on pixel-wise polarimetric target decomposition parameters [54]. They hardly considered the spatial patterns, which convey rich information in high-resolution SAR images [55].
Deep learning provides such a tool for automatically extract features that represent spatial patterns as well as polarimetric characteristics.
One large stream of studies will employ at least one type of unsupervised generative graphical models, such as DBN, SAE or RBM.
Xie et al. [56] first introduce multi-layer feature learning for PolSAR classification, where SAE is employed to extract useful features from a channel PolSAR image.
Geng et al. [57] propose a deep convolutional autoencoder (DCAE) to extract features and conduct classification automatically. The DCAE consists of a hand-crafted first layer of convolution, which contains kernels, such as gray-level cooccurrence matrix and Gabor filters, and a hand-crafted second layer of scale transformation, which integrates correlated neighbor pixels.
The rest layers are trained SAE. This approach is tested on high-resolution single-polarization TerraSAR-X images. Geng et al. [58] later propose a similar framework, called deep supervised and contractive neural network (DSCNN), for SAR image classification, which further includes the histogram of oriented gradient (HOG) descriptors as hand-crafted kernels. The trainable AE layers employ a supervised penalty, which captures the relevant information between features and labels, and a contractive restriction, which enhances local invariance. An interesting finding of Geng et al. [58] is that speckle reduction yields the worse performance and the authors suspect that speckle reduction might smooth out some useful information. The other stream of studies involves CNNs. Zhou et al. [65] apply CNN to PolSAR image classification, where covariance matrix is extracted as 6-real-channel data input. Duan et al. [66] propose to replace the conventional-pooling layer in CNN by a wavelet-constrained pooling layer.
The so-called convolutional-wavelet neural network is then used in conjunction with superpixels and Markov Random Field (MRF) to produce the final segmentation map.
Zhang et al. [53] propose a complex-valued (CV) CNN (cf. Fig. 8) specifically designed to process complex values in PolSAR data, i.e., the off-diagonal elements of coherency or covariance matrix. CV-CNN not only takes complex numbers as input but also employs complex weights and complex operations throughout different layers. A complex-valued backpropagation algorithm is also developed to train it. Fig. 9 shows an example of PolSAR classification using CV-CNN. [67] apply CNN to estimate ice concentration using SAR image during melt season. The labels are produced by visual interpretation by ice experts. It is tested on dual-pol RadarSat-2 data. Since the problem considered is regression of a continuous value, the loss function is selected as mean squared error. The final results suggest that CNN can produce a more detailed result then operational products.

3) Parameter inversion: Authors in
C. Interpretation of High-resolution Satellite Images 1) Scene Classification: Scene classification, which aims to automatically assign a semantic label to each scene image, has been an active research topic in the field of high-resolution satellite images in the past decades [68][69][70][71][72][73][74]. As a key problem in the interpretation of satellite images, it has widespread applications, including object detection [75,76], change detection [77], urban planning, land resource management, etc. However, due to the high spatial resolutions, different scene images may contain the same kinds of objects or share similar spatial arrangement. For example, both residential area and commercial area may contain buildings, roads and trees, but they are two different scene types. Therefore, the great variations in the spatial arrangements and structural patterns make scene classification a considerably challenging task.
Generally, scene classification can be divided into two steps: feature extraction and classification. With the growing number of images, to train a complicated nonlinear classifier is time-consuming. Hence, to extract a holistic and discriminative feature representation is the most significant part for scene classification. Traditional approaches are mostly based on the Bag-of-Visual-Words (BoVW) model [78,79], but their potential for improvement was limited by the ability of experts to design the feature extractor and the expressive power encoded.
The deep arhitectures discussed in Section II-B have been applied to the problem of scene classification of high-resolution satellite images and led to state-of-the-art performance [71,74,[80][81][82][83][84][85][86][87]. As deep learning is a multi-layer feature learning architecture, it can learn more abstract and discriminative semantic features as the depth grows and achieve far better classification performance compared with the mid-level approaches. In this section, we summarize the existing deep learning-based methods into the following three categories: -Using pre-trained networks. The pre-trained deep CNN on the natural image dataset, e.g., OverFeat [88], GoogLeNet [89], etc., have led to impressive results on scene classification of high-resolution satellite images by directly extracting the features from the intermediate layers to form global feature representations [81][82][83]87]. For example, [74,81] and [82] directly use the features from the fully-connected layers as the input of the classifier, while [83] takes the CNN as local feature extractor and combine it with feature coding techniques, such as BoVW [78] and Vector of Locally Aggregated Descriptors (VLAD) to generate the final image representation.
-Making a pre-trained model adapt to the specific conditions observed in a dataset under study, one can decide to fine-tune it on a smaller labeled dataset of satellite images. For example, [82] and [86] fine-tune some high-level layers of the GoogLeNet [89], etc., using the UC-Merced dataset [90] (see Section IV-C), obtaining better results than directly using only the pre-trained CNNs. This can be explained because the features learned are more oriented to the satellite images after fine-tuning, which can help to exploit the intrinsic characteristic of satellite images. Nonetheless, compared with the natural image dataset that consists of more than ten millions of samples, the scales of public satellite image datasets (i.e., UC-Merced dataset [90], RSSCN7 dataset [80], WHU-RS19 [91] are fairly smallfor example, up to several thousands -for which we cannot fine-tune the whole CNNs to make them more adaptive to satellite images.
-Training new networks. In addition to the above two ways to use deep learning methods for classifying satellite images, some researchers train the network using satellite images from scratch. For example, [82] and [86] train the networks by only using the existing satellite image dataset, which suffers a drop in classification accuracy compared with using the pre-trained networks as global feature extractors or fine-tuning the pre-trained networks.
The reason may lies in the fact that the large-scale networks usually contain millions of parameters to be learned. Thus, training them using the small-scale satellite image datasets will easily cause overfitting and local minimum problems. Consequently, some construct a new smaller network and train it from scratch using satellite images to better fit the satellite images [80,84,85,92]. However, such small-scale networks are often easily oriented to the training images, and the generalization ability decreases. For each satellite dataset, the network need to be retrained.
2) Object Detection: Object detection is another important task in the interpretation of highresolution satellite images [93]: one wishes to localize one or more specific ground objects of interest (such as building, vehicle, aircraft, etc.) within a satellite image and predict their corresponding categories as shown in Fig. 10. Due to the powerful ability of learning high-level (more abstract and semantically meaningful) feature representations, the deep CNNs are being explored in object detection systems in opposition to the more traditional proposals methods followed by a classifier based on handcrafted features [94,95]. Here, we review most existing works using CNNs for both specific and generic object detection.
Jin et al. [96] propose a vector-guided vehicle detection approach for IKONOS satellite imagery using a morphological shared-weight neural network, which learns the implicit vehicle model and incorporates both spatial and spectral characteristics, and classifies pixels into vehicles and non-vehicles. To address the problem of large-scale variance of objects, Chen et al. [97] propose a hybrid deep CNN model for vehicle detection in satellite images, which divides all feature maps of the last convolutional and max-pooling layer of CNN into multiple blocks of variable receptive field size or pooling size, to extract multi-scale features. Jiang et al. [98] propose a CNN-based vehicle detection approach, where a graph-based superpixel segmentation is used to extract image patches and a CNN model is trained to predict whether a patch contains a vehicle.
A few detection methods transfer the pre-trained CNNs for object detection. Zhou et al. [99] propose a weakly supervised learning framework to train an object detector, where a pre-trained CNN model is transferred to extract high-level features of objects and the negative bootstrapping scheme is incorporated into the detector training process to provide faster convergence of the detector. Zhang et al. [100] propose a hierarchical oil tank detector, which combines deep Zhang et al. [105] propose a weakly supervised learning approach using coupled CNNs for aircraft detection. The authors employ an iterative weakly supervised framework that simply requires image-level training data to automatically mine and augment the training data set from the original image, which can dramatically decrease human labor. A coupled CNN model, which is composed of a candidate region proposal network, and a localization network are developed to generate region proposals and locate the aircrafts simultaneously, which is suitable and effective for large-scale high-resolution satellite images.
For enhancing the performance of generic object detection, Cheng et al. [76] propose an effective approach to learn a rotation-invariant CNN (RICNN) to improve invariance to object rotation. In their paper, they add a new rotation-invariant layer to the off-the-shelf AlexNet 3) Image Retrieval: Remote sensing image retrieval aims at retrieving images with a similar visual content, with respect to a query image from a database [109]. A common image retrieval system needs to compute image similarity based on image feature representations, and thus the performance of a retrieval depends on the descriptive capability of image features to a large degree. Building image representation via feature coding methods (e.g., BoVW and VLAD) using low-level hand-crafted features has been proven to be very effective in aerial image retrieval [109,110]. Nevertheless, the discriminative ability of low-level features is very limited, and thus it is difficult to achieve substantial performance gain. Recently, a few works have investigated extracting deep feature representations from CNNs. Napoletano [111] extracts deep features from the fully-connected layers of the pre-trained CNN models, and the deep features prove to perform better than low-level features regardless of the retrieval system. Zhou et al. [112] propose a CNN architecture followed by a three-layer perceptron, which is trained on a large remote sensing dataset and able to achieve remarkable performance even with low dimensional deep features. Jiang et al. [113] present a sketch-based satellite image retrieval method by learning deep cross-domain features, which enables us to retrieve satellite images with hand-free sketches only.
Although there is still a lack of sufficient study of exploiting deep learning approaches for remote sensing image retrieval at present, in consideration of the great potential in learning high-level features of deep learning methods, we believe that more deep learning based image retrieval systems will be developed in the near future. It is also worth noticing that how to integrate the feedback from users into the deep learning retrieval scheme.

D. Multimodal Data Fusion
Data fusion is one of the fast-moving areas of remote sensing [114][115][116]: due to the recent increases in availability of sensor data, the perspectives of using big and heterogeneous data to study environmental processes have become more tangible.

1) Pansharpening and Super-Resolution:
Pansharpening is the task of improving the spatial resolution of multispectral data by fusing it with data characterized by sharper spatial information.
It is a special instance of the more general problem of super-resolution. Traditionally, the field was dominated by works fusing multispectral data with panchromatic bands [117], but more recently it has been extended to thermal [118] or hyperspectral images [119]. Most techniques rely either on projective methods, sparse models, or pyramidal decompositions. Using deep neural networks for pansharpening multispectral images is certainly an interesting concept, since most image acquired by satellite as the WorldView series or Landsat come with a panchromatic band.
In this respect, training data are abundant, which is in line with the requirements of modern CNNs.
A first attempt in this direction can be found in [120], where authors use a shallow network to upsample the intensity component obtained after the IHS of color images (RGB). Once the multispectral bands have been upsampled with the CNN, a traditional Gram-Schmidt transform is used to perform the pansharpening. The authors use a dataset of QuickBird images for their analysis. Even though this is interesting, in this paper, authors simply replace one operation (the nearest neighbours or bicubic convolution) with a CNN.
In [121], authors propose using a CNN to learn the pansharpening transform end-to-end, i.e., letting the CNN perform the whole pansharpening process. In their CNN, they stack upsampled spectral bands with the panchromatic band and then learn, for each patch, the high resolution values of the central pixel.
In [122] authors use a super-resolution CNN trained on natural images [123] as a pre-trained model and fine-tune it on a dataset of hyperspectral images. By doing so, they make an attempt at transfer leaning [124] between the domains of color (three bands, large bandwidths) and hyperspectral images (many bands, narrow bandwidths). Fine-tuning existing architectures, which have been trained on massive datasets with very big models, is often a relevant solution, since one makes use of discriminative strong features and only injects task-specific knowledge.
In [125], authors learn an upsampling of the panchromatic band via a stack of autoencoders: the model is trained to predict the full-resolution panchromatic image, from a downsampled version of itself (at the resolution of the multispectral bands). Once the model is trained, the multispectral bands are fed into the model one by one, therefore being upsampled using the data relationships learned from the panchromatic images.
2) Feature and Decision-level Fusion for image classification: Most current remote sensing literature, dealing with deep neural networks, studies the problem of image classification, i.e. the task of assigning each pixel in the image to a given semantic class (land use, land cover, damage level, etc.). In the following, we review recent approaches dealing with image classification problems, mostly at very-high resolution, using two strategies: feature-level fusion and decisionlevel fusion. In the last part of this section, we will also review works using different data sources to tackle separate, but related predictive tasks, or multi-task problems.
i) Feature-level fusion: using multiple sources simultaneously in a network. Other strategies to spatial upsampling have been proposed in the recent literature, including the direct use of upsampled activation maps as features to train the final classifier [128]. In [129], authors studied the possibility of visualizing uncertainty of predictions (applying the model of [130]): they stacked CIR, DSM and normalized DSM data as inputs to the CNN. Fig. 11. Deconvolution network proposed in [92]. The yellow and green part correspond to a fully convolutional network with a 9 × 9 pixels bottleneck; then a deconvolutional block (purple) leads to predictions of the same size as the input image (in [92], Besides dense predictions, other strategies have been proposed to include spatial information in deep neural networks: for example, authors of [58] extract different types of spatial filters and stack them in a single tensor; the tensor is then used to learn a supervised stack of autoencoders. They apply their models on the classification of SAR images, so the fusion here is to be considered between different types of spatial information. The neural network is then followed by a conditional random field, to decrease the effect of speckle noise inherent in SAR images. In [131], authors learn combinations of spatial filters extracted from hyperspectral image bands and DSMs: even if the model is not a traditional deep network, it learns a sequence of recombinations of filters, extracting therefore higher level information in an automatic way as deep neural networks do. I.e., it learns the right filters parameters (along with their combinations) instead of learning the filters coefficients themselves.
Data fusion is also a key component in change detection, where one would like to extract joint features from a bi-temporal sequence. The aim is to learn a joint representation, where both (co-registered) images can be compared: this area is especially interesting when methods can align data from multiple sensors (see [132,133]). Three studies employ deep learning to this end: - [134], where authors learn a joint representation of two images with Deep Belief Networks.
Feature vectors issued from the two image acquisitions are stacked and used to learn a representation, where changes stand out more clearly. Using such representation, changes are more easily detected by image differencing. Their approach is applied on optical images from the Chinese GaoFen-1 satellite and WorldView-2.
- [135], where the joint representation is learned via a stack of autoencoders using the single temporal acquisitions at each end of the encoder-decoder system. By doing so, they learn a representation useful for change detection at the bottleneck of the system (i.e. in the middle). The authors show the versatility of their approach by applying it to several datasets, including pairs of optical and SAR images and an example performing change detection between optical and SAR images.
-More recent work, instead, addresses the transferability of deep learning for change detection, while analyzing data of long time series for large-scale problems. For example, in [38], authors make use of an end-to-end RNN to solve the multi/hyper-spectral change detection task, since RNN is well known to be good at processing sequential data. In their framework, in [133] consider learning an ensemble of CNNs and then averaging their predictions: their proposed pipeline has two main streams, one processing the CIR data and another processing the DSM. They train several CNNs, using them as inputs the activation maps of each layer of the main model, as well as one fusing the CIR and DEM main streams as in [137]. By doing so, they obtain a series of land cover maps to nourish the ensemble, which improves performances by considering classifiers issued form different data sources and levels of abstraction. Compared to the previous one in this section, this model has the advantage of being entirely learned in an end-to-end fashion, but also incurs in extreme computational load and a complex architecture involving many skip connections and fusion layers.
-Decision fusion learned in the network: an alternative to an ad-hoc fusion (multiplication or averaging of the posterior maps), one could learn the optimal fusion. In [142], authors perform the fusion between two maps obtained by pre-trained models by learning a fusion network based on residual learning [4] logic: in their architecture, they learn how to correct the average fusion result by learning extra coefficients favouring one or the other map.
Their results show that such a learned fusion outperforms the, yet more intuitive, simple averaging of the posterior probabilities.
iii) Using CNN for solving different tasks. So far, only literature dealing with a single task (image classification) has been reviewed. But, besides it, one might want to predict other quantities or use the image classification results to improve the quality of related tasks as image registration. In this case, predicting different outputs jointly allows one to tighten feature representations with different meanings, therefore leading to another type of data fusion with respect to the one seen until now (that was mainly concerned by fusing different inputs).
Summarizing, we will talk about fusing outputs. Below, we discuss three examples from recent literature, where alternative tasks are learned together with image classification.
were producing and fusing an ensemble of land cover maps. In [143], that work was extended by including the idea of predicting object boundaries jointly with the land cover. The intuition behind this is that predicting boundaries helps to achieve sharper (and therefore more desirable) classification maps. In [143], authors learn a CNN to separately output edges likelihoods at multiple scales from CIR and height data. Then, the boundaries detected with each source are added as an extra channel to each source and an image classification network, similar to the one in [133] is trained. The predictions of such model are very accurate but the computational load involved becomes very high: authors report models involving up to 800 millions of parameters to be learned.
-Depth. Most approaches discussed above include the DSM as an input to the network.
But, often, such information is not available (and it is certainly not when working on historical data). A system predicting a height map form image data would indeed be very valuable, since it could generate reasonably accurate DSM for color image acquisitions.
This is known in vision as the problem of estimating depth maps [144] and has been considered in [145] for monocular subdecimeter images. In their models, authors use a joint loss function, which is a linear combination of a dense image classification loss and a regression loss minimizing DSM predictions errors. The model can be trained by traditional back-propagation by alternating over the two losses. Note that, in this case, the DSM is used as an output (contrarily with most approaches above) and is, therefore, not needed at prediction time.
-Registration. When performing change detection, one expects perfect co-registration of the sources. But, especially when working at very high resolution, this is difficulty to achieve.
For instance, think of urban areas, where buildings are tilted by the viewing angle. In their entry to the IEEE GRSS data fusion contest 2016 4 , authors of [146] learn jointly the registration between the images, the land use classification of each input and a change detection map with a conditional random field model. The land use classifier used is a twolayers CNN trained from scratch and the model is applied successfully either to pairs of VHR images or to datasets composed of VHR images and video frames from the International Space Station.

3) Fusing Heterogeneous Sources: Data fusion is not only about fusing image data with
the same viewpoint. Multimodal remote sensing data exceed these restrictive boundaries and approaches to tackle new, exciting problems with remote sensing are appearing in the literature.
A strong example is the joint use of ground-based and aerial images [147]: services, such as Google Street View and Flickr, provide endless sources of ground images describing cities from the human perspective. These data can be fused to overhead views to provide better object detection, localization, or recreation of virtual environments. In the following, we review a series of applications in this direction.
In [148], authors consider the task of detecting and classifying urban trees. To this end, they exploit Faster R-CNN [149], an object detector developed for general purpose object detection in vision. After detecting the trees in the aerial view and the Google Street View panoramas, they minimize a energy function to detect trees jointly in all sources, but also avoiding multiple and illogical detections (e.g. trees in the middle of a street lane). They use a trees inventory from the city of Pasadena to validate their detection model and train a fine grained CNN based on GoogleNet [89], to perform fine-grained classification of the trees species on the detections, with impressive results. Authors of [150] take advantage of an approach that combines CNN and MRF and can estimate fine grained categories (e.g., road, sidewalk, background, building and parking) by doing joint inference over both monocular aerial imagery and ground images taken from a stereo camera on top of a car.
Many papers in geospatial computer vision work towards cross-view image localization: when presented to a ground picture, it would be relevant to be able to locate it in space. This is very important for photo-sharing platforms, for which only a fraction of the uploaded photos comes with geo-location. Authors of [151,152] worked towards this aim, by training a cross-view Siamese network [153] to match ground images and aerial views. Siamese networks have also been recently applied to detect changes between matched ground panoramas and aerial images in [147]. Back to more traditional CNNs, authors of [154,155] study the specificity of images to refer to a given city: they study how much images of Charleston resemble those from San Francisco, and the other way around, by using the fully connected layers of Places CNN [156] and then translating this into differences in the respective aerial images. Moreover, in [154] they also present applications on image localization similar to the above, where the likelihood of localization is given by a similarity score between the features of the fully connected layer of Places CNN.

E. 3D Reconstruction
3D data generation from image data plays an important role for remote sensing. 3D data (e.g., in form of a DSM or DTM) is a basic data layer for further processing or analysis steps. The processing of image data from airborne sensors or satellite systems is a long-standing tradition.
In a typical 3D data generation workflow, two main steps must be performed. First, camera orientation, which means computing the position and orientation of the cameras that produced the image. This can be computed from the image data itself, by identifying and matching tiepoints and then performing camera-resectioning. The second step is triangulation with calculates the 3D measurements for point correspondences that get established through stereo matching.
The fundamental algorithms in this pipeline are of geometrical nature, the implementations are based on analytical calculations. So far, machine learning did not play a big role in this pipeline.
However, there are steps in this pipeline which recently could be improved significantly by using machine learning techniques.
1) Tie points identification and matching: For instance, during camera orientation, the identification and matching of tie-points has long been done manually by operators. The task of the operator was to identify corresponding locations in two or more images. This process has been automated by clever engineering of computer algorithms to detect point locations in images that will be easy to re-detect in other images (e.g. corners) as well as algorithms for computing similarities of image patches for finding a tie-point correspondence. Many different detectors and similarity measures have been engineered so far, famous examples are the SIFT [157] or SURF [158] features. However, all these engineered methods fall short of the last mile (i.e., they are still less accurate than humans). This is a domain where machine learning and, in particular, The work of Yi et al. [163] takes this idea one step further: the authors proposed a deep CNN to detect tie-point locations in an image and output a descriptor vector for each tie-point.
2) Stereo processing using convolutional neural networks: The second important step in this workflow is stereo matching, i.e., the search for corresponding pixels in two or more images. In this step, a corresponding pixel is sought for every pixel in the image. In most cases, this search can be restricted to a line in the corresponding image. However, current methods still make mistakes in this process. The semi-global matching (SGM) approach by Hirschmueller [164] acted as the gold-standard method for some considerable time.
Since 2002, the progress on stereo processing is tracked by the Middlebury stereo evaluation benchmark 5 . The benchmark allows to compare results of stereo processing algorithms to a carefully maintained ground truth. The performance of the different algorithms can be viewed as a ranked list. This ranking reveals that, today, the top performing method is based on CNNs.
Most stereo methods in this ranking proceed along the following main steps. First, a stereo correspondence search is performed by computing a similarity measure between image locations. This is typically done exhaustively for all possible depth values. Next, the optimal depth values are searched by optimization on the cost value. Different optimization schemes, convex optimization, local-optimization strategies (e.g. SGM), and probabilities methods (e.g. MRF inference) are used. Finally, typically some heuristic filtering is applied to remove gross outliers (e.g. left-right check In addition to similarity measures, a typical stereo processing pipeline contains other engineered decisions as well. After creating a so-called cost volume from the similarity measures, most methods use specifically engineered algorithms for finding the depths (e.g. based on  is also changing the way that 3D information is generated from image data. The traditional 3D generation process did neglect object information. 3D data was generated from geometric constraints only. Image data were treated as pure intensity values without any semantic meaning.
The availability of semantic information from CNN-based classification now makes it possible to utilize this information in the 3D generation process. CNN-based classification allows one to assign class labels to aerial imagery with unprecedented accuracy [170]. Pixels in images are then assigned labels like vegetation, road, building etc. This semantic information can now be used to steer the 3D data generation process. Class label specific parameters can be chosen for the 3D data generation process.
The latest proposal in this area, however, is a joint reconstruction of 3D and semantic information. This has been proposed in the work of Haene et al. [171], where 3D reconstruction is performed with a volumetric method. The area to be reconstructed is partitioned into small cells, the size of it defining the resolution of the 3D reconstruction. The reconstruction algorithm now finds the optimal partitioning of this voxel grid into occupied and non-occupied voxels which fits to the image data. The result is a 3D reconstruction of the scene. The work of Haene et al. also jointly assigns the 3D reconstruction to a class label for each voxel, e.g. vegetation, building, road, sky. Each generated 3D data point now also has a class label. The 3D reconstruction is semantically interpretable. This process is a joint process, the computation of the occupied and non-occupied voxels takes into account the class labels in the original images. If a voxel corresponds to a building pixel in the image, it is set to occupied with high probability. If a voxel corresponds to a sky pixel in the image, it has a high probability of being unoccupied.
On the other hand, if a set of voxels are stacked on top of each other, it is likely that these belong to some building, i.e. the probability for assigning the label class of building is increased for this structure. This semantic 3D reconstruction method has been successfully applied to 3D reconstruction from aerial imagery by Blaha et al. [172,173]. In their work they achieved a semantic 3D reconstruction of cities on large scales. The 3D model not only contains 3D data but also class labels. E.g., 3D structure that represents buildings gets the class label of building.
Even more, every building has even the roof structures labeled as roof. Fig. 14 shows an image of a semantic 3D reconstruction produced by the method of [172].
In summary, it can be said that CNNs quickly took on a significant role in 3D data generation. Utilizing

A. Tutorials
Some valuable tutorials for early deep learners, including books, survey papers, code tutorials, and videos, can be found at http://deeplearning.net/reading-list/tutorials/. In addition, we list two references [174,175] which provide some general recommendations for the choice of the parameters.

B. Open-source Deep Learning Frameworks
When diving deep into deep learning, choosing an open-source framework is of great importance. Fig. 15  learning frameworks can be found at http://www.cio.com/article/3193689/artificial-intelligence/ which-deep-learning-network-is-best-for-you.html.

C. Remote Sensing Data for Training Deep Learning Models
To train deep learning methods with good generalization abilities, one needs large datasets. This is true for both fine-tuning models and training small networks from scratch, while if we consider training large architectures, one should preferably resort to pre-trained methods [176].
In recent years, there is several datasets that have been made public and that can be used to train deep neural networks. Below is a non-exhaustive list.

1) Scene classification (one image is classified into a single label):
- html.
-The NWPU-RESISC45 dataset [178]. This dataset contains 31,500 aerial images spread over 45 scene classes. So far, it is the largest dataset for land use scene classification in terms of both total number of images and number of scene classes. The dataset can be obtained at http://www.escience.cn/people/JunweiHan/NWPU-RESISC45.html.
2) Image classification (each pixel of an image is classified into a label): -

D. Showcasing
Starting to work with CNNs from zero might seem a titanic task. The number of models available is large and setting up an architecture from zero is challenging. In this section, we point to three showcasing example that have been recently provided by remote sensing researchers 6 .
Each example uses a different deep learning library (and programming language).
-Deconvolution network in MatConvNet. The first example is released by the authors of [92] and corresponds to the architecture in Fig. 11. It exploits the MatConvNet library for MATLAB (http://www.vlfeat.org/matconvnet/). It provides a pre-trained network for both the Vaihingen and Potsdam datasets described above. The initial models are specific to remote sensing data and have been trained on each dataset separately. This example is mostly meant to show how to fine-tune an existing model in MatConvNet by training few extra iteration to improve the model weights. It can, of course, be trained from scratch by reinitializing the weights randomly. A function to test the additional images of the datasets is also provided. Overall, it allows one to reproduce the results in [92] which are similar to the last column of Fig. 12. By removing the deconvolutional part of the network and adding a fully connected layer at the bottleneck, one can reproduce the CNN-PC model. If instead, one adds a spatial upsampling layer (e.g. a spatial interpolation of the bottleneck), one can also reproduce the results of the CNN-SPL model of Fig. 12. In both cases, the models must be re-trained (or at least heavily fine tuned).
-Fully convolutional (SegNet) architecture in Caffè. The second example is released by the authors of [142] and exploits the Caffè library (http://caffe.berkeleyvision.org/). The model exploits the SegNet architecture from Kendall et al. [181]. Authors release the pre-trained model to reproduce the results of [142] on the Vaihingen dataset. The network configuration, database generation and training files are given in Python.
-AConvNet for SAR ATR in Caffe. The third example is released by the authors of [42]. It implements a CNN-based SAR target recognition and demonstrates via the MSTAR dataset.
It includes the model configuration file and the source code for training and testing, as well as a successfully trained CNN model.
-Residual Conv-Deconv Network in TensorFlow. The last example is released by the authors of [33,34] and shows how to build up a residual Conv-Deconv network for unsupervised spectral-spatial feature learning of hyperspectral data. It exploits TensorFlow (https://www. tensorflow.org/) and Keras (https://keras.io/) libraries. One can transfer the trained network for their own classification purpose by fine-tuning on the target data sets or obtain "free" object detection using the learned filters in the first residual block of the residual Conv-Deconv network.

V. CONCLUSION AND FUTURE TRENDS
In this paper, we reviewed the current state of the art in deep learning for remote sensing.
Thanks to the enormous success encountered in several areas of research, remote sensing is also surfing the wave of deep neural networks and observing a similar trend as in other fields: deep nets are solid models that tend to improve over classical approaches using hand crafted features.
Yet, this field is still relatively young and, in the upcoming years, rapid advancement of deep learning in remote sensing is expected. Technical challenges obviously remain ahead: • What are the further applications in remote sensing which can potentially benefit from deep learning? In general, deep nets are particularly beneficial for remote sensing problems whose physical models are complicated, e.g., nonlinear, or even not yet well understood, or/and cannot be generalized. Yet, so far, in varies remote sensing fields, most deep learning-related research has been focused on classification and detection-related tasks using a number of benchmark data sets.
• Is the transferability of deep nets sufficient to extract geo-information on a global scale?
Complex light scattering mechanisms in natural objects, various atmospheric scattering conditions, intra-class variability, culture-dependent features, and limited training samples make the use of deep learning for global tasks challenging [182]. To meet the need of large-scale applications, possible solutions are: never-ending learning [183], self-taught learning [184], etc.
• How to tackle problems raised by very limited annotated data in remote sensing?
-Is possible to learn deep hierarchical models for remote sensing image understanding in a weakly-supervised, semi-supervised or even unsupervised way? Here, we list a few inspiring work in machine learning and computer vision: [185], [186], and [34].
-How do we benchmark the fast-growing deep-learning algorithms in remote sensing applications? Some recent initiatives include 2017 IEEE GRSS Data Fusion Contest dataset 7 and Functional Map of the World Challenge dataset 8 .
• Fusion of physics-based modeling and deep neural network is a promising direction. Remote sensing imagery is a direct product of physics processes, such as light reflection, microwave scattering, etc. It has to resort to a synergy of the physics-based models which describe the a priori knowledge of the process behind imagery and newly develop artificial intelligence technologies.
Besides focusing on technical challenges, deep learning in remote sensing opens up opportunities for new applications, such as monitoring global changes or evaluating strategies for the reduction of resources consumption, in which remote sensing can make a difference. In this context, deep learning remains an incredible toolbox that allows researcher in remote sensing to exceed the boundaries of the field, to move beyond traditional small-scale benchmarking task and tackling large-scale, real-life problems with implicit models that generalize well. The data are now here, the hardware is ready, deep learning frameworks are openly available and it is now time to design models that are tailored to big remote sensing data and their multi-modal, geo-located, multi-aspect and multi-temporal aspects that were raised in the introduction.
On the other hand, commercial players are on the march to remote sensing and Earth observation. For example, Planet has launched about 140 small satellites which map the whole Earth daily. Standing on the paradigm shift from computational science to data-driven science, we, remote sensing experts, shall appropriately position ourselves among other data scientists, who are also trying to use deep learning for innovative remote sensing applications. This requires us, in turn and as mentioned before, to bring our domain expertises into deep learning to provide prior knowledge that is tailored to specific remote sensing problems.
Last but not least, we advocate for efforts of the community to share data and architectures, to be able to answer the challenges of the years to come.