Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing
  Imagery
  Clouds frequently cover the Earth's surface and pose an omnipresent challenge
to optical Earth observation methods. The vast majority of remote sensing
approaches either selectively choose single cloud-free observations or employ a
pre-classification strategy to identify and mask cloudy pixels. We follow a
different strategy and treat cloud coverage as noise that is inherent to the
observed satellite data. In prior work, we directly employed a straightforward
\emph{convolutional long short-term memory} network for vegetation
classification without explicit cloud filtering and achieved state-of-the-art
classification accuracies. In this work, we investigate this cloud-robustness
further by visualizing internal cell activations and performing an ablation
experiment on datasets of different cloud coverage. In the visualizations of
network states, we identified some cells in which modulation and input gates
closed on cloudy pixels. This indicates that the network has internalized a
cloud-filtering mechanism without being specifically trained on cloud labels.
Overall, our results question the necessity of sophisticated pre-processing
pipelines for multi-temporal deep learning approaches.
