Visualized Insights into the Optimization Landscape of Fully
  Convolutional Networks
  Many image processing tasks involve image-to-image mapping, which can be
addressed well by fully convolutional networks (FCN) without any heavy
preprocessing. Although empirically designing and training FCNs can achieve
satisfactory results, reasons for the improvement in performance are slightly
ambiguous. Our study is to make progress in understanding their generalization
abilities through visualizing the optimization landscapes. The visualization of
objective functions is obtained by choosing a solution and projecting its
vicinity onto a 3D space. We compare three FCN-based networks (two existing
models and a new proposed in this paper for comparison) on multiple datasets.
It has been observed in practice that the connections from the pre-pooled
feature maps to the post-upsampled can achieve better results. We investigate
the cause and provide experiments to shows that the skip-layer connections in
FCN can promote flat optimization landscape, which is well known to generalize
better. Additionally, we explore the relationship between the models
generalization ability and loss surface under different batch sizes. Results
show that large-batch training makes the model converge to sharp minimizers
with chaotic vicinities while small-batch method leads the model to flat
minimizers with smooth and nearly convex regions. Our work may contribute to
insights and analysis for designing and training FCNs.
