3D Instance Segmentation of MVS Buildings
  We present a novel framework for instance segmentation of 3D buildings from
Multi-view Stereo (MVS) urban scenes. Unlike existing works focusing on
semantic segmentation of an urban scene, the emphasis of this work lies in
detecting and segmenting 3D building instances even if they are attached and
embedded in a large and imprecise 3D surface model. Multi-view RGB images are
first enhanced to RGBH images by adding a heightmap and are segmented to obtain
all roof instances using a fine-tuned 2D instance segmentation neural network.
Roof instance masks from different multi-view images are then clustered into
global masks. Our mask clustering accounts for spatial occlusion and
overlapping, which can eliminate segmentation ambiguities among multi-view
images. Based on these global masks, 3D roof instances are segmented out by
mask back-projections and extended to the entire building instances through a
Markov random field (MRF) optimization. Quantitative evaluations and ablation
studies have shown the effectiveness of all major steps of the method. A
dataset for the evaluation of instance segmentation of 3D building models is
provided as well. To the best of our knowledge, it is the first dataset for 3D
urban buildings on the instance segmentation level.
