Modality Attention and Sampling Enables Deep Learning with Heterogeneous
  Marker Combinations in Fluorescence Microscopy
  Fluorescence microscopy allows for a detailed inspection of cells, cellular
networks, and anatomical landmarks by staining with a variety of
carefully-selected markers visualized as color channels. Quantitative
characterization of structures in acquired images often relies on automatic
image analysis methods. Despite the success of deep learning methods in other
vision applications, their potential for fluorescence image analysis remains
underexploited. One reason lies in the considerable workload required to train
accurate models, which are normally specific for a given combination of
markers, and therefore applicable to a very restricted number of experimental
settings. We herein propose Marker Sampling and Excite, a neural network
approach with a modality sampling strategy and a novel attention module that
together enable (i) flexible training with heterogeneous datasets with
combinations of markers and (ii) successful utility of learned models on
arbitrary subsets of markers prospectively. We show that our single neural
network solution performs comparably to an upper bound scenario where an
ensemble of many networks is na\"ively trained for each possible marker
combination separately. In addition, we demonstrate the feasibility of this
framework in high-throughput biological analysis by revising a recent
quantitative characterization of bone marrow vasculature in 3D confocal
microscopy datasets and further confirm the validity of our approach on an
additional, significantly different dataset of microvessels in fetal liver
tissues. Not only can our work substantially ameliorate the use of deep
learning in fluorescence microscopy analysis, but it can also be utilized in
other fields with incomplete data acquisitions and missing modalities.
