BLOOM-Net: Blockwise Optimization for Masking Networks Toward Scalable
  and Efficient Speech Enhancement
  In this paper, we present a blockwise optimization method for masking-based
networks (BLOOM-Net) for training scalable speech enhancement networks. Here,
we design our network with a residual learning scheme and train the internal
separator blocks sequentially to obtain a scalable masking-based deep neural
network for speech enhancement. Its scalability lets it dynamically adjust the
run-time complexity depending on the test time environment. To this end, we
modularize our models in that they can flexibly accommodate varying needs for
enhancement performance and constraints on the resources, incurring minimal
memory or training overhead due to the added scalability. Our experiments on
speech enhancement demonstrate that the proposed blockwise optimization method
achieves the desired scalability with only a slight performance degradation
compared to corresponding models trained end-to-end.
