Multi-modal Self-supervised Pre-training for Regulatory Genome Across
  Cell Types
  In the genome biology research, regulatory genome modeling is an important
topic for many regulatory downstream tasks, such as promoter classification,
transaction factor binding sites prediction. The core problem is to model how
regulatory elements interact with each other and its variability across
different cell types. However, current deep learning methods often focus on
modeling genome sequences of a fixed set of cell types and do not account for
the interaction between multiple regulatory elements, making them only perform
well on the cell types in the training set and lack the generalizability
required in biological applications. In this work, we propose a simple yet
effective approach for pre-training genome data in a multi-modal and
self-supervised manner, which we call GeneBERT. Specifically, we simultaneously
take the 1d sequence of genome data and a 2d matrix of (transcription factors x
regions) as the input, where three pre-training tasks are proposed to improve
the robustness and generalizability of our model. We pre-train our model on the
ATAC-seq dataset with 17 million genome sequences. We evaluate our GeneBERT on
regulatory downstream tasks across different cell types, including promoter
classification, transaction factor binding sites prediction, disease risk
estimation, and splicing sites prediction. Extensive experiments demonstrate
the effectiveness of multi-modal and self-supervised pre-training for
large-scale regulatory genomics data.
