Reprogramming Language Models for Molecular Representation Learning
  Recent advancements in transfer learning have made it a promising approach
for domain adaptation via transfer of learned representations. This is
especially when relevant when alternate tasks have limited samples of
well-defined and labeled data, which is common in the molecule data domain.
This makes transfer learning an ideal approach to solve molecular learning
tasks. While Adversarial reprogramming has proven to be a successful method to
repurpose neural networks for alternate tasks, most works consider source and
alternate tasks within the same domain. In this work, we propose a new
algorithm, Representation Reprogramming via Dictionary Learning (R2DL), for
adversarially reprogramming pretrained language models for molecular learning
tasks, motivated by leveraging learned representations in massive state of the
art language models. The adversarial program learns a linear transformation
between a dense source model input space (language data) and a sparse target
model input space (e.g., chemical and biological molecule data) using a k-SVD
solver to approximate a sparse representation of the encoded data, via
dictionary learning. R2DL achieves the baseline established by state of the art
toxicity prediction models trained on domain-specific data and outperforms the
baseline in a limited training-data setting, thereby establishing avenues for
domain-agnostic transfer learning for tasks with molecule data.
