Training Transformers for Information Security Tasks: A Case Study on
  Malicious URL Prediction
  Machine Learning (ML) for information security (InfoSec) utilizes distinct
data types and formats which require different treatments during
optimization/training on raw data. In this paper, we implement a
malicious/benign URL predictor based on a transformer architecture that is
trained from scratch. We show that in contrast to conventional natural language
processing (NLP) transformers, this model requires a different training
approach to work well. Specifically, we show that 1) pre-training on a massive
corpus of unlabeled URL data for an auto-regressive task does not readily
transfer to malicious/benign prediction but 2) that using an auxiliary
auto-regressive loss improves performance when training from scratch. We
introduce a method for mixed objective optimization, which dynamically balances
contributions from both loss terms so that neither one of them dominates. We
show that this method yields performance comparable to that of several
top-performing benchmark classifiers.
