VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection
This paper presents VulBERTa, a deep learning approach to detect security
vulnerabilities in source code. Our approach pre-trains a RoBERTa model with a
custom tokenisation pipeline on real-world code from open-source C/C++
projects. The model learns a deep knowledge representation of the code syntax
and semantics, which we leverage to train vulnerability detection classifiers.
We evaluate our approach on binary and multi-class vulnerability detection
tasks across several datasets (Vuldeepecker, Draper, REVEAL and muVuldeepecker)
and benchmarks (CodeXGLUE and D2A). The evaluation results show that VulBERTa
achieves state-of-the-art performance and outperforms existing approaches
across different datasets, despite its conceptual simplicity, and limited cost
in terms of size of training data and number of model parameters.