Toward the Detection of Polyglot Files
  Standardized file formats play a key role in the development and use of
computer software. However, it is possible to abuse standardized file formats
by creating a file that is valid in multiple file formats. The resulting
polyglot (many languages) file can confound file format identification,
allowing elements of the file to evade analysis.This is especially problematic
for malware detection systems that rely on file format identification for
feature extraction. File format identification processes that depend on file
signatures can be easily evaded thanks to flexibility in the format
specifications of certain file formats. Although work has been done to identify
file formats using more comprehensive methods than file signatures, accurate
identification of polyglot files remains an open problem. Since malware
detection systems routinely perform file format-specific feature extraction,
polyglot files need to be filtered out prior to ingestion by these systems.
Otherwise, malicious content could pass through undetected. To address the
problem of polyglot detection we assembled a data set using the mitra tool. We
then evaluated the performance of the most commonly used file identification
tool, file. Finally, we demonstrated the accuracy, precision, recall and F1
score of a range of machine and deep learning models. Malconv2 and Catboost
demonstrated the highest recall on our data set with 95.16% and 95.45%,
respectively. These models can be incorporated into a malware detector's file
processing pipeline to filter out potentially malicious polyglots before file
format-dependent feature extraction takes place.
