Content-Based Textual File Type Detection at Scale
  Programming language detection is a common need in the analysis of large
source code bases. It is supported by a number of existing tools that rely on
several features, and most notably file extensions, to determine file types. We
consider the problem of accurately detecting the type of files commonly found
in software code bases, based solely on textual file content. Doing so is
helpful to classify source code that lack file extensions (e.g., code snippets
posted on the Web or executable scripts), to avoid misclassifying source code
that has been recorded with wrong or uncommon file extensions, and also shed
some light on the intrinsic recognizability of source code files. We propose a
simple model that (a) use a language-agnostic word tokenizer for textual files,
(b) group tokens in 1-/2-grams, (c) build feature vectors based on N-gram
frequencies, and (d) use a simple fully connected neural network as classifier.
As training set we use textual files extracted from GitHub repositories with at
least 1000 stars, using existing file extensions as ground truth. Despite its
simplicity the proposed model reaches 85% in our experiments for a relatively
high number of recognized classes (more than 130 file types).
