Predicting the Reproducibility of Social and Behavioral Science Papers
  Using Supervised Learning Models
  In recent years, significant effort has been invested verifying the
reproducibility and robustness of research claims in social and behavioral
sciences (SBS), much of which has involved resource-intensive replication
projects. In this paper, we investigate prediction of the reproducibility of
SBS papers using machine learning methods based on a set of features. We
propose a framework that extracts five types of features from scholarly work
that can be used to support assessments of reproducibility of published
research claims. Bibliometric features, venue features, and author features are
collected from public APIs or extracted using open source machine learning
libraries with customized parsers. Statistical features, such as p-values, are
extracted by recognizing patterns in the body text. Semantic features, such as
funding information, are obtained from public APIs or are extracted using
natural language processing models. We analyze pairwise correlations between
individual features and their importance for predicting a set of human-assessed
ground truth labels. In doing so, we identify a subset of 9 top features that
play relatively more important roles in predicting the reproducibility of SBS
papers in our corpus. Results are verified by comparing performances of 10
supervised predictive classifiers trained on different sets of features.
