{'fraction_non_alphanumeric': 0.02095238095238095, 'fraction_numerical': 0.005079365079365079, 'mean_word_length': 5.380566801619433, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Detecting dubious research with SciBERT

Over the past few decades, the rate of scientific publication has increased dra-matically. At the same time, the number of retractions and failed replications of studies have also increased [1]. This replication crisis across many scientific disciplines has heightened scrutiny of scientific papers and their reported results [2]. However, the incentive for researchers to pursue replication of prior research is limited and are typically outweighed by the costs. As such, there is a need for computational methods for prioritizing replication efforts on scientific findings that are most dubious (i.e a high estimated probability of retraction) and highest impact (i.e. a large number of citations). The goal of this final project is to address the first desiderata by building a classifier for detecting dubious research articles from their titles and abstracts. Using the PubMed database, I retrieved over 10,000 titles and abstracts of retracted scientific papers along with matched negative controls. I finetuned a SciBERT model [3] to classify retracted papers from negative controls with only the titles or abstracts. Under finetuning with frozen SciBERT weights, these models outperformed a benchmark random forest classifier model trained on token frequency vectors. In combination with other metrics such as the number of citations, these models may be useful in prioritizing certain research articles to be investigated more thoroughly via replication efforts, which would be beneficial for the entire scientific ecosystem.