Evaluating recommender systems for AI-driven biomedical informatics
  Motivation: Many researchers with domain expertise are unable to easily apply
machine learning to their bioinformatics data due to a lack of machine learning
and/or coding expertise. Methods that have been proposed thus far to automate
machine learning mostly require programming experience as well as expert
knowledge to tune and apply the algorithms correctly. Here, we study a method
of automating biomedical data science using a web-based platform that uses AI
to recommend model choices and conduct experiments. We have two goals in mind:
first, to make it easy to construct sophisticated models of biomedical
processes; and second, to provide a fully automated AI agent that can choose
and conduct promising experiments for the user, based on the user's experiments
as well as prior knowledge. To validate this framework, we experiment with
hundreds of classification problems, comparing to state-of-the-art, automated
approaches. Finally, we use this tool to develop predictive models of septic
shock in critical care patients.
  Results: We find that matrix factorization-based recommendation systems
outperform meta-learning methods for automating machine learning. This result
mirrors the results of earlier recommender systems research in other domains.
The proposed AI is competitive with state-of-the-art automated machine learning
methods in terms of choosing optimal algorithm configurations for datasets. In
our application to prediction of septic shock, the AI-driven analysis produces
a competent machine learning model (AUROC 0.85 +/- 0.02) that performs on par
with state-of-the-art deep learning results for this task, with much less
computational effort.
