Veridical Data Science
  Building and expanding on principles of statistics, machine learning, and
scientific inquiry, we propose the predictability, computability, and stability
(PCS) framework for veridical data science. Our framework, comprised of both a
workflow and documentation, aims to provide responsible, reliable,
reproducible, and transparent results across the entire data science life
cycle. The PCS workflow uses predictability as a reality check and considers
the importance of computation in data collection/storage and algorithm design.
It augments predictability and computability with an overarching stability
principle for the data science life cycle. Stability expands on statistical
uncertainty considerations to assess how human judgment calls impact data
results through data and model/algorithm perturbations. Moreover, we develop
inference procedures that build on PCS, namely PCS perturbation intervals and
PCS hypothesis testing, to investigate the stability of data results relative
to problem formulation, data cleaning, modeling decisions, and interpretations.
We illustrate PCS inference through neuroscience and genomics projects of our
own and others and compare it to existing methods in high dimensional, sparse
linear model simulations. Over a wide range of misspecified simulation models,
PCS inference demonstrates favorable performance in terms of ROC curves.
Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook,
with publicly available, reproducible codes and narratives to back up human
choices made throughout an analysis. The PCS workflow and documentation are
demonstrated in a genomics case study available on Zenodo.
