Aspirations and Practice of Model Documentation: Moving the Needle with
  Nudging and Traceability
  Machine learning models have been widely developed, released, and adopted in
numerous applications. Meanwhile, the documentation practice for machine
learning models often falls short of established practices for traditional
software components, which impedes model accountability, inadvertently abets
inappropriate or misuse of models, and may trigger negative social impact.
Recently, model cards, a template for documenting machine learning models, have
attracted notable attention, but their impact on the practice of model
documentation is unclear. In this work, we examine publicly available model
cards and other similar documentation. Our analysis reveals a substantial gap
between the suggestions made in the original model card work and the content in
actual documentation. Motivated by this observation and literature on fields
such as software documentation, interaction design, and traceability, we
further propose a set of design guidelines that aim to support the
documentation practice for machine learning models including (1) the
collocation of documentation environment with the coding environment, (2)
nudging the consideration of model card sections during model development, and
(3) documentation derived from and traced to the source. We designed a
prototype tool named DocML following those guidelines to support model
development in computational notebooks. A lab study reveals the benefit of our
tool to shift the behavior of data scientists towards documentation quality and
accountability.
