PyHard: a novel tool for generating hardness embeddings to support
  data-centric analysis
  For building successful Machine Learning (ML) systems, it is imperative to
have high quality data and well tuned learning models. But how can one assess
the quality of a given dataset? And how can the strengths and weaknesses of a
model on a dataset be revealed? Our new tool PyHard employs a methodology known
as Instance Space Analysis (ISA) to produce a hardness embedding of a dataset
relating the predictive performance of multiple ML models to estimated instance
hardness meta-features. This space is built so that observations are
distributed linearly regarding how hard they are to classify. The user can
visually interact with this embedding in multiple ways and obtain useful
insights about data and algorithmic performance along the individual
observations of the dataset. We show in a COVID prognosis dataset how this
analysis supported the identification of pockets of hard observations that
challenge ML models and are therefore worth closer inspection, and the
delineation of regions of strengths and weaknesses of ML models.
