Improving Ethical Outcomes with Machine-in-the-Loop: Broadening Human
  Understanding of Data Annotations
  We introduce a machine-in-the-loop pipeline that aims to address root causes
of unwanted bias in natural language based supervised machine learning tasks in
the education domain. Learning from the experiences of students is foundational
for education researchers, and academic administrators. 21st-century skills
learned from experience are becoming a core part of college and career
readiness as well as the hiring process in the new knowledge economy.
Minoritized students demonstrate these skills in their daily lives, but
documenting, assessing, and validating these skills is a huge problem for
educational institutions. As an equity focused online platform, LivedX
translates minoritized students' lived experiences into the 21st century
skills, issues micro-credentials, and creates personal 21st century skills
portfolio. To automate the micro credential mining from the natural language
texts received from the students' submitted essays, we employed a bag-of-word
model to construct a multi-output classifier. Despite our goal, our model
initially exacerbated disparate impact on minoritized students. We used a
machine-in-the-loop model development pipeline to address the problem and
refine the aforementioned model to ensure fairness in its prediction.
