Auditing ImageNet: Towards a Model-driven Framework for Annotating
  Demographic Attributes of Large-Scale Image Datasets
  The ImageNet dataset ushered in a flood of academic and industry interest in
deep learning for computer vision applications. Despite its significant impact,
there has not been a comprehensive investigation into the demographic
attributes of images contained within the dataset. Such a study could lead to
new insights on inherent biases within ImageNet, particularly important given
it is frequently used to pretrain models for a wide variety of computer vision
tasks. In this work, we introduce a model-driven framework for the automatic
annotation of apparent age and gender attributes in large-scale image datasets.
Using this framework, we conduct the first demographic audit of the 2012
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) subset of ImageNet
and the "person" hierarchical category of ImageNet. We find that 41.62% of
faces in ILSVRC appear as female, 1.71% appear as individuals above the age of
60, and males aged 15 to 29 account for the largest subgroup with 27.11%. We
note that the presented model-driven framework is not fair for all
intersectional groups, so annotation are subject to bias. We present this work
as the starting point for future development of unbiased annotation models and
for the study of downstream effects of imbalances in the demographics of
ImageNet. Code and annotations are available at:
http://bit.ly/ImageNetDemoAudit
