Random Sampling in an Age of Automation: Minimizing Expenditures through
  Balanced Collection and Annotation
  Methods for automated collection and annotation are changing the
cost-structures of sampling surveys for a wide range of applications. Digital
samples in the form of images or audio recordings can be collected rapidly, and
annotated by computer programs or crowd workers. We consider the problem of
estimating a population mean under these new cost-structures, and propose a
Hybrid-Offset sampling design. This design utilizes two annotators: a primary,
which is accurate but costly (e.g. a human expert) and an auxiliary which is
noisy but cheap (e.g. a computer program), in order to minimize total sampling
expenditures. Our analysis gives necessary conditions for the Hybrid-Offset
design and specifies optimal sample sizes for both annotators. Simulations on
data from a coral reef survey program indicate that the Hybrid-Offset design
outperforms several alternative sampling designs. In particular, sampling
expenditures are reduced 50% compared to the Conventional design currently
deployed by the coral ecologists.
