Can you hear me $\textit{now}$? Sensitive comparisons of human and
  machine perception
  The rise of sophisticated machine-recognition systems has brought with it a
rise in comparisons between human and machine perception. But such comparisons
face an asymmetry: Whereas machine perception of some stimulus can often be
probed through direct and explicit measures, much of human perceptual knowledge
is latent, incomplete, or embedded in unconscious mental processes that may not
be available for explicit report. Here, we show how this asymmetry can cause
such comparisons to underestimate the overlap in human and machine perception.
As a case study, we consider human perception of $\textit{adversarial speech}$
-- synthetic audio commands that are recognized as valid messages by automated
speech-recognition systems but that human listeners reportedly hear as
meaningless noise. In five experiments, we adapt task designs from the human
psychophysics literature to show that even when subjects cannot freely
transcribe adversarial speech (the previous benchmark for human understanding),
they nevertheless $\textit{can}$ discriminate adversarial speech from closely
matched non-speech (Experiments 1-2), finish common phrases begun in
adversarial speech (Experiments 3-4), and solve simple math problems posed in
adversarial speech (Experiment 5) -- even for stimuli previously described as
"unintelligible to human listeners". We recommend the adoption of
$\textit{sensitive tests}$ of human and machine perception, and discuss the
broader consequences of this approach for comparing natural and artificial
intelligence.
