Assisted Perception: Optimizing Observations to Communicate State
  We aim to help users estimate the state of the world in tasks like robotic
teleoperation and navigation with visual impairments, where users may have
systematic biases that lead to suboptimal behavior: they might struggle to
process observations from multiple sensors simultaneously, receive delayed
observations, or overestimate distances to obstacles. While we cannot directly
change the user's internal beliefs or their internal state estimation process,
our insight is that we can still assist them by modifying the user's
observations. Instead of showing the user their true observations, we
synthesize new observations that lead to more accurate internal state estimates
when processed by the user. We refer to this method as assistive state
estimation (ASE): an automated assistant uses the true observations to infer
the state of the world, then generates a modified observation for the user to
consume (e.g., through an augmented reality interface), and optimizes the
modification to induce the user's new beliefs to match the assistant's current
beliefs. We evaluate ASE in a user study with 12 participants who each perform
four tasks: two tasks with known user biases -- bandwidth-limited image
classification and a driving video game with observation delay -- and two with
unknown biases that our method has to learn -- guided 2D navigation and a lunar
lander teleoperation video game. A different assistance strategy emerges in
each domain, such as quickly revealing informative pixels to speed up image
classification, using a dynamics model to undo observation delay in driving,
identifying nearby landmarks for navigation, and exaggerating a visual
indicator of tilt in the lander game. The results show that ASE substantially
improves the task performance of users with bandwidth constraints, observation
delay, and other unknown biases.
