A data-driven approach for learning to control computers
  It would be useful for machines to use computers as humans do so that they
can aid us in everyday tasks. This is a setting in which there is also the
potential to leverage large-scale expert demonstrations and human judgements of
interactive behaviour, which are two ingredients that have driven much recent
success in AI. Here we investigate the setting of computer control using
keyboard and mouse, with goals specified via natural language. Instead of
focusing on hand-designed curricula and specialized action spaces, we focus on
developing a scalable method centered on reinforcement learning combined with
behavioural priors informed by actual human-computer interactions. We achieve
state-of-the-art and human-level mean performance across all tasks within the
MiniWob++ benchmark, a challenging suite of computer control problems, and find
strong evidence of cross-task transfer. These results demonstrate the
usefulness of a unified human-agent interface when training machines to use
computers. Altogether our results suggest a formula for achieving competency
beyond MiniWob++ and towards controlling computers, in general, as a human
would.
