Unsupervised Doodling and Painting with Improved SPIRAL
  We investigate using reinforcement learning agents as generative models of
images (extending arXiv:1804.01118). A generative agent controls a simulated
painting environment, and is trained with rewards provided by a discriminator
network simultaneously trained to assess the realism of the agent's samples,
either unconditional or reconstructions. Compared to prior work, we make a
number of improvements to the architectures of the agents and discriminators
that lead to intriguing and at times surprising results. We find that when
sufficiently constrained, generative agents can learn to produce images with a
degree of visual abstraction, despite having only ever seen real photographs
(no human brush strokes). And given enough time with the painting environment,
they can produce images with considerable realism. These results show that,
under the right circumstances, some aspects of human drawing can emerge from
simulated embodiment, without the need for external supervision, imitation or
social cues. Finally, we note the framework's potential for use in creative
applications.
