{'fraction_non_alphanumeric': 0.022100338764316826, 'fraction_numerical': 0.007904500725923537, 'mean_word_length': 5.072477962781587, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Physically Embodied Deep Image Optimisation

Physical sketches are created by learning programs to control a drawing robot. A differentiable rasteriser is used to optimise sets of drawing strokes to match an input image, using deep networks to provide an encoding for which we can compute a loss. The optimised drawing primitives can then be translated into G-code commands which command a robot to draw the image using drawing instruments such as pens and pencils on a physical support medium.


Introduction
Recent progress in differentiable rasterisation of primitives (points, lines and curves) on a pixel raster allow us to build machines that turn digital raster images into continuous parametric representations and back into rasterised images again [8]. Using such an approach, we can model the physical act of drawing with a pen on paper.
This approach differs from other methods which use neural networks to generate painting-like images, such as style transfer [4] or GANs [2], which essentially map textures across pixel grids and do not model the physical process of drawing. Stroke based-rendering with some optimisation goal has been studied before the deep learning era [6]. In the field of robot art, Pix18 [7] excels at creating oil paintings with its own art subject, and with minimal human intervention.While recent advances to image generation have focused on reinforcement learning [3] or neural renderers [9,10], the optimisation of brushstrokes against loss functions parameterised by deep networks, which can then be directly interpreted by a drawing robot has been missing from the field.

Optimisation of Stroke Primitive Parameters with Different Losses
With the machinery defined in Mihai and Hare [8] it is possible to define a complete system that takes the parameters describing stroke primitives and rasterises those strokes into an image. As shown in fig. 1, by introducing a (pretrained) neural network to extract features from both the complete rasterised image and a fixed target image, coupled with an appropriate loss function computed between those features, it becomes possible to compute gradients with respect to the parameters of the primitives that created the rasterised image. Minimising this loss will adapt the underlying primitives to "shapes" that maximise the feature similarity of the target image. If the rasterisation function is differentiable with respect to the stroke primitives, then the minimisation problem can be solved using gradient descent. Further, the optimised stroke parameters can be directly transformed into instructions that control a robot (see appendix A) that can manipulate a drawing instrument like a pen or pencil over a support medium. This allows us to produce physical artefacts directly from the model.
In our experiments, we optimise the underlying primitive parameters by minimising either the MSE loss directly against the image pixels (the encoder network is the identity function) or a loss built upon features extracted from the intermediate layers of a deep convolutional network. For the latter loss which is inherently differentiable, features are extracted using a VGG16 network pretrained on StylisedImageNet dataset (SIN) [5] which has a shape bias, or the regular ImageNet dataset [1] which has been shown to be texture-biased.

Case Studies
We provide two case studies to demonstrate the effectiveness of our approach. Firstly we demonstrate the effects can be achieved by transforming photographs to line drawings by optimising against different intermediate representations of a deep network. Secondly, we demonstrate a variation of the idea of neural style transfer [4] where instead of generating raster images we directly create the underlying stroke information that allows an image to be drawn under different physical drawing constraints.
How does my deep network think an image should be sketched? We can use our technique to explore how different internal representations of pretrained encoder networks manifest themselves in terms of illustrations of photographs as shown in fig. 2. Interestingly this allows us to explore different artistic effects that arise, but also to qualitatively look at the variations between differing inductive biases and loss formulations. What if Edvard Munch's 'The Scream' was painted in the Pointillism technique? Or Cubism? Figure 3 showcases the results of optimising different primitives to fit a photo of the original "The Scream" by Edvard Munch 3 . We display results by optimising 2000 uniformly coloured points (figs. 3b and 3d) and 1000 coloured lines (figs. 3c and 3e) to fit the original image by minimising either an MSE or a perceptual loss (with SIN-pretrained VGG16). The contrast between the images with the same type of primitive parametrisation, but using a different loss, is striking. The perceptual loss captures the shape information rather well, while moving away from the colour or texture scheme of the original or the variant realised with MSE loss. The copies thus produced could be tagged as belonging to the Fauvism art movement. Changing the parametrisation of the drawings in the method described in section 2 gives us an idea of what the painting would've looked like if it were to have been drawn in a Pointillist style (figs. 3b and 3d), or a more abstract, Cubism-like style (figs. 3c and 3e), while at the same time, making it possible for the drawing to be physically produced on paper by a drawing agent. More examples can be found in appendix B.

B More Examples of Image Optimisation
In this section, we provide more examples of our approach for image-based optimisation. Figure 5 shows results of optimising 1000 lines and 500 curves parameterised as Catmull-Rom splines against two of Seurat's paintings, 'Une Baignade at Asni'eres' and 'A Sunday Afternoon on the Island of La Grande Jatte'. This also shows the differences in optimising with a perceptual loss based on features extracted with a VGG network pretrained on SIN against a simple reconstruction loss. Similarly, fig. 6 shows a comparison between the image optimisation results using different primitive parametrisations by minimising a perceptual loss.