Machine Learning-Based Prototyping of Graphical User Interfaces for
  Mobile Apps
  It is common practice for developers of user-facing software to transform a
mock-up of a graphical user interface (GUI) into code. This process takes place
both at an application's inception and in an evolutionary context as GUI
changes keep pace with evolving features. Unfortunately, this practice is
challenging and time-consuming. In this paper, we present an approach that
automates this process by enabling accurate prototyping of GUIs via three
tasks: detection, classification, and assembly. First, logical components of a
GUI are detected from a mock-up artifact using either computer vision
techniques or mock-up metadata. Then, software repository mining, automated
dynamic analysis, and deep convolutional neural networks are utilized to
accurately classify GUI-components into domain-specific types (e.g.,
toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates
a suitable hierarchical GUI structure from which a prototype application can be
automatically assembled. We implemented this approach for Android in a system
called ReDraw. Our evaluation illustrates that ReDraw achieves an average
GUI-component classification accuracy of 91% and assembles prototype
applications that closely mirror target mock-ups in terms of visual affinity
while exhibiting reasonable code structure. Interviews with industrial
practitioners illustrate ReDraw's potential to improve real development
workflows.
