Formal Analysis of Art: Proxy Learning of Visual Concepts from Style
  Through Language Models
  We present a machine learning system that can quantify fine art paintings
with a set of visual elements and principles of art. This formal analysis is
fundamental for understanding art, but developing such a system is challenging.
Paintings have high visual complexities, but it is also difficult to collect
enough training data with direct labels. To resolve these practical
limitations, we introduce a novel mechanism, called proxy learning, which
learns visual concepts in paintings though their general relation to styles.
This framework does not require any visual annotation, but only uses style
labels and a general relationship between visual concepts and style. In this
paper, we propose a novel proxy model and reformulate four pre-existing methods
in the context of proxy learning. Through quantitative and qualitative
comparison, we evaluate these methods and compare their effectiveness in
quantifying the artistic visual concepts, where the general relationship is
estimated by language models; GloVe or BERT. The language modeling is a
practical and scalable solution requiring no labeling, but it is inevitably
imperfect. We demonstrate how the new proxy model is robust to the
imperfection, while the other models are sensitively affected by it.
