Unsupervised Program Synthesis for Images By Sampling Without
  Replacement
  Program synthesis has emerged as a successful approach to the image parsing
task. Most prior works rely on a two-step scheme involving supervised
pretraining of a Seq2Seq model with synthetic programs followed by
reinforcement learning (RL) for fine-tuning with real reference images. Fully
unsupervised approaches promise to train the model directly on the target
images without requiring curated pretraining datasets. However, they struggle
with the inherent sparsity of meaningful programs in the search space. In this
paper, we present the first unsupervised algorithm capable of parsing
constructive solid geometry (CSG) images into context-free grammar (CFG)
without pretraining via non-differentiable renderer. To tackle the
\emph{non-Markovian} sparse reward problem, we combine three key ingredients --
(i) a grammar-encoded tree LSTM ensuring program validity (ii) entropy
regularization and (iii) sampling without replacement from the CFG syntax tree.
Empirically, our algorithm recovers meaningful programs in large search spaces
(up to $3.8 \times 10^{28}$). Further, even though our approach is fully
unsupervised, it generalizes better than supervised methods on the synthetic 2D
CSG dataset. On the 2D computer aided design (CAD) dataset, our approach
significantly outperforms the supervised pretrained model and is competitive to
the refined model.
