Graphical Models with Attention for Context-Specific Independence and an
  Application to Perceptual Grouping
  Discrete undirected graphical models, also known as Markov Random Fields
(MRFs), can flexibly encode probabilistic interactions of multiple variables,
and have enjoyed successful applications to a wide range of problems. However,
a well-known yet little studied limitation of discrete MRFs is that they cannot
capture context-specific independence (CSI). Existing methods require carefully
developed theories and purpose-built inference methods, which limit their
applications to only small-scale problems. In this paper, we propose the Markov
Attention Model (MAM), a family of discrete MRFs that incorporates an attention
mechanism. The attention mechanism allows variables to dynamically attend to
some other variables while ignoring the rest, and enables capturing of CSIs in
MRFs. A MAM is formulated as an MRF, allowing it to benefit from the rich set
of existing MRF inference methods and scale to large models and datasets. To
demonstrate MAM's capabilities to capture CSIs at scale, we apply MAMs to
capture an important type of CSI that is present in a symbolic approach to
recurrent computations in perceptual grouping. Experiments on two recently
proposed synthetic perceptual grouping tasks and on realistic images
demonstrate the advantages of MAMs in sample-efficiency, interpretability and
generalizability when compared with strong recurrent neural network baselines,
and validate MAM's capabilities to efficiently capture CSIs at scale.
