Hybrid Classification and Reasoning for Image-based Constraint Solving
  There is an increased interest in solving complex constrained problems where
part of the input is not given as facts but received as raw sensor data such as
images or speech. We will use "visual sudoku" as a prototype problem, where the
given cell digits are handwritten and provided as an image thereof. In this
case, one first has to train and use a classifier to label the images, so that
the labels can be used for solving the problem. In this paper, we explore the
hybridization of classifying the images with the reasoning of a constraint
solver. We show that pure constraint reasoning on predictions does not give
satisfactory results. Instead, we explore the possibilities of a tighter
integration, by exposing the probabilistic estimates of the classifier to the
constraint solver. This allows joint inference on these probabilistic
estimates, where we use the solver to find the maximum likelihood solution. We
explore the trade-off between the power of the classifier and the power of the
constraint reasoning, as well as further integration through the additional use
of structural knowledge. Furthermore, we investigate the effect of calibration
of the probabilistic estimates on the reasoning. Our results show that such
hybrid approaches vastly outperform a separate approach, which encourages a
further integration of prediction (probabilities) and constraint solving.
