Neural Theorem Provers Do Not Learn Rules Without Exploration
  Neural symbolic processing aims to combine the generalization of logical
learning approaches and the performance of neural networks. The Neural Theorem
Proving (NTP) model by Rocktaschel et al (2017) learns embeddings for concepts
and performs logical unification. While NTP is promising and effective in
predicting facts accurately, we have little knowledge how well it can extract
true relationship among data. To this end, we create synthetic logical datasets
with injected relationships, which can be generated on-the-fly, to test
neural-based relation learning algorithms including NTP. We show that it has
difficulty recovering relationships in all but the simplest settings. Critical
analysis and diagnostic experiments suggest that the optimization algorithm
suffers from poor local minima due to its greedy winner-takes-all strategy in
identifying the most informative structure (proof path) to pursue. We alter the
NTP algorithm to increase exploration, which sharply improves performance. We
argue and demonstate that it is insightful to benchmark with synthetic data
with ground-truth relationships, for both evaluating models and revealing
algorithmic issues.
