DiffRNN: Differential Verification of Recurrent Neural Networks
  Recurrent neural networks (RNNs) such as Long Short Term Memory (LSTM)
networks have become popular in a variety of applications such as image
processing, data classification, speech recognition, and as controllers in
autonomous systems. In practical settings, there is often a need to deploy such
RNNs on resource-constrained platforms such as mobile phones or embedded
devices. As the memory footprint and energy consumption of such components
become a bottleneck, there is interest in compressing and optimizing such
networks using a range of heuristic techniques. However, these techniques do
not guarantee the safety of the optimized network, e.g., against adversarial
inputs, or equivalence of the optimized and original networks. To address this
problem, we propose DIFFRNN, the first differential verification method for
RNNs to certify the equivalence of two structurally similar neural networks.
Existing work on differential verification for ReLUbased feed-forward neural
networks does not apply to RNNs where nonlinear activation functions such as
Sigmoid and Tanh cannot be avoided. RNNs also pose unique challenges such as
handling sequential inputs, complex feedback structures, and interactions
between the gates and states. In DIFFRNN, we overcome these challenges by
bounding nonlinear activation functions with linear constraints and then
solving constrained optimization problems to compute tight bounding boxes on
nonlinear surfaces in a high-dimensional space. The soundness of these bounding
boxes is then proved using the dReal SMT solver. We demonstrate the practical
efficacy of our technique on a variety of benchmarks and show that DIFFRNN
outperforms state-of-the-art RNN verification tools such as POPQORN.
