Consensus-based Interpretable Deep Neural Networks with Application to
  Mortality Prediction
  Deep neural networks have achieved remarkable success in various challenging
tasks. However, the black-box nature of such networks is not acceptable to
critical applications, such as healthcare. In particular, the existence of
adversarial examples and their overgeneralization to irrelevant,
out-of-distribution inputs with high confidence makes it difficult, if not
impossible, to explain decisions by such networks. In this paper, we analyze
the underlying mechanism of generalization of deep neural networks and propose
an ($n$, $k$) consensus algorithm which is insensitive to adversarial examples
and can reliably reject out-of-distribution samples. Furthermore, the consensus
algorithm is able to improve classification accuracy by using multiple trained
deep neural networks. To handle the complexity of deep neural networks, we
cluster linear approximations of individual models and identify highly
correlated clusters among different models to capture feature importance
robustly, resulting in improved interpretability. Motivated by the importance
of building accurate and interpretable prediction models for healthcare, our
experimental results on an ICU dataset show the effectiveness of our algorithm
in enhancing both the prediction accuracy and the interpretability of deep
neural network models on one-year patient mortality prediction. In particular,
while the proposed method maintains similar interpretability as conventional
shallow models such as logistic regression, it improves the prediction accuracy
significantly.
