SHORING: Design Provable Conditional High-Order Interaction Network via
  Symbolic Testing
  Deep learning provides a promising way to extract effective representations
from raw data in an end-to-end fashion and has proven its effectiveness in
various domains such as computer vision, natural language processing, etc.
However, in domains such as content/product recommendation and risk management,
where sequence of event data is the most used raw data form and experts derived
features are more commonly used, deep learning models struggle to dominate the
game. In this paper, we propose a symbolic testing framework that helps to
answer the question of what kinds of expert-derived features could be learned
by a neural network. Inspired by this testing framework, we introduce an
efficient architecture named SHORING, which contains two components:
\textit{event network} and \textit{sequence network}. The \textit{event}
network learns arbitrarily yet efficiently high-order \textit{event-level}
embeddings via a provable reparameterization trick, the \textit{sequence}
network aggregates from sequence of \textit{event-level} embeddings. We argue
that SHORING is capable of learning certain standard symbolic expressions which
the standard multi-head self-attention network fails to learn, and conduct
comprehensive experiments and ablation studies on four synthetic datasets and
three real-world datasets. The results show that SHORING empirically
outperforms the state-of-the-art methods.
