How could Neural Networks understand Programs?
  Semantic understanding of programs is a fundamental problem for programming
language processing (PLP). Recent works that learn representations of code
based on pre-training techniques in NLP have pushed the frontiers in this
direction. However, the semantics of PL and NL have essential differences.
These being ignored, we believe it is difficult to build a model to better
understand programs, by either directly applying off-the-shelf NLP pre-training
techniques to the source code, or adding features to the model by the
heuristic. In fact, the semantics of a program can be rigorously defined by
formal semantics in PL theory. For example, the operational semantics,
describes the meaning of a valid program as updating the environment (i.e., the
memory address-value function) through fundamental operations, such as memory
I/O and conditional branching. Inspired by this, we propose a novel program
semantics learning paradigm, that the model should learn from information
composed of (1) the representations which align well with the fundamental
operations in operational semantics, and (2) the information of environment
transition, which is indispensable for program understanding. To validate our
proposal, we present a hierarchical Transformer-based pre-training model called
OSCAR to better facilitate the understanding of programs. OSCAR learns from
intermediate representation (IR) and an encoded representation derived from
static analysis, which are used for representing the fundamental operations and
approximating the environment transitions respectively. OSCAR empirically shows
the outstanding capability of program semantics understanding on many practical
software engineering tasks.
