Long-Range Modeling of Source Code Files with eWASH: Extended Window
  Access by Syntax Hierarchy
  Statistical language modeling and translation with transformers have found
many successful applications in program understanding and generation tasks,
setting high benchmarks for tools in modern software development environments.
The finite context window of these neural models means, however, that they will
be unable to leverage the entire relevant context of large files and packages
for any given task. While there are many efforts to extend the context window,
we introduce an architecture-independent approach for leveraging the syntactic
hierarchies of source code for incorporating entire file-level context into a
fixed-length window. Using concrete syntax trees of each source file we extract
syntactic hierarchies and integrate them into context window by selectively
removing from view more specific, less relevant scopes for a given task. We
evaluate this approach on code generation tasks and joint translation of
natural language and source code in Python programming language, achieving a
new state-of-the-art in code completion and summarization for Python in the
CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for
user-experience-motivated tasks: code completion with normalized literals,
method body completion/code summarization conditioned on file-level context.
