Quantifying Long Range Dependence in Language and User Behavior to
  improve RNNs
  Characterizing temporal dependence patterns is a critical step in
understanding the statistical properties of sequential data. Long Range
Dependence (LRD) --- referring to long-range correlations decaying as a power
law rather than exponentially w.r.t. distance --- demands a different set of
tools for modeling the underlying dynamics of the sequential data. While it has
been widely conjectured that LRD is present in language modeling and sequential
recommendation, the amount of LRD in the corresponding sequential datasets has
not yet been quantified in a scalable and model-independent manner. We propose
a principled estimation procedure of LRD in sequential datasets based on
established LRD theory for real-valued time series and apply it to sequences of
symbols with million-item-scale dictionaries. In our measurements, the
procedure estimates reliably the LRD in the behavior of users as they write
Wikipedia articles and as they interact with YouTube. We further show that
measuring LRD better informs modeling decisions in particular for RNNs whose
ability to capture LRD is still an active area of research. The quantitative
measure informs new Evolutive Recurrent Neural Networks (EvolutiveRNNs)
designs, leading to state-of-the-art results on language understanding and
sequential recommendation tasks at a fraction of the computational cost.
