Integrating Discrete and Neural Features via Mixed-feature
  Trans-dimensional Random Field Language Models
  There has been a long recognition that discrete features (n-gram features)
and neural network based features have complementary strengths for language
models (LMs). Improved performance can be obtained by model interpolation,
which is, however, a suboptimal two-step integration of discrete and neural
features. The trans-dimensional random field (TRF) framework has the potential
advantage of being able to flexibly integrate a richer set of features.
However, either discrete or neural features are used alone in previous TRF LMs.
This paper develops a mixed-feature TRF LM and demonstrates its advantage in
integrating discrete and neural features. Various LMs are trained over PTB and
Google one-billion-word datasets, and evaluated in N-best list rescoring
experiments for speech recognition. Among all single LMs (i.e. without model
interpolation), the mixed-feature TRF LMs perform the best, improving over both
discrete TRF LMs and neural TRF LMs alone, and also being significantly better
than LSTM LMs. Compared to interpolating two separately trained models with
discrete and neural features respectively, the performance of mixed-feature TRF
LMs matches the best interpolated model, and with simplified one-step training
process and reduced training time.
