Exploring Kernel Functions in the Softmax Layer for Contextual Word
  Classification
  Prominently used in support vector machines and logistic regressions, kernel
functions (kernels) can implicitly map data points into high dimensional spaces
and make it easier to learn complex decision boundaries. In this work, by
replacing the inner product function in the softmax layer, we explore the use
of kernels for contextual word classification. In order to compare the
individual kernels, experiments are conducted on standard language modeling and
machine translation tasks. We observe a wide range of performances across
different kernel settings. Extending the results, we look at the gradient
properties, investigate various mixture strategies and examine the
disambiguation abilities.
