Noise Contrastive Estimation and Negative Sampling for Conditional
  Models: Consistency and Statistical Efficiency
  Noise Contrastive Estimation (NCE) is a powerful parameter estimation method
for log-linear models, which avoids calculation of the partition function or
its derivatives at each training step, a computationally demanding step in many
cases. It is closely related to negative sampling methods, now widely used in
NLP. This paper considers NCE-based estimation of conditional models.
Conditional models are frequently encountered in practice; however there has
not been a rigorous theoretical analysis of NCE in this setting, and we will
argue there are subtle but important questions when generalizing NCE to the
conditional case. In particular, we analyze two variants of NCE for conditional
models: one based on a classification objective, the other based on a ranking
objective. We show that the ranking-based variant of NCE gives consistent
parameter estimates under weaker assumptions than the classification-based
method; we analyze the statistical efficiency of the ranking-based and
classification-based variants of NCE; finally we describe experiments on
synthetic data and language modeling showing the effectiveness and trade-offs
of both methods.
