Analysis of Noise Contrastive Estimation from the Perspective of
  Asymptotic Variance
  There are many models, often called unnormalized models, whose normalizing
constants are not calculated in closed form. Maximum likelihood estimation is
not directly applicable to unnormalized models. Score matching, contrastive
divergence method, pseudo-likelihood, Monte Carlo maximum likelihood, and noise
contrastive estimation (NCE) are popular methods for estimating parameters of
such models. In this paper, we focus on NCE. The estimator derived from NCE is
consistent and asymptotically normal because it is an M-estimator. NCE
characteristically uses an auxiliary distribution to calculate the normalizing
constant in the same spirit of the importance sampling. In addition, there are
several candidates as objective functions of NCE.
  We focus on how to reduce asymptotic variance. First, we propose a method for
reducing asymptotic variance by estimating the parameters of the auxiliary
distribution. Then, we determine the form of the objective functions, where the
asymptotic variance takes the smallest values in the original estimator class
and the proposed estimator classes. We further analyze the robustness of the
estimator.
