Soft Truncation: A Universal Training Technique of Score-based Diffusion
  Model for High Precision Score Estimation
  Recent advances in diffusion models bring state-of-the-art performance on
image generation tasks. However, empirical results from previous research in
diffusion models imply an inverse correlation between density estimation and
sample generation performances. This paper investigates with sufficient
empirical evidence that such inverse correlation happens because density
estimation is significantly contributed by small diffusion time, whereas sample
generation mainly depends on large diffusion time. However, training a score
network well across the entire diffusion time is demanding because the loss
scale is significantly imbalanced at each diffusion time. For successful
training, therefore, we introduce Soft Truncation, a universally applicable
training technique for diffusion models, that softens the fixed and static
truncation hyperparameter into a random variable. In experiments, Soft
Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ
256x256, and STL-10 datasets.
