A Generalization Theory based on Independent and Task-Identically
  Distributed Assumption
  Existing generalization theories analyze the generalization performance
mainly based on the model complexity and training process. The ignorance of the
task properties, which results from the widely used IID assumption, makes these
theories fail to interpret many generalization phenomena or guide practical
learning tasks. In this paper, we propose a new Independent and
Task-Identically Distributed (ITID) assumption, to consider the task properties
into the data generating process. The derived generalization bound based on the
ITID assumption identifies the significance of hypothesis invariance in
guaranteeing generalization performance. Based on the new bound, we introduce a
practical invariance enhancement algorithm from the perspective of modifying
data distributions. Finally, we verify the algorithm and theorems in the
context of image classification task on both toy and real-world datasets. The
experimental results demonstrate the reasonableness of the ITID assumption and
the effectiveness of new generalization theory in improving practical
generalization performance.
