On-line Learning of an Unlearnable True Teacher through Mobile Ensemble
  Teachers
  On-line learning of a hierarchical learning model is studied by a method from
statistical mechanics. In our model a student of a simple perceptron learns
from not a true teacher directly, but ensemble teachers who learn from the true
teacher with a perceptron learning rule. Since the true teacher and the
ensemble teachers are expressed as non-monotonic perceptron and simple ones,
respectively, the ensemble teachers go around the unlearnable true teacher with
the distance between them fixed in an asymptotic steady state. The
generalization performance of the student is shown to exceed that of the
ensemble teachers in a transient state, as was shown in similar
ensemble-teachers models. Further, it is found that moving the ensemble
teachers even in the steady state, in contrast to the fixed ensemble teachers,
is efficient for the performance of the student.
