Collaborative and Privacy-Preserving Machine Teaching via Consensus
  Optimization
  In this work, we define a collaborative and privacy-preserving machine
teaching paradigm with multiple distributed teachers. We focus on consensus
super teaching. It aims at organizing distributed teachers to jointly select a
compact while informative training subset from data hosted by the teachers to
make a learner learn better. The challenges arise from three perspectives.
First, the state-of-the-art pool-based super teaching method applies
mixed-integer non-linear programming (MINLP) which does not scale well to very
large data sets. Second, it is desirable to restrict data access of the
teachers to only their own data during the collaboration stage to mitigate
privacy leaks. Finally, the teaching collaboration should be
communication-efficient since large communication overheads can cause
synchronization delays between teachers.
  To address these challenges, we formulate collaborative teaching as a
consensus and privacy-preserving optimization process to minimize teaching
risk. We theoretically demonstrate the necessity of collaboration between
teachers for improving the learner's learning. Furthermore, we show that the
proposed method enjoys a similar property as the Oracle property of adaptive
Lasso. The empirical study illustrates that our teaching method can deliver
significantly more accurate teaching results with high speed, while the
non-collaborative MINLP-based super teaching becomes prohibitively expensive to
compute.
