Embodied Self-supervised Learning by Coordinated Sampling and Training
  Self-supervised learning can significantly improve the performance of
downstream tasks, however, the dimensions of learned representations normally
lack explicit physical meanings. In this work, we propose a novel
self-supervised approach to solve inverse problems by employing the
corresponding physical forward process so that the learned representations can
have explicit physical meanings. The proposed approach works in an
analysis-by-synthesis manner to learn an inference network by iteratively
sampling and training. At the sampling step, given observed data, the inference
network is used to approximate the intractable posterior, from which we sample
input parameters and feed them to a physical process to generate data in the
observational space; At the training step, the same network is optimized with
the sampled paired data. We prove the feasibility of the proposed method by
tackling the acoustic-to-articulatory inversion problem to infer articulatory
information from speech. Given an articulatory synthesizer, an inference model
can be trained completely from scratch with random initialization. Our
experiments demonstrate that the proposed method can converge steadily and the
network learns to control the articulatory synthesizer to speak like a human.
We also demonstrate that trained models can generalize well to unseen speakers
or even new languages, and performance can be further improved through
self-adaptation.
