One-Shot Learning for Language Modelling
  Humans can infer a great deal about the meaning of a word, using the syntax
and semantics of surrounding words even if it is their first time reading or
hearing it. We can also generalise the learned concept of the word to new
tasks. Despite great progress in achieving human-level performance in certain
tasks (Silver et al., 2016), learning from one or few examples remains a key
challenge in machine learning, and has not thoroughly been explored in Natural
Language Processing (NLP).
  In this work we tackle the problem of oneshot learning for an NLP task by
employing ideas from recent developments in machine learning: embeddings,
attention mechanisms (softmax) and similarity measures (cosine, Euclidean,
Poincare, and Minkowski). We adapt the framework suggested in matching networks
(Vinyals et al., 2016), and explore the effectiveness of the aforementioned
methods in one, two and three-shot learning problems on the task of predicting
missing word explored in (Vinyals et al., 2016) by using the WikiText-2
dataset. Our work contributes in two ways: Our first contribution is that we
explore the effectiveness of different distance metrics on k-shot learning, and
show that there is no single best distance metric for k-shot learning, which
challenges common belief. We found that the performance of a distance metric
depends on the number of shots used during training. The second contribution of
our work is that we establish a benchmark for one, two, and three-shot learning
on a language task with a publicly available dataset that can be used to
benchmark against in future research.
