Realistic Evaluation of Transductive Few-Shot Learning
  Transductive inference is widely used in few-shot learning, as it leverages
the statistics of the unlabeled query set of a few-shot task, typically
yielding substantially better performances than its inductive counterpart. The
current few-shot benchmarks use perfectly class-balanced tasks at inference. We
argue that such an artificial regularity is unrealistic, as it assumes that the
marginal label probability of the testing samples is known and fixed to the
uniform distribution. In fact, in realistic scenarios, the unlabeled query sets
come with arbitrary and unknown label marginals. We introduce and study the
effect of arbitrary class distributions within the query sets of few-shot tasks
at inference, removing the class-balance artefact. Specifically, we model the
marginal probabilities of the classes as Dirichlet-distributed random
variables, which yields a principled and realistic sampling within the simplex.
This leverages the current few-shot benchmarks, building testing tasks with
arbitrary class distributions. We evaluate experimentally state-of-the-art
transductive methods over 3 widely used data sets, and observe, surprisingly,
substantial performance drops, even below inductive methods in some cases.
Furthermore, we propose a generalization of the mutual-information loss, based
on $\alpha$-divergences, which can handle effectively class-distribution
variations. Empirically, we show that our transductive $\alpha$-divergence
optimization outperforms state-of-the-art methods across several data sets,
models and few-shot settings. Our code is publicly available at
https://github.com/oveilleux/Realistic_Transductive_Few_Shot.
