Training on the test set? An analysis of Spampinato et al. [31]
  A recent paper [31] claims to classify brain processing evoked in subjects
watching ImageNet stimuli as measured with EEG and to use a representation
derived from this processing to create a novel object classifier. That paper,
together with a series of subsequent papers [8, 15, 17, 20, 21, 30, 35], claims
to revolutionize the field by achieving extremely successful results on several
computer-vision tasks, including object classification, transfer learning, and
generation of images depicting human perception and thought using brain-derived
representations measured through EEG. Our novel experiments and analyses
demonstrate that their results crucially depend on the block design that they
use, where all stimuli of a given class are presented together, and fail with a
rapid-event design, where stimuli of different classes are randomly intermixed.
The block design leads to classification of arbitrary brain states based on
block-level temporal correlations that tend to exist in all EEG data, rather
than stimulus-related activity. Because every trial in their test sets comes
from the same block as many trials in the corresponding training sets, their
block design thus leads to surreptitiously training on the test set. This
invalidates all subsequent analyses performed on this data in multiple
published papers and calls into question all of the purported results. We
further show that a novel object classifier constructed with a random codebook
performs as well as or better than a novel object classifier constructed with
the representation extracted from EEG data, suggesting that the performance of
their classifier constructed with a representation extracted from EEG data does
not benefit at all from the brain-derived representation. Our results calibrate
the underlying difficulty of the tasks involved and caution against sensational
and overly optimistic, but false, claims to the contrary.
