Post-hoc labeling of arbitrary EEG recordings for data-efficient
  evaluation of neural decoding methods
  Many cognitive, sensory and motor processes have correlates in oscillatory
neural sources, which are embedded as a subspace into the recorded brain
signals. Decoding such processes from noisy
magnetoencephalogram/electroencephalogram (M/EEG) signals usually requires the
use of data-driven analysis methods. The objective evaluation of such decoding
algorithms on experimental raw signals, however, is a challenge: the amount of
available M/EEG data typically is limited, labels can be unreliable, and raw
signals often are contaminated with artifacts. The latter is specifically
problematic, if the artifacts stem from behavioral confounds of the oscillatory
neural processes of interest.
  To overcome some of these problems, simulation frameworks have been
introduced for benchmarking decoding methods. Generating artificial brain
signals, however, most simulation frameworks make strong and partially
unrealistic assumptions about brain activity, which limits the generalization
of obtained results to real-world conditions.
  In the present contribution, we thrive to remove many shortcomings of current
simulation frameworks and propose a versatile alternative, that allows for
objective evaluation and benchmarking of novel data-driven decoding methods for
neural signals. Its central idea is to utilize post-hoc labelings of arbitrary
M/EEG recordings. This strategy makes it paradigm-agnostic and allows to
generate comparatively large datasets with noiseless labels. Source code and
data of the novel simulation approach are made available for facilitating its
adoption.
