Towards Deep Modeling of Music Semantics using EEG Regularizers
  Modeling of music audio semantics has been previously tackled through
learning of mappings from audio data to high-level tags or latent unsupervised
spaces. The resulting semantic spaces are theoretically limited, either because
the chosen high-level tags do not cover all of music semantics or because audio
data itself is not enough to determine music semantics. In this paper, we
propose a generic framework for semantics modeling that focuses on the
perception of the listener, through EEG data, in addition to audio data. We
implement this framework using a novel end-to-end 2-view Neural Network (NN)
architecture and a Deep Canonical Correlation Analysis (DCCA) loss function
that forces the semantic embedding spaces of both views to be maximally
correlated. We also detail how the EEG dataset was collected and use it to
train our proposed model. We evaluate the learned semantic space in a transfer
learning context, by using it as an audio feature extractor in an independent
dataset and proxy task: music audio-lyrics cross-modal retrieval. We show that
our embedding model outperforms Spotify features and performs comparably to a
state-of-the-art embedding model that was trained on 700 times more data. We
further discuss improvements to the model that are likely to improve its
performance.
