BigGAN-based Bayesian reconstruction of natural images from human brain
  activity
  In the visual decoding domain, visually reconstructing presented images given
the corresponding human brain activity monitored by functional magnetic
resonance imaging (fMRI) is difficult, especially when reconstructing viewed
natural images. Visual reconstruction is a conditional image generation on fMRI
data and thus generative adversarial network (GAN) for natural image generation
is recently introduced for this task. Although GAN-based methods have greatly
improved, the fidelity and naturalness of reconstruction are still
unsatisfactory due to the small number of fMRI data samples and the instability
of GAN training. In this study, we proposed a new GAN-based Bayesian visual
reconstruction method (GAN-BVRM) that includes a classifier to decode
categories from fMRI data, a pre-trained conditional generator to generate
natural images of specified categories, and a set of encoding models and
evaluator to evaluate generated images. GAN-BVRM employs the pre-trained
generator of the prevailing BigGAN to generate masses of natural images, and
selects the images that best matches with the corresponding brain activity
through the encoding models as the reconstruction of the image stimuli. In this
process, the semantic and detailed contents of reconstruction are controlled by
decoded categories and encoding models, respectively. GAN-BVRM used the
Bayesian manner to avoid contradiction between naturalness and fidelity from
current GAN-based methods and thus can improve the advantages of GAN.
Experimental results revealed that GAN-BVRM improves the fidelity and
naturalness, that is, the reconstruction is natural and similar to the
presented image stimuli.
