More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain
  Activity
  In the past few years, significant advancements were made in reconstruction
of observed natural images from fMRI brain recordings using deep-learning
tools. Here, for the first time, we show that dense 3D depth maps of observed
2D natural images can also be recovered directly from fMRI brain recordings. We
use an off-the-shelf method to estimate the unknown depth maps of natural
images. This is applied to both: (i) the small number of images presented to
subjects in an fMRI scanner (images for which we have fMRI recordings -
referred to as "paired" data), and (ii) a very large number of natural images
with no fMRI recordings ("unpaired data"). The estimated depth maps are then
used as an auxiliary reconstruction criterion to train for depth reconstruction
directly from fMRI. We propose two main approaches: Depth-only recovery and
joint image-depth RGBD recovery. Because the number of available "paired"
training data (images with fMRI) is small, we enrich the training data via
self-supervised cycle-consistent training on many "unpaired" data (natural
images & depth maps without fMRI). This is achieved using our newly defined and
trained Depth-based Perceptual Similarity metric as a reconstruction criterion.
We show that predicting the depth map directly from fMRI outperforms its
indirect sequential recovery from the reconstructed images. We further show
that activations from early cortical visual areas dominate our depth
reconstruction results, and propose means to characterize fMRI voxels by their
degree of depth-information tuning. This work adds an important layer of
decoded information, extending the current envelope of visual brain decoding
capabilities.
