{'fraction_non_alphanumeric': 0.03299845133621888, 'fraction_numerical': 0.014335067307310488, 'mean_word_length': 4.913007748297723, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.

. Brain Decoding and Image Reconstruction. For the first time, our proposed MinD-Vis is capable of decoding fMRI-based brain activities and reconstructing images with not only plausible details but also accurate semantics and image features (texture, shape, etc.), outperforming previous approaches. Left: Task overview. Middle: Comparison with benchmarks. Right: More reconstruction examples.

Abstract
Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.

Introduction
"What you think is what you see". Human perception and prior knowledge are deeply intertwined in one's mind [52]. Our perception of the world is determined not only by objective stimuli properties but also by our experiences, forming complex brain activities underlying our perception. Understanding these brain activities and recovering the encoded information is a key goal in cognitive neuroscience. Within this broad objective, decoding visual information is one of the challenging problems that are the focus of a large body of literature [22,28,35,68].
As a non-invasive and effective method to measure brain activities indirectly, functional Magnetic Resonance Imaging (fMRI) is usually used to recover visual information, such as the image classes [21,40]. With the help of recent deep learning models, it is intriguing if the original visual stimuli can be directly recovered from corresponding fMRI [2,47], especially with the guidance of biological principles [44,53]. However, due to the lack of fMRI-image pairs and useful biological guidance when decoding complex neural activity from fMRI directly, reconstructed images are usually blurry and semantically meaningless. Thus it is crucial to learn effective and biological-valid representations for fMRI so that a clear and generalizable connection between brain activities and visual stimuli can be established with a few paired annotations.
Moreover, individual variability in brain representations further complicates this problem. Individuals have unique brain activation patterns responding to the same visual stimulus (See Fig. 2). From the perspective of fMRI representation learning, a powerful brain decoding algorithm should robustly recognize features shared across the population over a background of individual variation [5,21]. On the other hand, we should also expect decoding variances due to the variation in individual perceptions. Therefore, we aim to learn representations from a large-scale dataset with rich demographic compositions and relax the direct generation from fMRI to conditional synthesis allowing for sampling variance under the same semantic category.
Self-supervised learning with pretext tasks in large datasets is a powerful paradigm to distill the model with context knowledge. A domain-specific downstream task (e.g. classification) is usually adopted to finetune the pre-trained model further [37,59], especially when the downstream dataset is small. Various pretext tasks are designed to benefit downstream tasks [23,67]. Among these methods, Masked Signal Modeling (MSM) has achieved promising results in both vision [18,63] and language understanding [8,38] recently. At the same time, the probabilistic diffusion denoising model has shown its superior performance in content generation and training stability [9]. A strong generation ability is also desired in our task to decode faithful visual stimuli from various categories.
Driven by the above analysis, we propose MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding, a framework that exploits the power of large-scale representation learning and mimics the sparse coding of information in the brain [14], including the visual cortex [57]. Different from [18], we use a much larger representation-to-data-space ratio to boost the information capacity of learned representations. Our contributions are as follows: • We propose Sparse-Coded Masked Brain Modeling (SC-MBM), designed under biological guidance as an effective brain feature learner for vision decoding.
• Augmenting the latent diffusion model with double conditioning (DC-LDM), we enforce stronger decoding consistency while allowing variance under the same semantics.
• Integrating the representation ability of SC-MBM with the generation ability of DC-LDM, MinD-Vis generates more plausible images with better preserved semantic information compared with previous methods.
• Quantitative and qualitative tests are performed on multiple datasets, including a new dataset that has not previously been used to evaluate this task.

Related Work
Conventional Decoding Methods Conventional methods rely on training with fMRI and corresponding hierarchical image features extracted by a pre-trained VGG [21,47]. During testing, the predicted image features will either be used for classification or fed into a generative model like GAN [46] to reconstruct the original stimulus. Instead of directly learning the limited training pairs, [2] enabled unsupervised learning on unpaired fMRI and images with a reconfigurable autoencoder design. [16] further extended this method to images from diverse semantic categories. However, just as with conventional approaches, fMRI is used directly for training and decoding. Authors in [32,34] took a different approach. A regression model was applied to extract latent fMRI representation, which was then used to finetune a pre-trained conditional bigGAN. Decoding is performed by conditionally sampling the bigGAN with the fMRI latent. With this method, the decoded images are more plausible and semantically meaningful.

Masked Signal Modeling
The power of MSM in learning representations from a large-scale dataset was first exploited in [8], which was later adapted to computer vision [18,61,63]. Successful applications to downstream tasks show that useful context knowledge is learned with MSM as a pretext task. In essence, MSM is a generalized denoising autoencoder that aims to recover the original data from the remaining after masking [4]. The portion of data to mask is different across data modalities, with an extremely high mask ratio (75%) usually used for visual signals [18]. In contrast, due to the disparity in information density, a low mask ratio (25%) is used in natural languages [8].
Diffusion Probabilistic Models Diffusion models [50] are emerging generative models that generate high-quality content.
In its basic form [20], the diffusion model is a probabilistic model defined by a bi-directional Markov Chain of states. Two processes are transiting through the chain: (i) The forward diffusion process gradually adds noise to the data until it is fully destroyed to an isotropic Gaussian noise; (ii) The reverse process recovers the corrupted data by modeling a posterior distribution p(x) at each state and eventually obtains a sample in the original data distribution [20,50,51]. Formally, assume a Markov Chain with a fixed length T , then the reverse conditional probability can be expressed as q(x t−1 |x t ), where t = 1,...,T and x t is obtained by corrupting the image x t−1 with Gaussian noise. After parameterization, this conditional probability can be learned by optimizing a variational lower bound which can be simplified to the following objective [20]: where θ (x t ,t) is a set of denoising functions that are usually implemented as UNets [9,42,43]. We refer readers to [20] for detailed descriptions of the diffusion models. Pre-train on fMRI with SC-MBM. We patchify, randomly mask the fMRI, and then tokenize them to large embeddings. We train an autoencoder (E MBM and D MBM ) to recover the masked patches. Stage B (right): Integration with the LDM through double conditioning. We project the fMRI latent (L fMRI ) through two paths to the LDM conditioning space with a latent dimension projector (P fMRI→Cond ). One path connects directly to cross-attention heads in the LDM. Another path adds the fMRI latent to time embeddings. The LDM operates on a low-dimensional, compressed version of the original image (i.e. image latent), however, the original image is used in this figure for illustrations.

Latent Diffusion Model (LDM)
Apart from the conventional diffusion models that generate samples in the original data space, another category of diffusion models that generate samples in the latent feature space has been proposed [42,49]. Operating in the latent feature space reduces the computational cost and introduces less spatial downsampling, giving better image synthesis quality. The LDM proposed in [42] consists of two components: (i) Vector Quantization (VQ) regularized [12] autoencoder that compresses images into lower-dimensional latent features and then reconstructs the images from features in the same space; (ii) UNet-based denoising model with attention modules. Incorporating attention mechanisms into the UNet allows the flexibility to condition image generation through key/value/query vectors during the Markov Chain transitions.

Motivation and Overview
In this subsection, we provide a detailed analysis of the fMRI data and elaborate on the motivations of our designs.
(i) fMRI measures the brain blood-oxygen-level-dependent (BOLD) changes as 3D voxels that serve as a proxy for the underlying changes in brain activity. Neighboring voxels often have similar amplitudes, indicating spatial redundancy in fMRI [54].
(ii) fMRI data is averaged across the time during which the stimulus is presented. A region of interest (ROI) of the averaged data is usually extracted as a 1D vector of voxels (in the visual processing hierarchy). The ROI size (voxel number) is generally smaller than the image size (pixel number). For example, [21] has about 4500 voxels (visual cortex), which is much smaller than a 256×256 RGB image. This creates a large difference in dimensionality when transforming fMRI into images.
(iii) fMRI data from different datasets may have significant domain shifts due to experimental conditions and scanner setups. Even with the same scan conditions, ROI size and location mismatch persist due to individual differences (See Fig. 2).
Driven by this analysis, we propose MinD-Vis, designed with two sequential stages as outlined in Fig. 3. Briefly, in Stage A, fMRI representations are learned by an autoencoder trained in a large fMRI dataset with masked signal modeling as a pretext task. The learned representations will be used as a condition to guide the image-generation process in the next stage. In Stage B, the pre-trained fMRI encoder is integrated with the LDM through cross-attention and time-step conditioning for conditional synthesis. In this stage, the encoder is jointly finetuned with cross-attention heads in the LDM using paired annotations.

Stage A: Sparse-Coded MBM (SC-MBM)
Activity in the human brain involves non-linear interactions among 86 billion neuronal cells in the brain and are thus highly complex [33,41]. The fMRI measuring the BOLD signals is an indirect and aggregate measure of neuronal activities, which can be analyzed hierarchically with functional networks [1,6,60]. These functional networks comprised of voxels of fMRI data have implicit correlations with each other in response to external stimuli [55,69]. Therefore, learning these implicit correlations by recovering masked voxels will equip the pre-trained model with a deep contextual understanding of the fMRI data.
Following [18], we divide the vectorized voxels into patches which will be subsequently transformed into embeddings using a 1D convolutional layer with a stride equal to the patch size. The hemodynamic response measured in fMRI BOLD signal causes spatial blurring, which creates spatial redundancy in fMRI data, like in natural images [11,48]. Due to the spatial redundancy, fMRI data can still be recovered even if a large portion is masked (See Fig. 4). Thus, in the first stage of MinD-Vis, we can mask a large portion of the fMRI patches to save computations without losing the learning power of masked modeling.
Masked Image Modeling (MIM) uses the embedding-topatch-size ratio around one [18], leading to a representation size similar to the original data size. However, we use a large embedding-to-patch-size ratio, which significantly increases the information capacity with a large fMRI representation space. This design also relates to the sparse coding of information in the brain, which has been proposed as a general strategy for the representation of sensory information [27].
We also adopt an asymmetric architecture as in [18]: the encoder is optimized to learn effective fMRI representations, while the decoder tries to predict the masked patches. Therefore, we make the decoder small in size, and it is discarded in Stage B as long as the pre-training converges.
Visual Encoding and Brain-Inspired Sparse Coding Here, we explain the biological basis of using SC-MBM to learn representations of visual stimuli in the brain from the perspective of visual encoding mechanisms. Theoretical and empirical studies suggest that visual stimuli are sparsely encoded in the primary visual cortex [33,39,57], with most natural images activating only a portion of the neurons in the visual cortex. This strategy increases information transmission efficiency and creates minimal redundancy in the brain [39]. As a result, visual information of natural scenes can be reconstructed from a small portion of data collected from the primary visual cortex via different imaging modalities, including fMRI [15,65]. This observation is interesting for the computer vision community because the sparse coding could be an efficient way for vision encoding in computer vision as well [27,64].
Sparse coding is an encoding strategy that in essence uses over-complete bases to represent data, where more locality is generally enforced to generate smoother representations [58,66]. In SC-MBM, fMRI data are divided into patches to introduce locality constraints. Then each patch is encoded into a high-dimensional vector space with a size much larger than the original data space, thus creating an over-complete space for fMRI representation (See Appendix). Emulating the brain vision encoding, SC-MBM can be a biologically-valid and effective brain feature learner for fMRI decoding.

Stage B: Double-Conditioned LDM (DC-LDM)
After the large-scale context learning in Stage A, the fMRI encoder transforms fMRI data into sparsely coded representations with locality constraints. To further decode visual contents from this abstract representation and allow for sampling variance, we formulate the decoding task as a conditional synthesis problem and approach it with a pre-trained LDM.
The LDM operates on the image latent space denoted by E(x) where x is an image in pixel space and E(·) is a VQ encoder. In our setting, we omit E(x) and use x directly to represent the latent variable of LDM for simplicity. Specifically, given the fMRI data z, we aim to learn the reverse diffusion process formulated by q(x t−1 |x t ,z). As proposed in [42], conditional information is applied through cross-attention heads in the attention-based Here, τ θ is the fMRI encoder with a suitable dimension projec- V are projector matrices with learnable parameters. Diversity and consistency are two opposite objectives when sampling a conditional generative model. Sampling diversity across various modalities such as label-to-image and text-to-image is very important in many image-generation tasks. However, the fMRI-to-image transition relies more on generation consistency-decoded images from similar brain activities are expected to be semantically similar. Thus, a stronger conditioning mechanism is desired to ensure such generation consistency, especially for probabilistic diffusion models.
In this way, we integrate the cross-attention conditioning with another conditioning method called the time steps conditioning [9] to provide stronger guidance for our task. In time steps conditioning, we add σ θ (τ θ (z)) to time step embeddings, where σ θ (·) is another suitable dimension projector. Time step embeddings are used in intermediate layers of the UNet, thus we have ϕ i (x t )=ϕ i (x t ,σ θ (τ θ (z))). We further reformulate the optimization objective Eq. (1) to a double conditioning alternation: We omit the parameterization symbol θ in τ(·) and σ(·) for simplicity. Additionally, we have τ(z)∈R M×dτ and σ(τ(z))∈ R 1×dt , where d τ and d t are the latent dimensions and time embedding dimension respectively, and M is a tunable parameter.
Finetuning After the fMRI encoder is pre-trained with SC-MBM, it is integrated with a pre-trained LDM through double conditioning. Commonly, the encoder's output is averaged, or a cls token is appended to produce a pooled 1D feature vector for downstream tasks [8,18]. This strategy is effective for tasks like prediction and classification, where learned knowledge  is expected to be distilled, producing distinguishable features. However, pooling into a 1D vector is inappropriate for retaining fMRI representations' sparsity and information capacity. Instead, we used convolution layers to pool the encoder's output into a latent dimension of R M×dτ as described in Eq. (2). The fMRI encoder, cross-attention heads, and projection heads are jointly optimized, while other parts are fixed. Finetuning the cross-attention heads is critical for bridging the pre-trained conditioning space and fMRI latent space. The finetuning is performed end-to-end with fMRI-image pairs, during which a clearer connection between the fMRI and image features will be learned through the large-capacity fMRI representations.

Datasets and Implementation
Datasets Three public datasets were used in this study: Human Connectome Project (HCP) 1200 Subject Release [56]; Generic Object Decoding Dataset (GOD) [21]; and Brain, Object, Landscape Dataset (BOLD5000) [5]. Our upstream pre-training dataset comprised fMRI data from HCP and GOD. Combining these two, we obtained 136,000 fMRI segments from 340 hours of fMRI scan, which is, by far, the largest fMRI pre-training dataset in the fMRI-image decoding task. The HCP dataset is commonly used in neuroscience research, containing only fMRI data. While the GOD is an fMRI-image paired dataset designed for fMRI-based decoding. The pairs in GOD were used for finetuning in our main analysis. The GOD consists of 1250 different images from 200 distinct classes, in which 1200 images were used as the training set, and the remaining 50 images were used as the testing set. The training set and testing set have no overlapping classes. The BOLD5000 dataset was used as the validation dataset in our study. It consists of 5254 fMRI-image pairs from 4916 distinct images, 113 images of which are used for testing. This is the first time that the BOLD5000 is used for fMRI decoding tasks.
Implementation The fMRI pre-training model is similar to ViT-Large [10] with a 1D patch embedder. We used a patch size of 16, embedding dimension of 1024, encoder depth of 24, and mask ratio of 0.75 as our Full model setting with an ImageNet class-conditioned pre-trained LDM. Different parameter choices are explored in our ablation study. Unless stated otherwise, the Full model is pre-trained for 500 epochs and finetuned for another 500. Results from the best model are reported. Images are generated at a resolution of 256 × 256 with 250 PLMS steps [30]. See Appendix for dataset and implementation details.

Evaluation Metric
N-way Classification Accuracy Following [16], we used the n-way top-1 and top-5 accuracy classification task to evaluate the semantic correctness of our results, where for multiple trials, top-1 and top-5 classification accuracies were calculated in n−1 randomly selected classes plus the correct one. Note that we did not consider the pixel-level metrics as we aimed to recover the semantically correct images in this work.
In [16], the authors generated a typical feature for each class selected and compared the distance between the reconstructed images and the typical features. However, this metric in [16] is hard to reproduce, and the semantic classification result largely depends on how the features are computed. Therefore, we propose a more straightforward and reproducible method, where a pre-trained ImageNet1K classifier [10,36] is used to determine the semantic correctness of generated images rather than handcrafted features. We describe this evaluation method in Algorithm D.1. Specifically, both ground-truth and generated images are input to the classifier first. Then we check for the generated image if the top-k classification in n selected classes matches the ground-truth classification. This metric does not require the ground-truth image to be from the ImageNet 1k classes. As long as semantic classification results of the ground-truth and the generated image match, it will be considered to be correct.

Fréchet inception distance (FID)
The FID [19] is a commonly used metric to assess image generation quality. In our experiments, we measured the FID between ground-truth images and generated images in the testing set. Note that FID is only used as a reference in our experiments due to the limited number of images available in GOD, which may lead to an underestimated distribution.

Results
Our main results are based on GOD which has no overlapping classes in the training and testing set. The training and testing were performed on the same subject, as individual differences remain a barrier when decoding at the group level [2,16,21,32,34]. To compare with the literature, we report results from Subject 3 here and leave other subjects in the Appendix.
We compared our results with Ozcelik et al. [34], Gaziv et al. [16] and Beliy et al. [16]. Gaziv et al. and Beliy et al. used the conventional method, which decoded images with higher pixel similarity but less plausibility and semantic details. On the other hand, Ozcelik et al. generated more plausible and semantically meaningful images using a pre-trained GAN. Based on the bestreconstructed samples of these methods (resized to 256×256), we performed a 1000-trial, n-way top-k accuracy identification task as described in Algorithm D.1. The experiment is repeated for n=50,100 and k =1,5 in the GOD testing set.
From  details, as could be observed in Fig. 5. For example, our method generated plausible details such as water and waves in the first and second images, drawings on the bowling ball, wheels of the carriage, etc., which were not present in the previous decoded images. The image quality is also reflected by the FID, where we achieved 1.67 with our best samples, while Ozcelik et al. and others achieved 2.36 or more with the best samples generated by their method. Interestingly, color mismatches are observed in some cases with the color difference well preserved. It can be explained with [3] which suggests the color category information is processed in the frontal lobes as a cognitive process, while the visual cortex only recognizes the difference in colors.

Generation Consistency
The consistency of our method was tested by decoding the same fMRI data multiple times with different random states. Five samplings with different random states were performed in the testing set for each fMRI. In the 50-way and the 100-way top-1 accuracy identification tasks, we achieved an average success rate across the five samplings of 0.2385±0.030 and 0.1736±0.029 respectively, which are statistically higher than the best sampling results from Ozcelik et al. by 21% and 35%. Regarding image quality, we achieved an average FID of 2.22±0.3 across the five samplings. The standard deviations across 5 samplings indicate that the generated images will always be in the same semantic category. It can also be seen in Fig. 7 where isomorphic samplings share similar details such as shape, color, texture, and semantics, matching with the ground-truth across trials.

SC-MBM Design
This section will discuss the ablation study on the SC-MBM pre-training stage with various important parameters. Results are summarized in Tab. 1. For all experiments in this section, the 50-way, top-1 accuracy semantic identification task was performed with the best models obtained from the finetuning of 500 epochs. Average results over five samplings were reported.
Testing Without SC-MBM To show that useful representations were learned with SC-MBM, we trained two models directly using the fMRI-image pairs without the SC-MBM pretraining. The first model consisted of an untrained fMRI encoder with the same architecture as the Full model. The second model consisted of an untrained fMRI encoder with a depth of only 2. The second model was designed to have fewer parameters, making it less likely to overfit the data. All the other settings were the same. The results correspond to Model 1 and 2 in Tab. 1, where the Full model significantly outperformed the other two models without the SC-MBM pre-training, showing that the pre-training is crucial. In fact, without SC-MBM these two models even failed to generate sensible images (See Appendix).

Model
Embedding Dim

Patch Embedding Dimension
Boosting the fMRI representation size using a large patch embedding matches the sparse coding mechanism of underlying visual information processing in the brain. Moreover, using a large patch embedding increases the information capacity of the representation. But larger embedding means more training parameters leading to a more data-hungry model. To balance this tradeoff, we tested SC-MBM models with different patch embedding dimensions ranging from 32 to 1280 (Model 3-9 in Tab. 1). We found that the accuracy generally increased as patch dimension increased, and accuracy peaked at 23.9% with 1024 patch embedding dimensions (full model), after which accuracy decreased as patch dimensions increased further.

Mask Ratios
We used a high mask ratio in SC-MBM due to high spatial redundancy in fMRI data. In Tab. 1 Model 10-14, we show that a high mask ratio does not impair the decoding performance initially, with the highest average accuracy achieved with a relatively high mask ratio of 0.75. Importantly, using a high mask ratio significantly reduces memory consumption since the encoder only operates over unmasked patches. This is an important consideration for fMRI as SC-MBM is more memory-intensive than MIM due to the higher embedding-to-patch-size ratio.

DC-LDM Finetuning Design
This section will discuss the ablation study on the DC-LDM finetuning designs from three perspectives: conditioning methods, optimization designs, and pre-trained LDMs. Here, all ablations used the same pre-trained fMRI encoder as the Full model. Only important parameters in the finetuning stage were varied. The 1000-trial, 50

Conditioning Methods
Here, we showed that the double conditioning method increased the conditioning strength in Tab. 2, where using only cross-attention conditioning achieved an identification accuracy of 15.6% (Model 1), which was significantly lower than the full method.
Optimizing LDM We proposed to finetune the fMRI encoder and the cross-attention heads jointly because the LDM was pre-trained in a different conditioning space. For example, for the ImageNet class-conditioning pre-trained LDM, the cross-attention heads were pre-trained to receive the class label information. To justify this choice, we tested on a model with the fMRI encoder finetuned and the cross-attention heads untouched. As shown in Model 2 in Tab. 2, the average identification accuracy dropped to 13.7% when only the fMRI encoder was finetuned, indicating stronger semantic guidance with the double conditioning. The visual quality and correspondence to the ground-truth of the generated images also decreased significantly (See Appendix).
Pre-trained LDM The pre-trained LDM determines the model's generative ability and the conditioning latent space to which the fMRI encoder would adapt. We considered three pretrained LDM provided by [42], which were trained on datasets with different conditioning tasks, i.e. ImageNet (label conditioning), LAION (text conditioning) [45] and OpenImages (layout conditioning) [24]. As shown in Model 3-4 Tab. 2, the ImageNet pre-trained LDM (used in the full model) showed the best performance in the same decoding task. Notably, images generated by models pre-trained on LAION and OpenImages were less visually favorable and plausible (See Appendix). This result is surprising because both LAION and OpenImages contain diverse images from various categories. We attribute the main reason for their poor performance to the complexity of their conditioning latent space. With limited training pairs, the class-conditioning latent space is easier to adapt to, compared with the latent space of the text-conditioning model and the layout-conditioning model.

Replication Dataset
We validated our method on BOLD5000 using the same pretrained fMRI encoder. Similarly, the pre-trained encoder was firstly finetuned for 20 epochs in the testing set of BOLD500 with wrap-around paddings to compensate for the unequal ROI size from the pre-training set, after which the model is further tuned with the fMRI-image training pairs in BOLD5000. All other settings were the same as the Full model. For the four subjects in BOLD5000, we achieved a 19% to 34% best accuracy in the 1000-trial, 50-way, top-1 accuracy semantic identification task (See Appendix). The generated images matched the ground-truth stimulus in both semantics and low-level features (Fig. 8). Our model accurately reconstructs images containing objects and animals, architecture, and landscapes.
Interestingly, we reconstructed similar images for some natural scenes with extra details that do not exist in the ground-truth stimulus. These extra details, for example, the river and the blue sky in Fig. 9, may reflect imagined scenery in the subject's mind when viewing the visual stimuli, which is captured in their brain activities. As reported in [21,47], features of imaginary images can also be decoded from the visual cortex.
To the best of our knowledge, this is the first work that performs fMRI decoding on BOLD5000. Additionally, adapt- Figure 9. Extra Features Decoded. Imagery-related details can be decoded with our method. e.g. the river and blue sky were decoded with natural scenery stimulus (top row); similar interior decorating of indoor environments was decoded when a house was presented (bottom row).
ing the same pre-trained model to this dataset shows that the SC-MBM pre-training indeed learns useful representations of brain recordings even when distinct domain shifts exist. These learned representations are shared and generalizable to datasets with different scanning protocols and preprocessing pipelines.

Discussion and Conclusion
Discussion In our experiments, we observed that image reconstruction quality varied across subjects, which is common in brain decoding research. This discrepancy may be driven by differences in the on-line processes of individual brains, namely differences in the visual perception process anatomically and functionally. Previous work observed that BOLD fluctuation is associated with individual task performance [25,26]. Better task performance and learning ability are usually linked with more hierarchical architecture in information processing. Thus it may be easier to decode subjects' brain activities with better information processing ability (i.e. good learners). Although our method can achieve satisfactory results in many cases, there are still some negative decoding samples (See Appendix). One possible reason is that the participants might have stimulus-independent imagery when they are in the scanner, which is reflected by the visual cortex. On the other hand, the training set and testing in the GOD have no overlapping classes, which means even though we can recover correct geometric information, the class information cannot be inferred in the testing set.
Conclusion We proposed a two-stage framework MinD-Vis to decode visual stimuli using only a few paired fMRI-image annotations from brain recordings. In Stage A, we used an fMRI pre-training scheme with masked modeling as a pretext task to learn generalizable context knowledge in an enlarged representation space. We employed a latent diffusion model with double conditioning in Stage B to generate plausible seen images from learned fMRI representations. We validated the decoding results of MinD-Vis on multiple datasets and showed that compared with previous methods, our model generates more plausible images that are semantically similar to the original stimuli, thus establishing a new state-of-the-art.  As discussed in the main text, we assume the failure cases are related to two reasons. On one hand, the GOD training set and testing set have no overlapping classes. That is to say, the model could learn the geometric information from the training but cannot infer unseen classes in the testing set. On the other hand, subjects might have some other stimuli-unrelated thoughts, which could be captured by fMRI and decoded by our method Human Connectome Project (HCP) 1200 Subject Release [56]: large-scale magnetic resonance imaging dataset used for pre-training. We utilized around 2000×15-min 3T resting state fMRI runs from 1091 subjects. The visual cortex (V1-V4) defined in [17] is used as the ROI, which gives approximately 4000 voxels.
Generic Object Decoding Dataset (GOD) [21]: human fMRI scans with 1250 distinct images from ImageNet as a visual stimulus. During the fMRI scan, subjects were instructed to fixate on a cross located at the center of the presented images. This dataset consists of 1250 natural images from 200 distinct classes from ImageNet, where 1200 images are used for training. The remaining 50 images from classes not present in the training data are used for testing. Each image in the training set is presented once to the subject during the scan, while each image in the testing set is presented 35 times. Following the preprocessing in [21], the 35 repetitions are averaged for each image to create a higher SNR fMRI sample for testing. This dataset is widely used in brain image decoding [2,13,16,32,46,47]. We used the manually defined ROIs (V1-V4, FFA, LOC, HVC) from the functional localizer runs provided in [21]. Altogether, the ROIs have around 4500 voxels per subject, with some individual variance as shown in Fig. 2.
Brain, Object, Landscape Dataset (BOLD5000) [5]: human fMRI study with 5,254 fMRI-image pairs from 4,916 distinct natural images (including various objects and indoor/outdoor scenes) from Scene UNderstanding (SUN) [62], Common Objects in Context (COCO) [29] and ImageNet [7]. In this dataset, 4,803 images are presented once, and 113 images are repeated twice or three times. The repeated data are also averaged to construct the testing set as in the GOD dataset. To the best of our knowledge, this dataset is the first time is applied to an image reconstruction task. The author also provided manually annotated ROIs based on a functional localizer. As a result of different scanning resolutions and ROI definition methods, the number of voxels in the defined ROIs is approximately 1,500 for each subject. Nonetheless, we show in our results that the pre-trained encoder can be directly applied to this dataset despite this difference in ROI definition and size.
Pre-training dataset Our upstream pre-training dataset is comprised of fMRI recordings from HCP and GOD. Following the processing step in target dataset [21], we averaged every 8.64 seconds (i.e. 12 time frames) of scans from HCP, which gives 130,000 fMRI time points. Including the training and testing fMRI in GOD, we have a pure fMRI dataset of 136,000 samples for pre-training. This pre-training dataset is, by far, the largest pre-training fMRI dataset used in this task.
To handle the different voxel numbers, all fMRI data are first padded to the maximum length in a wrap-around manner and then padded to the boundary of the patch size. Additionally, training fMRI is normalized to have zero mean and unit standard deviation. The testing samples are normalized with the mean and standard deviation from the training set.

C. Results on Different Subjects
The GOD consists of five different subjects, and the BOLD5000 consists of four subjects. The signal-to-noise ratio (SNR) is usually used to quantify the quality of a dataset. A higher SNR means better data quality. As reported by their authors respectively, the BOLD5000 has a much higher SNR than GOD. Within the GOD, the SNR differs among subjects as shown in Tab. C.1, where Subject 3 has a significantly higher SNR than the others. A higher SNR leads to better performance in our experiments, which has also been shown in various literature. Other than possible noise introduced during the scan, the SNR is also related to the subjects' on-line processing or information processing ability. Subjects with better information processing ability (i.e. better learners) will have a higher SNR during the scan under the same scanning conditions. D. More Implementation Details

D.1. Evaluation Metric Implementation
This algorithm performs the N-trial, n-way top-1 semantic classification test. It measures the semantic accuracy of generated images. We describe our evaluation method in Algorithm D.1, where the generated image and its corresponding ground-truth image are denoted by x andx respectively, and y is for the class label. This metric relies on a pre-trained ImageNet classifier to determine whether x andx belong to the same class rather than using handcrafted features to represent each class. This method is thus reasonable and easily reproducible. We used a pre-trained ResNet as the classifier. We also showed that using other model based pre-trained classifiers will not change the result of this metric.

D.2. SC-MBM Pre-training
In Masked Image Modeling (MIM) [18], images are divided into patches which are sequentially transformed into embeddings to adapt to a transformer-based architecture [10]. Following this practice, we divided fMRI voxels into patches which will be subsequently transformed into embeddings using a one-dimensional convolutional layer with a stride of the patch size.
A patch size of 16 and an embedding dimension of 1024 were used as the Full model. Notice that our embedding size to patch size ratio is much larger than that of MIM. For example in [18], the authors used a patch size of 16 and embedding dimension 768, which gave an embedding to patch dimension ratio: 768/(16·16·3) = 1, compared to ours: 1024/(16) = 64. This design largely expands the representation dimension of fMRI data, significantly boosting the information capacity of the fMRI representations. This design is justified by both considering the dimension gap between fMRI and natural images, as well as the hypothesis of sparse coding in the visual encoding process.
Following [63], we adopt an asymmetric architecture where the decoder is much smaller than the encoder. Before feeding patch embeddings to the encoder, a random portion is masked. We used a large mask ratio similar to the mask ratio used in MIM due to the similarity in information density between fMRI data and images. We additionally embed mask tokens and include positional embeddings along with the patch encodings at the end of the encoder and transform them into the decoder's embedding space via a linear projector. On the other hand, our decoder aims to recover the masked patches with the voxel value as the prediction target.
To train the data-hungry model like the ViT, we also applied random sparsification (RS) for data augmentation, where 20% of voxels in each fMRI were randomly selected and set to zero.
Hyperparameters used in the SC-MBM pre-training stage are listed in Tab. D.2. All other unlisted parameters are set to their defaults. The SC-MBM pre-training is performed on 8 RTX3090ti GPUs until the model converges. Examples of masked brain prediction are shown in Fig. D

D.3. DC-LDM Finetuning
The finetuning is performed by jointly optimizing the fMRI encoder and cross-attention heads in the LDM using the training set. Specifically, for an fMRI-image pair, the image will be encoded into the latent space via a VQ encoder, which will be subsequently used as an objective to train the fMRI encoder and cross-attention heads. In the forward pass, fMRI data is passed through the encoder, producing a patchified enlarged representation. Then this representation is projected into an intermediate space with a Operating in the image latent space, the computations needed for DC-LDM finetuning are small. All finetunings in our experiments are performed with a single RTX3090ti GPU for 500 epochs. The detailed hyperparameters are shown in Tab. D.3. All other unlisted parameters are set to their defaults. Please see [42]

E. Other Ablation Studies
Patch Sizes In [10], an image is divided into sixteen 16×16 patches which can be considered 16 words. Analogous to the fMRI data, the more words are used to describe the data, the higher accuracy of the resulting representation will be. Therefore, smaller patches will lead to better results if the number of voxels remains unchanged. This claim is justified by Tab. E.4. A continuous decrease in accuracy can be observed when the patch size is increased from 16 to 64. However, the minimal patch size applicable is constrained by available memory, as the number of patches increases drastically with smaller patch size.

Encoder Depth
The fMRI encoder depth is set to 24 in our Full model similar to the ViT-Large [10]. However, different depths lead to a different number of parameters and encoding capabilities. Usually, a deeper model is appreciated, but it comes with the need for more training samples as well. Therefore, considering the limited data, we explore whether a smaller model would have better results. To maintain an asymmetrical architecture, the depth of the SC-MBM decoder is kept at half of the encoder's depth.
A deeper fMRI encoder (as deep as 24 transformer blocks) gives the best result as shown in Tab. E.4.

Mask Stragtey
In [18], different masking strategies are tested for images, and the authors conclude that random masking is the best strategy for images. We explore in our ablation if it is the case for fMRI learning. For images, there are different strategies such as center masking and grid masking due to the geometric correlations among pixels in an image. For fMRI data, brain activities are reflected by the connectivity among groups of voxels (functional networks). Seven networks in the visual cortex are used in our study (i.e. V1-V4, FFA, PPA, and LOC), in which the V1, the primary visual cortex, consists of the most voxels and is the first stage of visual processing. Therefore we design a focus masking strategy similar to the center masking in images. In the center masking, pixels at the center of an image will be masked the most because the center of an image usually contains the richest information. Learning to recover the center potentially is beneficial to learning the underlying semantics of an image. Similar to the center masking, our focus masking in fMRI masks more patches in the V1 region than in other regions. However, in our experiments, the focus masking does not outperform the random masking strategy as shown in Tab. E.4.
Pretext Tasks As discussed, since fMRI voxels are correlated reflecting the underlying brain activities, masked modeling is a suitable learner for fMRI representations. With SC-MBM as a pretext task, self-supervised learning can be performed in a large unpaired fMRI dataset. On the other hand, considering a small part of paired fMRI are available in the training set. Therefore, it is intriguing if we can use the paired information in fMRI for the pre-training as well. So we include the image feature as another pretext task together with the masked modeling to guide the context learning. Specifically, the training set will be divided into two parts: the part with paired images; and the part without paired images. To construct a mini-batch, we randomly sample fMRI from these two parts.
In this design, another decoder is added to decode image features. The image features extracted from the second layer of a pre-trained VGG will be used as a target for this decoder. During training, the image feature reconstruction loss will be added to the MBM loss with a regularization term. However, adding the image guidance does not outperform the MBM only pre-training as shown in Tab. E.4.
Unequal Length Handle Due to individual variability, even in the same dataset, the voxel numbers of individuals are different. We need to handle this unequal length to include different subjects in the pre-training. The two most intuitive ways are considered: pad to the maximum length with a constant; cut to the minimum length. Besides padding with a constant, we pad the data in a wrap-around manner. From Tab. E.4 we can see that cutting the data gives the worst performance and wrap-around padding gives the best performance.
Crop Ratio In the finetuning, images are randomly center-cropped for augmentations. We tested different crop ratios, i.e. from 0 to 0.4. It is found that a crop ratio of 0.2 gives the best performance. Random cropping is an efficient augmentation in our task because the subjects' perceptions may be focused on different parts of the figure, even though they were instructed to fixate at the center of the image.  Table E.4. Other Design Ablations. The 1000-trial 50-way top-1 semantic classification accuracy is reported. All ablations are pre-trained for 500 epochs and then finetuned on GOD for another 500 epochs. Settings used in the Full model are colored in gray.

F. Extra Notes on Sparse-Coded Masked Brain Modeling
In our design, we use a large embedding-size-to-patch-size ratio to increase the information capacity of fMRI representations, which mimics the sparse coding mechanism underlying the encoding procedure of the visual cortex. Here, we provide a formal definition of the information capacity and explain the connection with the sparse coding mechanism.
Definition 1 (Data Representation) For a piece of data given by a one-dimensional code vector x∈R L , let f be an injective function that maps x from the data domain to a representation domain, namely f(x)=y, where y ∈RL is a representation of x.
Definition 2 (Information Capacity) For a random variable X, the Shannon entropy of X is upper bounded by its cardinality, which is given by H(X)≤log(|X |). We define log(|X |) as the information capacity of random variable X.
The inequality in Definition 2 can be easily proved with Jensen's inequality regardless of the distribution of X. Obviously, for a representation Y , if the dimension of Y is larger, the representation space will be larger. Hence, Y will have a larger information capacity.
To measure the change of information capacity after the representation mapping, we can simply divide the representation dimension by the data dimension, namely, R=L/L. In the context of masked modeling, we refer to R as the embedding-size-to-patch-size ratio. The essence of sparse coding is to use sets of over-complete bases to efficiently represent data [27]. Analogous to this over-completeness, we use a representation space that is much larger than the data space, namely higher R, to learn the representations of the fMRI. Data locality is included in the representations by dividing the fMRI time series into patches and transforming patches into embeddings.