{'fraction_non_alphanumeric': 0.03470303529089021, 'fraction_numerical': 0.015401798836047264, 'mean_word_length': 4.906029394745978, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Foreground Extraction via Deep Region Competition

We present Deep Region Competition (DRC), an algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background. In this work, we rethink the foreground extraction by reconciling energy-based prior with generative image modeling in the form of Mixture of Experts (MoE), where we further introduce the learned pixel re-assignment as the essential inductive bias to capture the regularities of background regions. With this modeling, the foreground-background partition can be naturally found through Expectation-Maximization (EM). We show that the proposed method effectively exploits the interaction between the mixture components during the partitioning process, which closely connects to region competition, a seminal approach for generic image segmentation. Experiments demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Moreover, we show empirically that DRC can potentially generalize to novel foreground objects even from categories unseen during training.


Introduction
Foreground extraction, being a special case of generic image segmentation, aims for a binary partition of the given image with specific semantic meaning, i.e., a foreground that typically contains identifiable objects and the possibly less structural remaining regions as the background. There is a rich literature on explicitly modeling and representing a given image as foreground and background (or more general visual regions), such that a generic inference algorithm can produce plausible segmentations ideally for any images without or with little supervision [1][2][3][4][5][6][7][8]. However, such methods essentially rely on low-level visual features (e.g., edges, color, and texture), and some further require human intervention at initialization [4,5], which largely limits their practical performance on modern datasets of complex natural images with rich semantic meanings [9,10]. These datasets typically come with fine-grained semantic annotations, exploited by supervised methods that learn representation and inference algorithm as one monolithic network [11][12][13][14][15][16]. Despite the success of densely supervised learning, the unsupervised counterpart is still favored due to its resemblance to how humans perceive the world [17,18].
Attempting to combine unsupervised or weakly supervised learning with modern neural networks, three lines of work surge recently for foreground extraction: (1) deep networks as feature extractors for canonical segmentation algorithms, (2) GAN-based foreground-background disentanglement, and (3) compositional latent variable models with slot-based object modeling. Despite great successes of these methods, the challenge of unsupervised foreground extraction remains largely open.
Specifically, the first line of work trains designated deep feature extractors for canonical segmentation algorithms or metric networks as learned partitioning criteria [19][20][21]. These methods (e.g., W-Net [19]) define foreground objects' properties using learned features or criteria and are thus generally bottle-necked by the selected post-processing segmentation algorithm [22,23]. As a branch of pioneering work that moves beyond these limitations, Yang et al. [24,25] have recently proposed a general contextual information separation principle and an efficient adversarial learning method that is generally applicable to unsupervised segmentation, separation and detection. GAN-based models [26][27][28][29][30][31] capture the foreground objectness with oversimplified assumptions or require additional supervision to achieve foreground-background disentanglement. For example, the segmentation model in ReDO [28] is trained by redrawing detected objects, which potentially limits its application to datasets with diverse object shapes. OneGAN [31] and its predecessors [29,30], though producing impressive results on foreground extraction, require a set of background images without foreground objects as additional inputs. Lastly, compositional latent variable models [32][33][34][35][36][37][38][39][40] include the background as a "virtual object" and induce the independence of object representations using an identical generator for all object slots. Although these methods exhibit strong performance on synthetic multi-object datasets with simple backgrounds and foreground shapes, they may fail on complex real-world data or even synthetic datasets with more challenging backgrounds [37,38]. In addition, few unsupervised learning methods have provided explicit identification of foreground objects and background regions. While they can generate valid segmentation masks, most of these methods do not specify which output corresponds to the foreground objects. These deficiencies necessitate rethinking the problem of unsupervised foreground extraction. We propose to confront the challenges in formulating (1) a generic inductive bias for modeling foreground and background regions that can be baked into neural generators, and (2) an effective inference algorithm based on a principled criterion for foreground-background partition.
Inspired by Region Competition [1], a seminal approach that combines optimization-based inference [41][42][43] and probabilistic visual modeling [44,45] by minimizing a generalized Bayes criterion [46], we propose to solve the foreground extraction problem by reconciling energy-based prior [47] with generative image modeling in the form of Mixture of Experts (MoE) [48,49]. To generically describe background regions, we further introduce the learned pixel re-assignment as the essential inductive bias to capture their regularities. Fueled by our modeling, we propose to find the foreground-background partition through Expectation-Maximization (EM). Our algorithm effectively exploits the interaction between the mixture components during the partitioning process, echoing the intuition described in Region Competition [1]. We therefore coin our method Deep Region Competition (DRC). We summarize our contributions as follows: 1. We provide probabilistic foreground-background modeling by reconciling energy-based prior with generative image modeling in the form of MoE. With this modeling, the foregroundbackground partition can be naturally produced through EM. We further introduce an inductive bias, pixel re-assignment, to facilitate foreground-background disentanglement.
2. In experiments, we demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Furthermore, we empirically show that using learned pixel re-assignment as the inductive bias helps to provide explicit identification for foreground and background regions.
3. We find that DRC can potentially generalize to novel foreground objects even from categories unseen during training, which may provide some inspiration for the study of out-of-distribution (OOD) generalization in more general unsupervised disentanglement.

Related Work
A typical line of methods frames unsupervised or weakly supervised foreground segmentation within a generative modeling context. Several methods build upon generative adversarial networks (GAN) [26] to perform foreground segmentation. LR-GAN [27] learns to generate background re-gions and foreground objects separately and recursively, which simultaneously produces the foreground objects mask. ReDO (ReDrawing of Objects) [28] proposes a GAN-based object segmentation model, based on the assumption that replacing the foreground object in the image with a generated one does not alter the distribution of the training data, given that the foreground object is correctly discovered. Similarly, SEIGAN [29] learns to extract foreground objects by recombining the foreground objects with the generated background regions. FineGAN [30] hierarchically generates images (i.e., first specifying the object shape and then the object texture) to disentangle the background and foreground object. Benny and Wolf [31] further hypothesize that a method solving an ensemble of unsupervised tasks altogether improves the model performance compared with the one that solves each individually. Therefore, they train a complex GAN-based model (OneGAN) to solve several tasks simultaneously, including foreground segmentation. Although LR-GAN and FineGAN do produce masks as part of their generative process, they cannot segment a given image. Despite SEIGAN and OneGAN achieving decent performance on foreground-background segmentation, these methods require a set of clean background images as additional inputs for weak supervision. ReDO captures the foreground objectness with possibly oversimplified assumptions, limiting its application to datasets with diverse object shapes.
On another front, compositional generative scene models [32][33][34][35][36][37][38][39][40], sharing the idea of scene decomposition stemming from DRAW [50], learn to represent foreground objects and background regions in terms of a collection of latent variables with the same representational format. These methods typically exploit the spatial mixture model for generative modeling. Specifically, IODINE [37] proposes a slot-based object representation method and models the latent space using iterative amortized inference [51]. Slot-Attention [38], as a step forward, effectively incorporates the attention mechanism into the slot-based object representation for flexible foreground object binding. Both methods use fully shared parameters among individual mixture components to entail permutation invariance of the learned multi-object representation. Alternative models such as MONet [36] and GENESIS [39] use multiple encode-decode steps for scene decomposition and foreground object extraction. Although these methods exhibit strong performance on synthetic multi-object datasets with simple background and foreground shapes, they may fail when dealing with complex real-world data or even synthetic datasets with more challenging background [37,38].
More closely related to the classical methods, another line of work focuses on utilizing image features extracted by deep neural networks or designing energy functions based on data-driven methods to define the desired property of foreground objects. Pham et al. [52] and Silberman et al. [53] obtain impressive results when depth images are accessible in addition to conventional RGB images, while such methods are not directly applicable for data with RGB images alone. W-Net [19] extracts image features via a deep auto-encoder jointly trained by minimizing reconstruction error and normalized cut. The learned features are further processed by CRF smoothing to perform hierarchical segmentation. Kanezaki [20] proposes to employ a neural network as part of the partitioning criterion (inspired by Ulyanov et al. [54]) to minimize the chosen intra-region pixel distance for segmentation directly. Ji et al. [21] propose to use Invariant Information Clustering as the objective for segmentation, where the network is trained to be part of the learned distance. As an interesting extension, one may also consider adapting methods that automatically discover object structures [55] to foreground extraction. Though being pioneering work in image segmentation, the aforementioned methods are generally bottle-necked by the selected post-processing segmentation algorithm or require extra transformations to produce meaningful foreground segmentation masks. Yang et al. [24,25] in their seminal work propose an information-theoretical principle and adversarial contextual model for unsupervised segmentation and detection by partitioning images into maximally independent sets, with the objective of minimizing the predictability of one set by the other sets. Additional efforts have also been devoted to weakly supervised foreground segmentation using image classification labels [56][57][58], bounding boxes [59,60], or saliency maps [61][62][63].

Methodology
Foreground extraction performs a binary partition for the image I to extract the foreground region. Without explicit supervision, we propose to use learned pixel re-assignment as a generic inductive bias for background modeling, upon which we derive an EM-like partitioning algorithm. Compared with prior methods, our algorithm can handle images with more complex foreground shapes and background patterns, while providing explicit identification of foreground and background regions.

Preliminaries
Adopting the language of EM algorithm, we assume that for the observed sample x P R D , there exists z P R d as its latent variables. The complete-data distribution is where p α pzq is the prior model with parameters α, p β px|zq is the top-down generative model with parameters β, and θ " pα, βq.
The prior model p α pzq can be formulated as an energy-based model, which we refer to as the Latentspace Energy-Based Model (LEBM) [47] throughout the paper: where f α pzq can be parameterized by a neural network, Z α is the partition function, and p 0 pzq is a reference distribution, assumed to be isotropic Gaussian prior commonly used for the generative model. The prior model in Eq. (2) can be interpreted as an energy-based correction or exponential tilting of the original prior distribution p 0 .
The LEBM can be learned by Maximum Likelihood Estimation (MLE). Given a training sample x, the learning gradient for α is derived as shown by Pang et al. [47], In practice, the above expectations can be approximated by Monte-Carlo average, which requires sampling from p θ pz|xq and p α pzq. This step can be done with stochastic gradient-based methods, such as Langevin dynamics [64] or Hamiltonian Monte Carlo [65].
An extension to LEBM is to further couple the vector representation z with a symbolic representation y [66]. Formally, y is a K-dimensional one-hot vector, where K is the number of possible z categories. Such symbol-vector duality can provide extra entries for auxiliary supervision; we will detail it in Section 3.4.

Generative Image Modeling
Mixture of Experts (MoE) for Image Generation Inspired by the regional homogenity assumption proposed by Zhu and Yuille [1], we use separate priors and generative models for foreground and background regions, indexed as α k and β k , k " 1, 2, respectively; see Fig. 1. This design leads to the form of MoE [48,49] for image modeling, as shown below.
Let us start by considering only the i-th pixel of the observed image x, denoted as x i . We use a binary one-hot random variable w i to indicate whether the i-th pixel belongs to the foreground region. Formally, we have w i " rw i1 , w i2 s, w ik P t0, 1u and ř 2 k"1 w ik " 1. Let w i1 " 1 indicate that the i-th pixel x i belongs to the foreground, and w i2 " 1 indicate the opposite.
We assume that the distribution of w i is prior-dependent. Specifically, the mixture parameter π ik , k " 1, 2, is defined as the output of a gating function π ik " p β pw ik " 1|zq " Softmaxpl ik q; l ik " h β k pz k q, k " 1, 2 are the logit scores given by the foreground and background generative models respectively; β " tβ 1 , β 2 u, z " tz 1 , z 2 u. Taken together, the joint distribution of w i is The learned distribution of foreground and background contents are p β px i |w ik " 1, z k q " p β k px i |z k q " Npg β k pz k q, σ 2 Iq, k " 1, 2 where we assume that the generative model for region content, p β k px i |z k q, k " 1, 2, follows a Gaussian distribution parameterized by the generator network g β k . As in VAE, σ takes an assumed value. We follow the common practice and use a shared generator for parameterizing π ik and p β k px i |z k q. We use separate branches only at the output layer to generate logits and contents. Generating x i based on w i 's distribution involves two steps: (1) sample w i from the distribution p β pw i |zq, and (2) choose either the foreground model (i.e., p β1 px i |z 1 q) or the background model (i.e., p β2 px i |z 2 q) to generate x i based on the sampled w i . As such, this distribution of x i is a MoE, wherein the posterior responsibility of w ik is Using a fully-factorized joint distribution of x, we have p β px|zq " ś D i"1 ř 2 k"1 π ik p β k px i |z k q as the generative modeling of x P R D . Apply the learned re-assignment on C Figure 2: Pixel re-assignment. The output of βp can be viewed as a learned re-assignment of the original background pixels that follows the mapped grid P αp,βp pz, Cq. Note that the reassignment function P αp,βp pz,¨q might not be injective. The final background image is generated via grid sampling.
Learning Pixel Re-assignment for Background Modeling We use pixel re-assignment in the background generative model as the essential inductive bias for modeling the background region. This is partially inspired by the concepts of "texture" and "texton" by Julez [45,67], where the textural part of an image may contain fewer structural elements in preattentive vision, which coincides with our intuitive observation of the background regions.
We use a separate pair of energy-based prior model α pix and generative model β pix to learn the re-assignment. For simplicity, we absorb α pix and β pix in the models for background modeling, i.e., α 2 and β 2 , respectively. In practice, the re-assignment follows the output of β pix , a shuffling grid with the same size of the image x. Its values indicate the re-assigned pixel coordinates; see Fig. 2. We find that shuffling the background pixels using the learned re-assignment facilitates the model to capture the regularities of the background regions.
Specifically, the proposed model with this essential inductive bias learns to constantly give the correct mask assignment, whereas most previous fully unsupervised methods do not provide explicit identification of the foreground and background regions; see discussion in Section 4.1 for more details.

Deep Region Competition: from Generative Modeling to Foreground Extraction
The complete-data distribution from the image modeling is where p α pzq " p α1 pz 1 qp α2 pz 2 q is the prior model given by LEBMs. α " tα 1 , α 2 u, and θ " tα, βu. w is the vector of pw i q, i " 1, ...D, whose joint distribution is assumed to be fully-factorized.
Next, we derive the complete-data log-likelihood as our learning objective: Of note, w and z are unobserved variables in the modeling, which makes it impossible to learn the model directly through MLE. To calculate the gradients of θ, we instead optimize E z"ppz|xq,w"ppw|x,zq rLpθqs based on the fact that underlies the EM algorithm: Therefore, the derived surrogate learning objective becomes J pθq " log p α pzq looomooon where J pθq " E w"p θ pw|x,zq rLpθqs is the conditional expectation of w, which can be calculated in closed form; see the supplementary material for additional details.
Eq. (11) has an intuitive interpretation. We can decompose the learning objective into three components as in Eq. (12). In particular, the second term ř D i"1 ř 2 k"1 γ ik log π ik has a similar form to the cross-entropy loss commonly used for supervised segmentation task, where the posterior responsibility γ ik serves as the target distribution. It is as if the foreground and background generative models compete with each other to fit the distribution of each pixel x i . If the pixel value at x i fits better to the distribution of foreground, p β1 px i |z 1 q, than to that of background, p β2 px i |z 2 q, the model tends to assign that pixel to the foreground region (see Eq. (7)), and vice versa. This mechanism is similar to the process derived in Zhu and Yuille [1], which is the reason why we coin our method Deep Region Competition (DRC).
Prior to our proposal, several methods [1,37,38] also employ mixture models and competition among the components to perform unsupervised foreground or image segmentation. The original Region Competition [1] combines several families of image modeling with Bayesian inference but is limited by the expressiveness of the pre-specified probability distributions. More recent methods, including IODINE [37] and Slot-attention [38], learn amortized inference networks for latent variables and induce the independence of foreground and background representations using an identical generator. Our method combines the best of the two worlds, reconciling the expressiveness of learned generators with the regularity of generic texture modeling under the framework of LEBM.
To optimize the learning objective in Eq. (11), we approximate the expectation by sampling from the prior p α pzq and posterior model p θ pz|xq9p α pzqp β px|zq, followed by calculating the Monte Carlo average. We use Langevin dynamics [64] to draw persistent MCMC samples, which iterates where t is the Langevin dynamics's time step, s the step size, and t the Gaussian noise. Qpzq is the target distribution, being either p α pzq or p θ pz|xq. ∇ z log Qpz t q is efficiently computed via automatic differentiation in modern learning libraries [68]. We summarize the above process in Algorithm 1.
Algorithm 1: Learning models of DRC via EM.
Input: Learning iterations T , initial parameters for LEBMs α p0q " tα p0q 1 , α p0q 2 u and generators β p0q " tβ p0q 1 , β p0q 2 u, θ p0q " tα p0q , β p0q u, learning rate ηα for LEBMs, η β for foreground and background generators, observed examples tx piq u N i"1 , batch size M , and initial latent variables Prior sampling for learning LEBMs: For each x piq , update z piq using Eq. (13), with target distribution πpzq " p α ptq pzq; 4 Posterior sampling for foreground and background generation: For each x piq , update z piq using Eq. (13), with target distribution Qpzq " p θ ptq pz|xq; Update foreground and background generators: During inference, we initialize the latent variables z for MCMC sampling from Gaussian white noise and run only the posterior sampling step to obtain z`. The inferred mask and region images are then given by the outputs of generative models p β k pw|z`q and p β k px|z`q, k " 1, 2, respectively.

Technical Details
Pseudo label for additional regularization Although the proposed DRC explicitly models the interaction between the regions, it is still possible that the model converges to a trivial extractor, which treats the entire image as the foreground or background region, leaving the other region null. We exploit the symbolic vector y emitted by the LEBM (see Section 3.1) for additional regularization. The strategy is similar to the mutual information maximization used in InfoGAN [69]. Specifically, we use the symbolic vector y inferred from z as the pseudo-class label for z and train an auxiliary classifier jointly with the above models; it ensures that the generated regions x k contain similar symbolic information for z k . Intuitively, this loss prevents the regions from converging to null since the symbolic representation y k would never be well retrieved if that did happen.
Implementation We adopt a similar architecture for the generator as in DCGAN [70] throughout the experiments and only change the dimension of the latent variables z for different datasets. The generator consists of a fully connected layer followed by five stacked upsample-conv-norm layers. We replace the batch-norm layers [71] with instance-norm [72] in the architecture. The energy-term in LEBM is parameterized by a 3-layered MLP. We adopt orthogonal initialization [73] commonly used in generative models to initialize the networks and orthogonal regularization [74] to facilitate training. In addition, we observe performance improvement when adding Total-Variation norm [75] for the background generative model. More details, along with specifics of the implementation used in our experiments, are provided in the supplementary material.

Experiments
We design experiments to answer three questions: (1) How does the proposed method compare to previous state-of-the-art competitors? (2) How do the proposed components contribute to the model performance? (3) Does the proposed method exhibit generalization on images containing unseen instances (i.e., same category but not the same instance) and even objects from novel categories?
To answer these questions, we evaluate our method on five challenging datasets in two groups: (1) Caltech-UCSD Birds-200-2011 (Birds) [76], Stanford Dogs (Dogs) [77], and Stanford Cars (Cars) [78] datasets; (2) CLEVR6 [79] and Textured Multi-dSprites (TM-dSprites) [80] datasets. The first group of datasets covers complex real-world domains, whereas the second group features environments of the multi-object foreground with challenging spatial configurations or confounding backgrounds. As to be shown, the proposed method is generic to various kinds of input and produces more competitive foreground-background partition results than prior methods.

Results on Foreground Extraction
Single  [31]. Of note, results of these two models on Dogs and Cars datasets may not be directly comparable to other listed methods, as the data used for training and evaluation could be different. We include these results as a rough reference since no official implementation or pretrained model are publicly available. § indicates unfair baseline results obtained using extra ground-truth information, i.e., we choose the best-matching scores from the permutation of foreground and background masks. †OneGAN is a strong weakly supervised baseline, which requires clean background images to provide additional supervision. We include this model as a potential upper bound of the fully unsupervised methods.
Single object in the wild In the first group of datasets, there is typically a single object in the foreground, varying in shapes, texture, and lighting conditions. Unsupervised foreground extraction on these datasets requires much more sophisticated visual cues than colors and shapes. Birds dataset consists of 11,788 images of 200 classes of birds annotated with high-quality segmentation masks, Dogs dataset consists of 20,580 images of 120 classes annotated with bounding boxes, and Cars dataset consists of 16,185 images of 196 classes. The latter two datasets are primarily made for fine-grained categorization. To evaluate foreground extraction, we follow the practice in Benny and Wolf [31], and approximate ground-truth masks for the images with Mask R-CNN [16], pre-trained on the MS COCO [9] dataset with a ResNet-101 [81] backend. The pre-trained model is acquired from the detectron2 [82] toolkit. This results in 5,024 dog images and 12,322 car images with a clear foreground-background setup and corresponding masks.
On datasets featuring a single foreground object, we use the 2-slot version of IODINE and Slotattention. Since ReDO, IODINE, and Slot-Attention do not distinguish foreground and background in output regions, we choose the best-matching scores from the permutation of foreground and background masks as in [28]. We observe that the proposed method and Grabcut are the only two methods that provide explicit identification of foreground objects and background regions. While the Grabcut algorithm actually requires a predefined bounding box as input that specifies the foreground region, our method, thanks to the learned pixel re-assignment (see Section 3.2), can achieve this in a fully unsupervised manner. Results in Table 1 show that our method outperforms all the unsupervised baselines by a large margin, exhibiting comparable performance even to the weakly supervised baseline that requires additional background information as inputs [31]. We provide samples of foreground extraction results as well as generated background and foreground regions in Fig. 3. Note that our final goal is not to synthesize appealing images but to learn foreground extractors in a fully unsupervised manner. As the limitation of our method, DRC generates foreground and background regions less realistic than those generated by state-of-the-art GANs, which hints a possible direction for future work. More detailed discussions of the limitation can be found in supplementary material. Multi-object scenes The second group of datasets contains images with possibly simpler foreground objects but more challenging scene configurations or background parts. Visual scenes in the CLEVR6 dataset contain various objects and often with partial occlusions and truncations. Following the evaluation protocol in IODINE and Slot-attention, we use the first 70K samples from CLEVR [79] and filter the samples for scenes with at most 6 objects for training and evaluation, i.e., CLEVR6. The TM-dSprites dataset is a variant of Multi-dSprites [80] but has strongly confounding backgrounds borrowed from Textured MNIST [32]. We generate 20K samples for the experiments. Similar to Greff et al. [37] and Locatello et al. [38], we evaluate on a subset containing 1K samples for testing. Note that IODINE and Slot-attention are designed for segmenting complex multi-object scenes using slot-based object representations. Ideally, the output of these models consists of masks for each individual object, while the background is viewed as a virtual "object" as well. In practice, however, it is possible that the model distributes the background over all the slots as mentioned in Locatello et al. [38]. We therefore propose two corresponding approaches (see the supplementary material for more details) to convert the output object masks into a foreground-background partition and report the best results of these two options for IODINE and Slot-attention in Table 1.
On the CLEVR6 dataset, we use the publicly available pretrained model for IODINE, which achieves a reasonable ARI (excluding background pixels) of 94.4 on the testing data, close to the testing results in Greff et al. [37]. We observe that IODINE distributes the background over all the slots for some of the testing samples, resulting in much lower IoU and Dice scores. We re-train the Slot-attention model using the official implementation on CLEVR6, as no pretrained model is publicly available. The re-trained model achieves a foreground ARI of 98.0 on the 1K testing samples, which we consider as a sign of valid re-implementation. Results in Table 1 demonstrate that the proposed method can effectively process images of challenging multi-object scenes. To be specific, our method demonstrates competitive performance on the CLEVR6 dataset compared with the SOTA object discovery method. Moreover, as shown empirically in Fig. 3, the proposed method can handle the strongly confounding background introduced in Greff et al. [32], whereas previous methods are distracted by the background and mostly fail to capture the foreground objects.  Table 2: Ablation study on Birds. *We replace the LEBM with encoders to perform amortized inference for the latent variables z within a variational framework as in VAE [83]. †We explore the possibility of using short-run MCMC [84] instead of persistent chain sampling.

Ablation Study
We provide ablation studies on the Birds dataset to inspect the contribution of each proposed component in our model. As shown in Table 2, we observe that replacing the LEBMs in the foreground and background models with amortized inference networks significantly harms the performance of the proposed method. In particular, the modified model fails to generate any meaningful results (marked as -in Table 2). We conjecture that LEBM benefits from the low-dimensionality of the latent space [47] and therefore enjoys more efficient learning. However, the inference networks need to learn an extra mapping from the high-dimensional image space to the latent space and require more elaborate architecture and tuning for convergence. Furthermore, we observe that the model that does not learn pixel re-assignment for background can still generate meaningful images but only vaguely captures masks for foreground extraction.

Generalizable Foreground Extraction
Extracting novel foreground objects from training categories We show results on generalizing to novel objects from the training classes. To evaluate our method, we split the Birds dataset following Chen et al. [28], resulting in 10K training images and 1K testing images. On Dogs and Cars datasets, we split the dataset based on the original train-test split [77,78]. This split gives 3,286 dog images and 6,218 car images for training, and 1,738 dog images and 6,104 car images for testing, respectively. As summarized in Table 3, our method shows superior performances compared with baselines; the performance gap between training and testing is constantly small over all datasets.   Extracting novel foreground objects from unseen categories To investigate how well our method generalizes to categories unseen during training, we evaluate the models trained in real-world single object datasets on the held-out testing data from different categories. We use the same training and testing splits on these datasets as in the previous experiments. Table 4 shows that our method outperforms the baselines on the Birds dataset when the model has trained on Dogs or Cars dataset, which have quite different distributions in foreground object shapes. Competitors like ReDO also exhibit such out-of-distribution generalization but only to a limited extent. Similar results are observed when using Dogs or Cars as the testing set. We can see that when the model is trained on Dogs and evaluated on Cars or vice versa, it still maintains comparable performances w.r.t. those are trained&tested on the same class. We hypothesize that these two datasets have similar distributions in foreground objects and background regions. In the light of this, we can further entail that the distribution of Dogs is most similar to that of Cars and less similar to that of Birds according to the results, which is consistent with our intuitive observation of the data. We provide a preliminary analysis of the statistics of these datasets in the supplementary material.

Conclusion
We have presented the Deep Region Competition, an EM-based fully unsupervised foreground extraction algorithm fueled by energy-based prior and generative image modeling. We propose learned pixel re-assignment as an inductive bias to capture the background regularities. Experiments demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes. We show empirically that learned pixel re-assignment helps to provide explicit identification for foreground and background regions. Moreover, we find that DRC can potentially generalize to novel foreground objects even from categories unseen during training. We hope our work will inspire future research along this challenging but rewarding research direction.

A.1 Caltech-UCSD Birds-200-2011
Birds dataset consists of 11,788 images of 200 classes of birds annotated with high-quality segmentation masks. Each image is further annotated with 15 part locations, 312 binary attributes, and 1 bounding box. We use the provided bounding box to extract a center square from the image, and scale it to 128ˆ128 pixels. Each scene contains exactly one foreground object.

A.2 Stanford Dogs
Dogs dataset consists of 20,580 images of 120 classes annotated with bounding boxes. We first use the provided bounding box to extract the center square, and then scale it to 128ˆ128 pixels. As stated in the paper, we approximate ground-truth masks for the pre-processed images with Mask R-CNN [16], pre-trained on the MS COCO [9] dataset with a ResNet-101 [81] backend. The pretrained model is acquired from the detectron2 [82] toolkit. We exclude the images where no dog is detected. We then manually exclude those images where the foreground object has occupied more than " 90% of the image, those with poor masks, and those with significant foreground distractors such as humans (see Fig. S1). The filtering strategy results in 5,024 images with a clear foreground-background setup and high-quality mask. Figure S1: Examples of excluded images. From left to right: (i) image with a foreground object that occupied too much space, (ii) image with a low-quality mask, and (iii) image with significant foreground distractors.

A.3 Stanford Cars
Cars dataset consists of 16,185 images of 196 classes annotated with bounding boxes. Though also being primarily designed for fine-grained categorization, it has a much clearer foreground-background setup compared with the Dogs dataset. We employ a similar process as used for Dogs dataset to approximate the ground-truth masks, and only exclude those images where cars are not properly detected. It finally produces 12,322 images for our experiments.

A.4 CLEVR6
CLEVR6 dataset is a subset of the original CLEVR dataset [79] with masks, generated by Greff et al. [37]. We follow the evaluation protocol adopted by IODINE [37] and Slot-attention [38], which takes the first 70K samples from CLEVR. These samples are then filtered to only include scenes with at most 6 objects. Additionally, we perform a center square crop of 192ˆ192 from the original 240ˆ320 image, and scale it to 128ˆ128 pixels. The resulting CLEVR6 dataset contains 3-6 foreground objects that could be with partial occlusion and truncation in each visual scene.

A.5 Textured Multi-dSprites
TM-dSprites dataset, which is based on the dSprites dataset [80] and Textured MNIST [32], consists of 20,000 images with a resolution of 128ˆ128. Each image contains 2-3 random sprites, which vary in terms of shape (square, circle, or triangle), color (uniform saturated colors), and position (continuous). The background regions are borrowed from Textured MNIST dataset [32]. The textures for the background are randomly shifted samples from a bank of 20 sinusoidal textures with different frequencies and orientations. We adopt a simpler foreground setting compared with the vanilla Multi-dSprites dataset used in [37], i.e., the foreground objects are not occluded as the dataset is designed to emphasize the background part. Some samples are presented in     Table S1. †K represents the pre-specified category number for latent variables. We use 200 for both the foreground and background LEBMs on real-world datasets, and 30 and 10 in the foreground and background LEBMs on multiobject datasets respectively.

C.1 Learning Objective
Derivation of Surrogate Learning Objective J pθq " E w"p β pw|x,zq rLpθqs is the conditional expec- where E is the conditional expectation of w. Recall that w ik P t0, 1u. The expectation becomes E rw ik s " 0ˆppw ik " 0|xi, zq`1ˆppw ik " 1|xi, zq S3 which is the posterior responsibility of w ik . We can further decompose J pθq into J pθq " log pαpzq loooomoooon as mentioned in the paper.
Understanding the Optimization Process Note that the surrogate learning objective is an expectation which is generally intractable to calculate. We therefore need to approximate the expectation by sampling from the distributions, and calculating the Monte Carlo average. In practice, this can be done by gradient-based MCMC sampling method, such as Langevin Dynamics [64].
Given x, we have p θ pz|xq9p β px|zqpαpzq. Note that Therefore, the log-likelihood of surrogate target distribution for the Langevin dynamics at the t-th step is which has the same form as J pθq. However, instead of updating parameters θ, Langevin dynamics updates the latent variables z with the calculated gradients.
The two-step learning process of the DRC models can be understood as follows: (1) in the first step, the algorithm optimizes J by updating latent variables z, where the posterior responsibility γ ik inferred at each step serves to gradually disentangle the foreground and background components, and (2) in the second step, the updated z is fed again into the models to generate the observation x, where the algorithm optimizes J by updating the model parameters θ.
It is worth mentioning that learning LEBMs requires an extra sampling step [47], as the gradients are given by the following δαpxq " E p θ pz|xq r∇αfαpzqs´E pαpzq r∇αfαpzqs , (20) where the second terms should be computed by sampling with pαpzq. We term this as prior sampling in the main paper.
Further Details on the Loss Functions For the generative models p β k px|z k q, k " 1, 2, we assume that x " g β k pz k q` , where g β k pz k q, k " 1, 2 are the generator networks for foreground and background regions, and is random noise sampled from a zero-mean Gaussian or Laplace distribution. Assuming a global fixed variance σ 2 for Gaussian, we have log p β k px|z k q "´1 2σ 2 }g β k pz k q´x} 2`C , k " 1, 2, where C is a constant unrelated to β k and z k . Similarly for Laplace distribution, we have log p β k px|z k q "´1 λ |g β k pz k qx |`C, k " 1, 2. These two log-likelihoods correspond to the MSE loss and L1 loss commonly used for image reconstruction, respectively.

C.2 Regularization
Pseudo Label Learning As mentioned in the paper, we exploit the symbolic vector y emitted by the LEBM for additional regularization. Let the target distribution of y k be P k given by pα k py|z k q, k " 1, 2, which represents the distribution of symbolic vector for foreground and background regions respectively. We can optimize the following objective as a regularization to our original learning objective: HpP k , Q k q "´xpα k py|z k q, log qτ k py|g β k pz k qqy, k " 1, 2, where qτ k , k " 1, 2 represents the jointly trained auxiliary classifier network (see Appendix B for architecture details) for foreground and background. g β k pz k q, k " 1, 2 represents the output of generator network. We set the weight of this regularization term to 0.1 for all the models.
Total Variation norm (TV-norm) Total Variation norm [75] is commonly used for image denoising, and has been extended as an effective technique for in-painting. We use TV-norm as a regularization for learning the background generator.
where Bxg β 2 pz2qph, wq and Byg β 2 pz2qph, wq represent the horizontal and vertical image gradients at the pixel coordinate ph, wq respectively. We set the weight of this regularization term to 0.01 for all the models.
Orthogonal Regularization We use orthogonal regularization [74] for the convolutional layers only. Let W be the flattened kernel weights of the convolutional layers, i.e., the size of W is CˆK where C is the output channel number. The orthogonal regularization is calculated according to where d is the Hadamard product. I denotes the identity matrix, and 1 denotes the matrix filled with ones. We set the weight of this regularization term to 0.1 for Birds models, and 1.0 for the rest of the models.

D Pytorch-style Code
We provide pytorch-style code to illustrate how the learning and inference in our model work.

E Evaluation Protocols
Intersecion of Union (IoU) The IoU score measures the overlap of two regions A and B by calculating the ratio of intersection over union, according to where we use the inferred mask and ground-truth mask as A and B respectively for evaluation.
Dice (F1) score Similarly, the Dice (F1) score is Higher is better for both scores.
Evalution As mentioned in the paper, IODINE [37] and Slot-attention [38] are designed for segmenting complex multi-object scenes using slot-based object representations. Ideally, the output of these models consists of masks for each individual object, while the background is viewed as a virtual "object" as well. In practice, however, it is possible that the model distributes the background over all the slots as mentioned in Locatello et al. [38]. Taking both cases into consideration (see Fig. S3 and Fig. S4), we propose two approaches to convert the multiple output masks into a foreground-background partition, and report the best results of these two options: (1) we compute the scores by making each mask as the background mask at a time, and then choose the best one; this works better when the background is treated as a virtual "object"; (2) we threshold and combine all the masks into a foreground mask; this is for when background is distributed to all slots. Figure S3: An example situation when using each individual mask as the background mask gives higher scores. Note that if we threshold the output of each individual slot and compose them, the result would be the mask shown in the last column. Figure S4: An example situation when thresholding&combining the output of each individual slot gives higher scores. We can see from the last column that the combined mask fits the foreground objects well.

F.1 More Examples
We provide more foreground extraction results of our model for each dataset; see Fig. S5, Fig. S6, Fig. S7 and Fig. S8. From top to bottom, we display: (i) observed images, (ii) generated images, (iii) masked generated foregrounds, (iv) generated backgrounds, (v) ground-truth foreground masks, and (vi) inferred foreground masks in each figure.

F.2 Failure Modes
We provide examples for illustrating typical failure modes of the proposed model; see Fig. S9. On Birds dataset, we observe that the method can perform worse on samples where the foreground object has colors and textures quite similar to the background regions. Although the method can still capture the rough shape of the foreground object, some details can be missing. On TM-dSprites dataset, we observe that the method may occasionally miss one of the foreground objects. We conjecture that the problem can be mitigated with more powerful generator and further fine-tuning on this dataset.

F.3 Baseline Results
GrabCut We provide results of GrabCut [5] on Birds dataset and TM-dSprites dataset, shown in Fig. S10.
We can see that GrabCut algorithm may fail when the foreground object and background region have moderately similar colors and textures. On TM-dSprites dataset, GrabCut algorithm outperforms other baselines, but is still inferior to the proposed method and exhibits a similar failure pattern.
ReDO We provide results of ReDO [28] on Birds dataset and TM-dSprites dataset, shown in Fig. S11. ReDO overall performs better than GrabCut on Birds dataset, while it may fail when the background regions become more complex. We can also observe that ReDO relies heavily on the pixel intensities for foreground-background grouping on TM-dSprites dataset.
IODINE On Birds dataset, we observe that IODINE [37] tends to use color as a strong cue for segmentation, see Fig. S12. On TM-dSprites dataset, IODINE is distracted by the background; see Fig. S13. These two findings are consistent with those reported in [37]; Slot-Attention On Birds dataset, Slot-attention learns to roughly locate the position of foreground objects, but mostly fails to provide foreground masks when the background region becomes complex; see Fig. S14. Similarly, we can observe that Slot-Attention tends to use color as a strong cue for segmentation. On TM-dSprites dataset, Slot-attention is distracted by the background; see Fig. S15. S11 Figure S5: Additional foreground extraction results on Birds dataset. S12 Figure S6: Additional foreground extraction results on Dogs dataset. S13 Figure S7: Additional foreground extraction results on Cars dataset. S14 Figure S8: Additional foreground extraction results on CLEVR6 and TM-dSprites datasets. S15 Figure S9: Typical failure modes on Birds and TM-dSprites. Figure S10: Results of GrabCut on Birds and TM-dSprites datasets. The first three columns are results from Birds dataset, and the last three are from TM-dSprites. From left to right, we display the observed image, ground-truth mask, and the foreground extraction results respectively. S16 Figure S11: Results of ReDO on Birds and TM-dSprites datasets. The first four columns are results from Birds dataset, and the last four are from TM-dSprites. From left to right, we display the observed image, ground-truth mask, mask from the first output channel and from the second channel respectively. Figure S12: Results of IODINE on Birds datasets. We provide the observed image, mask from the first slot and from the second slot respectively. Figure S13: Results of IODINE on TM-dSprites datasets. We provide the observed image and masks from four object slots respectively. Figure S14: Results of Slot-Attention on Birds datasets. We provide the observed image, mask from the first slot and from the second slot respectively. Figure S15: Results of Slot-Attention on TM-dSprites datasets. We provide the observed image and masks from four object slots respectively. S19 G Further Discussion

G.1 Preliminary Analysis of Real-world Datasets
We provide preliminary analysis of the statistics of the three real-world datasets. To measure the similarity of colors and textures for these datasets, we calculate the image histogram for the foreground objects and background regions of each dataset; see Fig. S16. To probe the similarity of shape distributions, we also provide the heatmap of foreground masks, as shown in Fig. S17. The heatmaps are calculated by overlapping the ground-truth masks and normalizing the summarized intensities with the maximum values. Despite the apparent difference in Birds vs Dogs and Cars, we can see that the data distribution of Birds dataset is more similar to that of Dogs dataset than to that of Cars dataset. We can also observe the similarity between the distributions of Dogs and Cars datasets. This could partly explain why the proposed method shows relatively strong performance on objects from unseen categories, i.e., it effectively combines the colors, textures and shapes for foreground extraction.   We explore the possibility of using our model for segmenting and disentangling multiple objects. As shown in Fig. S18, the proposed method can disentangle the foreground objects, while providing explicit identification of the background region. However, we find that the model occasionally distributes a single object into several slots based on the difference in texture and shading; see Fig. S19. We conjecture that this is due to the lack of objectness modeling. We would like to investigate more on this direction in future works. S20 Figure S19: Failure modes of energy-based slot representation model. Figure S20: Prior sampling results on Birds dataset.

G.3 Prior Sampling Results on Birds Dataset
We provide preliminary results of sampling from the learned energy-based priors, as shown in Fig. S20. Of note, the generated prior samples are generally less realistic compared with the posterior samples, as prior sampling does not involve the region competition between foreground and background components, which may lead to worse separation and the generation of foreground and background regions. We would further explore generating foreground and background in future work.