{'fraction_non_alphanumeric': 0.029839633447880872, 'fraction_numerical': 0.008619702176403208, 'mean_word_length': 4.990907531308973, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Perceived Overlap: A Prerequisite for VAE Disentanglement

Learning disentangled representations with variational autoencoders (VAEs) is often attributed to the regularisation component of the loss. In this work, we highlight the interaction between data and the reconstruction term of the loss as the main contributor to disentanglement in VAEs. We note that standardised benchmark datasets are constructed in ways that are conducive to learning what appear to be disentangled representations. We design an intuitive adversarial dataset that exploits this mechanism to break existing state-of-the-art disentanglement frameworks. Finally, we supply a solution that enables disentanglement by modifying the reconstruction loss, affecting how VAEs perceive distances between data points.


Introduction
A fundamental challenge in machine learning is discovering useful representations from highdimensional data that can then be used to solve subsequent tasks effectively. Recently, deep learning approaches have showcased the ability of neural networks to extract meaningful features from highdimensional inputs for tasks ranging from classification [13] to reinforcement learning [21]. While these learned representations have obviated the need for manual feature selection or preprocessing pipelines, they are often not semantically meaningful, which can negatively impact interpretability, fairness [16] and downstream task performance [17].
Prior work has therefore argued that it is desirable to learn a representation that is disentangled [1]. While there is no consensus on what constitutes a disentangled representation, it is generally agreed that such a representation should be factorised so that each latent variable corresponds to a single explanatory variable responsible for generating the data [3]. For example, a single image from a video game may be represented by continuous latent variables governing the x and y positions of the player or enemies, as well as categorical variables governing their clothing or appearance.
A common approach to discovering these representations are variational autoencoders (VAEs) [12], which are trained on unlabelled data to learn a lower-dimensional representation capable of reconstructing the given input. However, it has been proven that unsupervised methods cannot reliably learn representations without the introduction of supervision or inductive biases [17]. The recently introduced Ada-GVAE framework has partially overcome this problem by using a weakly supervised signal to discover these underlying factors [18], but there still remains room for improvement.
Interestingly, VAEs do not have an explicit mechanism that encourages the learning of disentangled representations, but it is theorised that this behaviour is related to the regularisation term and the information bottleneck principle [3,19,24]. However, despite this hypothesis, there is still no explicit reason for why the representations learnt by these frameworks should align with generative factors in the data. Nonetheless, these frameworks have been shown to produce disentangled representations (as measured by appropriate metrics [6,4,25]) when trained on synthetically generated data.
In this paper, we seek to understand why VAEs implicitly learn these disentangled representations by investigating the interaction between the reconstruction loss of the VAE and the input data. We find compelling evidence that disentanglement occurs not because of special algorithmic choices or the regularisation term, but because of how VAEs perceive distances between observations in the datasets themselves. In particular, we find that these standardised benchmarks are constructed in such a way that they accidentally encourage the models to learn what appear to be disentangled representations.
The main contributions of this paper are summarised as follows: (i) We introduce the idea of perceived distance, in terms of the VAE reconstruction loss, to measure overlap or similarity between dataset pairs. We show that perceived distances in existing datasets unintentionally correspond to the distances between ground-truth factors, and that VAEs learn these distances. This explains why learnt representations may appear disentangled. (ii) We reveal the ineffectiveness of state-of-the-art models by designing a simple adversarial dataset with constant perceived distance between elements, over which VAE-based frameworks fail to learn disentangled representations. (iii) We propose a solution to the adversarial dataset that simply modifies the reconstruction loss, and thus perceived distances across the dataset, so that the framework again captures the ground-truth factors. (iv) We contribute Disent, a PyTorch [22] disentanglement framework, with common models, metrics and datasets. 1

Background
Assume a dataset X = x (0) , ..., x (n) is a set of independent and identically distributed (i.i.d) observations x ∈ R N , generated by some random process involving an unobserved random variable z ∈ R D of lower dimensionality D N. Additionally, the true prior distribution z ∼ p * (z) and true conditional distribution x ∼ p * (x|z) are unknown. Variational autoencoders (VAEs) aim to learn this generative process. Unlike autoencoders (AEs), which consist of an encoder f φ (x) = z and decoder g θ (z) =x with weights φ and θ, VAEs instead construct a probabilistic encoder by using the output from the encoder or inference model to parameterise approximate posterior distributions z ∼ q φ (z|x). The approximate posterior is then sampled during training to obtain representations z, which are then decoded using the generative model to obtain reconstructionsx ∼ p θ (x|z).
A factorised Gaussian encoder [12] is commonly used. The posterior is modelled using a multivariate Gaussian distribution with diagonal covariance z ∼ N (µ φ (x), σ φ (x)), and the prior is given by the multivariate normal distribution p θ (z) = N (0, I), with a mean of 0 and diagonal covariance I. To enable backpropagation, the reparameterisation trick in Equation (1) is used to sample from the posterior distribution by offsetting the distribution means by scaled noise values.
VAEs maximise the evidence lower bound (ELBO) by minimising the loss given by Equation (4). VAE-based approaches often make slight modifications to this loss [8,26,9,14,4,10,18], but the terms of these modified loss functions can usually still be grouped into reconstruction and regularisation components, given by Equations 2 and 3 respectively. The regularisation term affects the representations learnt by the encoder, while the reconstruction term improves the outputs from the decoder. These terms usually contradict in practice, with strong regularisation leading to worse reconstructions but often better disentanglement [8,3].

Random sampling reorganises embeddings
Disentanglement in VAEs is generally attributed to the regularisation term in Equation (3); however, we highlight that regularisation only enables the underlying disentanglement mechanism. Disentanglement arises rather as a result of VAEs reorganising the latent space to minimise reconstruction mistakes as a result of random sampling from the probabilistic encoder during training. Through this mechanism, a VAE will place similar observations according to the reconstruction loss in Equation (2) close together in the latent space [3,19,27], as this action minimises sampling errors.
The regularisation term enables this interaction by controlling the overlap between latent distributions corresponding to different inputs. If these distributions overlap sufficiently, the decoder will often attribute a random sample to an incorrect input, see Figure 1. Thus, a mistake will be made during the decoding process, which encourages reorganisation to minimise the reconstruction error. Figure 1: Nearby distributions in the latent space that correspond to different inputs. The VAE reconstructs a sample from the middle distributions. Left: weaker regularisation leads to few sampling mistakes, resembling a lookup table [19]. Right: strong regularisation leads to more reconstruction mistakes, where samples are attributed to nearby distributions, encouraging reorganisation.

Related work
The following works are the most applicable to our research, falling under three general categories: (i) explanations for disentanglement, (ii) the role of the reconstruction loss in disentanglement, and (iii) problems with disentanglement. Firstly, Burgess et al. [3] relate VAEs to the information bottleneck principle. Which explains that random sampling leads to a local minimisation of the reconstruction loss which reorganises the latent space so that points close in pixel space are close in the latent space. Mathieu et al. [19] argue that VAEs do not explicitly encourage disentanglement through their design. Rather, they provide the explanation that the diagonal prior typically used in VAEs when combined with random sampling produces a similar effect to PCA. Secondly, Hou et al. [9] swap out the reconstruction loss of VAEs for a perceptual loss function, which can improve the representations learnt by the model. Zietlow et al. [27] extend the analysis of Mathieu et al. [19]. However, emphasis is placed on constructing adversarial datasets that hinder disentanglement performance using a mild transformation, obtained from trained models which achieve poor disentanglement scores. Finally, Locatello et al. [17] show that useful representations cannot be reliably learnt with unsupervised methods, unless inductive biases are introduced, and Gondal et al. [7] show that representations learnt on synthetic data often do not transfer well to real-world data.
Consider the 3D Shapes dataset [2] in Figure 2a, which contains observations of shapes fixed in the centre of the image with progressively changing attributes or factors such as size and colour. If, as humans, we are given unordered observations from a traversal along the size factor of 3D Shapes, it would be easy to order these observations using a perceived increase or decrease in the size of the shape. We might even say that the shapes in the images overlap by different amounts, considering shapes that are closer in size to possess more overlap, and thus also considering them as closer together in terms of distance. This idea naturally extends to VAEs ordering pairs of observations, and so we seek to investigate the correspondence between how these frameworks perceive distances over data points to reorganise the latent space and the ground-truth factors themselves.

Ground-truth distance
Synthetic datasets [2,15,20,23,7] used for benchmarking disentanglement frameworks are all generated from F ∈ N + ground-truth factors of variation. Each factor i ∈ [F] 2 represents some property about the data that can be varied, and has a dimensionality or size of f i > 0 where f i ∈ N + . The set of all factors used for generating the dataset is written as The full dataset is generated from this set of factors using some ground-truth generative process X = {g * (y) | y ∈ Y}. Examples of this generative process are given in Figure 2.
With this construction of synthetic datasets, it is fitting to describe the ground-truth distances between observations x (a) , x (b) ∈ X using the Manhattan or 1 distance between their corresponding groundtruth factors y (a) , y (b) ∈ Y, 3 as in Equation (5). It is important to note that this choice may not be optimal for single factors; rather, 1 distance naturally aligns with how the datasets are constructed.

Perceived distance
With the idea of ground-truth distances between observations, we need a distance measure between observations as perceived by VAE frameworks. We derive the perceived distance between dataset elements from the noisy sampling procedure and the chosen reconstruction loss in a VAE framework.
) be a (possibly incorrect) sample from the posterior distribution corresponding to some input element x (a) ∈ X . Since the regularisation term encourages latent distributions to overlap, this sample z (b) may be incorrectly attributed by the decoder to a distribution corresponding to some other element from the dataset . As the VAE objective consisting of the regularisation and reconstruction losses is jointly optimised, the decoder becomes better at reconstructing the inputs. In an ideal scenario, the inputs map to outputs (x → x), and reconstructions are samples from our dataset:x ∈ X . While this is not the case in practice due to the regularisation term, we derive the perceived distance in Equation (6) from this assumption that x → x. This allows us to directly compare the elements x (a) , x (b) ∈ X within a dataset using the reconstruction loss as a distance function: The perceived distance above depends on the choice of reconstruction loss. The usual choice in literature is the pixel-wise Mean Squared Error (MSE) for data that is assumed to be normally distributed. We assume that MSE loss is used throughout the rest of the paper, unless specified. However, analyses are similar for other pixel-wise losses, such as Binary Cross-Entropy (BCE).

Perceived distances correspond to ground-truth
In Section 4.1, all the ground-truth factors of a dataset are defined as the set In Equation (7), we now define a factor traversal Y (a,i) ⊂ Y as the ordered set of all the coordinates along a factor i ∈ [F] such that the set passes through a point y (a) ∈ Y. The number of elements in the traversal is equal to the size of the chosen factor |Y (a,i) | = f i , and each element in the traversal generates the same traversal Figure 2 gives examples of traversals.
We compute the distance matrixD (a,i) ∈ R fi×fi , for some distance function d, between pairwise elements along a factor traversal Y (a,i) , as written in Equation (8) using matrix notation.
To examine the ground-truth factors within our datasets, we compute the average distance matrix We plot these results in Figure 3 for both the groundtruth distance d gt and perceived distance d pcv . It is immediately obvious from these plots that the ground-truth distances and the distances perceived by a VAE may accidentally correspond.
(a) 3D Shapes [2] (b) Cars3D [23] (c) dSprites [20] (d) NORB [15] Figure 3: Distances in the ground-truth factor space naturally correspond to distances in the data space for current synthetic datasets. Top Row: Average ground-truth distance ( 1 ) matrices over factor traversals. Bottom Row: Average pixel-wise perceived distance (MSE) matrices over observations from the same factor traversals. Columns: Different ground-truth factors within each dataset.
Finally, we relate our work to Burgess et al. [3] by outlining a direct approach in the supplementary material for computing relative factor important using perceived overlap. A factor is considered more important if a VAE prefers to learn it before another factor.

VAEs learn perceived distances
We compute distance matrices over a trained β-VAE at various levels of the network, including the representation layer and reconstructions. At each level of the VAE, the learnt distances all correspond to the original perceived distances already present within the dataset, see Figure 4. Since VAEs reorganise the embedding space according to the perceived distances, combined with the results from Section 4.3, VAEs may discover structures that are similar to the underlying ground-truth factors. Our results in Figures 3 and 4 provide empirical evidence that VAEs mimic the distances already present in the dataset according to the reconstruction loss. To appear disentangled if the goal is factored representations, individual latent units will need to encode portions of the distances that correspond to factors within the data. However, it is known that VAEs with diagonal priors are rotationally invariant [19], thus the same distances between the means µ of latent distributions can be learnt for any arbitrary rotation of the latent space. This suggests that VAEs disentangle by accident, since ground-truth factors naturally correspond with distances in the dataset. If these perceived distances were to change such that they do not correspond to the ground-truth distances, VAEs might not be able to learn meaningful representations. This is highlighted by the fact that VAEs are already known to perform poorly on real-world data [7].

Adversarial dataset
In the previous section, we highlighted the striking similarity between the ground-truth distances and the perceived distances between observations in synthetic ground-truth datasets. This suggests that disentanglement occurs because latent distances accidentally correspond to ground-truth distances, when the latent space is reorganised to minimise reconstruction errors and perceived distances from the data space are captured.
Consider the example of a single chess piece moving across a chess board; there are no smooth transitions between grid points, since the piece is only valid when placed in the middle of squares.
We describe such a dataset as having constant perceived distance. This property is adversarial in nature as it is impossible for a VAE to order these observations using pixel-wise perceived distance. It is tempting to think that a harder case is if the perceived distances do not correspond to ground-truth distances; however, an (incorrect) ordering can still be found. Existing datasets such as Cars3D already satisfy this incorrect ordering, which may explain the generally worse disentanglement performance compared to other datasets, see Figure 3.
Formally, we say that a dataset has constant overlap when the pairwise distances over factor traversals are all equal. Let i ∈ [F] be a factor and y (a) ∈ Y be ground-truth coordinate vector. Then, for all elements over the factor traversal ∀y (b) ∈ Y (a,i) / y (a) , the corresponding perceived distance is constant such that d pcv (x (a) , x (b) ) = C f with C f ∈ R and C f > 0. Along factor traversals in such a dataset, no distinct ordering of elements can be found when a VAE tries to minimise the sampling error over the reconstruction loss. Going forward, we only consider the case where ∀f ∈ [F], C f = C for some C > 0.

XYSquares dataset
Taking inspiration from the chess piece example, we design a synthetic adversarial dataset called XYSquares, illustrated by Figure 5, to highlight VAEs' weakness to constant perceived distance. The dataset consists of three 8 × 8 pixel squares in a world of size of 64 × 64. This leaves 8 grid positions along each axis without any pixel-wise overlap. The three squares are each assigned a colour according to R (1, 0, 0), G (0, 1, 0) and B (0, 0, 1) to avoid any channel-wise overlap. With 6 ground-truth factors (three squares moving along two axes), each with 8 possible values, this gives a total dataset size of 8 6 = 262144 observations. The rightmost column of Figure 7 validates that this leads to a constant perceived distance between observation pairs in a factor traversal. Figure 5: Columns represent ground-truth factor traversals over our adversarial XYSquares dataset.

Experimental setup
We now investigate the performance of VAEs on our new dataset. In particular, we use the unsupervised β-VAE [8] and the state-of-the-art weakly supervised Ada-GVAE [18]. The β-VAE scales the VAE regularisation term with a coefficient β > 0, while the Ada-GVAE breaks symmetry and encourages shared latent variables between pairs of observations. This is achieved by averaging together latent distributions between observation pairs that are estimated to remain unchanged when the KL divergence is below some threshold. We note that if the weakly supervised Ada-GVAE performs poorly, then it is highly likely that another unsupervised method will also perform poorly.
We use the same Adam [11] optimiser and convolutional neural architecture as Burgess et al. [3].
To evaluate disentangled representations, we use the MIG [4] (Mutual Information Gap) and DCI Disentanglement [6] scores. MIG measures the mutual information between the highest and second highest latent units for each factor, and DCI Disentanglement measures how much each latent unit captures a ground-truth factor using a predictive model.
Finally, we perform an extensive hyper-parameter grid search for existing frameworks and datasets before running our own experiments. Hyperparameters include the learning rate, size of the latent dimension, training steps, batch size and β values. See the supplementary material for further details on all experiments conducted throughout the remainder of the paper.  Figure 6 shows that the disentanglement performance over XYSquares is extremely poor compared to existing datasets, even with the state-of-the-art Ada-GVAE. We are concerned only with the maximum score obtained for each model and dataset, as the graph is plotted over the hyper-parameter sweeps. This validates our adversarial dataset hypothesis in Section 5. Not only is the disentanglement performance poor, but much smaller values for β are needed when tuning the regularisation loss.

Varying levels of overlap
We have examined the effect of training on existing datasets with significant amounts of overlap, as well as our own adversarial dataset with constant overlap. However, we have not investigated increasing levels of overlap in datasets. To do so, we modify XYSquares by decreasing the spacing between grid points while keeping the number of grid points constant along each factor, ensuring the dataset size remains fixed at 8 6 = 262144 observations.
The original adversarial dataset with a spacing of 8 has a constant distance value of d pcv (x (a) , x (b) ) = C 8 . As the spacing s decreases from 8 → 1 over the datasets, the probability increases that any two observations re-sampled along a single factor traversal overlap p(d pcv (x (a) , x (b) ) < C 8 ) and should thus be placed closer together in the latent space. More overlap leads to more unique distance values which in turn allows for easier ordering of data points. We visualise this concept using ground-truth and perceived distance matrices in Figure 7. : Ground-truth distance matrices (far left) and pixel-wise perceived distance matrices (left to right) over factor traversals. The spacing between grid-points of XYSquares decreases from 8px to 1px, which improves the correlation between perceived distances and ground-truth distances.
We verify our statements through the experimental results in Figure 8, where the β-VAE and Ada-GVAE are trained on these datasets. As the spacing decreases and overlap is introduced, the disentanglement performance improves, since it is easier for a VAE to introduce an ordering over representations. Even for the XYSquares dataset with 1 pixel of overlap between grid points, an ordering of elements along factor traversals can be induced. However, the probability of a VAE encountering these scenarios in the latent space due to random sampling is low, and thus it is still not always easy for the model to learn disentangled representations over such a dataset.

Introducing overlap
The previous section focused on increasing overlap by changing the underlying dataset; however, this still does not solve the case for the original XYSquares dataset with constant pixel-wise perceived distance. Throughout this paper, we have provided evidence that VAEs disentangle based on their reconstruction loss, which happens to align with ground-truth factors of variation in common benchmark datasets. This correspondence is not optimal for all tasks and we propose that this leads to the poor disentanglement performance in these settings. Our solution is to choose a loss function that modifies perceived distances such that they also correspond to ground-truth distances.
The new loss function we choose cannot be a pixel-wise approach, as this does not capture the distances due to the spatial nature of the XYSquares dataset. For the sake of simplicity in this example, we convert the existing pixel-wise loss function into a spatially aware loss function by introducing a differentiable augmentation to its inputs. An appropriate augmentation for our dataset is a channel-wise box blur. The problem, however, is that the decoder needs to be able to reconstruct the data, and so purely replacing the pixel-wise loss with the augmented loss may not succeed. Rather, in Equation (9), we append the augmented term to the existing loss and scale it by a constant α > 0.

Augmented loss experiments
We choose a channel-wise box blur for our loss function with a radius of 31, or a total kernel size of 63 × 63. We implement large filters efficiently using the Fast Fourier Transform. The size of the filter ensures that if two observations have active pixels on opposite sides of the images, then overlap will still be introduced between them. We set α = 63 2 -while this appears large, a box blur kernel is normalised so that the sum of all its values is 1. We accordingly update our perceived distance measure and evaluate the new distances over the XYSquares dataset for each factor in Figure 9 after training and tuning β-VAEs. Finally, we compare the performance of the spatially-aware loss function in Figure 10 to the original pixel-wise loss. Our new loss significantly improves the disentanglement performance over the adversarial dataset. This is because it allows our models to capture perceived distances between observations that align with the ground-truth factors. Figure 10: MIG and DCI scores for Ada-GVAE and β-VAE using the MSE loss and our modified loss function. Introducing a spatially aware loss function allows us to capture ground-truth distances between observations and allows the models to disentangle the adversarial XYSquares dataset.
While our choice of loss may not be optimal for disentanglement of these specific x and y factors from our adversarial dataset, disentanglement results are impressive. This is important because it provides the intuition that changing the loss function changes perceived distances and affects the ability of VAE frameworks to learn disentangled representations. We leave learning or identification of optimal reconstruction losses for different datasets, to improve disentanglement, as future work.

Conclusion
In this paper, we demonstrated that there are fundamental characteristics of existing datasets that encourage VAEs to learn disentangled representations. Based on the observation that the perceived distances between pairs of data points correspond to their ground-truth distances, and that VAEs inherently learn these distances, we constructed an example dataset for which state-of-the-art models were unable to disentangle. Finally, we proposed an example modification to the VAE reconstruction loss that enabled successful learning of disentangled representations over this dataset by changing perceived distances to correspond to ground-truth distances. Our results highlight issues in current representation learning approaches. Disentanglement is largely accidental and careful choice of the reconstruction loss is needed to capture different ground-truth factors. This is impractical in the real world, since perceived distances cannot be a prerequisite for disentanglement. More advanced methods are therefore required that can uncover meaning within the data.

A Identifying factor importance
A factor in a dataset is considered more important if a VAE prefers to learn it before another factor. Burgess et al. [3] identify this order of importance through a slow increase of the information capacity of VAEs during training. We note that simply by looking at the average perceived distance between observations along factor traversals, this ordering can be determined. Factors with a greater average distance will minimise the error in the reconstruction loss due to random sampling the most when learnt first. These factors (or components thereof) will thus generally be preferred.
To compute the average perceived distance along a factor f , we sample a ground-truth coordinate vector y (a) ∈ Y and then another random different coordinate vector y (b) ∈ Y (a,i) over the traversal for factor f passing through y (a) . Note that y (a) = y (b) . Then, we compute the perceived distance between the corresponding observations d pcv (x (a) , x (b) ). We repeat this process to compute the expected perceived distance along factor traversals, given by Equation (10).
We determine the factor importance for dSprites as: d x ≈ 0.058 and d y ≈ 0.057 position, then d scale ≈ 0.025, then d shape ≈ 0.022, and finally d orientation ≈ 0.017. This aligns with the order determined by Burgess et al. [3]. Computing estimates over an entire dataset can be intractable-for our estimates, we sample at least 50000 pairs per factor.
Additionally, we compute the average perceived distance between any random pairs in the datasets (see Equation (11)) and find that the average distance is higher. For dSprites specifically, we have d ran ≈ 0.075. This suggests that the ground-truth factors correspond to axes in the data that minimise the reconstruction loss and is further evidence as to why VAEs appear to learn disentangled results.

A.1 Factor importance results
In Appendix A, we relate our work to Burgess et al. [3] by estimating the importance of different factors over the dSprites [20] dataset using the reconstruction loss (MSE) as the perceived distance function between observation pairs.
We compute and list the order of importance of factors from the remaining datasets in Table 1.
These importance values are computed as the average perceived distances between 50000 randomly sampled observation pairs taken along random factor traversals. Factors with higher average perceived distances will be prioritised by the model. For comparison, the average distance between any random pair in the dataset is also given. The average distances between pairs along factor traversals are usually less than the random distance, indicating that the ground-truth factors usually correspond to axes in the data that minimise errors.
We visualise the distribution of distances along factor traversals using cumulative frequency plots as in Figure 11. It is interesting to note the distinct shift in structure for the adversarial XYSquares dataset, since distance values are constant depending on the number of differing factors.

B Implementation details
In this section, we describe our various implementation details of the β-VAE [8] and Ada-GVAE [18] frameworks, as well as the handling and standardisation of the different ground-truth datasets.

B.1 Beta normalisation
For general consistency across datasets with different numbers of channels and models with different numbers of latent units, we implement beta normalisation as described by Higgins et al. [8].
Instead of taking the sum over the KL divergence in the regularisation term and the sum over elements in the reconstruction term of the VAE loss, we instead compute the means over elements in both terms and adjust the β value accordingly.  Figure 11: Cumulative proportion of perceived distance values between pairs sampled along factor traversals, compared to perceived distances between random pairs. Factors which are more important for the VAE to learn first to minimise the reconstruction loss have higher average perceived distances (lines shifted further to the right). This corresponds to the experimental results from Burgess et al. [3] which show that as the information capacity of a VAE is increased, it learns factors in order. For dSprites, this is x and y position, followed by scale, then shape, and finally orientation.

B.2 Symmetric KL
The original Ada-GVAE implementation uses the asymmetric KL divergence D KL (p || q) as the distance function between the corresponding latent units of observation pairs. The Ada-GVAE uses this distance measure to estimate which of these latent distributions should be averaged together.
We instead follow the approach of Dittadi et al. [5] and use the symmetric KL divergence to compute these distances between latent units, improving the averaging procedure and computation of the threshold. The symmetric KL divergence is defined in Equation (12).

B.3 Sampling Ada-GVAE pairs
The Ada-GVAE [18] framework introduces weak supervision by sampling pairs of observations such that there are always k ∈ [1, F] differing factors between them, where F is the total number of factors generating the dataset. We use the weaker but more realistic case for sampling each pair, where k is sampled uniform randomly from the range [1, F] as described in the original paper.

B.4 Dataset standardisation
For improved consistency and training performance, dataset observations are standardised. We first resize the observations to a width and height of 64 × 64 pixels using bilinear filtering if needed. Then the observations are normalised such that on average each channel of the image has a mean of 0 and a standard deviation of 1. Normalisation constants for each channel are precomputed across the entire dataset and are given in Table 2.  [17].

C.1 Model architecture
We use similar convolutional encoder and decoder models as Higgins et al. [8]. A full description of the basic VAE architecture is given in Table 3. The Gaussian encoder parameterises the mean and log variance of each latent distribution. The decoder uses the Gaussian derived Mean Squared Error (MSE) as the loss function. The number of input channels the encoder receives and the number of output channels the decoder produces depends on the dataset the model is trained on, this is either 1 or 3 channels.

C.2 Optimiser and batch size
Models are trained using the Adam [11] optimiser with a learning rate of 10 −3 . A batch size of 256 is used in the case of the β-VAE [8]. Similarly, in the case of the weakly-supervised Ada-GVAE [18], 256 observation pairs are sampled per batch using the strategy from Appendix B.3.

C.3 Experiment sweeps
Experiment plots and results are all produced from models trained over grid searches of hyperparameters. Grid search values are given in Table 4. If values are not specified in the hyper-parameter sweep, then default values from the corresponding section of the experiment or supplementary material are used.

C.4 Total compute
We estimate that approximately ∼ 1040 hours of compute across a computing cluster have been used to train the models needed to generate the plots and results presented throughout this paper.
Due to the inherent high variance of unsupervised VAE results, multiple runs using the same hyper-parameters but different random seeds are needed for comparing frameworks [17]. This susceptibility of unsupervised methods to the starting random seed makes extended comparisons between frameworks prohibitive due to the computational cost.