{'fraction_non_alphanumeric': 0.03379984951091046, 'fraction_numerical': 0.00966139954853273, 'mean_word_length': 4.9985556959740025, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RENs: Relevance Encoding Networks

The manifold assumption for high-dimensional data assumes that the data is generated by varying a set of parameters obtained from a low-dimensional latent space. Deep generative models (DGMs) are widely used to learn data representations in an unsupervised way. DGMs parameterize the underlying low-dimensional manifold in the data space using bottleneck architectures such as variational autoencoders (VAEs). The bottleneck dimension for VAEs is treated as a hyperparameter that depends on the dataset and is fixed at design time after extensive tuning. As the intrinsic dimensionality of most real-world datasets is unknown, often, there is a mismatch between the intrinsic dimensionality and the latent dimensionality chosen as a hyperparameter. This mismatch can negatively contribute to the model performance for representation learning and sample generation tasks. This paper proposes relevance encoding networks (RENs): a novel probabilistic VAE-based framework that uses the automatic relevance determination (ARD) prior in the latent space to learn the data-specific bottleneck dimensionality. The relevance of each latent dimension is directly learned from the data along with the other model parameters using stochastic gradient descent and a reparameterization trick adapted to non-Gaussian priors. We leverage the concept of DeepSets to capture permutation invariant statistical properties in both data and latent spaces for relevance determination. The proposed framework is general and flexible and can be used for the state-of-the-art VAE models that leverage regularizers to impose specific characteristics in the latent space (e.g., disentanglement). With extensive experimentation on synthetic and public image datasets, we show that the proposed model learns the relevant latent bottleneck dimensionality without compromising the representation and generation quality of the samples.


Introduction
Due to the rapidly evolving computational technologies, large amounts of unlabeled data are continuously generated. Considerable time, labor, and resources are dedicated to labeling, pre-processing, and transforming the unlabelled data for real-world supervised machine learning applications. As an alternative, unsupervised representation learning algorithms extract meaningful and discriminative representations that are amenable to the downstream task in the absence of labels. Unsupervised representation learning have found applications in several domains, such as computer vision [23,46,28,20,22], medical image analysis [42,49,26], and natural language processing [15,34] Deep generative models are widely used in unsupervised representation learning to learn informative representations of data and parameterize the underlying data manifold, enabling them to generate new samples from the distribution. Widely used methods include density estimation using flow-based models [10,52,40], generative adversarial networks (GANs) [41,14], and variational autoencoders [24,35]. This paper focuses on autoencoder-type architectures like VAEs that provide representation and sample generation capabilities.
Algorithms and models for unsupervised representation are based on the manifold assumption, where the data is assumed to be generated by varying a set of parameters. Usually, the number of parameters is much smaller than the dimensionality of the data. For example, different facial images can be generated using a finite set of parameters such as lighting, skin color, expressions, facial features, hair, etc. According to the manifold assumption: a set of N −samples X = {x 1 , ..., x N } in a D−dimensional data space, where x ∈ R D , is said to lie on or near a low-dimensional manifold M ⊂ R D with intrinsic dimensionality d I D that refers to the minimum number of parameters necessary to capture the entire information content present in the representation [13] Principal component analysis (PCA) was one of the earliest methods that established a relation between the data and the low-dimensional latent space. PCA provides a closed-form solution for determining the optimum latent space dimensionality. However, PCA is limited by linearity and imposes oversimplifying assumptions of Gaussianity on the data distribution. Moreover, traditional approaches such as PCA are not scalable for large datasets and require algorithmic treatments (e.g., online PCA [6,5]) for big data. Deep generative models overcome the limitations of PCA providing non-linearity and scalability using deep networks to parameterize the mapping to the latent space. In the case of deep generative models, the dimensionality of the latent space is typically defined upfront for each dataset at the design time. The design process may be under-or over-provision the number of dimensions for the application at hand. If the dimensionality is not predefined, this parameter is usually determined using time and resource-consuming cross-validation. A mismatch between latent dimensionality L and intrinsic dimensionality d I affects the quality of data representation and sample generation [37,31,38]. Studying and understanding the effects of dimensionality mismatch in the latent space becomes imperative to use deep generative models effectively. Multiple studies have targeted improving reconstruction quality, but very few have tried to analyze the impact of latent dimensionality mismatch. Wasserstein autoencoders, highlights that a mismatch between the latent dimensionality and the true intrinsic dimensionality leads to an infeasible optimization object [31,32]. For deterministic encoders and decoders, a high capacity bottleneck can cause curling of the manifold [44]. Whereas using a smaller bottleneck can cause lossy compression of data and deteriorate representation quality [9]. For deterministic encoders, studies have concluded that larger bottleneck dimensions are not always better [32,44]. For VAE, it was theoretically proved that increasing the bottleneck capacity beyond the intrinsic dimensionality does not improve the reconstruction quality [8].
We propose a principled framework grounded in probabilistic modeling to identify the optimal data-specific latent dimensionality without adding new hyperparameters. To empirically motivate the proposed model, we performed experiments withvanilla VAE [25] and dpVAE [3] to analyze the impact of latent dimensionality on the representation learning and sample generation tasks. See Figure 1. The two models were trained on the MNIST dataset with varying latent dimensions, while other parameters were kept the same. The Frechet inception distance (FID) score was used as the generation metric, and mean squared error (MSE) as the representation metric. The FID score and MSE curve indicate inferior performance when the bottleneck size is under-provisioned. We see the MSE performance improve with the bottleneck size, but FID scores suffers at higher dimensions such as 64 and 128. The lower MSE and increase in FID score at a large bottleneck size indicates over-fitting and loss of generalization. We can conclude that larger dimensions are not always guaranteed to produce the best-performing models. Hence, we need automated ways of identifying the dimensionality mismatch and informing the model about the intrinsic dimensionality of a given dataset. Our contributions are as follows: 1. Introduce relevance encoding networks (RENs): a framework that facilities the training of VAEs using a unified formulation to parameterize the data distribution and detect latentintrinsic dimensionalities mismatch. The formulation is general and can be adapted to state-of-art VAE-based methods such as regularized VAEs. This framework also provides a PCA-like ordering for the latent dimensions that conveys the variance of each latent dimension supported by the data manifold. 2 Related Work VAE [25] is a latent variable model specified by an encoder, decoder, and prior distribution on the latent space. The encoder maps the input to the latent space (inference), while the decoder reconstructs the original input from the latent space (representation). The prior enables sample generation from a tractable probabilistic distribution [11]. Several studies have suggested that using learnable prior can improve the performance of VAEs and reduce the impact of dimensionality mismatch [48,47,45,3,47]. Dai and Wipf rigorously analyzed the VAE objective under various scenarios of dimensionality mismatch [8]. A critical conclusion (see Theorem 5 in [8]) states that optimal reconstruction can be achieved when the latent bottleneck dimensionality matches the intrinsic dimensionality, and increasing the bottleneck capacity may negatively impact the generation process.
Very few methods have demonstrated ways of identifying and handling latent-intrinsic dimensionality mismatch. De Boom et al. illustrated the use of Generalized ELBO with constrained optimization (GECO) and the L0-augment-REINFORCE-merge (L0-ARM) gradient estimator [27,50] to shrink the latent bottleneck dimensionality of VAEs automatically [9]. The L0 norm was applied to a global binary gating vector that controlled the latent dimensionality. GECO was used to define a constraint on the reconstruction error to give it more weight during optimization until the desired level of accuracy is reached. Once the threshold is reached, the narrowing of the bottleneck is given priority. Heim N et.al. proposed the relevance determination in differential equations model (Rodent) model [16] that showed the use of automatic relevance determination (ARD) [43,4] priors to minimize the state size of the ordinary differential equation (ODE) and the number of non-parameters required to solve the problem using partial observations. They used a VAE-like architecture, where the encoder was a neural network, and the decoder was an ODE solver. Isotropic Gaussian was used with an ARD prior in the latent space, and a point estimate was used for the variance of the prior distribution. The Rodent model formulation is not fully probabilistic and only focused on solving the ODEs.
For autoencoders, several studies (e.g., [37,38]) have analyzed how deterministic and random encoder-decoder pairs perform in the presence of latent-intrinsic dimensionality mismatch. Studies by Rubenstein et al. revealed that the deterministic encoders start curling the manifold in the latent space when the latent dimensionality is higher than the intrinsic dimensionality. At the same time, random encoders fill the irrelevant dimension with noise while encoding useful information in the latent space. Random encoders start behaving like deterministic encoders if the dimensionality is increased further. Deterministic and random exhibit poor sample generation performance with an increase in the volume of the holes in the latent space. Mondal et al. studied the effect of dimensionality mismatch in the case of deterministic autoencoders [32,31]. Mathematically and empirically, Mondal et al. shows that having a fixed prior distribution, oblivious to the dimensionality of the true latent space, leads to the optimization infeasibility, and proposes masked autoencoders (MAAE) [32] as a potential solution. MAAE [32] introduced modifications to the autoencoder architecture to infer a mask at the end of the encoder to suppress noisy latent dimensions.
Existing approaches that identifies the relevant dimensions, introduces more hyperparameters that have to be tuned to identify the bottleneck size. Hence, the complexity of finding the optimum latent dimension remains the same. Moreover, these methods treat relevance determination as a separate task agnostic to the probabilistic formulation of deep generative models, making the solution less interpretable.The RENs framework facilities training of VAEs using a unified probabilistic formulation to parameterize the data distribution and detect latent-intrinsic dimensionality mismatch without adding new hyperparameters.

Background
Notation: We denote a set of N −observations in a D−dimensional data space as X = {x 1 , ..., x N } and their corresponding latent representations Z = {z 1 , ..., z N }. A representation learning model maps an observation x n ∈ R D to an unobserved latent representation z n ∈ R L in an L−dimensional latent space, where L D. Hereafter, we use boldface lowercase letters to denote vectors, boldface uppercase letters to denote matrices, and non-bold lowercase to denote scalars.
Variational Autoencoders (VAEs): VAEs are latent variable models that learn data representations in an unsupervised way by matching the learned model distribution p θ (x) to the true data distribution p(x). The generative (i.e., decoder) and inference (i.e., encoder) models in VAEs are jointly trained to maximize a tractable lower bound L(θ, φ) on the marginal log-likelihood E p(x) [log p θ (x)] of the training data. The structure of the learned latent representation is controlled via imposing a prior distribution on the latent space such as p(z) ∼ N (z; 0, I).
(1) where θ denotes the generative model parameters, φ denotes the inference model parameters, and q φ (z|(x) ∼ N (z; µ z (x), Σ z (x)) is the variational posterior distribution that approximates the true posterior p(z|x), with µ z (x) ∈ R L , Σ z (x) = diag(σ z (x)), and σ z (x) ∈ R L + dpVAE: Decoupled Prior for VAE: Maximizing the ELBO by optimizing the marginal loglikelihood does not guarantee good representation. With expressive generative models p θ (x|z), VAE can ignore the latent representation and not encode any information about the data in them, but still maximize the ELBO [18,1,7], this is the information preference phenomena of VAEs. It has been shown that data-driven (i.e., learned during training) priors help mitigate the information preference of VAEs [18,36,47]. Specifically, dpVAE [3] decouples the latent space that performs the representation z and the space that drives sample generation z 0 using a functional bijective mapping g η (z) = z 0 parameterized by the network parameters η, where p(z 0 ) ∼ N (z 0 ; 0, I). Affine coupling layers [10] are used to build a flexible bijection function g by stacking a sequence of K simple bijection blocks.
where η = {s 1 , ..., s k , t 1 , ...t k }, s k is the scaling network of the k−th block, b l k is the l−th element in b k vector; the binary mask used for partition the k−th block of the scaling s k and translation function t k of the k−th block and z l k is the k−th element of z k vectors Sigma Variational Autoencoders: It is common practice to consider the decoding distribution p θ (x|z) as a Gaussian with constant variance representing the data noise (tuned as a hyperparameter).
When using a fixed variance, a model with high variance will not retain enough information in the latent space to faithfully reconstruct samples and a model with low variance will generate poor samples as the KL divergence term becomes weaker [1,29]. The σ-VAE [39] model is a simple, yet effective solution for calibrating the decode variance by using a single learnable parameter σ in p θ,σ (x|z) = N (x; µ z (x), σ 2 I) The variance of the decoder is trainable and is learned along with the rest of the model parameters [39]. The formulation reduces the time and resources required to tune the variance for each model.
Considering the performance merits of dpVAE and σ-VAE, all our experiments will implement the VAE model with the decoupling architecture of dpVAE and σ-VAE formulation. This dovetails with RENs objective to reduce VAE hyperparamters that require extensive tuning for each datasets while improving sample generation and representation.

Relevance Encoding Networks
Larger latent bottleneck sizes does not guarantee a better VAE performance ( Figure 1). Therefore, to inform the model about intrinsic dimensionality, we introduce an automatic relevance determination (ARD) [4,43] hyperprior over the latent space prior p(z n ). The ARD hyperprior regularizes the latent space to discover relevant latent dimensions that are supported by the data, hence reducing the contribution of redundant dimensions. The ARD hyperprior provides relevance of each dimension in the latent representation and this relevance is defined via precision (i.e. inverse of variance). The ARD hyperprior pushes the precision of the spurious dimensions to infinity; thus, the variance of these dimensions is pushed to zero in the latent space. The latent prior is given by:

RENs Formulation
with ARD hyperprior given as: Here, α ∈ R L + is the relevance of the latent dimensions, and 1 L is an L−dimensional vector of ones. The concentration parameter a ∈ R + and the rate parameter b ∈ R + of the Gamma distribution are shared across all latent dimensions. The VAE prior now becomes: We introduce a relevance encoder to the VAE architecture that learns the variational posterior Z)) which approximates the true posterior p(α|X , Z). Here, ψ denotes the parameters of the relevance encoder network. The relevance of a latent dimension is a statistical property of the underlying latent distribution that is induced by the data distribution. Consequently, relevance cannot be estimated from a single sample, but instead requires access to a finite set of representative samples for the data and latent distributions. Hence, we formulate a set-input problem, where a set of instances is given as an input and the relevance encoder parameterizes the relevance for the entire set with permutation invariance. Taking the complete dataset into consideration, the joint probability of the training data can be expressed as, p(X , Z, α) = p(X |Z, α)p(Z|α)p(α) and the probability of the training data is p(X ) = N n=1 p(x n ). See Figure 2 and Figure 3 for the plate notation of the graphical model and the block diagrams of their architectures, respectively.
Samples of the Gamma distribution are reparameterized to enable gradient flow and network training in the presence of probabilistic layers. The derivatives are computed using the implicit reparame- terization approach [12]. This reparameterization is implemented in the TensorFlow Probability 1 .
Derivations for the ELBO can be found in Appendix A.1. The ELBO for VAE is: The ELBO for dpVAE is:

Network and Training Strategies
The ideal scenario would be feeding the relevance encoders all the training samples at once to generate a robust estimate of the relevance. However, the use of the entire dataset to estimate such relevance negatively impact the scalability provided by stochastic gradient descent training for large datasets. We train RENs via stochastic gradient descent and alternating optimization with two batch sizes. Each batch in the training dataset is broken down into r smaller batches of size batch_size/r. The alternating optimization is given by following steps: (i) The relevance encoder is kept fixed (i.e., not-trainable) and the smaller batches are used to update the VAE encoder φ and decoder θ r-times by keeping the α obtained from the previous iteration fixed. (ii) The original large batch is used to update the entire network end-to-end (i.e., encoder φ, decoder θ, and relevance encoder ψ), and the α value is updated. As mentioned in section 4.1, we use set formulation for the relevance encoder ψ, wherein the relevance encoder is fed in a set of data samples, and their latent representations and a single α is inferred. The response of the relevance encoder should be invariant to the ordering of the samples in the given batch. Therefore, we use DeepSets [51] to make the relevance encoder permutation invariant for a given batch. (see Figure 3(c)). Using this alternating optimization and DeepSets aggregator, REN is encouraged to learn the global statistics of latent representations induced by the data distribution in the data space. See Algorithm.1 for more details.

Toy Datasets
We use the circle and one-moon datasets for proof-of-concept experiments, both of which exhibit an intrinsic dimensionality of one, parameterized by the radius. We generated three different datasets for circle and one-moon distributions by varying the standard deviation of the zero-mean additive Gaussian noise to mimic data with different noise levels. For all our experiments, we implemented the relevance encoder with dpVAE [3] and σ-VAE framework [8], hereafter referenced as R-dpVAE. We tested the ability of the models to identify the intrinsic dimensionality along with their sample reconstruction capacity and realistic sample generation capability. The R-dpVAE model was compared with relevance factor VAE (RF-VAE) [21] and masked adversarial autoencoders (MAAE) [32]. Figure 4 shows the results for one-moon and circle datasets with additive Gaussian noise of zero-mean and 10% of the radius of the data manifold as standard deviation. Compared to RF-VAE and MAAE, R-dpVAE can regularize the latent space to discover latent dimensions relevance supported by the data while achieving the lowest mean square errors on the testing samples and suppressing the spurious dimension ( Figure 4.2 and 4.6). For R-dpVAE, the variance in the latent space is indicative of the relevance shown in plots (Figure 4.1c and 4.5c). For one-moon, relevance estimated by R-dpVAE is [0.2, 0.8] for zdim1 and zdim2, which correspond to the x and y-axis in the latent space. Therefore, the x-axis with low relevance has a larger variance, and the y-axis with high relevance has a low variance, correctly capturing a low-dimensional manifold where latent dimensionality equals the intrinsic dimensionality.
Although, the performance of RF-VAE (Figure 4.3 and 4.7) comes close to R-dpVAE with the estimated relevance, it fails to generate good quality samples as the relevance is not factored in the aggregate posterior, and the latent prior is still a standard normal distribution. MAAE (Figure 4.4 and 4.8) identifies the latent manifold but has a higher reconstruction error and generates bad quality samples due to the weak nature of regularization in the latent space; holes can be seen in the latent space (Figure 4.4b and 4.8b). Across all experiments, R-dpVAE models provide a tighter distribution for the reconstruction error than other methods. R-dpVAE is the best performing model consistently, even in the presence of higher noise levels that make learning the underlying one-dimensional manifold more challenging. Results with different noise levels can be found in Appendix A.3.

Image Dataset
We experimented with three image datasets: MNIST, Fashion MNIST, and dSprites. Similar to the toy experiments, we compared the performance of R-dpVAE with relevance factor VAE (RF-VAE) [21], masked adversarial autoencoders (MAAE) [32], with the addition of VAE regularized with L0 ARM and GECO (henceforth referenced as GECO) [9]. To set baselines for comparisons, we implemented vanilla VAE with σ-VAE framework and dpVAE with σ-VAE framework sans the relevance. The supplementary material includes details on all the models' implementation, architectures, and hyperparameters. We consider the following quantitative metrics to evaluate and compare the models: (1) Frechet Inception Distance (FID): This metric calculates the distance between feature vectors of the real and generated images [17]. Lower FID scores are better. Although the ground truth intrinsic dimensionality of image datasets is not known, we use previous studies [21,32,31,9] as points of reference. The design choices of latent dimensions in the experiments are also motivated by the reported latent dimensions in the relevant literature. For MNIST, studies reported the latent dimensionality between 7 and 10. Hence, we choose 16 as the base dimensionality to provide the models with enough degrees-of-freedom to discover the relevant latent dimensions and 32 as the over-provisioned model to assess the impact of a significant mismatch with the intrinsic dimensionality. Similarly, for Fashion MNIST, the choices were 32 and 64 for base and over-provisioned dimensionalities, and for dSprites (known to have 6 factors of variation [30]), the choices were 10 and 15. The MAAE and GECO models provide the number of active dimensions, whereas the RF-VAE model provides a relevance vector with values from 0 to 1. For RF-VAE, we estimate the dimensionality by calculating the number of dimensions with relevance values higher than the average of the vector. In the case of R-dpVAE, the relevance estimated by the relevance encoding network is the inverse variance as per Eq. 3. We compute the explained variance as the ratio of variance in a single dimension to the sum of all the variances. The number of dimensions required to explain 95% of the variability is considered the detected latent bottleneck dimensionality. Table 1 summarizes the FID scores of randomly generated images and the estimated latent dimensionality by each model for all three datasets. The proposed model (R-dpVAE) achieves lower FID scores on all three datasets compared to the other models while estimating bottleneck dimensionality in the same range as the other methods. R-dpVAE also achieves better FID scores than the baseline models that do not perform relevance determination. This performance boost via relevance encoding bolsters our argument for the necessity of identifying and fixing the latent and intrinsic dimensionality mismatch. Similar to the findings from the toy experiments, the models that do not modify the posterior based on the learned relevance of the latent space exhibit inferior sample generation. This is reflected in the higher FID scores of RF-VAE, GECO, and MAAE. The MAAE and GECO models show consistency in determining the effective bottleneck size in the case of MNIST and dSprites, irrespective of the provisioned bottleneck size of the model. Although the R-dpVAE model estimates the bottleneck size in the same range as MAAE and GECO and shows better generation capabilities, R-dpVAE does not estimate the same size across L for the same dataset. This behavior could be attributed to the use of random samples in the mini-batch used for updating the relevance encoder(ψ) and may not capture the entire data variation. Using a stratified sampling approach to generate mini-batch could be a potential solution. Table 2 summarizes the MSE of the testing (i.e., held-out) images for each model for all three datasets. While R-dpVAE does not achieve low MSE as other models, the overall performance in conjunction with the FID score indicates that R-dpVAE models generalized well with the use of calibrated decoders and relevance encoders. The use of fixed weight on the reconstruction term in models such as RF-VAE, MAAE, and GECO is hypothesized to cause lower MSE and higher FID scores. Figure 5 shows the reconstructed and generated images for the best performing models cross different L values for each method.    Table 1 for each method. We show (a) original images, (b) reconstructed images, and (c) randomly generated images (no cherry-picking).

Conclusion
Latent dimensionality mismatch can have a detrimental effect on the performance of deep generative models such as VAEs. We have introduced relevance encoding networks (RENs) to identify this mismatch and inform the model about the relevant bottleneck size. The RENs framework facilities training of VAEs using a unified probabilistic formulation to parameterize the data distribution and detect latent-intrinsic dimensionality mismatch. A key feature of the RENs framework is that it does not require any extra hyperparameter tuning for relevance determination and it provides a PCA-like ranking of the latent dimensions based on the learned, data-specific relevance. The proposed model is general and flexible to be incorporated in any state-of-the-art VAE-based models, including regularized variants of VAEs. Future directions include extending the formulation of RENs toward explainable VAEs.

A Appendix
A. p(z n |α); p(z n |α) ∼ N (z n ; 0, α −1 I) Considering the graphical model in Figure.3(d), the variational posterior is specified by q(Z, α|X ) q(Z, α|X ) = q ψ (α|Z, X )q φ (Z|X ) We begin with the defining the marginal likelihood of the training data, ELBO maximizes the marginal log likelihood of the training data: log p(X ) Applying Jensens inequality The first term is the reconstruction loss: The second term can be further simplified as follows, The ELBO for REN with VAEs: For dpVAE [3], please refer to the paper for the basic formulation. For dpVAE with REN the joint likelihood changes to, The second term for dpVAE with REN formulation becomes: The ELBO for REN with dpVAE: +E q(Z,α|X ) [log p(α)] + E q(Z,α|X ) [log p(g η (Z)) + log( ∂gη(Z)

A.3 Toy Dataset Results
The results in Figure 6 and Figure 7 shows the results for higher noise levels. The observations are the same as section 5.2. R-dpVAE is able to detect while maintaining low MSE even in the presence of noisy data. Figure 6: Sample reconstruction and sample generation outputs of R-dpVAE, RF-VAE, MaskAAE for one-moon and circle at 5% and 7% noise level.

B Broader Impacts
The paper proposes relevance encoding networks (RENs): a novel probabilistic VAE-based framework that uses the automatic relevance determination (ARD) prior in the latent space to learn the dataspecific bottleneck dimensionality. The proposed method is an algorithmic improvement and does not have direct societal impacts. However, unsupervised representation learning using deep generative models needs to be evaluated critically.
Unsupervised representation learning algorithms have emerged as alternatives for extracting meaningful and discriminative representations without labels. As the unsupervised representation learning algorithms rely solely on the training data, the model produced using such algorithms can only be as good as the quality of the training data used. The biases present in the training data can be propagated to the model. Hence, it is crucial to rigorously assess the quality of the training data, including biases, before using it for unsupervised algorithms.
With the requirement of labeled data removed, the unsupervised representation learning algorithms have found applications in various domains like-computer vision, medical image analysis, natural language processing, etc. When using the algorithms for sensitive applications, it is paramount to introduce humans into the loop for decision-making to avoid automation bias. C Architecture and Implementation Details C.1 RENs: Relevance Encoding Networks Figure 8 shows the architecture of the encoders and decoders used for all toy and image experiments. For toy datasets, the decoders were mirrored versions of the encoders. All the experiments used the same coupling layers and relevance encoder architecture shown in Figure 9. The learning rate for VAEs using a smaller batch was set 1e −3 and for an end to end training with the relevance encoders and a larger batch was set to 1e −5 . The large batch size for MNIST and Fashion MNIST was 100, and for dSprites, one-moon and circles were 128. In all the experiments, the larger batch was divided into four smaller batches (See Algorithm 1 for details). The number of samples in the training dataset of MNIST and Fashion MNIST was 60000, dsprites 663552, and toy experiments 4096. All image experiments were run for 100 epochs, and toy experiments were run for 1500 epochs. Seed used -42 for all experiments. All experiments use the same gamma prior p(α) = Γ(1e −3 , 1e −4 ). The experiments were run on single GPU(NVIDIA GeForce RTX 2080 Ti).

C.4 GECO: VAE regularized with L0 ARM and GECO
We used the code shared by the authors for the implementation after obtaining the consent for the use.

D.1 Image Datasets
In this section, we present more results for the image datasets. Figure 10 compares the performance of the various models on the dSprites dataset. Figure 11 shows the results of R-dpVAE on CELEBA dataset for L = 100 and L = 128. Figure 12 and Figure 13 shows the latent traversals of the R-dpVAE models. Each row represents a traversal from the image in the first column to the image in the second column. Figure 10: Results of the best performing models across L for dSprites from Table 1 for each method. We show (a) original images, (b) reconstructed images, and (c) randomly generated images (no cherry-picking).

E Limitations
Relevance encoding networks have the same limitations as variational autoencoders (VAE). The images generated using RENs and VAE can be blurry and unrealistic compared to those generated using GANS. As RENs seek to maximize a lower bound on the log-likelihood of the generative models, RENs inherit the limitations of maximum-likelihood (ML) [2]. Models like RENs and vanilla VAE optimize the (one-way) Kullback-Leibler (KL) divergence between the underlying data distribution and the model's distribution. Hence, do not penalize a model that generates data different from the training samples. Adversarial methods have been proposed in the literature that aims to alleviate this problem in VAE [33,19]. Similar techniques could be utilized for RENs to improve the quality of generated images.