{'fraction_non_alphanumeric': 0.02517162471395881, 'fraction_numerical': 0.0041326548037842826, 'mean_word_length': 4.83150766779526, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
De-Biasing Generative Models using Counterfactual Methods

Variational autoencoders (VAEs) and other generative methods have garnered growing interest not just for their generative properties but also for the ability to dis-entangle a low-dimensional latent variable space. However, few existing generative models take causality into account. We propose a new decoder based framework named the Causal Counterfactual Generative Model (CCGM), which includes a partially trainable causal layer in which a part of a causal model can be learned without significantly impacting reconstruction fidelity. By learning the causal relationships between image semantic labels or tabular variables, we can analyze biases, intervene on the generative model, and simulate new scenarios. Furthermore, by modifying the causal structure, we can generate samples outside the domain of the original training data and use such counterfactual models to de-bias datasets. Thus, datasets with known biases can still be used to train the causal generative model and learn the causal relationships, but we can produce de-biased datasets on the generative side. Our proposed method combines a causal latent space VAE model with specific modification to emphasize causal fidelity, enabling finer control over the causal layer and the ability to learn a robust intervention framework. We explore how better disentanglement of causal learning and encoding/decoding generates higher causal intervention quality. We also compare our model against similar research to demonstrate the need for explicit generative de-biasing beyond interventions. Our initial experiments show that our model can generate images and tabular data with high fidelity to the causal framework and accommodate explicit de-biasing to ignore undesired relationships in the causal data compared to the baseline.


I. INTRODUCTION
In many fields such as medicine and economics, an explainable model, in particular a causal model, is needed to elicit the effectiveness of interventions. This process makes diligent use of prior knowledge, usually in a structural causal model (SCM) that instantiates unidirectional relationships between the variables using a Directed Acyclic Graph (DAG) [1]. The confidence needed in a causal model needs to be much higher than in a statistical model as one needs to instantiate beliefs that are invariant and exist outside the domain of the data. Traditionally, this knowledge comes from experimentally derived results, or domain experts with experimental level knowledge. As such, there is a strong interest in the deep † Equal Contribution Department of Electrical and Computer Engineering University of California, Los Angeles learning community to integrate causal methods and information more directly with traditional deep learning architectures.
Although recent results show progress in causal deep learning, most methods focus on either causal discovery or the use of prior causal information alone [2]- [4]. Generative models have been crucial to solving many problems in modern machine learning [5]. Since the VAE's inception, many have found that the disentanglement of latent spaces can lead to better performance in generalizability and fine-tuned control over disentangled features. In addition, many techniques have been proposed in recent years as to how to improve disentanglement, largely based on factorization and independence techniques [6], [7].
Recently, an effective approach that blends the space of causal models with generative neural networks was displayed with the CausalVAE, which allowed the decoder to learn a causally disentangled representation of latent space variables. One of the key contributions in that paper was the inclusion of a Causal Layer. Most impressively, the CausalVAE enforced a causal structure on generating images to noticeably disentangle intentionally dependent latent variables via the use of a causal layer. This causal layer's disentanglement allows the CausalVAE to generate causal interventions. Specifically, when intervening on endogenous variables, the CausalVAE is able to generate images that are outside the normal bounds of the training dataset, as the intervention does not affect the exogenous variables [8].
Here, we combine the ideas of counterfactual causal reasoning and generative modeling by focusing on the causal layer of the CausalVAE. We modify the objective into learning a more refined, isolated causal structure that the latent space must go through, which we call the Causal Counterfactual Generative Model. This allows us to expand the use of the causal layer to more than just single interventions, to also to hypothesize and synthesize datasets of counterfactual causal models in interesting and useful ways.

II. RELATED WORK
Causal discovery has increasingly been the focus of deep learning methods which seek to reduce the combinatorial complexity of brute force searches for causal models from observational data. Progress in DAG search using continuously differentiable loss functions and reinforcement learning for score functions has started to integrate deep learning methods with causal discovery and identification [9] [2].
Building on initial deep causal discovery, causal generative models learn or use causal information for generating data and interventions. CausalGAN is a generative model that learns a prior Structural Causal Model (SCM) for images and label spaces and demonstrates how interventions in the latent space can generate causally intervened images [3]. DAG-GNN uses graph neural networks with a VAE architecture to extend causal discovery methods to more use-cases [4]. CausalVAE uses a causal layer in the middle of a VAE architecture to learn an implicit causal model that can also generate unseen images with latent space interventions [8]. Causal discovery with generative models capitalize on recent work in disentanglement to ensure the latent space has the necessary variable structure for causal identification [6]. Finally, causal generative models have been used to address the issue of fair or "de-biased" data sets such as DECAF, a causally aware GAN architecture applied explicitly to tabular data [10].
When causal models are known or hypothesized to contain measured confounders, statistical adjustment techniques have long been used to estimate causal effects when the structure is known or identifiable. Inverse Propensity Score Weighting (IPW), or advanced methods like Augmented IPW provide robust or doubly-robust ways to adjust for confounding bias [11].

A. Counterfactuals and Interventions
The SCM literature has long explored the benefits of interventions and counterfactual modeling once a causal model is known. Pearl introduces interventions using 'do-calculus' or the explicit setting of a variable to a specific value and calculating the resulting outputs [1]. In Figure 1 below, we introduce a 4-variable DAG with two exogenous and two endogenous variables. An intervention on the right shows how this is effectively breaking the parent nodes into the variable being intervened on, and explicitly setting it to a desired value (x), written using do-calculus notation do(x). This operation allows us to directly fix the value of a latent variable and asymmetrically propagate its value to other variables. Intervened parents should have their adjusted values impact child nodes, but intervened children should not adjust parent values.
do(x) Fig. 1: Example of a DAG on the left and a mutated counterfactual model on the right with an intervention setting the target variable to an explicit value x.

B. Counterfactual Models
Extending from the idea of interventions on instances of data, we define counterfactual models as a new model formed by removing a path deemed undesirable or a source of bias as seen in Figure 2. This could be a known bias present in the data generating process, or a desire to envision a new data distribution outside the training dataset with a specific graphical modification. Notice, unlike an intervention as in Figure 1, the target variable need not be set explicitly but still is a function of the other parent variables. This allows a data distribution to be generated in which the target is still a function of the remaining parent nodes, possibly simulating a "de-biased" or counterfactually constructed dataset, as opposed to explicit instantiations of the intervened variable.

C. Constructing a Causal Generative Model
Following the classic VAE model, given inputs x, we encode into a latent space z with distribution q φ where we have priors given by p(·) [5].
(1) In [8], the causal layer is described as a noisy linear SCM: which finds some causal structure of the latent space variables z with respect to a matrix A. By itself, A functions as the closest linear approximator for the causal relationships in the latent space of z.
A non-linear mask can be applied to the causal layer so that it can more accurately estimate non-linear situations as well. Suppose A is composed of column vectors A i . For each latent space concept i, define a non-linear function g i : R n → R and modify equation (2) such that where • is the Hadamard product. In this formulation, the view of A changes from one of function estimation to one of adjacency. That is, if A is viewed as a binary adjacency matrix, the g i functions take the responsibility of reconstructing z given only the the parents, dictated by A i • z. In the simplest case, if g i (v) = j v j , the summation of all the values of v, then Equation (3) degenerates back to Equation (2) [12].
Including the causal layer introduces many auxiliary loss functions that we mostly adopt [8]. First is a label loss (4), where the adjacency matrix A should also apply to the labels u. This loss is used in pre-training in its linear form to learn a form of A prior to learning the encoder and decoders. After pre-training, we apply a nonlinear mask f i that functions similarly to g i , but operates on the label space directly, but with the same A.
The latent loss tries to enforce the SCM, described by Equation (3).
Further enforcing the label spaces, we can define a prior p(z|u). We use the same conventions as in [8] and say that where u n ∈ [−1, 1] are normalized label values. This translates to an additional KL-loss.
Finally, we apply the continuous differentiable loss function (6) and apply a scheduling technique to enforce the DAG [4], [9]. The main use is that G is a DAG if and only if The scheduling is done via the augmented Lagrangian where at the end of every epoch, the scheduling update is where we set η = 2 and γ = 0.9.

D. Causal Estimation
There are numerous ways to estimate a causal effect once the model has been identified. Perhaps the most common and simplest is the Average Treatment Effect ( AT E), which is simply the difference of means between a population (index = i) treated and untreated group, assuming a binary intervention variable (D), and an outcome variable (Y ) as in equation (9).
This naïve method does not consider any confounding variables. One common way to adjust for such confounding bias is to use propensity scores (π(X i )), which is a model for how likely a sample is to receive the treatment based on the measured covariate factors. The inverse of the propensity score can then be used to weight each sample as in equation (10) and thus adjust for the bias of any measured confounders.
Finally, more recent developments in double-robust methods specify both an outcome model and an exposure/propensity score model which can provide accurate estimation if either one of the models is misspecified. Augmented IPW (AIPW) is a specific method that extends IPW below with a set of outcome models estimating the outcome variable as a function of the intervention and all covariates as introduced in [11].

A. Sun Pendulum Image Dataset
A toy pendulum image dataset is introduced in [8]. This dataset is generated by sweeping sun positions (x sun ) and pendulum angles (θ) to produce realistic shadow width (w shadow ) and shadow locations (x shadow ) from deterministic non-linear functions. Figure 3 shows the DAG for this model and an example generated image, in which the sun and pendulum variables are exogenous, and the shadow variables are endogenous. Thus any causal model will learn to reconstruct the shadow variables from the sun and pendulum variables. Such relationships in observational studies are often invertible as correlation has no directionality. Thus, without causal disentangling, an intervention on shadow position would likely adjust the sun position to match. This dataset is used to demonstrate causal generative model quality through reconstruction fidelity as well as causal learning by intervening on parent and child nodes, showing interventions only propagate forward from parents to children and not vice-versa [8].

B. Tabular National Study of Learning Mindsets Data
To analyze our methods in a tabular setting, we use a simulated dataset based on The National Study of Learning Mindsets [13]. This was a randomized study conducted in U.S. public high schools, the purpose of which was to evaluate the impact of a nudge-like intervention designed to instill students with a growth mindset on student achievement. We use a simulated subset of the data based on a model fit to the statistics of the original dataset (the actual dataset was not publicly released). The study includes measured outcomes via an achievement score, a binary treatment of a growth mindset educational intervention (not to be confused with a causal intervention), and 11 other potential confounding factors that could be parents of both the treatment and outcome. We select two of these confounding variables: an average measure of the fixed mindset at each student's school (inversely correlated with achievement and educational intervention) and the students' self-reported expectations of their own success (positively correlated with achievement and educational intervention). The full correlations between all four variables can be seen in Table I. Thus we maintain a hypothesized DAG structure as in Figure 4 identical to the pendulum model. Note that our interest is in regenerating the dataset with the treatment and targets as functions of the confounders, so we do not learn the effect of the intervention on the outcome. We will use our methods to generate datasets in which we can estimate the ATE to estimate our causal effect using a simple difference of means.

School Mindset Success Expectation
Intervention Achievement Score

Fig. 4: School Mindset DAG
The intuitive belief is that a naïve estimate of the ATE, calculated as the difference of means as in equation (9), would contain a positive bias due to the confounding variable of a student's own expectation. Students with higher expectations are more likely to participate in the growth mindset course (self-selection bias) but are also likely to have higher achievement anyway. Statistical adjustment techniques, such as Inverse Propensity Weighting (IPW), attempt to control for such confounders by measuring and weighting the effect based on the propensity to be treated. We will use such methods as a baseline for comparison, as we will first generate a dataset approximating the existing data distribution while learning some causal features. We will then employ a counterfactual model removing a confounding link and demonstrate a simulated dataset in which the naïve ATE aligns with the ATE measure using the statistical adjustment methods.

V. CAUSAL COUNTERFACTUAL GENERATIVE MODEL
We start from many of the same concepts as the original CausalVAE but begin by changing the enforced structure of the causal layer, allowing us to make direct modifications to the layer after training.

A. Limits of the CausalVAE for Counterfactuals
The causal layer in [8] has a purpose of passing some causal information about the latent space through from parents to children. However, there is a fundamental difference in how we would like to interpret our problem. Reiterating (2), Whatever causal structure is learned by A, there will always be a "leakage" of information via . This can be viewed as the output of a vanilla VAE, meaning that theoretically it can contain contribute everything for image generation. This leakage informs z without passing through the causal layer, so it weakens the need for A to learn all the causal structure of the problem. In the image space, this leakage of information improves generation and reconstruction and hence is desirable. However, it does not align with our objective of finding a good underlying causal structure. In the most extreme case, we could, in theory, find A = 0, which is still a valid DAG. In this case, no remaining causal information remains in the layer and the entire CausalVAE reverts back to a normal β-VAE.

B. Envisioning Bias-Free Models with CausalVAE
Here, we introduce CCGM as a modified and extended version of the CausalVAE, allowing for counterfactual models. In particular, we can directly manipulate the causal layer so that undesirable causal links learned from the data can be broken.
In CCGM, the encoder directly generates the output z, which is enforced to be standard-normally distributed. We pass this through our causal layer as one final mutatable bottleneck That is, it instantiates a linear SCM. One main distinction is that we solidify the structure of A by having exogenous and endogenous priors. This way, A can be split into a DAG term and a diagonal term: where B has 1 on the diagonal for exogenous variables and 0 if endogenous. This ensures that the trivial solution where A = I is never learned and enforces a causal relationship from the exogenous variables to the endogenous variables. Similarly, we add the non-linear mask to the causal layer just as in equation (3), but dropping the leakage.
When separating adjacency and estimation, we necessarily want to have a pre-training step for 5 epochs, where we train A to recognize the adjacency of the labels before applying the non-linear mask. After the pre-training, we apply training on both the A matrix and the non-linear mask, but there should be fewer changes as the mask should take care of the function approximations.
The ultimate goal of our work is to propose counterfactual causal models by directly manipulating this A matrix. The framework proposed by [8] requires one to retrain the entire model to generate a counterfactual A while fixing a path in the graph to zero, as their intervention method does not deal with the leakage. This is an expensive task in both time and computing power, making it unscalable for larger A's. Our method allows us to generate data about a hypothesized counterfactual space directly by breaking links in the causal graph, without the need to retrain the neural network.

C. General Structure of CCGM
While one could work with image-to-image VAEs, in our examples, we leverage as much tabular data as we can to reduce computational needs. In the pendulum example, we know the labels can be used as a perfect reconstruction of the data and so the labels that are provided act as at least a perfect bottleneck, containing more information than needed, in the reconstruction of images.
Furthermore, the label-to-label structure can be used as a pre-training step in determining a causal matrix. It then becomes a natural extension to apply the CCGM to tabular data. We no longer require a VAE setup, although we preserve the mild non-linear networks which allow for more complex causal functionality. Our experiments section will show a CCGM capable of generating tabular data with a reasonable representative distribution and a bias removed distribution. Note that noisy tabular data with hypothesized causal models (no known ground truth model or guarantee of endogenous/exogenous priors) present a new set of identification and estimation challenges.

VI. EXPERIMENTS
In this section, we evaluate the effectiveness of causal generative models on tabular and image datasets, by answering the following questions: (1) how does the performance of CCGM compare to the state of the art methods in reconstruction and causal logic; (2) how effective is CCGM for eliminating biases in image and and tabular datasets; and (3) how CCGM generates counterfactual models without extra training allowing for diverse and flexible data-generation. We compare the performance of CCGM and CausalVAE to generate counterfactual samples from a fixed causal model [8]. We further compare CCGM to advanced statistical adjustment methods for generating "de-biased" datasets vs. controlling biases statistically.

A. CausalVAE
Our experiments with the standard CausalVAE found that the model could handle interventions on specific latent space data, meaning that its decoder could causally disentangle some of the concepts. However, non-zero interventions did not appear to be working as intended. Figure 6 shows a sweep of interventions on the pendulum and sun position data, respectively, on the same image. Notice that the first intervention, corresponding to 0, works as intended. However, the pendulum does not change outside the 0 value, while the sun changes somewhat in an expected fashion, but the shadow does not respond.
Furthermore, we noticed little to no change in the results of certain interventions when generating counterfactual models, such as in Figure 5 below where a post training removal on the path from sun location to shadow position did not remove the effect propagation. These findings reflect that the CausalVAE was not designed to learn the full causal structure due to the leakage in .

B. Label-to-Label
Our initial experiments pertain to the label-to-label space, where we have four parameters (labels) that provide perfect information for image reconstruction.
We start with a set of labels u ∈ R n , where n = 4. These four labels correspond to the pendulum angle θ, the sun position x sun , the shadow width w shadow , and the shadow position x shadow , respectively such that u = [θ, x sun , w shadow , x shadow ] T . Then, u passes through an encoder to generate z ∈ R n , where we enforce the prior of p(z) ∼ N (0, I). This step allows us to sample new labels from the latent space drawn from a Gaussian distribution, as we see in the classic VAE [5]. Now, we subject z to the learned causal layer. Based on our designation of u, we set diag(A) = [1, 1, 0, 0], representing the exogenous and endogenous variables of u. Equation 13 is applied to to z, and the information of z should be preserved through the causal layer, even though the exact information of the endogenous variables is intentionally dropped.
Finally, this reconstructed latent space vectorẑ is passed into a decoder to reconstruct the original labelsû. For consistency of visualization and easy of human understanding, we pass these labels into a separate image generator to create all visualized images.
CCGM Generates Clean Label-to-Label Interventions. Since there are few parameters in the label-to-label space we are able to generate clean counterfactual models as well as interventions.
Our primary results for label-to-label is shown in Figures  7 and 8 sweeps on both θ and x sun , respectively. We take a true image and apply a range of interventions sampled from the range of the resultant sampling distribution (N (0, 1)) to generate counterfactual samples. Interventions on exogenous variables shows a response in the shadow variables, but the other exogenous variable should stay constant. Then, we apply the ideas of a counterfactual model. Instead of doing interventions on specific values, we break the link of x shadow with θ and x sun in A, respectively. Afterward, if we do the same interventions, the shadow position no longer responds to that intervention. While some of the results can be subtle, in Figure 7, the final image shows a noticeable difference in position before and after the counterfactual model and in Figure 8, the first and last interventions both show differences. The connection to shadow width remains, and so the shadow width still responds to the swept variable.

C. Label-to-Image
We then consider the more challenging problem of generating an entire image from the label information. Thus, we propose a label-to-image generative model based on the decoder of a VAE to use the disentanglement granted by the Causal Layer. We can use a pre-trained version of the A matrix coming from the label-to-label VAE to start our training of the causal generative model. For the sake of computational power, we keep the dataset in grayscale to reduce the image size by at least a factor of 3, but the physics aspects are still present.
Other than this additional pre-training step to learn the A matrix, the encoding step and the causal layer steps are still operating exactly the same as in the label-to-label VAE. We simply attach an image decoder after the causal layer. As in [8], we see that the images have mostly disentangled the endogenous variables and the encoder is able to create an image where interventions can happen. These images are displayed in Figure 9. Notice especially in the shadow position intervention that the sun position and pendulum angle have not changed. With CCGM, we can recreate the results from the label-to-label VAE in the label-to-image generator. These results are shown in Figures 10 and 11.
As in the label-to-label space, the top row of both of the figures shows the sweeps of θ and x sun in their latent space. With the interventions, the shadow responds accordingly and the complementary exogenous variable stays relatively consistent. The bottom row shows that the shadow position no longer responds to the interventions that are being enforced. In both Figure 10 and 11, the first and the last interventional images show noticeable movement from the non-counterfactual interventions.

D. Mindset Data
We begin by first training our CCGM method on the original Student Mindset data. In Figure 12, we see the generated achievement score from our model compared to the original dataset. The model is able to regenerate each of  the feature distributions. We take the top 30% as having the intervention (1) and lower 70% as no intervention (0) since this matches the base rates in the original dataset. The current CCGM operates on continuous variables and we treat the intervention values generated as a probability of treatment.
The results of an ATE on the generated dataset vs the three baseline measurements on the original dataset of ATE, IPW, and AIPW are shown in Figure 17. The generated dataset overestimates the ATE bias. We intervene on the path from student expectation to achievement scores, breaking the strongest positive correlation and see the CCGM "De-Bias" ATE estimation clearly drop below the advanced adjustment baseline methods. This makes intuitive sense since we leave the negatively correlated school mindset confounder which likely plays a small role in underestimating the ATE. Note that the advanced methods themselves are not necessarily ground truth, but they reflect an approximate ATE we would expect a "de-biased" dataset to have.
Further, Table II shows the full results for all models and multiple de-biasing schemes. It is clear that our generative model is able to produce a "de-biased" dataset accounting for the positive ATE bias for the Student Expectation confounder. Although removing both confounders does marginally increase the 95% bounds for out ATE as we would expect for a negative bias, it does not also do so when we only remove the school mindset. Here it becomes clear our model puts very little weight on this negative bias, a possible limitation of our model  with the noisy nature of this dataset and a point of interest for further inquiry. All results are shown with empirical standard deviations and 95% confidence interval bounds using bootstrap sampling methods (iters = 100).

VII. CONCLUSION
In this paper, we demonstrate the value of CCGM, an extension of previous causal generative model work that allows greater flexibility when considering counterfactual models and generating "out of distribution" data. We demonstrate the benefits of such a model on a simulated physics image dataset. We show the range of interventions and simulation of images outside of the training data, and outside of the ground-truth physics, with a simple adjustment after training the model. We then demonstrate results on a tabular dataset where groundtruth is not known. We show that we can learn the original data distributions, and simulate datasets which remove the impact of confounders in ways the make intuitive sense based on advanced statistical adjustment baselines. Much work is needed on refining the precision on noisy datasets, extending the framework to more complex causal models, and exploring the limitations based on the noise present and the target causal structure if known. We believe CCGM is a promising start within a growing field of work in causal generative models.  Fig. 17: Intervention probability distribution CCGM reconstruction example. Note that top 30% is used to bin data into binary before calculating ATEs.