{'fraction_non_alphanumeric': 0.04240206045818083, 'fraction_numerical': 0.01667344448962993, 'mean_word_length': 4.43160064791636, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Causal Knowledge Transfer from Task Affinity

Recent developments in deep representation models through counterfactual balancing have led to a promising framework for estimating Individual Treatment Effects (ITEs) that are essential to causal inference in the Neyman-Rubin potential outcomes framework. While Randomized Control Trials are vital to understanding causal effects, they are sometimes infeasible, costly, or unethical to conduct. Motivated by these potential obstacles to data acquisition, we focus on transferring the causal knowledge acquired in prior experiments to new scenarios for which only limited data is available. To this end, we first observe that the absolute values of ITEs are invariant under the action of the symmetric group on the labels of treatments. Given this invariance, we propose a symmetrized task distance for calculating the similarity of a target scenario with those encountered before. The aforementioned task distance is then used to transfer causal knowledge from the closest of all the available previously learned tasks to the target scenario. We provide upper bounds on the counterfactual loss and ITE error of the target task indicating the transferability of causal knowledge. Empirical studies are provided for various real-world, semi-synthetic, and synthetic datasets demonstrating that the proposed symmetrized task distance is strongly related to the estimation of the counterfactual loss. Numerical results indicate that transferring causal knowledge reduces the amount of required data by up to 95% when compared to training from scratch. These results reveal the promise of our method when applied to important albeit challenging real-world scenarios such as transferring the knowledge of treatment effects (e.g., medicine, social policy, personal training, etc.) studied on a population to other groups absent in the study.


INTRODUCTION
One of the most remarkable characteristics of humans is their ability to transfer causal knowledge learned in a scenario to other similar situations. In essence, natural neural systems can intuitively determine the similarity of scenarios and estimate average gains of an action in a new scenario from those of past experiences. It is highly desirable for artificial neural networks to have the same ability because of their numerous potential applications. For instance, mutations of old viruses often necessitate the development of new vaccines for treatment. To study the effect of new vaccine candidates, researchers will need to collect data from randomized control trials for both control and treatment groups. This can be both time-consuming and expensive. Therefore, it is exceptionally beneficial if a determination of the similarity of mutated viruses to old ones can be made, and the effects of vaccine candidates can be calculated based on this similarity and with potentially a small amount of data collected for the new scenario. It goes without saying that if such a transfer learning method can generally be developed, research on the effects of various treatments in many applications (e.g., medicines, personal training, social policy, etc.) can progress much faster.
Recently, there has been significant progress in transfer learning, especially in computer vision and natural language processing applications (Wang & Deng, 2018;Alyafeai et al., 2020). It has Figure 1: Inaccessibility to counterfactual data (which requires a "flipped" world where alternative treatments were assigned) makes transferring causal knowledge more challenging because we cannot have validation dataset to estimate test performance to avoid negative transfer. Figure 2: The overview of transfer learning in causal inference. The task distance is used to identify the closest task(s) from the set of prior tasks. The models and datasets from the relevant prior tasks are transferred to the target task.
been empirically observed that the representations learned for one task can help perform similar tasks (Yosinski et al., 2014), and that transfer learning significantly reduces the amount of required data Zhuang et al., 2021). While this is very promising, a challenge for transferring causal knowledge arises from statistical learning models' vulnerability to non-causal correlations. For example, camels and horses often exist in images with different background colors, and a classifier may learn to use these colors to classify these objects (Arjovsky et al., 2019;Geirhos et al., 2019;Beery et al., 2018). Another critical challenge for transferring causal knowledge is that, in practice, the performance of the trained model for estimating ITEs can never be computed. This is because counterfactual data can never be collected as shown in Figure 1. For example, to compute the effect of vaccination on an individual at some given time, she/he must be both vaccinated and not be given the vaccine, which is obviously impossible. This contrasts with conventional supervised learning problems, where practitioners often use a separate validation set to estimate the true accuracy.
The aforementioned challenges imply that much attention must be paid to selecting the appropriate source model to transfer from in causal knowledge transfer. Additionally, the similarity of scenarios must be calculated using a distance related to variations of counterfactual loss between scenarios. This motivates our work in this paper, where we propose a task distance between causal inference scenarios. The task distance is then used for transferring causal knowledge, as shown in Figure 2. Our contributions can be summarized as follows: 1. We first observe a special property of causal inference. Specifically, the absolute value of ITEs must be invariant to relabeling the treatment groups under the action of the symmetric group. Subsequently, we propose an intuitively appealing symmetrized Fisher task distance for which this property holds. 2. We provide both theoretical (e.g., Theorem 4) and empirical evidence (e.g., Figure 3) supporting the relevance of the symmetrized Fisher task distance to transferring causal knowledge.
3. We establish regret bounds in Section 4.2 on the learning of counterfactual outcomes and Individual Treatment Effects for target tasks in transfer learning scenarios. These bounds formally imply transferability for causal knowledge.
4. We present a representative set of causal inference datasets suitable for studying causal knowledge transfer. Some of these are well-established datasets in the literature, while others are derived from known causal relations in social sciences, physics, and mathematics.
5. We provide empirical evidence based on the above datasets that our methods can compute the ITEs for the target task with significantly fewer (up to 95% reduction) data points compared to the case where transfer learning is not performed.
The outline of this paper is given next. In Section 2, we briefly discuss existing prior work related to this paper. Next, we review the required mathematical concepts and backgrounds for this paper in Section 3. Section 4 presents our symmetrized Fisher task distance proposal and establishes theoretical bounds on its performance. Numerical experiments are provided in Section 5, demonstrating the performance of our method. Finally, we make our conclusions in Section 6.

RELATED WORK
Transfer learning is a crucial concept in training deep neural networks when encountering new target scenarios (please see Zhuang et al., 2021) and references therein). In this setting, prior learned models are used to increase the learning efficiency and decrease the required data. For instance, the parameters from a trained model may be used as initialization values for the target task. Many approaches in transfer learning (Silver & Bennett, 2008;Finn et al., 2016;Mihalkova et al., 2007;Niculescu-Mizil & Caruana, 2007;Luo et al., 2017;Razavian et al., 2014;Mallya & Lazebnik, 2018;Fernando et al., 2017;Rusu et al., 2016;Zamir et al., 2018;Kirkpatrick et al., 2017;Chen et al., 2018) have been proposed, analyzed and applied in various machine learning applications. Transfer learning techniques inherently assume that prior knowledge in the selected source model helps learn a target task. In other words, these methods often do not consider the selection of the base task to perform knowledge transfer. Consequently, in some rare cases, transfer learning may even degrade the performance of the model Zamir et al. (2018); Standley et al. (2020). In order to avoid potential performance loss during knowledge transfer to a target task, task affinity (or task similarity) is considered as a selection method that identifies a group of closest base candidates from the set of the prior learned tasks. Task affinity has been recently investigated and applied to various domains, such as transfer learning (Zamir et al., 2018;Dwivedi & Roig, 2019;Achille et al., 2019;Wang et al., 2019), neural architecture search (Le et al., 2021a;2022a;Le et al., 2021), few-shot learning (Pal & Balasubramanian, 2019;Le et al., 2022b), multi-task learning (Standley et al., 2020), and continual learning (Kirkpatrick et al., 2017;Chen et al., 2018). The related prior learned tasks are identified with similarity measures and then employed for knowledge transfer. Task affinity is inherently a non-commutative measure as it may be straightforward to transfer the knowledge from a more comprehensive task to a simpler task than the other way around (Standley et al., 2020;Le et al., 2021b).
While transfer learning and task affinity have been investigated in numerous application areas (as discussed above), their applications to causal inference have not been fully developed. Neyman-Rubin Causal Model and Pearl's Do-calculus are two popular frameworks for causal studies based on different perspectives. A central question in these frameworks is determining conditions for identifiability of causal quantities such as Average and Individual Treatment Effects. Past work considered estimators for Average Treatment Effect based on various methods such as Covariate Adjustment (a.k.a back-door adjustment) (Pearl, 2009;Rubin, 1978), weighting methods such as those utilizing propensity scores (Rosenbaum & Rubin, 1983), and Doubly Robust estimators (Funk et al., 2011).
With the emergence of Machine Learning (ML) techniques, more recent approaches to causal inference include the applications of decision trees (Wager & Athey, 2015), Gaussian Processes (Alaa & van der Schaar, 2017) and Generative Modeling (Yoon et al., 2018) to ITE estimation. In particular, deep neural networks have successfully learned ITEs and estimated counterfactual outcomes by data balancing in the latent domain .

MATHEMATICAL BACKGROUND
We first establish the notation and briefly review the required mathematical background.

CAUSAL INFERENCE
Let X ∈ X ⊂ R d be the covariates (features), T ∈ {0, . . . , M } be the treatment, and Y ∈ Y ⊂ R be the factual (observed) outcome. For every j ∈ {0, . . . , M } we define Y j to be the Potential Outcome that would have been observed if only treatment T = j, j ∈ {0, 1, · · · , M } was assigned. For example, in medical context, X is the individual information (e.g. weight, heart rate,etc), T is the treatment assignment (e.g., t = 0 when the individual didn't receive a vaccine candidate, and t = 1 where he/she did), Y is the outcome (e.g mortality data). A causal inference dataset is given by a set of factual observations where N is the number of samples. Without loss of generality, we can intuitively think of T = 0 as the case where no treatments are assigned. We present our results for M = 1 (binary case) in the sequel. However, our approach immediately applies to any positive integer M < ∞. In the binary case, we denote by control and treatment groups those assigned no treatments t = 0 and treatment t = 1. Definition 1 (ITE). The Individual Treatment Effect, also referred to as the Conditional Average Treatment Effect (CATE), is defined as: We assume that our data generation process respects overlap (i.e. 0 < p(t = 1|x) < 1 for all x ∈ X ) and conditional unconfoundedness (i.e. Y 1 ⊥ ⊥ Y 0 |X) (Robins, 1987). These assumptions assure that all underlying confounders are observed and there is a non-zero probability of observing both treatment assignments in each region of the covariate domain. These assumptions are sufficient conditions for the ITE to be identifiable (Imbens, 2004). We also assume that a true underlying function f (x, t) describes the causal relationship. In other words, the potential outcome y t for individual x with treatment t is f (x, t) + where is some additive measurement noise with mean zero. Extensions to non-additive and nonlinear noise models are straightforward and will not be discussed for the simplicity of the presentation. By definition τ (x) = f (x, 1) − f (x, 0). Letf (x, t) denote a hypothesis that estimates the true function f (x, t). Thus, the ITE function can then be estimated asτ (x) =f (x, 0) −f (x, 0). We let lf (x, t, y) denote a loss function that quantifies the performance off (·, ·). A possible example is lf (x, t, y) = (y −f (x, t)) 2 (L 2 loss) .
Definition 2 (Factual and Counterfactual Losses). For a hypothesisf and a corresponding loss function lf we define the factual and counterfactual losses respectively as lf (x, t, y) p(x, t, y)dxdtdy and We also define the factual loss for the treatment (t = 1) and control (t = 0) groups respectively as: t=1 Intuitively, the counterfactual loss corresponds to the expected loss value in a parallel universe where the roles of the control and treatment groups are exchanged. Definition 3. We define the Expected Precision in Estimating Heterogeneous Treatment Effect (PEHE) (Hill, 2011) as The value ε P EHE is often used as the performance metric for estimation of ITEs. Small factual and counterfactual losses (i.e. low ε P EHE ) are sufficient conditions for causal models to have good performance . Intuitively, this measures if a model has a good performance in predicting the effect both when the treatment is administered or not. Lower ε P EHE also implies that the model is good for predicting the ITEs. We note that the above measures of performance are not directly accessible in causal inference scenarios, because the calculation of the ground truth ITE values requires access to counterfactual values. In this light, we may resort to selecting a hypothesis that optimizes an upper bound instead, such as the one given in the following section (see Equation 2).

TARNET
Another challenging problem in causal inference is the selection bias arising from the difference between distributions of the treatment and the control group. TARNet  cleverly attempts to overcome this issue by creating a common representation space where the resulting representations of both the treatment and the control populations are forced to follow the same distribution. This can be thought of as emulating a Randomized Controlled Trial in the latent domain.
Formally, the TARNet model is defined as a pair of functions (Φ, h) where Φ : R d → R l is a representation function of the features and h : R l × {0, 1} → R is a function learning the two potential outcomes functions in the representation space. The hypothesis learning the true causal We denote the loss function lf by l (Φ,h) . TARNet uses integral probability metric (IPM) defined as where the supremum is taken over a given class of functions G to measure the distance between distributions. It is a consequence of Kantorovich-Rubinstein duality Villani (2009) that IPM reduces to 1-Wassertein distance when G is the set of 1-Lipschtiz functions as is the case in our numerical experiments.
TARNet  estimates the counterfactual outcomes by minimizing: The parameter α is referred to as the balancing weight since it controls the trade-off between the similarity of the representations in the latent domain, and the performance of the model on the factual data.

TASK REPRESENTATION
The ordered pair of a causal task T and its dataset D = (X, T ) will be denoted by (T , D), where dataset D itself consists of pair of covariates and their assigned treatments. Clearly D = D tr ∪ D te where D tr and D te are respectively the training and test sets.
We will mathematically formalize a sufficiently well-trained deep network representing a causal task-dataset pair (T , D) in the Appendix 7.3. From now on, we assume that all the previously trained models are sufficiently well-trained networks.

FISHER TASK DISTANCE
Here, we recall the definition of the Fisher Information matrix for a neural network, and well-defined Fisher task distance (Achille et al., 2019;Le et al., 2021b;2022b). Definition 4 (Fisher Information Matrix). For a neural network N θa with weights θ a trained on data D a , a given test dataset D b and the negative log-likelihood loss function L(θ, D), the Fisher Information matrix is defined as: where H is the Hessian matrix, i.e., H L(θ, D) = ∇ 2 θ L(θ, D), and expectation is taken w.r.t the data. It can proved that Fisher Information Matrix is asymptotically well-defined (Le et al., 2022b). In practice, we approximate the above with the empirical Fisher Information matrix: For completeness, we next review the task affinity score (Le et al., 2021b).
Definition 5 (Task Affinity Score (TAS)). Let (T a , D a ) and (T b , D b ) respectively denote the source and target task-dataset pairs. Let (respectively D tr b ) and D te a (respectively D te b ) be the training and test sets of dataset D b (respectively D b ), where the training for T a is performed using the source representation network N θa . Consider the Fisher information matrix H L(θ, D a ) of N θa with test data D te a . Let F a,a be the diagonal matrix of absolute values of elements of major diagonal of H L(θ, D a ) normalized to have unit trace. Let F a,b be constructed in an analogous manner but using the training data D tr b (instead of D te a ). The TAS from the source task T a to the target task T b is defined as: It can be proved that 0 ≤ T AS ≤ 1 where T AS = 0 denotes extreme similarity and T AS = 1 indicates extreme disimilarity. In Appendix 7.2, we prove under stringent assumptions that the order of TAS between candidate source tasks and the target task are preserved when a parallel universe experiment is performed in which the roles of the control and treatment groups are exchanged.

PROPOSED METHOD AND TRANSFERABILITY
We next propose a symmetrized Fisher task distance for causal inference.

LABEL-INVARIANT TASK AFFINITY
Our causal inference tasks are represented by TARNet type networks. We also restrict to the case, where all causal tasks under consideration have the same number of treatment labels M (e.g., be the source (respectively target) causal inference tasks. Clearly T a , T b ∈ {0, 1, . . . , M }.
Consider the symmetric group S M +1 consisting of all permutations of labels {0, 1, . . . , M }. For σ ∈ S M +1 , let T σ(b) denote the permutation of the target treatment labels under the action of σ. Let is the label-invariant task affinity distance between causal tasks T a and T b .
It follows from the above definition that the order of closeness of tasks under label-invariant task affinity closeness is robust to the architectural choice of the representation networks since task affinity distance has been shown to enjoy this property (Le et al., 2022b).

UPPER BOUND FOR TRANSFERRING CAUSAL KNOWLEDGE
In this section, we use superscripts T a and Sr to denote quantities related to target and source task respectively. Suppose that we have a model (Φ Sr , h Sr ) trained on a source causal inference task (T Sr , S Sr ). We apply the source model to a different target task described by (T T a , S T a ). For notational simplicity, we denote P (Φ(X)|T = t) by P (Φ(X t )) for t ∈ {0, 1}. We are interested in the performance of a well-trained source model when applied to a target task, i.e.
T a where τ T a is the individual treatment effect function of the target, Φ is the representation learning function, and h is the potential outcomes hypothesis. While it is difficult to estimate, this error can have an upper bound that only involves obtainable quantities if we make reasonable assumptions about relationship between source and target task (defined in the Assumption 4 below). We make following assumptions throughout this section: 1. Assumption 1: The loss function is non-negative, i.e. T a Φ,h (x, t, y) ≥ 0 for all (x, t, y) ∈ (X × {0, 1} × Y), 2. Assumption 2: Φ is injective (thus Ψ = Φ −1 exists on Im(Φ)) (We borrow this assumption from ), 3. Assumption 3: There exists a real function space G on R = Im(Φ) and a constant B T a Φ such that the function r → 1 4. Assumption 4: Causal Knowledge Transferability Assumption: There exists a function class G on Y such that y → l Φ,h (x, t, y) ∈ G and IP M G (P (Y Sr t |x), P (Y T a t |x)) ≤ δ for t ∈ {0, 1}.
Note that the causal knowledge transferability assumption implies that the outcome distributions (causal effects) of treatment t in source and target tasks need to be similar in order for transfer learning to be beneficial.
The main Theorem guarantees that causal knowledge can be transferred and is proved using two Lemmas that are stated below. These lemmas provide upper bounds on the factual and counterfactual losses for transferring causal knowledge and may be by themselves of independent interest. The proofs of these Lemmas and that of the Theorem are provided in the Appendix 7.2.
The above lemmas quantify the relationship between causality and transfer learning. In particular Lemma 2 bounds the inherently non-observable counterfactual loss by tractable quantities.
Φ Sr · IP M G (P (Φ Sr (X T a 0 )), P (Φ Sr (X T a 1 )) + 2δ) Theorem 1 implies that good performance on the target task is guaranteed if (1) the source model has small factual loss (first and second term in the upper bound) and (2) the distributions of control and treatment group features are similar in the latent domain (the rest three terms in the upper bound).

EXPERIMENTAL RESULTS
In this section, we first describe the datasets we have used for our empirical studies. Subsequently, we present empirical results about quantifying the gains of the proposed task distance and transfer learning.

CAUSAL INFERENCE DATASETS
We present a representative family of causal inference datasets suitable for studying causal knowledge transfer. Some of these are well-established datasets in the literature, while others are motivated by known causal structures in diverse areas such as social sciences, physics, health, and mathematics. Table 1 provides a brief description of the datasets used in our studies. A more detailed description is provided in Appendix 7.1.1. For each dataset, a number of corresponding causal inference tasks exist, which can be used to study transfer learning scenarios.

TASK DISTANCE AND COUNTERFACTUAL LOSS
Here, we show empirically the strong correlation between task distance (which only uses available data) and counterfactual loss (impossible to measure perfectly except for synthetic data). We show in Figure 3 that for different balancing weights α (see Equation 2), the correlation between the task distance and counterfactual error on the IHDP, RKHS, Movement, and Heat synthetic datasets for which counterfactuals are known. Our numerical results for the Jobs and Twins datasets verify that the proposed task distance can capture the symmetries within causal inference problems. We flip treatment labels (0 and 1) with the probability of flip = p (without any change to the features and the outcomes) independently for each control and test data point. In Figure 4, we demonstrate the trend of the symmetrized task distance between the original and the altered dataset by varying 0 ≤ p ≤ 1. The symmetry of task distance is evident (with some deviation due to limited training data for calculating the task distance). For all datasets, it can also be observed that the task distance trends are robust to variations in the balancing weight (also denoted as α in Equation 2).

COMPARISON OF PERFORMANCE WITH/WITHOUT TRANSFER LEARNING
Here we briefly discuss our experiments quantifying the impact of transferring causal knowledge on the size of required training data. In this experiment, we use Heat (Physics), Movement (Physics), IHDP, and RKHS datasets for which the counterfactual outcomes are available. We first fix a target causal inference task. For a wide range of balancing weights (α), we record the values of ε P EHE for the training of the model from scratch while increasing the size of training datasets (at the end training process). In this process, the training datasets are slowly expanded such that smaller training sets are subsets of larger ones. We then report the minimum ε P EHE achieved for each dataset size. For the Target task, we identify the closest source task and repeat the above process with a small amount of target task data. We then compare the performance with and without transfer learning to quantify the amount of data needed by transfer learning models to achieve the best possible performance without transferring causal knowledge. The results are summarized in Table 2, which demonstrates that transferring causal knowledge decreases the required amount of training data in this setting by a percentage between 75% and 95%.

CONCLUSION
In this paper, we proposed a method for causal transfer learning based on a task affinity framework.
To this end, we constructed a new task distance suitable for measuring the similarity of causal inference tasks. Given a new causal inference task, we transferred the causal knowledge from the closest available trained task. We provided theoretical analysis proving the transferability of causal knowledge. Extensive Simulations on a representative family of datasets provide empirical evidence demonstrating the gains of our method. Reductions as much as 95% in the amount of required training data for new scenarios were observed.

APPENDIX
Here, we provide the datasets description, theorems and the proofs for those theorems.  (2011) based on real covariates available from the Infant Health and Development Program (IHDP), studying the effect of development programs on children. The features in this dataset come from a Randomized Control Trial, and the potential outcomes were simulated using Setting "B" in (Hill, 2011), hence the word semi-synthetic. The dataset consists of 747 individuals (139 in the treatment group and 608 in the control group), each with 25 features. Hill generated the potential outcomes with Y 0 ∼ N (exp(β T ·(X +W )), 1), where W has the same dimension as X with all values = 0.5 and Y 1 ∼ N (β T (X +W )−ω, 1) with ω = 4. β is 25-element vector of regression coefficients randomly sampled from a categorical distribution with support (0, 0.1, 0.2, 0.3, 0.4) and respective probabilities µ = (0.6, 0.1, 0.1, 0.1, 0.1). We refer to the dataset generated according to these parameters as the base dataset.
We retain the base dataset and introduce 9 new settings according to Table 3 by varying µ and ω. We also generate 10 new datasets for each setting, each consisting of 747 individuals (139 in the treatment group and 608 in the control group) by running the same process but with different random samples of the aforementioned Gaussian distribution.  Jobs The original Jobs dataset (LaLonde, 1986) has 619 observations. The causal inference task is to learn the effect of participation/lack of participation in a specific professional training program (corresponding to receiving a treatment t = 1) at a time on the success in landing a job in the following three years. We generate a family of related datasets by randomly reverting the treatment assignments of the original dataset with various probabilities p ∈ [0, 1]. Specifically, to generate a dataset, we first choose a probability value p ∈ [0, 1], and then alter individuals (original) treatment assignment (i.e., 0 ↔ 1) with probability p. We choose values p ∈ {0 = 0/9, 1/9, 2/9, 3/9, 4/9, 5/9, · · · , 9/9 = 1}. Clearly, p = 0 corresponds to the original dataset, and p = 1 corresponds to all reverted treatment assignments. We choose the original Jobs dataset (LaLonde, 1986) as the base dataset for our experiments, as discussed in Section 7.1.2.
Twins The Twins dataset was first introduced by Louizos et al. (2017) based on the collected data about twins' births in the United States from 1989 to 1991. It is assumed that twins share significant parts of their features. We consider whether one of the twins was born heavier than the other as the treatment assignment and if he/she died in infancy (mortality) as the outcome. We divide the twins into two groups: In the treatment (respective control) group, we consider the outcome for the heavier (respectively lighter) twin as factual. In both groups, the outcome for the remaining twin is assumed to be counterfactual.
We first construct a base dataset by selecting a set of 2000 pairs of twins from the original dataset (Louizos et al., 2017). Then, each element is assigned to the treatment group according to a Bernoulli experiment with the probability of success q = 0.75.
Next, the base dataset is used to generate more datasets. In an analogous manner to that of the Jobs dataset, we generate a family of related datasets by randomly reverting the treatment assignments of the base dataset (0 ↔ 1) with corresponding probabilities p ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5, · · · , 1}. For instance, to generate dataset i = 1, 2, · · · , 11, we let p i = (i − 1)/10 revert the individual treatment assignments in the base dataset Bernoulli experiment with probability of success p i . Clearly, p = 0 corresponds to the original dataset while p = 1 corresponds to all treatment assignments reverted.
RKHS We generate 100 Reproducing Kernel Hilbert Space (RKHS) datasets, each having 2000 data points. For each dataset, we start by generating the treatment and the control populations X 1 , X 0 ∈ R 4 respectively from Gaussian distributions N (µ 1 , I 4 ) and N (µ 0 , I 4 ). We sample µ 1 ∈ R 4 and µ 0 ∈ R 4 respectively according to Gaussian distributions N (e e e, I 4 ) and N (−e e e, I 4 ) where e e e = [1, 1, 1, 1] T is the all ones vector.
Subsequently, we generate the potential outcome functions f 0 and f 1 with a Radial Basis Function (RBF) kernel K(·, ·) as described next.
We will refer to the first constructed dataset in the above as the base dataset.
Note that in the above, all the generated potential outcome functions are in the same RKHS.
Heat (Physics) Consider a hot object left to cool off over time in a room with temperature T 0 . A person is likely to suffer a burn if he/she touches the object at time u.
The causal inference task of interest is the effect of room temperature T 0 on the probability of suffering a burn. This family consists of 20 datasets; each includes 4000 observations with 2000 in each control and treatment group. The treatment in our setting is t = 1 when T 0 = 5, and t = 0 when T 0 = 25.
The treatment and control groups touching times are respectively sampled from two Chi-squared distributions χ 2 (5) and χ 2 (2) (intentionally in order to create artificial bias).
From the solution to Newton's Heat Equation (Winterton, 1999) the underlying causal structure is governed by the equation where T (u) is the temperature at time u and C, k > 0 are constants.
We let T 0 = 25 and C = 75 for all the control groups in the datasets. Similarly, we let T 0 = 5 and C = 95 for all the treatment groups in the datasets. We choose 20 values of k = {0.5, · · · , 2} uniformly spaced in [0.5, 2]. For each value of k, we generate a new dataset. The dataset corresponding to k = 0.5 is referred to as the base dataset.
Let T 0 (u) and T 1 (u) respectively denote the temperature at time u for the control and treatment groups. The potential outcomes Y 0 (u) and Y 1 (u) corresponding to the probability of suffering a burn at time t for respectively the control and treatment groups are given by Movement (Physics) Consider a falling person in the air encountering air resistance. Opening her/his parachute can change the air resistance and control its descent velocity. The causal inference task of interest is the effect of the air resistance (e.g., with t = 1 or without parachute t = 0) on the object's velocity at different times.
This family consists of 12 datasets. Each includes 4000 observations with 2000 in each treatment and control group. Here, the covariate is the time u, and the outcome is the velocity at time u. The treatment and control groups' times are respectively sampled from two Chi-squared distributions χ 2 (2) and χ 2 (5) (intentionally in order to create artificial bias).
The underlying causal structure is governed by an ordinary differential equation (ODE) with the following analytical solution describing the velocity of a person at time u: where C = k m , m and k are respectively the mass, and the air resistance constant, and g = 10 is the gravitational constant of earth. In the above v 0 = v(0) is the initial velocity at time u = 0. We assume v 0 = 0, corresponding to a free fall without initial velocity.

DETAILS OF EXPERIMENTS
In this paper, we first create various causal inference tasks from the above families of datasets. For each family of datasets (e.g. IHDP, Jobs, Twins), the base task is created from its base dataset. Similarly, we construct the other tasks from the remaining datasets in that family. In order to study the effects of transfer learning on causal inference, we define the source tasks and the target tasks as follows: • In the first experiment in Section 5.2, we choose the base task to be the source task and the other tasks to be the target tasks.
• In the second experiment in Section 5.3, we choose the base task to be the target task and the other tasks to be the source tasks.

PROOF OF LEMMAS AND THEOREMS
We will use the following known results  for causal inference. The proofs for these results are given in  .
For x ∈ X , t ∈ {0, 1}, with notational simplicity, we define Theorem 2 (Bounding The Counterfactual Loss). Let Φ be an invertible representation with inverse Ψ. Let p t=i Φ = p φ (r|t = i), i ∈ {0, 1} Let h : R × {0, 1} → Y be a hypothesis. Assume that there exists a constant B Φ > 0 such that for t = 0, 1, the function g Φ,h (r, t) := 1 BΦ · L h,Φ (Ψ(r), t) ∈ G. Here, we have Theorem 3 (Bounding the P EHE ). The Expected Precision in Estimating Heterogeneous Treatment Effect P EHE satisfies Next we relate the performance of target task T a,t=0 F (h, Φ) to that of a source task Sr,t=0 F (h, Φ). Without loss of generality, we present the proof for the case t = 0.
We make the following assumptions throughout the sequel.
We next upper bound Θ and Γ.
To bound Θ, we use the following inequality: With the above inequality: To bound Γ, we use the change of variable formula Combining the above upper bounds for Γ and Θ, we have We conclude that This concludes the proof.
Proof of Lemma 2 We apply Theorem 2 to establish an upper bound for the counterfactual loss of the target task and subsequently apply Lemma 1 .
This concludes the proof.
Proof of Theorem 1 By applying Theorem 3, we get T a P EHE (h, Φ) ≤ 2 T a,t=0 F (h, Φ) + T a,t=1 F (h, Φ) + B T a Φ IP M G P (Φ(X T a 0 )), P (Φ(X T a 1 )) After applying Lemma 1 to the first and second terms in the above: T a P EHE (Φ S , h S ) ≤ 2 Sr,t=0 F (Φ, h) + B T a Φ · IP M G P (X T a 0 ), P (X Sr 0 ) + δ + Sr,t=1 F (Φ, h) This concludes the proof.

TASK DISTANCE MATERIAL
Let P N θ (T , D te ) ∈ [0, 1] be a function that measures the performance of a given model N θ parameterized by θ ∈ R d on the test set D te of the causal task T . Definition 6 (ε-approximation Network). A model N θ is called an ε-approximation network for a task-dataset pair (T , D) if it is trained using the training data D tr such that P N θ (T , D te ) ≥ 1 − ε, for a given 0 < ε < 1.

TASK DISTANCE BETWEEN COUNTERFACTUAL TASKS
In the following section, we denote the pair a = (T a , D a ) by a F = (T a F , D a F ) (respectively a CF = (T a CF , D a CF )) whenever D a is sampled from the factual (respectively counterfactual) distribution. We refer to (T a F , D a F ) and (T a CF , D a CF ) as the corresponding factual and counterfactual tasks.
The following theorem proves that the order of proximity of tasks is preserved even if we go to a parallel universe where we observe the counterfactual tasks instead. In other words, a task, which is more similar to the target task when measured using factual data, remains more similar to the target task even when measured using counterfactual data.
and the TAS between the factual and the counterfactual can be arbitrarily small