{'fraction_non_alphanumeric': 0.038578961976131004, 'fraction_numerical': 0.013401530470639546, 'mean_word_length': 4.57638735352642, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Invariant Representations through Adversarial Forgetting

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks.


Introduction
Supervised machine learning models learn to associate a target y with underlying factors of data x that are informative of y. However, trained models often learn to associate irrelevant factors of x with y (Domingos 2012), e.g., stroke-width of text in optical character recognition. Learning incorrect associations between y and nuisance factors leads to overfitting. Models might also learn associations between the target y and factors that are correlated with y in collected training data due to external (sometimes historical) reasons, e.g., age, gender, and race in socio-economical prediction tasks. These biasing factors, which are sometimes known a priori, make trained models unfair to under-represented groups, posing ethical and legal challenges to the usage of such models. It is, therefore, imperative to develop models that are invariant to both nuisance and biasing factors.
Popular approaches for achieving invariance to undesired factors s have involved training with data augmentation through small perturbations to real x with respect to s (Bengio, Courville, and Vincent 2013). While these methods have been utilized for training deep neural networks (DNNs) (Ko et al. 2015;Krizhevsky, Sutskever, and Hinton 2012), in recent years methods have emerged that remove s from the latent representationsz, penalizing the model for the presence of s (Jaiswal et al. 2018;Jaiswal et al. 2019a;Li, Swersky, and Zemel 2014;Lopez et al. 2018;Louizos et al. 2016;Moyer et al. 2018;Xie et al. 2017;Zemel et al. 2013). These methods perform better than data augmentation, due to their approach of invariance by exclusion rather Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. than inclusion (Jaiswal et al. 2018;Jaiswal et al. 2019b;Hsu, Jaiswal, and Natarajan 2019).
We propose a novel framework for invariance in DNNs that promotes removal of information about s fromz through an adversarial forgetting mechanism. The working principle of the model is to map data x to an embedding z that encodes everything about x and use a forget-mask to filter out s information from z while retaining information about y to produce the invariantz. Specifically, an encoder network generates a latent code z from x, which is used to reconstruct x through a decoder. At the same time, a forgetgate network generates a mask m from x, which is multiplied elementwise with z to producez. The encodingz is used by a predictor to infer the target y. These components of the framework are trained adversarially with a discriminator that aims to predict s fromz. However, during training, gradients from the discriminator are allowed to only affect the training of the forget-gate. Finally, the framework is augmented with a regularizer that pushes the components m i of the forget-mask to be close to either 0 or 1, inducing disentanglement within the components of z and effective masking of s to produce an invariant representationz.
We show that the forgetting mechanism is equivalent to a bound on the mutual information I(z : z) and that coupled with the y-prediction task, can be interpreted as an information bottleneck. Further, by the data processing inequality, the forgetting mechanism bounds I(z : s). The generated mask can be manipulated through adversarial training to remove information about s from z. Empirical results show that the proposed framework exhibits state-of-the-art performance at inducing invariance to both nuisance and biasing factors across a diverse collection of datasets and tasks.
information bottleneck objective will remove all such nuisance s completely fromz (Achille and Soatto 2018b). The bottleneck objective is, however, difficult to optimize and has, hence, been approximated using variational inference in prior work (Alemi et al. 2016) or implemented as Information Dropout (Achille and Soatto 2018a).
The bottleneck objective cannot remove s fromz that are correlated with y (biasing factors). In such cases, it is necessary to force the exclusion of these biasing s fromz. Methods that employ such mechanisms additionally benefit the removal of nuisance s for which annotations are available. Moyer et al. (2018) achieve this by augmenting the bottleneck objective with I(z : s) and optimizing its variational bound, which we call the Conditional Variational Information Bottleneck (CVIB). The Variational Fair Autoencoder (VFAE) (Louizos et al. 2016) optimizes this objective indirectly as a Variational Autoencoder (VAE) (Kingma and Welling 2014) and uses Maximum Mean Discrepancy (MMD) (Gretton et al. 2007) to boost the removal of undesired s from the latent embedding. The Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al. 2005) constrained VAE (HCV) (Lopez et al. 2018) uses HSIC to enforce independence between the latent embedding and s. NN+MMD (Li, Swersky, and Zemel 2014) directly minimizes MMD as a regularizer for DNNs.
Invariance to nuisance variables has also been modeled as an implicit feature selection method within DNNs in the Unsupervised Adversarial Invariance framework (UAI) (Jaiswal et al. 2018;Jaiswal et al. 2019a) that splits data representation into an invariant embedding, which is used for predicting y, and a nuisance embedding through competitive training between prediction and reconstruction tasks combined with disentanglement between the two embeddings.
The Domain-Adversarial Neural Network (DANN) (Ganin et al. 2016) and the Controllable Adversarial Invariance (CAI) model (Xie et al. 2017) indirectly optimize an invariance objective using the gradient-reversal trick (Ganin et al. 2016) that penalizes a DNN if it encodes s. In the case of invariant representation learning for a supervised prediction task (which is the subject of this work), CAI, DANN and Fader Networks (Lample et al. 2017) reduce to the same model. In contrast, we propose a new forgetting mechanism that more explicitly "masks" s out of z, which encodes everything about x, to generate a new representationz that is maximally informative of y but invariant to s. Inspired by (1) the discovery of richer features in DNN classifiers upon augmentation with reconstruction objectives (Sabour, Frosst, and Hinton 2017;Zhang, Lee, and Lee 2016), and (2) the forgetting operation in Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) cells, the proposed framework adopts the idea of "discovery and separation of information" for invariance, which is foundationally different from the direct removal of undesired factors as in DANN and CAI.

Invariance through Adversarial Forgetting
Invariant representation learning aims to produce a mapping of data (x) to code (z) that is minimally informative of unde-sired factors of data (s) but maximally discriminative for the prediction task (y). There are two cases that arise from this formulation -(1) s is nuisance, i.e., there is little or no information shared between s and y asymptotically (e.g., pose in face recognition), and (2) s contains biasing factors, i.e., there is correlation between s and y, but for outside reasons, it is necessary to exclude these biases from the prediction process (e.g., gender, race, etc. in socio-economical prediction tasks using historical data). Invariance to s leads to robust models that generalize better on test data for case (1) (Jaiswal et al. 2018), and produces fair models that do not incorporate s while making predictions for case (2).
We induce invariance to s within a DNN using a novel approach of adversarial forgetting, which is inspired by forgetgates in Long Short-Term Memory (LSTM) cells in recurrent neural networks (Hochreiter and Schmidhuber 1997). Within the proposed framework, the model learns to embed everything about a data sample into an intermediate representation, which is transformed into an invariant representation through multiplication with a mask generated by an adversarial forgetting mechanism. Figure 1 shows the complete framework design. Data samples x are encoded into an intermediate representation z using an encoder E, while a forget-mask m (m i ∈ (0, 1)) is simultaneously produced from x through a forget-gate network F . The invariant representationz is then computed as the element-wise multiplicationz = z m. This is similar to how forget gates are used in LSTMs to "forget" certain information learned from past data. A decoder R is used to reconstruct x from z, such that E learns to encode everything about x into z. A predictor P infers y fromz, while an adversarial discriminator D tries to predict s fromz. Hence, the combined objectives of P and D aim to allow only factors of x that are predictive of y but not of s to pass from z toz.
The complete framework is trained with an adversarial objective such that the discriminator is pitted against all the other modules, as depicted with colors in Figure 1. The discriminator is allowed a more active role in the development of forget-masks m by allowing adversarial gradients from the discriminator to only flow to the forget-gate network and not to the encoder during training. This is illustrated in the figure with the break on the arrow between z and the multiplication operation. In order to further encourage the development of masks that truly filter out some information from z but retain everything else, a mask regularizer, in the form of m T (1 − m), is added to push components of m to either 0 or 1. We found the mask regularizer to always improve results in our experiments. The complete training objective can be written as shown in Equation 1.
The losses L y and L s are implemented as cross-entropy while mean squared error is used for L x in our experiments. The proposed model is trained using a scheduled update scheme similar to the training mechanism of the UAI model ( ). However, in practice, this is infeasible and training the discriminator much more frequently than the rest of the model (depending on the nature of the prediction task and the dataset) is sufficient to achieve good performance. This is especially true because the training of the discriminator is resumed from its previous state rather than starting from scratch after every update to the other modules. Therefore, the weights of the discriminator and the rest of the model are updated in the frequency ratio of k : 1. We found k = 10 to work well in our experiments. The model training does not incorporate the popular approach of gradient-reversal and instead follows (Jaiswal et al. 2019a). The targets of the discriminator D are set to the ground-truth s labels while updating D, but to random s values (sampled from the empirical s-distribution) when the parameters of the rest of the model are updated. Hence, D tries to predict the correct s during its training phase, but the rest of the model is updated to elicit random-chance performance at s-prediction, leading to the desired invariance to s. The model was implemented in Keras with TensorFlow backend. The Adam optimizer was used with 10 −4 learning rate and 10 −4 decay. The hyperparameters ρ, λ, and δ were tuned through grid search in powers of 10.
The proposed framework can also be extended to multitask settings, with z treated as the common encoding for tasks involving prediction of targets {y (1) , y (2) , . . . , y (n) } with corresponding undesired factors {s (1) , s (2) , . . . , s (n) }. Forget-gates F (j) are added to the framework, one for each prediction task y (j) , to generate associated masks m (j) and invariant representationsz (j) from z through adversarial training with discriminators D (j) , each of which tries to predict s (j) fromz (j) . Thus, the multi-task extension of the proposed model is intuitive and straightforward.

Characterizing Forgetting with Forget-gate
Forget-gates were introduced as components of LSTMs, where they cause "forgetting" of information from the past in the recurrence formulation conditioned on the input at a given step as well as the existing state. A forget-gate typically produces a mask m with m i ∈ (0, 1) that is multiplied elementwise with a latent encoding within an LSTM cell. Thus, the forget-gate can scale or remove information in each dimension of the encoding but not add information to it. Inspired by this formulation, we employ forget-gates to induce invariance to s, i.e., to "forget" s-related information. In this section, we characterize the erasure properties of the forget-gate in the proposed framework. Intuitively, if a mask element m i = 0, the information passed from z i tõ z i is also zero; likewise, if m i = 1, the information passed is complete. We characterize here the behavior for m i ∈ (0, 1), showing that under reasonable assumptions, there is a nontrivial forget regime besides zero. We show that the forgetgate acts as an information bottleneck, which can be manipulated for invariance to specific s.

Forget-gate
The proposed model generates a d-dimensional encoding z of x and a forgetting mask m with components m i ∈ (0, 1). These are multiplied element-wise to producez = z m. We consider the multiplication as a noisy operationz = z m + ε with a small ε in order to facilitate a theoretical analysis of "forgetting". We discuss the practicalities of ε later in Section 4.2. Assuming ε ∼ N (0, σ ε I), we get P (z|z) ∼ N (z m, σ ε I). In each dimension, we get: where H(ε i ) is constant with respect to z i and m i . Thus, the information passed from z i toz i is proportional to H(z i ).
Assuming that Var(z i ) is defined, the max-entropy Gaussian upper bound gives us the following in each dimension of the embedding (Cover and Thomas 2012): If m is non-random, the mutual information is: When m i → 1, assuming that Var(ε i ) Var(z i ), H(z i ) ≈ H(z i ). As m i → 0,z i = ε i , H(z i ) → H(ε i ) and I(z i : z i ) → 0. Importantly, when m i < V ar(ε i ) yet away from zero, the information loss is still non-trivial. The result in Equation 6 was derived assuming that m is fixed. We can extend this bound to random m, including those dependent on x as described in Section 3, by using the following identity (shown in Appendix A): This makes the bound on I(z i : z i ): Though more difficult to interpret, this has approximately the same characteristic as the fixed m i case. In order to extend this to the multivariate case (from a bound on I(z i : z i ) to that on I(z : z)), we first note that the max-entropy bound still holds for multivariate Gaussians, and by Hadamard's inequality, we can bound that distribution by its diagonal as: where Σ diag z has only the diagonal elements of Σ z and zero elsewhere. This gives us the bound: The max-entropy bound fails if there are degenerate elements of z, i.e. completely duplicate channels, but still holds on subsets of channels without duplicates. We have somewhat abused notation in this section; really, our bound is on I(z : (z, m)). While this is less intuitive, the distinction is necessary for data processing inequalities showing, e.g., that I(z : s) is controlled by the forget gate.

Practicalities of ε-noise and bottleneck-mask
With exact computation, information is only lost when m i = 0 because scaling by m i ∈ (0, 1) is an isomorphic map. In real computation, however, multiplication operations are not isomorphic due to imprecision in floating point arithmetic. These imprecisions, alongside commonly undertaken computational procedures (e.g., clipping) induce a non-trivial forgetting region, under which z i with "reasonable" variance may be forgotten. Thus, practically speaking, we do not need to add ε-noise artificially to erase information even when m i = 0. Hence, we do not need m i to be exactly zero to lose information. The background noise of computation is sufficient for I(z : z) to be smoothly controlled outside of zero m i . The proposed framework implements exactly this control on I(z : z), and thereby I(z : x). Further, since it also optimizes a distortion measure betweenz and y, this forms an information bottleneck (Alemi et al. 2016). For categorical y with cross-entropy loss, minimization of m i coupled with the prediction task is equivalent to the bottleneck objective from (Tishby, Pereira, and Bialek 1999). The goal of this work, however, is to generate invariant representations and not necessarily optimal bottleneck embeddings. In order to induce invariance to specific s, we learn the parameters of the bottleneck (forget-gate) so that it filters this information out of z (mechanism described in Section 3). This encourages the encoder and the forget-gate to generatez with minimal I(z : s). The adversary operates only on m and thus can be thought of as optimizing elementwise the channel between z andz, minimizing I(z : s) (or the equivalent general co-dependence term).
The masks generated by the forget-gate have an intuitive interpretation: for each component, it either allows information to pass from z i toz i or does not. The overall design of the proposed framework causes separation of factors of x that are correlated with s from those that are not, so that they occur in different components of z, allowing a componentwise mask to effectively include or exclude factors from the  ). However, for biasing s correlated with y, the forget-gate will choose whether to allow their inclusion inz based on the loss-weights in the objective.

Experimental Evaluation
The proposed framework is compared with NN+MMD, VFAE, CAI, CVIB and UAI. Performance is evaluated on two metrics: accuracy of predicting y fromz (A y ) using the jointly trained predictor and that of predicting s fromz (A s ) using a two-layer neural network trained post hoc. While a high A y is desired, for true invariance A s should be random chance for nuisances and the share of the majority s-class for biasing s. Mean and standard deviation are reported based on five runs, except when results are quoted from previous works. Relative improvements in error-rate (∆) are also reported with error-rate for A s defined as the gap between the observed A s and its optimal value. We further report evaluation results of the framework in a multi-task setting. Chairs. This is a dataset of 86,366 images of 1,393 types of chairs at 31 yaw and two pitch angles. We use the same version of this dataset as prior works, which is split into training and testing sets by picking alternate yaw angles, such that there is no overlap of angles between the two sets.

Robustness through invariance to nuisances
The chair type is treated as y and the yaw binned into four classes (front, left, right and back) as s. We use the same architecture for the encoder, the predictor, and the decoder as UAI (Jaiswal et al. 2018), i.e., two-layer networks. The discriminator is modeled as a two-layer network. Results of our experiments are summarized in Table 1. The proposed model achieves large improvements over the prior state-ofthe-art (UAI) on both A y and A s , with A s that is exactly random chance. Figure 2 shows the t-SNE visualization of z andz, exhibiting that z groups by orientation butz does not. Extended Yale-B. This is a dataset of face-images of 38 subjects captured under various lighting conditions. The prediction target y is the subject-ID, while the nuisance s is the lighting condition binned into five classes (four corners and frontal). For each subject, one image from each scategory is used for training and the rest of the dataset is used for testing (Jaiswal et al. 2018;Louizos et al. 2016;Xie et al. 2017). We use the same architecture for the encoder and the predictor as previous works (Jaiswal et al. 2018;Xie et al. 2017), i.e., one layer for each of these modules. The decoder and the discriminator are modeled with two layers each. Results of our experiments are shown in Table 2. The proposed model exhibits state-of-the-art performance on both A y and A s , with A s being exactly random chance. This shows that the proposed model is able to completely remove information about the lighting direction from the latent embedding while retaining high A y . Figure 3 shows the t-SNE visualizations of z andz and validates this claim as the clustering of the latent embedding completely changes from grouping by s for z to grouping by y forz.
MNIST-ROT. This is a variant of the MNIST (LeCun et al. 1998) dataset, which is augmented with digits rotated at ±22.5 • and ±45 • . The digit class is treated as y and the rotation angle as categorical s. We use the same architecture for the encoder, the predictor, and the decoder as UAI, i.e., two layers for the encoder and the predictor each and three layers for the decoder. The discriminator is modeled with two layers. Table 3 summarizes the results of our experiments on test images with rotation angles both seen and unseen (±55 • and ±65 • ) during training. Our model achieves not only the best A y but also A s that is exactly random chance, showing that it is able to filter out s while retaining more information about y, leading to more accurate y-predictions. The t-SNE visualizations of z andz shown in Figure 4 further validate this asz is clustered by y but has uniformly distributed s, while z shows distinct groups of s in each digit cluster.

Fairness through invariance to biasing factors
The Adult (Dheeru and Karra Taniskidou 2017) and German (Dheeru and Karra Taniskidou 2017) datasets have been popularly employed in fairness settings (Li, Swersky, and Zemel 2014;Louizos et al. 2016;Moyer et al. 2018;Xie et al. 2017). The former is an income dataset for classifying whether a person has more than $50,000 savings and the biasing s is age. The latter is used for predicting whether a person has a good credit rating, and gender is the biasing s. We use architectures similar to VFAE for encoders (two layers), predictors (one layer), and decoders (two layers), along with a two-layer discriminator. Results of experiments on these datasets are shown in Tables 4 and 5. The proposed model completely removes information about s, as reflected by A s being the same as the population share of the majority s-class for both the datasets. Previous works have also achieved perfect score on A s but the proposed model outperforms them on A y on both the datasets, showing that it is able to retain more information for predicting y while being fair by successfully filtering only biasing factors.

Invariance in multi-task learning
The proposed framework is evaluated on the dSprites (Matthey et al. 2017) dataset of shapes with independent factors: color, shape, scale, orientation, and position. The dataset was preprocessed following (Higgins et al. 2018), resulting in two classes for scale and four for position and orientation each. Shape (y (1) ) and scale (y (2) ) are treated as the prediction tasks, where shape is desired to be invariant to position (s (1) ) and scale to orientation (s (2) ). We use the same component networks that we used for Extended Yale-B. We compare results with a version of our model without the decoder, the maskers and the discriminators, i.e., both y (1) and y (2) are predicted from z. Evaluation could not be conducted on NN+MMD, VFAE, CAI and CVIB because they have onez and all tasks have to be invariant to the same s, and on UAI because it works only for a single y. Table 6 presents the results. The proposed framework achieves the same accuracies for y (1) and y (2) as the baseline, while maintaining random chance accuracies for s (1) (0.5) and s (2) (0.25) as opposed to significantly higher corresponding scores for the baseline. Hence, our framework works effectively in multi-task settings.

Conclusion
We have presented a novel framework for invariance induction in neural networks through "forgetting" of information related to unwanted factors. We showed that the forget-gate used in the proposed framework acts as an information bottleneck and that adversarial training encourages generation of forget-masks that remove unwanted factors. Empirical results show that the proposed model exhibits state-of-the-art  Acknowledgements. This work is based on research sponsored by the Defense Advanced Research Projects Agency (agreement number FA8750-16-2-0204). The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency or the U.S. Government.