{'fraction_non_alphanumeric': 0.040390674228103336, 'fraction_numerical': 0.012980466288594833, 'mean_word_length': 4.590172595984502, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linear Guardedness and its Implications

Previous work on concept identification in neural representations has focused on linear concept subspaces and their neutralization. In this work, we formulate the notion of linear guardedness -- the inability to directly predict a given concept from the representation -- and study its implications. We show that, in the binary case, the neutralized concept cannot be recovered by an additional linear layer. However, we point out that -- contrary to what was implicitly argued in previous works -- multiclass softmax classifiers can be constructed that indirectly recover the concept. Thus, linear guardedness does not guarantee that linear classifiers do not utilize the neutralized concepts, shedding light on theoretical limitations of linear information removal methods.


Introduction
Human-interpretable concepts, such as morphology (Vylomova et al., 2017), syntax (Linzen et al., 2016), semantics (Belinkov et al., 2017) and gender distinctions (Bolukbasi et al., 2016;Caliskan et al., 2017) were shown to be represented by neural models of text. In this paper, we study the ability to remove such concepts from neural representations, and the extent to which the information can be recovered later on. We follow the rich literature on information removal and bias mitigation, and formally define the notion of guardedness, the inability to directly predict a given concept from the representations. This paper focuses on the implications of guardedness of a set of representations on the behavior of subsequent classifiers that are trained over them.
Removing gender information, an objective similar to the condition we formalize as guardedness, has been previously studied as a way to mitigate bias in neural models. Information removal techniques are dominated by adversarial methods (Edwards and Storkey, 2015;Xie et al., 2017;Chen et al., 2018;Elazar and Goldberg, 2018;Zhang et al., 2018), that are applied during training. More recently, linear methods have emerged as an a tractable alternative (Bolukbasi et al., 2016;Dev and Phillips, 2019;Ravfogel et al., 2020;Dev et al., 2020;Kaneko and Bollegala, 2021). Bolukbasi et al. (2016) have pioneered linear inforamtion removal methods, by identifying a linear subspace that corresponds to gender, and erasing it from the representations. This operation approximates a condition of linear guardedness: the inability of linear predictors to recover the protected attribute from the representations.
Guardedness with respect to a protected attribute is an intrinsic measure of bias, in the sense that it concerns only with the representations of the model, and not with its predictions. However, users that apply the model in the real-world are often concerned not with the representations of the model, but rather with the fairness of its predictions (that is, whether it expresses extrinsic bias). Empirical work has shown that attempts to impose intrinsic fairness often fail in achieving extrinsic fairness (Goldfarb-Tarrant et al., 2021), and that mitigating extrinsic bias has a varied effect on extrinsic bias measures . In this work, we rigorously analyze the connections between linear guardedness and different extrinsic fairness measures, relying on the recently proposed method of Ravfogel et al. (2022) that is effective in achieving linear guardedness.
Concretely, we formalize the notion of linear guardedness, and study its implications. We ask: if X is a linearly guarded representation with respect to a protected attribute Z, and Y is a main-task prediction of a linear model over X, can we bound the extent to which Y indirectly leaks information on Z? We find a positive answer in the special case of |Z| = |Y| = 2. Conversely, for multiclass classification with softmax, we demonstrate that the main-task prediction can leak information on Z, pointing out to inherent limitations of the notion of linear guardedness, that was not observed by previous works on bias mitigation and fairness.

Information-Theoretic Guardedness
In this section, we present an information-theoretic approach to guardedness, which we couch in terms of V-information (Xu et al., 2020).
Setup. In our formalization, we consider a classification case, where X is vector-valued random variable over possible representations, Y is discrete random variable that denotes the main-task label (e.g., sentiment), and Z is a binary-valued random variable that denotes a protected attribute (e.g., gender). We assume that we observe samples from the joint distribution p( Intuitively, a set of representations is guarded if it is not possible to predict z n from x n using a specific predictive family. Note, however, that this does not mean that the mutual information I(X, Y) = 0 because we are only interested in a specific family, e.g., linear classifiers. Xu et al. (2020) framework based on V-information provides a natural encoding for this idea. In their setup, let V be any set of functions, e.g., all binary logistic regression classifiers. They define the conditional V-entropy as the best log likelihood a predictor from the family V can achieve in predicting Z from X, i.e., They further define V-information as a quantity similar to mutual information This is the best approximation of the mutual information realizable by classifier belonging to a given predictive family V. Note that in the case of logistic regression classifiers, (2) can be approximated by calculating the logistic loss of the classifier on a given set of examples, as H V (Z) is the entropy of the label distribution, and H V (Z | X) is the minimum achievable value of the logistic loss.

Guardedness
Having defined V-inforamtion, we can now define guardedness as a condition where the Vinformation is small. We say that representations X are guarded with respect to family of predictors V if predictors from that family cannot predict Z from X.
Definition 2.2 (Empirical Guardedness). In practice, we are given a dataset D. We say that the dataset is -empirically guarded if for any predictor in V, the cross-entropy loss is close to the entropy of the label distribution: |L(X, Z; θ) − H Z | ≤ .

Definition 2.3 (Linear Guardedness). A dataset is
-linearly guarded if it is -guarded for the family V of logistic regression classifiers.
In practice, according to this definition a dataset is linearly guarded if no linear classifier can perform better than a naive classifier that completely ignores X and always predicts Z according to the proportions of each label. Efficient algorithms have been proposed to approximate the condition of linear guardedness (Ravfogel et al., 2020(Ravfogel et al., , 2022. They identify an orthogonal projection matrix P that erases a low-dimensional linear subspace from the representation space, such that linear classifiers can no longer recover z n from P x n . In the following sections, we study the implications of this condition on the "fairness" of subsequent linear classifiers trained over the guarded representations. For instance, suppose we imposed guardedness on the last hidden representations of a transformer model, and then fitted a linear classifier over these representations to predict sentiment. Can the predictions of the sentiment classifier indirectly leak information on gender?

Binary Main-Task
We begin by analyzing the implications of information-theoretic guardedness on the independence of a binary linear-classifier trained over the guarded representations. Our analysis is based on the following claim that the composition of two logistic classifiers has a linear decision boundary.
Proof. To find the decision boundary, we require σ(βσ(α x)) = a for some a ∈ (0, 1). We first start with the definition of σ and the rest proceeds by algebraic manipulation: It is now self-evident that the decision boundary is represented by an equation linear in x. 1 Now, we show that when both the main task and the protected attributes are binary, and the predictive family we consider are the logistic functions followed by a hard decision rule, any linear classifier that predicts the main-task label cannot recover the information on Z.
Definition 2.5 (Discretized Logisitc Regression). The predictive family of logistic regression classifiers discretized with a parameter δ ∈ (0, 1) is defined as is a function that maps the probability value to one of two binary values: Claim 2.6. Let V δ be the predictive family defined in Definition 2.5. Let g * be a logistic regression classifier that predicts a binary target Y from X such that Y = g * (X). Then, if I V δ (X → Z) < , it also holds that > . Let f * ∈ V the predictor that realizes the infimum in the definition of I V δ ( Y → Z). By assumption, By assumption, Now, consider the function h * := f * • g * . By Claim 2.4, there is a linear logistic function q * ∈ V δ such that ∀x, q * (x) = h * (x). We get: The conclusion, This contradicts the assumption I V δ (X → Z) < .

The Multiclass Case
The above discussion shows that when both Z and Y are binary, (approximate) linear guardedness implies (approximate) independence of the main-task predictions Y that are generated by a logistic regression classifier. In previous works (Ravfogel et al., 2020;Elazar et al., 2021) it has been argued that if linear guardedness is imposed on the lasthidden representation of the model, the last linear layer (a layer whose output produces, e.g., a distribution over the vocabulary in a language model) cannot recover the information, since it is linear. Indeed, Gonen et al. (2020) (2022) have used linear interventions to assess the influence of concepts on the behavior of language models, where |Y| 2. 2 We note that there is a key flaw in this argument. If the data is linearly guarded, it is easy to see that the logits that are linearly calculated from the guarded representation cannot encode the information. However, multiclass classification is usually performed by a softmax classifier. While the decision boundary of the softmax classifier for every pair of labels is linear, 3 the softmax is a nonlinear function applied over the logits. This non-linearity is at times sufficient to break guardedness.
To see that, consider the data in R 2 presented in Fig. 1. The data is linearly guarded. We will examine a 4-class linear softmax classifier trained over the space (|Y| = 4), which is adversarially built to recover the Z information. Assume the parameter matrix Θ ∈ R 4×2 of this classifier is composed of 4 rows θ 1 , θ 2 , θ 3 , θ 4 such that , for some positive α. These directions encode the quadrant of a point: when the norm of the parameter vectors is large enough (i.e, for a large enough α), the softmax score of class i will be close to 1 if, and only if, the input is in the ith quadrant. Given the information on the quadrant, the data is rendered perfectly linearly separable. Thus, the labels Y predicted by a multiclass softmax classifier can 3 Class i will have higher softmax probability than class j iff (θi − θj) x > 0. recover the linear separation according to Z.
This argument can be generalized to a separation which is not axis-aligned (Fig. 2).
Lemma 2.7. Consider any linearly guarded data that is separated to K pure regions (with respect to Z) R 1 , . . . , R K , that are defined by the intersection of subspaces θ 1 , . . . , θ K (Fig. 2). 4 There is a Kclass linear softmax classifier that encodes the region of a point x in its probabilities Softmax(Θx).
Proof. By assumption, we can identify each region by its orientation relative to each subsapces. For instance, for region number 1 in Fig. 2, a point belongs to that region if and only if θ 1 x < 0 while θ 2 x > 0. Formally, for each point x, let f (x) be a vector that stores the orientation of x relative to each of the θ i : We also define a "region identifier" binary vector for each region I j ∈ {−1, 1} K such that: Thus, a point x is in region R j if, and only if, Consider a K-class softmax classifier with a parameter vector Θ that contains, in its jth row, the vector θ * j := α K i=0 I j [i]θ i -that is, we sum over all θ i and give positive weight to a vector θ i if a positive dot product with it is a necessary condition to belonging to region j. Additionally, we scale the parameter vector by some positive scalar α.
For a large enough α, coordinate j in the softmax defined above will be arbitrary close to 1 if, and only if, the point x belongs to region j: The ratio between the softmax value in coordinate j and any other coordinate k is given by Note that K subsapces can define more than K regions, but we can always add "dummy subspaces" that are outside the data distribution.
Note that the weight I j [i]−I k [i] will be 0 if θ i x > 0 is a necessary condition to both region j and region k (or if it rules out both of them); otherwise, it will be 2 if θ i x > 0 is a necessary condition to region j, and will be -2 if θ i x < 0 is a necessary condition for region j. Therefore, if the point x is in region j, we only sum over positive values in the exponent, and the ratio becomes arbitrarily large for a large enough α -that is, the softmax probability of class j approaches 1.
Claim 2.8. When there are K pure regions that are defined by intersection of subspaces, the predictions of a K-class softmax classifier can break guardedness and fully recover Z.
Proof. By Lemma 2.7, it is possible to construct a softmax classifier that encodes region identity. Given the region identity, it is trivial to linearly classify according to Z.
This counterexample demonstrates that one should be cautious when arguing on the implications of linear guardedness in when multiclass softmax classifiers are applied over the guarded representations. When linear guardedness with respect to a binary Z is imposed, there may still exist a set of k > 2 linear separators that together separate Z well.

Accuracy-Based Guardedness
We now define an accuracy-based notion of guardedness and discuss its implications. Note that the information-theoretic notion of guardedness described above does not directly concern with accuracy: consider a binary logistic classifier on a balanced data, that always attributes a probability of 1 2 + δ to the correct class, for some small δ. The cross-entropy loss of such a classifier will be arbitrary close to the entropy log(2), even though it has a perfect accuracy. This disparity calls for an accuracy-based notion of guardedness: Definition 3.1 (Accuracy-based Guardedness). A data X is linearly guarded with respect to a protected attribute Z if for any linear classifier q θ = Z it holds that P When focusing on accuracy in predicting Z, it is natural to consider Independence or Demographic Parity of the main-task classifiers that is trained over the representations. We define the independence gap as GAP ind : where dist is some measure of difference between the two distributions. Following , we use the L 1 metric, which is linearly related to the total variation distance. Accordingly, we define

GAP ind
We prove that if the data is (approximately) linearly guarded, and globally balanced with respect to Z (P [Z = z] = P [Z = ¬z] = 0.5, then the prediction of any linear binary main-task prediction is (approximately) independent of Z. Note that is true regardless of any imbalance of the protected attribute Z within each main-task class y ∈ Y: the data only needs to be globally balanced.
Proof. See Appendix App. A.1 for the proof.

Imbalanced Protected Attribute
The relation between accuracy-based guardedness and independence relies on the fact P [Z = z] = P [Z = ¬z] = 0.5. Surprisingly, for the imbalanced case, linear guardedness does not guarantee that the predictions Y of the main-task classifier are -independent of Z. As a counterexample, consider a dataset sampled from the distribution in Tab. 1.
Further, assume that the dataset is generated by the following generative process: So the 2-dimensional input vectors contain the Y and Z values themselves, with some normal noise. Claim 3.4. The transformation X ←X, that is realized by x n ← [x n [0], 0] (zeroing-out the second coordinate in each x n ), renders the dataset X linearly guarded.
Proof. After zeroing out the components that encode z n , the vectors only contain the information y n . By Tab. 1, the majority class in Z is Z = 1 (with P [Z = 1] = 0.75]). That is, a majority classifier will always predict Z = 1. Further, Z = 1 is also the majority class when conditioning on Y = 1 and on Y = 0 individually (P Thus, knowing Y provides no additional information that is useful in predicting Z, and thus knowing Y cannot improve Z prediction accuracy over the majority accuracy of 0.75.
Claim 3.5. Linear guardedness on the guarded dataset that is generated according to Claim 3.4 does not imply independence.
Proof. Consider a case where the predictions Y of the main-task classifier overX equal to Y (such perfect classifier exists when the noise parameter is small enough). By Tab. 1, the distribution of Y-and thus, also the distribution of Y, a perfect classifier-is not independent of Z (e.g. P [Y = 1] = 0.55 while P [Y = 1 | Z = 1] = 0.6).

Experimental Evaluation
We evaluate the extent to which the labels induced by linear main-task classifiers-either binary or multicalss-leak inforamtion about protected attribute.
Data. We perform experiments on gender bias mitigation on the Bias in Bios dataset of De-Arteaga et al. (2019), that is composed of short biographies, annotated by both gender (protected attribute) and profession (main-task labels). We embed each biography with the [CLS] representation in the final hidden representation of pre-trained BERT.
Approximating linear guardedness. To approximate the condition of linear guardedness, we use RLACE (Ravfogel et al., 2022). The method is based on an minimax game between a linear predictor that aims to predict the concept of interest and an orthogonal projection matrix that aims to prevent the recovery of the concept. We run RLACE with linear logistic-regression predictors. The process results in a projection matrix P that, empirically, prevents logistic classifiers from predicting gender after the transformation x n ← P x n . Note that while RLACE is based on a continuous minimax game, Claim 2.6 concerns with the family V δ of discretized logistic regression classifiers (Definition 2.5). As we do not have a technique to induce guardedness against this predictive class, we use the soft-probability logistic classifiers adversaries as an (empirically good) approximation.
Quantifying leakage. To quantify the extent to which subsequent classifiers trained on the guarded representations can leak inforamtion, we train either binary or softmax logistic regression on top of the guarded representations to get predictions Y, followed by an additional logistic classifier that predicts Z from Y. We estimate the information Y leaks on Z by the logistic loss of the second classifier, according to Definition 2.2. We consider two types of main-task classifiers Y: (1) Y prof , the predictions of a logistic profession classifier trained on the projected data; and (2) Y adv , the predictions of a logistic classifier that is adversarially constructed to predict gender, as described below.
See App. A.2 for complete details on our experimental setting.

Binary main-task
We start by evaluating the case where both Z and Y are binary.
Experimental Setting. To simulate a binary main-task classification, we randomly sample 15 pairs of professions from the dataset (see App. A.2), and run binary classification on the examples belonging to these professions alone. After running RLACE, we get a separate gender-neutralization matrix P for each pair of professions, we project the representations x n ← P x n and verify that logistic classifiers trained on the projected data get accuracy of less than 2% above majority-accuracy.
For each pair of professions, we evaluate several quantities: • Evaluating I V δ (X → Z). To get an upper bound on the linearly-present information the representations contain with respect to the protected attribute, we fit logistic regression classifiers on the original (un-guarded) representations, and test different values of the thresholds δ to estimate the V information.
• Evaluating I V δ ( Y adv → Z). In order to simulate the extent to which any binary labeling that is induced by a logistic regression classifier can leak inforamtion on Z on guarded data, we train a "neural network" that is the composition of two logistic regression functions of the form σ(βσ(α x) + γ).
Recall from Claim 2.4 that this very-simple neural network has a linear decision boundary. This network is trained to recover the protected attribute Z from the (approximately) guarded data. In inference time, we derive hard "pseudo-labels" from the inner sigmoid Y adv = Sign(σ(α x) and train a logistic regression to predict Z from them. To the extent it succeeds, it shows that there is binary labeling that leaks information about Z. This adversarial procedure aims to identify such labeling.
indicates that there is binary labeling from a linear logistic classifiers that leaks inforamtion on Z, it does not necessarily mean that common main-task classifiers also leak that information. To estimate the difference, we fit logistic regression classifiers to predict the profession Y prof from the guarded data, and measure the V-information between their predictions and Z.
Results. The results are presented in Fig. 3, for the 15 pairs of professions we experimented with (each curve is a separate pair), the three quantities listed above, and different values of the threshold δ on the x-axis. We observe that, unsurprisingly, the red curve, corresponding to I V (X → Z)-the mutual information as evaluated from the original representations-shows high V-information values for certain thresholds, indicating that BERT representations do encode gender distinctions. The blue curve, corresponding to I V δ ( Y adv → Z), measures the ability of adversarially-constructed binary main-task classifier to recover the gender information. It is significantly lower than the red curve, but is nonzero -indicating that RLACE does not, in fact, protect against any linear adversary. 5 Finally, the green curve, corresponding to I V δ ( Y prof → Z), measures the amount of leakage we get in practice from main-task classifiers that are trained on profession prediction. In that case, the numbers are significantly lower, showing that RLACE does provide descent guardedness in practice. 5 When we train several logistic regression classifiers on the representations after RLACE, we converge to "bad" solutions that have high loss. Yet, the linear model which is equivalent to the composition of two logistic models (Claim 2.4), shows better-than-random performance. We leave the study of this interesting observation to a followup work.

The Multiclass Case
We have shown that RLACE provides reasonable, albeit not perfect, protection against binary logistic regression adversaries, in line with the conclusions of Claim 2.6. We now turn to experiments on multiclass classification. According to § 2.3, we expect to see that guardedness is broken with a large enough number of main-task labels.
Experimental setting. Similarly to the estimation of I V δ ( Y adv → Z) in the binary case, we train a binary linear logistic classifier (predicting Z) from the representations of a linear layer with softmax activation and K hidden units. K corresponds to the number of pseudo-labels. The network is trained end-to-end to recover the protected attribute, and in inference time we apply argmax on the softmax probability vector, and fit a linear logistic Z classifier on top of the 1-hot vector. We consider different values of K: 2, 4, 8, 16, 32, 64.

Results
The results are shown in Fig. 4. For all professions, we find a softmax classifier whose predicted labels are highly predictive of the protected attribute (V-information of approximately 1 for δ = 0.999). Indeed, for most professions 4 or 8 hidden units are enough. This indicates that there are labeling schemes of the data, to 4 or 8 labels, that are induced by a linear softmax classifier, and recover almost all inforamtion on Z.

Discussion
Even if a set of representations is linearly guarded, at times one can still adversarially construct a multiclass softmax classifier that recovers the information. These results stem from the disparity between the manifold in which the concept resides, and the expressivity of the (linear) intervention we preform: softmax classifiers can access inforamtion which is inaccessible to a purely linear classifier. Thus, interventions that are aimed at achieving guardedness should consider the specific adversary against which one aims to protect.

Related Work
Techniques for information removal are generally divided into adversarial methods and post-hoc linear methods. Adversarial methods (Edwards and Storkey, 2015;Xie et al., 2017;Chen et al., 2018;Elazar and Goldberg, 2018;Zhang et al., 2018) use a gradient-reversal layer during training to induce representations that do not encode the protected attribute. However, Elazar and Goldberg (2018) have shown that these methods fail in exhaustively removing all the information associated with the protected attribute. Linear methods have been proposed as a tractable alternative, where one identifies a linear subspace that capture the concept of interest, and neutralizes it using algebraic techniques. Different methods have been proposed for the identification of the subspace, e.g. PCA (Bolukbasi et al., 2016), orthogonal-rotation (Dev et al., 2020, classification-based (Ravfogel et al., 2020) and adversarial approaches (Ravfogel et al., 2022). Different definitions have been proposed for fairness (Mehrabi et al., 2021), but they are mostly extrinsic -they concern only with the predictions of the model, and not with its representation space. Intrinsic bias measures-that do focus in the representation space of the model-have been also extensively studied. These measures quantify, for instance, the extent to which word-representation space is encodes gender distinctions (Bolukbasi et al., 2016;Caliskan et al., 2017;Kurita et al., 2019;Zhao et al., 2019). The relation between extrinsic and intrinsic bias measures is understudied, but recent works have empirically demonstrated either a relatively weak or inconsistent correlation between the two (Goldfarb-Tarrant et al., 2021;Cao et al., 2022;.
To the best of our knowledge, this work is the first to define guardedness as an intrinsic bias measure, and study its implications on extrinsic bias.

Conclusions
We have formulated the notion of guardednessthe inability to directly decode a concept from the representation. We show that in general, linear guardedness with respect to a binary protected attribute does not prevent a subsequent multiclass linear classifier trained over the guarded representations to leak information on the protected attribute. In contrast, when the main-task is binary, we can bound that leakage. These results are explained by a disparity between the manifold in which the gender concept is encoded (which is, empirically, not solely linear) and the linear intervention we perform to erase it. Thus, these results open an avenue for alternative methods that would directly target the manifold in which the concept is represented.

Limitations
Our theoretical analysis targets a specific notion of information leakage, and it is likely that it does not apply to alternative ones. While the V-informationbased approach seems natural, future work should consider alternative extrinsic bias measures as well as alternative notions of guardedness. Additionally, our focus is on the linear case, which is tractable and important -but limits the generality of our conclusions. We hope to extend this analysis to other predictive families in future work.

Ethical Considerations
The empirical experiments in this work involve the removal of binary gender information from a pretrained representation. Beyond the fact that gender a non-binary concept, this task may have real-world applications, in particular such that relate to fairness. We would thus like remind the readers to take the results with a grain of salt and be extra careful when attempting to deploy methods such as the one discussed here. Regardless of any proofs, care should be taken to measure the effectiveness of bias mitigation efforts in the context in which they are to be deployed, considering, among other things, the exact data to be used and the exact fairness metrics under consideration.
Proof. Assume in contradiction that the independence gap (defined in (10)), Consider the following cases: Case 1: P [ Y = y|Z = z] > P [ Y = y|Z = ¬z] + 2 . Then, construct the following Z-classifier: Note that provided that f θ (x) is linear, so is q θ (x). The accuracy of q θ over the entire population is: In contradiction to the linear guardedness assumption.
Case 2: P [ Y = y|Z = ¬z] > P [ Y = y|Z = z] + 2 . The proof is similar, just inverse the definition of q θ . q θ (x) = Z = ¬z, if f θ (x) = y z, otherwise The accuracy of q θ over the entire population is: In contradiction to the guardedness assumption.

A.2 Experimental Setting
In this appendix, we describe the settings of our experiments ( § 4).
Data. We use the same train-dev-test split of the biographies dataset used by Ravfogel et al. (2020), resulting in training, evaluation and test sets of sizes 255,710, 39,369, and 98,344, respectively. We reduce the dimensionality of the representations to 256 using PCA. The dataset is composed of short biographies, annotated with both gender and profession. We randomly sampled 15 pairs of professions from the dataset: (professor, attorney), (journalist, surgeon), (physician, nurse), (professor, physician), (psychologist, teacher), (attorney, teacher), (physician, journalist), (professor, dentist), (teacher, surgeon), (psychologist, surgeon), (photographer, surgeon), (attorney, psychologist), (physician, teacher), (professor, teacher), (professor, psychologist) Optimization. We run RLACE (Ravfogel et al., 2022) with a simple SGD optimization, with a learning rate of 0.005, a weight decay of 1e −5 and a momentum of 0.9, chosen by experimenting with the development set. We use a batch size of 128. The algorithm is based on adversarial game between a predictor that aims to predict gender, and an orthogonal projection matrix adversary that aims to prevent gender classification. We choose the adversary which yielded highest classification loss. All training is done on a single NVIDIA GeForce GTX 1080 Ti GPU.
Estimating V-information. After running RLACE, we get an approximately linearly-guarded representations by projecting x n ← P x n , where P is the orthogonal projection matrix returned by RLACE. We validate guardedness by training logistic regression classifiers over the projected representations; they achieve accuracy less than 2% above the majority accuracy. Then, to estimate I V δ ( Y adv → Z), we train a "neural network" with either single sigmoid hidden neurons ( § 4.1) or K = 2, 4, 8, 16, 32, 64 hidden neurons with softmax activations ( § 4.2). The networks are trained to recover binary gender for 25000 batches of size 2048. Optimization is done with Adam with the default parameters. Upon convergence, we derive hard labels from the sigmoid or softmax layer by taking the argmax, and then fit a linear logistic classifier over the "pseudo-labels" we get.
We use the loss of the logistic classifier to estimate I V δ ( Y adv → Z), according to Definition 2.2.