{'fraction_non_alphanumeric': 0.04918758142621863, 'fraction_numerical': 0.024974803962537793, 'mean_word_length': 4.400504447099429, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
KNIFE: Kernelized-Neural Differential Entropy Estimation

Mutual Information (MI) has been widely used as a loss regularizer for training neural networks. This has been particularly effective when learn disentangled or compressed representations of high dimensional data. However, differential entropy (DE), another fundamental measure of information, has not found widespread use in neural network training. Although DE offers a potentially wider range of applications than MI, off-the-shelf DE estimators are either non differentiable, computationally intractable or fail to adapt to changes in the underlying distribution. These drawbacks prevent them from being used as regularizers in neural networks training. To address shortcomings in previously proposed estimators for DE, here we introduce KNIFE, a fully parameterized, differentiable kernel-based estimator of DE. The flexibility of our approach also allows us to construct KNIFE-based estimators for conditional (on either discrete or continuous variables) DE, as well as MI. We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks. Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classification, and textual fine-tuning demonstrate the effectiveness of KNIFE-based estimation. Code can be found at https://github.com/g-pichler/knife.


Introduction
Learning tasks requires information (Principe et al., 2006) in the form of training data. Thus, information measures (Shannon, 1948) (e.g. entropy, conditional entropy and mutual information) have been a source of inspiration for the design of learning objectives in modern machine learning (ML) models (Linsker, 1989;Torkkola, 2006). Over the years, a plethora of estimators have been introduced to estimate the value of the aforementioned measures of information and they have been applied to many different problems, including information and coding theory, limiting distributions, model selection, design of experiment and optimal prior distribution, data disclosure, and relative importance of predictors (Ebrahimi et al., 2010). In these applications, traditional research focused on both developing new estimators and obtaining provable guarantees on the asymptotic behavior of these estimators (Liu et al., 2012;Verdú, 2019).
However, when used for training deep neural networks, additional requirements need to be satisfied. In particular, the estimator needs to be differentiable w.r.t. the data distribution (R1), computationally tractable (R2), and able to rapidly adapt to changes in the underlying distribution (R3). For instance, Mutual Information (MI), a fundamental measure of dependence between variables, only became a popular (standalone or regularizing) learning objective for DNNs once estimators satisfying the above requirements were proposed (Poole et al., 2019;Barber & Agakov, 2003). Although MI is notoriously difficult to estimate in high dimensions (Kraskov et al., 2004;Pichler et al., 2020;McAllester & Stratos, 2020), these estimators have demonstrated promising empirical results in unsupervised representation learning (Krause et al., 2010;Bridle et al., 1992;Hjelm et al., 2019;Tschannen et al., 2020), discrete/invariant representations (Hu et al., 2017;Ji et al., 2019), generative modelling (Chen et al., 2016;Zhao et al., 2017), textual disentangling (Cheng et al., 2020b;Colombo et al., 2021), and applications of the Information Bottleneck (IB) method (Mahabadi et al., 2021;Devlin et al., 2018;Alemi et al., 2016) among others. Compared to MI, Differential Entropy (DE) has received less attention from the ML community while offering a potentially wider range of applications than MI.
In this paper, we focus on the problem of differentiable, arXiv:2202.06618v1 [cs.LG] 14 Feb 2022 flexible and efficient DE estimation satisfying simultaneously (R1),(R2),(R3) as this quantity naturally appears in many applications (e.g. reinforcement learning (Shyam et al., 2019;Hazan et al., 2019;Ahmed et al., 2019;Kim et al., 2019), IB (Alemi et al., 2016), mode collapse (Belghazi et al., 2018)) and can be used to estimate MI. Traditional estimators of DE often violate at least one of the requirements (R1) -(R3) listed above (e.g. k-nearest neighbor based estimators violate (R1)). Consequently, the lack of appropriate DE estimators for arbitrary data distributions forces deep learning researchers to either restrict themselves to special cases where closed-form expressions for DE are available (Shyam et al., 2019) or use MI as a proxy (Belghazi et al., 2018). In this work, we introduce a Kernelized Neural dIFferential Entropy (KNIFE) estimator, that satisfies the aforementioned requirements and addresses limitations of existing DE estimators (Schraudolph, 2004;McAllester & Stratos, 2020). Stemming from recent theoretical insights (McAllester & Stratos, 2020) that justify the use of DE estimators as building blocks to better estimate MI, we further apply KNIFE to MI estimation. In the context of deep neural networks with high dimensional data (e.g. image, text), KNIFE achieves competitive empirical results in applications where DE or MI is required.
1. We showcase limitation of the existing DE estimators proposed in (Schraudolph, 2004;McAllester & Stratos, 2020) with respect to desirable properties required for training deep neural networks. To address these shortcomings, we introduce KNIFE, a fully learnable kernel-based estimator of DE. The flexibility of KNIFE allows us to construct KNIFE-based estimators for conditional DE, conditioning on either a discrete or continuous random variable.
2. We prove learnability under natural conditions on the underlying probability distribution. By requiring a fixed Lipschitz condition and bounded support we are not only able to provide an asymptotic result, but also a confidence bound in the case of a finite training set. This extends the consistency result by (Ahmad & Lin, 1976).
3. We validate on synthetic datasets (including multi-modal, non-Gaussian distributions), that KNIFE addresses the identified limitations and outperforms existing methods on both DE and MI estimation. In particular, KNIFE more rapidly adapts to changes in the underlying data distribution. 4. We conduct extensive experiments on natural datasets (including text and images) to compare KNIFE-based MI estimators to most recent MI estimators. First, we apply KNIFE in the IB principle to fine-tune a pretrained language model. Using KNIFE, we leverage a closed-form expression of a part of the training objective and achieve the best scores among competing MI estimators. Second, on fair textual classification, the KNIFE-based MI estimator achieves near perfect disentanglement (with respect to the private, discrete label) at virtually no degradation of accuracy in the main task. Lastly, in the challenging scenario of visual domain adaptation, where both variables are continuous, KNIFEbased MI estimation also achieves superior results. Code for KNIFE and to reproduce results can be found at .

Existent Methods and Related Works
DE estimation. Existing methods for estimating DE fit into one of three categories (Beirlant et al., 1997;Hlaváčková-Schindler et al., 2007;Verdú, 2019): plug-in estimates (Ahmad & Lin, 1976;Györfi & Van der Meulen, 1987), estimates based on sample-spacings (Tarasenko, 1968), and estimates based on nearest neighbor distances (Kozachenko & Leonenko, 1987;Tsybakov & Van der Meulen, 1996;Berrett et al., 2019). Our proposed estimator falls into the first category and we will thus focus here on previous work using that methodology. Excellent summaries of all the available methods can be found in the works (Beirlant et al., 1997;Hlaváčková-Schindler et al., 2007;Wang et al., 2009;Verdú, 2019). In (Ahmad & Lin, 1976), a first nonparametric estimator of DE was suggested and theoretically analyzed. It builds on the idea of kernel density estimation using Parzen-Rosenblatt windowing (Rosenblatt, 1956;Parzen, 1962). More detailed analysis followed (Joe, 1989;Hall & Morton, 1993) but the estimator remained essentially unchanged. Unfortunately, this classical literature is mostly concerned with appropriate regularity conditions that guarantee asymptotic properties of estimators, such as (asymptotic) unbiasedness and consistency. Machine learning applications, however, usually deal with a fixed-often very limited-number of samples.
Differentiable DE estimation. A first estimator that employed a differential learning rule was introduced in (Viola et al., 1996). Indeed, the estimator proposed therein is optimized using stochastic optimization, it only used a single kernel with a low number of parameters. An extension that uses a heteroscedastic kernel density estimate, i.e., using different kernels at different positions, has been proposed in (Schraudolph, 2004). Still the number of parameters was quite low and varying means in the kernels or variable weights were not considered. Although the estimation of DE remained a topic of major interest as illustrated by recent works focusing on special classes of distributions (Kolchinsky & Tracey, 2017;Chaubey & Vu, 2021) and nonparametric estimators (Sricharan et al., 2013;Kandasamy et al., 2015;Moon et al., 2021), the estimator introduced in (Schraudolph, 2004) was not further refined and hardly explored in recent works.
Differentiable MI estimation. In contrast, there has been a recent surge on new methods for the estimation of the closely related MI between two random variables. The most prominent examples include unnormalized energy-based variational lower bounds (Poole et al., 2019), the lower bounds developed in (Nguyen et al., 2010) using variational characterization of f-divergence, the MINE-estimator developed in (Belghazi et al., 2018) from the Donsker-Varadhan representation of MI which can be also interpreted as an improvement of the plug-in estimator of (Suzuki et al., 2008), the noise-contrastive based bound developed in (van den Oord et al., 2018) and finally a contrastive upper bound (Cheng et al., 2020a). (McAllester & Stratos, 2020) point out shortcomings in other estimation strategies and introduce their own Differences of Entropies (DOE) method.

KNIFE
In this section we identify limitations of existing entropy estimators introduced in (Schraudolph, 2004;McAllester & Stratos, 2020). Subsequently, we present KNIFE, which addresses these shortcomings.

Limitations of Existing Differential Entropy Estimators
Consider a continuous random vector X ∼ p in R d . Our goal is to estimate the DE h(X) := − p(x) log p(x) dx.
Given the intractability of this integral, we will rely on a Monte-Carlo estimate of h(X), using N i.i.d. samples Unfortunately, assuming access to the true density p is often unrealistic, and we will thus construct an estimatep that can then be plugged into (1) instead of p. Ifp is smooth, the resulting plug-in estimator of DE is differentiable (R1).
Assuming access to an additional-ideally independentset of M i.i.d. samples E = {x m } M m=1 , we build upon the Parzen-Rosenblatt estimator (Rosenblatt, 1956;Parzen, 1962)p where w > 0 denotes the bandwidth and κ is a kernel density. The resulting entropy estimator when replacing p in (1) by (2) was analyzed in (Ahmad & Lin, 1976). In (Schraudolph, 2004), this approach was extended using the kernel estimator where A := (A 1 , . . . , A M ) are (distinct, diagonal) covariance matrices and κ A (x) = N (x; 0, A) is a centered Gaussian density with covariance matrix A.
The DOE method of (McAllester & Stratos, 2020) is a MI estimator that separately estimates a DE and a conditional DE. For DE, a simple Gaussian density estimatê are the training parameters, the diagonal covariance matrix A and the mean µ.
While both SCHRAU. and DOE yield differentiable plug-in estimators for DE, they each have a major disadvantage. The strategy of (Schraudolph, 2004) fixes the kernel mean values at E, which implies that the method cannot adapt to a shifting input distribution (R3). On the other hand, DOE allows for rapid adaptation, but its simple structure makes it inadequate for the DE estimation of multi-modal densities. We illustrate these limitations in Section 3.1.

KNIFE Estimator
In KNIFE, the kernel density estimate is given bŷ where θ := (A, a, u) and the additional parameters 0 ≤ u = (u 1 , u 2 , . . . , u M ) with 1·u = 1 and a = (a 1 , . . . , a M ) are introduced. Note thatp KNIFE (x; θ) is a smooth function of θ, and so is our proposed plug-in estimator KNIFE combines the ideas of (Schraudolph, 2004;McAllester & Stratos, 2020). It is differentiable and able to adapt to shifting input distributions, while capable of matching multi-modal distributions. Thus, as we will see in synthetic experiments, incorporating u m and shifts a m in the optimization enables the use of KNIFE in non-stationary settings, where the distribution of X evolves over time.
Learning step: Stemming from the observation that, by the Law of Large Numbers (LLN), we propose to learn the parameters θ by minimizing h KNIFE , where E may be used to initialize a. Although not strictly equivalent due to the Monte-Carlo approximation, minimizing h KNIFE can be understood as minimizing the Kullback-Leibler (KL) divergence in (6), effectively minimizing the gap between h KNIFE and h(X). In fact, h KNIFE can also be interpreted as the standard maximum likelihood objective, widely used in modern machine learning. It is worth to mention that the KNIFE estimator is fully differentiable with respect to θ and the optimization can be tackled by any gradient-based method (e.g., Adam (Kingma & Ba, 2014) or AdamW (Loshchilov & Hutter, 2017)).

Convergence Analysis
Note that the classical Parzen-Rosenblatt estimator h(D x ; w), where (2) is plugged into (1), is a special case of KNIFE. Thus, the convergence analysis provided in (Ahmad & Lin, 1976)[Theorem 1] also applies and yields sufficient conditions for h KNIFE (D x , θ) → h(X). In Appendix D, we extend this result and, assuming that the underlying distribution p is compactly supported on X = [0, 1] d and L-Lipschitz continuous, the following theorem is proved.
Theorem 1. For any δ > 0, there exists a function ε(N, M, w) such that, with probability at least where M and N denote the number of samples in E and D x , respectively.
The precise assumptions for Theorem 1 and an explicit formula for ε(N, M, w) are given in Theorem 2 in Appendix D. For instance, Theorem 1 provides a bound on the speed of convergence for the consistency analysis in (Ahmad & Lin, 1976, Thm. 1).

Estimating Conditional Differential Entropy and Mutual Information
Similar to (McAllester & Stratos, 2020), the proposed DE estimator can be used to estimate other information measures.
In particular, we can use KNIFE to construct estimators of conditional DE and MI. When estimating the conditional DE and MI for a pair of random variables (X, Y ) ∼ p, we not only use D x = {x n } N n=1 , but also the according i.i.d. samples D y = {y n } N n=1 , where (x n , y n ) are drawn according to p.
Conditional Differential Entropy. We estimate conditional DE h(X|Y ) by considering θ to be a parameterized function Θ(y) of y. Then all relations previously estab-lished naturally generalize and p KNIFE (x|y; Θ) :=p KNIFE (x; Θ(y)), . (9) Naturally, minimization of (6) is now performed over the parameters of Θ. If Y is a continuous random variable, we use an artificial neural network Θ(y), taking y as its input. On the other hand, if Y ∈ Y is a discrete random variable, we have one parameter θ for each y ∈ Y, i.e., Mutual Information. To estimate the MI between random variables X and Y (either discrete or continuous), recall that MI can be written as I(X; Y ) = h(X) − h(X|Y ). Therefore, we use the marginal and conditional DE estimators (5) and (9) to build a KNIFE-based MI estimator 3. Experiments using Synthetic Data

Differential Entropy Estimation
In this section we apply KNIFE for DE estimation, comparing it to (3), the method introduced in (Schraudolph, 2004), subsequently labeled "SCHRAU.". It is worth to mention that we did not perform the Expectation Maximization algorithm, as suggested in (Schraudolph, 2004), but instead opted to use the same optimization technique as for KNIFE to facilitate a fair comparison.

GAUSSIAN DISTRIBUTION
As a sanity check, we test KNIFE on multivariate normal data in moderately high dimensions, comparing it to SCHRAU. and DOE, which we trained with the exact same parameters. We estimate the entropy h(X) = d 2 log 2πe of X ∼ N (0, I d ) for d = 10 and d = 64. The mean error and its empirical standard deviation are reported in Table 1 over 20 runs. KNIFE yielded the lowest bias and variance in both cases, despite DOE being perfectly adapted to matching a multivariate Gaussian distribution. Additional experimental details can be found in Appendix A.1.
In order to use a DE estimation primitive in a machine learning system, it must be able to adapt to a changing input distribution during training (R3). As already pointed out in Section 2.1, this is a severe limitation of SCHRAU., as redrawing the kernel support E can be either impractical or at the very least requires a complete re-training of the entropy estimator. Whereas in (4), the kernel support a is trainable 0.7301 ± 0.0428 9.8919 ± 0.1604 KNIFE 0.0461 ± 0.0139 2.8045 ± 0.0796 Table 1: Results on normal data with dimension d.
and it can thus adapt to a change of the input distribution. In order to showcase this ability, we utilize the approach of (Cheng et al., 2020a) and successively decrease the entropy, observing how the estimator adapts. We perform this experiment with data of dimension d = 64 and repeatedly multiply the covariance matrix of the training vectors with a factor of a = 1 2 . The resulting entropy estimation is depicted in Figure 1. It is apparent that SCHRAU. suffers from a varying bias. The bias increases with decreasing variance, as the kernel support is fixed and cannot adapt as the variance of D x shrinks. DOE is perfectly adapted to a single Gaussian distribution and performs similar to KNIFE. [t]

TRIANGLE MIXTURE
KNIFE is able to cope with distributions that have multiple modes. While (3) is also capable of matching multi-modal distributions, DOE is unable to do so, as it approximates any distribution with a multivariate Gaussian. We illustrate this by matching a mixture of randomly drawn triangle distributions. The resulting estimated PDFs as well as the ground truth when estimating the entropy of a 1-dimensional mixture of triangles with 10 components can be observed in Figure 2 (left). With increasing dimension the difficulty of this estimation rises quickly as in d dimensions, the resulting PDF of independent c-component triangle mixtures has c d modes. To showcase the performance of KNIFE in this challenging task, we ran 10 training runs for DE estimation of 2-component triangle mixtures in 8 dimensions. An example training run is depicted in Figure 2 (right).

Mutual Information Estimation
Multivariate Gauss We repeat the experiments in (Cheng et al., 2020a), stepping up the MI I(X d ; Y d ) between d i.i.d. copies of joint normal random variables (X, Y ) by increasing their correlation coefficient, i.e., (X, Y ) are multivariate Gaussian with correlation coefficient ρ i in the i-th epoch. A training run is depicted in the top of Figure 3. As in (Cheng et al., 2020a), we also repeat the experiment, applying a cubic transformation to Y . The estimation of MI between d i.i.d. copies of X and Y 3 can be observed in the middle row of Figure 3. The MI is unaffected by this bijective transformation. In Appendix A.3, the bias and variance are depicted separately.

Sum of Uniformly Distributed Variables
In order to test the ability of KNIFE to adapt to distributions substantially different from the Gaussian kernel shape, we apply it in MI estimation of I(X d ; Y d ) with uniformly distributed data. To this end, let X and E be centered, uniformly distributed random variables with E[ Figure 3 (bottom). Details about the source distribution as well as details of the experiments can be found in Appendix A.3.

Experiments on Natural Data
In this section, we benchmark our proposed KNIFE-based MI estimator on three practical applications, spanning textual and visual data. We reproduce and compare our method to the most recent MI estimators including MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010), InfoNCE (van den Oord et al., 2018), CLUB (Cheng et al., 2020a), and DOE (McAllester & Stratos, 2020). We do not explicitly include the SMILE estimator (Song & Ermon, 2019) in our comparison as it has the same gradient as NWJ.
Common notation: In all following applications, we will use Φ ψ : X → Z to denote an encoder, where X is the raw input space (i.e., texts or images), and Z denotes a lower dimensional continuous feature space. Additionally, we will use C ψ : Z → Y to denote a shallow classifier from the latent space Z to a discrete or continuous target space Y for classification or regression, respectively. We will use ψ to denote the parameters of both models, Φ ψ and C ψ . CE denotes the cross entropy loss.

Information Bottleneck for Language Model Finetuning
IB has recently been applied to fine-tune large-scale pretrained models (Mahabadi et al., 2021) such as BERT (Devlin et al., 2018) and aims at suppressing irrelevant features in order to reduce overfitting.
Problem statement. Given a textual input X ∈ X and a target label Y ∈ Y, the goal is to learn the encoder Φ ψ and classifier C ψ , such that Φ ψ (X) retains little information about X, while still producing discriminative features, allowing the prediction of Y . Thus, the loss of interest is: where λ controls the trade-off between the downstream and the compression terms.
Setup. Following (Mahabadi et al., 2021) Details on the architecture of µ ψ and Σ ψ can be found in Appendix C. The classifier C ψ is composed of dense layers. To minimize L, the second part of the objective (11)

Fair Textual Classification
In fair classification, we would like the model to take its decision without utilizing private information such as gender, age, or race. For this task, MI can be minimized to disentangle the output of the encoder Z and a private label S ∈ S (e.g., gender, age, or race).
Problem Statement. Given an input text X, a discrete target label Y and a private label S, the loss is given by where λ controls the trade-off between minimizing MI and CE loss. In this framework, a classifier is said to be fair or to achieve perfect privacy if no statistical information about S can be extracted from Φ ψ (X) by an adversarial classifier.
Overall, a good model should achieve high accuracy on the main task (i.e., prediction of Y ) while removing information about the protected attribute S. This information is measured by training an offline classifier to recover the protected attribute S from Φ ψ (X).
Setup. We compute the second term of (12) with competing MI estimators, as well as the model from (Elazar & Goldberg, 2018), which will be referred to as "Adv", as it utilizes an adversary to recover the private label from the latent representation Z. For KNIFE-based MI estimation, we use two DE estimators (as S is a binary label), following the approach outlined in Section 2.4. All derivations are detailed in Appendix C. We follow the experimental setting from (Elazar & Goldberg, 2018; Barrett et al., 2019) and use two datasets from the DIAL corpus (Blodgett et al., 2016) (over 50 million tweets) where the protected attribute S is the race and the main labels are sentiment or mention labels. The mention label indicates whether a tweet is conversational or not. We follow the official split using 160 000 tweets for training and two additional sets composed of 10 000 tweets each for development and testing. In all cases, the labels S and Y are binary and balanced, thus a random guess corresponds to 50% accuracy.
Results. Figure 4 gathers results on the fair classification task. The upper dashed lines represent the (private and main) task accuracies when training a model with only the CE loss (case λ = 0 in (12)). This shows that the learned encoding Φ ψ (X) contains information about the protected attribute, when training is only performed for the main task. On both the sentiment and mention task, we observe that a KNIFEbased estimator can achieve perfect privacy (see Figures 4b  and 4d) with nearly no accuracy loss in the main task (see Figures 4a and 4c). The other MI estimators exhibit different behavior. For sentiment labels, most MI estimators fail to reach perfect privacy (CLUB, NWJ, DOE, and Adv) while others (InfoNCE) achieve perfect privacy while degrading the main task accuracy (10% loss on main accuracy). For mention labels, CLUB can also reach perfect privacy with almost no degradation of the accuracy of the main task. Overall, it is worth noting that KNIFE-based MI estimation enables better control of the degree of disentanglement than the reported baselines.

Unsupervised Domain Adaptation
In unsupervised domain adaptation, the goal is to transfer knowledge from the source domain ( Problem Statement. The learner is given access to labeled images from a source domain (x s , y) ∼ (X S , Y ) ∈ X S × Y and unlabeled images from a target domain x t ∼ X T ∈ X T . The goal is to learn a classification model {Φ ψ , C ψ } that generalizes well to the target domain. Training models on the supervised source data only results in domain-specific latent representations Φ ψ (X) leading to poor generalization (when X is chosen randomly from {X S , X T }). In order to make the latent representations as domain-agnostic as possible, we follow the informationtheoretic method proposed by (Gholami et al., 2020), and used in (Cheng et al., 2020a). The idea is to learn an additional binary model {Φ d ν , C d ν }, whose goal it is to guess the domain D ∈ {0, 1} of X. The latent representation learned by Φ d ν will therefore contain all the domain-specific information that we would like the main encoder Φ ψ to discard. In other words, we would like Φ ψ (X) and Φ d ν (X) to be completely disentangled, which naturally corresponds to the minimization of I(Φ ψ (X); Φ d ν (X)). Concretely, the domain classifier is trained to minimize the CE between domain labels D and its own predictions, whereas the main classifier is trained to properly classify support samples while minimizing the MI between Φ ψ (X) and Φ d ν (X). Us- Setup.
The different MI estimators are compared based on their ability to guide training by estimating I(Φ ψ (X); Φ d ν (X)) in (13). We follow the setup of (Cheng et al., 2020a)  Results. Results are presented in Table 3. The KNIFEbased estimator is able to outperform MI estimators in this challenging scenario where both Φ ψ (X) and Φ d ν (X) are continuous.

Concluding Remarks
We introduced KNIFE, a fully learnable, differentiable kernel-based estimator of differential entropy, designed for deep learning applications. We constructed a mutual information estimator based on KNIFE and showcased several applications. KNIFE is a general purpose estimator and does not require any special properties of the learning problem. It can thus be incorporated as part of any training objective, where differential entropy or mutual information estimation is desired. In the case of mutual information, one random variable may even be discrete.
Despite the fundamental challenges in the problem of differential entropy estimation, beyond limitations arising from the use of a finite number of samples, KNIFE has demonstrated promising empirical results in various representation learning tasks.
Future work will focus on improving the confidence bounds given in Theorem 1. In particular, tailoring them towards KNIFE using tools from (Birge & Massart, 1995;Singh & Poczos, 2014). Another potential extension is direct estimation of the gradient of entropy, whenp KNIFE (x; θ) has been learned (Mohamed et al., 2020;Song et al., 2020). This could be applied after the learning phase of KNIFE and is left for future work.

Acknowledgements
This work was granted access to the HPC resources of IDRIS under the allocation 2021-101838 made by GENCI.  (Figures 4a and 4c) and private task (Figures 4b and 4d) for both mention and sentiment labels. Results of MINE are not reported because of instabilities that prevent the network from converging. Figures 4b and 4d are obtained by training an offline classifier to recover the protected attribute S from Φ ψ (X).   Zhao, S., Song, J., and Ermon, S. InfoVAE: Information maximizing variational autoencoders. arXiv, abs/1706.02262, 2017.

A. Experimental Details of Experiments with Synthetic Data
Implementation of KNIFE in PyTorch (Paszke et al., 2019) is rather straightforward. The constraint on the weights u can be satisfied by applying a softmax transformation. The covariance matrices were parameterized by the lower-triangular factor in the Cholesky decomposition of the precision matrices, guaranteeing the definiteness constraint to be satisfied.

A.1. Differential Entropy Estimation of Gaussian Data
In Section 3.1.1, the estimation of the entropy h(X) = d 2 log 2πe for X ∼ N (0, I d ) was performed with the hyperparameters given in Table 4. The mean error and its empirical standard deviation are reported in Table 1 over 20 runs, where an independently drawn evaluation set with the same size as the training set is used. At d = 10 we have the entropy h = d 2 log 2πe = 14.19, while for the higher dimension, d = 64 we find h = 90.81. In the experiment depicted in Figure 1, entropy is decreased after every epoch by letting where G is a standard normal random variable, resulting in an decrease of the DE by ∆ = − d 2 log a ≈ 22.18 for a = 1 2 with every epoch. We start at h(X 0 ) = d 2 log 2πe ≈ 90.81 and successively decrease until h(X 4 ) = h(X 0 ) + 4∆ ≈ 2.1. Additional parameters can be found in Table 5.
Computational Resources. Training was performed on an NVidia V100 GPU. Taken together, training for the first experiments of entropy estimation in dimensions d = 10, 64, as well as the experiment depicted in Figure 1 used GPU time of less than 5 minutes.

A.2. Differential Entropy Estimation of Triangle Mixtures
In Section 3.1.2, we perform an estimation of the entropy of c-component triangle mixture distributions. The PDF of such a c-component triangle-mixture, is given by where Λ s (x) := 1 s max{0, 2 − 4s|x|} is a centered triangle PDF with width s > 0. The scales s = (s 1 , . . . , s c ) and weights w = (w 1 , . . . , w c ) satisfy 0 < s i , w i < 1 and c i=1 w i = 1. Before the experiment, we choose w uniformly at random from the c-probability simplex and the scales are chosen uniformly at random in [0.1, 1.0]. An example for c = 10 is the true PDF depicted in Figure 2 (left). For d > 1, we perform the estimation on d i.i.d. copies. Note that the triangle mixture with c components in d-dimensional space has c d modes, i.e., the support can be partitioned into c d disjoint components.    The parameters of the experiment yielding Figure 2 (left) are given in Table 6, while the details of the experiment depicted in Figure 2 (right) can be found in Table 7. In the latter experiment, over ten runs, entropy was estimated to an accuracy of 1.6563 ± 0.8528 by KNIFE, accurate to 2.4445 ± 0.5439 using (3) and with an accuracy of 7.1070 ± 2.7984 by DOE. This is the mean absolute error and its empirical standard deviation over all 10 runs, where the evaluation set was drawn independently from the training set and has the same size as the training set.
Computational Resources. Training was performed on an NVidia V100 GPU. Training in d = 1 dimension, that resulted in Figure 2 (left) can be performed in seconds, while all training required for producing Figure 2 (right) used approximately 1.5 hours of GPU time.

A.3. Mutual Information Estimation
In Section 3.2, we estimate I(X d ; Y d ) and ] are independent and Y is given by Y = ρ i X + 1 − ρ 2 i E. In both cases, ρ i is chosen such that I(X d ; Y d ) = 2i in the i-th epoch.
All neural networks are randomly initialized. The bias, variance, and MSE during training as a function of the MI, can be observed in Figure 5.
The estimation is performed in 10 runs, randomly choosing the training meta-parameters as proposed by (McAllester & Stratos, 2020). In Figure 3 (bottom), we present the best run for each method, selected by distance from the true MI at the end of training. The bias, variance, and MSE during training, as a function of the MI, can be observed in Figure 6. Details about the source distribution as well as details of the experiments can be found in Table 8. During experimentation it turned out to be beneficial to train the parameters Θ and θ in (10) separately and substantially increase the learning rate for the training of θ. Thus, we increase the learning rate for the training of θ by a factor of 10 3 .
Model Architecture for Θ. We utilize the feed-forward architecture, also used in (McAllester & Stratos, 2020). It is a simple architecture with two linear layers, one hidden layer using tanh activation, immediately followed by an output layer. The number of neurons in the hidden layer is a meta-parameter selected randomly from {64, 128, 256} for each training run. Three models with this architecture are used for the three parameters (A, a, u), as described by (4), where only the output dimension is changed to fit the parameter dimension.
Computational Resources. Training was performed, using about 6 hours of GPU time on an NVidia V100 GPU to carry out the experiment depicted in Figure 3 (bottom).   (3). The mean error and standard deviation is computed using and independently drawn evaluation set that has the same size as the training set. The best training run is depicted in Figure 7 and the details can be found in Table 9.  In Section 4, we rely on two different types of models: pretrained (e.g., fine tuning with VIBERT) and randomly initialized (e.g., in fair classification and domain adaptation). When working with randomly initialized networks the parameters are updated. However, it is worth noting that in the literature the pretrained model parameters (i.e. ψ) are not always updated (see (Ravfogel et al., 2020)). In our experiments: (i) We always update the parameters (even for pretrained models), and (ii) we did not change the way the parameters were updated in concurrent works (to ensure fair comparison). Specifically, • for language model finetuning (Appendix C.2), we followed (Mahabadi et al., 2021) and did a joint update; • for the fair classification task (Appendix C.3), we followed common practice and used the algorithm described in Algorithm 1 which rely on an alternated update; • for the domain adaptation task (Appendix C.4), we followed common practice and used a joint method.

C.2. Information Bottleneck for Language Model Finetuning
For this experiment we follow the experimental setting introduced in (Mahabadi et al., 2021) and work with the GLUE data 2 .
Model Architecture. We report in Table 10, the multilayer perceptron (MLP) used to compute the compressed sentence representations produced by BERT. Variance and Mean MLP networks are composed of fully connected layers.
Model Training. For model training, all models are trained for 6 epochs and we use early stopping (best model is selected on validation set error). For IB, λ is selected in {10 −4 , 10 −5 , 10 −6 } and K is selected in {144, 192, 288, 384}. We follow (Alemi et al., 2016) where the posterior is averaged over 5 samples and a linear annealing schedule is used for λ. Additional hyper-parameters are reported in Table 11. Dataset Statistics. Table 12 reports the statistics of the dataset used in our finetuning experiment.
Computational Resources. For all these experiments we rely on NVidia-P100 with 16GB of RAM. To complete the full grid-search on 10 seeds and on the three datasets, approximately 1.5k hours are required.

C.3. Fair Textual Classification
In this section, we gather the experimental details for the textual fair classification task.

C.3.1. DETAILS OF THE KNIFE-BASED ESTIMATOR
In this experiment, we estimate the MI between a continuous random variable, namely Z = Φ ψ (X), and a discrete variable, denoted by S ∈ S = {1, 2, . . . , |S|}. We follow the strategy outlined in Section 2.4 for estimating the conditional DE h(Z|S). However, we will reuse the estimate of the conditional PDFp(z|s; Θ) to compute an estimate of the DE as wherep(s) = 1 N |{n : s n = s}| is used to indicate the empirical distribution of S in the training set D s . 3 In our experiments, with |S| = 2, we found that estimating the DE h(Z) based on the KNIFE estimator learnt for h(Z|S) increases the stability of training. We adopted the same strategy for DOE.

C.3.2. EXPERIMENTAL DETAILS
Model Architecture. For the encoder, we use a bidirectionnal GRU with two layers with hidden and input dimension set to 128. We use LeakyReLU as the activation function. The classification head is composed of fully connected layers of input dimension 256. We use a learning rate of 0.0001 for AdamW. The dropout rate is set to 0.2. The number of warmup steps is set to 1000.
Computational Resources. For all these experiments, we rely on NVIDIA-P100 with 16GB of RAM. Each model is trained for 30k steps. The model with the lowest MI is selected. The training of a single network takes around 3 hours.

C.4. Unsupervised Domain Adaptation
We follow the experimental setup given in (Cheng et al., 2020a) as closely as possible, i.e., we pick hyperparameters given in the paper, or if not provided, those set in the code: 4 Model Training. We use Adam optimizer for all modules with a learning rate of 0.001. Batch size is set to 128. We set the weighting parameter λ = 0.1. The original code of (Cheng et al., 2020a) uses 15 000 training iterations, but we found most methods had not properly converged at this stage, and hence use 25 000 iterations instead. Similar to other experiments, we set the kernel size M = 128.
Model Architecture. Table 13 summarizes the architectures used for the different modules. For the MI network of each method, the best configuration, based on the validation set of the first task MNIST → MNIST-M, is chosen among 4 configurations: with or without LayerNorm and with ReLU or tanh activation.
Computational Resources. For these experiments, we used a cluster of NVIDIA-V100 with 16GB of RAM. Each training (i.e., 25k iterations) on a single task requires on average 2 hours. Given that we have 6 tasks, and repeat the training for 3 different seeds, on average 36 hours computation time is required for each method.

D. Bounding the Error
In the following, fix L > 0 and let P L be the set of L-Lipschitz PDFs supported 5 on X := [0, 1] d , i.e., X p(x) dx = 1, and ∀x, y ∈ R d : |p(x) − p(y)| ≤ L x − y for p ∈ P L , where 6 x := k |x k |.
Assume p ∈ P L and let κ be a PDF supported on X . In order to show that estimation of h(X) is achievable, we use a standard Parzen-Rosenblatt estimatorp(x; w) := 1 M w d M m=1 κ x−X m w , as in (2). The entropy estimate is then defined by the empirical average h(D x ; w) := − 1 N N n=1 logp(X n ; w).
Note that it is easily seen that p max ≤ L 2 and C 1 ≤ max p max log 2 p max , 4e −2 by our assumptions. The requirement C 2 , K max < ∞ represents a mild condition on the kernel function κ.
We can now show the following. Theorem 2. With probability greater than 1 − δ we have if the expression in the logarithm is positive.
In particular, the estimation error approaches zero as N → ∞ if w = w(N ) → 0, M = M (N ) → ∞ are chosen such that N w → 0, We prove Theorem 2 in several Lemmas.
Case x < y. Here, we can write y = a + b and x = y − c = a + b − c for b ≥ 0 and 0 ≤ c ≤ δ < a.
Proof of Theorem 2. We apply Lemma 3 N times and use the union bound to show that with probability greater than 1 − δ 3 we have for every n ∈ [N ] Similarly, by Lemma 4, we have with probability greater than 1 − δ 3 that for all n ∈ [N ].
Again by the union bound, we have that with probability greater than 1 − 2δ 3 both (49) and (50) hold for all n ∈ [N ], and thus, by Lemma 5, we obtain and apply Chebychev's inequality, showing that with probability greater than 1 − δ 3 , The union bound and the triangle inequality applied to (54) and (59) yields the desired result.

E. Libraries Used
For our experiments, we built upon code from the following sources.