{'fraction_non_alphanumeric': 0.04850292785327588, 'fraction_numerical': 0.016130814274665783, 'mean_word_length': 4.539290085679315, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rethinking Positive Sampling for Contrastive Learning with Kernel

Data augmentation is a crucial component in unsupervised contrastive learning (CL). It determines how positive samples are defined and, ultimately, the quality of the representation. While efficient augmentations have been found for standard vision datasets, such as ImageNet, it is still an open problem in other applications, such as medical imaging, or in datasets with easy-to-learn but irrelevant imaging features. In this work, we propose a new way to define positive samples using kernel theory along with a novel loss called decoupled uniformity. We propose to integrate prior information, learnt from generative models or given as auxiliary attributes, into contrastive learning, to make it less dependent on data augmentation. We draw a connection between contrastive learning and the conditional mean embedding theory to derive tight bounds on the downstream classification loss. In an unsupervised setting, we empirically demonstrate that CL benefits from generative models, such as VAE and GAN, to less rely on data augmentations. We validate our framework on vision datasets including CIFAR10, CIFAR100, STL10 and ImageNet100 and a brain MRI dataset. In the weakly supervised setting, we demonstrate that our formulation provides state-of-the-art results.


Introduction
Contrastive Learning (CL)[37, 2, 3, 5, 8] is a paradigm designed for representation learning which has been applied to unsupervised [8,11], weakly supervised[46, 17] and supervised problems [31]. It gained popularity during the last years by achieving impressive results in the unsupervised setting on standard vision datasets (e.g. ImageNet) where it almost matched the performance of its supervised counterpart [8,25]. The objective in CL is to increase the similarity in the representation space between positive samples (semantically close) while decreasing the similarity between semantically distinct negative samples. Despite its simple formulation, it requires the definition of a similarity function (that can be seen as an energy term [35]), positives and negatives. Similarity functions are defined on latent representations of an encoder f ∈ F (CNN [9] or Transformer [7] for vision datasets) and and is usually expressed as a Euclidean scalar product (e.g. InfoNCE [37]). In supervised learning [31], positives are simply images belonging to the same class while negatives are images belonging to different classes. In unsupervised learning [8], since labels are unknown, positives are usually defined as transformed versions (views) of the same original image (a.k.a. the Figure 1: Illustration of the proposed method. Each point is an original imagex. Two points are connected if they can be transformed into the same augmented image using a distribution of augmentations A. Colors represent semantic (unknown) classes and light disks represent the support of augmentations for each samplex, A(·|x). From an incomplete augmentation graph (1) where intraclass samples are not connected (e.g. augmentations are insufficient or not adapted), we reconnect them using a kernel defined on prior information (either learnt with generative model or given as auxiliary attributes). The extended augmentation graph (3) is the union between the (incomplete) augmentation graph (1) and the kernel graph (2). In (2), the gray disk indicates the set of pointsx that are close to the anchor (blue star) in the kernel space.
anchor) and negatives are the transformed versions of all other images. As a result, the augmentation distribution A used to sample both positives and negatives is crucial [8] and it conditions the quality of the learnt representation. The most-used augmentations for visual representations involve aggressive crop and color distortion. Cropping induces representations with high occlusion invariance [39] while color distortion may avoids the encoder f to take a shortcut [8] while aligning positive sample representations and fall into the simplicity bias [43]. Nevertheless, learning a representation that mainly relies on augmentations comes at a cost: both crop and color distortion induce strong biases in the final representation [39]. Specifically, dominant objects inside images can prevent the model from learning features of smaller objects [10] (which is not apparent in object-centric datasets such as ImageNet) and few, irrelevant and easy-to-learn features, that are shared among views, are sufficient to collapse the representation [10] (a.k.a feature suppression). Finding the right augmentations in other visual domains, such as medical imaging, remains an open challenge [17] since the anatomical differences between two classes (e.g., pathological and healthy) can be quite subtle and difficult to capture with usual transformations. If the augmentations are too weak or inadequate to remove irrelevant signal w.r.t. a discrimination task, then how can we define positive samples? In our work, we propose to integrate prior information, learnt from generative models or given as auxiliary attributes, into contrastive learning, to make it less dependent on data augmentation. Using the theoretical understanding of CL through the augmentation graph, we make the connection with kernel theory and introduce a novel loss with theoretical guarantees on downstream performance. Prior information is integrated into the proposed contrastive loss using a kernel. In the unsupervised setting, we leverage pre-trained generative models, such as GAN [21] and VAE [32], to learn a prior representation of the data. We provide a solution to the feature suppression issue in CL [10] and also demonstrate SOTA results with weaker augmentations on visual benchmarks. In the weakly supervised setting, we use instead auxiliary/prior information, such as image attributes (e.g. birds color or size) and we show better performance than previous conditional formulations based on these attributes [46]. In summary, we make the following contributions: 1. We propose a new framework for contrastive learning allowing the integration of prior information, learnt from generative models or given as auxiliary attributes, into the positive sampling 2. We derive theoretical bounds on the downstream classification risk that rely on weaker assumptions for data augmentations than previous works on CL 3. We empirically show that our framework can benefit from the latest advances of generative models to learn a better representation while relying on less augmentations 4. We show that we achieve SOTA results in the unsupervised and weakly supervised setting

Related Works
In a weakly supervised setting, recent studies [17,46] have shown that positive samples can be defined conditionally to an auxiliary attribute in order to improve the final representation, in particular for medical imaging [17]. From an information bottleneck perspective, these approaches essentially compress the representation to be only predictive of the auxiliary attributes. This might harm the performance of the model when these attributes are too noisy to accurately approximate the true semantic labels for a given downstream task. Furthermore, the benefit of adding data augmentations when conditioning with respect to these attributes is not theoretically discussed.
In an unsupervised setting, recent approaches [19,54,55,36] used the encoder f , learnt during optimization, to extend the positive sampling procedure to other views of different instances (i.e. distinct from the anchor) that are close to the anchor in the latent space. In order to avoid representation collapse, a support set [19], a momentum encoder [36] or another small network [54] can be used to select the positive samples. In clustering approaches [36, 6], distinct instances with close semantics are attracted in the latent space using prototypes. These prototypes can be estimated through Kmeans [36] or Sinkhorn-Knopp algorithm [6]. All these methods rely on the past representation of a network to improve the current one. They require strong augmentations and they essentially assume that the closest points in the representation space belong to the same latent class in order to to better select the positives. This inductive bias is still poorly understood theoretically [42] and may depend on the visual domain. Our work also relates to generative models for learning representations. VAE [32] learns the data distribution by mapping each input to a Gaussian distribution that we can easily sample to reconstruct the original image. GAN [21] instead sample directly from a Gaussian distribution to generate images that are classified by a discriminator in a min-max game. The discriminator representation can then be used [41] as feature extractor. Other models (ALI [18], BiGAN [15] and BigBiGAN[16]) learn simultaneously a generator and an encoder that can be used directly for representation learning. All these models do not require particular augmentations to model the data distribution but they perform generally poorer than recent discriminative approaches [9] for representation learning. A first connection between generative models and contrastive learning has emerged very recently [30].
In [30], authors study the feasibility of learning effective visual representations using only generated samples, and not real ones, with a contrastive loss. Their empirical analysis is complementary to our work. Here, we leverage the representation capacity of the generative models, rather than their generative power, to learn prior representation of the data.
3 Constrastive Learning with Decoupled Uniformity

Problem setup
The general problem in contrastive learning is to learn a data representation using an encoder f ∈ F : X → S d−1 that is pre-trained with a set of n original samples (x i ) i∈[1..n] ∈X , sampled from the data distribution p(x) 1 These samples are transformed to generate positive samples (i.e., semantically similar tox) in X , space of augmented images, using a distribution of augmentations A(·|x). Concretely, for eachx i , we can sample views ofx i using x ∼ A(·|x i ) (e.g., by applying color jittering, flip or crop with a given probability). For consistency, we assume A(x) = p(x) so that the distributions A(·|x) and p(x) induce a marginal distribution p(x) over X . Given an anchorx i , all views x ∼ A(·|x j ) from different samplesx j =i are considered as negatives. Once pre-trained, the encoder f is fixed and its representation f (X ) is evaluated through linear evaluation on a classification task using a labeled dataset .K], with K the number of classes. Linear evaluation. To evaluate the representation of f on a classification task, we train a linear classifier g(x) = W f (x) (f is fixed) that minimizes the multi-class classification error.

Objective
The popular InfoNCE loss [38,37], often used in CL, imposes 1) alignment between positives and 2) uniformity between the views (x i.i.d. ∼ A(·|x)) of all instancesx [48]-two properties that correlate well with downstream performance. However, by imposing uniformity between all views, we essentially try to both attract (alignment) and repel (uniformity) positive samples and therefore we cannot achieve a perfect alignment and uniformity, as noted in [48]. Moreover, InfoNCE has been originally designed for only two views (i.e., one couple of positive) and its extension to multiple views is not straightforward. Previous works have proposed a solution to either the first [45] or second [51] issue. Here, we propose a modified version of the uniformity loss, presented in [48], that solves both issues since it: i) decouples positives from negatives, similarly to [51] and ii) is generalizable to multi-views as in [45]. We introduce the Decoupled Uniformity loss as: is called a centroid of the views ofx. This loss essentially repels distinct centroids µx through an average pairwise Gaussian potential. Interestingly, it implicitly optimizes alignment between positives through the maximization of ||µx|| 2 , so we do not need to explicitly add an alignment term. It can be shown (see Appendix), that minimizing this loss brings to a representation space where the sum of similarities between views of the same sample is greater than the sum of similarities between views of different samples. From a physics point-of-view, we are trying to find the equilibrium state of |X | particles linked with a pair Gaussian potential energy. We will study its main properties hereafter and we will see that prior information can be added during the estimation step of these centroids.

Geometrical Analysis of Decoupled Uniformity
.n] such that n ≤ d + 1, any optimal encoder f * minimizingL d unif achieves a representation s.t.: 1. (Perfect uniformity) All centroids (µx i ) i∈[1..n] make a regular simplex on the hyper-sphere Proof in Appendix. Theorem 1 gives a complete geometrical characterization when the batch size n set during training is not too large compared to the representation space dimension d. By removing the coupling between positives and negatives, we see that Decoupled Uniformity can realize both perfect alignment and uniformity, contrary to InfoNCE. Remark. The assumption n ≤ d + 1 is crucial to have the existence of a regular simplex on the hypersphere S d−1 . In practice, this condition is not always full-filled (e.g SimCLR [8] with d = 128 and n = 4096). Characterizing the optimal solution of L d unif for any n > d + 1 is still an open problem [4] but theoretical guarantees can be obtained in the limit case n → ∞ (see below). Theorem 2. (Asymptotical Optimality) When the number of samples is infinite n → ∞, then for any perfectly aligned encoder f ∈ F that minimizes L d unif , the centroids µx forx ∼ p(x) are uniformly distributed on the hypersphere S d−1 . Proof in Appendix.
Empirically, we observe that minimizers f ofL d unif remain well-aligned when n > d + 1 on realworld vision datasets (see Appendix). Decoupled uniformity thus optimizes two properties that are nicely correlated with downstream classification performance [48]-that is alignment and uniformity between centroids. However, as noted in [49,42], optimizing these two properties is necessary but not sufficient to guarantee a good classification accuracy. In fact, the accuracy can be arbitrary bad even for perfectly aligned and uniform encoders [42]. Most recent theories about CL [49, 24] make the hypothesis that samples from the same semantic class have overlapping augmented views to provide guarantees on the downstream task when optimizing InfoNCE [8] or Spectral Contrastive loss [24]. This assumption, known as intra-class connectivity hypothesis, is very strong and only relies on the augmentation distribution A. In particular, augmentations should not be "too weak", so that all intra-class samples are connected among them, and at the same time not "too strong", to prevent connections between inter-class samples and thus preserve the semantic information. Here, we prove that we can relax this hypothesis if we can provide a kernel (viewed as a similarity function between original samplesx) that is "good enough" to relate intra-class samples not connected by the augmentations (see Fig. 1). In practice, we show that generative models can define such kernel. We first recall the definition of the augmentation graph [49], and intra-class connectivity hypothesis before presenting our main theorems. For simplicity, we assume that the set of imagesX is finite (similarly to [49,24]). Our bounds and theoretical guarantees will never depend on the cardinality |X |.
3.4 Intra-class connectivity hypothesis Definition 3.2. (Augmentation graph [24,49]) Given a set of original imagesX , we define the augmentation graph G A (V, E) for an augmentation distribution A through 1) a set of vertices V =X and 2) a set of edges E such that (x,x ) = e ∈ E if the two original imagesx,x can be transformed into the same augmented image through A, i.e supp A(·|x) ∩ supp A(·|x ) = ∅. Previous analysis in CL make the hypothesis that it exists an optimal (accessible) augmentation module A * that fulfills: Assumption 1. (Intra-class connectivity [49]) For a given downstream classification task D = X × Y ∀y ∈ Y, the augmentation subgraph, G y ⊂ G A * containing images only from class y, is connected. Under this hypothesis, Decoupled Uniformity loss can also tightly bound the downstream supervised risk for a bigger class of encoders than prior work [49]. To show it, we define a measure of the risk on a downstream task D. While previous analysis [49, 1] generally used the mean cross-entropy loss (as it has closer analytic form with InfoNCE), we use a supervised loss closer to decoupled uniformity with the same guarantees as the mean cross-entropy loss (see Appendix). Notably, the geometry of the representation space at optimum is the same as cross-entropy and SupCon [31] and we can theoretically achieve perfect linear classification. Definition 3.3. (Downstream supervised loss) For a given downstream task D =X × Y, we define the classification loss as: L sup (f ) = log E y,y ∼p(y)p(y ) e −||µy−µ y || 2 , where µ y = E p(x|y) µx.
Remark. This loss depends on centroids µx rather than f (x). Empirically, it has been shown [20] that performing feature averaging gives better performance on the downstream task.

∼ A(·|x)
Theorem 3. (Guarantees with A * ) Given an optimal augmentation module A * , for any -weak aligned encoder f ∈ F we obtain: where D is the maximum diameter of all intra-class graphs G y (y ∈ Y). Proof in the Appendix. In practice, the diameter D can be controlled by a small constant in some cases [49] (typically ≤ 4) but it remains specific to the dataset at hand. Furthermore, we observe (see Appendix) that f realizes alignment with small error during optimization of L d unif (f ) for augmentations close to the sweet spot A * [45] on CIFAR-10 and CIFAR-100. In the next section, we study the case when A * is not accessible or very hard to find.

Reconnect the disconnected: extending the augmentation graph with kernel
Having access to optimal augmentations is a strong assumption and, for many real-world applications (e.g medical imaging [17]), it may not be accessible. If we have only weak augmentations (e.g., supp A(·|x) supp A * (·|x) for anyx), then some intra-class points might not be connected and we would need to reconnect them to ensure good downstream accuracy (see Theorem 7 in Appendix).
Augmentations are intuitive and they have been hand-crafted for decades by using human perception (e.g., a rotated chair remains a chair and a gray-scale dog is still a dog). However, we may know other prior information about objects that are difficult to transfer through invariance to augmentations (e.g., chairs should have 4 legs). This prior information can be either given as image attributes (e.g., age or sex of a person, color of a bird, etc.) or, in an unsupervised setting, directly learnt through a generative model (e.g., GAN or VAE). Now, we ask: how can we integrate this information inside a contrastive framework to reconnect intra-class images that are actually disconnected in G A ? We rely on conditional mean embedding theory and use a kernel defined on the prior representation/information. This allows us to estimate a better configuration of the centroids in the representation space, with respect to the downstream task, and, ultimately, provide theoretical guarantees on the classification risk.
Example. If we work with large natural images, assuming that we know a prior z(x) about our images (e.g., given by a generative model), we can compute KX using z through KX (x,x ) =K(z(x), z(x )) whereK is a standard kernel (e.g., Gaussian or Cosine).
To link kernel theory with the previous augmentation graph, we need to define a kernel graph that connects images with high similarity in the kernel space.
For kernels with constant norm (e.g., the standard Gaussian, Cosine or Laplacian kernel), it is in fact an equivalence. Intuitively, it means that we connect two original points in the kernel graph if they have small distance in the kernel space. We give now our main assumption to derive a better estimator of the centroid µx in the insufficient augmentation regime. Assumption 2. (Extended intra-class connectivity) For a given task D =X × Y, the extended

union between augmentation graph and -kernel graph) is class-connected for all y ∈ Y.
This assumption is notably weaker than Assumption 1 w.r.t augmentation distribution A. Here, we do not need to find the optimal distribution A * as long as we have a kernel KX such that disconnected points in the augmentation graph are connected in the -kernel graph. If K is not well adapted to the data-set (i.e it gives very low values for intra-class points), then needs to be large to re-connect these points and we will see that the classification error will be high. In practice, this means that we need to tune the hyper-parameter of the kernel (i.e., σ for a RBF kernel) so that all intra-class points are reconnected with a small .

Conditional Mean Embedding
Decoupled Uniformity loss includes no kernel in its raw form. It only depends on centroids µx = E A(x|x) f (x). Here, we show that another consistent estimator of these centroids can be defined, using the previous kernel KX . To show it, we fix an encoder f ∈ F and require the following technical assumption in order to apply conditional mean embedding theory [44,33].
iid ∼ A(x,x). Assuming 3, a consistent estimator of the centroid is: Intuition. This theorem says that we can use representations of images close to an anchorx, according to our prior information, to accurately estimate µx. Consequently, if the prior is "good enough" to connect intra-class images disconnected in the augmentation graph (i.e. fulfills Assumption 2), then this estimator allows us to tightly control the classification risk . From this theorem, we naturally derive the empirical Kernel Decoupled Uniformity loss using the previous estimator.
.n] . We define the empirical kernel decoupled uniformity loss as: The computational cost added is roughly O(n 3 ) (to compute the inverse matrix of size n × n) but it remains negligible compared to the back-propagation time using classical stochastic gradient descent. Importantly, the gradients associated to α i,j are not computed.

Contrastive Learning with Kernel
We show here thatL d unif (f ) can tightly bound the supervised classification risk for well-aligned encoders f ∈ F.
Theorem 5. We assume 2 and 3 hold for a reproducible kernel KX and augmentation distribution A.
. For any -weak aligned encoder f ∈ F: .n] and D is the maximal diameter of all sub-graphsG y ⊂G where y ∈ Y. We noted λ min (K n ) > 0 the minimal eigenvalue of K n .
Interpretation. Theorem 5 gives a tight bound on the classification loss L sup (f ) with few assumptions. In the special case = 0 and A = A * (i.e the augmentation graph is class-connected, a stronger assumption than 2), we retrieve the standard bounds of Theorem 3. As before, we don't require perfect alignment for f ∈ F and we don't have class collision term (even if the extended augmentation graph may contain edges between inter-class samples), contrarily to [1]. Also, the estimation error doesn't depend on the number of views (which is low in practice))-as it was always the case in previous formulations [49, 1, 24] -but rather on the batch size n. Contrarily to CCLK [46], we don't condition our representation to weak attributes but rather we provide better estimation of the conditional mean embedding conditionally to the original image. Our loss remains in an unconditional contrastive framework driven by the augmentations A and the prior KX on input images.

Experiments
Here, we study several problems where current contrastive frameworks fail at learning robust visual representations. In unsupervised learning, we show that we can leverage generative models to outperform current self-supervised models when the augmentations are insufficient to remove irrelevant signal from images. In weakly supervised, we demonstrate the superiority of our unconditional formulation when noisy auxiliary attributes are available. Implementation details in Appendix.

Generative models as prior -Evading feature suppression
Previous investigations [10] have shown that a few easy-to-learn irrelevant features not removed by augmentations can prevent the model from learning all semantic features inside images. We propose here a first solution to this issue. RandBits dataset [10]. We build a RandBits dataset based on CIFAR-10. For each image, we add a random integer sampled in [0, 2 k − 1] where k is a controllable number of bits. To make it easy to learn, we take its binary representation and repeat it to define k channels that are added to the original RGB channels. Importantly, these channels will not be altered by augmentations, so they will be shared across views. We train a ResNet18 on this dataset with standard SimCLR augmentations [8] and varying k. For kernel decoupled uniformity, we use a β-VAE representation (ResNet18 backbone, β = 1) to define K V AE (x,x ) = K(µ(x), µ(x )) where µ(·) is the mean Gaussian distribution ofx in the VAE latent space and K is a standard RBF kernel. Table 1 shows the linear evaluation accuracy computed on a fixed encoder trained with various contrastive (SimCLR, Decoupled Uniformity and Kernel Decoupled Uniformity) and non-contrastive (BYOL and β-VAE) methods. As noted previously [10], β-VAE is the only method insensitive to the number of added bits, but its representation quality remains low compared to other discriminative approaches. All contrastive approaches fail for k ≥ 10 bits. This can be explained by noticing that, as the number of bits k increases, the number of edges between intra-class images in the augmentation graph G A decreases. For k bits, on average N/2 k images share the same random bits (N = 50000 is the dataset size). So only these images can be connected in G A . For k = 20 bits, < 1 image share the same bits which means that they are almost all disconnected, and it explains why standard contrastive approaches fail. Same trend is observed for non-contrastive approaches (e.g. BYOL) with a degradation in performance even faster than SimCLR. Interestingly, encouraging a disentangled representation by imposing higher β > 1 in β-VAE does not help. Only our K V AE Decoupled Uniformity loss obtains good scores, regardless of the number of bits. Figure 2: Empirical verification of our theory. The optimal * to add 100 edges between intra-class images is correlated with the downstream accuracy, as suggested by Theorem 5. We use k = 20 bits and an RBF kernel.  Table 1: Linear evaluation accuracy (in %) after training on RandBits-CIFAR10 with ResNet18 for 200 epochs. For VAE, we also use a ResNet18 backbone. Once trained, we use its representation to define the kernel K V AE in kernel decoupled uniformity loss.

Towards weaker augmentations
Color distortion (including color jittering and gray-scale) and crop are the two most important augmentations for SimCLR and other contrastive models to ensure a good representation on ImageNet [8]. Whether they are best suited for other datasets (e.g medical imaging [17] or multi-objects images [10]) is still an open question. Here, we ask: can generative models remove the need for such strong augmentations ? We use standard benchmarking datasets (CIFAR-10, CIFAR-100 and STL-10) and we study the case where augmentations are too weak to connect all intra-class points. We use a trained VAE to define K V AE as before and a trained DCGAN where z(·) denotes the discriminator output of the penultimate layer.  Table 2: When augmentation overlap hypothesis is not full-filled, generative models can provide a good kernel to connect intra-class points not connected by augmentations. * For VAE and DCGAN, we did not use augmentations during training since they model the true data distribution.
In Table 2, we observe that our contrastive framework with DCGAN representation as prior is able to match the performance of SimCLR on CIFAR100 within 200 epochs by applying only crop augmentations and flip. Additionally, when removing almost all augmentations (crop and color distortion), we approach the performance of the prior representations of the generative models. This is expected by our theory since we have an augmentation graph that is almost disjoint for all points and thus we only rely on the prior to reconnect them. ImageNet100. Since we are still not matching supervised performance on ImageNet using contrastive models, it means the augmentation graph is not entirely classconnected and there is still room for improvement.   Table 4: We report linear evaluation AUC(%) using a 5-fold leave-siteout CV scheme to avoid possible bias on acquisition site.

Weakly Supervised Learning
Now we assume to have access to image attributes that correlate well with true semantic labels (e.g birds color or size for birds classification  Table 5: If images attributes are accessible (e.g birds color or size for CUB200), they can be leveraged as prior in our framework to improve the representation.

Conclusion
In this work, we have showed that we can integrate prior information into CL to improve the final representation. In particular, we draw connections between kernel theory and CL to build our theoretical framework. We demonstrate tight bounds on downstream classification performance with weaker assumptions than previous works. Empirically, we show that generative models provide a good prior when augmentations are too weak or insufficient to remove easy-to-learn noisy features. We also show applications in medical imaging and in the weakly supervised setting. We hope that CL will benefit from the future progress in generative modelling with our theoretical framework. [2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.

A Broader Impact
This work provides a technical advancement in the field of unsupervised representation learning, that already has great impact throughout many applications. In particular, in the medical context, including prior information can be of great value in order to convince clinicians to use AI solutions for computer-aided diagnosis. This work is also a theoretical advancement in representation learning for deep learning (DL) applications. We believe that it is crucial to provide theoretical guarantees of current DL systems in order to improve their reliability and robustness.

B More Empirical Evidence
In this section, we provide additional empirical evidence to confirm several claims and arguments developed in the paper. We empirically show here that Decoupled Uniformity optimizes alignment, even in the regime when the batch size n > d + 1, where d is the representation space dimension. We use CIFAR-10 and CIFAR-100 datasets and we optimize Decoupled Uniformity (without kernel) with all SimCLR augmentations with d = 128 and we vary the batch size n. We report the alignment metric defined in [48] as L align = E A(x|x)A(x |x)p(x) ||f (x) − f (x )|| 2 .

B.2 Multi-view Contrastive Learning with Decoupled Uniformity
When the intra-class connectivity hypothesis is full-filled, we showed that Decoupled Uniformity loss can tightly bound the classification risk for well-aligned encoders (see Theorem 3). Under that hypothesis, we consider the standard empirical estimator of µx ≈ V v=1 f (x (v) ) for V views. Using all SimCLR augmentations, we empirically verify that increasing V allows for: 1) a better estimate of µx which implies a faster convergence and 2) better SOTA results on both small-scale (CIFAR10, CIFAR100, STL10) and large-scale (ImageNet100) vision datasets. We always use batch size n = 256 for all approaches with ResNet18 backbone for CIFAR10, CIFAR100 and STL10 and ResNet50 for ImageNet100. We report the results in Table 6.   Table 6: A better approximation of centroids µx (i.e. increasing number of views) when augmentation overlap hypothesis is (nearly) full-filled implies faster convergence. All models are pre-trained with batch size n = 256. We use ResNet18 backbone for CIFAR10, CIFAR100, STL10 and ResNet50 for ImageNet100. We report linear evaluation accuracy (%) for a given number of epochs e.

B.3 Longer training on ImageNet100
A longer training time is beneficial for our method when using the BigBiGAN representation pretrained on ImageNet as prior (K GAN Decoupled Unif). In particular, we show that at 400 epochs, we obtain SOTA results. We always use batch size n = 256 in this experiment and we use the kernel K GAN (x,x ) = K(z(x), z(x )) where K is an RBF kernel and z(·) is the representation given by BigBiGAN's encoder pre-trained on ImageNet.
Model epochs ImageNet100 SimCLR ( Table 7: Using prior information given by BigBiGAN encoder pre-trained on ImageNet for Kernel Decoupled Uniformity provides better representation.

B.4 Influence of temperature and batch size for Decoupled Uniformity
InfoNCE is known to be sensitive to batch size and temperature to provide SOTA results. In our theoretical framework, we assumed that f (x) ∈ S d−1 but we can easily extend it to f ( We show here that Decoupled Uniformity does not require very large batch size (as it is the case for SimCLR) and produce good representations for t ∈ [1, 5].
Datasets t = 0.1 t = 0.5 t = 1 t = 2 t = 5 t = 10  Table 9: Linear evaluation accuracy (%) after training for 200 epochs with a batch size n, ResNet18 backbone and latent dimension d = 128. Decoupled Uniformity is less sensitive to batch size than SimCLR thanks to its decoupling between positives and negatives, similarly to [51].

B.5 Kernel choice on RandBits experiment
In our experiments on RandBits, we used RBF Kernel in Decoupled Uniformity but other kernels can be considered. Here, we have compared our approach with a cosine kernel on Randbits with k = 10 and k = 20 bits. There is no hyper-parameter to tune with cosine. From Table 10, we see that cosine gives comparable results for k = 10 bits with RBF but it is not appropriate for k = 20 bits.

C Geometrical Considerations about Decoupled Uniformity
In this section, we provide a geometrical understanding of Decoupled Uniformity loss from a metric learning point of view. In particular, we consider the Log-Sum-Exp (LSE) operator often used in CL as an approximation of the maximum. We consider the finite-samples case with n original samples ( iid ∼ A(·|x i ) for each samplex i . We make an abuse of notations and set i ). Then we have: j ) and s(·, ·) = f (·), f (·) 2 is viewed as a similarity measure. From a metric learning point-of-view, we shall see that minimizing Eq. 5 is (almost) equivalent to looking for an encoder f such that the sum of similarities of all views from the same anchor (s + i and s + j ) are higher than the sum of similarities between views from different instances (s − ij ): where is a margin that we suppose "very big" (see hereafter). Indeed, this inequality is equivalent to − > 2s − ij − s + i − s + j for all i = j, which can be written as : This can be transformed into an optimization problem using the LSE (log-sum-exp) approximation of the max operator: Thus, if we use an infinite margin (lim →∞ ) we retrieve exactly our optimization problem with Decoupled Uniformity in Eq.5 (up to an additional constant depending on n).

D Additional general guarantees on downstream classification D.1 Optimal configuration of supervised loss
In order to derive guarantees on a downstream classification task D when optimizing our unsupervised decoupled uniformity loss, we define a supervised loss that measures the risk on a downstream supervised task. We prove in the next section that the minimizers of this loss have the same geometry as the ones minimizing cross-entropy and SupCon [31]: a regular simplex on the hyper-sphere [22]. More formally, we have: Lemma 6. Let a downstream task D with C classes. We assume that C ≤ d + 1 (i.e., a big enough representation space), that all classes are balanced and the realizability of an encoder f * = arg min f ∈F L sup (f ) with L sup (f ) = log E y,y ∼p(y)p(y ) e −||µy−µ y || 2 , and µ y = E p(x|y) µx. Then the optimal centroids (µ * y ) y∈Y associated to f * make a regular simplex on the hypersphere S d−1 and they are perfectly linearly separable, i.e min (wy) y∈Y ∈R d E (x,y)∼D 1(w y · µ * y < 0) = 0. Proof in the next section.
This property notably implies that we can realize 100% accuracy at optima with linear evaluation (taking the linear classifier g(x) = W * f * (x) with W * = (µ * y ) y∈Y ∈ R C×d ).

D.2 General guarantees of Decoupled Uniformity
In its most general formulation, we tightly bound the previous supervised loss by Decoupled Uniformity loss L d unif depending on a variance term of the centroids µx conditionally to the labels: Theorem 7. (Guarantees for a given downstream task) For any f ∈ F and augmentation A we have: where Var(µ j x |y) = E p(x|y) (µ j x − E p(x |y) µ j x ) 2 , y = arg max y ∈Y Var(µ j x |y ) and µ j x is the j-th component of µx = E A(x|x) f (x). Proof in the next section.
Intuitively, it means that we will achieve good accuracy if all centroids (µx)x ∈X for samplesx ∈X in the same class are not too far. This theorem is very general since we do not require the intra-class connectivity assumption on A; so any A ⊂ A * can be used.

E Experimental Details
Code will be released upon acceptance of the manuscript. We provide a detailed pseudo-code of our algorithm as well as all experimental details to reproduce the experiments run in the manuscript.

BHB [17]
This dataset is composed of 10420 3D brain MRI images of size 121 × 145 × 121 with 1.5mm 3 spatial resolution. Only healthy subjects are included.

BIOBD [28]
It is also a brain MRI dataset including 662 3D anatomical images and used for downstream classification. Each 3D volume has size 121 × 145 × 121. It contains 306 patients with bipolar disorder vs 356 healthy controls and we aim at discriminating patients vs controls. It is particularly suited to investigate biomarkers discovery inside the brain [27].

E.4 Contrastive Models
Architecture. For all small-scale vision datasets (CIFAR-10 [34], CIFAR-100 [34], STL-10 [13], CUB200-2011 [47] and UT-Zappos [53]), we used official ResNet18 [26] backbone where we replaced the first 7 × 7 convolutional kernel by a smaller 3 × 3 kernel and we removed the first maxpooling layer for CIFAR-10, CIFAR-100 and UTZappos. For ImageNet100, we used ResNet50 [26] for stronger baselines as it is common in the literature. For medical images on brain MRI datasets (BHB [17] and BIOBD [28], we used DenseNet121 [29] as our default backbone encoder, following previous literature on these datasets [17]. Following [8], we use the representation space after the last average pooling layer with 2048 dimensions to perform linear evaluation and use a 2-layers MLP projection head with batch normalization between each layer for a final latent space with 128 dimensions. Batch size. We always use a default batch size 256 for all experiments on vision datasets and 64 for brain MRI datasets (considering the computational cost with 3D images and since it had little impact on the performance [17]).
Optimization. We use SGD optimizer on small-scale vision datasets (CIFAR-10, CIFAR-100, STL-10, CUB200-2011, UT-Zappos) with a base learning rate 0.3 × batch size/256 and a cosine scheduler. For ImageNet100, we use a LARS [52] optimizer with learning rate 0.02 × √ batch size and cosine scheduler. In Kernel Decoupled Uniformity loss, we set λ = 0.01 √ batch size and t = 2. For SimCLR, we set the temperature to τ = 0.07 for all datasets following [51]. Unless mentioned otherwise, we use 2 views for Decoupled Uniformity (both with and without kernel) and the computational cost remains comparable with standard contrastive models.
Training epochs. By default, we train the models for 200 epochs unless mentioned otherwise for all vision data-sets excepted CUB200-2011 and UTZappos where we train them for 1000 epochs, following [46]. For medical datasets, we perform pre-training for 50 epochs, as in [17]. For linear evaluation, we use a simple linear layer trained for 300 epochs with an initial learning rate 0.1 decayed by 0.1 on each plateau.
Augmentations. We follow [8] to define our full set of data augmentations for vision datasets including: RandomResizedCrop (uniform scale between 0.08 to 1), RandomHorizontalFlip and color distorsion (including color jittering and gray-scale). For medical datasets, we use cutout covering 25% of the image in each direction (1/4 3 of the entire volume), following [17].
Training. For VAE, we use PyTorch-lightning pre-trained model for STL-10 3 and we optimize VAE for CIFAR-10 and CIFAR-100 for 400 epochs using an initial learning rate 10 −4 and SGD optimizer with a cosine scheduler. We use the same pipeline on RandBits dataset. For DCGAN, we optimize it using Adam optimizer (following [41]) and base learning rate 2 × 10 −4 .
Finally, since ||µ i || = 1 then we have equality in the Jensen's inequality Since || · || is strictly convex on the hyper-sphere, then f must be constant on supp A(·|x i ), for allx i so f must be perfectly aligned.
Theorem 2. (Asymptotical Optimality) When the number of samples is infinite n → ∞, then for any perfectly aligned encoder f ∈ F that minimizes L d unif , the centroids µx forx ∼ p(x) are uniformly distributed on the hypersphere S d−1 .
PROOF. Let f ∈ F perfectly aligned. Then all centroids µx = f (x) lie on the hypersphere S d−1 and we are optimizing: So a direct application of Proposition 1. in [48] shows that the uniform distribution on S d−1 is the unique solution to this problem and that all centroids are uniformly distributed on the hyper-sphere.

F.3 Optimality of Supervised Loss
Lemma 6. Let a downstream task D with C classes. We assume that C ≤ d + 1 (i.e., a big enough representation space), that all classes are balanced and the realizability of an encoder f * = arg min f ∈F L sup (f ) with L sup (f ) = log E y,y ∼p(y)p(y ) e −||µy−µ y || 2 , and µ y = E p(x|y) µx. Then the optimal centroids (µ * y ) y∈Y associated to f * make a regular simplex on the hypersphere S d−1 and they are perfectly linearly separable, i.e min (wy) y∈Y ∈R d E (x,y)∼D 1(w y · µ * y < 0) = 0.
PROOF. This proof is very similar to the one in Theorem 1. We first notice that all "labelled" centroids µ y = E p(x|y) µx are bounded by 1 (||µ y || ≤ E p(x|y) E A(x|x) ||f (x)|| = 1 by Jensen's inequality applied twice). Then, since all classes are balanced, we can re-write the supervised loss as: We have: with equality if and only if C y=1 µ y = 0 and ∀y ∈ [1..C], ||µ y || = 1. By strict convexity of u → e −u , we have: with equality if and only if all pairwise distance ||µ y − µ y || are equal (equality case in Jensen's inequality for strict convex function), C y=1 µ y = 0 and ||µ y || = 1. So all centroids must form a regular C −1-simplex inscribed on the hypersphere S d−1 centered at 0. Furthermore, since ||µ y || = 1 then we have equality in the Jensen's inequality ||µ y || = ||E p(x|y)A(x|x) f (x)|| ≤ E p(x|y)A(x|x) ||f (x)|| = 1 so f must by perfectly aligned for all samples belonging to the same class: ∀x,x ∼ p(·|y), f (x) = f (x ).

F.4 Generalization bounds for decoupled uniformity
Theorem 7. (Guarantees for a given downstream task) For any f ∈ F and augmentation distribution A, we have: where Var(µ j x |y) = E p(x|y) (µ j x − E p(x |y) µ j x ) 2 and µ j x is the j-th component of µx = E A(x|x) f (x).
F.5 Generalization bound under intra-class connectivity assumption Theorem 3. Assuming 1, then for any -weak aligned encoder f ∈ F: Where D is the maximum diameter of all intra-class graphs G y (y ∈ Y).
(1) follows from Jensen's inequality and by definition of µx.
(2) follows because f is -weak aligned and So we have ||µx − µx || ≤ 2 D and we can conclude by Theorem 7 (right inequality).

F.6 Conditional Mean Embedding Estimation
Let f ∈ F fixed.
PROOF. Let mx = E p(x|x) f (x), f (·) ∈ H X be the conditional mean embedding operator. According to Theorem 6 in [44] and the assumption ∀g ∈ H X , E p(x|·) g(x) ∈ HX , this estimator can be approximated by:mx with α i defined previously in the theorem. This estimator converges with RKHS norm to mx at rate O( 1 √ nλ + λ). So we need to link mx,mx with µx,μx. We have: (1) holds by the reproducing property of kernel K X in H X . We can similarly obtain: Again, (1) by reproducing property of K X . And finally: By pooling these 3 equalities, we have: We can conclude since ||mx −mx|| ≤ O(λ + (nλ) −1/2 ).
F.7 Generalization bound under extended intra-class connectivity hypothesis Theorem. Assuming 3 and 2 holds for a reproducible kernel KX and augmentation distribution A.
PROOF. Let a, b, c ∈X . We consider the distance d(x, y) = K(x, x) + K(y, y) − 2K(x, y) (it is a distance since K is a reproducible kernel so it can be expressed as K(·, ·) = φ(·), φ(·) ). We will distinguish two cases.