{'fraction_non_alphanumeric': 0.02842126140420468, 'fraction_numerical': 0.013288377627925426, 'mean_word_length': 4.650678023086406, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CLIPping Privacy: Identity Inference Attacks on Multi-Modal Machine Learning Models

As deep learning is now used in many real-world applications, research has focused increasingly on the privacy of deep learning models and how to prevent attackers from obtaining sensitive information about the training data. However, image-text models like CLIP have not yet been looked at in the context of privacy attacks. While membership inference attacks aim to tell whether a specific data point was used for training, we introduce a new type of privacy attack, named identity inference attack (IDIA), designed for multi-modal image-text models like CLIP. Using IDIAs, an attacker can reveal whether a particular person, was part of the training data by querying the model in a black-box fashion with different images of the same person. Letting the model choose from a wide variety of possible text labels, the attacker can probe the model whether it recognizes the person and, therefore, was used for training. Through several experiments on CLIP, we show that the attacker can identify individuals used for training with very high accuracy and that the model learns to connect the names with the depicted people. Our experiments show that a multi-modal image-text model indeed leaks sensitive information about its training data and, therefore, should be handled with care.


Introduction
For a long time, machine learning models were trained on single modalities. In recent years, however, research has started to focus on the combination of multiple modali-* Contact: dominik.hintersdorf@cs.tu-darmstadt.de ties, such as text and images, to create even more powerful models. Large multi-modal models are pushing the benchmarks in many applications like text-guided image generation [47,40,49,65], visual question answering [11], image captioning [38] or even content moderation [51].
One of the milestones for multi-modal machine learning was the introduction of Contrastive Language-Image Pre-training [44], or CLIP in short, which has opened up a whole new area of unsupervised pre-training. Instead of requiring large datasets of labeled images to pre-train a model, CLIP uses image-text pairs, collected from the internet, to train a vision model and a text transformer simultaneously in a contrastive learning fashion. During training, the visual and text embeddings of images and their captions are forced to be close to each other, while maximizing the distance to other embeddings of text examples in the training set. As a result, CLIP is learning the visual concepts depicted on the images through the accompanied texts.
Trained on massive data, the resulting multi-modal models are quite powerful. However, even though the data is publicly accessible, it also covers sensitive information like names, addresses, and phone numbers crawled from the internet at scale. Unfortunately, as we will show, this makes multi-modal models not only more powerful but also easier to attack than single modality approaches. When attacking a large language model such as GPT-2 [7], the attacker has to make hundreds or thousands of queries to the model, whereas when attacking a multi-modal model one can simply ask the model to name the person in an image. In other words, attackers being able to combine different modalities are even more powerful and, in turn, the models pose an even more severe risk to privacy.
To demonstrate this trade-off, we propose a novel attack, called Identity Inference Attacks (IDIAs  in a membership inference attack the attacker wants to infer whether a sample was used for training the model, an identity inference attack takes a broader view and ask whether a particular person was part of the training data. As researchers are developing more and more capable models, with even the goal of artificial general intelligence (AGI), one has to think about and discuss possible trade-offs between utility and user privacy. In particular, if a model has a high level of knowledge and can answer even complicated questions, an attacker, if posing the right questions, can extract sensitive information.
On the other hand, we should be aware of "Clearview AI" [19] and "Ever" [32] moments for large multi-modal models. Clearview AI is scraping billions of images of individuals, and offers its service to law enforcement for police investigations, to prevent human trafficking, or offers identity verification for other businesses, while also failing to comply with the data protection laws in Canada, Australia, and parts of Europe and has therefore been fined by multiple countries [35,33,34]. The company Everalbum, provider of the now shut down cloud photo storage app "Ever", was training facial recognition models on users' photos without their consent and was ordered by the US Federal Trade Commission to delete all models and algorithms trained on this user data [32,62]. This shows that it is crucial to check where data is coming from and whether the individual data owners have given consent before training a model.
Likewise for multi-modal models like CLIP [44], DALL-E [47] or Imagen [49], even if the data from largescale datasets are publicly accessible, it is still questionable whether the individuals depicted have given their consent to be used for training a machine learning model. Therefore, it is critical to investigate whether multi-modal models leak information about their training data. If this should turn out to be the case, those models could not only be misused to identify or infer sensitive information about individuals, but would also potentially violate data protection laws. This shows that attacks on multi-modal models have a severe social impact and go beyond the academic world. Information leakage like for example names, addresses, phone numbers or emails, and connections with other modalities like images could be exploited by an attacker for various malicious purposes.
To summarize, this paper focuses on the privacy risks of multi-modal image-text systems and investigates whether multi-modal models leak sensitive information: 1. We propose a new kind of privacy attack, called identity inference attack (IDIA), which aims to extract sensitive information about specific identities from a trained multi-modal zero-shot classifier.
2. Exhaustive experiments demonstrate that an attacker can infer with very high accuracy whether a person was used for training and that multi-modal models indeed leak information about their training data.
3. IDIA shows that image-text models are so powerful that they easily learn to connect the names with the depicted individuals during training, even if the person appears very few times in the training data. This may imply a novel trade-off: the more "emergent" abilities, the more privacy leakage.  [39] and German celebrities [25,24].
We proceed as follows. We start off by touching upon related work and background in Sec. 2. In Sec. 3 we introduce identity inference attacks and present our exhaustive empirical evaluation in Sec. 5. Before concluding, we are discussing our results and possible implications.

Background and Related Work
To begin, we start by giving an overview over multimodal systems, particularly CLIP. We further introduce related attacks on machine learning systems and discuss research on the privacy of multi-modal systems.

Multi-Modal Systems as Zero-Shot Classifiers
Multi-modal learning describes the process of training machine learning models to use information from different modalities at the same time, e.g., written text, images, or audio. Early approaches trained models based on deep Boltzmann machines [56] or LSTM networks [46]. Instead of training a standard image classifier directly on a labelled training set, CLIP [44] introduced a novel contrastive pretraining scheme. The approach of Radford et al. [44] relies on public data scraped from millions of websites. More specifically, the training data consists of image-text pairs, where the images are accompanied by their corresponding captions. CLIP is trained by jointly optimizing an image feature extractor M img and a text encoder M text to learn to match images with their corresponding textual description. While the image feature extractor can be any standard neural network for computer vision tasks, for example, a ResNet [18] or a vision transformer (ViT) [10], a transformer is usually used as the text encoder. The original CLIP model was trained on a non-public dataset with about 400 million image-text pairs, scraped from public websites.
After training the multi-modal model on these imagetext pairs, the authors demonstrated that CLIP achieves high zero-shot prediction accuracies on various tasks. Zero-shot prediction in this context means to use the model for predictions on unseen datasets and tasks. Given an image and multiple text prompts, the model will predict the text prompt, best matching the visual concept on the image.
To achieve this, the model is queried with an image x to make a prediction on, together with multiple text prompts Z = [z 1 , z 2 , ..., z n ] which contain descriptions of possible visual concepts. Based on the computed embeddings of both inputs, the pairwise cosine similarity score between every text prompt z i and the input image x is computed and scaled by a temperature term τ : Finally, the cosine similarities are normalized using a softmax function and the text prompt with the highest probability is predicted. This makes CLIP a very versatile zero-shot classifier, since the labels and the number of possible classes for the classification can be chosen freely. Radford et al. [44] further demonstrated that carefully selected text prompts, also known as prompt engineering, often improve the zero-shot prediction accuracy. For example, instead of using cat as a text prompt, embedding it into the prompt a photo of a cat helps the model to understand the context better.

Privacy and Security of Machine Learning Models
Various privacy and security attacks against machine learning systems have been proposed. Arguably best known in the field of security attacks are adversarial attacks [59,17]. These attacks introduce fine-tuned noise to images to manipulate a model's prediction, while avoiding humans being able to detect these manipulations by visual inspection. Subsequent research in this area adapted adversarial attacks to other kinds of models, such as NLP [43,68] or multi-modal [13,41] models, but also demonstrated their applicability and security risks in production-ready systems [58,37].
Related to adversarial attacks but more focused on the privacy instead of security aspect of machine learning models are model inversion attacks [14]. Similar to adversarial attacks, most of the approaches are altering input images, maximizing the output probabilities of a certain class and are trying to reconstruct specific training samples [67] or class representatives [63,57] based on a trained model's learned knowledge. But leakage of training data is not only limited to vision models. Carlini et al. [7] demonstrated that large natural language models like GPT-2 memorize training data and leak sensitive information. Other attacks that try to gain information about the training data are property inference attacks [2,15], inferring properties about the training data, or model extraction attacks [27,22], which aim to create a clone of a trained model by querying it with black box access only.
Most related to our approach, however, are membership inference attacks (MIAs), first introduced by Shokri et al. [54]. By performing MIAs, the adversary tries to identify the specific samples used to train a machine learning model. Successfully inferring this information might indeed lead to a privacy breach, for example in domains like medicine or facial recognition. If, for example, a model was trained on health data from a certain hospital, inferring membership of a data sample would leak the information that the person to which this data sample belongs, was once a patient in this hospital. This could leak private information such as where the person lives or whether the person has had certain illnesses.
Describing MIAs more formally, the adversary has access to some data point x and a target model M target , which has been trained on an unknown training set S target training . The goal of the attack is then to answer whether the data point was in the training data x ∈ S target training (member) or was not in the training data x / ∈ S target training (non-member). In the literature, it is usually assumed that the attacker has access to a non-overlapping shadow dataset [54] S shadow training , which the attacker can use to prepare the attack. Most proposed attacks [54,50,29,9] train so-called shadow models M shadow on the shadow dataset S shadow training . These models are used to mimic the behavior of M target but with the advantage that the adversary knows the training data S shadow training used to train M shadow . With this knowledge, an inference model h, which can be a simple threshold or a more complex model, is trained on the outputs of M shadow on training and non-training samples. The trained inference model h is then applied to M target and some unseen data x to predict the membership status.
As one of the first MIAs, Shokri et al. [54] trained multiple shadow models to fit h. The proposed attack only requires access to the prediction scores of M target . Later, Salem et al. [50] improved the attack process and demonstrated that training a single shadow model is sufficient to perform the attacks while achieving comparable results. While the attacks proposed by Salem et al. use the prediction scores of M target , Yeom et al. [64] exploited the fact that neural networks for image classification are trained to minimize some loss on their training samples and, therefore, propose to distinguish between members and non-members based on their loss. Carlini et al. [5] recently proposed a novel likelihood ratio attack that trains multiple models, half of them on data sets which contain the data sample x to infer its membership status, and half of them without. Subsequently, the models are used to approximate the loss distribution on the models' predictions on x and perform a likelihood ratio test to predict the membership.
However, in reality, often the attacker has only label-only access to the model and cannot get the prediction scores for each class or calculate the model's loss for a sample. Therefore, Choquette-Choo et al. [9] and Li and Zhang [29] introduced label-only MIAs, which rely solely on the predicted labels to infer the membership status. The intuition behind these attacks is that trained models are seemingly more robust in predicting samples from the training data than predicting unseen data.
Whereas the literature on MIAs using shadow models attest high success rates for inferring the membership status of samples, they all make the assumption that the attacker knows the data distribution of the training data. Hintersdorf et al. [20] recently argued that this information is usually not available to the adversary and that the actual privacy risk resulting from MIAs under realistic conditions, where the attacker does not know the distribution of the training data, is overestimated. The authors demonstrated that the overconfidence of neural networks acts as an implicit defense against MIAs, and that there exists a trade-off between mitigating overconfidence and the susceptibility against MIAs.
Moving slightly beyond MIA but staying uni-modal, Li et al. [28] observed that embeddings of a person used for training, say, a LuNet for image classification, result in more compact clusters than images of other individuals in metric embedding learning. Likewise, other approaches on text [55] and audio data [36] exploit the uniqueness of the data samples of a person to check whether their data was used for training.
Finally, investigating the privacy of vision models trained with contrastive-learning, Liu et al. [30] have shown that these models are susceptible to membership inference attacks. The proposed attack exploits the fact that during training, the augmentations of a single image are close to each other in the embedding space, while the augmented images of samples not used for training are more distant to each other. Even though they also looked at vision models pre-trained using CLIP, they are only investigating the privacy of the learned vision model with an attacker that has access to the calculated embeddings of the images.
To summarize, IDIA is the first image-text attack exploiting that CLIP models easily learn to connect the names with the depicted individuals during training, even if the person appears very few times in the training data.

Privacy and Security of Multi-Modal Systems
The idea to combine different modalities to perform privacy attacks is not new. Jaiswal and Provost [23], for example, have shown that multi-modal representations unintentionally encode additional information like gender or identity. Removing the gender and speaker information from the representations using adversarial learning seems to reduce the information leakage. Similarly, Rahman et al. [45] have shown that combining different modalities like image and text information, locations, and hashtags in social media posts, an attacker can achieve higher accuracy in inferring relationships in social networks than when just using a single modality.
As multi-modal systems can also be used as data retrieval models, Zhang et al. [66] have proposed a privacy protection method to prevent an adversary to search for and combine private information in released data such as photos. While training a retrieval system, the authors propose to modify the original data using adversarial perturbations to prevent images from being retrieved by an unauthorized party, not having access to the trained system.
In the area of security of multi-modal models, Carlini et al. [6] demonstrated that image-text systems trained in a contrastive manner are susceptible against simple backdoor and data poisoning attacks. The authors demonstrated that by adding a small trigger to only 0.0001% of the dataset, causes the model to misclassify its input when presented with the trigger.
To analyze how CLIP is learning to abstract the visual concepts during training, Goh et al. [16] have examined how the neurons of the vision model of CLIP react to certain stimuli. They have discovered that the model has multimodal neurons that are activated by different images that describe the same visual concept. Apparently, there are neurons that react to the visual concept that are attributed to certain people. As an example, the "Jesus Christ neuron" is activated by pictures of Jesus Christ but also by pictures of other Christian symbols like crosses or crowns of thorns.
Related to the analysis of the neurons of CLIP, Thies [61] demonstrated with a very limited example in a short blog post that CLIP does not only store general concepts of images and texts, but also some information about people's identities in the training data. Similarly, the original CLIP paper also noted that the model can be used to classify celebrities from the CelebA dataset [31].
However, to date, no systematically in-depth research has been conducted in the privacy context of zero-shot multi-modal classifiers such as CLIP. With our work, we want to close this research gap and demonstrate that new privacy concerns go hand in hand with the raise of multimodal models trained on large amount of public data from the internet.

Identity Inference Attacks
Now we are ready to introduce our identity inference attack (IDIA) as depicted in Fig. 1. In an IDIA, the adversary is given black-box access to a multi-modal image-text model together with some pictures of a person whose name is known. Additional information, for example about the architecture or the training algorithm used to train the multimodal model, is not available to the attacker. The goal of the attacker is to infer whether the model was trained using some images of this specific person.
More formally, given images I of an individual and its name N , the attacker can query a target model M with the images x i ∈ I and text prompts P with N ∈ P containing potential names. After receiving the prediction z x,i from the model, the attacker applies the decision function f : z x,i × P → {0, 1}, which outputs 1 if the correct name for the person depicted on x i was predicted and 0 otherwise. The identity inference attack score S(I, P ) is then calculated using the following function: To infer whether the person was in the training set, the attacker applies the identity inference attack prediction function Q, which takes the attack score, applies a threshold τ to it and outputs 1 if the person is predicted to be in the training set and 0 otherwise: Setting, e.g., the threshold as τ = 0.5, will cause Q predicting the person to be in the training set, if more than half of the images available to the attacker are predicted correctly. In Sec. 6 we will discuss how a different threshold might influence the results of the attack.

Attack Protocol
Identity inference attacks (IDIAs) assume a threat model that we describe now in detail and that also serves as our experimental setup.
Attacker's Capabilities. In our attack, we consider an attacker who has black-box label-only access to a trained CLIP model [44]. Performing privacy attacks in a black-box label-only setting is usually the hardest and most challenging setting for the adversary. The only information given to the attacker are a certain number of images of an individual and the name of the person.
Datasets. To evaluate our attack, we are attacking CLIP models that were pre-trained on the LAION-400M [52,21] dataset and the Conceptual Captions 3M [53] dataset. For the identities for which we want to infer whether they are in the training data, we have taken images of actors and actresses from the FaceScrub dataset [39], consisting of facial images of 530 actresses and actors. Both the LAION-400M dataset and the Conceptual Captions 3M dataset contain images and text pairs extracted from millions of webpages. While the LAION-400M dataset contains 400 million image-text pairs, the Conceptual Captions dataset contains 3.3 million image-text pairs. Because only the URLs of the images in the Conceptual Captions dataset are provided and some of these images are not available anymore, we were able to download 2.8 million images out of the 3.3 images URLs given. Because there is also a Conceptual Captions dataset containing 12 million images [8], we are referring to the dataset used in this work as Conceptual Captions 3M (CC3M).
Target Models. In our experiments, we evaluate our IDIA on several CLIP models with different image feature extractors. In our experiments on the LAION-400M dataset, we attack models with the three vision transformer architectures ViT-B/32, ViT-B/16 and ViT-L/14 [10] used as image feature extractors, which were all pre-trained by the OpenCLIP project [21]. The different image feature extractors used for the experiments on the CC3M dataset are ResNet-50 [18], ViT-B/32 [10] and a ResNet-50x4. ResNet-50x4 is a ResNet-50, which was scaled accordingly using EfficientNet-style scaling techniques [60]. During all the experiments, the text encoder of the CLIP model is identical to the original paper [44].
With this at hand, one can prepare attacks and their evaluation. This includes the dataset analysis and training the target models. Our code, as well as the pre-trained models, are available on GitHub 1 .
Analyzing and Creating Datasets. To evaluate the attack, we first have to analyze which entities are in the dataset and which are not. This enables the correct calculation of false-positive and false-negative predictions. To search for entities within such large dataset with millions of image-text pairs, we used clip-retrieval [3], an open-source implementation for image retrieval based on the image and text embeddings calculated by CLIP.
To analyze the LAION-400M dataset, we retrieved the 200 most similar images together with their caption for each image in the FaceScrub dataset, resulting in roughly 7.5 million retrieved image-text pairs. We then checked whether the name of the actor or actress is present in one of the captions of the retrieved images. For analyzing the Conceptual Captions 3M (CC3M) dataset, a similar procedure as for analyzing the LAION-400M dataset was used. However, while creating the CC3M dataset, Sharma et al. [53] have anonymized entity names in the image captions by rephrasing the captions or by replacing them, e.g., names with a <PERSON> tag. Therefore, it is not possible to search for the individual names in the image captions. To still be able to analyze which individuals are already present in the dataset, we trained a ResNet-50 on the FaceScrub dataset to classify the faces of the 530 identities. The ResNet-50 used, was trained with a batch size of 128 for 100 epochs on a single A100 GPU using the cropped images of the FaceScrub dataset. 90% of the dataset was used to train the model, while the other 10% was used as validation set. All images were resized to 224x224 pixels and center cropped. During training, randomly resized cropping, color jitter and horizontal flipping were applied as data augmentation techniques. We used Adam [26] with a learning rate of 0.003, which was reduced by 0.1 after 75 and 90 epochs.
The trained ResNet-50 achieved a test accuracy of 91% on the FaceScrub test set. We used OpenCV's [4] Haar cascade face detector to detect faces in the similar images which were retrieved from the CC3M dataset and used the trained ResNet-50 to predict whether the person in question was present in the images. In addition to the trained ResNet-50 for filtering the images, we used the CLIP model pre-trained on LAION-400M to predict whether the person is present in the retrieved images of the CC3M dataset. We then chose the 200 individuals with the lowest number of correct predictions and mixed the first 100 of these individuals with the CC3M dataset, while the other half of the individuals was not used for training the target models of our experiments.
Training Target Models on CC3M. To train the CLIP models on the CC3M dataset, we used the code of the opensource project OpenCLIP [21]. As can be seen in Tab. 1, we trained CLIP models with three different vision model architectures. The hyperparameters of the vision and text model architectures are the same as in the original CLIP paper [44]. While the text transformers used as an encoder for the ResNet-50 and the ViT-B/32 architecture models use 12 layers, 8 heads and have a width of 512, the text transformer for the ResNet-50x4 architecture is slightly larger and uses 12 layers, 10 heads and a width of 640. Because we want to investigate the influence of the number of occurrences of a person on the success of the attack, we trained five models with 75, 50, 25, 10 and 1 data sample occurrences per person for each of the architectures. All the models were trained on 8 A100 GPUs for 50 epochs on the CC3M dataset with a per GPU batch size of 128, a learning rate of 1e −3 and a weight decay parameter of 0.1.
The top-1 and top-5 ImageNet zero-shot prediction accuracy of the trained models can be seen in Tab. 1. While the two ResNets achieve comparable results of a top-1 accuracy of around 20%, the vision transformer ViT-B/32 achieves a top-1 accuracy of 14.3% on the ImageNet [48] validation set. We suspect that 2.8 million data points is not sufficient for the vision transformer to fully learn the visual concepts of the images. Even though the ResNet-50 is much smaller, it seems to generalize well and performs even better than the bigger vision transformer model. The image-to-text rank of the models describes how often the model is predicting the 8.01% ± 0.31% 6.13% ± 0.22% 4.94% ± 0.25% Table 2: Attackers can be confident in their predictions. The mean true-positive rate (TPR), true-negative rate (TNR), falsepositive rate (FPR) and false-negative rate (FNR) and their standard deviation of the identity inference attack on the CLIP models [21] pre-trained on the LAION-400M dataset.
correct text prompt for a given validation image. Rank@1 means that the ground truth caption for the image was the predicted one, while rank@5 means that the ground truth predicted caption was under the top-5 predicted ones. As with the zero-shot accuracy, one can see that the ResNet-50 and the ResNet-50x4 achieve comparable results. While the ResNet-50 achieves an image-to-text rank@1 of around 30%, the ResNet-50x4 achieves an image-to-text rank@1 of around 33%, meaning that in about a third of the cases the correct caption to a given image in the validation set was predicted.

Attack Results
In this section, we will present the results of our experiments conducted on the models trained on the LAION-400M and the CC3M dataset.

Experimental Results on LAION-400M
We evaluated the attack on three models with different sizes. The top-1 accuracies of the models on the validation set of ImageNet [48] can be seen in Tab. 2. The zero-shot top-1 accuracy of the models are all above 60%, which indicates that the models have learned meaningful visual concepts during training and can connect the visual concepts depicted on the images with given text descriptions. The results of the top-1 accuracy are comparable with those reported in the original CLIP paper [44]. For performing the attacks, the adversary has the name and 30 images of each person. During each attack, the adversary queries the model with each image and provides the model with 546 names as text prompts, 530 of which are the names of the entities in the FaceScrub dataset and 16 of which are the names of the German actors and actresses. In all our experiments, we have used only the names of the individuals as text prompts and conducted no prompt engineering.
The outcome of IDIA on the CLIP models pre-trained on the LAION-400M dataset can be seen in Tab. 2. The mean true-positive rate (TPR), true-negative rate (TNR), falsepositive rate (FPR) and false-negative rate (FNR) together with the standard deviation for the IDIA with 30 samples per person available to the attacker is depicted. Because some individuals have more than 30 images available, the attack was run 20 times, with randomly sampling 30 images from all available images for each person.
Looking at the results of the attacks, one can see that the models learn to recognize individual people. As can be seen, the mean true-positive rate of the attack on all models is consistently above 91% while the mean false-positive rate is quite low on the ViT-B/32 and ViT-B/16 models. The mean true-positive rate and the mean false-positive rate seems to increase, with the model size. Accordingly, the mean true-negative rate and mean false-negative rate decrease with increased model size.
The lower the false-positive rate, the higher the confidence of the adversary can be in the positive predictions. In a setting of a privacy attack, this is especially interesting since low false-positive rates are increasing the potential leakage of sensitive information. High false-positive rates, on the other hand, are reducing the risks of privacy attacks since the attacker cannot be confident in the results of the attacks. For the smaller ViT-B/32 and ViT-B/16 models, the attacker can be sure that the entities with positive predictions are indeed in the training data because of a falsepositive rate below 6%.
Interestingly, and against our initial expectation, the false-positive rate seems to considerably increase with the model size. Our intuition is that the larger a model's capacity, the higher the chance that the model is remembering the images and the names of a person. However, we suspect that the increase in the false-positive rate is due to false positives in our analysis of the LAION-400M dataset and incorrectly labelling entities as not occurring in the dataset during our analysis. It is very unlikely that the model is predicting the correct name out of 546 names more than half of the time if it has never seen the person during training. Because of the considerable size of this dataset, we unfortunately cannot verify this assumption. However, the low false-positive rate in the experiments on the CC3M dataset indicate that this assumption is most likely correct. Interestingly, sample selection does not seem to have a high impact on the results. While the standard deviation of the false-positive and true-negative rates is 0%, the standard deviation of the false-negative rate and the true-positive rate is below 0.31% in all three experiments. This indicates that the selection of an individual's images, used for the attack, has almost no effect on the success of the attack.

Influence of Number of Attack Samples.
To investigate how the number of samples available to an adversary influences the success of the attack, we varied the number of images available to the attacker. The results for all the models pre-trained on LAION-400M can be seen in Fig. 3.
As can be seen in all three plots, the number of samples does not make a big difference when having more than 5 images of a person available for the attack. An adversary with 5 samples achieves similar attack results in terms of true-positive rate as an adversary with 30 samples. However, it even seems that having more samples increases the number of false-positive predictions for the attack on the largest model ViT-L/14. As can be seen in Fig. 3c, the falsepositive rate is increasing with the number of samples, while the true-positive rate plateaus. As explained earlier, this is most likely caused by incorrectly labelled entities during the dataset analysis and might indicate that having more samples increases the effectiveness of the attack on larger models.
As for the results of the other two models seen in 3a and Fig. 3b, the false-positive rate as well as the false-negative rate stay very low for all sample sizes. This shows that an attacker with only a few images of a person can perform the attack and be confident in the positive predictions, which indicates that the adversary can confidently infer information about the training data.
In all three plots, there can be seen a dip in the truepositive rate with a sample size of 2. We assume that this is due to the attack predicting individuals to be in the training set only when for more than half of the images, the name of the person was correctly predicted. As a result, when having two images to perform the attack, in numerous instances there are conflicting predictions causing the IDIA to predict the individuals not to be in the training set. This causes the mean true-positive rate to decrease rapidly, while the mean false-negative rate increases.

Experimental Results on CC3M
Although the experiments with the models pre-trained on the LAION-400M dataset show impressive results, due to the size of the dataset, the analysis of the dataset is error-prone and the number of entities included and not included in the training set is very unbalanced. Therefore, we conducted additional results on the CC3M dataset, where the setting of the experiments can be controlled more thoroughly and the circumstances under which the attack is successful can be investigated more clearly.
We are especially interested in answering the following two research questions: 1. What is the influence of the number of images the attacker has of the individuals to perform the attack?
2. How does the number of images per person in the training dataset affect the success of the attack?
The results of IDIA on the CLIP models trained on the CC3M dataset can be seen in Tab Table 3: Attackers can be confident in their predictions. The mean true-positive rate (TPR), true-negative rate (TNR), falsepositive rate (FPR) and false-negative rate (FNR) and their standard deviation of the identity inference attack on the CLIP models trained on CC3M.
in the table are the mean and standard deviation values of the attack with 30 samples per person over 20 attacks. As stated, the mean true-positive rate is consistently above 63% for all models, while the mean false-positive rate is always below 1%. While the mean true-positive rate seems to increase, the mean false-negative rate seems to decrease with the model size. Interestingly, the size of the model seems to have no effect on the true-negative rate or the false-positive rate. Even though the number of parameters for the vision model has almost tripled and the number of parameters for the text encoder almost doubled from the ResNet-50 to the ResNet-50x4, both the true-negative rate and the false-positive rate have not changed very much, which is why the behavior of IDIA for unseen individuals stays the same.
In contrast to the experiments on the LAION-400M dataset, the false-positive rate is not increasing with the model size. This indicates that our assumption about incorrect labelled entities during the analysis of the LAION-400M dataset is most likely correct.
The number of parameters of the vision model seems to have a positive effect on the true-positive rate of the attack. Comparing the mean true-positive rate of the ResNet-50 with the mean true-positive rate of the ViT-B/32 model, one can see that the number of trainable parameters has more than doubled and as a result, the mean true-positive rate increased by 3%. However, comparing the results of the ViT-B/32 with the ResNet-50x4, one can see that the mean false-negative rate decreased by more than 8% while the true-positive rate increased by 8%. It seems that some samples that were incorrectly classified as not being in the training set, are now correctly predicted. We have two possible explanations on why this might be the case.
Our first theory concerns the number of parameters in the text encoder. Comparing the number of parameters of the ViT-B/32 model with the number of trainable parameters of the scaled ResNet-50x4 model, one can see that while the parameters of the vision model stays roughly the same, the number of parameters of the text encoder is increased by a factor of only more than 1.5. This might have a signifi-cant effect on the true-positive and false-negative rate. We suspect that the size of the text encoder has higher impact on the attack than the vision model, as a main component of the attack are the names of the individuals. While larger vision models can differentiate faces of different individuals better, a larger text encoder can recognize and memorize names better.
A second possible explanation for the higher truepositive rate and lower false-negative rate might be that the dataset of 2.8 million image-text pairs was too small for the vision transformer to learn the visual concepts well enough. As a result, the image features might not be extracted as well as with the ResNet-50x4 model, which results in a lower true-positive rate. The lower zero-shot top-1 accuracy on ImageNet might be one indication of this.
Investigating the cause for this behavior and the importance of the complexity of the vision and text models, using other architectures, is an interesting avenue for future work.
Influence of Number of Attack Samples. As with the experiments on the LAION-400M dataset, we are interested in what influence the number of samples available to the attacker has on the success of the attack. The results for the models trained with 75 image-text pairs for each person contained in the training set can be seen in Fig. 4. Similar to the experiments on the LAION-400M dataset, the accuracy and mean true-positive rate are increasing with more samples being available for the attacker. In contrast to the experiments on the LAION-400M dataset, the number of samples is influencing the success of the attack even more. On all models, the accuracy as well as the true-positive rate are increasing with the number of samples and does not plateau with more than 5 samples as with the LAION-400M experiments. The number of samples available to the attacker does not seem to effect the false-positive rate. Even if the attacker has only one sample, the false-positive rate is zero. As a result, the attacker can be confident in the positive predictions the attack is doing, even when performing the attack with only a single image per person.
To answer our first research question, it seems that   Even though the true-positive rate is just above 20%, the mean false-positive rate as well as the mean true-negative rate are 0% and 100%, respectively. As in the previous experiments, this indicates that the adversary still can learn information about the training data even if the person is present only 10 times in the training data. Reducing the number of training samples per person to only 1 among the 2.8 million training samples and using 30 samples to attack each identity, leads to a true-positive rate of 3.75% on the ResNet-50, 4.85% on the ViT-B/32 model and even of 5.7% on the ResNet-50x4, as can be seen in Fig. 5. Even though these true-positive values are very low, it shows that it is theoretically possible to identify a person on a single image among 2.8 million images.
Increasing the number of samples per person seems to have diminishing influence on the success of the attacks for the ResNet-50 and the ViT-B/32 model. While the mean TPR is increasing by as much as 18% when increasing the number of samples from 10 to 25, the true-positive rate is increasing only slightly for most of the models when the number of samples is increased from 50 to 75. However, this effect of diminishing returns is not as pronounced with the ResNet-50x4 model seen in Fig. 5b. Contrary to the other two models, the true-positive rate and the false-negative rate seem to improve with more training samples available without diminishing returns. While both the ResNet-50 and the ViT-B/32 have the same number of parameters for the text encoder, the ResNet-50x4 model has a much bigger text encoder. This supports our hypothesis that the text encoder is an important part for the success of the attack. The bigger text encoder being able to memorize the names of the individuals better might be a possible explanation for the improvements of the attack with increasing the number of training samples per person.
This leaves the question open, what influence the number of samples available to the attacker has on the success of the attack, depending on the number of samples per person that were present in the training data. In Fig. 6, the influence of the number of samples available to the attacker and the number of samples in the training data per person on the true-positive rate can be seen. As illustrated in the heatmaps of all three architectures trained on the CC3M dataset, the number of samples available to the attacker has the highest influence on the true-positive rate of IDIA when the identities were present 75 times in the training data set. Even though the TPR is increasing with the number of samples when the individuals appeared only 10 times in the training set, the effect is much less pronounced. As a general observation, one can say that the more often a person appears in the training set, the higher the influence of the number of samples of a person available to the attacker on the success of the attack.

Discussion
Choosing the Threshold. Our experiments have shown that multi-modal models recognize and memorize people and their names. In our experiments, we have chosen the threshold for predicting when a person was part of the training data as τ = 0.5. Therefore, if the name of the individual is predicted correct more than half of the times, the attacker predicts that the person was part of the training data. We selected this threshold because we wanted to ensure a very strict attack scenario, with the attacker having as few as possible information. However, the attacker could also search for a more suitable threshold. In our attack scenario, the only information the attacker has, are some images of a person and the name. In many privacy attacks, it is assumed that the attacker has much more information than that. For membership inference attacks, for example, it is assumed that the adversary has access to data from the same data distribution as the training data to train a shadow model. During the preparation of the attack, this shadow model is used to choose a threshold for predicting members and non-members. Assuming a similar scenario for our identity inference attack, where the attacker has additional information, one could fine-tune the threshold τ to even further reduce the false-negative rate. For example, the attacker could train an additional multi-modal model and use it as a shadow model, or the adversary has information about some entities being part of the training data. As a result, the privacy leakage of IDIA could be even more severe.
Trade-Off between Power and Privacy. To put it simple, in our proposed identity inference attack, the adversary is asking whether the model knows the person. Using the prediction, the attacker can reveal whether the individual was part of the training data. As researchers around the world are striving for artificial general intelligence (AGI) machine learning models are projected to be even more capable. However, as our experiments on the LAION-400M dataset have shown, multi-modal models capable of learning visual concepts and performing high accuracy zero-shot classification tasks are very susceptible to identity inference attacks. While the ViT-B/32 model trained on the LAION-400M dataset achieved a zero-shot top-1 accuracy of 62.9%, the true-positive rate of the attack is above 90% while the false-positive rate is 0%. Even though our experiments are not conclusive and future work has to investigate this further, we assume that there is a trade-off between the "emergent" abilities of machine learning models and their susceptibility to privacy attacks.

Possible Countermeasures
Our identity inference attack is exploiting the fact that the model is memorizing the visual appearance of a person together with its name. A possible counter measure to prevent these privacy attacks is to mask out all names in the dataset, similar to the approach applied during the creation of the Conceptual Captions 3M dataset [53]. However, it is questionable if this will fully prevent privacy attacks. If the name of a person is memorized, other attributes about this individual are most likely memorized too. One could think of other textual attributes than the name of a person. Taking celebrities, for example, one could think of films or TV-series in which they played a role. Instead of using the name of the person, one could use the name of the TV-series to infer whether the celebrity was in the training data.
With models getting more powerful, one should think about whether some kind of decency could be taught to these multi-modal models. If the model is aware that the information it is about to share is private, it could refuse to give an answer, similar to a doctor refusing to talk about the health status of other patients. With the DALL-E [47] API, OpenAI is already doing a first step in this direction. According to the content policy [42] of DALL-E, it is prohibited to generate public figures like celebrities or politicians. Trying to do so, the image generation is refused.

Conclusion
We have introduced a new type of privacy attack on multi-modal image-text models, called identity inference attack (IDIA). Through several experiments, we have demonstrated that using images and the name of the people, an attacker can predict with very high accuracy whether a person was used to train the model, to which (s)he has black-box access. Specifically, our evaluation on models trained on the LAION-400M dataset has shown that even models trained on 400 million data points and generalized well enough to achieve a zero-shot accuracy of more than 60% on the ImageNet validation dataset still leak information. Our experiments on the CC3M dataset have shown that because of the very low false-positive rate, the adversary can be confident in the results of the attack, causing privacy leakage even if the identity appeared as few as only a single time in a training set of 2.8 million images-text pairs.
Our experimental results suggest that posing the right questions, an attacker can extract sensitive information about the training data of a multi-modal system and that there might exist a trade-off between the "emergent" abilities of a machine learning model and privacy. Overall, our results suggest that multi-modal image-text models indeed leak sensitive information about the training data, and that the models should be handled with care.
IDIAs provide a number of interesting avenues for future work. While the evaluation of the IDIAs in this work was limited to 3 model architectures on two datasets each, it is important to investigate under which circumstances the attacks are successful. The complexity and the interplay between the vision model and the text encoder are most likely vital to the success of IDIAs. Investigating the influence of the size of the vision and the text model would be an interesting avenue for future work.
Even though we have shown that multi-modal zero-shot models like CLIP are susceptible to privacy attacks, these kinds of attacks are not limited to multi-modal modals with zero-shot capabilities. As text-to-image generation models like DALL-E [47] or Imagen [49] are built on the same principle to learn the visual concepts and connect them to texts, it is very likely that those models as well suffer from training data leakage. Attacking such a generative model, the adversary would need even less information. Looking at the generated images would suffice because if the model does not know a certain concept it cannot generate it. Similarly, with visual question answering models like MAGMA [12] or Flamingo [1] the attacker could simply ask the model the right questions like "Who can be seen on the photo?" or "What is the person's name?".