{'fraction_non_alphanumeric': 0.02950921236935675, 'fraction_numerical': 0.009535610386811767, 'mean_word_length': 5.0276830654327, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fingerprinting of DNN with Black-box Design and Veriﬁcation

—Most of the existing veriﬁcation techniques for model integrity belong to private veriﬁability, which only provides a veriﬁcation service to honest parties. In contrast, public veriﬁability is intended for a broader range of users (i.e., ﬁngerprinting patterns are available to all parties, including adversaries), and accounts for the possibility of untrusted service providers ( e.g. , cloud platforms) featuring in the Cloud and MLaaS services. None of the current veriﬁcation approaches are feasible for public veriﬁability due to the inability to meet the new properties such as lightweight, smooth appearance and anti-counterfeiting. In this paper, we propose PublicCheck, a practical watermarking technique for ensuring the integrity of cloud-enabled services via public veriﬁcation. The intuition behind PublicCheck is that a model’s decision boundary could serve as an inherent ﬁngerprint for watermarking. The challenge is how to capture the decision boundary with a limited number of available samples. PublicCheck achieves this by generating encysted sample ﬁngerprints, a set of smoothly transformed and augmented inputs, enclosed around the model’s decision boundary to capture the inherent model ﬁngerprints. PublicCheck assumes that the watermarking sample design should accommodate cases where the model’s knowledge is unavailable or limited for the ﬁngerprinting design, in lieu of the gradient/parameter knowledge required by the state-of-the-art white-box watermarking approaches, as well as black-box veriﬁcation through the public API. PublicCheck is implemented and extensively evaluated against various model integrity attacks and model compression attacks. We show that PublicCheck can detect model integrity breaches with 100% accuracy within ﬁve black-box model accesses. Our evaluation of the appearance of encysted sample (augmented data points permeating the decision boundary) reveals that our structural perturbation results in a smooth appearance in the pixel space. We also demonstrate the feasibility and efﬁciency of generating a plethora of encysted samples for ﬁngerprinting.


Introduction
Cloud-enabled Machine Learning as a Service (MLaaS) has shown enormous promise to transform how deep learning models are developed and deployed. Model agents, who facilitate the commercialization of machine learning based frameworks or licensed products, feature the entire pipeline from downstream end-users (clients) to upstream cloud platforms. Their role is to tout to commercialize by deploying pre-trained models that they did not necessarily develop themselves for profit to cater to broader end users' demands. One of the benefits of such a business mode is the isolation of development, deployment, and the use of models, e.g., the details of the deployment model are unnecessary to be known by model agents and clients. For example, Atrium 1 (model agent) recently created a Machine Learning Model Broker service to facilitate machine learning adoption. Model agents enable machine learning models to be trained and wrapped up for solutions customized to individual characteristics with a licensed but confidential treasure trove of training data often through two commercial frameworks. One is Google Prediction API which hides all white-box details and the other is Amazon ML which offers only a limited set of options [1]. Hence, the MLaaS business ecosystem now includes four players. Model developer (the original owner), model agent (the broker, e.g., Atrium) who facilitates model adoption without white-box access, service provider (Google/Amazon) where the model is deployed with commercial APIs with black-box query access, and model client who commercially uses the model by querying those service provider APIs.
On orthogonal path, potential risks and security threats are emerging with MLaaS, since the pre-trained models can be maliciously modified through Trojan or backdoor attacks [2]- [5], or can be degraded [6] and even backdoored [7] by model compression to save storage cost during deployment. To protect the model integrity and benefits of end-users, it is imperative for model agents, service providers, and end-users to verify whether the deployed model has been tampered with. To achieve this, verification keys, such as hash value or (watermarking sample, label) pairs, are designed to detect/verify surrogate models, derived from the source of original or licensed models (e.g., model developers or model agents). During verification, if a deployed model outputs a different label on a watermarking sample, the model will be considered modified. Most of the existing verification techniques for model integrity belong to private verifiability [8], which only provides a verification service to honest parties, such as the model developers who have white-box access to the training data and model parameters. In contrast, public verifiability is intended for a wider range of users, including model agents, service providers, and clients. The public verification releases model developers from the workload of conducting or maintaining the verification service, enabling a more flexible business mode with model agents involved. Most importantly, it also allows untrusted service providers (e.g., cloud platforms) to exist in the pipeline (see Figure 1).
To preserve the security of clients, a verification key in public scenarios should be: (i) disposable (designed to be used once per user); (ii) indistinguishable (to avoid malicious collection); and (iii) anti-counterfeiting (to protect clients from malicious service providers). Therefore, towards public verification, a feasible (watermarking) approach should satisfy the following three properties: • Lightweight. To enable large-scale verification key generation, a lightweight approach is desired to ensure the affordable computational cost to model agents. • Smooth appearance. In public verification, all users, including the adversary, have access to verification keys. A verification key should be indistinguishable from the training data to avoid being detected, reversed, and then cracked by the adversary. For image samples, one feasible approach is to ensure that the adding of patterns is smooth in pixel space so that it blends in with the original manuscript. • Anti-counterfeiting. Even if an adversary obtains a large number of verification keys, a feasible public verification design should ensure that the adversary cannot apply or fit these keys in a surrogate model. It is important to incorporate randomness and uncertainty into the process of generating verification keys toward secure and longterm verification. Furthermore, a public verification approach should accommodate a black-box setting, as white-box knowledge is not always available to model agents. This is substantiated by the fact that the licensed model obtained by the model agent is often protected or encapsulated as executable files. Additionally, model files may be encrypted by compilation, shelling, or confusion, making them difficult to decompose. Further, a model could be non-gradient-based, such as simulated annealing and evolutionary strategies, in which the gradient matrix or Hessian matrix cannot be obtained, and stochastic gradient descent (SGD) cannot be used for optimizing. In such situations, the watermarking samples can only be generated in a black-box manner (i.e., model agents only have access to a few training samples and the prediction results of a licensed model).
From a technical point of view, existing verifiability approaches could be categorized into hash-based verification [9], trigger-based watermarking [8], and adversarialbased watermarking [6]. However, none of them could fulfill the aforementioned properties and be applied or transferred to public verification. Hash-based verification is limited because adversaries or dishonest cloud providers may provide unreliable attestation results to clients, leading them to believe that the models are intact when they are not. Trigger-based watermarking creates a backdoor on the target model using a set of backdoor samples that incorporate well-designed triggers, resulting in the misclassification of target labels. The (backdoor sample, target label) pairs extend the classifier's decision boundary and are adopted as the fingerprinting of the model, as shown in Figure 2(a). The adversarial-based watermarking produces (adversarial example, target label) pairs as model fingerprinting by incorporating well-designed noise on original samples, resulting in misclassification of target labels as shown in Figure 2(b). Therefore, in trigger-based and backdoor-based watermarking approaches, model agents require white-box knowledge of the model, such as gradients/parameters. The generation of watermarking samples through model fine-tuning and backward propagation demands a significant computational resource, let alone the backdoor embedding also causes model degradation. Meanwhile, the smooth appearance of the watermarking samples (detailed in Appendix D), as well as the anti-counterfeiting property (a malicious service provider can embed the same backdoor or conduct adversarial training to bypass the verification, demonstrated in Section 4.8), are not guaranteed. We present an overview of model verification approaches in Table 1. Our solution. This research seeks the answers to the following research question: How to enable MLaaS users to publicly verify the underlying DNN model's integrity while satisfying the three aforementioned properties? To the best of our knowledge, the proposed PublicCheck is the first practical methodology for public integrity verification with only black-box access to the target MLaaS models during both watermarking design and inference. The intuition is that manipulation of the model will manifest as a shift in the decision boundary, changing the prediction of certain points in the general vicinity of the decision boundary. In view of this, the model's decision boundary could serve as an inherent fingerprint for watermarking. The challenge is how to capture the decision boundary with a limited number of available samples. A possible solution would be to augment some available data points to be dense around the decision boundary to capture the model's behavior in the vicinity of the boundary in fine granularity. Generative models are promising methods for such data augmentation, such as GANs [12], [13], VAEs [14], [15] and VQVAEs [16], [17].
We define the encysted sample as a set of augmented data points that encyst the classifier's decision boundary via a generative model, as demonstrated in Figure 2(c). A set of pairs with an encysted sample and its prediction serve as the fingerprint for the model. Our contributions. The key contributions are as follows: • We propose the first watermarking approach for public integrity verification of DNNs via encysted sample fingerprinting. • We develop a feature manipulation generative model to produce a large number of encysted samples for fingerprinting at a low cost, where data samples could be controllably manipulated along with a semantic attribute. Randomness and uncertainty are integrated into the generation via latent perturbation, while smoothness selection, a metric that correlates with human perception, is used to filter candidate samples, aiming at the smoothness requirement. The proposed approach only requires blackbox access to the MLaaS models to perform watermarking design, which can be applied to model-agnostic scenarios, with no assumptions on network architecture, hyperparameters, and training methods. • We implement and evaluate our approach against numerous integrity attacks, including model compression of different models (e.g., BadNet, TojanNN, and Clean Label attacks, and model compression of ResNet and VGG-16 models). The results show that our methodology can achieve 100% accuracy on model integrity detection, with a zero false-positive rate and low overhead (less than ten black-box model accesses). The appearance evaluation of encysted samples reveals that they are smooth for human perception. We also demonstrate the feasibility of generating a large number of encysted samples for fingerprinting, reducing the generation time from 300 seconds to 1 second compared to the existing adversarial perturbation-based approaches.

Problem Statement and Threat Model
We consider the public verification of models deployed in the cloud to provide prediction services via APIs, for any user roles, including the model agent, service provider, and any model clients. This is a typical MLaaS service provided by service providers, such as Amazon SageMaker. The process of the public verification is illustrated in Figure 1(b).

Problem Statement
The model agent U A has the licensed model f that produces a label for a given input. The verification service starts with obtaining the fingerprint of a model which is Here V S is the minimum number of samples required to conduct efficient verification, and S should be as large as possible to allow for multi-time disposable public verification. A secure server would be appropriate for maintaining PulicCheck model and storing fingerprint samples for public verification.
The licensed model is then uploaded and stored on a platform maintained by the cloud service provider U P , and then deployed to an endpoint instance to provide an API querying service for public users. Considering that the deployed model f on the cloud platform may differ from the licensed model f (which will be detailed in the threat model), it is important for the model provider U M and clients U C who purchased the API to verify whether the model f is the same as f . A set of verification keys are used to verify the integrity of the deployed model. During verification inference, the user obtains FM = {(s i , r i )} V i=1 from the secure server, where the repository of model fingerprints resides, and then queries the API of the deployed model f with watermarking samples

Threat Model
There are three main service procedures during model deployment: model transition between agents and servers, model storage in the cloud, and inquiry services via APIs (see Figure 1). We assume that the model integrity could be compromised at any procedure, and no assumptions are made regarding how the model is modified, whether by an external attack or internal manipulation. Adversary's goal. The adversary's primary goal is to derive a surrogate model that can be verified as the original licensed model provided by the model agent. The adversary may embed backdoor or adversarial misclassification behaviors in the modified model for malicious purposes. An untrusted cloud service provider may deploy degraded models with lower cost (e.g., model compression) without informing the   [2], [8] Adversarial-based watermarking [6], [10], [11] PublicCheck (Ours) : the property or requirement is not satisfied by the approach; : the property or requirement is satisfied by the approach. clients. A secondary goal is to counterfeit the verification keys or the verification process to prevent the surrogate model from being recognized by clients. For example, as shown in Figure 1(a), in private verification, only the model developer can determine whether the deployed model is intact, while the clients can be fooled by the counterfeit verification keys issued by the service provider. Adversary's knowledge. Existing DNN watermarking approaches [6], [8], [9] assume the trust service provider, which would be too strong and might even backfire when facing the untrusted cloud provider who has white-box access to all the parameters and run-time information, resulting in the feasibility to breach the watermarking patterns, such as the backdoor or adversarial misclassification behaviors. To dive into a more practical scenario, we assume that the adversaries, including the service provider, have white-box knowledge of the deployed models. We also assume that an adversary has white-box access to the fingerprint verification samples that are stored publicly in the secure server. Adversary's capacity. We summarize the adversary's capacity with respect to potential vulnerabilities and security risks in MLaaS as follows. We also present these threats in Figure 1(a) with red arrows. Note that, in private validation, the vulnerable flow covers the entire path from the original model to client verification. • Transmission vulnerability. The adversary can modify a model while it is transmitted from the model agent to the service provider by exploiting vulnerabilities in cloud network protocols or service interfaces, i.e., during upload procedure in Figure 1(b). • Storage vulnerability. Adversaries can exploit ML model storage vulnerabilities to substitute a compromised model for the safe one. • Dishonest cloud. If the cloud service provider is dishon-est, it may violate the service level agreement secretly for financial gains, such as compressing model data to save space without informing the customer or maliciously modifying the deployment model. • Counterfeited verification keys. After modifying the model, the adversary may further forge the verification key to make the client believe that the deployed model is intact, for example, by publishing a fake hash code, or embedding the watermarking sample into the modified model through the model fine-tuning or adversarial training.

PublicCheck System Design
To address the lightweight, smooth appearance, anticounterfeiting, and black-box setting in both the watermarking design and verification approach, we present a practical watermarking methodology with public verifiability based on encysted samples, named PublicCheck.
As explained previously, the intuition behind Public-Check is that a minor manipulation in the model will manifest as a shift in the decision boundary, and the prediction around the decision boundary could be easy to be flipped, as shown in Figure 2(c). A slight change in the decision boundary indicates an integrity violation in the model. The model's decision boundary could be used to generate its fingerprint for watermarking. However, it is challenging to capture the decision boundary with a limited number of available samples. To address this issue, PublicCheck first utilizes a small number of training samples as referenced samples to conduct (i) data augmentation densely around the decision boundary of the target model by using generative deep neural networks, incorporating lightweight criteria to (ii) control the augmentation; then (iii) select

Data Augmentation
We utilize attribute manipulation based generation for efficient data augmentation, whereby data samples are augmented along a semantic feature axis. Variational autoencoder (VAE)-based generative models are developed to represent samples in the high-dimensional pixel space into a low-dimensional latent space via latent codes. Combined with disentanglement, they are capable of providing such a controllable mechanism. Our objective is to develop latent representations that encode distinct attributes of the data as separate latent codes. The disentangled latent representation indicates that changes in one part of a latent code correspond to changes in one attribute of data only, while remaining relatively invariant to changes in other attributes. We intro-duce generative models with two levels of disentanglement in an unsupervised learning manner. These are attribute-level disentanglement based on low-dimensional latent representation vector and abstraction-level disentanglement based on high-dimensional latent representation.
As for relatively low-fidelity and simple images, such as handwritten digits MNIST, the VAE-based generative model can be used to learn a simple representation of data in the latent space, e.g., a 20-dimensional latent code vector. In addition, attribute-level disentanglement can be attained by forcing the distribution of latent representations to be factorial via loss terms, leading to independent distributions across dimensions. An individual latent code reveals only a particular semantic attribute, such as one latent code that controls the thickness of the digits. To better balance the trade-offs between reconstruction quality and disentanglement, we adopt the Total Correlation [14] and adopt a human perceptual evaluation Learned Perceptual Image Patch Similarity (LPIPS) [18] as reconstruction error. Details are attached in Appendix A.1. The loss of VAE incorporated with a TC term is given as follows: The first term L Rec is the reconstruction error based on LPIPS, which assesses whether the latent codes z is informative enough to reassemble the original instance. The second part is a regularization term, to push Encoder q φ (z|x) to match the prior distribution p(z), e.g., a Gaussian distribution. The third part is the TC term, to measure the dependence for multiple random variables, with υ = 40.
For high-fidelity and complicated images, such as highresolution human face images, or multi-domain images such as CIFAR-100, the dimension of the latent codes should be largely extended to maintain more features. Therefore, the disentanglement strategies used for disentangle-VAE are not applicable. Accordingly, we provide an abstractionlevel disentanglement strategy to handle high-fidelity and complicated images. We extend the scale of the latent codes and divide them into a style (high abstraction) level and a texture (low abstraction) level latent representation, via two separate encoders. To extend the latent representation, we apply the Vector Quantized strategy used in works [16], [17]. In general, VQ-based autoencoders consist of three components, encoder En, decoder De, and codebook C. The codebook C can be viewed as a common feature dictionary shared between the encoder and decoder, consisting of K categorical embedding items with D dimensions. The encoder is a non-linear mapping from the input instance x in the pixel space to the latent representation z e (x), consisting of latent embedding vectors with D dimensions. Vector Quantization is to map z e (x) to a discrete latent matrix with each element representing the index of the nearest embedding items in the codebook for each latent embedding vector of z e (x). The decoder reconstructs back to pixel space using the queried embedding items z q (z) corresponding to the discrete latent index matrix via another non-linear function, as shown in Figure 4. Details of VQ To achieve abstraction-level disentanglement, we apply two separate (encoder, codebook) pairs to model texture information and style information, respectively. A global encoder En g and codebook C g is applied to capture highabstractive information, such as style, shape, and geometry, and a local encoder En l and codebook C l is used to capture low-abstractive information, such as texture, color or background. Specifically, the local encoder initially maps the input instance into local latent representation z l e (x) using En l , followed by conducting global encoder En g to map z l e (x) into z g e (x). Then we transfer the z g e (x) and z l e (x) into a discrete latent matrix z g (x) and z l (x) via vector quantization by nearest neighbor searching on the global codebook C g and C l , respectively. z l q (x) and z g q (x) are the queried/retrieved embedding items according to the discrete latent matrix. The global decoder De g is then applied to recover the latent representation z g q (x) back to a representation s l with the same size as z l e (x). The composited z l q (x) = s l + z l q (x) are fed into the decoder De l . Finally, the local decoder De l takes as input all levels of the quantized latent representation back to the original image size. Here, we also replace the default pixel-wise reconstruction evaluation with the perceptual evaluation metric LPIPS. The generative models are assumed to be trained on the part of the training data of the target model or the similar publicavailable datasets.

Control Augmentation
We next utilize the disentangled generative model, which has been pre-trained on clean data, to model the probability distribution of encysted samples from a given image and conduct data augmentation via latent perturbation to address the Lightweight requirement. Specifically, we first transform the clean data to its latent space representation (line 5 of Algorithm 1). Our next step is to identify the latent code that controls specific attributes. Then, using black-box inference of the target model, we add perturbations to the selected latent code until a change in the prediction of the reconstructed image is observed. We define the marginal value of the encysted boundary (e.g., µ) as the minimum perturbation value in the selected latent code that caused a prediction change (lines 7).
The decision boundary of a model is its inherent fingerprinting on the training data. Therefore, we aim to conduct data augmentation around the decision boundary of the target model, through the use of generative deep neural networks on a limited number of available reference samples.
We apply an attribute manipulation-based generation to enable controllable augmentation, whereby data samples are augmented along a semantic feature direction via adding perturbation into the latent space. Based on the disentangled latent representation derived from the Generative Model with Disentanglement, encysted sample augmentation is conducted via adding certain perturbation ∆z into specific latent codes that clearly control a semantic attribute, followed by reconstructed to the pixel space via decoder De(z + ∆z). The perturbation will be added to the selected elements of the latent code vector as attribute-level disentanglement, or the entire low abstraction latent representation as abstraction-level disentanglement.
Encysted samples are defined as reconstructed copies from perturbed latent representations, where noise is from a given distribution within a given scale (e.g., normal distribution N (µ, σ)). Given a target model f (x), we define the marginal value of the encysted boundary (e.g., µ) as the minimum value that changes the prediction (i.e., argmax ∆Z f (De(Z + ∆Z)) = f (De(Z))). Scale σ defines the encysted noise range with the maximum noise scale (upper bound µ + σ) and the minimum scale (lower bound µ − σ). Encysted samples ES = {es 1 , · · · , es n } are defined as reconstructed copies using the perturbation sampled from normal distribution N (µ, σ). Within the encysted noise range, the predictions of the reconstructed encysted samples using the perturbed latent codes are easy to be flipped. An encysted sample derived from x that produces a changed prediction is referred to as the outer encysted sample, and vice versa, the inner encysted sample. Depending on the randomly sampled noise, reconstructed encysted samples are either the inner or outer samples.
To meet the lightweight criteria of fingerprinting sample selection, we develop two strategies to effectively decide the encysted noise distribution N (µ, σ) for the generation of the encysted sample, under the limitation of the knowledge about the target model to be verified. Fingerprinting design with few training samples. To address the black-box knowledge setting of a model agent in Section 2, we conduct the black-box knowledge fingerprinting design with a few training samples as follows. The model agent in this scenario has access only to a few training samples and the prediction results of a licensed model, without access to any other knowledge about the target model, such as its parameters and structure during fingerprinting design. This strategy aims to determine the outer bound of noise, namely, the minimum noise scale added to the latent representation that changes the prediction of the corresponding reconstructed image. The perturbation of the latent representation is considered as the optimization   of ∆Z using the C&W loss [19]: where Z is the latent codes for input x, the f (·) y reveals that the output of the function f is y. The minimum loss is achieved when f (De(Z + ∆Z)) y max c =y f (De(Z + ∆Z)) c , i.e., we reach the outer bound of the noise ∆Z. Thus, finding the outer bound of latent perturbation for the input data x is defined as follows: Here, ∆ max is the condition to terminate the perturbation searching.
In the BlackBox knowledge case, Equation 3 is optimized via only accessing the inputs and outputs of the function f (e.g., a classifier). Therefore, the Smooth Evolution Strategies (SES) is adopted to optimize Equation 3 via search gradients. We define the latent noise ∆Z to be from an isometric normal distribution with mean µ and standard deviation σ, denoted by N ∆Z | µ, σ 2 I . Then we use parameters ι as {µ, σ} [20]- [22]. SES defines a search distribution p(·|ι) on L (·), De(Z + ∆Z) is denoted by · to simplify notation, followed by optimization under this distribution using the following objective: (4) Gradient descent is then used to optimize the Equation 4 by figuring out the Jacobian of J(ι). Here, σ is considered as a hyperparameter, and the loss J(·) will be only optimized with respect to µ. The parameters ι will be updated via a gradient descent step with a learning rate α as follows: Finally, we generate the encysted samples by recon-structing the perturbed latent representation using the noise ∆Z sampled from N ∆Z | µ, σ 2 I .
Fingerprinting design with substitutive model training. In this scenario, the model agent is assumed to have a substitutive model with respect to the target model, even with different structures and parameters. It is practical to train such a substitutive model using similar training datasets like those used to train the target model. This strategy aims to obtain an approximation for the outer bound of noise using the given pair of substitutive model M for the target model f and its attacked version M • . Namely, the minimum noise scale added to the latent representation distinguishes the prediction of M from that of M • on the same reconstructed image. Formally, the perturbation of the latent representation is the approximation of ∆Z using loss: The minimum loss is achieved when M(De(Z + ∆Z)) y max c =y M • (De(Z + ∆Z)) c , namely, we reach one approximation of as outer bound of the noise ∆Z. Finding the outer bound of latent perturbation for the input data x is defined as: We define the latent noise ∆Z from an isometric normal distribution with mean µ and standard deviation σ, denoted by N ∆Z | µ, σ 2 I . In this case, we directly use the approximation of ∆Z x outb as the µ x for each input x, while σ is considered as a hyperparameter. Finally, we generate the encysted samples by reconstructing the perturbed latent representation using the noise ∆Z sampled from N ∆Z | µ, σ 2 I , targeting function f .

Sample Selection
We then apply the sample selection (lines 9-16 of Algorithm 1) to enhance the smoothness of the augmented encysted samples. Encysted samples for fingerprinting are perceived to be smooth to the human with the original inputs, and it is infeasible to recognize the fingerprinting patterns in the latent space from the pixel space. Therefore, we apply the filtering strategies to select suitable encysted samples for fingerprinting to meet the smoothness criteria using an adaptive threshold to build a pool of smooth samples for random selection. Finally, the model's fingerprint could be represented as a prediction vector for a small set of encysted samples for fingerprinting. During verification inference, the model is verified to be intact only when the response of the test model on the encysted sample set is equal to the fingerprint prediction vector (lines 19-20 of Algorithm 1). A visual illustration of our PublicCheck method is provided in Figure 5.
Following augmentation of the encysted samples using perturbation scales derived from the two strategies above, the next step is to use the filtering strategies to select suitable encysted samples to be fingerprinted to meet the smoothness criterion. We retain only the encysted samples with a higher similarity evaluation based on LPIPS. A fixed threshold for perceptual similarity is difficult to establish for different data types, so we propose an adaptive approach for assessing smoothness. Given a small set of reference instances (generally around 10) randomly selected for each class, denoted by RI c , the LPIPS loss metric is then applied to calculate the similarity between every two instances of the same class. We use the average similarity as the adaptive threshold ξ c of each class for the filtering. During the filtering, given a reconstructed encysted sample candidate es i , the adaptive threshold ξ c and reference instance set RI c according to its predicted class via f (es i ) = c are used to decide the filtering results. Specifically, we calculate the LPIPS similarity between the es i and each sample in RI c . The encysted sample es i will be selected as fingerprinting sample only if all distances are smaller than the threshold ξ c . Formally, the encysted sample for fingerprinting is satisfied only when LP IP S(es i , ri) ≤ ξ c , ∀ri ∈ RI c .

Evaluation of PublicCheck
We evaluate the performance of PublicCheck on three datasets: MNIST [23], CIFAR-10 and CIFAR-100 [24], and FFHQ [25], [26], while the target classifiers are LeNet (5 layers), ResNet-18 [27], and VGG-16 [28], respectively, as they have different complexity. Extended details of the datasets and classification models used in our experiments, as well as the accuracy of the original, clean models, are provided in Appendix B. We randomly sample 20% of each dataset to use for training the generative model. The VAE-based generative model (attribute-level disentanglement) is used for MNIST, and VQ-based generative model (abstraction-level disentanglement) is used for CIFAR and FFHQ. Detailed architecture and hyperparameters for the generative models are given in Appendix C. Only a small set of instances (around 20) are sampled from the remaining 80% of the dataset, to use as the reference samples for augmentation, watermarking design, and evaluation. The 80% data split is also used to train the substitutive model in Section 3.2.

Evaluation Goals and Metrics
Our first experimental goal is to determine the accuracy of PublicCheck for integrity breach detection. We test the accuracy under various attacks, as well as under model compression. In general, the less information we can derive from the underlying target model, the more difficult it is to verify its integrity. Therefore, we assume the worst-case scenario of API access, where only the Top-1 classification label (i.e., the most likely label) can be obtained.
The model fingerprint, denoted by (ES, Y ), is defined to be the Top-1 classification prediction of the original model on the fingerprinting set of encysted samples ES. The fingerprint is created by the model agent. Given a set of fingerprinting pairs {(es 1 , y 1 ), (es 2 , y 2 ), · · · , (es M , y M )} ⊂ (ES, Y ) and the predictionsŶ = {ŷ 1 ,ŷ 2 , · · · ,ŷ M } of these given encysted samples by the black-box API model, we define an integrity breach to be successfully detected if: ∃ŷ i ∈Ŷ such thatŷ i = y i . We report the detection rate under both few sample and substitutive model knowledge cases in Section 4.3. We investigate the performance of our PublicCheck under more restricted scenarios in Section 4.4, in which the availability of reference samples is limited. The overheads associated with designing and implementing fingerprinting samples are in Section 4.5. We evaluate the performance of our method when varying the hyperparameters, such as noise scale, in Section 4.6. The smoothness of the encysted sample for fingerprinting is evaluated in Section 4.7. Finally, we evaluate the performance of our watermarking approach under an adaptive attack in Section 4.8. We compare our approach to the baseline (randomly selected original reference samples) and the adversarial based approaches (Sensitive Sample [6] as a typical work) for the private verification of model integrity. We allow white-box knowledge on target models for Sensitive Sample design.

Experimental Setups
We mimic two adversarial settings of the integrity breach in the experiments. The first is backdoor attacks that compromised the model during model transition or storage in the cloud. In this setting, we assume that the target model is known and under the attacker's control. The second scenario we investigate is dishonest cloud vendors that conduct model compression to save storage space.
(i) For the backdoor attacks, we use the TrojanZoo [29] platform to deploy the BadNet [30], TrojanNN [5], and CleanLabel [4] attacks on the image classification models, which are typical backdoor attacks that compromise the training data to change the model's behavior at test time.  We select these attacks as they have diverse complexities, e.g., the LeNet on MNIST has the simplest structure with 5 layers and the ResNet18 on CIFAR-10 is more complex with 18 layers. We assume the attacker attempts to maximize the degradation of prediction accuracy, as well as the possibility of misclassifying a sample as a target class. A backdoored model typically reacts to trigger-embedded inputs with such malicious functions, but otherwise behaves normally. In our implementation, the attack success rate of each of these 3 adversarial models was more than 95%. The default settings of these attacks are given in Appendix E.
(ii) To mimic a dishonest cloud vendor, we apply weight pruning to compress the models on three datasets. Pruning starts by learning the connectivity via regular network training. Next, all connections with weights below a threshold are removed, followed by retraining the network to learn the final weights for the remaining sparse connections. The network is pruned by retaining only important connections, with between 15% of weights removed for the LeNet model on the MNIST dataset, 20% removed for the ResNet model on the CIFAR-10 dataset, and around 24% removed for the VGG16 model on the FFHQ dataset. Pruning resulted in only a minor change in the accuracy of each model (< 2%). Note that, in the real-world, a dishonest cloud provider may compress the models with more weights reduced to lower down the storage cost. Here we try to illustrate the performance of PublicCheck in a worst case of model compression detection (i.e., least changes in models).

Evaluation on Integrity Breach Detection
Evaluating the black-box setting of fingerprinting design with few training samples. Under this setting, the design of the encysted sample for fingerprinting is only based on the encysted noise that is restricted by µ and userspecific scale σ described in Section 3.2. In our experiment, we set the default perturbation scale as σ = 0.05. Results of the detection success rate against three attacks are reported in Figure 6. As shown, the detection rate rises when increasing the number of encysted samples used for verification. Even when only 2 encysted samples were used for verification, the detection rate of integrity breaches was above 93% for MNIST, 97.9% for CIFAR-10, and almost 100.0% for FFHQ.
Our results suggest that the detection rate increases when the target model grows in size and complexity. The reason is that the complex model has a more complicated decision boundary, and our method has more sources to design the fingerprinting. With 5 encysted samples for fingerprinting, we achieve 100% detection accuracy for all the datasets and attacks we tested. Furthermore, the overhead of model integrity validation is minimal, with only 5 API queries.
To evaluate the effectiveness of our watermarking approach on a dataset with a larger number of classes, we also applied an evaluation on CIFAR-100. As shown in Figure 7(a), using only 1 encysted sample for fingerprinting, the detection rate of an integrity breach was above 88% for CIFAR-100 for all three attacks, as opposed to 76% for CIFAR-10. Using 2 encysted samples increased the detection accuracy for CIFAR-100 to 98%, and using 4 samples resulted in 100% accuracy. As a result of the more complex decision hyperplane, the detection rate increases when the classification task has more classes.
Besides the above three adversarial attacks, we also evaluate the breach detection accuracy under model compression, as shown in Figure 7(b). Using only 2 encysted samples for fingerprinting, the integrity breach detection rate for the compressed models was around 97% for MNIST, 98.0% for CIFAR-10 and FFHQ, and 98.6% for CIFAR-100. We again observe that the detection rate increases with model complexity. Using 5 encysted samples to fingerprint the model, we achieved 100% detection accuracy for all datasets we tested against model compression, demonstrating the minimal overhead associated with verification. For the dataset with larger number of classes, e.g., CIFAR-100, using only 1 encysted sample for fingerprinting, the detection rate of the integrity breach could reach more than 90% under model compression, compared to 57% for CIFAR-10. 4 encysted samples of CIFAR-100 are able to achieve 100% detection accuracy. This also confirms the effectiveness of our watermarking approach when applied to a dataset with more classes. Comparison with baseline. When using randomly selected samples instead of our approach to verify three modified models attacked by TrojanNN, BadNet, and CleanLabel, we found the detection rates are around 10-20%, which are even lower on compressed models (between 10-15%). Comparison with SOTA. In this section we compare our approach to the Sensitive Sample [6] as the state of the art (SOTA) for the private verification of model integrity. For detection of adversarial attacks, the performance of our PublicCheck under the few sample knowledge setting is similar to that of Sensitive Sample under the white-box knowledge setting. For model compression, the change of parameter values under model compression is intuitively more significant than the change under attacks, since a large fraction of the parameters are removed, with values reduced to zero instead of being just slightly modified. Therefore, one would anticipate that the detection rate for model compression is higher than for adversarial attacks, which is demonstrated by our above results.
We next compare the performance of PublicCheck and sensitive samples [6] under the same model compression settings on the CIFAR-10 classifier. Using only two samples, PublicCheck achieves a 98% detection rate for CIFAR-10, compared to around 84.4% for sensitive samples, and below 10% for the randomly selected training samples. Five samples are sufficient to achieve 100% detection for our approach. In contrast, 10 samples are not enough to achieve 100% detection rate for the sensitive sample approach.
It is interesting to note that the sensitive samples approach performs much worse on model compression than on attacks. This has demonstrated the major limitation of the sensitive samples approach under the public verification of the model integrity scenario. The reason is due to the design of the adversarial perturbation used to reveal the changes in parameters of the target model. Namely, because it is based on the activation or gradient of parameter updates during training. At the same time, the model compression approaches such as pruning also utilize similar signals to remove parameters with lower activation, resulting in degradation of detection of the model change. Due to this, the gradient-based perturbation for fingerprinting was not able to detect the model changes caused by attacks or compression approaches that utilize similar gradient information.
These results indicate that our PublicCheck can achieve state-of-the-art performance for model integrity verification against diverse attacks and model compression under the few sample knowledge settings. The advantage of our approach is that a high detection accuracy can be achieved without requiring any knowledge of model parameters or structure and with a lower cost associated with designing fingerprint samples-only forward procedures are required.

4.3.2.
Evaluating the black-box setting of fingerprinting design with substitutive model training. In this section, we evaluate PublickCheck when more information about the model is available. Under this setting, the design of the encysted sample for fingerprinting is based on the substitutive model of the target model and its attacked versions (Section 3.2). Figure 8 shows that, as in the few sample setting, the success detection rate rises when increasing the number of encysted samples for fingerprinting. When using only one encysted sample for fingerprinting, the breach detection rate against adversarial attacks is higher in the substitutive model setting than in the few sample setting, by 22% for MNIST, and roughly 11% for CIFAR-10. When using two or more encysted samples, the substitutive model setting continues to outperform the few sample settings by around 1% until both approaches achieve 100% detection accuracy when using five encysted samples for fingerprinting. This was observed for all datasets and attacks tested. Additionally, we also evaluate the successful detection rate of model integrity breaches caused by model compression under the substitutive model setting. Using only one encysted sample for fingerprinting, the detection rate of the integrity breach increased by 15% to 20% in the substitutive model setting compared to the few sample knowledge setting.
Our results indicate that our PublicCheck can perform better against diverse attacks and model compression under substitutive model settings compared to the few sample knowledge settings, randomly selected training samples, and sensitive samples, particularly in the model compression case. The results demonstrate that the detection rate increases with more information about the target model, when substitute models are available. As five encysted samples for fingerprinting can obtain the 100% accuracy of model integrity verification for both the few sample and the substitutive model knowledge settings, the former seems to be the more reasonable choice due to no assumption on the model itself. These results also demonstrate the advantage of our PublicCheck in terms of practicality and performance under under limited knowledge.

Performance under Restricted Setups
We examine the performance under more restrictive circumstances. Our default assumption for the few sample knowledge settings (used in the previous section) is that the set of reference samples from the training data that are used for creating the encysted samples contain most of the classes in the dataset. However, in practice, the model agent may not have access to training samples from all classes.
Here we examine this more limited setting in which a small set of reference samples covering only a few categories is used. This is a more practical application scenario. In the following, we report our investigation on the impact of the limited categories on integrity violation detection accuracy. We compare the performance of PublicCheck in detecting attacked models on MNIST and CIFAR-10 when encysted samples are generated by using only a subset of classes, namely, by randomly sampling 1, 2, 3, or 5 of the classes, instead of using all ten classes in each dataset.
Next, we evaluate whether the model integrity breach detection rate could be 100% within the limited number of API queries. Results are averaged over these two datasets and all 3 TrojanZoo attacks. We report the results in Table 2. In general, we observe an increase in accuracy as the number of classes increases. Although the detection rate of our PublicCheck under 1, 2, 3, or 5 classes restriction has a slight drop compared to all-classes available scenario, it is demonstrated that even when watermark samples are created for one class only, PublicCheck still achieves 100% accuracy when using seven watermark samples for verification. It shows that five samples are enough to achieve 100% detection accuracy using PublicCheck, assuming the reference samples used to generate encysted samples for fingerprinting consist of five randomly selected classes. Furthermore, the variance in detection accuracy decreases as the number of encysted samples increases. Increasing the number of different classes used in the encysted samples had little impact on the variance. It demonstrates our approach's generalization ability, relying on the well-encysted and finegranularity measure of the decision boundary using the attributed augmentation in the latent space. Specifically, due to the structural perturbation of PublicCheck, each encysted sample is generated from a randomly sampled perturbation, which lies very close to a random part of the decision boundary of the classification model. The prediction on the encysted sample for fingerprinting is used to distinguish whether the boundary of the verified model has the right disturbed in that localized region. Every additional watermarking sample then may test a different localized region of the decision boundary, thereby increasing the detection ability of PublicCheck.

Performance across Various Noise Scales
To conduct the augmentation, we add a perturbation to the latent code that controls specific attributes. Generally, the generation of encysted samples aims to produce inner and outer encysted samples as close to the decision boundary as possible, via reconstructing the selected latent code after adding randomly sampled noise from the given distribution N (µ, σ) under the few sample knowledge settings.
As the noise scale σ should be decided in advance as a hyperparameter, we now investigate "How scaling the noise impacts the accuracy". Intuitively, a smaller noise scale, σ, could result in a better performance of the fingerprinting samples, since more minor noise leads to inner and outer samples being closer to the decision boundary. We conduct experiments to evaluate the detection rate of our method when varying the noise scale σ. The results are reported in Table 3, confirming the intuition above that a smaller noise scale brings a higher detection rate. Even under the broader noise scale, i.e., 0.1 or 0.5, six or nine encysted samples for fingerprinting are sufficient to achieve 100% detection accuracy, demonstrating the effectiveness of our method with the flexible settings. Furthermore, the variances of detection accuracy against different attacks decrease as the number of encysted samples increases. Furthermore, the variance of detection accuracy are also reduced as the scale of noise decreases. Generally, there is a trade-off between accuracy and overheads of computation. If smaller steps are taken, it takes more iterations to find the marginal value of the encysted boundary (e.g., µ). Accordingly, a broader noise scale could improve the performance of a fast and flexible fingerprinting design, which is suitable for resourceconstrained devices. We demonstrate this argument in the next section with respect to the runtime overheads.

Overheads for Fingerprints Design
We examine the run time as an overhead measure to demonstrate the practicality of PublicCheck. We mimic the end-users as a laptop with GPU resources (NVIDIA Quadro RTX 4000 8GB and i7 9900 16G CPU), and the other has CPU resources only (i5 8265U 8GB CPU). Two types of consumed time will be examined. One is the generation of one individual encysted sample, and the other is the generation and selection of one smooth encysted sample (which we refer to as the generation of one smooth encysted sample). The run time to generate a single encysted sample for fingerprinting for three given datasets under various noise scale settings is reported in Table 4. Given the onceoff trained generator, the encysted noise range, and the default noise level of 0.05 used in our experiments, the generation time of each encysted augmentation is less than 2s for all testing datasets, including the high-fidelity images such as FFHQ images. For MNIST and CIFAR datasets, the time are 0.33s and 0.83s for each encysted augmentation, respectively. We also report the generation time for each augmentation of high-fidelity FFHQ images for CPU-Only end user, which is around 15s per augmentation.
To ensure the smooth appearance requirement under public verification, a further filtering procedure is conducted on the set of augmented encysted samples. The overhead for the filtering procedure is minimal, less than 0.2s for selecting one smooth encysted sample for fingerprinting. The total time for the combination of augmentation and filtering of one encysted sample for fingerprinting varies from roughly 2 seconds for MNIST, CIFAR-10, and CIFAR-100, to roughly 9 seconds for FFHQ, under 0.05 noise level for GPU end users. For CPU-only users, runtime is similar to GPU users' for simple cases like MINIST and CIFAR-10 but can be longer for tasks involving larger class numbers or high fidelity inputs. A possible solution for CPU users is to perform parallel processing for LPIPS evaluation across classes and the generation of encysted samples. Once watermarking samples have been generated, the time required to verify a deployed model is less than 0.2 s per sample. Therefore, we can expect model verification to take less than one second when we have five watermarking samples. Thus, our solution can be deployed in real-time applications.
Comparison with SOTA. We select the Sensitive Sample [6] as the SOTA for the private verification of model integrity. By comparison, we also tested the time to generate sensitive samples on the same dataset, taking human face images as an example under the GPU settings. It took 2964.5s to generate 100 samples, and then an additional 725 seconds to select the best 10 examples from those 100 according to its Maximum Active-Neuron Cover Sample Selection, giving a total time of 3689.5 seconds to generate 10 samples. Using a filtering ratio of selecting the 10 best samples from a pool of 100 generated samples, the time to generate a single watermark sample is roughly 369 seconds, compared to around 1s for our PublicCheck. Reducing this filtering procedure would reduce the generation time at the cost of degradation of the performance for the selected sensitive samples. However, since the time to generate one candidate of the sensitive sample (with no filtering) is still of the order of 29.6 seconds, revealing that the generation of PublicCheck is still a lot faster.

Evaluation of Smoothness
We demonstrate the smoothness of the generated encysted sample for fingerprinting in this section. We first visualize the produced encysted samples for fingerprinting in Figure 12. As shown, there are few artifacts introduced in the encysted samples compared to the original reference samples. Generally, the generated samples' small texture or color temperature is modified via manipulating the corresponding latent codes. Thus, from human perception, these generated encysted samples are smooth. These results confirm the advantages of our method under the public verification scenarios. In addition to this, we also report the quantitative evaluation of the smoothness of encysted samples for fingerprinting, using the LPIPS metric in Figure 9. The average LPIPS values of the encysted samples for MNIST and CIFAR-10 among classes are approximately 0.2, and 0.45 and 0.55 for CIFAR-100 and FFHQ, respectively. The smoothness of encysted samples for fingerprinting is obvious and consistent among different classes for all these three datasets (FFHQ is a male and female class), demonstrating that they are smooth in pixel space. Comparison with SOTA. For the randomly augmented encysted samples, around 25% of them satisfy the smoothness evaluation in terms of the adaptive threshold (0.57). In contrast, we demonstrate that only 1% of sensitive sam-ples meet the adaptive smoothness threshold (LPIPS=0.65). Considering the time cost, generating one smooth sensitive sample takes more than 3600s (1h), which is infeasible in practice. Our PublicCheck only takes 9 seconds to generate one smooth encysted sample for fingerprinting. Besides, the visualization of the successful sensitive samples is given in Appendix D, which does not seem smooth for human perception despite having a small pixel-wise error level.

Evaluation of Adaptive Attacks
Since encysted samples are visible to the public, including adversaries, we also evaluate the performance of our watermarking approach when faced with adaptive attacks. This setting assumes that an extreme adversary can collect a large number of publicly available encysted samples for all classes and then use these samples to prune/compress a model or introduce backdoors, such as the untrusted cloud service provider. Adaptive attacks are similar to adversarial training based defense strategies. Specifically, for a given dataset (CIFAR-10 or MNIST), N encysted samples (N = 10, 50, 1000, 2000) for each class, derived from our watermarking approach for the target model TM, are publicly released. And then, the adaptive adversaries combine these watermarking samples into the training data in order to construct the compressed model TM . Next, our watermarking approach is used to generate new encysted samples to distinguish TM from TM. We demonstrate that the detection accuracy for TM is not affected after applying adversarial training with different amounts of publicly released encysted samples. The robustness is based on uncertainty and randomness in the generation of encysted samples. The released augmented encysted samples could be regarded as randomly sampled points distributed around uncertain areas of the decision boundary of the target model, which is not effective for changing the decision boundary during adversarial training. We also test the performance of the Sensitive Sample with the same settings. The results indicate that the detection accuracy sharply decreases when the network is trained with sensitive samples along with their clean pairs. Based on adversarial training with N = 1000, the attacked model could bypass more than 80% of sensitive samples, resulting in a similar performance to the randomly selected training samples. In other words, adversarial training has successfully learned the distribution of fingerprinting patterns created by Sensitive Samples.

Related Work
Model poisoning attacks. This stream of works focuses on violating DNN model integrity by inserting backdoors or trojans into the model itself. Gu et al. [30] proposed BadNet, a poisoning attack that generates a poisoned model by retraining the original one with a poisoned training dataset. The attacked model's behavior is similar to the clean one except when the backdoor trigger is encountered within the input data. Worse, their backdoor remains effective even after the transfer learning. Liu et al. [5] further enhanced this attack by manipulating only a subset of weights to insert a backdoor. Chen et al. [3] introduced a similar attack where the attacker does not need to have access to the model. Instead, the attacker re-engineers the model from scratch and trains it with a poisoned dataset. This stream of work is very promising in a black-box setup to determine if the incoming input is benign or adversarial. However, they cannot determine if the integrity of the DNN model itself is maintained or violated by poisoning attacks. Watermarking. It can be categorized in terms of the purpose into (i) protecting the Intellectual Property (IP) of DNN models, and (ii) protecting the integrity of the DNN models, which is where our work belongs. (i) The former focuses on claiming the ownership of the models by developing a persistent watermark, but not protecting the model integrity in the sense of the question -'has the model been changed?' In fact, the aim is the opposite; that is, to build a robust watermark that cannot be removed easily by model thieves. Examples include: embedding a small number of bits into deep layers by Uchide et al. [31], [32], infusing a verifiable watermark in black-box settings by Zang et al. [33], inserting a backdoor to claim the ownership by Adi et al. [8]. (ii) The latter aims at protecting the integrity of the DNN models while being used. There are few works introduced so far. Abuadbba et al. [34] proposed a fragile watermark based mechanism that requires white-box access during verification. Another option is to design adversarial perturbation for a DNN as its fingerprinting patterns [6], [10], [11]. Utilizing adversarial examples has the primary advantage of eliminating the need for training or re-training and enabling the black-box inference capability. The major shortcomings of this approach are the computation cost of the backward propagation and the requirement of whitebox knowledge of gradient information. Furthermore, these approaches only allow verification by an honest party-they are only privately verifiable due to their technical limitations of lacking randomness/uncertainty, infinite generation, or imperceptibility. On the contrary, the MLaaS scenario has a large number of users, which necessitates the need for public verifiability with those characteristics. To the best of our knowledge, this is the first work to promote the public verification of model integrity towards practice by leveraging the advantages of a lightweight, smoothness, and anti-counterfeiting properties while only using black-box access during both inference and design procedures.

Conclusion
We propose a practical DNN watermarking scheme Pub-licCheck, for ensuring the integrity of cloud-enabled models with public verifiability with limited knowledge for design. We show that PublicCheck can withstand model integrity attacks and compression. We also show PublicCheck can achieve three important properties that (i) [Lightweight] only five randomly selected encysted watermarking samples suffice to achieve a 100% verification and zero false-positive rate, without any extra coverage-guarantee mechanism; (ii) [Smoothness] our structured perturbation in the latent space has reasonable controllability through the generative model and disentanglement strategies; (iii) [Anti-counterfeiting] linear perturbations in the latent space will result in nonlinear structural change in the pixel space. These augmented encysted samples could be scattered over the vicinity of the decision boundary of the target model in an uncertain manner and multi-class reference samples for random augmentation will further help prevent blind spots on the decision boundary. We hope that PublicCheck developed in this paper could be amenable to future-generation verification in deep neural networks.
cally, the first term is the gradient of the reconstruction error Dist(x − De(z q )), which will be back-propagated through the decoder, and to the encoder using the straight-through gradient estimator. Here, we replace the default pixel-wise reconstruction evaluation with the perceptual evaluation metric Learned Perceptual Image Patch Similarity (LPIPS) [18]. Besides, two additional terms are applied in the loss function to align the codebook's embedding items with the encoder's output. The codebook loss, which only applies to the codebook variables, brings the selected codebook close to the output of the encoder. The commitment loss, which only applies to the encoder weights, encourages the output of the encoder to stay close to the chosen codebook vector to prevent it from fluctuating too frequently from one code vector to another. The exponential moving average updates are used to update the codebook, as a replacement for the codebook loss (the second loss term in Eq. 10 ): Here, n (t) i is the number of vectors in E(x) in the minibatch quantized to codebook item C i at timestamp t, and γ is a decay parameter with a value between 0 and 1 (default 0.99).
Although the training time of VQ generative model takes hours for 256x images, that is a once-off procedure on the server-side that could be carried out in parallel across several GPUs or by using the pre-trained model for transfer learning to further reduce the training time. Besides, the generative could be reused for the similar datasets, e.g. the model pretrained on the FFHQ face dataset could be used for other human face datasets.

Appendix B. Datasets and Settings
Here, we provide more details about the three datasets and three DNN model architectures we use. (1) MNIST consists of 28×28 grayscale handwritten digit images from 10 classes, i.e., digit 0-9, and has a training set of 55000 instances and a test set of 10000 instances. (2) The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class, including airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. (3) To evaluate the effectiveness of our watermarking approach on a dataset of a larger class number, we also used the CIFAR-100 dataset. This dataset contains 100 classes containing 600 images each. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. The images are resized to 64px in our experiments. (4) Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, consisting of 70,000 high-quality 1024×1024 resolution images from Flickr and associated with considerable variation in terms of age, ethnicity, and image background. The images are resized to 256px in our experiments. The target classifier for MNIST using the LeNet, consisting of three sets of convolution layers with a combination of average pooling, followed by two fully connected layers. At last, a Softmax function which classifies the images into respective class. For the CIFAR-10 data, the target classifier is adopted the ResNet-18 [27], consisting of 18 convolutional layers utilizing skip connections or shortcuts to jump over some layers to address vanishing gradients. For the FFHQ data, the target classifier is used the VGG-16 [28], consisting of 16 convolutional and fully connected layers mostly have 3×3 filters. Table 5 shows the classification models used in our experiments for each of these datasets, and the corresponding model accuracy.
Our DNN techniques are implemented using PyTorch, backdoor attacks are deployed in the TrojanZoo [29] platform, and the PublicCheck in Python. The number of instances that are randomly sampled as reference samples is generally 3-4 times of required number of the encysted sample for fingerprinting. For example, if four encysted samples for fingerprinting are desired, accordingly, we randomly select 20 reference samples to conduct augmentation, followed by randomly selecting 4 augmented encysted samples that satisfied the smooth appearance. Each entry in the results of performance for watermarking is averaged over 1000 repetitions of augmentation and selection.

Appendix C. Architectures and Hyperparameters
The VAE-based generative model (attribute-level disentanglement) is used for MNIST, and the architecture and hyperparameters are similar to the settings in [14]. The encoder consists of four strided convolutional layers with steps of 2 and a window size of 4 * 4, followed by two fully connected layers with 128,2*10 hidden units respectively. In the decoder, there are one fully connected layers (128 hidden units), followed by four transposed convolutions with stride 2 and window size 4 × 4. The VQ-based generative model(abstraction-level disentanglement) is used for CIFAR and FFHQ. As the CIFAR-10 images are with low fidelity, we combine the top and bottom encoder into one, i.e. the ordinary VQ-based autoencoder in Appendix A.2 is used. The architecture and hyperparameters of the ordinary VQbased autoencoder are similar to the settings in [16] and that of two-encoder VQ-based generative model (abstractionlevel disentanglement) for FFHQ are similar to the settings in [17]. The encoder consists of two strided convolutional layers with steps of 2 and a window size of 4 * 4, followed by two residual 3 * 3 blocks (implemented as ReLU, 3x3, ReLU, 1x1), all of which have 128 hidden units. In the decoder, there are two residual 3 × 3 blocks, followed by two transposed convolutions with stride 2 and window size 4 × 4. Default hyperparameters are summarized in the following table. Appendix D.
Visualization of Sensitive Samples, Triggered Images of Backdoor Attacks, and Our Encysted Samples for Fingerprinting A fingerprinting instance is generally created from an original example by adding a trigger or pixel perturbation in existing works. In this section, we demonstrate that such pixel-level fingerprinting patterns can be distinguished by human perception in Figures 10 and 11. Figure 10: Visualization of Sensitive Samples [6] for private verification of model integrity (Second row). First row is the corresponding original images. Figure 11: Visualization of triggers (Trojan Square and Trojan Watermark) used in Backdoor attacks [5] Appendix E.

Relevant DNN Attacks and Settings
While promising, the MLaaS model (and outsourced models, in general) poses critical security concerns, specifically relating to the integrity of the served model by the cloud provider at the time of queries. Here are a few examples of those potential attacks: Model Compression attack. The attacker's (cloud provider's) aim is to compress the DNN model with negligible accuracy reduction, to minimize required cloud storage for profit. Examples of (a) Encysted samples for MNIST on the latent code that control the weight of digit (attribute-level disentanglement) (b) Encysted samples for FFHQ on the texture latent cods Figure 12: Visual Examples of few Encysted samples for fingerprinting on MNIST and FFHQ datasets (abstractionlevel disentanglement). The first row is original reference samples, second and last row is the inner (same prediction with the reference) and outer (different prediction) encysted samples, respectively, as close to the decision boundary.
compression techniques that may achieve this are quantization [36], pruning [37], low precision [38] and architecture optimization [39]. In the later one, the service provider may use single-layer instead of a multi-layer neural network, to minimize its costs [40]. Neural network trojan attack the cloud provider could be compromised by malware that injects a trojan into the model so that the model misclassifies samples containing a specific trigger [5], [30]. Targeted poisoning attack It is an attack in which the attacker retrains the model by using at least one sample and corresponding label (not reflecting the ground truth) [41] to force the model to misclassify. Adversarial Samples: Another active related stream of work worth to be discussed is adversarial samples [42]. The focus of this stream is not poisoning the model itself but rather manipulating the input data in such a way that lead clean DNN models to misclassification at the testing time. Attacks: Fast gradient sign method by Goodfellow et al. [43], basic iterative method by Kurakin et al. [44], DeepFool by Moosavi et al. [45], Jacobianbased Saliency map method by Papernot et al. [46], and DeepXplore method by Pei et al. [47] are a few examples of these attacks.
Settings for implemented attacks. We summarize the default setting of the backdoor attacks used in our evaluation in Table 7. In this section, we explore a potential management solution for encysted sample images based on blockchain and the InterPlanetary File System (IPFS). Storage step. Initially, the model owner registers basic information on the resource storage page, including the resource model ID, the name of the owner to which it belongs, the time, etc. Then, the PublicCheck module is called for encysted samples for fingerprinting using the black-box access to the model. The generated images are stored in IPFS, which is a protocol and peer-to-peer network for storing and sharing data in a distributed file system [48]. IPFS uses content-addressable block storage to uniquely identify each file within a global namespace connecting all computing devices. IPFS first hashes the produced images and uses the hash values (e.g., SHA256, RIPEMD160) as the file's unique identification and address in the IPFS network. Once the file is stored in the IPFS network, the server only needs to store the file's address in IPFS, which greatly reduces the storage overhead. Besides, since the address is found through the file's hash, which will be changed when the file's contents are maliciously tampered with. At the same time, these basic information provided by the owner, responded IPFS address, and timestamps, will be processed as fixed-length hex strings. These hex strings are then saved to the blockchain by using the smart contract. Acquisition step. Following the querying and browsing of the basic information about the resource, the end user may request information associated with API purchases and model deployments as identity information. The management system then invokes the associated smart contracts, reads the hash address in the basic information about files, and downloads the resource files stored in IPFS, without providing a user download interface. In addition, the user's identity information is extracted and translated into a digital fingerprint code, which is then embedded in the resource file to create a resource file containing fingerprint information. After embedding, the public key in the fingerprint certificate is used to encrypt the resource file, and a download interface is provided to the user. The user downloads the resource file through the interface, and decrypts the resource file with the private key to obtain the digital resource embedded with fingerprint information. As encysted sample images could be once-off, an inventory threshold will be set to trigger a new round of generation.