{'fraction_non_alphanumeric': 0.03168671348119308, 'fraction_numerical': 0.01115900716123201, 'mean_word_length': 4.428665785997358, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explaining Latent Representations with a Corpus of Examples

Modern machine learning models are complicated. Most of them rely on convoluted latent representations of their input to issue a prediction. To achieve greater transparency than a black-box that connects inputs to predictions, it is necessary to gain a deeper understanding of these latent representations. To that aim, we propose SimplEx: a user-centred method that provides example-based explanations with reference to a freely selected set of examples, called the corpus. SimplEx uses the corpus to improve the user's understanding of the latent space with post-hoc explanations answering two questions: (1) Which corpus examples explain the prediction issued for a given test example? (2) What features of these corpus examples are relevant for the model to relate them to the test example? SimplEx provides an answer by reconstructing the test latent representation as a mixture of corpus latent representations. Further, we propose a novel approach, the Integrated Jacobian, that allows SimplEx to make explicit the contribution of each corpus feature in the mixture. Through experiments on tasks ranging from mortality prediction to image classification, we demonstrate that these decompositions are robust and accurate. With illustrative use cases in medicine, we show that SimplEx empowers the user by highlighting relevant patterns in the corpus that explain model representations. Moreover, we demonstrate how the freedom in choosing the corpus allows the user to have personalized explanations in terms of examples that are meaningful for them.


Introduction and related work
How can we make a machine learning model convincing? If accuracy is undoubtedly necessary, it is rarely sufficient. As these models are used in critical areas such as medicine, finance and the criminal justice system, their black-box nature appears as a major issue [1,2,3]. With the necessity to address this problem, the landscape of explainable artificial intelligence (XAI) developed [4,5]. A first approach in XAI is to focus on white-box models that are interpretable by design. However, restricting to a class of inherently interpretable models often comes at the cost of lower prediction accuracy [6]. In this work, we rather focus on post-hoc explainability techniques. These methods aim at improving the interpretability of black-box models by complementing their predictions with various kinds of explanations. In this way, it is possible to understand the prediction of a model without sacrificing its prediction accuracy.
Feature importance explanations are undoubtedly the most widespread type of post-hoc explanations. Popular feature importance methods include SHAP [7,8,9], LIME [10], Integrated Gradients [11], Contrastive Examples [12] and Masks [13,14,15]. These methods complement the model prediction for an input example with a score attributed to each input feature. This score reflects the importance of each feature for the model to issue its prediction. Knowing which features are important for a model prediction certainly provides more information on the model than the prediction by itself. However, these methods do not provide a reason as to why the model pays attention to these particular features.
Another approach is to contextualize each model prediction with the help of relevant examples. In fact, recent works [16] have demonstrated that human subjects often find example-based explanations more insightful than feature importance explanations. Complementing the model's predictions with relevant examples previously seen by the model is commonly known as Case-Based Reasoning (CBR) [17,18,19]. The implementations of CBR generally involve models that create a synthetic representation of the dataset, where examples with similar patterns are summarized by prototypes [20,21,22]. At inference time, these models relate new examples to one or several prototypes to issue a prediction. In this way, the patterns that are used by the model to issue a prediction are made explicit with the help of relevant prototypes. A limitation of this approach is the restricted model architecture. The aforementioned procedure requires to opt for a family of models that rely on prototypes to issue a prediction. This family of model might not always be the most suitable for the task at hand. This motivates the development of generic post-hoc methods that make few or no assumption on the model.
The most common approach to provide example-based explanations for a wide variety of models mirrors feature importance methods. The idea is to complement the model prediction by attributing a score to each training example. This score reflects the importance of each training example for the model to issue its prediction. This score will typically be computed by simulating the effect of removing each training instance from the training set on the learned model [23]. Popular examples of such methods include Influence Functions [24] and Data-Shapley [25,26]. These methods offer the advantage of being flexible enough to be used with a wide variety of models. They produce scores that describe what the model could have predicted if some examples were absent from the training set. This is very interesting in a data valuation perspective. However, in an explanation perspective, it is not clear how to reconstruct the model predictions with these importance scores.
So far, we have only discussed works that provide explanations of a model output, which is the tip of the iceberg. Modern machine learning models involve many convoluted transformations to deduce the output from an input. These transformations are expressed in terms of intermediate variables that are often called latent variables. Some treatment of these latent variables is necessary if we want to provide explanations that take the model complexity into account. This motivates several works that push the explainability task beyond the realm of model outputs. Among the most noticeable contributions in this endeavour, we cite Concept Activation Vectors that create a dictionary between human friendly concepts (such as the presence of stripes in an image) and their representation in terms of latent vectors [27]. Another interesting contribution is the Deep k-Nearest Neighbors model that contextualizes the prediction for an example with its Nearest Neighbours in the space of latent variables, the latent space [28]. An alternative exploration of the latent space is offered by the representer theorem that allows, under restrictive assumptions, to use latent vectors to decompose a model's prediction in terms of its training examples [29]. In each case, SimplEx highlights the role played by each feature of each corpus example in the latent space decomposition. SimplEx centralizes many functionalities that, to the best of our knowledge, constitute a leap forward from the previous state of the art. (1) SimplEx gives the user freedom to choose the corpus of examples whom with the model's predictions are decomposed. Unlike previous methods such as the representer theorem, there is no need for this corpus of examples to be equal to the model's training set. This is particularly interesting for two reasons: (a) the training set of a model is not always accessible (b) the user might want explanations in terms of examples that make sense for them. For instance, a doctor might want to understand the predictions of a risk model in terms of patients they know. (2) The decompositions of SimplEx are valid, both in latent and output space. We show that, in both cases, the corpus mixtures discovered by SimplEx offer significantly more precision and robustness than previous methods such as Deep k-Nearest Neighbors and the representer theorem. (3) SimplEx details the role played by each feature in the corpus mixture. This is done by introducing Integrated Jacobians, a generalization of Integrated Gradients that makes the contribution of each corpus feature explicit in the latent space decomposition. This creates a bridge between two research directions that have mostly developed independently: feature importance and example-based explanations [19,30]. In Section C of the supplementary material, we report a user-study involving 10 clinicians. This study supports the significance of our contribution.

SimplEx
In this section, we formulate our method rigorously. Our purpose is to explain the black-box prediction for an unseen test example with the help of a set of known examples that we call the corpus. We start with a clear statement of the family of black-boxes for which our method applies. Then, we detail how the set of corpus examples can be used to decompose a black-box representation for the unseen example. Finally, we show that the corpus decomposition can offer explanations at the feature level.

Preliminaries
Let X ⊆ R d X be an input (or feature) space and Y ⊆ R d Y be an output (or label) space, where d X and d Y are respectively the dimension of the input and the output space. Our task is to explain individual predictions of a given black-box f : X → Y. In order to build our explainability method, we need to make an assumption on the family of black-boxes that we wish to interpret. Assumption 2.1 (Black-box Restriction). We restrict to black-boxes f : X → Y that can be decomposed as f = l • g, where g : X → H maps an input x ∈ X to a latent vector h = g (x) ∈ H and l : H → Y linearly maps 1 a latent vector h ∈ H to an output y = l(h) = Ah ∈ Y. In the following, we call H ⊆ R d H the latent space. Typically, this space has higher dimension than the output space d H > d Y . Remark 2.1. In the context of deep-learning, this assumption requires that the last hidden layer maps linearly to the output. While it is often the case, it is crucial in the following since we will use the fact that linear combinations in latent space correspond to linear combinations in output space. Our purpose is to gain insights on the structure of the latent space. Remark 2.2. This assumption is compatible with regression and classification models, we just need to clarify what we mean by output in the case of classification. If f is a classification black-box that predicts the probabilities for each class, it will typically take the form in Assumption 2.1 up to a normalizing map φ (typically a softmax): f = φ • l • g. In this case, we ignore 2 the normalizing map φ and define the output to be y = (l • g) (x).
Our explanations for f rely on a set of examples that we call the corpus. These examples will typically (but not necessarily) be a representative subset of the black-box training set. The corpus set has to be understood as a set of reference examples that we want to use as building blocks to interpret unseen examples. In order to index these examples, it will be useful to denote by [n 1 : n 2 ] the set of natural numbers between the natural numbers n 1 and n 2 with n 1 < n 2 . Further, we denote [n] = [1 : n] the set of natural numbers between 1 and n ≥ 1. The corpus of examples is a set C = {x c | c ∈ [C]} containing C ∈ N * examples x c ∈ X . In the following, superscripts are labels for examples and subscripts are labels for vector components. In this way, x c i has to be understood as the component i of corpus example c.

A corpus of examples to explain a latent representation
Our purpose is to understand a prediction f(x) for an unseen test example x with the help of the corpus. How can we decompose the prediction f(x) in terms of corpus predictions f(x c )? A naive attempt would be to express x as a mixture of inputs from the corpus C: x = C c=1 w c x c with weights w c ∈ [0, 1] that sum to one C c=1 w c = 1. The weakness of this approach is that the signification of the mixture weights is not conserved if the black-box f is not a linear map: Fortunately, Assumption 2.1 offers us a better vector space to perform a corpus decomposition of the unseen example x. We first note that the map g induces a latent representation of the corpus Similarly, x has a latent representation h = g(x) ∈ H. Following the above line of reasoning, we could therefore perform a corpus decomposition in latent space h = C c=1 w c h c . Now, by using the linearity of l, we can compute the black-box output of this mixture in latent space: l( In this case, the weights that are used to decompose the latent representation h in terms of the latent representation of the corpus g(C) also reflect the way in which the black-box prediction f(x) can be decomposed in terms of the corpus outputs f(C). This hints that the latent space H is endowed with the appropriate geometry to make corpus decompositions. More formally, we think in terms of the convex hull spanned by the corpus. Definition 2.1 (Corpus Hull). The corpus convex hull spanned by a corpus C with latent representation 3. This is the set of latent vectors that are a mixture of the corpus latent vectors.
At this stage, it is important to notice that an exact corpus decomposition is not possible if h ∈ CH(C). In such a case, the best we can do is to find the elementĥ ∈ CH(C) that best approximates h. If H is endowed with a norm · H , this corresponds to the convex optimization problem h = arg miñ By definition, the corpus representationĥ of h can be expanded 3 as a mixture of elements from g(C):ĥ = C c=1 w c h c . The weight can naturally be interpreted as a measure of importance in the reconstruction of h with the corpus. Clearly, w c ≈ 0 for some c ∈ [C] indicates that h c does not play a significant role in the corpus representationĥ of h. On the other hand, w c ≈ 1 indicates that h c generates the corpus representationĥ by itself.
At this stage, a natural question arises: how can we know if the corpus approximationĥ is a good approximation for h? The answer is given by the residual vector h −ĥ that measures the shift between the latent representation h = g(x) and the corpus hull CH(C). It is natural to use this residual vector to detect examples that cannot be explained with the selected corpus of examples C. Definition 2.2 (Corpus Residual). The corpus residual associated to a latent vector h ∈ H and its corpus representationĥ ∈ CH(C) solving (1) is the quantity Figure 2: Corpus convex hull and residual.
In Section A.1 of the supplementary material, we show that the corpus residual also controls the quality of the corpus approximation in output space Y. All the corpus-related quantities that we have introduced so far are summarized visually in Figure 2. Note that this Figure is a simplification of the reality as C will typically be larger than 3 and d X , d H will typically be higher than 2. We are now endowed with a rigorous way to decompose a test example in terms of corpus examples in latent space. In the next section, we detail how to pull-back this decomposition to input space.

Transferring the corpus explanation in input space
Now that we are endowed with a corpus decompositionĥ = C c=1 w c h c that approximates h, it would be convenient to have an understanding of the corpus decomposition in input space X . For the sake of notation, we will assume that the corpus approximation is good so that it is unnecessary to draw a distinction between the latent representation h of the unseen example x and its corpus decompositionĥ. If we want to understand the corpus decomposition in input space, a natural approach [11] is to fix a baseline input x 0 together with its latent representation h 0 = g(x 0 ). Let us now decompose the representation shift h − h 0 in terms of the corpus: With this decomposition, we understand the total shift in latent space h − h 0 in terms of individual contributions from each corpus member. In the following, we focus on the comparison between the baseline and a single corpus example x c together with its latent representation h c by keeping in mind that the full decomposition (2) can be reconstructed with the whole corpus. To bring the discussion in input space X , we interpret the shift in latent space h c −h 0 as resulting from a shift x c −x 0 in the input space. We are interested in the contribution of each feature to the latent space shift. To decompose the shift in latent space in terms of the features, we parametrize the shift in input space with a line γ c : [0, 1] → X that goes from the baseline to the corpus example: Together with the black-box, this line induces a curve in latent space g • γ c : [0, 1] → H that goes from the baseline latent representation h 0 to the corpus example latent representation h c . Let us now use an infinitesimal decomposition of this curve to make the contribution of each input feature explicit. If we assume that g is differentiable at γ c (t), we can use a first order approximation of the curve at the vicinity of t ∈ (0, 1) to decompose the infinitesimal shift in latent space: where we used γ c i (t) = x 0 i + t · (x c i − x 0 i ) to obtain the second equality. In this decomposition, each input feature contributes additively to the infinitesimal shift in latent space. It follows trivially that the contribution of the input feature corresponding to input dimension i ∈ [d X ] is given by In order to compute the overall contribution of feature i to the shift, we let δt → 0 and we sum the infinitesimal contributions along the line γ c . If we assume 4 that g is almost everywhere differentiable, this sum converges to an integral in the limit δt → 0 . This motivates the following definitions. Definition 2.3 (Integrated Jacobian & Projection). The integrated Jacobian between a baseline (x 0 , h 0 = g(x 0 )) and a corpus example 1]. This vector indicates the shift in latent space induced by feature i of corpus example c when comparing the corpus example with the baseline. To summarize this contribution to the shift h − h 0 described in (2), we define the projected Jacobian where ·,· is an inner product for H and the normalization is chosen for the purpose of Proposition 2.1. Remark 2.4. The integrated Jacobian can be seen as a latent-space generalization of Integrated Gradients [11]. In Section A.3 of the supplementary material, we establish the relationship between the two quantities: IG c i = l(j c i ).
x c Figure 3: Integrated Jacobian and projection.
We summarize the Jacobian quantities in Figure 3. By inspecting the figure, we notice that projected Jacobians encode the contribution of feature i from corpus example c to the overall shift in latent space: p c i > 0 implies that this feature creates a shift pointing in the same direction as the overall shift; p c i < 0 implies that this feature creates a shift pointing in the opposite direction and p c i = 0 implies that this feature creates a shift in an orthogonal direction. We use the projections to summarize the contribution of each feature in Figures 1 , 8 & 9. The colors blue and red indicate respectively a positive and negative projection. In addition to these geometrical insights, Jacobian quantities come with natural properties.
Proposition 2.1 (Properties of Integrated Jacobians). Consider a baseline (x 0 , h 0 = g(x 0 )) and a test example together with their latent representation (x, h = g(x)) ∈ X × H. If the shift h − h 0 admits a decomposition (2), the following properties hold.
Proof. The proof is provided in Section A.4 of the supplementary material.
These properties show that the integrated Jacobians and their projections are the quantities that we are looking for: they transfer the corpus explanation into input space. The first equality decomposes the shift in latent space in terms of contributions w c j c i arising from each feature of each corpus example. The second equality sets a natural scale to the contribution of each feature. For this reason, it is natural to use w c p c i to measure the contribution of feature i of corpus example c.

Experiments
In this section, we evaluate quantitatively several aspects of our method. In a first experiment, we verify that the corpus decomposition scheme described in Section 2 yields good approximations for the latent representation of test examples extracted from the same dataset as the corpus examples. In a realistic clinical use case, we illustrate the usage of SimplEx in a set-up where different corpora reflecting different datasets are used. The experiments are summarized below. In Section B of the supplementary material, we provide more details and further experiments with time series and synthetic data. The code for our method and experiments is available on the Github repository https://github.com/JonathanCrabbe/Simplex. All the experiments have been replicated on different machines.

Precision of corpus decomposition
Description Metrics We are interested in measuring the precision of the corpus approximation in latent space and in output space. To that aim, we use the R 2 score in both spaces. In this way, R 2 H measures the precision of the corpus approximationĥ with respect to the true latent representation h. Similarly, R 2 Y measures the precision of the corpus approximationŷ = l(ĥ) with respect to the true output y = l(h). Both of these metrics satisfy −∞ < R 2 ≤ 1. A higher R 2 score is better with R 2 = 1 Y score for the output approximation Baselines We compare our method 5 (SimplEx) with 3 baselines. A first approach, inspired by [28], consists in using the K-nearest corpus neighbours in latent space to build the latent approximation h. Building on this idea, we introduce two baselines (1) KNN Uniform that takes the average latent representation of the K-nearest corpus neighbours of h in latent space (2) KNN Distance that computes the same average with weights w c inversely proportional to the distance h − h c H . Finally, we use the representer theorem [29] to produce an approximationŷ of y with the corpus C. Unlike the other methods, the representer theorem does not allow to produce an approximation in latent space.
Datasets We use two different datasets with distinct tasks for our experiment: (1) 240,486 patients enrolled in the American SEER program [31]. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. We train a multilayer perceptron (MLP) for this task. Since this task is simple, we show that a corpus of C = 100 patients yields good approximations.
(2) 70,000 MNIST images of handwritten digits [32]. We consider the multiclass classification task of identifying the digit represented on each image. We train a convolutional neural network (CNN) for the image classification. This classification task is more complex than the previous one (higher d X and d Y ), we show that a corpus of C = 1, 000 images yields good approximations in this case.

Results
The results for SimplEx and the KNN baselines are presented in Figure 4 & 5. Several things can be deduced from these results: (1) It is generally harder to produce a good approximation in latent space than in output space as R 2 H < R 2 Y for most examples (2) SimplEx produces the most accurate approximations, both in latent and output space. These approximations are of high quality with R 2 ≈ 1. (3) The trends are qualitatively different between SimplEx and the other baselines. The accuracy of SimplEx increases with K and stabilizes when a small number of corpus members contribute (K = 5 in both cases). The accuracy of the KNN baselines increases with K, reaches a maximum for a small K and steadily decreases for larger K. This can be understood easily: when K increases beyond the number of relevant corpus examples, irrelevant examples will be added in the decomposition. SimplEx will typically annihilate the effect of these irrelevant examples by setting their weights w c to zero in the corpus decomposition. The KNN baselines include the irrelevant corpus members in the decomposition, which alters the quality of the approximation. This suggests that K has to be tuned for each example with KNN baselines, while the optimal number of corpus examples to contribute is learned by SimplEx. (4) The standard deviations indicate that the performances of SimplEx are more consistent across different runs. This is particularly true in the prostate cancer experiment, where the corpus size C is smaller. This suggests that SimplEx is more robust than the baselines. (5) For the representer theorem, we have R 2 Y = −(6.6 ± 6.1) · 10 7 for the prostate cancer dataset and R 2 Y = −(7.2 ± 6.6) for MNIST. This corresponds to poor estimations of the black-box output. We propose some hypotheses to explain this observation in Section B.1 of the supplementary material.

Significance of Jacobian Projections
Description The purpose of this experiment is to check if SimplEx's Jacobian Projections are a good measure of the importance for each corpus feature in constructing the test latent representation h. In the same setting as in the previous experiment, we start with a corpus C of C = 500 MNIST images. We build a corpus approximation for an example x ∈ X with latent representation h = g(x) ∈ H. The precision of this approximation is reflected by its corpus residual r C (h). For each corpus example x c ∈ C, we would like to identify the features that are the most important in constructing the corpus decomposition of h. With SimplEx, this is reflected by the Jacobian Projections p c i . We evaluate these scores for each feature i ∈ [d X ] of each corpus example c ∈ [C]. For each corpus image x c ∈ C, we select the n most important pixels according to the Jacobian Projections and the baseline. In each case, we build a mask m c that replaces these n most important pixels by black pixels. This yields a corrupted corpus image x c cor = m c x c , where denotes the Hadamard product. By corrupting all the corpus images, we obtain a corrupted corpus C cor . We analyse how well this corrupted corpus approximates h, this yields a residual r Ccor (h).
Metric We are interested in measuring the effectiveness of the corpus corruption. This is reflected by the metric δ cor (h) = r Ccor (h) − r C (h) . A higher value for this metric indicates that the features selected by the saliency method are more important for the corpus to produce a good approximation of h in latent space. We repeat this experiment for 100 test examples and for different numbers n of perturbed pixels.
Baseline As a baseline for our experiment, we use Integrated Gradients, which is close in spirit to our method. In a similar fashion, we compute the Integrated Gradients IG c i for each feature i ∈ [d X ] of each corpus example c ∈ [C] and construct a corrupted corpus based on these scores.

Results
The results are presented in the form of box plots in Figure 6. We observe that the corruptions induced by the Jacobian Projections are significantly more impactful when few pixels are perturbed. The two methods become equivalent when more pixels are perturbed. This demonstrates that Jacobian Projections are more suitable to measure the importance of features when performing a latent space reconstruction, as it is the case for SimplEx.

Use case: clinical risk model across countries
Very often, clinical risk models are produced and validated with the data of patients treated at a single site [33]. This can cause problems when these models are deployed at different sites for two reasons: (1) Patients from different sites can have different characteristics (2) Rules that are learned for one site might not be true for another site. One possible way to alleviate this problem would be to detect patients for which the model prediction is highly extrapolated and/or ambiguous. In this way, doctors from different sites can make an enlightened use of the risk model rather than blindly believing the model's predictions. We demonstrate that SimplEx provides a natural framework for this set-up.
As in the previous experiment, we consider a dataset D USA containing patients enrolled in the American SEER program [31]. We train and validate an MLP risk model with D USA . To give a realistic realization of the above use-case, we assume that we want to deploy this risk model in a different site: the United Kingdom. For this purpose, we extract D UK from the set of 10,086 patients enrolled in the British Prostate Cancer UK program [34]. These patients are characterized by the  Figure 8, both the American and the British doctors make the same observation: the model relates the test patient to corpus patients that are mostly misclassified by the model. With the help of SimplEx, both doctors will rightfully be sceptical with respect to the model's prediction.
In Figure 9, something even more interesting occurs: the two corpus decompositions suggest different conclusions. In the American doctor's perspective, the prediction for this patient appears perfectly coherent as all patients in the corpus decomposition have very similar features and all of them are rightfully classified. On the other hand, the British doctor will reach the opposite conclusion as the most relevant corpus patient is misclassified by the model. In this case, we have a perfect illustration of the limitation of the transfer of a risk model from one site (America) to another (United Kingdom): similar patients from different sites can have different outcomes. In both cases, since the test patient is British, only the decomposition in terms of C UK really matters. In both cases, the British doctor could have anticipated the misclassification of each patient with SimplEx.

Discussion
We have introduced SimplEx, a method that decomposes the model representations at inference time in terms of a corpus. Through several experiments, we have demonstrated that these decompositions are accurate and can easily be personalized to the user. Finally, by introducing Integrated Jacobians, we have brought these explanations to the feature level.
We believe that our bridge between feature and example-based explainability opens up many avenues for the future. A first interesting extension would be to investigate how SimplEx can be used to understand latent representations involved in unsupervised learning. For instance, SimplEx could be used to study the interpretability of self-expressive latent representations learned by autoencoders [35]. A second interesting possibility would be to design a rigorous scheme to select the optimal corpus for a given model and dataset. Finally, a formulation where we allow the corpus to vary on the basis of observations would be particularly interesting for online learning.

A Supplement for Mathematical Formulation
In this supplementary section, we give more details on the mathematical aspects underlying SimplEx.

A.1 Precision of the corpus approximation in output space
If the corpus representation of h ∈ H has a residual r C (h), Assumption 2.1 controls the error between the black-box prediction for the test example f(x) = l(h) and and its corpus representation l(ĥ). Proposition A.1 (Precision in output space). Consider a latent representation h with corpus residual r C (h). If Assumption 2.1 holds, this implies that the corpus prediction l(ĥ) approximates l(h) with a precision controlled by the corpus residual: where · Y is a norm on Y and l op = inf λ ∈ R + : l(h) Y ≤ λ h H ∀h ∈ H is the usual operator norm.
Proof. The proof is immediate: where we have successively used the linearity of l, the definition of the operator norm · op and Definition 2.2.

A.2 Uniqueness of corpus decomposition
As we have mentioned in the main paper, the corpus decomposition is not always unique. To illustrate, we consider the following corpus representation: Consider the following vector in the corpus hull: h = 0.75 · h 1 + 0.25 · h 2 . We note that this vector can also be written as h = 0.5 · h 1 + 0.5 · h 3 . In other words, the vector h ∈ g(C) admits more than one corpus decomposition. This is not a surprise for the attentive reader: by paying a closer look to g(C), we note that h 3 is somewhat redundant as it is itself a combination of h 1 and h 2 . The multiplicity of the corpus decomposition results from a redundancy in the corpus representation.
To make this reasoning more general, we need to revisit some classic concepts of convex analysis. To establish a sufficient condition that guarantees the uniqueness of corpus decompositions, we recall the definition of affine independence.
If a set of vectors is not affinely independent, it means that one of the vectors can be written as an affine combination of the others. This is precisely what we called a redundancy in the previous paragraph. We now adapt a well-known result of convex analysis to our formalism: Proposition A.2 (Uniqueness of corpus decomposition). If the corpus representation g(C) = {h c | c ∈ [C]} is a set of affinely independent vectors, then every vector in the corpus hull h ∈ CH(C) admits one unique corpus decomposition.
Proof. The existence of a decomposition is a trivial consequence of the definition of CH(C). We prove the uniqueness of the decomposition by contradiction. Let us assume that a vector h ∈ CH(C) admits two distinct corpus decompositions: is not affinely independent, a contradiction.
This shows that affine independence provides a sufficient condition to ensure the uniqueness of corpus decompositions. If one wants to produce such a corpus, a possibility is to gradually add new examples in the corpus by checking that the latent representation of each new example is not an affine combination of the previous latent representations. Clearly, the number of examples in such a corpus cannot exceed d H + 1.

A.3 Integrated Jacobian and Integrated Gradients
Integrated Gradients is a notorious method used to discuss feature saliency [11]. It uses a black-box output to attribute a saliency score to each feature. In the original paper, the output space Y is assumed to be one-dimensional: d Y = 1. We shall therefore relax the bold notation that we have used for the outputs so far. In this way, the black-box is denoted f and the latent-to-output map is denoted l. Although the original paper makes no mention of corpus decompositions, it is straightforward to adapt the definition of Integrated Gradients to our set-up: Definition A.2 (Integrated Gradient). The Integrated Gradient between a baseline x 0 an a corpus example x c ∈ X associated to feature In the main paper, we have introduced Integrated Jacobians: a latent space generalization of Integrated Gradients. We use the word generalization for a reason: the Integrated Gradient can be deduced from the Integrated Jacobian but not the opposite 6 . We make the relationship between the two quantities explicit in the following proposition. Proposition A.3. The Integrated Gradient can be deduced from the Integrated Jacobian via Proof. We start from the definition of the Integrated Gradient: where we have successively used: Assumption 2.1, the linearity of the partial derivative, the linearity of the integration operator, the linearity of l and the definition of Integrated Jacobians.
Note that Integrated Jacobians allow us to push our understanding of the black-box beyond the output. There is very little reason to expect a one dimensional output to capture the model complexity. As we have argued in the introduction, our paper pursues the more challenging ambition of gaining a deeper understanding of the black-box latent space.

A.4 Properties of Integrated Jacobians
We give a proof for the proposition appearing in the main paper.
Proposition 2.1 (Properties of Integrated Jacobians). Consider a baseline (x 0 , h 0 = g(x 0 )) and a test example together with their latent representation (x, h = g(x)) ∈ X × H. If the shift h − h 0 admits a decomposition (2), the following properties hold.
Proof. Let us begin by proving (A). By using the chain rule for a given corpus example c ∈ [C], we write explicitly the derivative of the curve g • γ c with respect to its parameter t ∈ (0, 1): where we used γ c i (t) = x 0 i + t · (x c i − x 0 i ) to obtain the second equality. We use this equation to rewrite the sum of the Integrated Jacobians for this corpus example c: where we have successively used: the linearity of integration, the explicit expression for the curve derivative, the fundamental theorem of calculus and the definition of the curve g • γ c . We are now ready to derive (A): where we have successively used the exact expression for the sum of Integrated Jacobians associated to corpus example c and the definition of the corpus decomposition of h. We are done with (A), let us now prove (B). We simply project both members of (A) on the overall shift h − h 0 . Projecting the left-hand side of (A) yields: where we used the linearity of the projection operator. Projecting the right-hand side of (A) yields: By equating the projected version of both members of (A), we deduce (B).

A.5 Pseudocode for SimplEx
We give the pseudocode for the two modules underlying SimplEx: the corpus decomposition (Algorithm 1) and the evaluation of projected Jacobians (Algorithm 2). Evaluate loss: Update pre-weights:w ← Adam step (L [w]); end Return normalized weights: w ← softmax [w]; Return corpus residual: Where we used a vector notation for the pre-weights and the weights: w = (w c ) C c=1 . For the Adam optimizer, we use the default hyperparameters in the Pytorch implementation: lr = 10 −3 ; β 1 = .9 ; β 2 = .999 ; eps = 10 −8 . Note that this algorithm is a standard optimization loop for a convex problem where the normalization of the weights is ensured by using a softmax.
When the size of the corpus elements to contribute has to be limited to K, we use a similar strategy as the one used to produce extremal perturbations [14,15]. This consists in adding the following L 1 term to the optimized loss L: where vecsort is a permutation operator that sorts the components of a vector in ascending order. The notation vecsort d refers to the d th component of the sorted vector. This regularization term will impose sparsity for the C − K smallest weights of the corpus decomposition. As a result, the optimal corpus decomposition only involves K non-vanishing weights. We now focus on the evaluation of the Projected Jacobian.
Set the evaluation input: end Apply the appropriate pre-factor: This algorithm approximates the integral involved in the definition of the Projected Jacobian with a standard Riemann sum. Note that the definition of X 0 implies that the baseline vector x 0 is broadcasted along the first dimension of the matrix. More explicitly, the components of this matrix are X 0 . Also note that the projected Jacobians can be computed in parallel with packages such as Pytorch's autograd. We have used the notation to denote the conventional Hadarmard product. In our implementation, the number of bins N b is fixed at 200, bigger N b don't significantly improve the precision of the Riemann sum.

A.6 Choice of a baseline for Integrated Jacobians
Throughout our analysis of the corpus decomposition, we have assumed the existence of a baseline x 0 ∈ X . This baseline is crucial as it defines the starting point of the line γ c that we use to compute the Jacobian quantities. What is a good choice for the baseline? The answer to this question depends on the domain. When this makes sense, we choose the baseline to be an instance that does not contain any information. A good example of this is the baseline that we use for MNIST: an image that is completely black x 0 = 0. Sometimes, this absence of information is not well-defined. A good example is the prostate cancer experiment: it makes little sense to define a patient whose features contain no information. In this set-up, our baseline is a patient whose features are fixed to their average value in the training 7 set x 0 = |D train | −1 x∈Dtrain x. In this way, a shift with respect to the baseline corresponds to a patient whose features differ from the population average.

B Supplement for Experiments
In this section, we give more details on the experiments that we have conducted with SimplEx. All our experiments have been performed on a machine with Intel(R) Core(TM) i5-8600K CPU @ 3.60GHz [6 cores] and Nvidia GeForce RTX 2080 Ti GPU. Our implementation is done with Pytorch 1.8.1.

B.1 Details for the corpus precision experiment
Metrics We give the explicit expression for the two metrics used in the experiment. By keeping the notation of Section 3, we assume that we are given a set of test samples T ⊂ X . For each test representation h ∈ g(T ) and test output y ∈ f(T ), we build build an approximationĥ andŷ with  several methods. To evaluate the quality of these approximations, we use the following R 2 scores: These R 2 scores compare the approximation method to a dummy approximator that approximates every representation and output with their test average. A negative value for R 2 indicates that the approximation method performs more poorly than the dummy approximator. Ideally, the R 2 score should be close to 1.

SEER Dataset
The SEER dataset is a private dataset consisting in 240,486 patients enrolled in the American SEER program [31]. All the patients from the SEER dataset have been de-identified. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. Each patient in the dataset is represented by the couple (x, z), where x contains the patient features and z ∈ {0, 1} 2 is a vector indicating the patient mortality. The features characterizing each patient are their age, PSA, Gleason score, clinical stage and which, if any, treatment they are receiving. These features are summarized in Table 1. The original dataset is severely imbalanced as 93.8% patients survive or have a mortality unrelated to cancer. We extract a balanced subset of 42,000 patients that we split into a training set D train of 35,700 patients and a test set D test of 6,300 patients. We train a multilayer perceptron (MLP) for the mortality prediction task on D train .
MNIST Dataset MNIST is a public dataset consisting in 70,000 MNIST images of handwritten digits [32]. We consider the multiclass classification task of identifying the digit represented on each image. Each instance in the dataset is represented by the couple (x, z), where x contains the image itself and z ∈ {0, 1} 10 is a vector indicating the true label for the image. The images are characterized by 28 × 28 pixels with one channel. The dataset is conventionally split into a training set D train of 60,000 images and a test set D test of 10,000 images. We train a convolutional neural network (CNN) for the image classification task on D train . Yann LeCun and Corinna Cortes hold the copyright of MNIST dataset, which is a derivative work from original NIST datasets. MNIST dataset is made available under the terms of the Creative Commons Attribution-Share Alike 3.0 license.
Prostate Cancer Model The model that we use for mortality prediction on the SEER dataset is a simple MLP with two hidden layers. Its precise architecture is described in Table 2. The model is trained by minimizing the following loss: where we have introduced a L 2 regularization, as required by the representer theorem. The regularization coefficient is chosen to be λ = 10 −5 (a bigger λ significantly decreases the performance of the model on the testing set). We train this model with Adam (default Pytorch hyperparameters) for 5 epochs. Across the different runs, the accuracy of the resulting model on the test set ranges between 85 − 86%. Batch Norm  3  3  Only acts on Age, PSA  and Comorbidities  Dense 1  26  200  ReLU  Dropout  200  200  Dense 2  200  50  ReLU  Dropout  50  50 Output: h = g(x) Linear 50 2 Output: y = l(h) Softmax 2 2

Layer Input Dimension Output Dimension Activation Remark
Output: p = f(x)

MNIST Model
The model that we use for image classification on the MNIST dataset is the CNN represented in Figure 10. Its precise architecture is described in Table 3. The model is trained by minimizing the loss (3) with λ = 10 −1 . We train this model with Adam (default Pytorch hyperparameters) for 10 epochs. Across the different runs, the accuracy of the resulting model on the test set ranges between 94 − 96% (note that the weight decay decreases the performances, training the same model with λ = 0 yields a test accuracy above 99%).
Representer theorem Previous works established that the pre-activation output of classification deep-networks can be decomposed in terms of contributions arising from the training set [29]. In our   set-up, where the neural network takes the form f = φ • l • g, the decomposition can be written as where λ is the L 2 regularization coefficient used in the training loss (3). In our work, we decompose the same output in terms of a corpus C that is distinct from the training set D train . In this experiment, the corpus is a random subset of the training set C ⊂ D train (in our implementation of the representer theorem, the true labels associated to the corpus example are included). To give a corpus approximation of the output with the representer theorem, we restrict the above sum to the corpus: The R 2 Y score reported in the main paper measure the quality of this approximation. It turns out that R 2 Y < 0 in both experiments, which indicates that the representer theorem offers poor approximations. We have two explanations: (1) As previously mentioned, the representer theorem assumes that the decomposition involves the whole training set. By making a decomposition that involves a subset C of the training set, we violate a first assumption of the representer theorem. (2) The representer theorem assumes that the trained model f θ * corresponds to a stationnary point of the loss: ∇ θ L train | θ * = 0. This assumption is rarely verified in non-convex optimization problems such as the optimization of deep networks.
Plots with an alternative metric In Figures 11 & 12, we report the norm of the error associated to each method as a function of the number of active corpus members K. With this metric, SimplEx remains the most interesting approximation method in latent and output space. Note that for SimplEx, when K = C, the error in latent space is equivalent to the corpus residual r C (h).
On the measure of consistency In our experiments, we use the standard deviation of each metric across different runs to study the consistency of the corpus approximations. Table 4 details the parts of the experiment that are modified from one run to another.

B.2 Details for the clinical use case
Description In this subsection, we detail the detection of British patients that is described in Section 3.3 of the main paper.   CUTRACT Dataset The CUTRACT dataset is a private dataset consisting in 10,086 patients enrolled in the British Prostate Cancer UK program [34]. All the patients from the CUTRACT dataset have been de-identified. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. Each patient in the dataset is represented by the couple (x, z), where x contains the patient features and z ∈ {0, 1} 2 is a vector indicating the patient mortality. The features characterizing the patient are the same as for the SEER dataset. These features are summarized in Table 5. Once again, the full dataset is unbalanced, we then choose D UK as a balanced subset of 2,000 patients. This dataset is private.

Metrics
Model We use the same mortality predictor as in the previous experiment (see Table 2). The only difference is that no weight decay is included in the optimization (λ = 0).
Note on the prostate cancer datasets In the medical literature on prostate cancer [42], the grade of a patient can be deduced from the Gleason scores in the following way: We noted that this relationship between the grade and the Gleason score was not always verified among the patients in our two prostate cancer datasets. After discussing with our curator, we understood that the data of some patients was collected by using a different convention. Further, some of the data was missing and has been imputed in a way that does not respect the above rule. Clearly, those details are irrelevant if we use this data to train a model to illustrate the functionalities of SimplEx as it is done in our paper. Nonetheless, it should be stressed that this inconsistency with the medical literature implies that Model As in the previous experiment, we train the CNN from Table 3. In contrast with the previous experiment, we set the weight decay to zero: λ = 0. The resulting model has more than 99% accuracy on the test set.
Results Results are shown in Figure 13. This suggests that the difference between the two dataset is encoded in their latent representation. SimplEx and the baselines offer similar performances in this case.

B.4 Experiments with synthetic time series
In this section, we describe some further experiments we have performed with synthetic time series. The two parts of this subsection mirror the experiments that we have performed in Section 3 of the main paper. Metrics We use the same metrics as in Section 3.1 of the main paper. We run the experiment 5 times to report standard deviations across different runs.
Baselines We use the same baselines as in Section 3.1 of the main paper.
Data generation We generate data from the following AR(2) generating process.
where ϕ 1 = .7, ϕ 2 = .25 and t ∼ N (0, 0.1). The initial condition for the time series are sampled independently: x 1 , x 2 ∼ N (0, 1). Each instance in the dataset D consists in a couple (x, y) ∈ D of sequences x = (x t ) T t=1 and y = (y t ) T t=1 . For each time step, the target indicates the value of the time series at the next step: y t = x t+1 for all t ∈ [1 : T ]. We generate 10,000 such instances that we split into 9,000 training instances D train and 1,000 testing instances D test .
Model We train a two layer LSTM to forecast the next value of the time series at each time step. The precise model architecture is described in   Figure 14: Precision of corpus decomposition for AR (avg ± std). loss: where x 1:t ≡ (x t ) T t=1 . We train this model with Adam (default Pytorch hyperparameters) for 20 epochs. Across the different runs, the average RMSE of the resulting model on testing data is always 0.1. This corresponds to ideal performances for a deterministic model due to the noise term t in the AR model.

Results
The results of this experiment are shown in Figure 14 & 15. As in Section 3.1 of the main paper, SimplEx offers significantly better and more consistent results across different runs.

B.4.2 Detection of oscillating time series
Description This experiment mirrors the EMNIST detection experiment from Section B.3. We train a LSTM with a training set extracted from the previous AR dataset D train ⊂ D. Next, we sample a corpus C ⊂ D train of size C = 1, 000. We are now interested in investigating if the  Metrics We use the same metrics as in Section B.2. We run the experiment 5 times to report standard deviations across different runs.
Baselines We use the same baselines as in Section B.2.
Data generation We generate D as in the previous experiment. The time series inD are sampled from the following AR(2) process: where ϕ 1 , ϕ 2 and t are defined as in (4). The initial condition for the time series are sampled independently:x 1 ,x 2 ∼ N (0, 1). The only difference between D andD lies in the extra minus sign from (5) compared to (4). This gives an extra oscillating behaviour to the time series fromD. We generate 1,000 such instances that we use for testing purpose.
Model We use the same LSTM as in the previous experiment.

Results
The results of this experiment are shown in Figure 16. As in Section 3.3 of the main paper, the difference between D andD is imprinted in their latent representation. Once again, SimplEx offers the best detection scheme.

B.5 A comparison with Influence Functions
Description We note that there is no standard way to reconstruct the explicit black-box output f(x) with the influence scores [24] for an input x ∈ X ⊂ R d X . In contrast, SimplEx allows to explicitly decompose a black-box prediction in terms of contributions arising from each corpus example: f(x) = C c=1 w c l(h c ). An interesting question to ask is the following: can we interpret influence scores as reconstruction weights in latent space? To explore this question, we propose the following procedure. First, we compute the influence score i c ∈ R for the prediction f(x) and for each corpus example x c ∈ C. Then, we extract the helpful examples from the corpus: C help = {x c ∈ C | i c > 0}. In the same spirit as in Section 3.1 of the main paper, we select the K most helpful examples from C help . We denote their corpus indices as I = {c 1 , c 2 , . . . , c K } ⊂ [C]. Finally, we make a corpus decomposition with weights proportional to the influence score: Metrics We study the quality of influence-based corpus decomposition C c=1 w c h c as an approximation of the test example's latent representation h = g(x). Therefore, we use the same metrics as in Section 3.1 of the main paper.
Baseline We consider SimplEx as a baseline.
Dataset We perform the experiment with the MNIST dataset.

Results
We report the result of this experiment in Figure 17 (average +/-standard deviation over 5 runs). This confirms that influence functions scores are not suitable to decompose the latent representations in terms of the corpus.

C User Study
We have conducted a small scale user study with SimplEx. The purpose of this study was to identify if the functionalities introduced by SimplEx are interesting for the clinicians. In total, 10 clinicians took part in the study.
Let us now describe the study. With the SEER Prostate Cancer dataset that is described in the paper, we have performed a SimplEx corpus decomposition presented in Figure 22. The decomposition involved 1 test patient that we called Joe and 2 corpus patients that we called Bill and Max. Our classification model predicted that Joe will die of his prostate cancer. Bill died of his prostate cancer and Max survived. The SimplEx corpus weights were as follows: 66% for Bill, 34% for Max. For both Bill and Max, the Jacobian Projections were given and presented as a measure of importance for each of their features in order to relate them to Joe.
After presenting this explanation to the clinician, we gradually brought their attention to its various components. We made several statements related to SimplEx's functionalities and asked the clinicians if they agree/disagree on a scale from 0 to 5, where 0 corresponds to strongly disagreeing, 3 corresponds to a neutral opinion and 5 corresponds to strongly agreeing.
The first two statements were related to the weights appearing in the corpus decomposition. The purpose was to determine if those are important for the clinicians and if there is an additional value in learning these weights, as is done in SimplEx. The first statement was the following: "The value of the weights in the corpus decomposition is important". The results were the following: 6 of the clinicians agreed (1 strongly), 1 remained neutral and 3 disagreed (1 strongly). The second statement was the following: "Some valuable information is lost in setting the weights to a uniform value" (the doctors are given the KNN Uniform equivalent of SimplEx's explanation in the presented case). The results were the following: 5 of the clinicians agreed (3 strongly), 3 remained neutral, 2 strongly disagreed. We conclude that the majority of the clinicians found the weights to be important. Most of them found that hard-coding the weights as in the KNN Uniform baseline hides some valuable information.
The third statement was related to the Jacobian Projections. The purpose was to determine if the Jacobian Projections provide valuable information for interpretability. The statement was the following: "Knowing which feature increases the similarity/discrepancy between two patients is important". The results were the following: 9 of the clinicians agreed (5 strongly), 1 disagreed. We conclude that the Jacobian Projections constitute a crucial part of SimplEx's explanations.
The fourth statement was related to the freedom of choosing the corpus. The purpose was to determine if the flexibility of SimplEx is useful in practice. The statement was the following: "It is important for the clinician to be able to choose the patients in the corpus that is used for comparison". The results were the following: 4 of the clinicians agreed (1 strongly), 1 remained neutral, 5 disagreed (3 strongly). Clearly, the clinicians are more divided on this point. However, this additional freedom offered by SimplEx comes at no cost. A clinician that desires explanations in terms of patients they are familiar with can use their own corpus. A clinician that is happy with explanations in terms of any patients can use a corpus sampled from training data.
The last statement was related to the use of SimplEx in order to anticipate misclassification, as it is suggested in Section 3.2 of the main paper. The statement was the following: "If Bill had not died due to his prostate cancer, this would cast doubt on the mortality predicted for Joe". The results were the following: 6 of the clinicians agreed (2 strongly), 1 remained neutral, 3 disagreed (2 strongly). We conclude that, for the majority of the clinicians, SimplEx's explanations affect their confidence in the model's prediction.