{'fraction_non_alphanumeric': 0.025403481398410453, 'fraction_numerical': 0.006192403335121166, 'mean_word_length': 4.904563120771556, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
To what extent should we trust AI models when they extrapolate?

Many applications affecting human lives rely on models that have come to be known under the umbrella of machine learning and artificial intelligence. These AI models are usually complicated mathematical functions that map from an input space to an output space. Stakeholders are interested to know the rationales behind models' decisions and functional behavior. We study this functional behavior in relation to the data used to create the models. On this topic, scholars have often assumed that models do not extrapolate, i.e., they learn from their training samples and process new input by interpolation. This assumption is questionable: we show that models extrapolate frequently; the extent of extrapolation varies and can be socially consequential. We demonstrate that extrapolation happens for a substantial portion of datasets more than one would consider reasonable. How can we trust models if we do not know whether they are extrapolating? Given a model trained to recommend clinical procedures for patients, can we trust the recommendation when the model considers a patient older or younger than all the samples in the training set? If the training set is mostly Whites, to what extent can we trust its recommendations about Black and Hispanic patients? Which dimension (race, gender, or age) does extrapolation happen? Even if a model is trained on people of all races, it still may extrapolate in significant ways related to race. The leading question is, to what extent can we trust AI models when they process inputs that fall outside their training set? This paper investigates several social applications of AI, showing how models extrapolate without notice. We also look at different sub-spaces of extrapolation for specific individuals subject to AI models and report how these extrapolations can be interpreted, not mathematically, but from a humanistic point of view.


INTRODUCTION
Can we trust the decisions of AI models? How do we know if we can trust it? To answer these questions, we need to know (1) the specific tasks for which a model is intended, and (2) the data on which it has been trained. In this paper, we focus on these two aspects with illustrative examples in the results section.
There has been a considerable effort to promote accountability and transparency in the use of machine learning models. Much research has focused on developing interpretable models for various tasks. Another branch of study has focused on developing methods to demystify black-box models that are hard to interpret. Some researchers and policymakers have proposed that any model output should be accompanied by an explanation understandable to the person affected. Stakeholders should be able to understand what goes into an AI decision. Regulations informed by these suggestions have recently started taking effect. For instance, the European Union has drafted and passed the General Data Protection Regulation (GDPR) [47]. The regulation responds to cases in which people were deprived of any explanation about consequential decisions made by the models for them. An example of such deprivation is in recidivism prediction used for bail decisions [38]. Interpretability is helpful to the concerned party as well as policymakers and scholars attempting to map and measure models' attitudes towards demographic groups to expose and overcome their biases and shortcomings [54].
In this paper, we show that when making decisions or predictions about new samples, machine learning models often have to extrapolate outside the samples on which they have been trained. Extrapolation refers to the type of estimation that goes beyond the convex hull of training set. For example, a model that has learned to predict the likelihood of a person defaulting on their next credit card payment may often need to project outside the original observation range provided by the training set. For researchers interested in the social implications of extrapolation, we provide three case studies in the results section to translate the math problem into a human problem. In translating the math problem, we provide accessible examples that require no specialized knowledge to understand.
The extent of extrapolation is crucial information about the model's decision. We recommend the extent of extrapolation be reported in favor of accountability and transparency. We propose that AI models automatically log their extrapolation.
Mathematically, extrapolation is a concept defined in contrast to interpolation. A given model, M, is trained on a finite set of samples, i.e., a training set. Future decisions/predictions of a model will be based on what it has learned from its training samples. This automatic learning process has come to be known as artificial intelligence (AI) and machine learning. Moreover, because learning is automatic via complex mathematical procedures, the resulting models are often hard to interpret, making them black-box models.
A training set, however large, will have a finite set of samples, and those samples form a convex hull. Here, we denote the training set by D and the convex hull of the training set by H . Eventually, M will be used to make predictions/decisions about samples other than the ones in D . These other samples are referred to as testing samples.
Whenever M makes a decision for a new sample , that process is either interpolation or extrapolation. If is inside the convex hull of D , then the operation performed by M is an interpolation; otherwise, it is called extrapolation. A convex hull is the smallest interval containing all the data abscissae [3].
Consider, for example, a case where a model is trained to make clinical decisions for patients. The model would be trained on data from previous patients gathered in a training set. Now, assume that patients' age in the training set varies between 25 and 60. When the model receives information about a patient younger than 25 or older than 60, it would need to extrapolate outside the convex hull of its training set. This seems straightforward, but extrapolation may happen even for patients aged between 25

Our contributions
Our contributions can be summarized as the following: (1) We show that for several standard datasets used in social applications of AI, a considerable portion of testing samples fall outside the convex hull of the training set. Evidently, the functional task of machine learning models often involves extrapolation.
(2) We show that the extent of extrapolation for testing samples is not negligible. We further analyze the patterns of the required extrapolations.
(3) We propose a new requirement for achieving the explainability of AI models: any extrapolation performed by AI models should be reported to the relevant parties affected by the outcome. For example, when a model used for medical decision making has to extrapolate to decide for a new patient, the model should report, to the physician in charge, the fact that the new patient falls outside the convex hull of the data it has been trained on, namely, the model has extrapolated.

Our plan
First, we review the literature on the accountability and transparency of AI models to show the current roadblocks that we will help clear. We draw on the literature of extrapolation and the geometry of datasets. Second, in Section 3, we present the formulation for investigating the extent of extrapolation for any testing sample. Our systematic approach is model agnostic because it promotes transparency by analyzing the geometry of data. In our formulation, we consider domains with a combination of categorical, integer, and continuous variables. In Section 4, we present our results on several datasets widely used in research. Most research published on some of these datasets has discussed bias inherent in the data, but few offer ways to audit how much the models extrapolate. We propose a solution. When extrapolation happens, the model should have a self-auditing system to report the extent of extrapolation. In section 5, we discuss the policy implications of our work, followed then by the conclusion.

Studies on promoting accountability and transparency
The past few years have seen promising progress in promoting accountability and transparency in using AI models.
Some have made headway in developing inherently interpretable models. Such models need to be designed and tailored for individual tasks, e.g., financial lending [12], embryo selection [1], recidivism prediction [55], etc. This judicious use of models is advantageous, especially for high-stakes decision-making. For such applications, the common wisdom has been to avoid black-box models because they are hard to interpret. Many interpretation methods have failed to explain the rationale behind decisions of AI models reliably. In fact, some widely used interpretation methods such as LIME [33] may even produce explanations that contradict the models' decisions. Because of that, Rudin [35] and others have suggested that black-box models should not be used in high-stakes applications. In support of this approach, studies show that in some learning tasks, deep networks and other black-box models do not have an accuracy advantage over simple models, especially if one performs a modest pre-processing on the data [36].
However, one appeal of machine learning, and more often of deep learning, is automation (i.e., learning from data without feature engineering or sophisticated model design). It is known that neural networks are universal approximators [14,44], meaning that they can approximate any function over a domain with bounded error. As a result, for many learning tasks, black-box models such as neural networks are frequently used to avoid feature engineering, or else prevent transparency and keep the models proprietary [37]. In the meantime, better methods have been developed for interpreting, auditing, and debugging black-box models [53]. Some of the proposed interpretation methods are model agnostic [54]. Some others are specific to certain model types, e.g., decision tree models [28]. There are also studies focused on making the explanations better understandable by non-experts [40,49] using counterfactuals and interactive user interfaces.
One of the motivations for using machine learning models is that these models can learn by extracting hidden patterns and structures in the data. Based on that learning, they can generalize and make decisions and predictions about unseen data with reasonable accuracy, sometimes more than humans. If we agree that a model has learned from a dataset, should we expect that the model will generalize well on any related data? What if the model has to extrapolate outside the convex hull of its training data? Studies in psychology and cognitive science show that humans and machines are usually less confident in their decisions and predictions when they extrapolate instead of interpolating, and a correlation exists between human confidence and the correctness of predictions [43].
We need computational methods and procedures to interpret the models to promote accountability and transparency.
It is also important to mandate auditing for black-box models and explanations for outputs. In that spirit, rules and regulations have been proposed such that decisions made by AI models be accompanied with clear explanations for individuals affected by decisions [48]. One culmination of the many years of advocacy is the European Union's GDPR. Other organizations, including the World Health Organization (WHO), have adopted guidelines for ethics and governance of AI in their field [32].
Building on these endeavors, we suggest that the geometry of data should also be part of the routine procedures for auditing and explaining AI models. The geometry of data is an aspect of data that has received less attention, especially in instances where models have to extrapolate outside the convex hull of their training set. The formulation section and the results section demonstrate this point. There, we show that AI models extrapolate frequently and significantly outside the range of what they have encountered in the training process. We believe that routine and automatic reporting about the extent of extrapolation can promote transparency and accountability.

Studies on extrapolation and geometry of datasets
Geometry is one important aspect of data which also intimately relates to statistics and other mathematical properties.
Many studies in the AI literature have focused on the geometric understanding of data, e.g., [13,27]. Geometry helps understand models' functional behavior and distinguish whether that involves interpolation or extrapolation. This approach provides clues to the functional operation of a model in relation to the training data. By definition, when a function, M, operates on input outside its convex hull of training set, H , it extrapolates. In other words, a function has to extend outside its H to reach an input and process it.
The distinction between interpolation and extrapolation has long been part of machine learning literature. For example, Haffner [23] developed extrapolating support vector machines (SVMs) to classify datasets such as MNIST.
Vincent and Bengio [46] used the distance to convex hulls for classification with K-Nearest Neighbor algorithms.
Extrapolation also has a rich literature in many fields of study, including pure and applied mathematics [17,41], cognitive science [21], psychology [42], neuroscience [22], etc. Mathematicians have studied how a learned function can be extended in certain ways outside a domain, e.g., Whitney's extension problem [18]. Yousefzadeh and Mollick [52] recently reviewed the cognitive psychology literature that studies extrapolation and learning.
However, extrapolation has been neglected in recent literature on the mathematical understanding of black-box models and in the scholarship on accountability and transparency of AI models. In recent literature, extrapolation usually implies using a trained model for a task different from the task it has been trained on. For a model trained for object recognition, extrapolation may imply using it to classify hand-written digits or radiology images of the liver, i.e., out-of-distribution samples. With that thinking, Xu et al. [51] concluded that neural networks are not good at extrapolation.
Indeed, there is a connection between extrapolation and the topic of out-of-distribution detection [26]. But extrapolation does not only happen for out-of-distribution samples. While a model may need to extrapolate to process an input, being outside the H does not necessarily imply that input is out-of-distribution. Let us be clear: we do not imply that all samples outside the convex hull of the training set should be considered out-of-distribution. Extrapolation and learning frequently supplement each other, but evidence suggests that both humans and AI models are more susceptible to mistakes when they extrapolate as opposed to when they interpolate [9,39,43,50]. This is sometimes reported as extrapolation tasks being more difficult to perform correctly [15,21,29]. Given this, we caution that one should be informed, and perhaps be more cautious and less confident, about the instances where AI models extrapolate.
Some studies assume that machine learning models generalize merely by interpolating between training samples [50], but geometric evidence does not support that presupposition. Other studies assume that learning is solely about interpolation [5]. Yet, this presumption neither explains how machine learning models generalize, nor aligns with human understanding of learning. Recently, Balestriero et al. [4] concludes that "learning almost always amounts to extrapolation" in any high dimensional space (more than 100 dimensions). We show that even in datasets with as few as 23 dimensions and more than a million samples, extrapolation is not negligible for a considerable portion of samples.
This is the third case study in the results section. Therefore, the number of dimensions might not be a good guide to determine whether a model is interpolating or extrapolating. Instead, for any given sample, one should solve an optimization problem (described in Section 3) to verify whether the sample is inside or outside the H . If the sample is outside the H , one should examine the directions of extrapolation.
All the above literature review amounts to our suggestion that interpretation frameworks should report if the model has extrapolated for each testing sample, the features involved in extrapolation, and the extent of extrapolation in those features. With such information, an expert and the affected parties will stay informed. We further suggest that directions of extrapolation be investigated systematically, using advanced techniques in numerical linear algebra [20] to identify the dominant patterns and most influential features.

FORMULATION
Our trained model, M, is a function that maps inputs to outputs. It follows that M has a domain and a range. The domain of the model may be bounded or unbounded depending on the nature of the variables. For example, for a variable representing the age of people, the domain will be lower-bounded by zero. It is also sensible to assume that the age variable is upper-bounded by 120 or so to represent the cap of life expectancy. The domain may be discrete for certain variables, such as a categorical variable that represents the occupation of people. We denote the domain of M by Ω. Ω could be comprised of discrete and continuous spaces.

Roozbeh Yousefzadeh and Xuenan Cao
For example, consider two samples in D representing two patients, one female and one male. Patient information may include their blood test measurements. A convex combination of these two patients could be points that are neither male nor female, rather it could be 50% female and 50% male. Similarly, we can consider three training samples for a dataset about people's salaries and occupations: a nurse, a farmer, and a programmer. Convex combination of these samples could be a point representing someone who is 20% nurse, 50% farmer, and 30% programmer. These convex combinations might not always be meaningful for certain types of studies. It is not hard to imagine someone being a nurse, a farmer, and a programmer simultaneously, but that might not be representative of samples we have in D . In fact, we might have extracted some datasets from the census where each sample has only one occupation. In such a case, it could be more sensible to consider points in the convex hull that only have a single occupation. We will explain how such points may be computed.
For continuous features, however, we may have different thinking. For example, if we have three samples, all nurses, with varying ages of 30, 40, and 50 years old, a convex combination of them could be a sample whose occupation is nurse, with age somewhere between 30 and 50. Therefore, we may only be interested in certain convex combinations, representing points with discrete values for certain categorical variables such as occupation. Conversely, sometimes we might want to relax that requirement for some categorical features and consider mixed categorical variables. This would be reflected in how we define the Ω.
The discussion above signifies that all points in H will be within the bounds of Ω, but they may not be a member of it when Ω is discrete in certain dimensions. Therefore, we would be interested in points that belong to the intersection of H and Ω. Note that we may want to relax some or all of the discrete variables to study different dimensions of extrapolation. Perhaps, we want to relax the gender variable to include convex combinations of female and male samples. Moreover, for a female testing sample, we may want to compute the closest convex combination of males to her. We can do so by modifying the Ω. In the next section, we formulate an optimization problem that can solve these questions.

Projecting a query point to a convex hull
Given a point , in the feature space Ω, we would like to find the closest point to it on the H . Distance can be measured using any desired norm. Here, we use the 2-norm distance and minimize it via the objective function Our constraint, in a broad description, is that the solution lies within the intersection of H and Ω ℎ ∈ H ∩ Ω.
When Ω represents a continuous space, H ∩ Ω will be equivalent to H , but when Ω includes discrete variables, their intersection may only be a subset of each. To implement constraint (2), we break it into several smaller constraints, some of them ensuring that ℎ belongs to H , and other ensuring that ℎ belongs Ω.
First, we relate ℎ to the samples in training set where D is the training set, formed as a matrix where rows represent samples and columns represent features.
The next two constraints ensure that ℎ belongs to the convex hull of D.
where 1 denotes a column vector of ones.
Minimizing the objective function (1) subject to constraints (3)-(5) will lead to the point on H closest to .
Additionally, when Ω has discrete features, we have to ensure that Minimizing the objective function (1) subject to constraints (3)-(6) will then lead to the point on H ∩ Ω closest to . We denote this projection with a function ℎ = P ℎ ( , H , Ω).
From the optimization point of view, this is a constrained least squares problem [20]. Since our optimization problem is convex, we are guaranteed to find its solution. There are fast algorithms to solve this problem efficiently [30], especially when Ω is continuous. In such cases, a Gradient Projection (GP) algorithm could be a good choice as it can rapidly change its active set of constraints. At each iteration of the GP algorithm, we first compute the Cauchy point (the feasible minimizer along the direction of derivatives), and then perform a subspace minimization for the samples that are not binding in constraint (5). Gradient projection algorithm may be applied to the dual form of the problem as well, which could be computationally beneficial if the solution is composed of a large portion of samples in D .
When Ω has categorical variables, it would be helpful to use discrete optimization algorithms in tandem with the gradient projection method. One approach we have successfully used is to first relax the discrete space into a continuous space, and then gradually impose the discrete requirement using a homotopy method. In separate work, we shall discuss these optimization algorithms. Our code is published online.
The above statistical formulation shows how we can measure the extent of extrapolation through geometric analysis.
In which dimensions and socially consequential ways do models project the query point (testing sample) onto the convex hull? This section measures the distance of projection in several social applications of AI to show how models extrapolate without our knowledge. We also provide narrative examples to illustrate the real-life impact of extrapolation.
In the following cases, extrapolation happens in multiple dimensions -both continuous features (such as a person's age and the months of trading, which continues on a spectrum) and categorical features (such as the country of origin and gender). A testing sample outside the convex hull requires the model to project in both categorical and continuous variables. The fact that the training set is finite necessitates extrapolation in areas against social conventions. In the following case studies, the extrapolation happens to the extent that both categorical and continuous variables have to change for the query point to reach the convex hull.
The first case highlights extrapolations that are not negligible because it goes against our social norms related to gender, race, and social-economic status. The second case highlights the high frequency of extrapolation. The third case shows how models may need to extrapolate for a large clinical dataset. Together, these results showcase frequently used, widely studied, and high-stake datasets. By showcasing the non-negligible extrapolations, we make a crucial distinction between our contributions and the existing literature: the critical insight here is not that datasets contain biased materials. Instead, we emphasize that without our proposed systematic approach, models trained on such datasets may freely and significantly extrapolate without notice.

The First Case: Adult Income Dataset
Adult income dataset [25] is a standard public dataset that predicts individuals' annual income based on census data.
The categorical variables include race, gender, native country, education, marital status, work class, and occupation.
Age, years of education, capital, and work hours per week are continuous features.
We choose to study this dataset because, first, it is widely used to study social applications of AI [16], and second, the extent of extrapolation for this dataset has not been subject to research from the perspective of promoting transparency and accountability. Scholars interested in the social implications of income prediction will benefit from this analysis and the practical methods we suggest to investigate and report the extent of extrapolation.
There are 16,281 testing samples and 32,561 training samples in this dataset. We see that 52% of testing samples are considerably outside the convex hull of the training set, as shown in Figure 1. For 7% of the samples, there is not even a continuous path to the H , meaning that for at least 1200 people, a model must extrapolate in categorical features and continuous features to reach the samples. In other words, for those testing samples, there is not a single training sample that has the same gender, race, country of origin, work class, and marital status. Figure 2 shows the distribution of distance to H for all testing samples that are outside the H with a continuous path to reach it, i.e., the 45% of samples shown in Figure 1. As we can see, the extent of extrapolation is significant for most of these samples. represent features. We project these samples to the intersection of H and Ω using equation (7) to obtain . Directions betweem the samples and the convex hull can be obtained via The matrix of directions, V, has full rank, and its 2-norm condition number is 2 × 10 11 . This condition number implies that there is some redundancy in the matrix of directions. Using rank-revealing QR factorization [10], we identify that the workclass of "Work without pay" is the redundant feature. Dropping this feature from the V will reduce its condition number to 4 × 10 3 . Singular value decomposition (SVD) of V indicates that there are 5 predominant patterns in directions. Analyzing the spectrum of directions indicates that there are 2 major clusters in V.
Story Time: what happens to a person subject to extrapolation? We now explain, in more detail, one of the testing samples that are outside the training set with no continuous path to the convex hull, i.e., one of the samples in the 7% described in the pie chart in Figure 1. For illustration, we call this sample Anong.
Anong is a 55-year-old unmarried woman, with no child, originally from Thailand and now working in the United Suppose the HR wants to relax the country of origin further to include both citizens and immigrants. In that case, the model finds the closest point represented by a group of single white women around the same age as Anong, who works less per week, averaging 41 hours per hour. What if the model also relaxes the categorical differences between man and woman (because gender really should not be considered in making a salary-related decision, although in reality, it is unfortunately not often the case). This time, the model extrapolates based on a group that averages at 42-year-old, extrapolates in the dimension of working hours per week. Anong is not a singular case. We find many query points requiring the model to extrapolate either in continuous features or both continuous and categorical features.
In social norms, extrapolating in the dimensions of categorical and continuous variables (gender, country of origin, and age) is shockingly excessive. The statistically significant adjustments are predetermined not by the bias embedded in the training set but by the fact that models have to extrapolate. We are not arguing that the bias in the datasets does not matter -they matter a lot. But here, we are concerned with the explainability issue, which we tackle with the tool to analyze the extent of extrapolation. The problem is not that we do not understand a model trained on this dataset. In fact, many have published on the bias in the models trained on this dataset [6]. The more persistent issue is that models have been extrapolating salary recommendations without informing the users and policymakers. This case shows what we understood the least, and the model does the most in many cases of extrapolation.

The Second Case: FICO dataset
The FICO dataset is a real-world financial model for determining the risk associated with traders who sell and buy goods, currency, and stock. A model trained on this dataset would predict the risk of traders. This dataset was the subject of challenge/competition at the 2018 NeurIPS conference [19]. The best accuracy achieved on this dataset was by a simple model as opposed to a black-box model such as neural networks [11]. This dataset does not have any demographic information about individuals. Instead, all the 23 features are about financial transactions. This feature helps cut the weeds around models making mistakes in categorical variables, which we saw in the preceding case study about Anong. Moreover, we specifically chose this dataset because it was the subject of a challenge and further studies [37]. As a result, the dataset is relatively well-known in the AI research community. This study shows that extrapolation is still frequent and significant even for datasets locating a highly specific topic, with only continuous variables and no demographic information.
Amongst the testing samples (traders), approximately half (54 percent) of them are outside the convex hull. Of the 54 percent, the histogram in Figure 3 shows the distribution of distance to the convex hull. Note that in this dataset, we do not have any categorical variables. All variables are continuous, such as those denoting trade frequency, delinquency, and trade open time. Let us now pick a random trader at the average distance to the hull, which we show as testing point #7 in Table 1.
Comparing the query point and its projection to the convex hull, we see that the model extrapolates moderately in the dimensions related to delinquency (percentage of trades never delinquent, and months since most recent delinquency).
The extrapolation in the dimensions related to delinquency might seem very significant. However, if we look at other variables crucial for understanding the risk associated with a trader, we see a different story. As shown in Table 1, we see that the model has to extrapolate much more significantly in several key dimensions, such as those related to trading frequency (e.g. the months since the trader's oldest trade is 30 months less and the trader's total number of credit accounts twice as few) and the amount of installment transactions (e.g. net fraction installment burden).

The Third Case: Veterans Aging Cohort Study
We now consider a very large real-world dataset containing 1,048,575 samples of electronic health record (EHR). This dataset is part of the Veterans Aging Cohort Study (VACS) [24] at the veterans healthcare system. Dataset includes Manuscript submitted to ACM blood tests, and whether the patient has Hepatitis C or not. All patients in this dataset are HIV+ and they have been on Highly Active Anti-retroviral Therapy (HAART) for more than a year.
Having more than a million samples provides the chance to investigate how extrapolation may occur by models used in the field of medicine. Indeed VACS study has been the subject of research from different perspectives, and predictive models have been developed based on this data [45]. Of the samples that are outside the hull, this sample is relatively typical as its distance is close enough to the average distance. Table 2 shows the information for this sample, and also its projection to the convex hull of training set. Many of the features for this patient, whom we call Bob, are noticeably far-off the closest point on the convex hull.
Bob is 59 of age and underweight (his Body Mass Index is less than 16). For the sake of illustration, we describe the closest point on the convex hull in a figurative way as a 54-year-old man whose information is provided in the far right column in Table 2. This younger man, which is a convex combination of many samples in the training set, has BMI of 19.9 which can be considered healthy for this age. If the model generates health-related predictions for the underweight Bob, it will have to extrapolate based on younger and regular weight patients in the training set. If we look at other variables, we see further more severe implications of extrapolations. In medical terms, an estimated glomerular filtration rate (EGFR) below 60 would mean kidney disease for any given person, and Bob has an EGFR of only 5.
Creatinine, another indicator of kidney function, is significantly higher than his closest projection on the convex hull.
Third, a Fibrosis-4 index higher than 3.25 indicates significant liver fibrosis, and Bob falls into that category. Bob and his projection to the convex hull would be treated as two entirely different cases to medical experts. Yet, a model trained on our rather large training set has no choice but to extrapolate from less ill patients to estimate the health condition of a much sicker patient like Bob. We can logically conclude that the extent of extrapolation is notable. And there are many other patients in the dataset falling outside the convex hull of training. Any given model would need to extrapolate for them.
To promote transparency and accountability, a model should inform the clinicians and physicians whenever it extrapolates, i.e., let the users of the model know when a given sample is not similar to the samples it has seen before.
A trained model may make a correct or incorrect prediction for Bob. Either way, it is reasonable that a human expert should be notified about the extrapolation performed by the model, and review the recommendation/evaluation of the model made for such patient.

POLICY IMPLICATIONS AND FUTURE WORK
Many different applications affecting humans use AI models. There have been various calls for regulating AI models to make them explainable. Notably, the European Union has drafted and passed the General Data Protection Regulation (GDPR), which requires decisions made by automated systems to be accompanied by clear explanations. Our work, in this paper, suggests that extrapolation should be an inherent part of such explanations.
The postulation that a machine learning model can automatically learn from the data and generalize well seems overly optimistic. Frequently, it has been reported that AI models may exercise various sorts of unacceptable behaviors and biases [2,31]. Sometimes such behaviors are rooted in the bias in the data. AI models may even exacerbate those biases or develop new preferences [34]. But, what if the model has to extrapolate to significant degrees for a considerable portion of its testing samples? The crucial insight here is not that datasets contain bias, but rather, algorithms can freely and significantly extrapolate in many dimensions.
From the public policy perspective, our study shows the benefit of creating AI pipelines that automatically report to relevant entities whenever they extrapolate significantly. In high-stakes decision-making, it might be sensible to impose regulations requiring AI models' extrapolated decisions/predictions to be sent for human expert review.

Privacy issues
Reporting that a testing sample falls outside the convex hull and that a model needs to extrapolate to process it, in our view, is crucial information that should be shared with the entities affected by the output of AI models. However, other questions arise. In specific applications of machine learning, it may not be possible to share information about training samples.
Consider, for example, an AI model that recommends clinical decisions based on patient data. If the model has to extrapolate for a particular patient, it would be useful to let the physician know that the model has extrapolated. Suppose physicians have the necessary authorization to access other patients' data. In that case, it may be useful for them to see the information about other relevant patients, i.e., the convex combination of samples that form the P ℎ ( , H , Ω) for patient . But, if we want to report the extrapolation to the patient him/herself, it might be best practice to just report that the model had to extrapolate, and perhaps, accompany that with additional information that extrapolation happened in which dimensions, e.g., age and BMI. But, it may not be proper nor legal to share which training samples were closest to the patient's data, as it may violate the privacy of other patients.
Our formulated auditing procedures require access to the training samples, but due to privacy issues, training sets could be protected from auditing procedures. In such cases, one approach could be to approximate the convex hull of training set using a different set of points other than the original set. There are several efficient methods available for approximating convex hulls with theoretical bounds on the maximum difference between the approximated convex hull and the original one [7,8]. Approximation can also address the first privacy concern discussed above.
The extent to which an AI model should report its extrapolation is an important topic that can be the subject of further studies on privacy and transparency.

Implications for model selection
We also observe, empirically, that extrapolation is less significant for learning tasks where deep networks have not shown an advantage over simple models. On the other hand, the extent of extrapolation is larger for learning tasks where deep networks have an advantage. This suggests that the geometry of data, and specifically, the relative extent of extrapolation, might be a useful guide for model selection. When the extent of extrapolation is small, simple models may be more appealing, especially if they achieve an accuracy as good as deep networks, because simple models are computationally less expensive and easier to interpret. When the extent of extrapolation is more significant, more sophisticated models such as deep networks might be more likely to achieve better accuracy. These implications may be studied in future studies in the context of model selection literature.

CONCLUSIONS
This paper promotes transparency in AI by focusing on the neglected factor of extrapolation. Countering the common belief that AI models do not extrapolate much, we show the frequency and the non-negligible extrapolation through geometric analyses of several social applications of AI related to salary, trading, and health care.
Having reviewed the literature on accountability and transparency, we present our auditing approach to clear the current roadblocks. Most research published on these datasets has discussed bias inherent in the training set, but few offer ways to audit how much the models extrapolate. This paper proposes a practical solution: when extrapolation happens (which has been happening so far without notice), the model should have a self-auditing system to report the extent of extrapolation. If extrapolation is statistically excessive, the model should generate a report indicating to what extent and in which features has the model extrapolated.
We illustrate with examples how socially consequential extrapolation can be and how to add a robust auditing step to the AI pipeline. The extrapolations we have examined can happen in multiple dimensions -both continuous features and categorical features. A testing sample outside the convex hull requires the model to project in both categorical and continuous variables. Even in cases where categorical variables are not involved, extrapolation is still non-negligible.
We use a systematic method of geometric analysis in developing an auditing approach that can be applied automatically to any dataset/model. Our systematic approach is specifically useful in considering domains with a combination of categorical, integer, and continuous variables. We suggest that models be audited with respect to extrapolation, and reporting of extrapolation becomes part of regulations such as GDPR.