{'fraction_non_alphanumeric': 0.031142416208302007, 'fraction_numerical': 0.014304934509912548, 'mean_word_length': 4.848530432770192, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena

Interpretable machine learning (IML) is concerned with the behavior and the properties of machine learning models. Scientists, however, are only interested in models as a gateway to understanding phenomena. Our work aligns these two perspectives and shows how to design IML property descriptors. These descriptors are IML methods that provide insight not just into the model, but also into the properties of the phenomenon the model is designed to represent. We argue that IML is necessary for scientific inference with ML models because their elements do not individually represent phenomenon properties; instead, the model in its entirety does. However, current IML research often conflates two goals of model analysis -- model audit and scientific inference -- making it unclear which model interpretations can be used to learn about phenomena. Building on statistical decision theory, we show that IML property descriptors applied on a model provide access to relevant aspects of the joint probability distribution of the data. We identify what questions such descriptors can address, provide a guide to building appropriate descriptors and quantify their epistemic uncertainty.


Introduction
Scientists increasingly use machine learning (ML) in their daily work. This development is not limited to natural sciences like the geosciences (Reichstein et al. 2019) or material science (Schmidt et al. 2019), but also extends to social sciences such as education science (Luan and Tsai 2021) and archaeology (Bickler 2021).
When building predictive models for problems with complex data structures, ML outcompetes classical statistical models in both performance and convenience. Impressive recent examples of successful prediction models in science include the automated particle tracking at CERN (Farrell et al. 2018), or DeepMind's AlphaFold, which has essentially solved the protein structure prediction challenge CASP (Senior et al. 2020). In such examples, some see a paradigm shift towards theory-free science that "lets the data speak" (Kitchin 2014, Anderson 2008, Mayer-Schönberger and Cukier 2013, Spinney 2022. Indeed, prediction is one of the core aims of science (Luk 2017, Douglas 2009), but so are, as philosophers of science and statisticians emphasize, explanation and knowledge generation (Salmon 1979, Longino 2018, Shmueli et al. 2010. Focusing exclusively on prediction may therefore represent a historical step back (Toulmin 1961, Pearl 2018. What hinders scientists from using ML models to gain real-world insights is model complexity and an unclear connection between model and phenomenon -the so-called opacity problem (Boge 2022, Sullivan 2020. Interpretable machine learning (IML, also called XAI, for eXplainable artificial intelligence) aims to solve the opacity problem by analyzing individual model elements or inspecting specific model properties (Molnar 2020). Different stakeholders with different goals hold diverse expectations of IML (Zednik 2021), including scientists (Roscher et al. 2020), ML engineers (Bhatt et al. 2020), regulatory bodies (Wachter et al. 2017), and laypeople (Arrieta et al. 2020). Due to this plurality, IML has been criticized for lacking a proper definition (Lipton 2018).
Nevertheless, scientists increasingly use IML for inferring which features are predictive of e.g. crop yield (Shahhosseini et al. 2020, Zhang et al. 2019, personality traits (Stachl et al. 2020), or seasonal precipitation (Gibson et al. 2021). Although researchers are aware that their IML analyses remain just model descriptions, it is often implied that the explanations, associations, or effects found also extend to the corresponding real-world properties. Unfortunately, drawing inferences with IML can currently be epistemically problematic because the interpretation methods are not designed for that purpose (Molnar et al. 2022). In particular, the difference between model-only versus phenomenon explanations is often unclear (Chen et al. 2020, and a theory to quantify the uncertainty of interpretations is lacking (Molnar et al. 2020a, Watson 2022. Contributions. In this paper, we present an account of scientific inference with IML inspired by ideas from philosophy of science and statistical inference. While we focus on supervised learning on identically and independently distributed (i.i.d.) data, we briefly discuss other learning scenarios in Section 5.3. Our key contributions are: 1.
We argue that ML cannot profit from the traditional approach to scientific inference via model elements because its parameters do not represent phenomenon properties (Section 3). While current IML methods aim to restore representationality of the model as a whole, they conflate the model audit and scientific inference goals of interpretation. 2.
We identify the properties that IML methods need to fulfill to provide access to aspects of the conditional probability distribution P(Y | X), where X describes predictor variables and Y the target (Section 4). We call methods that are suitable for inference IML property descriptors. We provide a guide to build such descriptors starting with a phenomenon question about X and Y and evaluating whether it can be addressed, followed by an answer to this question with ML models and finite data, and conclude with the quantification of epistemic uncertainty. We illustrate our approach using conditional partial dependence plots (cPDP) as an example IML descriptor.
Terminology. For the purposes of our discussion below, a phenomenon is a real-world process whose aspects of interest can be described by random variables. Observations of the phenomenon are drawn from the unknown joint distribution induced by the random variables and form the dataset or just data. A ML model is a mathematical model optimized with the aid of a learning algorithm applied on the collected data in order to accurately predict unknown or withheld phenomenon observations, i.e. to generalize beyond the initial data. Here we focus on the supervised learning setting. Finally, scientific inference is the process of rationally deriving conclusions about a phenomenon from data (via ML, or other types of models). We employ inference to imply investigating unobserved variables and parameters similar to statistical inference, i.e. in a more general sense than is common in some of the ML literature, where it is used exclusively as a synonym for prediction. The knowledge gained by scientific inference can build the basis of scientific explanations. These brief conceptual remarks are meant to reduce ambiguity in our usage: we lay no claim as to their universality.

Related Work
Whether and how ML models, and specifically IML, can help obtain knowledge about the world is a debated topic among philosophers of science, statisticians, and also the IML community.
Philosophy of Science. It has been argued that ML models are only suitable for prediction because their parameters are instrumental and lack meaning (Bailer-Jones and Bailer-Jones 2002, Bokulich 2011). On the other hand, Sullivan (2020) argues that nothing prevents us from gaining real-world knowledge with ML models as long as the link uncertainty -the connection between the phenomenon and the model -can be assessed. Cichy and Kaiser (2019) and Zednik and Boelsen (2022) claim that IML can help in learning about the real world, but they remain vague about how model and phenomenon are connected. Like Watson (2022), we explain that IML methods relying on conditional sampling are faithful to the phenomenon. However, while he assigns IML inferences to the causal phenomenon level, we clarify that, without additional assumptions, such inferences only reveal associational relationships (Räz 2022).
Our work makes precise that ML models can be described as epistemic representations of a certain phenomenon that allow us to perform valid inferences (Contessa 2007)

via interpretations.
Statistical Modeling and Machine Learning.  describes ML (algorithmic modeling) and statistics (data modeling) as two approaches for reaching conclusions from data. On a medical example he shows that post-hoc analysis of ML models can allow more correct inferences about the underlying phenomenon than standard, inherently interpretable data models. Our paper gives an epistemic foundation for such post-hoc analyses. Shmueli et al. (2010) distinguishes statistics and ML by their goals -prediction (ML) and explanation (statistics). Like Hooker and Mentch (2021), we argue against such a clear distinction and offer steps to integrate the two fields. This paper builds on ideas from Molnar et al. (2021), where they introduce ground-truth and confidence intervals for partial dependence plots (PDP) and permutation feature importance (PFI) of arbitrary ML models. Our work generalizes these ideas to arbitrary IML methods and draws the connection to the underlying phenomenon.
Interpretable Machine Learning. IML as a field has been widely criticized for being ill-defined, mixing different goals (e.g. transparency and causality), conflating several notions (e.g. simulatability and decomposability), and lacking a proper measure of success (Doshi-Velez and Kim 2017, Lipton 2018. Some even argued against the central IML leitmotif of analyzing trained ML models post hoc in order to explain them (Rudin 2019). In this paper, we show that, if we focus on interpretations for scientific inference, a clear foundation including a proper theory of success can be provided and these criticisms can be partially addressed.

Scientific Inference and Elementwise Representationality
The goal of this paper is to analyze and describe how we can conduct scientific inference on ML models using IML methods. This section explains why inference with ML models cannot be done as in traditional scientific models and why current IML methods do not generally address the problem. The next section describes our solution and illustrates it with a complete example from question formulation to uncertainty quantification.

ML Models are not Elementwise Representational
In scientific modeling, there is a paradigm that many models implicitly follow -we call it the paradigm of elementwise representationality. Definition. A model is elementwise representational (ER) if all model elements (variables, relations, and parameters) represent an element in the phenomenon (components, dependencies, properties). Figure 1 depicts the relationship between ER models and the phenomenon: 1 variables describe phenomenon components; mathematical relations between variables describe structural, causal or associational dependencies between components; parameters specify the mathematical relations and describe properties of the component dependencies.  The main elements of a traditional, ER model, are shown in encoding-decoding correspondence to the phenomenon elements they represent (Stachowiak 1973). Phenomenon and model elements are illustrated with a simple example of two bodies in gravitational interaction and its classical, Newtonian mechanistic description.
knowledge and an underlying scientific theory. Variables are selected carefully and sparsely during model construction, and the relations are constrained to a relation class with few free parameters. When ER models need to account for an additional phenomenon aspect, they are gradually extended so that large parts of the "old" model are preserved  (Frigg and Nguyen 2021). However, only those aspects of the phenomenon that have a model counterpart can be analyzed with this approach. Fortunately, as described above, ER models can be extended to account for further relevant aspects identified by the scientist.
Running example: 2 Linear Model. Suppose a researcher, we call her Laura, wants to study what attributes influence students' grades in mathematics. Specifically, she wants to research how language skills and math skills are associated.
She uses data from Cortez and Silva (2008), who collected a dataset 3 , encompassing 32 student attributes in Portuguese schools including math/Portuguese grades, age, parents' education, etc.
Laura starts with a classical ER model -a linear model with one predictor and one target variable. She selects the student grade in Portuguese X p and in mathematics Y as her proxy variables for students language and math skills respectively. 4 Based on her background knowledge, she assumes that the true relationship can be described as Y = β 0 + β 1 X p + with β 0 , β 1 ∈ R and an error ∼ N(µ, σ 2 ). Laura centers X p by the average student grade in 2 Since the physical model from Figure 1 is a mechanistic causal model (Schölkopf et al. 2021), we switch henceforth to an illustrative associational model from the social sciences that compares more fairly with current associational ML models. We strongly simplify things in this example and do not claim that it reflects social science methodology or that ML is even required. 3 see Appendix B for more details. 4 In the Portuguese grading scheme, the range is 0-20, where 0 is the worst and 20 the best grade.
Portuguese and obtains the prediction model that minimizes the mean-squared-error (MSE), Laura's model is ER: she can interpretβ 0 = 10.46 as the predicted math grade for an average Portuguese student (if x p = 12.55) 5 andβ 1 = 0.77 as the strength of association between the Portuguese grade and the math grade.
Laura can analyze the model to draw scientific inferences about the underlying phenomenon, for example, with 95% confidence intervals 6 for her estimatesβ 0 andβ 1 with [10.05; 10.88] and [0.63; 0.91] respectively. The inference she draws is on these parameters; Laura can only draw conclusions about the phenomenon if the model is ER and highly predictive. Laura may conclude fromβ 1 that language skills and math skills are strongly and positively related.
To reach a more expressive and predictively accurate model, Laura can also extend the model to include additional features, relations, or interaction terms. As long as she preserves ER, she can directly draw scientific inferences from analyzing model elements. Indeed, these inferences are only as valid as the modeling assumptions (e.g. target normality, homoscedasticity, or linearity).
ML Models are generally not ER. ER makes model elements interpretable and allows to reason about the effects of model or even real-world interventions; as such, ER models suit our image of science as an endeavor aimed at understanding. However, as mentioned above, for ER we usually require background knowledge on which components are relevant, and we need to severely restrict the class of relations that can be considered for the given phenomenon.
These difficulties might lead scientists to either limit their investigations to phenomena that are already well studied or, as  argued, to develop overly simple models for complex phenomena and possibly draw wrong conclusions.
ML models excel for complex problems with an unbounded number of components that display ambiguous and entangled relationships i.e. ML models are highly expressive (Gühring et al. 2020). ML models are subject-domain independent (Bailer-Jones and Bailer-Jones 2002), this means that we do not necessarily need subject-domain background knowledge in modeling. Instead, ML modeling only requires specifying a broad model class and a set of hyperparameters. The choice of these hyperparameters is data-domain specific i.e. they reflect inductive biases that allow for efficient learning.
The gain in generality and convenience with ML comes at a price -ML models are generally not ER. As also argued in Boge (2022), Bokulich (2011), Bailer-Jones and Bailer-Jones (2002), ML models (e.g. artificial neural networks) contain model elements such as weights, activation functions, or network structure that have no corresponding phenomenon counterpart.
Running example: Artificial Neural Network (ANN). Suppose Laura is dissatisfied with her linear model and fits a dense three-layer neural network to predict math grades using all available features. 7 She reduces the test-set MSE from 16.0 in the linear-one-variable model case to 8.9. A formal description of the model is given by: where model elements are the values of the weight matrices W 1 , W 2 , W 3 and bias vectors b 1 , b 2 , b 3 , and the activation functions σ 1 , σ 2 , σ 3 . Unlike in the linear model above, it is highly unclear what these parameters correspond to in our data or phenomenon. While the input vector x is still representational, the weights, activation functions, or threelayer architecture are very hard or even impossible to interpret: A high value of weight W (3,2) 1 might have a positive, neutral, or negative effect on the target, dependent on all other model elements; the activation function only reflects the currently popular heuristics in model training; and the particular three-layer architecture is a result of model selection based on predictive performance and rules of thumb, but with little phenomenon-based rationale.

Scientific Inference in Light of Current IML
We have argued so far that: i) If models are ER, they allow for scientific inference.
ii) ML models are generally not ER.
How can we still do scientific inference with ML models? We discuss two strategies to enable scientific inference with ML: We argue that the first strategy, namely restoring ER, fails because ML models are designed to represent in a distributed manner; the second strategy, embracing holistic representationality, is highly promising but current attempts conflate different goals of model analysis. This discussion sets the stage for the next section, where we show how a holistic account of representationality can enable scientific inference.
Restore ER. One strategy towards scientific inference with ML is to challenge Proposition ii) and show that ML models are ER too. Researchers in this camp argue that individual elements in ML have a natural phenomenon counterpart, but this counterpart only becomes evident when these model elements are extensively scrutinized. 8 This would be surprising: ER is not enforced in state-of-the-art techniques and, even worse, some methods such as training with dropout purposefully discourage ER in order to gain robustness (Srivastava et al. 2014); ML models like ANNs are designed for distributed representation (Buckner andGarson 2019, McClelland et al. 1987).
It has been claimed that model elements represent high-level constructs constituted from low-level phenomenon components that are often called concepts (Buckner 2018, Olah et al. 2020 If this is the case, model elements or aggregates of such elements can be reconnected to the phenomenon; ER would be restored by the representations of coarse-grained phenomenon components. Research on neural networks supports that some model elements are associated with concepts (Mu and Andreas 2020, Voss et al. 2021, Kim et al. 2018, Olah et al. 2017, however, often these elements are neither the only associated elements nor exclusively associated with one concept as shown in Figure   Research on the representational correlates of model elements seems indeed fascinating. However, current ML models that do not enforce ER will rely on distributed representations and cannot be reduced to logical concept machines.
The associative connection between model elements and phenomenon concepts should not be confused with their equivalence. Analyzing single model elements will therefore be a hopeless enterprise.

Embrace Holistic Representationality.
An alternative route to scientific inference is to accept that ER is wellsuited for scientific inference and that ML models are not ER but reject that ER is the only approach for scientific inference. To choose this route, one must offer an alternative path for drawing scientific inference with ML models that goes beyond the analysis of model elements.
Our approach is to regard the model as representational of phenomenon aspects only as a whole -we call this holistic representationality (HR). HR implicitly underlies large parts of the current research program in IML: Modelagnostic methods, in particular, analyze the entire ML model simply as an input-output mapping (Scholbeck et al. 2019); In the same spirit, many model-specific IML methods like gradient or path-based feature attribution treat ML models as mappings with additional useful properties such as differentiability (Alqaraawi et al. 2020).
Model-agnostic and model-specific methods share the idea that relevant model properties such as the effects or importances of variables can be derived by analyzing the model just as a functional mapping. Initial definitions of, for example, global feature effects (Friedman et al. 1991) and feature importance  or local feature contribution (Štrumbelj and Kononenko 2014) and model behavior (Ribeiro et al. 2016) have been presented. However, many researchers have pointed out that these methods lead to counterintuitive results for dependent or interacting features and offered alternative definitions (Apley and Zhu 2020, Strobl et al. 2008, Molnar et al. 2020b, Goldstein et al. 2015, König et al. 2021b, Janzing et al. 2020, Slack et al. 2020, Alqaraawi et al. 2020. We believe that these controversies stem from a lack of clarity about the goal of model analysis. Are we interested in model properties to learn about the model (model audit) or do we want to use these model properties as a gateway to learn about the underlying phenomenon (scientific inference)? These two goals must not be conflated. Scientific inference and model audit should be viewed as two different but interacting goals. In each of them, we take different stances toward the ML model: The auditor adopts a skeptical attitude of the model, she has ground-truth information or normative standards to check the model against; the scientist adopts a trusting attitude, she wants to learn from the model. Both cases describe a knowledge asymmetry (Gobet 2018, Rosser et al. 2008) but in opposite directions. Auditing the model is an indispensable step for scientists to gain enough trust in it. Only after several rounds of auditing and improvement should the researcher rely on the model to draw scientific conclusions.

Scientific Inference with IML Property Descriptors
We just argued that ML models are generally not ER and therefore do not allow for scientific inference in the standard way. HR offers a viable alternative, but currently different goals of model analysis are conflated. In this section, we show that a HR perspective enables scientific inference using IML methods. Particularly, we show that certain IML methods -we call them IML property descriptors -can represent phenomenon properties.  Figure 3: IML property descriptions distill phenomenon properties from HR models. Instead of explicitly encoding phenomenon properties as parameters like for ER models, HR models (e.g. ML models) encode phenomenon properties in the whole model. We propose that these encoded properties can be read out with IML property descriptions external to the model. In this way, IML offers an indirect route to scientific inferences through model analysis.

IML Property Descriptions
the phenomenon and the model; we propose to draw this link using statistical decision theory, which shows what optimal ML models can holistically represent. Clearly, an approximate ML model will not provide an answer if even the optimal model cannot. However, if a question can in principle be answered with the optimal model, an ML model trained on data can approximate the answer in practice. The problem then becomes to quantify the approximation error.

ML Representationality and Optimal Predictors
Which aspects of a phenomenon ML models can represent even under ideal circumstances depends on the data, the learning paradigm, and the loss function. For identically and independently distributed (i.i.d.) data used for supervised learning, optimal predictors from statistical decision theory provide an answer (Hastie et al. 2009, p18-22). Besides the advanced theory available in this setting, supervised learning on i.i.d. data is the most popular ML setup in practical applications. We briefly discuss representationality and scientific inference in the case of unsupervised and causal learning in Section 5.3.
Basic Notation. We assume that the random variables X 1 , . . . , X n and Y fully characterize the phenomenon. We write the joint feature vector as X (X 1 , . . . , X n ) with X Range(X) and Y Range(Y). X and Y jointly describe the phenomenon. L describes a loss function L(Y, m(X)) : X × Y → R + andm a model in the set M of mappings from X to Y. Table 1 shows the optimal predictors for standard loss functions.

Problem
Loss The optimal predictors for standard loss functions can be derived from P(Y | X).

Supervised learning.
Supervised learning seeks to find an optimal predictor m by using a learning algorithm 12 I:∆ → M that selects a a modelm from a set M with the aid of a dataset D (x (1) , y (1) ), . . . , (x (k) , y (k) ) with D in the set of datasets ∆ drawn i.i.d. from the joint distribution, i.e. (x (i) , y (i) ) ∼ (X, Y). Instead of the EPE itself, the learning algorithm minimizes the empirical risk on the test data (i.e. on data not used to trainm), which is a finite-data estimate of the EPE.

IML Property Descriptors
We have just argued that ML models, when considered as a whole, approximate phenomenon aspects that can be derived from the conditional distribution P(Y | X). IML property descriptors can help to investigate these aspects by describing their relevant properties.
Five Steps Towards IML Methods for Inference. Our proposal consists of the five steps in Figure 4, which we now discuss in detail. For each of the five steps, we provide an inference example based on the prediction of student grades in mathematics. In what follows, we assume that we have a supervised learning ML modelm that approximates a phenomenon aspect described by the optimal predictor m.
Step 1) Formalize Scientific Question. Science starts by formulating a question. To address it with ML, this question has to be formalized. Exemplary questions that can already be addressed with IML methods following the scheme below are discussed in Section 4.3. Note that IML for scientific inference only helps answer questions that concern the association between X and Y.  Formally. We denote the formalized question by Q.
Example. Suppose Laura wants to find out how students' language skills are related to their math skills. She approaches this problem by asking how students' expected math grades are related to their Portuguese grades. Laura formalizes this question as the conditional expectation i.e. Q = E Y|X p [Y | X p ], where Y, X p respectively stand for the math and Portuguese grade variable. 13 Step 2) Establish Question Identifiability. Many scientific questions cannot be addressed using an ML model.
ML models can only help answer questions that could theoretically be addressed with the optimal predictor. We call a question that the optimal predictor can help answer together with additional probabilistic knowledge (e.g. aspects of P(X, Y)) identifiable. A constructive strategy to establish identifiability relative to some probabilistic knowledge is to think of the transformations, using solely the probabilistic knowledge, that take the optimal predictor into the question Q. Of course, it is desirable to keep to a minimum the amount of probabilistic knowledge required to identify the question.
Formally. The optimal predictor is denoted by m. We say that a question is identifiable relative to probabilistic knowledge K if we can compute Q from m and K.
Example. Assume that Laura has trained her neural network, which, unlike the simple linear model presented above, takes into account all available features X, to minimize the MSE loss, i.e. m(X) = E Y|X [Y | X]. Is Laura's question identifiable? For specific values of X, the optimal predictor allows to compute the expected value of Y i.e.
The only difference to Q is that m takes into account features besides the Portuguese grade, that we denote X −p . If we have access to the conditional distribution P(X −p | X p ) (required 14 probabilistic knowledge K), we can integrate these other features out by taking the expected value Thus, Q is identifiable via m given K = P(X p | X −p ).
Step 3) Design IML Property Descriptor. It is not enough to identify a question. We need a way to estimate an answer for ML models -we need IML property descriptors. An IML property descriptor describes a continuous function that applies the transformation from the question identification step above to a given ML model and outputs an element of the space Q. Thus, given the optimal predictor, an IML property descriptor outputs an answer to Q.
Continuity guarantees that if our ML model is close to the optimal model, our answer is approximately correct. We call the application of a property descriptor to a specific ML model, g K (m), a model property description.
Formally. An IML property descriptor is a continuous function g K (w.r.t. metrics d M and d Q ) 15 that identifies Q using probabilistic knowledge K: The output space Q remains unspecified to account for the variety of scientific questions; Q could denote a set of real numbers, vectors, functions, probability distributions, etc.
Example. The property descriptor describes the transformations that identify Q, i.e.
This is indeed a property descriptor because conditional expectation is continuous on M, and Q is identifiable given K = P(X −p | X p ). Note that Equation (4.1) describes the well-known conditional partial dependence plot, or cPDP, also known as M-plot (Molnar 2020, Apley and Zhu 2020).
Step 4) Estimate IML Property Descriptor. Often we lack access to relevant probabilistic knowledge K. Instead, we have a finite amount of data on which we can evaluate our ML mapping, which we call the evaluation data. It may bundle up our training and test data D (see Section 4.1), as well as additionally available (unlabeled) data, and artificially generated data. The IML property description estimator describes a way to estimate property descriptions with access only to the ML model plus the evaluation data. 15 The function d M is a metric on the function space M, d M (m 1 , m 2 ) X L(m 1 (x), m 2 (x)) P X (x) dx for m 1 , m 2 ∈ M, while d Q describes a metric appropriate for the space Q.
Formally. We denote the evaluation dataset by D * and the random process that generates it by D * . We callĝ D * : M → Q the IML property description estimator if it is an unbiased estimator of g K i.e.
Example. Laura's evaluation dataset D * is her initial training and test dataset D augmented by artificial instances created by the following manipulation: Laura makes six copies of the data, and jitters the Portuguese grade by 1, −1, 2, −2, 3 or −3 respectively. This augmentation strategy reflects how Laura understands the Portuguese grade as noisy based on her background knowledge of how much student performance varies daily and teachers grade inconsistently. Let the students with (jittered) Portuguese grade i be D * |x p =i (x ∈ D * | x p = i), then, we can define the IML property description estimator at i as the conditional mean (unbiased estimator of the conditional expectation): The estimated answer to Laura's question is plotted in Figure 5. The plot on the left suggests that math grade is only strongly dependent on Portuguese grades in the interval 8 − 17. However, as we show in the next step, we must also take into account that we have very sparse data in some regions (e.g. very few students scored below 8) before confirming this first impression. Step 5) Quantify Uncertainty. We have shown how we can estimate Q using an approximate ML model paired with a suitable evaluation dataset. But how good is our estimate? Two steps involve approximations: 1. Applying the IML property descriptor to the ML modelm instead of the optimal model m; we call the resulting error model error The model error depends on the ML modelm we obtained from the learning algorithm trained on the given dataset.
2. Applying the IML property description estimator on our evaluation dataset D * instead of computing the true model property description based on K directly; we call this error estimation error The estimation error depends on the evaluation dataset D * .
In theory, the model error and the estimation error can be separated. In practice, however, they are statistically dependent because the training and the evaluation data overlap. Generally, neither the model error nor the estimation error can be computed perfectly; this would require access to the optimal model m and infinitely many data instances.
Nevertheless, we can quantify in expectation how large the two errors are.
An intuitive approach to quantifying the expected errors is to decompose them into bias and variance contributions.
The two decompositions below quantify the range in which the true phenomenon property descriptions are most likely to lie.
Formally. For the bias-variance decomposition, we assume the metric d Q to be the squared error. 16 Considering the dataset that we entered into the learning algorithm as a random variable D, we can decompose the expected ME [m] error as follows Variance wherem I(D) is the output of a machine learning algorithm I for dataset D (Section 4.1). Considering the evaluation data as a random variable D * , we can decompose the expected EE D * error as follows The bias term vanishes because the property description estimator is by definition unbiased w.r.t. the IML property descriptor.
Example. Laura obtains different cPDPs ( Figure 5) for different models with similar performance, as well as for different selections of evaluation data, how much can she then rely on these cPDPs?
The estimates of the variances of the cPDP by Molnar et al. (2021) allow to calculate pointwise confidence intervals ( Figure 6). We can define a confidence interval that only incorporates the estimation uncertainty by 16 A bias-variance decomposition is also possible for other loss functions, including the 0-1 loss (Domingos 2000). and a confidence interval that incorporates both model and estimation uncertainty by For the combined confidence interval we require a strong and unfortunately not testable assumption to be satisfiedunbiasedness of the ML algorithm. Unbiasedness implies that, in expectation over training sets, the ML algorithm learns the optimal model, i.e. m = E D [m]. 17 Figure 6 shows that for students with Portuguese grades between 8 and 17, Laura can be very confident in her model and the relationship it identifies between math and Portuguese grade. 18 However, both for Portuguese grades below 8 or above 17, the true value might be far off from our estimated value using a given model, as we can see from the width of the confidence intervals. For these grade ranges, gathering more data may reduce Laura's uncertainty. Summary. Figure 7 gives an overview of all functions and spaces involved in IML for scientific inference. We started from a phenomenon and formalized a scientific question Q about it. Using a learning algorithm I on dataset D from the phenomenon, we learned an ML modelm that approximates the optimal model m. We then set out to answer Q fromm. We defined a property descriptor g K , that is, a function that allows to compute Q from m given K, respectively approximates Q fromm given K. Because g K requires probabilistic knowledge about P(X, Y), we introduced a property description estimatorĝ D * , a function estimating Q solely from finite data, the evaluation set D * .
Finally, we showed how the expected error of our estimation steps can be quantified with confidence intervals CI ME [m] and CI EE[ D * ] .
17 Since unbiasedness is tied to a specific context, there is no conflict with the no-free-lunch theorems (Sterkenburg and Grünwald 2021). 18 We used resampling techniques to estimate the two variances. In real-data settings it is generally not possible to always sample new data for the model training and the evaluation. Although resampling may result in an underestimation of the variance, our goal here is simply to illustrate the process of quantifying uncertainty for a concrete IML method.

Models Datasets Descriptions
Figure 7: From datasets to inferences via ML models. Mappings are represented by arrows and sets are represented by filled circles, with confidence regions in green shades. Practical IML descriptionsĝ D (m) are approximate, uncertainty-aware answers to a question Q that are built from a modelm fit on D and an evaluation dataset D * .

Property Descriptors and Current IML Methods
Many questions that can be answered based on the conditional probability distribution P(Y | X) are widely relevant.
The goal of practical IML research for inference should be to define relevant descriptors and provide accessible implementations of these descriptors, including quantification of uncertainty. To find out which specific questions are relevant to scientists, and therefore what descriptors are necessary, IML researchers, statisticians and scientists must closely interact.
In Table 2 we present a few examples of elementary inference questions that can in principle be addressed by existing IML methods i.e. these methods can operate as property descriptors already. We distinguish between global and local phenomenon questions: global questions concern general associations, local questions concern associations for a specific instance. The last column highlights current IML methods that provide approximate answers, albeit often without uncertainty quantification. Note how we ultimately require conditional versions of existing marginal IML methods, which suggests that marginal sampling, which generates unrealistic instances, is inadequate in scientific inference.

Discussion
ER models enable straightforward scientific inference because their elements represent something about the underlying phenomenon. While ML models are generally not ER, IML can offer an indirect route to scientific inference, provided model properties have a corresponding phenomenon counterpart. We have shown how phenomenon representation can be achieved through optimal predictors and described how to practically construct IML property descriptors following five-steps: the first two steps clarify what questions we can address with IML, step three and four show how to answer them with ML models and finite data, and step five allows to evaluate how certain the answers are. We pointed out 19 Only defined on phenomenon if P(X p = p, X −p = x −p ) > 0. 20 Only with the right similarity metric that accounts for the realistic constraint. m X p (X p ) cPDP (Apley and Zhu 2020) Conditional Contribution: How much worse can Y be predicted from X if we hadn't known X p ?
) cPFI (Fisher et al. 2019) Fair Contribution: What is the fair share of feature X p in the prediction of Y?
SAGE (Covert et al. 2020)  m X (X p , x −p ) ICE-curve (Goldstein et al. 2015) 19 Conditional Contribution: How much worse can Y be predicted from X = x if we hadn't known X p ?
no method yet Fair Contribution: What is the fair share of feature conditional Shapley Values (Aas et al. 2021) Relevant Value: Under which realistic conditions similar to X = x can we expect to observe relevant value y rel ?
arg min (Dandl et al. 2020) 20 Table 2: Global and local formalized questions and matching IML property descriptors. Note that questions are relative to a specific loss function L; for a set S ⊆ {1, . . . , n}, the term m X S describes the optimal predictor of Y w.r.t. loss function L and random variable(s) X i with i ∈ S . d X and d Y describe suitable metrics on X and Y respectively. that some current IML methods can already be seen as IML property descriptors.
Is the lack of elementwise representationality specific to ML models? No, ML shows only an extreme case. In fact, there is a continuum between fully ER models and HR-only models: Some scientific models contain elements that are difficult or impossible to interpret e.g. the wave function in physics (Callender 2015); complex classical statistical models like generalized additive models also contain elements that are difficult to interpret. Our main message is: the five-step approach can be used to extend inference to any non ER model (whether ML or not).
One could argue that science should only rely on ER models (Rudin 2019). Indeed, it would be great if we could always build models from simple to complex and keep ER from beginning to end. However, more and more problems seem to be very difficult to tackle with this approach ; Interpretable but inaccurate models (w.r.t. to the phenomenon) are not a solution . In situations where we cannot construct accurate ER models because we lack background knowledge or the phenomenon is very complex, scientific inference with ML models may thus be the only viable alternative.

Implications
Adopting a phenomenon-centric perspective on IML allows us to answer a variety of questions that were puzzling from a model-centric perspective: Which questions can be addressed with IML property descriptors?
IML property descriptors can help retrieve relevant phenomenon properties i.e. properties derived from the conditional distribution P(Y | X). Which phenomenon properties are relevant is context-specific and up to researchers to identify. While formulating questions, researchers must be aware that supervised ML models are only representational of associative structure and not the underlying causal mechanism (see Section 5.3).
Why use (I)ML for inference? Supervised ML can help draw scientific inference when sampling from X is easy but sampling from Y is difficult, e.g. when Y is hard to measure or determined only in the future. In such situations, analyzing both the model and the data with IML methods can allow for better conclusions than analyzing just the data -the ML model fills the gaps by interpolation. Extrapolation to out-of-distribution data is generally not a strength of ML and can lead to incorrect conclusions; such extrapolations should only be trusted if the learning algorithm incorporated a powerful and suitable inductive bias.
When sampling from X is difficult or the property of interest can be computed more reliably by other means, we advice against using IML for inference.

How important is model performance in inference?
If the model is a poor approximation or representation of the modeled phenomenon, the conclusions we draw from that model are unreliable (Cox 2006, Good andHardin 2012).
Thus, a good fit is vital for gaining reliable knowledge.
Note that even for the optimal model, there remains the so-called Bayes error rate, an irreducible error arising from the fact that X does not completely determine Y (Hastie et al. 2009). Thus, high error does not necessarily flag a low-quality model, but rather may indicate that X provides insufficient information about Y.
What kind of data should be used for IML? Many IML methods (e.g. Shapley Values, LIME, etc.) rely on probing the ML model on permuted data (Scholbeck et al. 2019). These artificial "data" may never occur in the real world. This may be useful to audit the model, but if we want to learn about the world, artificial data is supposed to credibly supplement observations. Our analysis therefore substantiates the criticism of Hooker and Mentch (2021), Hooker et al.
(2021), Mentch and Hooker (2016) concerning the permutation of features irrespective of the dependency structure in the data.

Open Problems
There are several open issues that we have not addressed: What about non-tabular data? For some data types, such as images, audio, or video data, it is extremely difficult to formulate scientific questions only in terms of low-level features such as pixels or audio frequencies. To follow our approach, we need a translation of high-level concepts (e.g. objects in images or words in audio) that scientists can use to formulate their questions into low-level features (e.g. pixels or audio frequencies) that the model works with.
Such translations are notoriously difficult to find; deep learning may help here (Jia et al. 2013, Zaeem and Komeili 2021, Koh et al. 2020).
How to assess if data is realistic? In IML, we often need to augment our data. However, using unrealistic data is highly problematic for scientific inference, as mentioned earlier.

Other Forms of Scientific Inference With ML
In this paper, we focused exclusively on scientific inference with supervised learning ML models on i.i.d. data. For this setting, there is sufficient theory in both statistical decision theory and IML research to provide secure epistemic foundations for scientific inference. We have explained what we can learn about the conditional distribution of Y given X. We can even learn that X contains little information about Y to predict it, which is scientifically interesting (Taleb 2005, Shmueli et al. 2010. However, many questions that scientists regularly face are of a different nature and go beyond conditional distributions.

Unsupervised Learning.
Unsupervised learning is concerned with estimating aspects of the joint distribution P(X 1 , . . . , X n ). Unsupervised learning is hard as it typically targets a high-dimensional joint distribution and, often, lacks a clear measure of success (Hastie et al. 2009, p486). In principle, our five-step guide is also applicable to unsupervised learning, however, we lack a theoretical counterpart to optimal predictors.
Causal Learning. The observational joint probability distribution is interesting, but it remains on rung one of Judea Pearl's ladder of causation -the associational level (Pearl and Mackenzie 2018). What scientists are often much more interested in is answering causal questions such as average treatment effects (rung 2) or counterfactual questions (rung 3) (Salmon 1998, Woodward andRoss 2021). Laura may be interested not only whether students' language and math skills are associated (rung 1), but also in whether the provision of tutoring in Portuguese affects students' math skills (rung 2) or whether a specific student (who is not a native Portuguese speaker) would have done better in math if she had received Portuguese tutoring at a young age (rung 3).
Supervised ML models only represent aspects of the observational distribution (rung 1) and therefore generally do not allow answering causal questions. As a consequence, the IML descriptors of the models also generally do not allow causal insight into the data. Many IML papers that discuss causality (Schwab and Karlen 2019, Janzing et al. 2020, Wang et al. 2021, Heskes et al. 2020) are only concerned with causal effects on the model's prediction, which do not necessarily translate into a causal insight into the phenomenon.
In order to answer causal questions, causal models should be used instead. 21 To learn a causal model, we must gather interventional data and/or make strong, untestable assumptions. Causal inference constitutes thus a challenging problem and remains an active area of research (Heinze-Deml et al. 2018, Kalisch and Bühlmann 2014, Constantinou et al. 2021, Peters et al. 2017).
In certain situations, ML models can nevertheless be useful for causal inference. Firstly, if all predictor variables are causally independent and the prediction target is caused by the features, the causal model interpretation implies the causal data interpretation. Secondly, associative models in combination with IML can help estimate causal effects even in the absence of causal independence if they are in principle identifiable by observation. For example, the partial dependence plot coincides with the so-called adjustment formula and therefore identifies a causal effect if the backdoor criterion is met (and the model optimally predicts the conditional expectation) (Zhao and Hastie 2021). Thirdly, when there is access to observational and interventional data during training, training ML models with invariant risk minimization yields models that predict accurately in interventional environments (Peters et al. 2016, Pfister et al. 2021, Arjovsky et al. 2019). For such intervention-stable models, IML methods that provide insight into the effect of interventions on the prediction also describe causal effects on the underlying real-world components (König et al. 2021a).
Another way in which ML supports causal inference is by facilitating practical scientific inference relying on complex mechanistical models, frequently implemented as numerical simulators. Indeed simulators can represent complex, causal, dynamics in an ER fashion, but often at the price of an intractable likelihood and thus expensive inference. A variety of new ML methods for likelihood-free inference on simulators (Cranmer et al. 2020) allows to estimate a full posterior distribution over ER parameters for increasingly complex models.
While supervised learning learns from a fixed dataset, reinforcement learning (RL) systems are designed to act and can therefore assess the effect of interventions. As such, RL models can be designed to provide causal interpretations (Bareinboim et al. 2015, Zhang and Bareinboim 2017, Gasse et al. 2021.

Conclusion
Traditional scientific models were designed to satisfy elementwise representationality. This allowed scientists to directly inspect model elements to learn about Nature. Although ML models do not satisfy elementwise representationality, we have showed that it is still possible to learn about the phenomenon using them. All we need to do is to interrogate the model with suitable IML property descriptors. We have shown how such descriptors must be designed to enable scientific inference.

Appendix B Dataset
Figure 8 gives a descriptions of the different features and is copied from Cortez and Silva (2008). In our trained models, we only used the final G3 student grades. The data was collected during 2005 and 2006 from two public schools, from the Alentejo region in Portugal. The database is collected from a variety of sources from both school reports and questionnaires. Cortez and Silva (2008) integrated the information into a mathematics dataset (with 395 examples) and a Portuguese language dataset (649 records).