{'fraction_non_alphanumeric': 0.035326086956521736, 'fraction_numerical': 0.013847528290649195, 'mean_word_length': 4.560960463672117, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explaining Adverse Actions in Credit Decisions Using Shapley Decomposition

When a financial institution declines an application for credit, an adverse action (AA) is said to occur. The applicant is then entitled to an explanation for the negative decision. This paper focuses on credit decisions based on a predictive model for probability of default and proposes a methodology for AA explanation. The problem involves identifying the important predictors responsible for the negative decision and is straightforward when the underlying model is additive. However, it becomes non-trivial even for linear models with interactions. We consider models with low-order interactions and develop a simple and intuitive approach based on first principles. We then show how the methodology generalizes to the well-known Shapely decomposition and the recently proposed concept of Baseline Shapley (B-Shap). Unlike other Shapley techniques in the literature for local interpretability of machine learning results, B-Shap is computationally tractable since it involves just function evaluations. An illustrative case study is used to demonstrate the usefulness of the method. The paper also discusses situations with highly correlated predictors and desirable properties of fitted models in the credit-lending context, such as monotonicity and continuity.


Introduction
In credit lending, an adverse action (AA) occurs when a financial institution declines an application for credit. This covers several situations: i) a refusal to grant credit in substantially the amount or terms requested in an application, ii) a termination of an account or an unfavorable change in the terms of an account; or iii) a refusal to increase the amount of credit available to an applicant. The institution is then required to provide clear reasons for the AA or to let the applicant know that they are entitled to request an explanation (Regulation B of the Equal Credit Opportunity Act, www.federalreserve.gov/boarddocs/supmanual/cch/fair_lend_reg_b.pdf; and Fair Credit Reporting Act, www.consumerfinance.gov/compliance/compliance-resources/otherapplicable-requirements/fair-credit-reporting-act/). Section 6 discusses requirements of shape constraints such as monotonicity and continuity in the fitted model. The paper concludes with some remarks.
There are several approaches in the literature for explaining the fitted models using machinelearning algorithms. Some of these, such as Local Interpretable Model-Agnostic Explanations or LIME (Ribeiro, Singh, & Guestrin, 2016), involve fitting a local linear model at a point of interest and using the coefficients of the fitted model to explain the result locally. These techniques will not work for the AA explanation problem when the declined and accepted points, and , are far apart and a local linear model does not provide a good fit.

Problem Formulation 2.1 Background
Let ( ) be the predicted PoD of an applicant with predictor = ( 1 , … ), obtained by fitting a model to historical data. Let be the suitably chosen threshold. The credit-decision algorithm will approve a future loan application with attribute * if ( * ) ≤ and decline otherwise. In practice, ( ) is often developed in terms of a link function such as Since the model is likely to be simpler or more interpretable in terms of ( ), we will develop explanation in terms of the differences in ( ) rather than ( ). However, the arguments will also work with ( ). The AA explanation involves providing information to the applicant why their loan is declined. A common approach is to compare their loan characteristic, , with , those of another applicant whose loan would be approved ( Figure 1). Specifically: a) For each customer who was declined, one selects a reference point for comparison in the "acceptance" region of the decision space. It can be the same point for all declined applicants, or it can depend on the particular declined applicant; b) Decompose the difference in default probabilities between the declined and accepted applicants and attribute them to differences in the predictors.
Mathematically, this involves decomposing the difference as where ( , ) is the contribution of the −th predictor to the difference. This will allow the institution to attribute the AA to important predictors. In the rest of the paper, we denote ( , ) as simply .

Selecting a reference point for comparison
Consider the illustrative diagram in Figure 1 that portrays the decision boundary separating regions where loans are declined from where loans are accepted. Here = ( 1 , 2 ) denotes the loan application that has been declined. We briefly consider a few options for selecting a reference point for comparison in the "accept" region. In practice, this choice will depend also on many other practical considerations. We discuss it here only to provide some concreteness. Any reference point will work for our methodology. a) A common practice is to choose the reference point at some high values of the predictors in the acceptance region, typically maximum or close to maximum values. Arguably, a more reasonable choice would be some percentiles (such as median) of these predictors. b) From the loan applicant's perspective, the most favorable point is the one with the shortest distance from to the decision boundary. This would provide the applicant with information on the smallest changes they have to make to be successful (sometimes called counterfactuals). There are, however, some potential challenges with this approach: i) the decision boundary is estimated from data and one should also consider the inherent variability; and ii) the reference point will vary with the declined loan and hence may be challenging to explain to multiple customers. c) The above choices assume that the loan applicant will be able to improve their credit worthiness equally well for the different attributes. In practice, they may be able to manipulate a subset more easily than others. In this case, one can consider reference points that have the shortest distance in a lower dimensional subspace of the attributes. Figure 1 shows this reference point in terms of the shortest distance in 1 −dimension.
In the rest of this paper, we treat the reference point generically and do not consider any particular choice. We will use the following notation:  = ( 1 , … ) denotes the point, and values of the predictors, for the declined application; and  = ( 1 , … ) denotes the reference point, and values of the predictors, for the successful application that has been selected for comparison. This point can depend on but we will suppress any dependence.

AA Explanation in Important Special Cases
We start with special situations and develop the explanation method from first principles.

Additive Models with No Interactions
Suppose the fitted model is linear without interactions: The decomposition in Eq (2)

Two-Factor Linear Model with Interaction
The explanations become less obvious in the presence of even simple interactions. Consider a two-factor linear model with interaction: ( ) = 0 + 1 1 + 2 2 + 12 1 2 . Now, [ ( ) − ( )] = 1 ( 1 − 1 ) + 2 ( 2 − 2 ) + 12 ( 1 2 − 1 2 ). (3) The difference in interactions, last term on the right-hand side in Eq (3), involves both 1 and 2 and has to be further decomposed and allocated to each separately. Before providing a solution to this special case, we consider a more general model with two factors in the next section.  Similarly,

General Model with Two Factors
To compute the attributions, we have to do only four function evaluations (at the four corners of the rectangle in Figure 2) and compute the differences in Eq (4) and Eq (5).

Revisiting the Simple Case in Section 3.2
Applying Equations (4 -5) to the interaction term in Eq (3), we get 12 ( 1 2 − 1 2 ) = Note the symmetry in the decomposition on the right-hand side in Eq (6). Combining the main effects in Section 3.2 with the interactions, we get the decompositions for the two factors as

factors with Second Order Terms Only
The results in Section 3.4 generalize easily to factors. To ease into the general case, suppose first = 3. Then, We can apply the decomposition in Section 3.3 to each of the three pairwise interactions. For example, This extends in a straightforward manner to models with predictors of the form The attribution for ( is fixed in the summation below) is: To compute the decompositions. We have to do a total of ( ℎ 2) × 4 function evaluations: ( ℎ 2) sub-models of two-factors, and each requiring four evaluations.

Two-factor Models Expressed as Orthogonal Main Effects and Interactions
The -factor interaction model in Eq (8) can be re-expressed as main effects (GAM) and two-factor functional interactions that are constrained to be orthogonal: These have been called GA2M or GAMI in the literature. We can express the AA explanations in Eq (9) in terms of main effects and interactions as

Model with Three Factors
. Consider the path in Figure 3 from to There are six such paths in Figure 3, and [ ( ) − ( )] can be written in terms of the function evaluations of the points on those paths, as in Eq (10). We can also write the difference on the left hand-side as an average of the expressions for the six paths. To get the contribution 1 for 1 , we collect all the terms involving differences in 1 and 1 with the settings of ( 1 , 2 ) fixed. Take the term in Eq (10) is an example. Rearranging terms, Using similar arguments, The computation in this case require function evaluations at 2 3 = 8 corner points of the cube in Figure 3.

General Models with Only Three-Order Terms
Consider the case with K factors and suppose the model depends on three-order submodels only: The results in the last section can be generalized in a straignforward way to get the attribution for ℓ , the ℓ −th variable, as the sum across ( , ) with ≠ and ℓ fixed: Now, we have a total of( ℎ 3) × 8 function evaluations: ( ℎ 3) sub-models of three factors with each requiring 8 function evaluations.

General Case
So far, we have developed AA explanation for special cases from first principles in an intuitive manner. We now consider the general case of an arbitrary model with factors, and show that our results are special cases of Shapley decomposition (Shapley, 1953) and Baseline Shapley (Sundararajan and Najmi, 2020).

Shapley Decomposition
Given an arbitrary function ( ) of predictors, the goal is to decompose the difference [ ( ) − ( )] into individual components of each predictor. This problem was addressed in Shapley (1953) in the context of cooperative games. His formulation was as follows. There are K players involved in a coalition in a cooperative game, and the goal is to distribute the gains from the game among the players. Let = {1, 2, … , }, { } be the player of interest, \ denote a subset that does include { }, is any subset of , | | is its cardinality, and ( ) is some value function for the game that is associated with the set . Shapley (1953) showed that the unique distribution scheme that satisfies several desirable axioms is for = 1, … , . The summation in Eq (11) is over all possible subsets , and the combinatorial coefficients arise from the number of such subsets. Owen (2014) proposed using Eq (11) with the value function ( ) = ( ( ( )| )) as a measure of global variable importance in complex statistical and ML models. This is an alternative to the commonly used variable importance analysis based on permutation (Breiman, 2001). However, when the sample size and the number of variables (and hence the number of possible subsets) are large, computing the value function (variances of the conditional expectation) is challenging. This has impeded the use of Global Shapley for variable importance analysis.
Consider a different (local) explanation problem in ML algorithms. Given a fitted model ( ) and a point of interest * in the prediction space, the goal is to decompose ( * ) into contributions of the variables. Lundberg and Lee (2017 proposed using Eq (11) with the value function as ( ) = ( ( )| ) and referred to it SHapley Additive explanation (or SHAP). While this is simpler to compute than global Shapley, it is still time-consuming when , and hence the number of sub-models, is large. This has led to the development of various approximations in the literature such as Kernel and Tree SHAP (Lundberg, Erion, and Lee, 2018), and the use of marginal expectations.
Sundararajan and Najmi (2020) proposed another alternative to local explanation called Baseline-Shapley (referred to as B-Shap). Instead of using integrations, it involves computing just the difference of the functions at each data point vs at an average reference point, and hance is easier to compute. We adapt in in a straightforward manner for AA explanation in the next section.

Baseline-Shapley (B-Shap) for AA Explanation
We use a slightly different notation in this section. Let = \{ }, the subset of without { }. Then, , the AA explanation for the −th variable is: This is just Eq (11)  As noted earlier, B-Shap involves only function evaluations, so it is computationally less expensive. For a model with −variables, there are 2 function evaluations. When the model structure is simpler, as in the special cases in Section 3, the computations will be even less demanding.

Highly Correlated Predictors
Given the automated nature of feature selection and feature engineering in ML algorithms, modelers tend to include a large number of predictors in the model. So it is likely that some of the predictors in the model will have moderate to high correlation. If so, some values involved in computing the differences in Eq (12) may lie outside of the data envelope where the fitted model is unreliable. It would make sense then to treat the correlated predictors jointly in making attributions. To make this notion concrete, suppose we have three predictors ( 1 , 2 , 3 ) and ( 1 , 2 ) are strongly correlated. In this case, we can view ( 1 , 2 ) as a single predictor 1 and treat it as a two-dimensional attribution problem with predictors ( 1 , 3 ). We take 1 = ( 1 , 2 ) and 1 = ( 1 , 2 ). We will see an illustration of this joint attribution to the illustrative example in Section 5. Of course there are other ways to treat this problem, including use of principal components or other dimension-reduction techniques.

Illustrative Application
We use an illustrative example to demonstrate the application of the results. We first have to develop the model for PoD from historical data. We use a simulated dataset that is intended to mimic application to credit cards. The predictors and their marginal distributions were obtained from credit bureau data. Their correlation structure as well as the input-output model were simulated, but they were intended to mimic real-world behavior. Readers interested in just the demonstration of AA explanation can skip the next two sub-sections and go directly to Section 5.3.

Description of Dataset
We simulated historical information on 50,000 current customers from the financial institution for a comparable credit product. The response was binary with = 1 if account defaulted during an 18-month period and = 0 otherwise. There was also information on 10 credit-related predictors at the time the customers applied for the product. The first two columns of Table 1 provide the variable names and descriptions. Subject-matter expertise suggested that the relationship between PoD and six of the 10 variables should be monotone. This information provided the third column of Table 1. For our analysis, we standardized x1 = average card balance, x2 = credit age, and x4 = total balance. We applied a log transformation to x8 = total amount currently past due. Finally, in order for the direction of monotonicity to be the same for all variables, we flipped the sign of x2 = credit age, and called it credit age flip. The correlation structure of the variables is given in Figure 4, and we see strong correlation structure among subsets of variables. This is not surprising since x1 and x4 are both measures of balance, x3 and x5 are measures of utilization, x6-x8 are measures of delinquency (past due), and x9 and x10 are measures of number of inquiries. This is typical in credit applications where multiple dimensions of the same latent variable are collected and used in the model. We simulated the correlation structure to mimic this real world scenario.

Model Development, Performance, and Diagnostics
We fitted two different models: an unconstrained feedforward neural network (FFNN) and a monotone NN (mono-NN) that incorporates the shape constraints in Table 1. The mono-NN algorithm is based on the two-stage approach in Liu, Han, Zhang and Liu (2020).
The dataset with 50,000 observations was divided into training (80%), validation (10%), and testing (10%). We tuned the hyper-parameters of the FFNN algorithm and ended up with three layers and nodes [23,35,15] and learning rate of 0.004. The mono-NN also had three layers with nodes [35,15,5] and learning rate 0.001.  Table 2 shows the predictive performances. Mono-NN has a lower training AUC but higher test AUC, indicating it generalizes to the test dataset better. It also has a smaller gap between train and test AUCs, suggesting it might be more robust. Figure 5 are variable importance plots for the mono-NN model. The left panel shows results for all 10 predictors. Credit age (x2) is by far the most important followed by x9 = number of credit inquiries in last 12 months. Note, however, that x10 = number of credit card inquiries in the last 12 months is the least important. This may be because x9 and x10 are highly correlated. Similar issues arise with x6, x7, and x8 (measures of delinquency or accounts past due), whose effects may be distributed. To address possible interpretation problems from high levels of correlation, we collapsed the 10 to get five predictors that measure intrinsically different quantities. The right panel in Figure 5 shows the joint variable importance. Credit age remains the most important variable, but now delinquency (accounts past due) and inquiries are showing up collectively as important. These results are more meaningful in this application.  Figure 6 shows the partial-dependence plots (PDPs) describing one-dimensional input-output relationships: x-axis corresponds to the values of the variables and y-axis corresponds to the logodds of PoD. Recall from Table 1 that the Mono-NN was constrained to be monotone in x2 and x6-x10, and the PDPs retain this shape constraint. In fact, x7, x9 and x10 are mostly linear. In addition, the shape for x3 is also monotone. The average cash balance = x1 and utilization = x5 have quadratic behavior, while total balance = x4 has a more complex pattern.

Figure 7. Two-Dimensional PDPs for Selected Predictors for Mono-NN
We also examined the interactions among the predictors using H-statistics and twodimensional PDPs (Friedman, 2001). Figure 7 shows the two pairs of variables for which there was some degree of interaction. The plots shown are slices of one-dimensional PDPs for the variables on the x-axis with the colored curves corresponding to different slices of the other variable. If there is no interaction, the curves will be parallel. The left panel shows interaction between x3 (percent of all open credit products with over 50% utilization) and x5 (percentage of open credit cards with over 50% utilization). They are both measuring credit utilization. The right panel shows interaction between x1 (average monthly debt) and x4 (total debt standardized). These two are both measuring debt.

Adverse Action Explanation Using B-Shap
Finally, we turn to the demonstration of AA explanation in this example. For illustrative purposes, we chose the PoD threshold as = 0.25 . In other words, any future credit card application with ( * ) ≤ 0.25 will be approved; otherwise it will be declined.
The reference point in the acceptance space was selected as the 75 th percentile of each predictor (see second column in Table 3). Its PoD is ( ) = 0.016. We selected two points in the declined region: (third column) and (fifth column) with ( ) = 0.294 and ( ) = 0.858. Columns 4 and 6 in Table 3 Table 3. Values in parenthesis are percentages, component values divided by the total difference.
Consider Column 4 first. There is no difference in the values of x6, x7, and x8 (number of accounts and amount past due) between and , so they do not contribute anything. The values of x3 and x5, which deal with utilization, are not very different, so their contributions are also small. The values of x1, average balance, are quite different for the declined and reference points, but we see from Figure 5 that this variable is not as important. So the contribution is relatively small (3.5%). On the other hand, the values of x2 in the first and third columns are quite different. Moreover, Figure 5 shows it is an important variable in the model. So the contribution is the largest, in absolute sense and percentage of about 60. The next biggest contribution is from x9, number of credit inquiries in the past 12 months, with a contribution of about 31%. Note that x9 has the second largest important score in the left panel of Figure 5. Surprisingly, the difference for x10 between the first and third columns is even larger, and yet the contribution is much smaller. This is due to the fact that x10 has the lowest score in the left panel of Figure 5. While this may be reasonable (credit inquiries in the past 12 months are more important than past 24 months), it may also be an artifact due to the high correlation between the two variables (see Figure 4). Therefore, we collapsed the highly correlated variables and considered the same subset of five variables on the right panel in Figure 5. We then computed the joint B-Shap explanation for these variables as described in Section 4.3. The second column in Table 4 provides these joint attributions. These five predictors are more interpretable in terms of measuring distinct underlying measures of creditworthiness, and it is easier to explain the results to applicants in terms of these. The final AA explanation for this case is simple: the two important predictors are credit age and number of inquiries, with the former much more important. Decision makers can now use the numerical results, as needed, to provide further information to the applicants.

M-NN
The comparison between and , the first and fourth columns, can be done in a similar manner. The difference in the defaults probabilities is now much larger. After decomposing the difference (column 5), we see that the biggest contributors are x2 (credit age) and x6, x7, and x8 (number of accounts and amount) past due. Variables x6-x8 are highly correlated (see Figure 4), so we again computed joint attributions for the variables after collapsing them into five. The third column in Table 4 shows the joint decomposition and attributions. Number of accounts/amount past due is by far the most important with credit age also as a significant contributor, and the explanation is again quite straightforward even though the underlying model is complex.

Discussion: Shape Constraints with Credit-Decision Models
In credit decision applications, the fitted model should satisfy certain desirable properties. We discuss two of them in this section,

Monotonicity
For the illustrative application in Section 5, subject-matter knowledge suggested that six of the 10 variables have a monotone relationship with PoD. Thus, we fitted a shape-constrained NN where the fitted model satisfied the monotone properties. We argue that this should be a general requirement for all models that deal with credit decisions, namely, the fitted model should satisfy shape constraints for relevant predictors. If we expect POD to be increasing or decreasing with an underlying measures of credit worthiness, then the fitted model should satisfy the property (see also Chen et al., 2022). Decisions based on models that violate such basic requirements will not provide reasonable AA explanations.
There are techniques and algorithms in the literature to fit monotone versions of ML algorithms. Liu et al. (1992) proposed an iterative NN algorithm that fits a penalized loss function (with a penalty for monotonicity), checks if the property is satisfied, increases the penalty if not, and iterates. This is the algorithm used in our example in Section 6. See also Cano et al. (2019) for other methods.

Continuity
Another desirable property is that the fitted POD should vary continuously as a function of continuous predictors. To illustrate this, consider the illustrative example in Figure 8 with two continuous predictors 1 and 2 . Suppose we fit a discontinuous (piecewise constant) tree model which yields the fitted values and decision boundary in Figure 8. Here 's are the constants corresponding the fitted PoDs, and the red line is the decision boundary separating the "accept" and "reject" regions. Take, for example, the red and green points ( 1 , 2 ) and ( 1 , 2 ), in Figure 8, corresponding to declined and accepted applications. Their PODs are 3 and 5 , and this difference will remain the same regardless of how close the two points are to each other but remain in opposite sides of the decision region. There is no way one can rationalize such a decision rule and explain it to applicants. One of the commonly used approaches in credit-scoring models does in fact bin the continuous variables and fits piecewise constant models. This has been justified as a way to do feature engineering. As we argue above, such models cannot be justified if one has to provide meaning AA explanations. More broadly, tree-based ML algorithms such as Gradient Boosting and Random Forests all use piecewise constant fitting. Since they are ensemble models are based on many trees, the discontinuities may be small. However, they still pose conceptual challenges for AA explanation.

Concluding Remarks
This article proposes the use of B-Shap as a model-agnostic method for explaining adverse actions in credit decisions. It decomposes the difference of the PoD or its log-odds, evaluated at two points, into components corresponding to the different predictors. The method was developed from first principles. It is intuitive, so easy to understand and explain. The computations are easy for lower-order models. For arbitrary models, the computations are tractable even for dimensions as high as 20. When the number of predictors is larger, one would expect them to have significant correlations among subsets. In such case, as we have shown through our example, it is more meaningful to collapse them into fewer, distinct measures of credit-worthiness and do AA explanation in the reduced space.