{'fraction_non_alphanumeric': 0.030903977971761674, 'fraction_numerical': 0.0072939246587380635, 'mean_word_length': 4.538449059052564, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weight-of-evidence 2.0 with shrinkage and spline-binning

In many practical applications, such as fraud detection, credit risk modeling or medical decision making, classification models for assigning instances to a predefined set of classes are required to be both precise as well as interpretable. Linear modeling methods such as logistic regression are often adopted, since they offer an acceptable balance between precision and interpretability. Linear methods, however, are not well equipped to handle categorical predictors with high-cardinality or to exploit non-linear relations in the data. As a solution, data preprocessing methods such as weight-of-evidence are typically used for transforming the predictors. The binning procedure that underlies the weight-of-evidence approach, however, has been little researched and typically relies on ad-hoc or expert driven procedures. The objective in this paper, therefore, is to propose a formalized, data-driven and powerful method. To this end, we explore the discretization of continuous variables through the binning of spline functions, which allows for capturing non-linear effects in the predictor variables and yields highly interpretable predictors taking only a small number of discrete values. Moreover, we extend upon the weight-of-evidence approach and propose to estimate the proportions using shrinkage estimators. Together, this offers an improved ability to exploit both non-linear and categorical predictors for achieving increased classification precision, while maintaining interpretability of the resulting model and decreasing the risk of overfitting. We present the results of a series of experiments in a fraud detection setting, which illustrate the effectiveness of the presented approach. We facilitate reproduction of the presented results and adoption of the proposed approaches by providing both the dataset and the code for implementing the experiments and the presented approach.


Introduction
Classification is a well-studied machine learning task that concerns the assignment of instances to a set of outcomes. Classification models support the optimization of managerial decision making across a variety of operational business processes. For instance, fraud detection models classify instances, such as transactions or claims, as fraudulent or nonfraudulent [39]. This allows for the efficient and effective allocation of limited inspection capacity by selecting the most suspicious cases for investigation by a human fraud analyst [5]. Credit risk models, on the other hand, assess the risk connected with providing credit to customers, and this risk can be used to construct optimal portfolios of loans or other lines of credit [3,6].
A wide variety of classification models have been proposed in the literature. These proposals range from very complex models including neural networks, support vector machines and ensemble methods to more elementary models such as logistic regression and decision trees [17]. Some of the more complex models have been shown to outperform the simpler classification techniques in various real-life classification tasks [7,34,21,13,48].
In industry, however, simple logistic regression currently remains among the most frequently used approaches for developing classification models across various fields of application [37,4,21,13,11]. Its popularity may be explained by the presence of industry regulations, e.g., the Basel regulatory framework for the banking industry, which requires the resulting model to be both interpretable [26] and accurate. Logistic regression is widely perceived as offering the best balance between both objectives. Other possible explanations are the broad expertise and experience in using logistic regression that exists in industry, but follow-the-herd behavior and some degree of inertia and resistance to change may explain its enduring popularity. Moreover, the superior performance of the more complex models can strongly depend on the task at hand. On tabular datasets, which are commonly encountered in the context of credit scoring and healthcare analytics, they have been shown to provide only marginal performance gains [21,29].
Aside from the development of classification models and techniques for learning them, a different approach to improving the final model is to focus on pre-and post-processing. In contrast to studies on learning models and post-processing techniques [40,19], relatively few studies focus on preprocessing data. The goal of preprocessing is to optimally prepare the data (e.g., through transformation) to maximize the predictive power and out-ofsample performance, or, importantly, to improve the interpretability of the resulting model. Specifically, we identify a lack of approaches that allow us to optimally transform nonlinear patterns and categorical variables with high cardinality for incorporation in linear models to achieve an interpretable yet powerful classifier [27]. Currently, the weight-of-evidence (WOE) approach appears to be frequently used to this end, as it offers a good balance between interpretability and predictive power, and it is complementary with and similar to logistic regression [36,2]. For categorical variables with many categories, however, WOE may lead to overfitting. Moreover, WOE does not have an integrated binning approach for optimally merging categories or discretizing continuous predictors.
In this article, we present an integrated WOE-based approach for optimally transforming predictor variables, which mainly improves upon the existing WOE approach in cases with nonlinear predictor variables (continuous or ordinal) and categorical predictor variables with high cardinality.
The goal of this preprocessing method is to maximize both the predictive power and interpretability of logistic regression models (and more generally, generalized linear models). The proposal is based on generalized additive models in combination with exact univariate k-means clustering and shrinkage estimation. The presented approach is experimentally evaluated; an illustration of the use of the proposed approach and an indication of its merits are provided. An open source implementation of the method is provided in the digital annex to this paper to enable peer researchers to reproduce and verify the presented results and allow practitioners to adopt the method for practical use. This paper is structured as follows. In the following section, we present the standard methodology that uses logistic regression and weight-of-evidence, and we expand upon this approach in Section 3. In Section 4, we present experimental results obtained from a fraud detection case and a credit risk case, and in Section 5, we conclude the paper and present directions for future research.

Background methodology
Consider a model with a binary response Y and p continuous predictors X = (X 1 , . . . , X p ). The goal is to model the conditional mean p The classical logistic regression model, which is part of the family of generalized linear models (GLMs) [28], assumes a linear relationship between the predictor variables and the log-odds of the event Y = 1. More specifically, we have where β 0 denotes an intercept and β = (β 1 , . . . , β p ) denotes a vector of model parameters. This model can be reformulated in terms of probabilities as P (Y = 1|X = x) = 1 1 + e −(β 0 +βx) .

continuous variables with nonlinear effects on the log-odds
Categorical variables are often one-hot encoded (also known as "dummy encoding"), after which they can be included in the model as numerical variables. This has the drawback that a categorical variable with N categories leads to N −1 variables. If N is large, this leads to considerable variability in the estimation process and usually many insignificant predictors. One way to avoid this problem is by converting the categorical variable into a continuous variable by using a weight-of-evidence transformation. The weightof-evidence (WOE) transformation of a categorical predictor is commonly defined as follows. Suppose that we have a category j with N j elements.
Denote by P j the number of true cases in our category and by F j the number of false cases in our category. Additionally, let P be the total number of true cases in the data and F be the total number of false cases in the data.
The WOE value of category j is then given by: The WOE transformation usually provides an elegant solution, but since it is based on the estimation of a proportion, its variance can be high when there are categories with few observations, which is common for categorical variables with high cardinality.
Continuous variables are modeled by logistic regression as having linear effects on the log-odds of the response. While this is often reasonable, there can be variables that do not satisfy this assumption. This happens unexpectedly but sometimes by design, as illustrated in the following example. Suppose that we want to predict whether a transaction is fraudulent based on a single predictor X t that characterizes the time at which the transaction was made (i.e., taking values within [0, 24)). Now suppose that we make the reasonable assumption that the influence of the time on the probability of a transaction being fraudulent is roughly continuous, and we interpret X t being close to 24 as X t being close to 0. Then, we would have In terms of log-odds, this would imply that β 0 = lim which clearly can only be satisfied when β 1 = 0. In other words, under the assumptions above, the only relationship that can be fit is a constant relationship, which is not of much interest. This example illustrates that some variables display nonlinear relationships with the response by design.
One way to incorporate the nonlinear effects of continuous predictors on the log-odds is to use the generalized additive model (GAM, [16,44]) for logistic regression: where f 1 , . . . , f p are arbitrary smooth functions of the predictor variables Throughout the remainder of the article, we make the assumption that the conditional expectation of Y can indeed be adequately modeled through a GAM in the predictor variables. This assumption entails that the nonlinear effects are sufficiently smooth functions of the predictors. Furthermore, we assume that the number of variables p is considerably smaller than the number of observations n which guarantees stability in fitting the model. In case this last assumption would not be met, one could resort to regularized GAMs and apply the propsed methodology in that setting.

Methodology
In the following, we describe our proposal to address the issues described in the previous section. The underlying goal is to develop a powerful predictive model while maintaining interpretability by allowing the incorporation of nonlinear effects within a GLM and by improving upon the traditional WOE-based binning process. i.e., each element in a given category j is assigned the value

(Local) shrinkage of WOE
wherep j denotes the proportion of successes (e.g., fraudulent transactions) in category j. The equivalence with the earlier definition in Eq. 1 can be seen as follows. With the notation introduced before, we have that WOE j = log F . Therefore, both values differ by only a constant, which typically does not play a role in most statistical or machine learning models. As an example, the constant disappears in the intercept of a GLM. It is worth noting that sometimes categories withp = 0 orp = 1 can occur, and these lead to undefined WOE values. In those cases, we can slightly adjust the WOE by introducing a small offset c with 0 < c < 1 and replacep = 0 withp = c n j andp = 1 witĥ p = 1 − c n j . Note that this offset disappears as the number of observations in the category becomes large (i.e., when n j → ∞). We use c = 0.01 by default.
In practice, categories are often merged to avoid this boundary case, but this merging introduces a certain level of arbitrariness. In particular, it raises the question as to whether all possible combinations of categories should be considered as possible merging candidates. Additionally, this technique does not use the performance or quality of the final model for evaluating which merges are most interesting. We thus prefer working with a small offset, after which we can deal with the WOE values in a rigorous way.
For a category with a small number of observations n j , the estimation of p j (and the corresponding WOE j ) has a high variance, often yielding unreliable estimates. This is more likely to occur in categorical variables with many levels. To address this issue, we consider the shrinkage estimation of the proportion of successes in each category j. The shrinkage estimator of a proportion is given by [24]: wherep denotes the proportion of successes calculated over all possible values of j (i.e., over all categories). We thus effectively shrink the proportion of successes towards the sample mean. The shrinkage coefficient b j determines the amount of shrinkage: b j = 0 corresponds to no shrinkage, whereas b j = 1 corresponds to taking the population proportion. The value of b j is chosen to minimize the expected mean squared error over all estimated proportions, which is given by The minimum is given by where v = var(p) is the sampling variance ofp, v j denotes the sampling variance ofp j and σ 2 equals the between-area variance (i.e., var j (p j )) [24].
By plugging the shrinkage estimator into the WOE calculation, we obtain the shrinkage estimator of the WOE values: for each category j. In the rest of the paper, we denote the WOE transformation based on the shrinkage estimation of the proportions by SWOE(·).
In addition to the global shrinkage method described above, which shrinks proportions towards the overall proportion in the data, we consider shrinking the proportions locally. More specifically, we cluster the WOE values using the weighted k -means approach [25,23], where the weights are taken as inversely proportional to the sampling variability of the WOE values.
Note that by the central limit theorem and delta method, it holds that The asymptotic variance of the WOE estimates is thus 1/(np(1 − p)). Denoting the WOE values with z 1 , . . . , z n , we therefore solve the optimization problem given bŷ Note that these weights are small for categories with very few observations, which makes it more likely that these categories are put in the same cluster as other categories. Clustering the WOE values induces local shrinkage, since WOE values that are close together tend to end up in the same cluster and receive a WOE value that is a weighted average of the WOE values in the cluster. In addition to achieving less variability in the estimation of the WOE values, we also obtain a natural "fusing" of similar categories resulting in a categorical variable with fewer categories. This allows for easier interpretation and visualization of the effect of the categorical variable. In the rest of the paper, we denote the WOE transformation based on clustered estimation of the proportions by CWOE(·).
To allow for nonlinear effects of the predictor variables on the log-odds, we revisit the approach of [18] and start from the generalized additive model (GAM) of Eq. 2. After fitting the GAM, the goal is to discretize the fitted spline functions into a limited number of bins. These can then be used as inputs for a classical logistic regression model. As such, we can capture nonlinear effects while greatly improving the interpretability of the model.
Our approach differs from others in three main ways. First, we unite different binning types in one framework consisting of "constrained" and "un- Let us consider an example. Suppose that x j is a variable characterizing the age of a person making a transaction. After fitting the model in Eq. 2, we obtain a smooth function f j (x j ) that linearly influences the log-odds.
Suppose that we want to create bins for this transformed variable. If we apply unconstrained binning, the binning of f j (X j ) would be independent of the value of X j . This means that the resulting bins may combine different age groups. We could have a bin of ages {0 − 20, 80+} and another bin of ages {21 − 79}. While this may be fine in some situations, there may also be situations where the binning process is required to be contiguous in x j to enable a user to interpret or explain the model. This means that the categories cannot "jump" over ages. An example of such a binning result is We would like to emphasize that the choice of binning is primarily a question of user preferences.
Unconstrained binning is arguably the easiest problem. Given a predictor x = x 1 , . . . , x n where i = 1, . . . , n ranges over the observations, consider the transformed values z i = f (x i ). We want to find K disjoint binsB 1 , . . . ,B K for the original observations x 1 , . . . , x n such that within each bin, the corresponding values of z i are roughly homogeneous. This is a univariate clustering problem for which many approaches have been proposed. We propose to optimize the weighted k-means objective function: where w i ≥ 0 are weights such that n i=1 w i = n andz k denotes the mean of all z i values with i ∈ B k (i.e., the cluster center). We choose the weights to be inversely proportional to the variance of the fitted spline function at point x i . Once we obtain the binsB 1 , . . . ,B K , we can transform the original predictor x = x 1 , . . . , x n toz k 1 , . . . ,z kn , where k i denotes the cluster to which observation i = 1, . . . , n is assigned. Alternatively, we can include the predictor as a categorical variable with the categories equal to the cluster memberships. We choose not to do this to avoid the creation of many dummy variables.
The weighted k-means clustering problems can be solved exactly in O(n log(n)) time using dynamic programming. Finally, note that the kmeans approach with all weights equal to 1 is equivalent to Fisher's natural breaks algorithm [12] used in [18]. The issue of choosing the number of clusters K is a challenge in cluster analysis, and a multitude of heuristic approaches exist. Among the more popular methods are the gap statistic [38] and the silhouette coefficient [33]. While these can be used in our setting, they do not take our primary goal of building a solid predictive model into account. Therefore, we adopt a hyperparameter tuning approach and determine the value of K by evaluating the quality of the resulting logistic regression model, aligning the clustering process with our overall objective.
We address this issue in more detail in Section 3.2.
We now turn to the problem of constrained binning. Consider again the transformed variable z i = f (x i ). In contrast to the unconstrained binning scenario, the value of x i now influences the clustering of the z i values.
Suppose without loss of generality that the values of x i are ordered in the relevant order (e.g., the observed ages are listed in ascending order). We are now interested in K bins B 1 , . . . , B K , which each contain disjoint subsets of x 1 , . . . , x n such that if x i , x j ∈ B k for certain i < j ∈ {1, . . . , n}, then Of course, we still want the bins to contain homogeneous values for the corresponding transformed values z i . This problem is equivalent to fitting a step function to the set of bivariate points (x i , z i ), i.e., we look for a piecewise-constant approximation of z i within the clusters of x i . This problem has been considered in many areas, including function approximation, time series analysis and cluster analysis. In the same spirit as the weighted K-means approach, we propose to optimize the weighted K-segments objective function: which is the exact same objective as that of the weighted k-means problem, with the added constraint that the bins need to be contiguous. The weights w i ≥ 0 are again chosen to be inversely proportional to the variance of the fitted function at point x i .
The k-segments clustering can be found exactly in O(n 2 ) time using dynamic programming, but an approximate O(n log(n)) algorithm exists [41]. Alternatively, one could use (evolutionary) regression trees to bin the z i values. However, they are typically slower to compute and do not guarantee a global optimum of the objective function. We propose to simplify the problem as follows. A simple AIC for univariate k-means clustering is [30] AIC = WCSS k + 2k, where k equals the number of clusters and WCSS k denotes the within-cluster sums of squares (i.e., the value of the k-means objective) when clustering into k clusters.

Model building
One could use this formula to select the number of clusters for each clustering problem, but this would not take the performance of the final model into account. Therefore, we adapt this criterion by introducing a parameter that balances the strength of the fit with the number of clusters: As λ increases, we encourage the algorithm to use fewer bins or clusters.
When there are only continuous variables that need to be preprocessed, we propose to use two tuning parameters, λ c and λ uc , for the constrained and unconstrained effects, respectively. Note that it is necessary to distinguish between these two effects since the constrained problem will have a naturally higher WCSS. To tune the model, we thus use the procedure outlined in Table 1.
Tuning of λ c and λ uc Step 1 Bin the unconstrained nonlinear continuous effects using the number of bins k that yields the minimal value of the objective in Eq. 4 with λ = λ uc .
Step 2 Bin the constrained nonlinear continuous effects using the number of bins k that yields the minimal value of the objective in Eq. 4 with λ = λ c .
Step 3 Fit a GLM using the binned effects, possibly including other variables.
Step 4 Evaluate the GLM using the AIC.  Table 2, after which the value of λ cat yielding the lowest AIC of the resulting GAM is retained.
Once λ cat has been determined, we proceed by tuning λ c and λ uc using Tuning of λ cat Step 1 For each categorical variable, find the number of clusters associated with the value of λ = λ cat .
Step 2 Cluster the WOE values of all categorical variables using the appropriate number of clusters found in the previous step.
Step 3 Train the GAM using the splines for the continuous variables and the clustered WOE values for the categorical variables.
Step 4 Evaluate the GAM using the AIC. the previous procedure in Table 1. Note that instead of using the AIC, the tuning parameters can also be tuned using other performance criteria, such as a measure of prediction accuracy, on a validation set (if available) or through cross validation. This requires more data to be available and more computation time but is likely to better guard against the overfitting of the training data. The parameters λ co and λ ca yielding the lowest out-ofsample prediction errors are then retained, and the final model is fit using these values.
We now analyze the computational complexity of the whole pipeline including the tuning procedure. Suppose the data consists of n observations in p dimensions in addition to a univariate response. Furthermore, assume that the continuous variables can be split up in p uc unconstrained nonlinear effects, p c constrained nonlinear effects, and p l linear effects. Also denote the number of categorical variables with p cat so that p = p uc + p c + p l + p cat .
Finally, We assume that the lengths of the grids for the tuning parameters are given by G cat , G uc and G c .
The time complexity can now be analyzed by splitting up the procedure in 2 steps, the first being the tuning of λ cat for the categorical variables, and the second the tuning of the parameters λ uc and λ c for the continuous nonlinear effects.
Step 1 requires, for each value of λ cat , the preprocessing of p cat variables and the fitting of one GAM on X. The preprocessing requires O(n log(n)) time for each categorical variable due to the exact univariate k-means optimization. In total, we thus obtain O(G cat (C GAM + p cat n log(n))), where C GAM denotes the computational cost of fitting a GAM to the data. Note that step 1 is only needed when the WOE values need to be binned. In case shrinkage estimation is used, there is no need for the tuning parameter λ cat and the complexity becomes O(C GAM ). The complexity of fitting a GAM depends on the fitting algorithm and the number of smoothing parameters, but O(np 2 ) is a reasonable assumption given a fixed number of iterations until convergence (see [46,22] for a discussion).
Step 2 requires the separate preprocessing of the constrained and unconstrained effects through spline-binning. This requires O((G c p c + G uc p uc ) n log(n)) time. Additionally, for each combination of λ uc and λ c , the fitting of one GLM is required, which leads to an additional O(G c G uc np 2 ) cost.
Combining the computational cost of both steps together, we obtain a total of O(G cat np 2 + p cat n log(n) )+O((G c p c + G uc p uc ) n log(n))+O(G c G uc np 2 ).
If we assume the sizes of the grid to be constant for increasing n and p, and we further assume that at least one of p uc , p c , p l , p cat is O(p) (which is a worst-case scenario), we obtain an overall complexity of O(n log(n)p + np 2 ).
While this is a manageable complexity, the constant factor may be quite high if the grids for parameter tuning are fine. That said, the optimization over a grid can be easily parallelized to allow for efficient yet precise parameter tuning.

Data
We evaluate our proposal on two datasets. The first is a dataset on The response is a binary variable indicating fraudulent transactions, of which there are 73 in this dataset (i.e., roughly 1 %). Table 3

Experimental design
To illustrate the advantages of the proposed method in several ways, we set up three experiments. The first two are conducted on the credit card fraud data and are meant to illustrate the model building process step-bystep while emphasizing the enhanced interpretability and superior results of the resulting model. The third experiment is a complete comparison of the proposed method on the credit risk data using cross validation. For our experiments, we make use of the R packages mgcv [43], Ckmeans.1d.dp [41], cellWise [31], hmeasure [1], xgboost [9], caret [20] and ROCR [35].

Experiment 1: the effect of spline binning on the fraud dataset
In the first experiment, we use only the continuous variables to predict fraudulent transactions. To quickly scan for the variables that may have potential nonlinear effects on the response, we fit a GAM on the continuous predictors. Fig. 3 shows the results, indicating that the amount and time variables are likely to influence the log-odds of fraud in a nonlinear way.
Note that the time variable is a typical example of an inherent nonlinear effect, as discussed in Section 3. Denoting by p the probability of fraud, we train the following GAM on the training data: where f 1 is a thin-plate regression spline [45] and f 2 is a cyclic cubic regression spline [44], which captures the periodic nature of the time effect.
In the second step, the continuous effects f 1 (amount) and f 2 (time) are discretized (i.e., approximated by step functions) using the strategy described in Section 3.2 to obtain f(amount) and f(time). The amount variable is discretized using constrained binning, whereas we use unconstrained binning for the time variable.
Finally, a classical logistic regression model is fit to the transformed variables: The results are evaluated based on different criteria. In addition to the AIC on the training set, we also evaluate the AUC, the weighted Brier score and the H-measure obtained on the test set. The AUC is the wellknown area under the receiver operating curve (also equivalent to a linearly transformed Gini coefficient). The classical Brier score is the mean squared error between the predicted probabilities and observed responses, i.e., 1 n n i=1 (p i − y i ) 2 . This measure is clearly inadequate for imbalanced classification tasks, as it gives equal importance to each individual prediction. We therefore use weights that are inversely proportional to the prior Note that these weights make the predictions of all fraudulent cases together as important as those of all regular transactions. The H-measure is a more recently developed alternative to the AUC that avoids dependence on the classifier and is therefore more reliable. It requires the severity ratio as an input, for which we take the recommended ratio of the class priors (π 1 /π 0 ); see [14,15] for details.

Experiment 2: complete approach on the fraud dataset
In the second experiment, we consider the complete fraud dataset (including the categorical variables) with the goal of evaluating the different treatment combinations of the categorical and continuous variables. For the combination of discretized splines with the shrinkage estimation of the WOE values, we first convert the categorical variables into continuous variables using shrinkage estimators. Then, we proceed as in Experiment 1, with the difference being that the GAM now includes the transformed categorical variables: where f 1 is a thin-plate regression spline and f 2 is a cyclic cubic regression spline, which captures the periodic nature of the time effect.
For the combination of the clustered WOE values with the discretized splines, we follow the strategy outlined in Section 3.2. We thus first optimize the number of clusters for each of the categorical variables using the approach in Table 2. Afterwards, we proceed as in Experiment 1 but now with the GAM: For the evaluation, we use the same performance measures as in the previous experiment: the AIC, AUC, weighted Brier score and H-measure.

Experiment 3: complete approach on the credit risk dataset
In this experiment, we use the same approach as in Experiment 2 in that we compare the combinations of spline binning with the different treatments of categorical variables. We again use the strategy outlined in Section 3.2, including the clustering of the categorical variables as in Table 2 when cWOE is used. All of the continuous variables are fit as a binned spline in the model, with the exception of QUANT ADDITIONAL CARDS, as it is supported on a very discrete domain. The PAYMENT DAY variable, which indicates the day of the month on which the eventual payments will be made, is fitted with a cyclic spline, as it is natural to expect cyclic behavior from this variable. As there is no predefined split for the training and test data, we evaluate our proposal using 10-fold cross validation and evaluate the performance of the method on each fold using the AIC, AUC, weighted Brier score and H-measure.

Experiment 1
The initial fit of the GAM of Eq. 5 yields the estimatesβ 0 = −19.518 and overfit the training data. By discretizing the resulting spline functions, we gain robustness against this overfitting. Table 5 shows a comparison of the performances. We additionally add the results of the classical GLM. We see that the GLM with spline binning (SB) outperforms the classical GLM on all levels. The most significant difference is found in the H-measure, with an increase of almost 15 %. As a reference, we add the performance of XGBoost (XGB) [8] to the table, which does not provide a significant improvement over the GLM-based approaches on these data.

Experiment 2
In the second experiment, we compare the different combinations of our proposed preprocessing techniques. The results of this comparison are presented in   to Australia, China, South Africa and Chile have neutral risk levels. The GAM fit with the optimal value of λ cat no longer displays a non-linear effect for the amount variable, as was the case in Experiment 1. This means that the inclusion of the categorical variables resolves the nonlinearity issue for this variable, and we can treat it as a linear effect. The time variable, however, still displays a nonlinear relationship with the response, as shown in Fig. 7.  Table 7, which suggests that all predictors have significant contributions to the final model, with the exception of the amount variable.

Experiment 3
The results for the final experiment are summarized in Table 8. It is clear that the absolute differences are not as pronounced as those in the previous example. This is not very surprising, as a significant number of predictor variables that carry a lot of signal are either binary or enter the model linearly, and in both cases, the effect of the proposed approach is limited.
Nevertheless, all the differences are statistically significant, as verified by the Wilcoxon rank test [42], which yields p-values between 0.002 and 0.036 for testing the performance of sWOE + SB against the alternatives in terms of the AUC, wbrier and H-measure. These differences can produce significant cost savings in practical business settings. As in the previous example, the XGBoost classifier does not seem to improve upon a GLM-based approach for these data.

Discussion
The

Conclusion
We propose and study two advanced techniques for preprocessing data before applying regression. The first method considers the treatment of WOE values, which we propose to estimate using shrinkage estimators for the proportions. Alternatively, the original WOE values can be clustered for improved interpretability. Second, we study the discretization of continuous variables through the binning of spline functions. This allows for captur-ing nonlinear effects in predictor variables and yields highly interpretable predictors that take only a small number of discrete values.
Through three different experiments on a fraud detection dataset, we illustrate the advantages of using these advanced preprocessing techniques.
In particular, the out-of-sample performance of the model is improved us-

Software availability
An implementation of the proposed pipeline as well as a script reproducing the results in the paper can be found in the GitHub repository https://github.com/JakobRaymaekers/WOE2.0.