{'fraction_non_alphanumeric': 0.026578808937124505, 'fraction_numerical': 0.007129211772197459, 'mean_word_length': 4.99096035826837, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comparative Evaluation of Quantification Methods

Quantification represents the problem of predicting class distributions in a dataset. It also represents a growing research field in supervised machine learning, for which a large variety of different algorithms has been proposed in recent years. However, a comprehensive empirical comparison of quantification methods that supports algorithm selection is not available yet. In this work, we close this research gap by conducting a thorough empirical performance comparison of 24 different quantification methods on 40 data sets, considering binary as well as multiclass quantification settings. We observe that no single algorithm generally outperforms all competitors, but identify a group of methods including the Median Sweep and the DyS framework that performs best in the binary setting. We also find that tuning the underlying classifiers has in most cases only a limited impact on the quantification performance. For the multiclass setting, we observe that a different, broad group of algorithms yields good performance, including the Generalized Probabilistic Adjusted Count, the readme method, the energy distance minimization method, the EM algorithm for quantification, and Friedman's method. More generally, we find that the performance on multiclass quantification is inferior to the results obtained in the binary setting. Our results can guide practitioners who intend to apply quantification algorithms and help researchers to identify opportunities for future research.


Introduction
Quantification is the problem of estimating the distribution of target labels on unseen (test) data. That is, after being trained on a dataset with known class labels, we want to estimate the number of instances of each class in a set of instances with given features but no target labels. In contrast to traditional classification tasks, we are not interested in individual predictions, but only in aggregated values on a group level. Additionally, we also assume that the distribution of the features (but not their associations with the labels) could have shifted between training and application. For this problem setting, previous research has established that training a classification algorithm and counting instance-wise predictions does not generally yield accurate estimates [12,27]. This has given rise to a relatively young, but vivid research field within the machine learning community. As an increasing number of researchers are becoming aware of this issue, a growing number of novel methods have been proposed.
While a first review of existing quantification methods has been given by González et al. [15], and recent publications also provide broader frameworks for quantification learning [20,21], a thorough, empirical, and independent comparison of quantification methods has not been presented yet. In this work, we aim to fill this research gap by providing a comparison of 24 different quantification algorithms over 40 datasets. Apart from assessing approaches for the binary class setting, we also include experiments for the multiclass quantification setting, which has received limited attention in quantification research so far. For each dataset and algorithm, we evaluate several degrees of distribution shifts between training data and test data with varying training set sizes. Further, we evaluate the effect of tuning the classifiers that the majority of existing quantification methods are making use of. Thus, overall, our experimental results report results from more than 5 million algorithm runs.
Our experiments with binary target labels show that there is not a single algorithm that outperforms all others-but we identify a group of algorithms including the Median Sweep [12] and the DyS framework [20] that on average perform significantly better. We also show that algorithms that are based on optimizing a classifier for the quantification problem show on average worse performance, implying that their benefits in practice might be restricted to particular scenarios. Further, we observe that tuning the hyperparameters of the base classifiers does not produce significant improvements in the resulting predictions of the corresponding quantifiers.
In the multiclass setting, we find a broader group of algorithms which show a significantly better average performance than the rest, with the Generalized Probabilistic Adjusted Count [3,10], readme [17], energy distance minimization [19], the EM algorithm for quantification [25], and Friedman's method [13] leading in the averaged rankings. These algorithms share the characteristic that they naturally allow for multiclass quantification. By contrast, extending predictions from binary quantifiers to the multiclass case in a one-vs-all fashion does not appear to yield competitive results, even when using strong base quantifiers such as the Median Sweep or the DyS framework. Further, we show that fine-tuning the hyperparameters of the underlying base classifiers appears to have significant positive effects in this multiclass setting for most of the quantifiers, in particular for the EM algorithm and Friedman's method, which appear to outperform the rest. More generally, we observe significantly weaker performance for the multiclass case, corroborating that multiclass quantification constitutes a harder research problem and might need more research attention in the future.
Overall, our results guide practitioners towards the most propitious quantification approaches for certain applications, and help researchers to identify promising future research avenues.
In the following, we first briefly introduce the quantification problem and describe how it conceptually differs from the classification problem. Afterwards, Section 3 gives an overview of the algorithms included in our experimental comparison, providing a summary of the state-of-the-art in quantification. Next, in Section 4, we provide a thorough description of the experimental setup of our comparison, before giving an in-depth presentation of the experimental results in Section 5. Finally, in Section 6, we discuss the results of our experiments, before closing with a conclusion in Section 7.

The Quantification Problem
Quantification is a supervised machine learning problem that aims to estimate the distribution of class labels in a test set instead of predicting the class of individual instances. Throughout this paper, we use the following notations. For training, we are given a dataset of instances D T rain , for which we know the values of multiple (categorical or continuous) features X and the corresponding target label Y . Letting L denote the number of possible values of the target label, we distinguish between the binary case, i.e., there are exactly L = 2 possible values for the target label, and the multiclass case, i.e., there are L > 2 options for the target label. The goal is then to train a model on training data, that allows to predict the distribution of the target label in some test data D T est , for which only the values of the features X are known. In the following, we will often use the term prevalence for the relative frequency of single labels in training or test data. We formally denote the distributions of X and Y in the training set by P T rain (X) and P T rain (Y ), and their distribution in the test set by P T est (X) and P T est (Y ). Since in the binary case, the full distribution is already specified by the share of one class, we will denote for shorter notation the instances from one arbitrary class as positives, and label their prevalence in training and test data as pos T rain and pos T est , respectively.
In contrast to traditional classification, a shift of the distribution of the target label Y , i.e., a difference between the class probabilities P T rain (Y ) in the training set and the class probabilities P T est (Y ) in the test set, is expected. However, it is assumed that the conditional distributions P (X|Y ) are stable between training and test sets. Furthermore, compared to classification, it is also more common to expect contradictory labels for training or test instances with the exact same feature values.
A trivial approach to quantification, known as Classify and Count (CC) method, applies an arbitrary classification method trained on the training data to the test data and predicts the distribution of the predicted labels. However, this has been theoretically and empirically shown to achieve insufficient results in many scenarios [12,27].

Algorithms for Quantification
We first outline the quantification algorithms under consideration. Following a previous categorization [15], we distinguish between (i) adjusted count adaptations, (ii) distribution matching methods, and (iii) adaptations of traditional classification algorithms. An overview of the considered algorithms is given in Table 1. In that table, the column Continuous indicates whether an algorithm is also able to handle datasets with continuous features, and Multiclass indicates whether an algorithm is not restricted to binary quantification. Next to that, this table also lists abbreviations that we are often going to use when discussing these algorithms.  [28] No Yes Classify and Count CC Forman [12] Yes Yes Probabilistic Classify and Count PCC Bella et al. [3] Yes Yes SVM perf using KLD loss SVM-K Esuli et al. [8] No Yes SVM perf using Q-measure-loss SVM-Q Barranquero et al. [2] No Yes Nearest-Neighbor-Quantification PWK Barranquero et al. [1] Yes No Quantification Forest QF Milli et al. [22] Yes No AC-corrected Quantification Forest QF-AC Milli et al. [22] No No Table 1: Overview of considered quantification algorithms. For each algorithm, we denote its abbreviation, a reference, and its ability to handle multiclass labels and continuous features. Regarding the multiclass setting, we distinguish between algorithms that can naturally handle this setting (Yes), algorithms that require a one-vs.-rest approach (OVR), and algorithms that were not considered in our multiclass experiments (No).

Adjusted Count Adaptations
The trivial Classify and Count (CC) method just applies an arbitrary classifier c(X) on the test data and counts the number of respective predictions. The core idea behind the Adjusted Count (AC) approach is to adjust these results post hoc for potential biases. This is accomplished by exploiting the assumption that the likelihood P (X|Y ) of the features X given the target class Y does not vary between training and test data, and thus the true-positive rate (tpr) and false positive rate (f pr) of a classifier, which correspond to the distributions P (c(X)|Y ), are expected to be identical between training and test data. Letting ppos T est denote the predicted prevalence of positives by the CC method, we can express this quantity in terms of the true prevalence of positives pos T est and the (mis)classification rates tpr and f pr via ppos T est = pos T est · tpr + (1 − pos T est ) · f pr, which we can solve for pos T est to obtain the AC estimation pos T est = ppos T est − f pr tpr − f pr .
In practice it can occur that the estimate falls out of the feasible interval [0, 1], hence in such cases one would have to clip the outcome to the boundary values.
Based on that idea, a few variations of the AC method have been introduced in the literature, and the following methods will be included in our experiments: 1. Adjusted Count (AC). As described above, we estimate the true positive and false positive rates from the training data and utilize them to adjust the output of the CC method [11]. 2. Probabilistic Adjusted Count (PAC). This method adapts the AC approach by using average classconditional confidences from a probabilistic classifier instead of true positive and false positive rates [3]. 3. Threshold Selection Policies (TSX, TS50, TSMax, MS). The core idea of these variations is to shift the decision boundary (e.g., classify an instance as positive if the original estimate c(x) is larger than 0.7) of the underlying classifier in order to get numerically more stable AC estimates. Different strategies involve using the threshold that maximizes the denominator tpr − f pr (TSMax), a threshold for which we have f pr = 1 − tpr (TSX), a threshold at which tpr ≈ 0.5 holds TS50, or, as in the Median Sweep (MS) method, using an ensemble of such threshold-based methods and taking the median prediction [12].

Distribution Matching Methods
The majority of existing quantification methods can be categorized as distribution matching algorithms. These algorithms are all implicitly based on the assumption that the distribution of the features X conditioned on the distribution of the target labels Y does not change between training data and test data. Under that assumption, with j , j ∈ {1, . . . , L}, denoting the possible values of the labels Y , the law of total probability yields that As in this equation, both the left-hand distribution P T est (X) and the conditional distributions P T rain (X|Y = j ) on the right-hand side can be seen as represented by given training and test data, only the sought-for probabilities P T est (Y = j ) are left as unknowns. To estimate these target probabilities, there are two main issues to be worked out from a methodological point of view. First, estimating or modeling the distributions P T rain (X|Y = j ) and P T est (X) is not at all trivial. There can be an arbitrary amount of features X, and the training data usually does not provide nearly enough samples to accurately represent the distribution of the feature space, even more when conditioning on the target labels Y . Second, once an estimation of the distributions P T est (X) and P T rain (X|Y = j ) has been made, there are also various ways to predict the target probabilities P T est (Y = j ) from these estimations.
The methods discussed in this chapter tackle these issues in various ways. One basic approach to tackle the first issue has, for instance, already been introduced when discussing the Adjusted Count. In the Adjusted Count approach, information on the distribution of the features X was derived by the means of applying a classifier c(X) and considering the distribution of their outputs P (c(X)) instead of P (X). That way, Equation 2 would be transformed to the set of linear equations However, there are also methods that do not apply classifiers, and instead, for instance, estimate P (X) based on the distributions of single features, or in terms of distances between individual instances in the data.
Regarding the second issue, most of the presented methods translate Equation 2 into a set of linear equations, and then minimize some distance function between the left-and right-hand-side expressions, subject to the constraint that L j=1 P T est (Y = j ) = 1 has to hold. This common pattern has already been noted by Firat [10]. Among all the methods of this category, we are comparing the following methods: 1. Generalized Adjusted Count Models (GAC, GPAC). As described above, the most simple work-around to avoid estimating P (X) is to apply a classifier to build a system of linear questions as in Equation 3, and solve it via constrained least-squares regression [10]. That approach can be considered as a Generalized Adjusted Count (GAC) method, which also naturally includes the multiclass case. Similarly, one can obtain the Generalized Probabilistic Adjusted Count (GPAC) method, by utilizing the confidence scores from probabilistic classifiers as in the PAC method. 2. The DyS Framework (DyS, FMM). More recently, Maletzke et al. [20] proposed the DyS framework, in which the main idea is to utilize confidence scores resulting from the decision functions of a binary classifier. More precisely, the confidence scores obtained on the training data are divided into bins, and then the probability that the confidence of an instance ends up in that bin is estimated from the training set. Thus, in our context the number of linear equations we obtain from Equation 2 equals the chosen number of bins, which, next to the distance function that this set of equations is optimized on, can be seen as a parameter of this framework. A main drawback of this framework is that it only works for the binary case, and that many of the distance functions that were proposed and evaluated for this framework are not convex, requiring methods such as ternary search to estimate the optimal solution. Since in their evaluation, using the Topsøe distance [6] has proven to yield consistently good results, we are applying this setup as DyS method in our experiments. Furthermore, it is noteworthy that Forman's mixture model (FMM) [12] can also be regarded as a method of this framework that uses Manhattan distance along with a particularly large number of bins.
3. readme. Instead of applying a classifier, the readme method [17] directly models the distribution of the feature space by counting the co-occurences of the individual features X i . This method was originally developed to be used on document-term matrices in the context of text mining. Due to sparsity issues, it requires all features to be binned, and only a subset of all features is projected on to form the system as in Equation 2, which is then optimized by means of general least-squares regression. In practice, one samples an ensemble of such subsets and averages the corresponding results in the end to obtain the final prediction.
4. Hellinger Distance Minimization (HDx, HDy). González-Castro et al. [16] have proposed two methods that are based on minimizing the Hellinger distance to match distributions in Equation 2. Similar to the readme method, in the HDx method one considers the distributions of single features X i , while in the HDy method, one applies classifiers as in the GAC method to form the system in Equation 3.

5.
Friedman's Method (FM). Similar to the GPAC method, Friedman [13] proposed to utilize the confidence scores from probabilistic classifiers. However, instead of averaging the class-conditional confidence scores, he proposes to utilize the fraction of class-conditional confidence scores that are above and below the observed class prevalences in the training data.
6. Energy Distance Minimization (ED). As the name of this method suggests, its core idea is to minimize the energy distance between the left-hand and right-hand side distribution in Equation 2. In that context, the distribution of the feature space is intrinsically modeled by the Euclidean distances between individual instances, and therefore no classifiers or additional parameters are required [19].
7. The EM Algorithm for Quantification (EM). This method applies the classic EM algorithm [5] on the outputs of probabilistic classifiers to adjust them for potential distribution shift between the target class distributions in training and test data. While quantification was not the main focus in the original proposal of the algorithm [25], the sought-for target class prevalences are obtained as a side-product. 8. CDE Iteration (CDE). The class distribution estimation (CDE) iterator [29] applies principles from costsensitive classification to account for changes in class distributions between training and test data. For that purpose, the misclassification costs are updated iteratively, and in the original proposition of the algorithm, the underlying classifier is retrained in every iteration step. In our experiments, we use the more efficient variant proposed by Tasche [28], in which each iteration rather updates the decision threshold of an underlying probabilistic classifier. For this variant, Tasche has also proven that the iteration will eventually converge.

Classifiers for Quantification
Classifiers for quantification apply established classification methods to the setting of quantification. The main approach behind most of these methods is to optimize such established classifiers based on a loss function that minimizes the quantification error, and then estimate the class distributions based on the predictions of the individual instances. Thus, these approaches are all in some sense variants of the CC method. In our experiments, we included the following methods: 1. Classify and Count (CC). This trivial approach applies a classifier and counts the number of times each class is predicted [12].
2. Probabilistic Classify and Count (PCC). This approach takes probabilistic predictions, i.e., continuous values between zero and one, and averages the predictions of all instances to estimate the class prevalences [12,3].
3. SVM perf optimization (SVM-Q, SVM-K). This pair of methods applies the so-called SVM perf classifier, which is an adaptation of traditional support vector machines which can be optimized for multivariate loss functions [18]. Based on this algorithm, multiple classifiers with different quantification-oriented loss functions have been proposed. For instance, Esuli et al. [8] have proposed to use the Kullback-Leibler Divergence (SVM-K), while Barranquero et al. [2] have developed the Q-measure for that purpose (SVM-Q).
4. Nearest-Neighbor based Quantification (PWK). Barranquero et al. [1] have adapted nearest-neighbor based classification to the setting of quantification. In their k-NN approach, they apply a weighting scheme which applies less weight on neighbors from the majority class.

5.
Quantification Forests (QF, QF-AC). The decision tree and random forest classifiers have been adapted for quantification by Milli et al. [22]. Other than in the traditional approach, the authors propose that the split in each decision tree is made based on a quantification-oriented loss functions. Since in their original proposition, applying the AC method on the predictions from these random forests yielded particularly strong results, we include both the Quantification Forests as well as the AC adaption of them in our experiments.

Multiclass Quantification
In the literature on quantification, multiclass quantification has received surprisingly little attention so far, despite Forman [12] pointing out that this task is a much harder task than binary quantification. In our comparative evaluation, we also take a closer look into this scenario. In contrast to classification, approaches based on a one-vs.-one comparison of two target labels are to the authors' knowledge not used for quantification, potentially since it is unclear how to aggregate the results. Therefore, methods for multiclass quantification can be broadly distinguished into two categories: 1. Natural Multiclass Quantifiers. Like in classification, some quantification algorithms can naturally handle the multiclass setting as well. This is the case for most distribution matching methods, as by Equation 2, there is no constraint on the number of classes that are summated. Further, quantification-oriented classifiers such as Quantification Forests can handle the multiclass setting as well, since the underlying classifier allows for it.

2.
One-vs.-Rest Quantifiers. Traditional quantification methods such as the count adaptation methods as introduced by Forman [12] have been designed for the binary setting only. To extend such methods to the multiclass setting, one can estimate the prevalence of each individual class in a one-vs.-rest fashion, and then normalize the resulting estimations afterwards so that they sum to 1. Next to all count adaptation methods, we also applied this strategy for the distribution matching methods from the DyS framework, as these do not naturally generalize to the multiclass setting.
An overview regarding which multiclass strategy is used for each quantification algorithm is provided in Table 1. For the SVM-K, SVM-Q and QF-AC methods we did not conduct any multiclass experiments, as the underlying implementations did not provide a multiclass feature. Furthermore, for the CDE iterator we did not run multiclass experiments either, since the individual one-vs.-rest predictions yielded extreme predictions of either 0 or 1 regularly.

Experimental Setup
Overall, we compare 24 algorithms on 40 datasets. In the following, we provide details regarding the datasets, sampling protocols, algorithmic parameters, and evaluation measures. The implementation of the algorithms and experiments can be found on GitHub 1 .

Datasets
We applied all algorithms on a broad range of 40 datasets collected from the UCI machine learning repository 2 and from Kaggle 3 . An overview of the datasets that we used in our experiments is provided by Table 2. It includes information on the amount of instances per dataset N , the number of features D, the number L of classes of the corresponding target variable, and an indicator about whether or not a dataset contains non-categorical (continuous) features. The non-categorical aspect is relevant for some quantification algorithms such as readme that require a finite feature space.
Out of the 40 datasets, 17 had a non-binary set of target labels, or were even regression datasets. The regression datasets were converted to both multi-class and binary datasets by binning the values of the target variable. This was usually done with the abstract goal of achieving groups of similar size with respect to the number of instances to allow for a more robust basis for potential shifts in the following steps. The cut-off points for the bins were determined manually after looking at the distribution of the target values. Furthermore, the real multiclass datasets were also converted to binary datasets. In these cases, we kept the most populated class as is, and merged the other classes into a single class, like in a one-vs.-rest classification problem. By doing so, we preserved meaningful class semantics that classifiers and quantifiers could recognize. All datasets have been preprocessed the same way as for standard classification, including dummy coding their non-ordinal features, rescaling their continuous features, and removing missing values. Furthermore, to enable the application of algorithms that require a finite feature space, we created a variation of each dataset in which all non-categorical features were binned. All algorithms that could handle a non-finite feature space    Table 3: List of training distributions P T rain (Y ) and test distributions P T est (Y ) considered for experiments in the multiclass setting, ordered by number of target classes L. In both the columns for training and test distribution, each row represents a distribution of instances that was sampled from the corresponding data. For instance, assuming that for a dataset with L = 3 classes, the corresponding labels are given by Y ∈ {1, 2, 3}, the first row among the column of training distribution indicates that in our experiments, we have sampled training sets where Label 1 had a prevalence of 0.2, Label 2 had a prevalence of 0.5, and Label 3 had a prevalence of 0.3. For each combination of training and test distributions, we generated ten test scenarios by taking different samples.
were run on the non-binned datasets. While one may argue that due to these alterations in the datasets the results would be less comparable, the binning procedure ultimately simulates the loss of information that one would have to accept when applying such restricted algorithms in the first place.
Overall, these datasets represent a wide range of domains, and are differently shaped in terms of their number of instances as well as in the design of their feature spaces.

Sampling Strategy
As we aim to evaluate quantifiers under a large set of diverse conditions, we chose a sampling approach in which we varied (i) the training distribution, (ii) the test distribution and (iii) the (relative) sizes of training and test datasets. In both distributions, we sampled from the full interval [0, 1], including also very unbalanced and thus presumably difficult settings with only very few (or even no) positive labels in the training and/or test set.
Concerning the multiclass case, we have considered datasets with a varying number of L ∈ {3, 4, 5} different target classes. For each of these values of L, we fixed a set of three training and five test class distributions, representing relatively uniform as well as polarized class distributions, which can be seen in Table 3.
Both in the binary and the multiclass settings, we considered splits with relative amounts of training versus test data samples in {(0.1, 0.9), (0.3, 0.7), (0.5, 0.5), (0.7, 0.3)}, thereby simulating scenarios in which we have little as well as relatively much information at hand to train our models. We omitted splits with 90% training data to save computational resources, as the computational complexity of most algorithms in our experiments is determined by the size of training data rather than test data. Even without this particular split, in the binary setting we obtain a large number of 288 combinations of training distributions, test distributions, training/test-splits, and in the multiclass setting we obtained 60 of such combinations for each dataset.
To collect experimental data from each dataset which satisfies these constraints, we use undersampling, selecting from a given dataset as many data instances as possible. We illustrate this sampling strategy with an example: Assume a dataset with 1000 instances and a binary class attribute, consisting of 700 positive and 300 negative instances. As an example evaluation scenario, we aim to sample data with an 80%/20% split in training and test sets, and with a 60%/40% distribution of positive and negative instances in both training and test sets. Splitting the 300 negative instances randomly 80/20, we have 0.8 · 300 = 240 negative instances available for training and 0.2 · 300 = 60 instances available for the test data. To obtain a 60/40 distribution of positives and negatives in the training data, we therefore have to choose 240 : 40 60 = 360 positive instances to include in the training data, which we randomly sample from the full set of positive instances. The positives for the test data are randomly sampled analogously. Note that the instance count for each label imposes a constraint on the number of sampled instances with other labels. Overall, we use the maximum number of instances for each label that satisfies all constraints.
This sampling procedure can lead to a relatively small subset compared to the whole corpus in cases where the class distributions we aim to sample strongly deviates from the natural class distributions in the given dataset. This makes the quantification task comparatively more challenging in these settings.
To address possible variances in the drawn samples, we made ten independent draws for each combination of distributions that could occur within our protocol, and ran all corresponding algorithms on each of these draws. To ensure the reproducibility of all these draws, we used a set of ten fixed seeds for the random number generators. In total, for the binary setting, we therefore performed 2880 draws per dataset, which, considering that we applied 24 algorithms on 40 datasets, yielded 2,764,800 experiments for that setting. Including 204,000 additional experiments in the multiclass case and 2,666,520 more experiments on tuned alternative base classifiers (cf. Section 5.3), we conducted for this paper a total number of more than 5 million experiments in our evaluation.

Algorithms and Parameter Settings
In our experiments, we compared all algorithms described in Section 3 and listed in overview Table 1. Except for the SVM perf -based quantifiers and Quantification Forests, all algorithms were implemented from scratch in Python 3, using scikit-learn as base implementation for the underlying classifiers, and the package cvxpy [7] to solve constrained optimization problems. For the SVM perf algorithm, we used the corresponding open source software package by Joachims [18], and adapted the code that Esuli et al. [9] have used as a baseline for their QuaNet method to connect Joachim's C++ implementation to Python. Regarding Quantification Forests, we used the original implementation that was kindly provided by the authors [22].
As the focus of this work is on a general comparison of quantification algorithms, we initially fixed for all algorithms a set of default parameters based on which the main experiments were conducted. When choosing the hyperparameters of each model, we followed recommendations of the original papers where possible. For all quantification methods that required a base classifier, we used the same logistic regression classifier for each dataset split. The logistic regression model was chosen since it is one of the most established and popular base classifiers and also provides calibrated class probability scores that are required for quantification methods such as the PAC, EM, or FM methods. That way, results from different quantifiers could not be biased by differences in the underlying classification performances. We acknowledge that fine-tuning the hyperparameters of each quantifier for each dataset could overall improve the performance, but argue that fixing parameters once allows for a fairer comparison of individual approaches and makes larger numbers of algorithm runs computationally feasible. However, since one could suspect a strong dependence of the quantification performances from the performance of the underlying classifiers, we further conducted a series of experiments in which we used stronger classifiers with tuned parameters, see Sections 4.3.2 and Section 5.3. Below, we will first outline the parameter settings for the main experiments before giving details on the experiments in which we used tuned classifiers.

Parameter Settings in the Main Experiments
For the main experiments, we made the following choices regarding the algorithms' hyperparameters: • As mentioned above, for all methods that utilize a classifier to perform quantification, we used the logistic regression classifier with the default L-BFGS solver along with its built-in probability estimator as provided by scikit-learn and set the number of maximum iterations to 1000. We always used stratified 10-fold cross-validation on the training set when estimating the misclassification rates or computing the set of scores and thresholds that were needed by the quantifiers.
• In all adaptations of the Adjusted Count that apply threshold selection policies, namely the TSX, TS50, TSMax and MS methods, as well as in Forman's mixture model (FMM), we reduced the sets of scores and thresholds obtained from cross-validation by rounding to two decimals. In the MS algorithm, we followed Forman's recommendation to only use models that yield a value of at least 0.25 in the denominator of Equation 1.
• In Forman's mixture model (FMM), we also reduced the sets of scores and thresholds obtained from crossvalidation by rounding to two decimals. Afterwards, we have chosen a large number of 100 bins, approximating Forman's original proposal of using a single bin for each confidence score occurring in the data. • For the DyS algorithm, we chose to divide its confidence scores into 10 bins, as this number of bins appeared to yield consistently strong results in the study by Maletzke et al. [20]. • For the EM algorithm and the CDE iterators, we chose ε = 10 −6 as the convergence parameter, and limited the number of iterations to a maximum of m = 1000 iterations, which was reached only very rarely. • For the readme algorithm, we set the size of each feature subset to log 2 (D) + 1 , with D denoting the number of features in X. We considered an ensemble of 50 subsets, which were all drawn uniformly. • In the QF and QF-AC algorithms, we used the weka-based implementation that has kindly been provided by the authors. We left all parameters at their default values, including the size of the forest of 100 trees. • For both the SVM-Q and the SVM-K method, we chose C = 1 as the regularization coefficient, which was however decreased to C = 0.1 when there were more than 10,000 training samples. This adaptation was chosen as in our experiments we observed that when large amounts of training data were present, a higher regularization parameter would slow down the convergence of the optimization significantly. • For the PWK algorithm, we chose a neighborhood size of k = 10, and a weighting parameter of α = 1, as different weight values did not yield significantly better results in the study by Barranquero et al. [2]. • In the rare case that in one-vs.-rest quantification all individual class prevalences were predicted as 0, we returned the uniform distribution as prevalence estimation.

Parameter Settings in the Experiments on Tuned Classifiers
Many quantification methods rely on the predictions of an underlying base classifier to form their class prevalence estimations. Since the quality of these underlying classification models could have a strong impact on the performance of the quantifier, we evaluated the impact of applying more advanced classification methods with tuned hyperparameters in our second set of experiments. For that purpose, we conducted experiments with four classification models, namely random forests, AdaBoost, RBF-kernel support vector machines, and logistic regression models. For each of these classifiers, we conducted a grid search to optimize the hyperparameter on every single dataset split in our experiments. Due to scalability issues, which in particular occurred for the SVM classifier, we however restricted ourselves to the 24 datasets which have not more than 10,000 instances in total. After having determined their optimal parameter configuration for each dataset split, we used each of the four classification models with their optimal parameterization as base classifiers for the quantification methods.
For the CC, AC, GAC, and HDy methods, all four classification models could be applied, as these only require the pure (mis)classification rates from the training data for their estimations. For all quantifiers which require scores from a classifier's decision function, namely the threshold selection policies TS50, TSX, TSMax, and MS as well as the DyS and FMM method, we only used the support vector classifier and the logistic regression model, since AdaBoost and random forests do not provide calibrated decision functions or confidence scores. Furthermore, for all quantifiers which require probability scores, we only applied the tuned logistic regression model, since it is the only method that really provides calibrated probability scores.
In addition to these experiments on tuned classifiers, we also ran additional experiments on the same datasets with the SVM-K and SVM-Q methods, in which we applied an RBF-kernel instead of the default linear kernel. Due to computational reasons, hyperparameter optimization was however not feasible when using this kernel, and we thus resorted to a default parameter configuration where we chose C = 1 as the regularization coefficient and γ = 1/D with D denoting the amount of training features. Regarding the exact parametrizations of the algorithms, we made the following choices: • All parameters of the quantification methods which do not regard the underlying classifiers were kept as described in Section 4.3.1. • For the logistic regression classifier, in the grid search we varied the regularization weight C within the set Regarding the tree size, we kept a fixed high number of 1000 trees, since it is well-established that choosing a high number of trees yields more reliable results than any lower number of trees.

Evaluation
Next, we describe the error measures that we used in our evaluation, as well as the procedure used to rank the quantification algorithms and determine statistically significant differences in the performance of the algorithms we have compared.

Error Measures for Quantification
The choice of performance measures for quantification is in itself not a trivial issue, and for a thorough review and discussion of existing quantification measures we point to a recent survey by Sebastiani [26]. To evaluate the quantification performances in our experiments, we focus here on the absolute error (AE) and the normalized Kullback-Leibler divergence (NKLD). In the following, we let p ∈ R L denote the true distribution of labels Y in an unseen test set, and θ ∈ R L denote the distribution of labels Y that has been predicted from a given quantifier on the test set. The absolute error between true distribution p and estimated distribution θ is then given by whereas the normalized Kullback-Leibler divergence between p and θ is defined as denoting the Kullback-Leibler divergence. Since the Kullback-Leibler divergence is not defined when θ i = 0 and p i = 0 for some i ≤ L, we smoothed the distributions by a small value ε = 10 −8 to avoid this problem.
We chose the AE score due to its interpretability as well as its robustness against outliers. Note that this definition corresponds directly to the total variation between the predicted categorical probability distribution and the ground-truth distribution. In contrast to related studies as conducted by [16], we do not use the Mean Absolute Error (MAE), i.e., we do not divide by the number L of predicted classes. This avoids having different upper bounds for the error depending on L, which would make results less comparable, specifically in multiclass quantification settings. The NKLD score-in contrast to the AE score-particularly punishes quantifiers which marginalize the minority class. Both measures are bounded to the same interval for both binary and multiclass quantification, with both values obtaining their minimum (and optimal value) at 0, and the maximum of the AE1 score being 2, while the maximum NKLD value is 1.

Statistical Evaluation of Performance Rankings
Regarding the actual comparison of the given quantifiers, we adapted a statistical procedure established by Demšar [4], who, in the context of classification, suggested to conduct comparisons of multiple algorithms by statistical tests in a two-step approach that is based on the performance rankings of all algorithms considered with respect to a number of datasets they were applied on.
Within that two-step approach, at first a Friedman test [14] is conducted on the null hypothesis that all algorithms perform equally well over a given ensemble of datasets with respect to a chosen error measure. If that null hypothesis is rejected, one may follow up with the Nemenyi post-hoc test [24] to compare the performance rankings of each algorithm per dataset with each other, and determine which algorithms differ from each other in a statistically significant way. The margin of statistical significance is modeled by the critical distance value, which is determined by both the number of algorithms and datasets that are considered as well as the chosen significance level α.
While in classification the underlying rankings would usually be obtained based on a cross-validated accuracy score, in our context we averaged the quantification errors obtained from all the settings in our protocol over each dataset. Based Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. On the right, we plot the distributions of rankings with a Nemenyi post-hoc test at 5% significance. For each algorithm, we depict the average performance rank over all algorithms. Horizontal bars indicate which average rankings do not differ to a degree that is statistically significant. The critical difference (CD) was 5.6973. Overall, DyS, MS, and TSMax appear to work best in general.
on these average errors, for each dataset, we then determined a ranking of our algorithms for this dataset. To account for outliers, we also averaged the resulting scores via the mean and not the median value, which by design of our error measures became more noticeable when averaging with respect to the NKLD errors.

Results
This section presents the results of our extensive experimental evaluation for binary quantification (i.e., labels with exactly two values) and multiclass quantification (i.e., labels with more than two values). For both types, we start by first showing the main results, which aggregate the performance of each algorithm over all datasets and settings. Then, we show more detailed results for subtypes of scenarios, i.e., for different shifts (differences between training and test distributions) and for varying amounts of training data. Finally, we compare the performance of all presented algorithms in the multiclass case, which is a setting that has not received much attention yet.

Binary Quantification
We first describe our results for binary quantification, that is, quantification with binary class labels.

Overall Results
We show the general performance results of all quantification algorithms over all datasets in Figure 1 and Table 4. The letter-value plots in Figures 1a) and 1c) represent the respective distributions of absolute error (AE) and normalized Kullback-Leibler divergence (NKLD) scores resulting from all experiments. Colors in the plot indicate algorithm categories, i.e., count adaptation-based algorithms are shown in blue, distribution matching methods in orange, and adaptations of traditional classification algorithms are shown in green. The plots in 1b) and 1d) depict the average performance ranks of all algorithms per dataset along with the critical differences between average ranks, which indicate whether the difference in average ranks is statistically significant according to the Nemenyi post-hoc test [4]. Here, horizontal bars show which average rankings do not differ to a degree that is statistically significant. Tables 4a) and 4b) complement these plots by providing the average absolute errors (AE) and normalized Kullback-Leibler divergences (NKLD) for all scenarios per algorithm and dataset. Based on these averages, the rankings for the plots 1b) and 1d) have been compiled. Further, for each algorithm, a total average error score over all datasets is provided.
Overall, under both the NKLD and the AE metric, we observe substantial differences between the algorithms. While there is no single individual best algorithm for all cases, the results suggest that there is a group of algorithms that perform particularly well compared to the rest. First and foremost, the two threshold selection methods TSMax and MS, and the Topsøe distance optimization from the DyS-Framework share the top three aggregated ranks with respect to both the AE and the NKLD. When additionally considering the overall distribution of error scores, the MS and DyS methods even appear to have a slight edge over the TSMax approach. Further, with respect to both measures, the TSX threshold selection policy as well as Friedman's method (FM) do not fall behind these top methods to a degree that is statistically significant with respect to the Nemenyi post-hoc test.
This general impression is confirmed by Table 4, where we see that the MS algorithm takes the top rank on most datasets with respect to the AE score, whereas for the NKLD score, the DyS method is most dominant in these rankings. Interestingly, even overall very good methods can have severe issues for specific datasets. For example, the MS method can in general be considered as very good, but faces severe issues for the nurse dataset. This dataset only contains binary features, which causes the underlying SVM classifier to extreme decision scores, which may lead to issues due to the rounding of values. Contrarily, methods such as FM or ED appear to show a relatively robust performance over all datasets, leading to the lowest total average error scores with respect to the AE and NKLD, respectively. However, these algorithms hardly ever show the best performance on a single specific dataset. Considering the overall distribution of error scores, the PAC/GPAC methods also appear to yield relatively robust performance over all datasets, even though top ranks with respect to specific datasets are hardly ever achieved.
When considering the performance of basic algorithms such as (Probabilistic) Classify and Count and Adjusted Count, we observe that these baselines are clearly outperformed by the top algorithms. By contrast, the Probabilistic Adjusted Count (PAC) algorithm shows similar performance to these top algorithms according to the AE scores, but appear to be weaker in terms of their ranks according to NKLD scores. Moreover, all algorithms that we have categorized as classifiers for quantification and also the CDE iterator consistently show the worst performances with respect to both measures.

Influence of Distribution Shift
In the context of quantification, a shift of the distribution of the target labels Y between training and test set is assumed. It could be expected that the size of distribution shift affects the difficulty of the quantification task, as we assume that stronger shifts make accurate quantification more challenging. For that reason, we now take a closer look at the impact of this distribution shift to find out which methods are more or less sensitive to the amount of distribution shift. In that context, we categorize all settings into three scenarios, namely a minor shift, a medium shift, and a major shift in these distributions. More precisely, we consider the shift to be • minor, if the distribution shift is lower than 0.4 in L 1 distance, • medium, if the distribution shift is bigger or equal to 0.4 and lower than 0.8 in L 1 distance, • major, if the distribution shift is bigger or equal to 0.8 in L 1 distance.
We depict the aggregated performance of the quantification algorithms under these three kinds of shifts in Figure 2. Unsurprisingly, we can observe that the performance of all quantification algorithms generally deteriorates with an increasing shift in target distributions. In that regard, the effect appears to be the strongest for classification-based approaches, in particular for the Quantification Forests and the PCC method. The only exception to this principle appears to be the nearest-neighbor-based PWK quantifier, which with respect to the NKLD error appears relatively robust towards distribution shift, as can also be seen by the average rankings (cf. Appendix). Contrarily, the algorithms which yielded the best overall performances, that is, the DyS framework and, in particular, the MS method, also appear   the most robust to larger shifts. Next to these two algorithms, also the other threshold selection policies and Friedman's method (FM) appear relatively robust to distribution shift. Strong performance under minor shift is also achieved by the PAC/GPAC algorithms, which, however, appear to fall significantly behind the top two methods for increasing shifts.
For all other algorithms, the decrease in performance appears to be in between the aforementioned robust algorithms and the classify and count-based quantifiers. Aside from the performance of the PWK method, the overall rankings appear mostly unaffected from distribution shift. That implies that even though the overall performance deteriorates, the same methods perform well independent of the amount of shift.

Influence of Training Set Size
Next, we consider the performance of quantification algorithms when relatively few training samples are given. For that purpose, we restrict the experimental data to only those cases in which the given data was split into 10% training and 90% test samples. The overall distribution of error scores with respect to the AE and NKLD scores can be found in Figure 3. We observe that in general, the performance of all algorithms seems to be worse compared to the results when not restricting ourselves to little training data, which is also to be expected intuitively. However, again the methods which yielded the overall best performances, such as MS, DyS, TSMax, TSX, FM, and PAC/GPAC also appear the most robust towards this scenario. The average performance rankings of all algorithms per dataset (see Figure 12 in the Appendix) are mostly in line with the general setting. An exception to this is that the DyS framework falls somewhat behind the MS method with few training data samples.

Multiclass Quantification
Next, we present results for multiclass quantification, i.e., quantification for labels with more than two values.

Overall Results
Tables 5a) and 5b) as well as Figure 4 present the main results for multiclass quantification. Compared to the binary case, we obtain substantially different results. First of all, the overall prediction performance is much worse, as both the AE scores and the NKLD scores appear multiple times higher on average. For instance, AE values below 0.1 and NKLD values below 0.01 are widespread in the binary case, whereas in the multiclass case, such scores are only rarely achieved. Instead, the average AE values of each algorithm over all experiments are mostly around the interval [0.3,0.4], which is three to four times higher than the average AE values of the best algorithms in the binary case. The second main difference regards the algorithms which appear to work best: Algorithms such as the DyS-Framework, the Median Sweep (MS), and the other threshold selection policies, which have worked very well for binary quantification, appear comparatively weak in their performance. By contrast, the best performances seem to be achieved by distribution matching algorithms which also naturally extend to the multiclass setting, namely the GPAC, ED, FM, EM, readme, and HDx methods. In that context, the GPAC, ED, EM, and FM methods show the best performances with respect to the absolute error, whereas the readme, ED, EM, and FM, but also the classification-based PWK method take the best average rankings with respect to the NKLD. This relatively wide range of strongly performing algorithms is also underlined in Tables 5a) and 5b), where we observe a relatively high fluctuation with regard to the best performance on specific datasets. In addition, from the overall distributions of errors in Figures 4a) and 4c) it becomes apparent that these algorithms also have strong differences in the variance of their performances. In particular, the GPAC method appears to have a much higher variance in its error scores compared to the rest, whereas the ED and readme methods display the lowest variance in their performances. However, the given results also have one big similarity with the results from the binary setting, i.e., all algorithms which are based on the classify and count principle display subpar performances even when optimizing quantification-based loss functions.

Impact of Distribution Shift
As in the binary case, we also investigate the effect that the shift of the distribution of the target labels Y between training and test sets has on the resulting quantification performance. Since we have less experimental data than in the binary case, this time we only distinguish between a minor shift and a major shift. We consider the shift to be • minor, if the distribution shift is lower than 0.5 in L 1 distance, • major, if the distribution shift is bigger or equal to 0.5 in L 1 distance.
The results of multiclass quantification under these scenarios are shown in Figure 5. Similar to the binary case, we observe that the algorithms which appeared to work best in general, also appear the most robust with respect to high distribution shifts. In particular, the GPAC method appears almost unaffected by a high shift in its average performance,   Table 5: Main results for multiclass quantification. We show the averaged error scores for all scenarios per algorithm and dataset. Overall, distribution matching methods that naturally generalize to the multiclass setting perform better than one-vs.-rest or classify and count-based approaches.
consistently achieving higher performance ranks under higher shifts, although significant variance is present in its performance. By contrast, all methods which apply the classify and count principle are again the most susceptible to higher error rates when applied in scenarios with higher shifts between training and test distribution.

Influence of Training Set Size
Finally, we consider the performance of the given algorithms when the given data was split into 10% training and 90% test samples. As before, this serves to investigate the impact of having a relatively small set of training data. The distributions of error scores with respect to the AE and NKLD metrics can be found in Figure 6. Compared to the general distribution of error scores from all experiments, the performance when only small training sets are given does deteriorate. In particular, we observe that the GPAC is much less competitive than in the general scenario, in particular with respect to the NKLD score. Conversely, the EM and ED algorithms, and, with respect to the NKLD score, also the readme method appear to be most robust towards this setting. This implies that those algorithms could be recommended if only small training sets are available.  In addition, we present results of the RBF-K and RBF-Q methods, which are variants of the SVM-K and SMV-Q that use an RBF-kernel instead of a linear one. We observe that except for the CC, PCC, GAC and CDE methods, tuning the classifiers does not seem to have a significant positive effect on the outcome.

Impact of Alternative Classifiers and Tuning
Finally, we present the results from our experiments with quantifiers that applied tuned base classifiers. We begin with the results on binary data, before closing with the results on multiclass data.

Experiments on Binary Data
In Figure 7, we show the scores of all quantifiers using different tuned base classifiers aggregated over all considered datasets, see Section 4.3.2. As a baseline, we also include the results from the quantifiers that are based on the default logistic regressor. These results yield a few key findings. First, for most algorithms, tuning the base classifier does not seem to have a significant positive effect. Instead, for the best performing algorithms MS, DyS, TSX, FM, and TSMax, the performance even appears to deteriorate. The few exceptions where tuning classifiers appear to benefit the predictions include the CC, PCC, CDE, and GAC. While the first two directly apply the classify and count principle, where it is to be expected that more accurate classification will tend to yield more accurate quantification, the results for the CDE and GAC methods pose as outliers. Regarding the SVM-K and SVM-Q methods, we observe that applying the alternative RBF-kernel appears to have a slight positive effect, but these RBF-K and RBF-Q variants still show inferior performance compared to most of the other quantifiers, while at the same time coming at very high computational costs. Overall, the given results also appear consistent over both the AE and the NKLD metrics.

Experiments on Multiclass Data
The results of our experiments on quantification with tuned base classifiers in the multiclass setting can be found in Figure 8. Other than in the binary setting, we observe that tuning classifiers appears to have a significant positive effect for most quantifiers. This effect is particularly pronounced for methods which utilize probability scores from their underlying classifiers, namely the EM, PAC, GPAC, FM, and CDE methods, but also very notable for the classify and count-based CC and PCC methods. In that context, the EM method with tuned logistic regressors as its base classifiers even appears to significantly outperform the other quantifiers in this experimental setting. Again, these trends are also mostly consistent over both error metrics that we considered, although with regard to the NKLD, the benefits of tuning base classifiers even appear a little more distinct.

Discussion
Next, we discuss the main results of our study and potential limitations of our study.

Discussion of Results
Our experiments yield substantially different results for the binary case compared to the multiclass case, both in terms of overall quality of performance, and in terms of which algorithms perform best. In the binary case, we identified a few algorithms which appear to work particularly well with respect to both absolute error and NKLD, namely the MS, DyS, and TSMax methods. These methods stand out both in terms of their ranks and their overall error distribution (though DyS and MS have a slight edge over TSMax in that regard). Next to these three algorithms, other methods show similarly strong performances, at least with respect to one of the two measures that were considered. In that regard, Friedman's method has shown very strong performances with respect to the AE score, while the ED method appears to work particularly well with respect to the NKLD measure.
The strong performance of the MS and TSMax methods indicate that the simple idea behind the Adjusted Count approach, even when using a rather unsophisticated baseline classifier, can still yield very decent results as long as numerical stability, i.e., a big denominator in Equation 1, is ensured. In that regard, the MS method also benefits from the policy that all thresholds for which the denominator is below 0.25 are excluded. A similar argument can be made for the superiority of the DyS framework compared to other distribution matching methods that utilize predictions from classifiers. More precisely, the approach of binning confidence scores into more than just two classes, which ultimately adds more equations to the system in Equation 2, also appears to yield more robust results. However, using too many bins as in the FMM method might lead to deteriorating performance.
Interestingly, all quantification methods that do not apply a classifier do not appear to be as competitive compared to the best methods. Since this also holds for fully discrete datasets such as dota this is likely not due to feature binning. While the readme method might yield stronger improvement through an added parameter optimization step, the ED and HDx methods do not have any parameters to optimize. On the other hand, for most quantifiers that apply base classifiers to form their predictions, we observe that tuning parameters of this base classifiers hardly seems to have a positive effect on the quality of the resulting predictions. For the quantifiers that utilize misclassification rates, an explanation might be that having somewhat higher misclassification rates may actually yield more numerical stability in the predictions. Next to the CDE method, only the classify and count-based methods CC and PCC posed on exception to this pattern, but it has also to be expected that for this kind of algorithm, optimizing the base classifiers would beneficial. However, these methods displayed the general trend of yielding worse performance overall than the other classes of quantifiers, even after hyperparameter tuning. Similarly, classifiers that optimize quantification-oriented loss-functions also tended to show worse performance than the majority of other quantifiers. This is overall another strong indicator that pure classification without adjustments for potential distributions shifts does not perform well for quantification. The reason for that is that under a shift in the target distribution, predictions are strongly biased towards the training distribution, as exemplified in our experiments. This practical outcome is also clearly in line with Forman's Theorem [12], which states that when a distribution shift is given, a bias in the CC estimates towards the training distribution is to be expected. This finding bears some contrast to a recent discussion of this kind of approach by Moreo and Sebastiani [23], who have reassessed the performance of the Classify and Count approach, and found that when doing careful optimization of hyperparameters, such quantification-oriented classification approaches would deliver near-state-of-the-art performance, although still being inferior. Our experimental outcomes suggest that this type of approach should be used only carefully for quantification, as a vulnerability towards distribution shifts in theory as well as in experimental results can be observed clearly. Further, aside from the base classifiers, we did not optimize hyperparameters for all other algorithms in our experiments either, and we would expect positive effects in that regard if, for instance, the number of bins in the DyS framework was optimized.
Considering the multiclass case, results are qualitatively different. Most notably, error scores were significantly higher than in binary quantification. Another key difference is that methods such as DyS, MS or TSMax, which have excelled in binary quantification, only showed mediocre performance in the multiclass case. By contrast, distribution matching methods that naturally extend to the multiclass setting appeared to work best, although there was no clear best-performing algorithm. This appears to indicate that generalizing binary quantification results to the multiclass case via a one-vs.-rest approach is not an optimal strategy for multiclass quantification. A reason for this might be buried in the normalization of the results from the one-vs.-rest quantifications, in which small errors in the predictions of individual class prevalences might get amplified. Further, we found that, unlike in the binary case, fine-tuning hyperparameters of the underlying base classifiers appears to have a significant positive effect on the quality of the resulting predictions for many of the given quantifiers. In particular for the EM method, a strong effect could be observed, and with tuned base classifiers this method appeared to clearly outperform the other quantifiers. However, compared to the other experimental analyses, we had a relatively small number of only nine multiclass datasets on which we performed this hyperparameter tuning, so we urge being careful about the generalizability of these findings.
Ultimately, our results indicate that quantification in the multiclass setting is a much harder task than in the binary setting. This points to multiclass quantification as a high-potential problem for future research efforts.

Limitations
This paper presents an extensive empirical comparison of state-of-the-art quantification methods. As such, our results are necessarily affected by some experimental design choices.
First of all, in our experiments, we relied on default parameters of the individual algorithms and did not perform extensive hyperparameter optimization for the quantification algorithms on each dataset. While on the one hand this is due to computational considerations-we have performed more than 295,000 experiments with 10 sampling iterations each, making extra hyperparameter optimization steps infeasible-this also reflects the performance an "uninitiated" data analyst would achieve using those methods off-the-shelf. Obviously, some methods such as DyS and readme that feature more hyperparameters could profit more from optimizations. However, we evaluated the effects of tuning the underlying classifiers, see Section 5.3. These experiments had to be limited to small and medium-sized dataset due to computational requirements.
Regarding the choice of datasets, training set sizes and distribution shifts, we aimed to cover a wide range of possible application scenarios. However, unintended biases in the selection of the datasets or scenarios might lead to biases in the aggregated results. Yet, we would like to note that this study does not include algorithms from the authors or collaborators, such that the authors do not have stakes in any particular outcome. In that regard, the authors stress that an on average subpar performance of a specific quantification method does not imply that this method might not excel in other particular scenarios.

Conclusions
In this study, we have conducted a thorough experimental comparison of 24 quantification methods over 40 datasets, involving more than 5 million algorithm runs. In our experiments, we have both considered the binary and the multiclass case in quantification, and have also specifically considered the impact of shifting target label distributions between training and test data, as well as the impact of having relatively small training sets. In the binary case, we have identified two methods which generally appear to work best, namely, the distribution matching approach from the DyS framework [20] as well as the Median Sweep method [12]. Regarding the multiclass case, a group of distribution matching methods, which naturally extend to multiclass quantification, appeared to be generally superior to the other evaluated algorithms.
We provide further evidence that the multiclass setting in general is much harder to solve for established quantification methods, as the obtained error scores consistently were multiple times higher than in the binary case. However, in contrast to the binary setting, tuning the base classifiers appears to have a distinctive positive effect on the resulting quantifications. This indicates a certain potential for future research on this specific setting. In addition, our results demonstrate that algorithms that are based on the classify and count principle, even when the underlying classifier is optimized for quantification, on average exhibit worse performance compared to other specialized solutions. Overall, we hope our findings provide guidance to practitioners in choosing the right quantification algorithm for a given application and aid researchers in identifying promising directions for future research.
In the appendix, we present additional analytical results regarding the ranking of algorithms. We compute the average ranks of all algorithms aggregated per data set filtered by several conditions. Then, we apply a Nemenyi post-hoc test at 5% significance. In the individual plots, we then show for each algorithm the average performance rank over all algorithms. Horizontal bars indicate which algorithm's average rankings do not differ to a degree that is statistically significant, cf. [4].
Complementing the results from Section 5.1, Figure 9 displays the distributions of rankings under varying shifts between training and target data. Figure 10 displays the rankings of quantification methods when only few training samples are given. In both figures, we observe that the rankings are very similar to the general cases. However, we observe a stronger distinction in the average ranks for high shifts and few training data. Figure 11 and Figure 12 complement the results from Section 5.2 by presenting additional rankings in the multiclass settings. Figure 11 displays the distributions of rankings of quantification algorithms under minor and major shifts between training and target data. Figure 12 displays the rankings of multiclass quantifiers when only settings with few training samples are considered.
Finally, we present additional results from our experiments on quantification using tuned classifiers. Table 6 displays the average error scores of all considered algorithms per dataset in the binary setting, whereas Table 7 shows the corresponding results in the multiclass setting.  Figure 9: Impact of distribution shifts on algorithm rankings for binary labels. We plot the distributions of rankings separated by minor, medium, and major shifts with a Nemenyi post-hoc test at 5% significance.  Figure 11: Impact of distribution shifts on algorithm rankings for multiclass target labels. We plot the distributions of rankings separated by minor, medium, and major shifts with a Nemenyi post-hoc test at 5% significance.  Table 6: Main results for binary quantification using tuned classifiers. We show the averaged error scores over all scenarios per algorithm and data set.  Table 7: Main results for multiclass quantification using tuned classifiers. We show the averaged error scores over all scenarios per algorithm and data set.