The Risk of Data Science – Calculating a Classification Value at Risk

Classification algorithms are widely used for detecting a diverse amount of possibly negative events, like fraud or insolvencies.
The reliability of the classifications is generally measured with key figures like accuracy, precision or recall.

Companies using classification algorithms, however, are generally also interested in the actual financial extent of the damage.
Here, they also are often not only interested in the expected extent, but also in the possible extent for negative worst case scenarios.
Hence, classical risk figures for classification predictions (like the Value at Risk; VaR) are very useful.

The follwing text describes an approach for obtaining a Classification VaR (for False Negatives) from a conventional confusion matrix.

The calculation is based on the following assumptions:

  • The False Negative rate is known from the model validation
  • False Negatives are independent and binomially distributed
  • The extent (for single events) and distribution of the possible damages is known

Based on these assumptions, the Classification VaR can be calculated via a Monte Carlo simulation. Here, one has to take care that the calculations can become very fast very time consuming, since many scenarios are necessary to capture high confidence levels.

RiskDataScience developed a Monte Carlo simulation for calculating the Classification VaR and applied it to the following case.

Starting from a known confusion matrix, the appropriate TP, TN, FP and FN rates are calculated. It is assumed that the detection of harmful events (True Positives) prevents the company from financial losses. However, damages are caused due to undetected False Negatives.

The damage extents of the example belong to several known damage classes, each one occuring with a known probability of 10%.

The new – unknown – sample consists of 17,926 instances. In order to calculate the Classification VaR at a confidence level of 95%, a Monte Carlo simulation with 1,000 scenarios was started. In each scenario, the value for each instance was randomly obtained and summed up. The scenarios were sorted and the VaR was directly retrieved from the appropriate scenario.

In this case, the Classification VaR at 95% is € 42 mn, while the mean expected loss would be just € 36 mn; hence one expects at least an additional damage of € 6 mn in the 5% worst case scenarios.

The method can be enhanced to capture more complex cases.

We have developed respective methods and tools and support our customers in obtaining an overall risk perspective of the data science procedures in use.


Dr. Dimitrios Geromichalos
Founder / CEO
RiskDataScience UG (haftungsbeschränkt)
Theresienhöhe 28, 80339 München
Telefon: +4989244407277, Fax: +4989244407001
Twitter: @riskdatascience