{'fraction_non_alphanumeric': 0.03651016346373454, 'fraction_numerical': 0.009918757545107508, 'mean_word_length': 4.229483023374851, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Risk-Averse Action Selection Using Extreme Value Theory Estimates of the CVaR

The Conditional Value-at-Risk (CVaR) is a useful risk measure in machine learning, finance, insurance, energy, etc. When the CVaR confidence parameter is very high, estimation by sample averaging exhibits high variance due to the limited number of samples above the corresponding threshold. To mitigate this problem, we present an estimation procedure for the CVaR that combines extreme value theory and a recently introduced method of automated threshold selection by Bader et al. (2018). Under appropriate conditions, we estimate the tail risk using a generalized Pareto distribution. We compare empirically this estimation procedure with the naive method of sample averaging, and show an improvement in accuracy for some specific cases. We also show how the estimation procedure can be used in reinforcement learning by applying our method to the multi-armed bandit problem where the goal is to avoid catastrophic risk.


Introduction
In the stochastic multi-armed bandit problem, a learning agent is presented with the repeated task of selecting from a number of choices (arms), each providing independent and identically distributed rewards. The agent has no prior knowledge of the reward distributions. Through feedback observation of the reward with a combination of exploration and exploitation, the agent attempts to identify the arm with the most favorable reward distribution-see Sutton and Barto (1998) for a description of such a setting.
In the traditional version of the problem, the most favorable distribution maximizes the expected reward over time. However, more recent generalizations of this problem have been considered in the literature where the expectation objective is replaced by other metrics aimed at measuring risk.
For instance, Sani et al. (2012) and Yu and Nikolova (2013) address the multi-arm bandit problem with a risk-averse agent. The risk considered may either be instantaneous, i.e., considering risk for a single draw of a reward, or cumulative, i.e., considering jointly all subsequent rewards.
The agent may be interested in minimizing the impact of a rare catastrophic loss. Risk measures targeted at quantifying exposure to extreme losses are well studied in the risk management literature. A popular example introduced by Rockafellar and Uryasev (2002) is the Conditional Value-at-Risk (CVaR) which measures the average loss given that the latter exceeds a given quantile of its distribution. Theoretical properties of the CVaR risk measure are studied for instance in Acerbi and Tasche (2002) and Sarykalin et al. (2008). Note that not all risk measures are exclusively targeting catastrophic risk; other measures also quantify the impact of moderate unfavorable outcomes, see for instance the semi-variance. Nevertheless, the objective of the current paper is to tackle extreme risk minimization, which makes CVaR a suitable choice in this context.
An important challenge that the agent faces when using the CVaR as the objective function in the multi-armed bandit context is the estimation of the CVaR from a finite sample of observations. If an extreme quantile confidence level is given for the CVaR, the sparsity of observations lying in the tail of the distribution can yield imprecise results in common calculation methods such as sample averaging. We propose to employ results from extreme value theory (see for instance McNeil et al., 2005) to obtain better estimates of the CVaR. In particular, the Pickands- Balkema-de Haan Theorem presents a parametric approximation of the tail data using a generalized Pareto distribution (GPD) (Pickands III et al. (1975), Balkema and De Haan (1974)). The theorem states that by selecting an appropriate threshold, the distribution of tail data beyond that threshold can be well approximated by the GPD. The parametric modeling of the tail distribution is often referred to the peaks over threshold approach, which is investigated for instance in Simiu and Heckert (1996), Ferreira and Guedes Soares (1998), Frigessi et al. (2002), Beguería and Vicente-Serrano (2006) and Zhao et al. (2018).
We combine the peaks over threshold methodology with the recent work of Bader et al. (2018) for automated threshold selection via ordered goodness-of-fit tests to estimate tail distributions and approximate the CVaR. An application to a risk-averse multi-armed bandit problem is then presented.
The current paper is divided as follows. In Section 2, the notation used in the current work is introduced and the risk-averse bandit problem is defined along with the CVaR risk measure. In Section 3, a background on extreme value theory (EVT) is provided, and an approach to estimate the CVaR using EVT is illustrated. In Section 4, the statistical estimation procedures used for the CVaR calculation is discussed, including the automated threshold selection procedure.
In Section 5, details of the multi-arm bandit policy in a risk-averse setting are discussed. In Section 6, results from numerical simulations comparing statistical estimation procedures for the CVaR in the multi-arm bandit setting are shown.

Problem Formulation
This section described the multi-armed bandit framework and the problem that is tackled in the current paper.

The multi-armed bandit framework
The multi-armed bandits framework involves a finite horizon multi-stage decision setting, where an agent makes decisions at stages t = 1, . . . , T . Let K ≡ {1, ..., k} denote a set of arms, which are possible actions that can be taken at each stage. The outcome of each draw from a bandit is considered a cost to the agent (i.e., the larger the value that is sampled, the most unfavorable the outcome is considered). For t = 1, . . . , n, define the k-dimensional random vector X t ≡ (X t 1 , . . . , X t k ) where X t j denotes the cost incurred if the arm j is selected at stage t.
Vectors X 1 , . . . , X n are assumed to be independent and identically distributed. Therefore, for all arms i = 1, . . . , k, cost variables X 1 i , . . . , X n i are i.i.d. copies of some random variable X i . Let {F 1 , . . . , F k } denote the respective cumulative distribution functions (CDF) of X 1 , . . . , X k ; these distribution functions are unknown to the agent.
The sequence of selected arms is denoted by a ≡ (a 1 , . . . , a n ) where a t is the random variable taking values in 1, . . . , k denoting the arm selected at time t. When an arm a t is selected at time t, its associated cost X t at is observed, but the costs associated with all other arms {X t i : i = a t } remain unobserved.
The selection of one of the k arms at each time step is decided through a policy. A policy is a mapping that returns the probabilities of selecting any action at the next stage given the agent's current state. The policy evolves over time as new samples are obtained and results in a sequence of policies π 1 , . . . , π n where, for a given t, the function π t : t−1 i=1 (K × R) → [0, 1] k takes as input all previous realizations of actions and costs, (a 1 , X 1 a 1 ), . . . , (a t−1 , X t−1 a t−1 ) , and maps them into probabilities of selecting any possible next-stage action a t .
Policies considered in the current paper attempt to identify the arm with the least risk, as quantified through a risk measure. Let χ denote a set of random variables. For a given confidence level α ∈ (0, 1), let ρ α : χ → R denote a law-invariant 1 risk measure.
Since the cost probability distributions are a priori unknown, every time an arm is sampled, the estimate of the risk associated with the sampled arm is refined. The notationρ t α (X i ) is used to refer to the estimate of ρ α (X i ) after the first t stages. The least risky arm is denoted i * = arg min i∈K ρ α (X i ).

The CVaR risk measure
Various risk measures have been considered in the bandit problem literature, for instance the variance in Sani et al. (2012), or the Value-at-Risk (VaR) and Average Value-at-Risk (AVaR) in Yu and Nikolova (2013). The Conditional-Value at-Risk (CVaR), which is a synonym of the AVaR, is used in the current work. Hence, we assume that the agent focuses on minimizing the risk of a catastrophic loss. For a given random variable Y along with its CDF F Y , the quantile of confidence level α of the distribution of Y is defined as This allows to define in turn the CVaR as in Rockafellar and Uryasev (2002) as the mean of the of Y which has the following CDF: Typical values of α are 0.95, 0.99 or 0.999.
The current paper considers exclusively one risk measure: the CVaR with a given confidence level α. If the random variable Y is absolutely continuous, it can be shown that which gives and intuitive interpretation to the CVaR. Without loss of generality, the current work will only consider absolutely continuous variables for simplicity.
Note that all results in the current work could be easily generalized to consider the optimization of a risk-reward tradeoff by selecting an objective function of the form instead of the purely risk-centric framework ρ α ≡ CV aR α .

Sample CVaR estimation
Since for each arm j the CDF F X j is unknown, it must be estimated from costs previously sampled from the arm j. Consider an i.i.d. sample S t = {y 1 , . . . , y t } of observations drawn from a distribution F Y . For every y ∈ R, the sample CDF estimator is defined aŝ The sample CDF can be plugged into the definition of the quantile and the CVaR to obtain naive estimators of these quantities. Let {y (1) , . . . , y (t) } be the set of order statistics, i.e., the observations sorted in non-decreasing order. Then, the naive quantile estimator iŝ and in turn the naive CVaR estimator is (2.3) A confidence interval for the sample CVaR estimate can be obtained through bootstrapping as described in Appendix A.1. Such confidence intervals can be useful to design upper-confidencebound (UCB) action selection schemes as described in Sutton and Barto (1998). Such schemes are left out-of-scope of the current paper.

Estimating the CVaR through extreme value theory
The use of the sample CDF to estimate CV aR α can be problematic when the sample size is small and the confidence level α is large. The scarcity of sampled observations lying in the tail of the distribution can lead to a volatile estimate of the tail distribution and thus of the CVaR. We therefore turn to extreme value theory, which was developed in an attempt to estimate the tail distribution from scarce samples by exploiting the asymptotic behavior of the tail distribution above increasingly high quantiles. This section shows how to use extreme value theory to approximate the CVaR, and in turn to estimate the approximation from i.i.d. observations.
Proofs for some of the results are given in Appendix B.

The Pickands-Balkema-de Haan theorem and CVaR approximation
For a random variable Y with CDF F Y and a given threshold u > ess inf Y , the excess distribution function K u is defined for z > 0 as Note that the domain of K u is [0, ess sup Y ). The z-values are referred to as the threshold excesses.
Given that Y has exceeded some high threshold u, this function represents the probability that it exceeds the threshold by at most z. When F Y is unknown, K u cannot be calculated directly.
Finding an approximation to this function is the motivation of the subsequent results, which can Definition 3.1 (GPD). The generalized Pareto distribution (GPD) with two parameters ξ ∈ R and σ > 0 is a continuous probability distribution with PDF Over its support, the CDF is given by Let Y be a random variable whose distribution function is F Y . The following theorem, known as the Pickands-Balkema-de Haan Theorem, states that under certain conditions and for any large enough u, the threshold exceedances CDF z → K u (z) is well approximated by the GPD. Two additional definitions are needed to state the theorem.
Definition 3.2 (GEV). The generalized extreme value (GEV) distribution with single parameter for all y ∈ R.
Theorem 3.1 (Pickands-Balkema-de Haan). Consider a real value ξ and a random variable Y such that y max ≡ ess sup Y ≤ ∞ and that F Y ∈ MDA(H ξ ). Then there exists a positive function The property F Y ∈ MDA(H ξ ) for some ξ holds for a large class of distributions, in particular it holds for all common continuous distributions (e.g., uniform, normal, Student-T, exponential, beta).
Using the Pickands-Balkema-de Haan Theorem, we now proceed to derive our main result, which is an approximator of the CVaR.
where β is the function specified in Theorem 3.1.

Statistical estimation of the CVaR approximation
In practice, using the CVaR approximation (3.3) requires identifying suitable values for the threshold u and parameters ξ and σ = β(u) from a sample of observations S t = {y 1 , . . . , y t }. Such considerations are discussed in the current section.
4.1 Estimating (ξ, σ) for a given threshold u First, assume that the threshold u is pre-determined, and that parameters ξ and σ are estimated based on such a choice u. The maximum likelihood approach for the estimation of such parameters is a typical procedure. Consider the set of excesses over the threshold u defined by Elements of Z u are independent, identically distributed and approximately distributed as GPD(ξ, σ) with σ = β(u) for some mapping β by Theorem 3.1. The maximum likelihood estimator entails solving the following optimization problem: where g ξ,σ is defined in (3.1). Such an optimization must conducted numerically as closed-form solutions to this problem are not available. In the current paper, since we want to consider integrable distributions (so that the CVaR exists), and thus the constraint ξ < 1 is imposed when the maximum likelihood optimization (4.1) is applied.
This leads to an estimate of CV aR α (Y ) of based on (3.3): where (ξ u ,σ u ) are obtained from (4.1). An asymptotic confidence interval for the CVaR estimate can be derived by combining the asymptotic maximum likelihood variance of parameter estimates and the delta method, see Appendix A.2.
The misspecification of the tail distribution, i.e. the fact that the conditional tail distribution is not exactly a GPD distribution in general, causes the estimator (4.2) to be asymptotically biased (i.e. as the number of samples tends to infinity) in general. The construction of the confidence interval based on the delta method also disregards the conditional tail distribution misspecification issue, which leads to a loss in precision.

Estimating the extreme quantile q α with EVT
The calculation of CV aR α (Y ) requires determining its quantile q α . A first possibility would be to use the naive estimate given by (2.2). However, extreme value theory can also be used for such purpose.
Assume that the threshold u that is used in the CVaR estimation procedure is smaller than the quantile of interest, i.e. q α ≥ u. Denoteq α as the approximation of q α , and recall (2.1) which definesF t Y as the empirical CDF generated by S t = {y 1 , . . . , y t }, a sample from i.i.d.
copies of Y . The following results gives the approximation formula for q α which relies on the Pickands-Balkema-de Haan theorem theorem. Without loss of generality, only the result for ξ > 0 is provided.
Corollary 4.1. Assume that q α ≥ u and that F Y ∈ MDA(H ξ ) for some ξ > 0. Then the quantile q α of the distribution of Y can be approximated througĥ where estimatesξ andσ are provided by (4.1).

Choosing the threshold u
The selection of a suitable threshold u is a much harder problem. The choice of u is a balancing act between bias and variance: if u is too low, the asymptotic approximation of the tail of the distribution by the GPD shall not be sufficiently accurate. If u is too high, too few threshold excesses are available in Z u , and the GPD parameters estimates will be unreliable.
Multiple approaches for setting the threshold u were considered in the literature, see for instance Scarrott and MacDonald (2012) (j) ≡ Gθ i (z j ) for 1 < j < n i is applied, where G is the GPD CDF from (3.2).
The AD statistic is then .
Corresponding p-values for each test statistic can then be found by referring to a lookup table (Choulakian and Stephens (2001)). Finally, using the p-values p 1 , . . . , p l calculated for each test, the ForwardStop rule of G'Sell et al. (2016) is used to choose the threshold. This is done by calculating a cutoffk where γ is a chosen significance parameter. Under this rule, the threshold uk F +1 is chosen. If nô k F exists, then no rejection is made and u 1 is chosen.
Thus, summarizing the overall tail distribution estimation procedure, the threshold and GPD parameter estimates are respecitvely provided by

Multi-armed bandits policies
The current section outlines the proposed policies that are investigated in the simulation study of the next section for the context of multi-armed bandits problems.
For each considered policy, after each stage t, an estimate CV aR t α (X j ) is available for all arms.
Such estimates can be used to determine the action at the subsequent stage. The CVaR estimates for all arms allow defining an -greedy policy which is now described. Consider the following deterministic sequence ≡ { t } n t=1 containing real numbers in [0, 1]. The sequence is referred to as a schedule. t defines the probability of making an exploratory move at stage t instead of exploiting knowledge (i.e. selecting the perceived least risky action). Typically, the schedule is a decreasing sequence so as to progressively reduce the amount of exploration as the costs distribution estimated become more precise. Let Π t,j be the probability of selecting action j at stage t. Such quantities characterize the policy followed by the agent. The -greedy policy entails choosing the action at stage t according to the following rule: In other words, at stage t such a policy entails choosing randomly uniformly across all arms with a probability t , or selecting the greedy action (i.e. the one with the least estimated risk) with probability 1 − t . When more than a single action reach the minimal estimated risk among all arms (i.e. when the arg min set is not a singleton), the arm with the minimum index is selected to break the tie.
To determine the estimates CV aR t α (X j ), two methodologies are compared. The first estimation approach contemplated is the naive sample CVaR estimation stemming from (2.2)-(2.3). This approach is referred to subsequently as the Sample Average (SA) method. The second methodology considered involves the extreme value theory estimator outlined in Section 3 and Section 4. The description of such an approach referred to as the Extreme Value Theory (EVT) method is provided next.
For each arm j, let S j t = {y j s : a s = j, s = 1, . . . , t} be the sample containing all rewards sampled from arm j between stage 1 and t. The number of elements of the set S j t is denoted N j t ≡ t s=1 1 {as=j} . Before stage 1, all CVaR estimates are set to zero: Subsequently, each time an action j is selected at some stage t, the associated CVaR estimate is refined based on the new cost outcome generated by arm j. To update the CVaR estimate, a threshold u t j is selected based on observations S j t . The set of threshold exceedances over the threshold u t j computed from the set S j t are then used to estimate the corresponding Generalized Pareto distribution parameters as indicated in (4.1). This allows using (4.2) as the updated CVaR estimate CV aR t α (X j ), where the quantile q α is estimated according to (4.3). For all other arms i.e. for all = a t , the CVaR estimate is left untouched i.e. CV aR t α (X ) ≡ CV aR t−1 α (X ).
Throughout the rest of the paper, it is assumed that the reward distribution associated with each arm satisfies the MDA assumption, i.e. that for all j = 1, . . . , k, there exists ξ j < 1 such that . Such an assumption is not very restrictive as it holds for a very large class of distributions. The integrability assumption underlying ξ j < 1 is neither very restrictive in practice. This implies that the estimate (4.2) is valid to approximate the CVaR associated with any arm j, i.e. CV aR α (X j ), provided the threshold u is sufficiently large for each arm.

Simulation Studies
In this section, both Sample Average (SA) and Extreme Value Theory (EVT) CVaR estimation methods described in the previous section are compared within a simulation study. Two simulation experiments will be conducted. The first is a pure statistical estimation problem where i.i.d. costs from a single arm are sequentially observed, and the cost distribution CVaR estimated based on both respective methods are updated every time a new observation becomes available. This allows evaluating the statistical accuracy of both methods. The second simulation experiment embeds the two respective CVaR estimation methods within a multi-armed bandit problem so as to assess their suitability for sequential action selection. The current section provides details about these experiments and outlines numerical results obtained.

Single-arm CV aR α estimation experiment
The single-arm problem where all costs are i.i.d. samples from an unknown distribution is first considered. The estimation performance of the SA and EVT methods is compared. The experiments consists in performing M = 1,000 independent runs. Each run consists in sequentially sampling n = 5000 independent costs from the single arm, and every time a new sample is observed the CVaR estimates are updated according to both respective methods.
Three families of distributions are considered for the arm costs: GPD, Weibull (WE) and lognormal (LN). The density of the last two is given by Such distributions are chosen since the exact value of the CV aR α can be derived exactly, see Norton et al. (2019) for formulas which we repeat for completeness purposes. If X follows a Weibull(κ, λ) distribution, then where Γ(a, b) = ∞ b p a−1 e −p dp is the upper incomplete gamma function Moreover, if X follows a lognormal(µ, σ) distribution, then where Φ and Φ −1 are respectively the standard normal CDF and its inverse. For the GPD distribution, the tail distribution is exactly GPD distributed as explained in Lemma B.1, and therefore the EVT approximation of the CV aR α is asymptotically unbiased (i.e. as the number of stages tends to infinity). For the Weibull and lognormal distributions, the EVT approximation is clearly biased, and the simulation experiments shall help investigating whether the reduction in variance provided by the EVT in comparison to the SA method is sufficient to offset the bias of the former method.
The performance of estimates is assessed using two metrics. For m = 1, . . . , M , denote the stage-t estimate of the arm j CV aR α for run m by m CV aR t α (X j ). The first is the commonly used root-mean-square error (RMSE): Since the RMSE is sensitive to outliers, a second metric is also considered: the percentage of times that the EVT CV aR α estimate is closer to the true value of the CV aR α than the SA estimate across all runs. We refer to this metric as Fraction Closer subsequently.
To summarize the simulation procedure, for each run m, at each stage t, calculations are performed on the first t observations {x 1 , . . . , x t } with the following procedure: 1. For the EVT estimate, consider a set of candidate thresholds u 1 , . . . , u l . 3. Calculate CV aR α estimates using the SA and EVT methods.

For
The confidence level of the CVaR in the simulation experiments is set to α = 0.999. A high confidence level is considered since the scarcity of observations is more important for such levels; this is where the EVT method is most likely to outperform the SA counterpart and prove the most useful. In all simulations, at stage t, u 1 and u l are respectively set to the 0.7 and α confidence level sample quantiles of the empirical distribution of costs sampled previously in the run from the arm. The number of threshold considered is set l = 50, and the threshold u j is set as the empirical cost distribution quantile with confidence levelα j =α 1 + (α l −α 1 ) j−1 l , j = 1, . . . l; equally spaced threshold confidence levels spanning the interval [0.7, α] are used. The ForwardStop rule confidence level γ was set to 0.1.
To provide additional stability to the EVT approach, a small modification to the threshold procedure was applied. Whenever for a given candidate threshold u the maximum likelihood estimates (4.1) for exceedances are such thatξ u > 0.9, the threshold u was automatically discarded. This is due to the expression 1 − ξ found at the denominator of the CVaR approximation 3.3 which can make the estimate explode whenξ u is close to one. Although this comes at the expense of generating some additional bias when the ξ associated with the limiting distribution is greater than 0.9, this modification to the algorithm never reduced its performance in some unreported tests performed by the authors. A general observation which can be made is that for most of the tested parameter configurations which are reported above, the EVT method tends to be underperform the SA and exhibit a lesser stability in earlier stages in terms of RMSE. However, at subsequent stages, the EVT estimate tends to stabilize and eventually provides a better performance than the SA estimate. The same phenomenon is observed when looking at the Fraction Closer metric. An interesting observation is that EVT starts outperforming the SA according to the Fraction Closer earlier than it does in terms RMSE. Since the RMSE is very sensitive to large errors contrarily to the Fraction Closer, this tends to indicate that the EVT approach can lead to larger errors than the SA before it stabilizes. This could partly be due to a large EVT estimator variance in early stages when the estimateξ is not very precise and can take values close the the 0.9 limit that was set; this would lead to very large CVaR estimates due to the reciprocal of 1 − ξ found in (3.3) as mentioned previously.

Best Arm Selection in a multi-armed bandits simulation
In the current section, results from the outcome of a 5-armed testbed simulation inspired from Sutton and Barto (1998) are provided. This experiment is analogous to the one from Section 6.1, except there are now k = 5 arms from which to sample costs instead of one. The cost distribution is different for each arm, and thus a distinct estimate for the CVaR is formed for each of the arms.
The arm selection policy considered is the -greedy one described in Section 5. To encourage exploration, a fully random arm selection is used for the first 1000 stages, whereas for subsequent stages the exploration probability is set to 0.1. This entails using ( = 1), while is set to 0.1 thereafter, which corresponds to the schedule The performance metric considered for the multi-arm bandit experiments is referred to as the Percent Best Action which represents the percentage of time across all runs the less risky arm is selected at a given stage t. Figure 4 provides values obtained for that metric for each of the three experiments at all stages of the simulation.
The main lesson obtained from the multi-armed bandits simulation results is qualitatively the same than for the single-arm experiment: for early stages, the SA method performs better than the EVT, but the EVT eventually catches up and outperforms the SA in its ability to select the less risky arm. This clearly demonstrates the usefulness of considering an EVT estimation method for the CVaR when considering a multi-armed bandit action selection framework.

Conclusion
The current work proposes the inclusion of risk estimates based on extreme value theory within sequential decision problems to allow a risk-averse agent performing his action selection. More precisely, in a multi-armed bandits framework, the risk-averse agent is assumed to attempt minimizing the CVaR of costs he incurs. The Pickands-Balkema-de Haan theorem from EVT is invoked to form parametric estimates of the CVaR of costs associated with each respective action; information about the asymptoptic behavior of the tail of the cost distribution is leveraged so as to form a parametric estimation of that tail distribution relying on the Generalized Pareto distribution. An important step in the formation of such estimate is the determination of a suitable threshold above which the tail distribution is deemed sufficiently close to its asymptotic distribution. The novel methodology based on sequential goodness-of-fit tests provided by Bader et al. (2018) in the context of threshold selection for extreme values modeling is used for such purposes.
The estimation method based on EVT is compared within some numerical experiments to a more naive approach for the estimation of the CVaR relying on sample averaging. The first numerical Theory (EVT) CVaR estimation methods in three 5-armed bandit testbed simulations. The first, second and third experiments involves respectively GDP, lognormal and Weibull distributed arm costs. The goal of the agent is to select actions so as to minimize the T CE 0.999 of the arm cost. The distribution parameters of the various arms is provided for within each subpanel's caption.
simulation involves the estimation of the cost CVaR in a context where a single action is available.
It purely is an estimation problem as not action selection is involved. Results show that the EVT approach can outperform its sample averaging counterpart in some specific cases by leading to a smallest root-mean-square error for the cost CVaR estimate. The second simulation involves the classic 5-armed testbed experiment from Sutton and Barto (1998) where the agent uses an -greedy policy to attempt minimizing the CVaR he incurs (instead of maximizing rewards as in the classic case). Results from this experiment show that using EVT based estimates for the CVaR can help the agent performing better action selection and reducing the risk it incurs.

B Proofs
The following Lemma Lemma B.1 can then be used to obtain the CVaR of a Generalized Pareto distribution.
Since q α ≥ u, Y − u ≥ q α − u implies that Y ≥ u. Furthermore, the CDF of Y − u given Y ≥ u is approximately G ξ,β(u) for some mapping β by Theorem 3.1.
Proof of Corollary 4.1: First, from Theorem 3.1, the distribution of Y − u given Y > u is approximately GPD. Using this approximation, since q α ≥ u would have no atoms in a neighborhood around q α and therefore α ≈ F Y (q α ). absolutely continuous. This implies by conditioning that .
Isolatingq α in the latter expression directly leads to (4.3).