Risk-Averse Action Selection Using Extreme Value Theory Estimates of the
  CVaR
  In a wide variety of sequential decision making problems, it can be important
to estimate the impact of rare events in order to minimize risk exposure. A
popular risk measure is the conditional value-at-risk (CVaR), which is commonly
estimated by averaging observations that occur beyond a quantile at a given
confidence level. When this confidence level is very high, this estimation
method can exhibit high variance due to the limited number of samples above the
corresponding quantile. To mitigate this problem, extreme value theory can be
used to derive an estimator for the CVaR that uses extrapolation beyond
available samples. This estimator requires the selection of a threshold
parameter to work well, which is a difficult challenge that has been widely
studied in the extreme value theory literature. In this paper, we present an
estimation procedure for the CVaR that combines extreme value theory and a
recently introduced method of automated threshold selection by
\cite{bader2018automated}. Under appropriate conditions, we estimate the tail
risk using a generalized Pareto distribution. We compare empirically this
estimation procedure with the commonly used method of sample averaging, and
show an improvement in performance for some distributions. We finally show how
the estimation procedure can be used in reinforcement learning by applying our
method to the multi-arm bandit problem where the goal is to avoid catastrophic
risk.
