Hedging using reinforcement learning: Contextual $k$-Armed Bandit versus
  $Q$-learning
  The construction of replication strategies for contingent claims in the
presence of risk and market friction is a key problem of financial engineering.
In real markets, continuous replication, such as in the model of Black, Scholes
and Merton (BSM), is not only unrealistic but it is also undesirable due to
high transaction costs. A variety of methods have been proposed to balance
between effective replication and losses in the incomplete market setting. With
the rise of Artificial Intelligence (AI), AI-based hedgers have attracted
considerable interest, where particular attention was given to Recurrent Neural
Network systems and variations of the $Q$-learning algorithm. From a practical
point of view, sufficient samples for training such an AI can only be obtained
from a simulator of the market environment. Yet if an agent was trained solely
on simulated data, the run-time performance will primarily reflect the accuracy
of the simulation, which leads to the classical problem of model choice and
calibration. In this article, the hedging problem is viewed as an instance of a
risk-averse contextual $k$-armed bandit problem, which is motivated by the
simplicity and sample-efficiency of the architecture. This allows for realistic
online model updates from real-world data. We find that the $k$-armed bandit
model naturally fits to the Profit and Loss formulation of hedging, providing
for a more accurate and sample efficient approach than $Q$-learning and
reducing to the Black-Scholes model in the absence of transaction costs and
risks.
