DeepScalper: A Risk-Aware Deep Reinforcement Learning Framework for
  Intraday Trading with Micro-level Market Embedding
  Reinforcement learning (RL) techniques have shown great success in
quantitative investment tasks, such as portfolio management and algorithmic
trading. Especially, intraday trading is one of the most profitable and risky
tasks because of the intraday behaviors of the financial market that reflect
billions of rapidly fluctuating values. However, it is hard to apply existing
RL methods to intraday trading due to the following three limitations: 1)
overlooking micro-level market information (e.g., limit order book); 2) only
focusing on local price fluctuation and failing to capture the overall trend of
the whole trading day; 3) neglecting the impact of market risk. To tackle these
limitations, we propose DeepScalper, a deep reinforcement learning framework
for intraday trading. Specifically, we adopt an encoder-decoder architecture to
learn robust market embedding incorporating both macro-level and micro-level
market information. Moreover, a novel hindsight reward function is designed to
provide the agent a long-term horizon for capturing the overall price trend. In
addition, we propose a risk-aware auxiliary task by predicting future
volatility, which helps the agent take market risk into consideration while
maximizing profit. Finally, extensive experiments on two stock index futures
and four treasury bond futures demonstrate that DeepScalper achieves
significant improvement against many state-of-the-art approaches.
