Bi-level Off-policy Reinforcement Learning for Volt/VAR Control
  Involving Continuous and Discrete Devices
  In Volt/Var control (VVC) of active distribution networks(ADNs), both slow
timescale discrete devices (STDDs) and fast timescale continuous devices
(FTCDs) are involved. The STDDs such as on-load tap changers (OLTC) and FTCDs
such as distributed generators should be coordinated in time sequence. Such VCC
is formulated as a two-timescale optimization problem to jointly optimize FTCDs
and STDDs in ADNs. Traditional optimization methods are heavily based on
accurate models of the system, but sometimes impractical because of their
unaffordable effort on modelling. In this paper, a novel bi-level off-policy
reinforcement learning (RL) algorithm is proposed to solve this problem in a
model-free manner. A Bi-level Markov decision process (BMDP) is defined to
describe the two-timescale VVC problem and separate agents are set up for the
slow and fast timescale sub-problems. For the fast timescale sub-problem, we
adopt an off-policy RL method soft actor-critic with high sample efficiency.
For the slow one, we develop an off-policy multi-discrete soft actor-critic
(MDSAC) algorithm to address the curse of dimensionality with various STDDs. To
mitigate the non-stationary issue existing the two agents' learning processes,
we propose a multi-timescale off-policy correction (MTOPC) method by adopting
importance sampling technique. Comprehensive numerical studies not only
demonstrate that the proposed method can achieve stable and satisfactory
optimization of both STDDs and FTCDs without any model information, but also
support that the proposed method outperforms existing two-timescale VVC
methods.
