Action-Constrained Reinforcement Learning for Frame-Level Bit Allocation
  in HEVC/H.265 through Frank-Wolfe Policy Optimization
  This paper presents a reinforcement learning (RL) framework that leverages
Frank-Wolfe policy optimization to address frame-level bit allocation for
HEVC/H.265. Most previous RL-based approaches adopt the single-critic design,
which weights the rewards for distortion minimization and rate regularization
by an empirically chosen hyper-parameter. More recently, the dual-critic design
is proposed to update the actor network by alternating the rate and distortion
critics. However, the convergence of training is not guaranteed. To address
this issue, we introduce Neural Frank-Wolfe Policy Optimization (NFWPO) in
formulating the frame-level bit allocation as an action-constrained RL problem.
In this new framework, the rate critic serves to specify a feasible action set,
and the distortion critic updates the actor network towards maximizing the
reconstruction quality while conforming to the action constraint. Experimental
results show that when trained to optimize the video multi-method assessment
fusion (VMAF) metric, our NFWPO-based model outperforms both the single-critic
and the dual-critic methods. It also demonstrates comparable rate-distortion
performance to the 2-pass average bit rate control of x265.
