Optimizing the Long-Term Average Reward for Continuing MDPs: A Technical
  Report
  Recently, we have struck the balance between the information freshness, in
terms of age of information (AoI), experienced by users and energy consumed by
sensors, by appropriately activating sensors to update their current status in
caching enabled Internet of Things (IoT) networks [1]. To solve this problem,
we cast the corresponding status update procedure as a continuing Markov
Decision Process (MDP) (i.e., without termination states), where the number of
state-action pairs increases exponentially with respect to the number of
considered sensors and users. Moreover, to circumvent the curse of
dimensionality, we have established a methodology for designing deep
reinforcement learning (DRL) algorithms to maximize (resp. minimize) the
average reward (resp. cost), by integrating R-learning, a tabular reinforcement
learning (RL) algorithm tailored for maximizing the long-term average reward,
and traditional DRL algorithms, initially developed to optimize the discounted
long-term cumulative reward rather than the average one. In this technical
report, we would present detailed discussions on the technical contributions of
this methodology.
