Deep Reinforcement Learning Based Networked Control with Network Delays
  for Signal Temporal Logic Specifications
  We apply deep reinforcement learning (DRL) to design of a networked
controller with network delays to complete a temporal control task that is
described by a signal temporal logic (STL) formula. STL is useful to deal with
a specification with a bounded time interval for a dynamical system. In
general, an agent needs not only the current system state but also the past
behavior of the system to determine a desired control action for satisfying the
given STL formula. Additionally, we need to consider the effect of network
delays for data transmissions. Thus, we propose an extended Markov decision
process using past system states and control actions, which is called a $\tau
d$-MDP, so that the agent can evaluate the satisfaction of the STL formula
considering the network delays. Thereafter, we apply a DRL algorithm to design
a networked controller using the $\tau d$-MDP. Through simulations, we also
demonstrate the learning performance of the proposed algorithm.
