Constraint-Guided Reinforcement Learning: Augmenting the
  Agent-Environment-Interaction
  Reinforcement Learning (RL) agents have great successes in solving tasks with
large observation and action spaces from limited feedback. Still, training the
agents is data-intensive and there are no guarantees that the learned behavior
is safe and does not violate rules of the environment, which has limitations
for the practical deployment in real-world scenarios. This paper discusses the
engineering of reliable agents via the integration of deep RL with
constraint-based augmentation models to guide the RL agent towards safe
behavior. Within the constraints set, the RL agent is free to adapt and
explore, such that its effectiveness to solve the given problem is not
hindered. However, once the RL agent leaves the space defined by the
constraints, the outside models can provide guidance to still work reliably. We
discuss integration points for constraint guidance within the RL process and
perform experiments on two case studies: a strictly constrained card game and a
grid world environment with additional combinatorial subgoals. Our results show
that constraint-guidance does both provide reliability improvements and safer
behavior, as well as accelerated training.
