On Value Functions and the Agent-Environment Boundary
  When function approximation is deployed in reinforcement learning (RL), the
same problem may be formulated in different ways, often by treating a
pre-processing step as a part of the environment or as part of the agent. As a
consequence, fundamental concepts in RL, such as (optimal) value functions, are
not uniquely defined as they depend on where we draw this agent-environment
boundary, causing problems in theoretical analyses that provide optimality
guarantees. We address this issue via a simple and novel boundary-invariant
analysis of Fitted Q-Iteration, a representative RL algorithm, where the
assumptions and the guarantees are invariant to the choice of boundary. We also
discuss closely related issues on state resetting and Monte-Carlo Tree Search,
deterministic vs stochastic systems, imitation learning, and the verifiability
of theoretical assumptions from data.
