Showing Your Offline Reinforcement Learning Work: Online Evaluation
  Budget Matters
  In this work, we argue for the importance of an online evaluation budget for
a reliable comparison of deep offline RL algorithms. First, we delineate that
the online evaluation budget is problem-dependent, where some problems allow
for less but others for more. And second, we demonstrate that the preference
between algorithms is budget-dependent across a diverse range of
decision-making domains such as Robotics, Finance, and Energy Management.
Following the points above, we suggest reporting the performance of deep
offline RL algorithms under varying online evaluation budgets. To facilitate
this, we propose to use a reporting tool from the NLP field, Expected
Validation Performance. This technique makes it possible to reliably estimate
expected maximum performance under different budgets while not requiring any
additional computation beyond hyperparameter search. By employing this tool, we
also show that Behavioral Cloning is often more favorable to offline RL
algorithms when working within a limited budget.
