Alternative Function Approximation Parameterizations for Solving Games:
  An Analysis of $f$-Regression Counterfactual Regret Minimization
  Function approximation is a powerful approach for structuring large decision
problems that has facilitated great achievements in the areas of reinforcement
learning and game playing. Regression counterfactual regret minimization (RCFR)
is a simple algorithm for approximately solving imperfect information games
with normalized rectified linear unit (ReLU) parameterized policies. In
contrast, the more conventional softmax parameterization is standard in the
field of reinforcement learning and yields a regret bound with a better
dependence on the number of actions. We derive approximation error-aware regret
bounds for $(\Phi, f)$-regret matching, which applies to a general class of
link functions and regret objectives. These bounds recover a tighter bound for
RCFR and provide a theoretical justification for RCFR implementations with
alternative policy parameterizations ($f$-RCFR), including softmax. We provide
exploitability bounds for $f$-RCFR with the polynomial and exponential link
functions in zero-sum imperfect information games and examine empirically how
the link function interacts with the severity of the approximation. We find
that the previously studied ReLU parameterization performs better when the
approximation error is small while the softmax parameterization can perform
better when the approximation error is large.
