Parameter-Free Online Convex Optimization with Sub-Exponential Noise
  We consider the problem of unconstrained online convex optimization (OCO)
with sub-exponential noise, a strictly more general problem than the standard
OCO. In this setting, the learner receives a subgradient of the loss functions
corrupted by sub-exponential noise and strives to achieve optimal regret
guarantee, without knowledge of the competitor norm, i.e., in a parameter-free
way. Recently, Cutkosky and Boahen (COLT 2017) proved that, given unbounded
subgradients, it is impossible to guarantee a sublinear regret due to an
exponential penalty. This paper shows that it is possible to go around the
lower bound by allowing the observed subgradients to be unbounded via
stochastic noise. However, the presence of unbounded noise in unconstrained OCO
is challenging; existing algorithms do not provide near-optimal regret bounds
or fail to have a guarantee. So, we design a novel parameter-free OCO algorithm
for Banach space, which we call BANCO, via a reduction to betting on noisy
coins. We show that BANCO achieves the optimal regret rate in our problem.
Finally, we show the application of our results to obtain a parameter-free
locally private stochastic subgradient descent algorithm, and the connection to
the law of iterated logarithms.
