Learning Augmented Index Policy for Optimal Service Placement at the
  Network Edge
  We consider the problem of service placement at the network edge, in which a
decision maker has to choose between $N$ services to host at the edge to
satisfy the demands of customers. Our goal is to design adaptive algorithms to
minimize the average service delivery latency for customers. We pose the
problem as a Markov decision process (MDP) in which the system state is given
by describing, for each service, the number of customers that are currently
waiting at the edge to obtain the service. However, solving this $N$-services
MDP is computationally expensive due to the curse of dimensionality. To
overcome this challenge, we show that the optimal policy for a single-service
MDP has an appealing threshold structure, and derive explicitly the Whittle
indices for each service as a function of the number of requests from customers
based on the theory of Whittle index policy.
  Since request arrival and service delivery rates are usually unknown and
possibly time-varying, we then develop efficient learning augmented algorithms
that fully utilize the structure of optimal policies with a low learning
regret. The first of these is UCB-Whittle, and relies upon the principle of
optimism in the face of uncertainty. The second algorithm, Q-learning-Whittle,
utilizes Q-learning iterations for each service by using a two time scale
stochastic approximation. We characterize the non-asymptotic performance of
UCB-Whittle by analyzing its learning regret, and also analyze the convergence
properties of Q-learning-Whittle. Simulation results show that the proposed
policies yield excellent empirical performance.
