Using Non-Stationary Bandits for Learning in Repeated Cournot Games with
  Non-Stationary Demand
  Many past attempts at modeling repeated Cournot games assume that demand is
stationary. This does not align with real-world scenarios in which market
demands can evolve over a product's lifetime for a myriad of reasons. In this
paper, we model repeated Cournot games with non-stationary demand such that
firms/agents face separate instances of non-stationary multi-armed bandit
problem. The set of arms/actions that an agent can choose from represents
discrete production quantities; here, the action space is ordered. Agents are
independent and autonomous, and cannot observe anything from the environment;
they can only see their own rewards after taking an action, and only work
towards maximizing these rewards. We propose a novel algorithm 'Adaptive with
Weighted Exploration (AWE) $\epsilon$-greedy' which is remotely based on the
well-known $\epsilon$-greedy approach. This algorithm detects and quantifies
changes in rewards due to varying market demand and varies learning rate and
exploration rate in proportion to the degree of changes in demand, thus
enabling agents to better identify new optimal actions. For efficient
exploration, it also deploys a mechanism for weighing actions that takes
advantage of the ordered action space. We use simulations to study the
emergence of various equilibria in the market. In addition, we study the
scalability of our approach in terms number of total agents in the system and
the size of action space. We consider both symmetric and asymmetric firms in
our models. We found that using our proposed method, agents are able to swiftly
change their course of action according to the changes in demand, and they also
engage in collusive behavior in many simulations.
