Learning to Play Pong using Policy Gradient Learning
  Activities in reinforcement learning (RL) revolve around learning the Markov
decision process (MDP) model, in particular, the following parameters: state
values, V; state-action values, Q; and policy, pi. These parameters are
commonly implemented as an array. Scaling up the problem means scaling up the
size of the array and this will quickly lead to a computational bottleneck. To
get around this, the RL problem is commonly formulated to learn a specific task
using hand-crafted input features to curb the size of the array. In this
report, we discuss an alternative end-to-end Deep Reinforcement Learning (DRL)
approach where the DRL attempts to learn general task representations which in
our context refers to learning to play the Pong game from a sequence of screen
snapshots without game-specific hand-crafted features. We apply artificial
neural networks (ANN) to approximate a policy of the RL model. The policy
network, via Policy Gradients (PG) method, learns to play the Pong game from a
sequence of frames without any extra semantics apart from the pixel information
and the score. In contrast to the traditional tabular RL approach where the
contents in the array have clear interpretations such as V or Q, the
interpretation of knowledge content from the weights of the policy network is
more illusive. In this work, we experiment with various Deep ANN architectures
i.e., Feed forward ANN (FFNN), Convolution ANN (CNN) and Asynchronous Advantage
Actor-Critic (A3C). We also examine the activation of hidden nodes and the
weights between the input and the hidden layers, before and after the DRL has
successfully learnt to play the Pong game. Insights into the internal learning
mechanisms and future research directions are then discussed.
