Solving Large-Scale Extensive-Form Network Security Games via Neural
  Fictitious Self-Play
  Securing networked infrastructures is important in the real world. The
problem of deploying security resources to protect against an attacker in
networked domains can be modeled as Network Security Games (NSGs).
Unfortunately, existing approaches, including the deep learning-based
approaches, are inefficient to solve large-scale extensive-form NSGs. In this
paper, we propose a novel learning paradigm, NSG-NFSP, to solve large-scale
extensive-form NSGs based on Neural Fictitious Self-Play (NFSP). Our main
contributions include: i) reforming the best response (BR) policy network in
NFSP to be a mapping from action-state pair to action-value, to make the
calculation of BR possible in NSGs; ii) converting the average policy network
of an NFSP agent into a metric-based classifier, helping the agent to assign
distributions only on legal actions rather than all actions; iii) enabling NFSP
with high-level actions, which can benefit training efficiency and stability in
NSGs; and iv) leveraging information contained in graphs of NSGs by learning
efficient graph node embeddings. Our algorithm significantly outperforms
state-of-the-art algorithms in both scalability and solution quality.
