FuzzerGym: A Competitive Framework for Fuzzing and Learning
  Fuzzing is a commonly used technique designed to test software by
automatically crafting program inputs. Currently, the most successful fuzzing
algorithms emphasize simple, low-overhead strategies with the ability to
efficiently monitor program state during execution. Through compile-time
instrumentation, these approaches have access to numerous aspects of program
state including coverage, data flow, and heterogeneous fault detection and
classification. However, existing approaches utilize blind random mutation
strategies when generating test inputs. We present a different approach that
uses this state information to optimize mutation operators using reinforcement
learning (RL). By integrating OpenAI Gym with libFuzzer we are able to
simultaneously leverage advancements in reinforcement learning as well as
fuzzing to achieve deeper coverage across several varied benchmarks. Our
technique connects the rich, efficient program monitors provided by LLVM
Santizers with a deep neural net to learn mutation selection strategies
directly from the input data. The cross-language, asynchronous architecture we
developed enables us to apply any OpenAI Gym compatible deep reinforcement
learning algorithm to any fuzzing problem with minimal slowdown.
