Analog Circuit Design with Dyna-Style Reinforcement Learning
  In this work, we present a learning based approach to analog circuit design,
where the goal is to optimize circuit performance subject to certain design
constraints. One of the aspects that makes this problem challenging to
optimize, is that measuring the performance of candidate configurations with
simulation can be computationally expensive, particularly in the post-layout
design. Additionally, the large number of design constraints and the
interaction between the relevant quantities makes the problem complex.
Therefore, to better facilitate supporting the human designers, it is desirable
to gain knowledge about the whole space of feasible solutions. In order to
tackle these challenges, we take inspiration from model-based reinforcement
learning and propose a method with two key properties. First, it learns a
reward model, i.e., surrogate model of the performance approximated by neural
networks, to reduce the required number of simulation. Second, it uses a
stochastic policy generator to explore the diverse solution space satisfying
constraints. Together we combine these in a Dyna-style optimization framework,
which we call DynaOpt, and empirically evaluate the performance on a circuit
benchmark of a two-stage operational amplifier. The results show that, compared
to the model-free method applied with 20,000 circuit simulations to train the
policy, DynaOpt achieves even much better performance by learning from scratch
with only 500 simulations.
