Trajectory Balance: Improved Credit Assignment in GFlowNets
  Generative Flow Networks (GFlowNets) are a method for learning a stochastic
policy for generating compositional objects, such as graphs or strings, from a
given unnormalized density by sequences of actions, where many possible action
sequences may lead to the same object. Prior temporal difference-like learning
objectives for training GFlowNets, such as flow matching and detailed balance,
are prone to inefficient credit propagation across action sequences,
particularly in the case of long sequences. We propose a new learning objective
for GFlowNets, trajectory balance, as a more efficient alternative to
previously used objectives. We prove that any global minimizer of the
trajectory balance objective can define a policy that samples exactly from the
target distribution. In experiments on four distinct domains, we empirically
demonstrate the benefits of the trajectory balance objective for GFlowNet
convergence, diversity of generated samples, and robustness to long action
sequences and large action spaces.
