Homogeneous Linear Inequality Constraints for Neural Network Activations
  We propose a method to impose homogeneous linear inequality constraints of
the form $Ax\leq 0$ on neural network activations. The proposed method allows a
data-driven training approach to be combined with modeling prior knowledge
about the task. One way to achieve this task is by means of a projection step
at test time after unconstrained training. However, this is an expensive
operation. By directly incorporating the constraints into the architecture, we
can significantly speed-up inference at test time; for instance, our
experiments show a speed-up of up to two orders of magnitude over a projection
method. Our algorithm computes a suitable parameterization of the feasible set
at initialization and uses standard variants of stochastic gradient descent to
find solutions to the constrained network. Thus, the modeling constraints are
always satisfied during training. Crucially, our approach avoids to solve an
optimization problem at each training step or to manually trade-off data and
constraint fidelity with additional hyperparameters. We consider constrained
generative modeling as an important application domain and experimentally
demonstrate the proposed method by constraining a variational autoencoder.
