A Universal Approximation Result for Difference of log-sum-exp Neural
  Networks
  We show that a neural network whose output is obtained as the difference of
the outputs of two feedforward networks with exponential activation function in
the hidden layer and logarithmic activation function in the output node (LSE
networks) is a smooth universal approximator of continuous functions over
convex, compact sets. By using a logarithmic transform, this class of networks
maps to a family of subtraction-free ratios of generalized posynomials, which
we also show to be universal approximators of positive functions over
log-convex, compact subsets of the positive orthant. The main advantage of
Difference-LSE networks with respect to classical feedforward neural networks
is that, after a standard training phase, they provide surrogate models for
design that possess a specific difference-of-convex-functions form, which makes
them optimizable via relatively efficient numerical methods. In particular, by
adapting an existing difference-of-convex algorithm to these models, we obtain
an algorithm for performing effective optimization-based design. We illustrate
the proposed approach by applying it to data-driven design of a diet for a
patient with type-2 diabetes.
