Adaptive n-ary Activation Functions for Probabilistic Boolean Logic
  Balancing model complexity against the information contained in observed data
is the central challenge to learning. In order for complexity-efficient models
to exist and be discoverable in high dimensions, we require a computational
framework that relates a credible notion of complexity to simple parameter
representations. Further, this framework must allow excess complexity to be
gradually removed via gradient-based optimization. Our n-ary, or n-argument,
activation functions fill this gap by approximating belief functions
(probabilistic Boolean logic) using logit representations of probability. Just
as Boolean logic determines the truth of a consequent claim from relationships
among a set of antecedent propositions, probabilistic formulations generalize
predictions when antecedents, truth tables, and consequents all retain
uncertainty. Our activation functions demonstrate the ability to learn
arbitrary logic, such as the binary exclusive disjunction (p xor q) and ternary
conditioned disjunction ( c ? p : q ), in a single layer using an activation
function of matching or greater arity. Further, we represent belief tables
using a basis that directly associates the number of nonzero parameters to the
effective arity of the belief function, thus capturing a concrete relationship
between logical complexity and efficient parameter representations. This opens
optimization approaches to reduce logical complexity by inducing parameter
sparsity.
