Two-argument activation functions learn soft XOR operations like
  cortical neurons
  Neurons in the brain are complex machines with distinct functional
compartments that interact nonlinearly. In contrast, neurons in artificial
neural networks abstract away this complexity, typically down to a scalar
activation function of a weighted sum of inputs. Here we emulate more
biologically realistic neurons by learning canonical activation functions with
two input arguments, analogous to basal and apical dendrites. We use a
network-in-network architecture where each neuron is modeled as a multilayer
perceptron with two inputs and a single output. This inner perceptron is shared
by all units in the outer network. Remarkably, the resultant nonlinearities
often produce soft XOR functions, consistent with recent experimental
observations about interactions between inputs in human cortical neurons. When
hyperparameters are optimized, networks with these nonlinearities learn faster
and perform better than conventional ReLU nonlinearities with matched parameter
counts, and they are more robust to natural and adversarial perturbations.
