Decoupling Gating from Linearity
  ReLU neural-networks have been in the focus of many recent theoretical works,
trying to explain their empirical success. Nonetheless, there is still a gap
between current theoretical results and empirical observations, even in the
case of shallow (one hidden-layer) networks. For example, in the task of
memorizing a random sample of size $m$ and dimension $d$, the best theoretical
result requires the size of the network to be $\tilde{\Omega}(\frac{m^2}{d})$,
while empirically a network of size slightly larger than $\frac{m}{d}$ is
sufficient. To bridge this gap, we turn to study a simplified model for ReLU
networks. We observe that a ReLU neuron is a product of a linear function with
a gate (the latter determines whether the neuron is active or not), where both
share a jointly trained weight vector. In this spirit, we introduce the Gated
Linear Unit (GaLU), which simply decouples the linearity from the gating by
assigning different vectors for each role. We show that GaLU networks allow us
to get optimization and generalization results that are much stronger than
those available for ReLU networks. Specifically, we show a memorization result
for networks of size $\tilde{\Omega}(\frac{m}{d})$, and improved generalization
bounds. Finally, we show that in some scenarios, GaLU networks behave similarly
to ReLU networks, hence proving to be a good choice of a simplified model.
