Bayesian Perceptron: Towards fully Bayesian Neural Networks
  Artificial neural networks (NNs) have become the de facto standard in machine
learning. They allow learning highly nonlinear transformations in a plethora of
applications. However, NNs usually only provide point estimates without
systematically quantifying corresponding uncertainties. In this paper a novel
approach towards fully Bayesian NNs is proposed, where training and predictions
of a perceptron are performed within the Bayesian inference framework in
closed-form. The weights and the predictions of the perceptron are considered
Gaussian random variables. Analytical expressions for predicting the
perceptron's output and for learning the weights are provided for commonly used
activation functions like sigmoid or ReLU. This approach requires no
computationally expensive gradient calculations and further allows sequential
learning.
