Accurate and Efficient Hyperbolic Tangent Activation Function on FPGA
  using the DCT Interpolation Filter
  Implementing an accurate and fast activation function with low cost is a
crucial aspect to the implementation of Deep Neural Networks (DNNs) on FPGAs.
We propose a high-accuracy approximation approach for the hyperbolic tangent
activation function of artificial neurons in DNNs. It is based on the Discrete
Cosine Transform Interpolation Filter (DCTIF). The proposed architecture
combines simple arithmetic operations on stored samples of the hyperbolic
tangent function and on input data. The proposed DCTIF implementation achieves
two orders of magnitude greater precision than previous work while using the
same or fewer computational resources. Various combinations of DCTIF parameters
can be chosen to tradeoff the accuracy and complexity of the hyperbolic tangent
function. In one case, the proposed architecture approximates the hyperbolic
tangent activation function with 10E-5 maximum error while requiring only 1.52
Kbits memory and 57 LUTs of a Virtex-7 FPGA. We also discuss how the activation
function accuracy affects the performance of DNNs in terms of their training
and testing accuracies. We show that a high accuracy approximation can be
necessary in order to maintain the same DNN training and testing performances
realized by the exact function.
