{'fraction_non_alphanumeric': 0.028389339513325607, 'fraction_numerical': 0.008111239860950173, 'mean_word_length': 5.22042042042042, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Research and design of activation function hardware implementation methods

Activation function is an important part of neural network. Many artificial intelligence accelerators specially design hardware component supporting for activation function. In this article, we analyze normal activation functions, study kinds of activation function hardware implementation methods, and propose a parallel look-up table based piecewise linear fitting method. In many-core AI processor, we determine the table precision by memory size, and propose the basic algorithm and detailed structural design scheme. After the design is completed, we verify its correctness and evaluate its performance. The result shows that proposed method meets the need of correctness, and can finish the parallel computation of sixteen activation function fitting results within thirteen cycles. It greatly improves the efficiency of activation function hardware design.


INTRODUCTION
Recently, Artificial Intelligence technology develops rapidly, Deep Neural Network is going to be stateof-the-art in many applications such as computer vision [1], nature language processing [2]. Nowadays main stream neural network includes Convolutional Neural Network [1], Recurrent Neural Network [3], Long Short Term Memory [4], Generative Adversarial Nets [5]. As a nonlinear function, activation function plays a very important role in lots of neural networks. Activation function which commonly used in neural network includes Sigmoid, Tanh, Softmax, Relu. No matter training or inference, we have to use activation function in forward and backward propagation. Activation function is of great significance to the convergence of neural networks.
Activation function is nonlinear, making it hard to compute using the floating point multiply-add component. We need an efficient hardware implementation method.
Currently mainstream artificial intelligence chips specially design hardware component supporting for activation function, such as TPU [6], DianNao [7]. The overall architecture of DianNao is shown in Figure 1. In DianNao, the layers in CNN are divided into three levels. The first level is Multiply. The second level is Add/Max. The third level is Transfer Function (i.e., nonlinear function like Sigmoid). Convolutional layers and fully connected layers are composed of three levels, and pooling layers are composed of the first two levels. In the Transfer Function level, DianNao uses look-up table based linear fitting method to achieve hardware supporting for activation function. Many-core AI chip is composed of many-core processor architecture, including Management Processing Element and Computing Processing Element. Computing Processing Element contains general-purpose computing component and AI acceleration core. Convolution is finished in AI acceleration core, and the results are sent to general-purpose computing component to finish activation function computation. The activation function component is in general-purpose computing component, realized by parallel look-up table based piecewise linear fitting method.
In this article, we analyze normal activation functions, study kinds of activation function hardware implementation methods. In many-core AI processor, we propose a parallel look-up table based piecewise linear fitting method, design the detailed architecture, and do some experimental verification. The result shows this architecture can accomplish activation function.
The hardware implementation of Relu function is simple, and Softmax function is composed of exponential function and logarithmic function. So, in this article, we only consider Sigmoid, Tanh, Log(x) and Exp(x).
In the process of neural network training, we have to use activation function and derivation of activation function in forward and backward propagation. The hardware implementation of activation function includes look-up table method, Taylor expansion method, piecewise linear function method [9], etc.
Look-up table method is most intuitive and universal. Within the allowable range of error, it stores sampling value of nonlinear function. For any input number, this method directly accesses corresponding result. Look-up table method match input and output, getting the result in one memory access, but it costs too much memory.
Taylor expansion method uses Taylor expansion formula to simulate nonlinear function. Normal activation function is infinitely derivative, and Taylor expansion formula can approach the real value. But the computing units are finite, and high order operations are too complex. Taylor expansion method is rarely used. Its expression is in (5).
Piecewise linear fitting method divides the nonlinear function, and uses multiple sections of linear function to fit the nonlinear function within the allowable range of error. The simplest piecewise linear fitting method is equal division. It divides the independent variable interval equally, and stores the slope and bias value per section. We access the corresponding slope and bias value by independent variable, and use floating point operation to get the fitting result. Its expression is in (6).

MANY-CORE AI CHIP
Many-core AI chip is composed of many-core processor architecture, including Management Processing Element and Computing Processing Element. Computing Processing Element contains general-purpose computing component, AI acceleration core and Local Data Memory. The architecture of Computing Processing Element is shown in Figure 2. Proposed hardware implementation of activation function is completed in general-purpose computing component.

Basic algorithm and overall architecture
Nonlinear function such as Sigmoid and Tanh is hard to directly solve, so piecewise linear function fitting method can reduce implementation cost. Linear fitting coefficients store in the table, and we can transfer the input single precision 32-bit number to get the signed offset address in the table.
The basic algorithm of look-up table based piecewise linear fitting method is shown as follow. Firstly, according to the exponent of input data, we align the 24-bit mantissa. Then we cut out the result based on instruction flag, combining the sign of input number, and get 8-bit signed number.

Alignment Module
Alignment Module aligns the mantissa part of input data according to the exponent part of input data, getting 14-bit immediate result (7-bit integer and 7-bit fraction). The logic diagram of Alignment Module is shown in Figure 4.
Firstly, we transpose mantissa using the sign of input data, getting 24-bit number which need to be aligned. When the sign is zero, 24-bit number combines 1-bit 1 and 23-bit mantissa; When the sign is one, 24-bit number combines 1-bit 0 and 23-bit inverted mantissa.  Then, we align 24-bit number using the exponent of input data. After alignment, data need to be truncated, so the maximum align number is related to truncation. According to 7-bit integer, 7-bit fraction and position of decimal point, the maximum left align number is 8, and the maximum right align number is 7. The

Post-process Module
Post-process Module calculates 14-bit table biased address using 14-bit immediate result and instruction flag. The logic diagram of Post-process Module is shown in Figure 5. We split instruction flag, high three bits i_type [5:3] representing integer, low three bits i_type [2:0] representing fraction. Then we decode split instruction flag using decoder and get 14-bit data. 14-bit data "AND" with 14-bit immediate result, and align using fraction flag, truncating 7-bit data, combining inverted input data sign ~i_src[31] and 6 zeros, getting 14-bit table biased address. Normal look-up table operations output fixed value.

Exception Detection Module
Exception Detection Module makes exception judgment about instruction exception flag and input data exceeding linear section exception flag, these two exception are only effective during expanded parallel look-up table operations. The logic diagram of Exception Detection Module is shown in Figure 6. Instruction exception flag. High three bits of instruction flag represent integer and low three bits of instruction flag represent fraction. If these two number are both zero or sum of them is less than or equal to 8, instruction has no exception. Or else instruction exception flag is 1.
Input data exceeding linear section exception flag. Input data exceeding linear section means input single precision data exceeds hardware representable largest linear section range, then input data exceeding linear section exception flag is 1. There are some conditions. Condition 1 is left alignment.

Experimental Verification
We implement parallel look-up table based activation function hardware in many-core AI processor, and verify its correctness and evaluate its performance.
After finishing the design of architecture and the code, we verify the correctness of this component using PSL language, including module level test, function points test and random number test. The results show that this component meets the need of correctness.
Parallel look-up table based piecewise linear fitting method implements activation function. Look-up table instruction takes 3 cycles, 1cycle for address calculation, 1cycle for memory access, 1 cycle for table look-up. Combing 6 cycles for two look-up table instructions and 7 cycles for floating point multiply-add instruction, 16 activation function fitting results are finished within 13 cycles totally. Comparing to 3 cycles for direct look-up table method, our method is far more efficient.

SUMMARY
In this article, we propose a parallel look-up table piecewise based linear fitting method for activation function hardware implementation, and use it in many-core AI processor.
We analyze normal activation functions and kinds of activation function hardware implementation methods, and propose our method. Then we determine the table precision by many-core AI processor memory size, and propose the basic algorithm and detailed structural design scheme. Finally, we do some experimental verification.
The results show that proposed method meets the need of correctness, and can finish the parallel computation of 16 activation function fitting results within 13 cycles, greatly improving the efficiency of activation function hardware design.