{'fraction_non_alphanumeric': 0.04443534350363469, 'fraction_numerical': 0.013694071874680046, 'mean_word_length': 4.614975567691865, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores

Accelerating the neural network inference by FPGA has emerged as a popular option, since the reconfigurability and high performance computing capability of FPGA intrinsically satisfies the computation demand of the fast-evolving neural algorithms. However, the popular neural accelerators on FPGA (e.g., Xilinx DPU) mainly utilize the DSP resources for constructing their processing units, while the rich LUT resources are not well exploited. Via the software-hardware co-design approach, in this work, we develop an FPGA-based heterogeneous computing system for neural network acceleration. From the hardware perspective, the proposed accelerator consists of DSP- and LUT-based GEneral Matrix-Multiplication (GEMM) computing cores, which forms the entire computing system in a heterogeneous fashion. The DSP- and LUT-based GEMM cores are computed w.r.t a unified Instruction Set Architecture (ISA) and unified buffers. Along the data flow of the neural network inference path, the computation of the convolution/fully-connected layer is split into two portions, handled by the DSP- and LUT-based GEMM cores asynchronously. From the software perspective, we mathematically and systematically model the latency and resource utilization of the proposed heterogeneous accelerator, regarding varying system design configurations. Through leveraging the reinforcement learning technique, we construct a framework to achieve end-to-end selection and optimization of the design specification of target heterogeneous accelerator, including workload split strategy, mixed-precision quantization scheme, and resource allocation of DSP- and LUT-core. In virtue of the proposed design framework and heterogeneous computing system, our design outperforms the state-of-the-art Mix&Match design with latency reduced by 1.12-1.32x with higher inference accuracy. The N3H-core is open-sourced at: https://github.com/elliothe/N3H_Core.


INTRODUCTION
In the past decade, Deep Neural Networks (DNN) has succeeded in various computer vision and natural language processing tasks [14]. Accelerating the DNN with domain-specific accelerator [4,5,15] has been widely explored to counter the excellent computing power demanded by the mammoth DNN model that is growing exponentially [1,12]. Compared to other off-the-shelf hardware accelerators, FPGA can quickly adapt to the fast-evolving algorithms with outstanding performance (e.g., latency, throughput, etc.), owing to its reconfigurability. It is known that the dominant operation in the DNN inference is the Multiplication and Accumulation (MAC) [4]. As the Digital Signal Processing (DSP) units in the FPGA can perform MAC efficiently, prior FPGA-based DNN accelerator designs heavily use the DSP resources to construct the processing elements (aka. DSP-core) as DNN inference engine. In contrast, as another significant on-chip resource of FPGA, Look-Up Table (LUT) is merely utilized to build the peripheral computing units that perform the computations of batch-normalization, activation, etc.
Thanks to recent advances in DNN compression algorithms [6,11,18,27], parameters of DNN can be converted from 32-bit floating point to extremely low bit-width (e.g., < 4-bits) with negligible inference accuracy degradation, but significantly simplify the computation complexity and mitigate the on-/off-chip data access bottleneck (aka. "memory wall") [32,36]. Moreover, researchers are aware that the layers of DNN own different sensitivity to the quantization noise resulted from different bit-width. Thus, the DNN quantization with varying bit-width emerges as an alleged compression scheme to minimize the overall model size. Note that, DSP-Core cannot efficiently support MAC operations with varying bit-width, due to the overhead caused by flexibility. Although a recent design proposed by Yaman et. al. called BISMO [29] can efficiently support mixed-precision MACs via a bit-serial computing method, the inference latency of BISMO is still much higher than the pure DSP-based bit-parallel counterpart, to achieve the identical accuracy for low bit-width quantization. For example, our preliminary experiment of 8-bit quantized ResNet-18 on ImageNet dataset [8,16] shows the inference latency of DSP-based and LUTbased (i.e., BISMO-based) design are 182ms and 309ms respectively. Thus, with the available DSP/LUT resources in the pool, an FPGA system that supports fully-flexible bit-width and outperforms other DSP-centric designs [15] is desired.
Furthermore, to maximize the performance of FPGA-based DNN accelerator, prior works [30,37] have conducted the design space exploration. These studies either use the roof-line model to limit the design space or build the latency/performance model w.r.t the workload and architecture design parameters to explore the design space. While the former one cannot reflect the design parameters of the architecture, the latter does not impose constraints on parameters sufficiently, resulting in a enormous design space.
In summary, the preliminary architecture design of the FPGAbased DNN accelerator mainly counters the following drawbacks: 1) Rich on-chip LUT resource are not well utilized; 2) A highperformance architecture supporting mixed-precision operation is absent; 3) Lacking the systematic design methodology for the single-chip heterogeneous system. As the countermeasure, in this work, we propose a heterogeneous computing architecture, which fully utilizes the on-chip resources (including LUT, DSP, BRAM, etc.) for computing instead of control. Our proposed architecture consists of LUT-core and DSP-core that work jointly as a highperformance computing system. Such two computing cores handle computation with operands in flexible and fixed bit-width, respectively. To ease the manpower to optimize the complicated system configuration on large design space, we develop the end-to-end optimization framework as a systematic design methodology for the proposed single-chip heterogeneous system. Our contributions can be summarized as: • To fully exploit the on-chip resource of the target FPGA, we propose a heterogeneous DNN accelerator architecture called N 3 H-Core that consists of DSP-and LUT-centric computing units (aka. DSP-core and LUT-core, respectively). • For maximum throughput and minimum latency, We design the DSP-and LUT-core to operate w.r.t the predefined instructions in the intra-layer asynchronous fashion. • We construct scalable and adaptable cost model across different DNNs and FPGA, to precisely estimate the resource utilization, inference latency, and other metrics-of-interests. • Through the cost model of the N 3 H-Core, we apply the Reinforcement Learning (RL) technique to build the end-toend optimization framework that automatically generate the architecture configurations (resource allocation), data flow (layer-wise workload split ratio), and DNN (quantization bit-width of weights and activations) respectively. Thus, given the target DNN and FPGA platform, an agile and optimal design can be rapidly achieved.

PRELIMINARY AND RELATED WORKS 2.1 DNN Inference Acceleration on FPGA
Recently, many studies [15,28,31] have been conducted to explore using FPGA in DNN acceleration for flexibility and high computation performance. Those designs can be generally categorized into bit-parallel and bit-serial counterparts, specified as follow.  [31] proposes a latency-centric optimizer to explore design space efficiently.
[28] presents a systematic design space exploration methodology to maximize the throughput of an OpenCL-based FPGA accelerator, taking source available on-chip as constraints. [3] proposes a heterogeneous architecture that utilizes DSP and LUT resources. The logarithmic quantization is used to replace the multiplication with bit-shift. However, the architecture of [3] is not bit-width flexible and without automated architecture optimization.
2.1.2 Bit-serial Acceleration. In bit-parallel scheme, accelerator is designed to support highest operand bit-width, even when some operands are in low bit-width, thus lead to great hardware overhead.
To mitigate that, bit-serial scheme computes one binary multiplication per cycle which can compose operations on higher bit-width operands. Given integer matrix and , their multiplication is: where × is decomposed into weighted sum of the binary matrix multiplication. Such computation transform is hardware-friendly and can be efficiently implemented with LUT on FPGA, which is adopted in BISMO design [29]. Based on Eq. (1), BISMO propose a GEMM core (depicted in Figure 1 composed of a M-by-N DPU array, where each DPU unit performs the binary vector dot-product via the XNOR and pop-count. According to the bit-serial computing fashion, the computing latency of BISMO is proportional to the operand bit-width of and . Thus, such computing architecture is prone to compute with the low bit-width operands.

Model Quantization
DNN model quantization has emerged as a mandatory technique for high-performance DNN inference. Thanks to the advances in model quantization algorithm [6,7,18,19,21,38], the activations and weights in 32-bit floating-point (fp32) can be quantized into extreme low bit-width with negligible inference accuracy degradation, using uniform [20] or non-uniform quantizer [23] in quantizationaware training [18,19] or post-training quantization [21]. In this work, we focus on the model quantization using bits -bit uniform quantizer, and its quantization function can be expressed as: where is the step size.ˆ= −2 bits −1 andˆ= 2 bits −1 − 1 are the upper and lower bound for the bits integer representation. clip(·,ˆ,ˆ) = min(max(·,ˆ),ˆ) is the clipping function, and round() denotes the round function. With the uniform quantizer adopted, two quantization schemes (uniform precision and mixed precision) are usually applied upon the entire DNN mode. Uniform Precision. Uniform precision quantization assigns identical bit-width to each layer of target DNN. Researchers [7,13,38] have shown that INT8, even INT4, can retain accuracy on many neural networks. For the general-purpose computing hardware (e.g., CPU or GPU), due to the hardware constraint, only INT4 and INT8 are supported. Thus, uniform precision quantization is a relatively general method, as most hardware platforms only support computation (e.g., MAC) in specific bit-width.
Mixed Precision. As the prior investigations [2,33,34] reveal the fact that layers of DNN own different redundancy, the quantization with varying bit-width on different layers emerges as a potential scheme to minimize the model size of target DNN. [39] determines the layer-wise bit-width based on the entropy of weights and activations. [9] utilizes the second-order information, i.e., Hessian matrix, to determine the bit-width, where the block with higher top Hessian eigenvalue will be assigned with higher bit-width. Elthakeb et al. [10] use the sample efficiency of Proximal Policy Optimization (PPO) to find the optimal bit-width, which is the first to use RL for quantization. Wang et al. [33] predict the layer-wise bit-width via Deep Deterministic Policy Gradient (DDPG) algorithm with the target hardware taken into consideration.

ARCHITECTURE OF N H-CORE
In this work, we propose a heterogeneous neural network accelerator called N 3 H-Core, which fully exploits both the DSP and LUT resources in a heterogeneous fashion. Its architecture, data flow and cost and latency models are specified as follows.

Architecture Overview of N 3 H-Core
As depicted in Fig. 2, N 3 H-Core architecture consists of 1 instruction generator, heterogeneous computing cores (i.e., 2 LUT-core and 3 DSP-core), 4 address generator, 5 peripheral buffers. 1 Instruction Generator. N 3 H-Core is an intra-layer asynchronous and inter-layer synchronous instruction-based neural network accelerator. Thus, the instructions adopted can be generally separated into two categories: operation instruction and synchronization instruction, which jointly work for accurate pipelining computation within LUT-and DSP-core. Within each layer, Operation Instructions are generated asynchronously and specifically for the LUT-Core and DSP-Core, consisting of three types {Fetch, Execute, Result} that corresponds to three operating phases of LUT-and DSP-core, respectively. 1) In Fetch phase, Direct-Memory Access (DMA) module loads weight and activation data from the off-chip DDR to on-chip buffers; 2) In Execute phase, data is transmitted from buffers to computing cores for multiplication and accumulation; 3) In Result phase, the calculation results from the LUT-and DSP-core are written back to the shared result buffer. Thanks to the synchronous instructions (Sync), two cores are synchronized after they complete the assigned workloads, then launch the computation of successive layer without conflicts.
Based on prior designs [22,29], we propose a new schedule for matrix multiplication and design Fetch, Execute and Result engines to run the corresponding instructions, as described in Fig. 3. Referring to Eq. (1), we assume the activation buffers possess the capacity of the activation matrix , and weight buffer is the half capacity of weight matrix . To begin with the first part 1 of and , R0 is fetched by fetch engine, then 0 is fetched as well. Meanwhile, execute and fetch engines are in waiting status, i.e., Wait-Fetch (WF) and Wait-Execute (WE). When fetch instructions is completed, the fetch engine sends a synchronization signal to activate the execute engine, i.e., Sync-Execute (SE). After the execute engine receives the signal, it immediately transfer from WF status to run the execute instruction of L0×R0. While the L0×R0 is executing, the fetch engine start to load following binary matrix L1 for pipeling. After fetching the first tile of weight data R0 and R1, the weight buffer is filled, so the fetch engine is in WE status, waiting for a synchronization signal from execute engine. After the signal is received, the fetch engine continues to fetch the next tile of weight data R2 and R3. Once the R2 is fetched, the execute engine multiplies the pre-loaded binary matrices of in a sequence. When all the multiplications are done, the result engine receives the synchronization signal (Sync-Execute, SE) from the execute engine, then shift from Wait-Execute (WE) status to run Result instruction that store the computation result back to DDR. In this way, the design bridges the gap between data movement and processing.
For instruction compatibility for DSP-and LUT-core, all instructions are 128-bits composing different segments for information representation. For Fetch and Result instructions, they consist of base address (16-bits), stage control (3-bits), read /write range (1-bit) (a) Instruction synchronization via queue. of on-chip buffer and base address (32-bits), offset (24-bits), and write/read range (16-bits) of the DDR. For Execute instruction, it includes the on-chip buffer data address, and the GEMM-Core parameters listed in Table 1. For sync instruction, it includes the current (1-bit) and next state (2-bits) of each Engine and the flag (3bits) indicating the sent of synchronization token. Heterogeneous Computing Core. To fully utilize the on-chip resource of a target FPGA for boosted computing power, we develop a heterogeneous computing core that consists of 2 LUT-core and 3 DSP-core. Given a computing task (e.g., inference with one layer of DNN), the computation workload is split w.r.t a designated ratio, then performed by the LUT-and DSP-core asynchronously. Note that, the optimization of workload split ratio will be discussed in Section 5.3. For the LUT-core, we adopt the open-sourced BISMO 2 [29] as its backbone, which is a GEMM kernel with run-time adjustable operand bit-width in virtue of the bit-serial computing mechanism. In contrast to the LUT-core operating in the bit-serial fashion, the DSP-core is developed for bit-parallel computing. Overall, the heterogeneous computing core incorporates prior-neglected rich LUT resources and introduces a more flexible weight-activation quantization scheme for accuracy lossless mapping. 5 Peripheral Buffer. To avoid the frequent off-chip memory access that hampers the overall inference latency and improve the throughput via pipelining, the proposed architecture allocates each computing core with input/weight/result buffers. Weight and activation data are fetched via Direct Memory Access (DMA) to the dedicated buffers, then transferred to the computing cores. Once the computation is done, the result stored in the result buffer is then written to DDR as the activation of the next layer.
Memory Control. Two workload partitions allocated to two computing cores are stored at different regions on DDR to make the

Heterogeneous Computing Core
3.2.1 LUT-Core. Through the conversion function (i.e., im2col), the computation of various parametric layers (e.g., convolution layer, fully-connected layers) can all be computed via the matrix multiplication (GEMM). Fig. 1 illustrates the architecture of LUT-Core. In LUT-Core, the buffers allocated to the row are designated as the activation buffer (indexed from 0 to −1 in Fig. 1), the activation buffers have the depth of L,buf , whose interface width is . Similarly, the buffer allocated to the column is weight buffers indexed from 0 to −1, and the depth is L,buf , whose interface width is as well. Each DPU reads -bits from the activation buffers and -bits from weight buffers simultaneously, then store the binary dot multiplication results as partial sum. The LUT-Core can process × × bits in parallel. As listed in Table 1, { , , , L,buf , L,buf } are five important configuration parameters significantly affect the latency, which will be explored at design time.
3.2.2 DSP-Core. The architecture detail of DSP array is illustrated in Fig. 4. Note that, the DSP core performs the GEMM in tiling fashion as well. In Execute phase, a tile of input activation data and weight data are read from buffers into register arrays. The register array of activation data has a reg,row rows and a reg,col columns. Correspondingly, the register array of weight data has a reg,col rows 3 and w reg,col columns. For activation buffer, the size of each row of activation register array is same as the activation buffer. Thus, the total bit-width of each buffer is a reg,col × a , where a is the activation bit-width. During the computing, each row of the activation array is filled with an activation buffer per cycle. For weight buffer, the bit-width of each buffer is a reg,col × w-D , and each two columns of the weight register array is filled with a weight buffer, thus it requires two cycles to fill the weight register array with weight buffer.

Cost and Latency Model
To minimize the latency by finding the optimal architecture configuration, we build the cost and latency models here for rapid design space exploration. The cost model simulates the resource utilization of LUT, BRAM, and DSP on FPGA, while the architecture configuration in Table 1 is constrained by available on-chip resource. Besides, the latency model describes the end-to-end inference latency.
DSP-Core Cost Model. As described in Section 3.2.2, the number of activation buffer equals to the a reg,row , and each buffer has the depth of a buf with the interface of a reg, col × a . Since our DSPcore is designed for 4-bit multiplication, the quantized activation bit-width ranges from 2 to 4 bits. When the activation bit-width a is less than 4, we pad zeros to make it 4-bits for storing in buffers. The BRAMs forming buffers are 1024 in-depth and 36-bits wide, and we use the 32-bits for convenience. Thus, the BRAM size consumed by DSP-Core is formulated as: where ⌈ a reg,col × 4/32⌉ · ⌈ a D,buf /1024⌉ is the BRAM size that one activation buffer consumes, and a reg,row is the size of activation buffers. Likewise, since we allocate data in one buffer to two columns of the weight register array, the amount of weight buffers is w reg,col /2, and ⌈ a reg,col × 4/32⌉ · ⌈ w D,buf /1024⌉ is the BRAM size that one weight buffer consumed. For LUT usage of DSP-Core, it is mainly consumed by the instruction generator and the control module, which is about 1000 as a constant (LUT D-core = 1000). The utilization rate of DSP resources is predefined as 100% at design time. The DSP-Core utilizes all the DSP resource available on chip, which means DSP D-core = DSP Available .
LUT-Core Cost Model. Since we adopt the BISMO [29] as the backbone of LUT-core, we inherits its cost model as: LUT L-core ( , , ) = · · ( + + ) + (4) where { , , , } = {1.17, 120.1, 44.1, 718} are the fitting coefficient. From Eq. (4), the LUT utilization is proportional to , , and . We use 1024 in-depth and 36-bits wide BRAMs to build the buffer, and we only use 32-bits of the original interface. As mentioned before, there are activation buffers and each of them is comprised of ⌈ /32⌉ · ⌈ L,buf /1024⌉ BRAMs. Similarly, the BRAMs consumed by weight buffer is calculated by Eq. (5). DSP-Core Latency Model. As the DSP-core is an instructionoriented computing unit whose instruction pipeline is organized in a specific pattern, we build its latency model w.r.t its instruction pipeline. We focus on the execute engine and count the latency as following queues: wait , run , rst and sig . These four parts of latency is relevant to the parameters { a reg,row , a reg,col , w reg,col , a D,buf , a D,buf } in Table 1. Then, we trace the instructions streams and accumulate the latency of each phase on execute engine. Note that, wait is the latency of waiting for fetching data (i.e., phases denoted by 0 in Fig. 3), run is the latency of execution (phases denoted by 1), sig is the latency of sending signal (indicated by 2), and rst is the latency of result phase. By accumulating each latency, the computational latency of one layer is formulated as: To fully utilize the DSP resource on-chip, we adopt a simple simulation method that a reg,col , w reg,col are fixed to 16, and only a reg,row can be modified that determines the latency performance. Therefore, it ensures that DSPs are fully-utilized by modifying only one parameter, and a simple core design method is provided at the same time. Then, the calculation of Eq. (6) can be simplified as: LUT-Core Latency Model. Likewise, the latency model of LUT-Core is built in the same manner via describing and simulating the instruction streams. However, the main difference is that LUT-Core computes the MACs in a bit-serial fashion, whose latency is determined by the bit-width of activation ( a ) and weight ( w-L ) matrices. Thus, the parameters that relevant to the LUT-Core latency is { a bits , w L,bits , , , , a L,buf , w L,buf }, and the latency model of LUT-Core is formulated as: Note that, the weight buffer reads one tile at once. Thus, the capacity of the weight buffer is the size of one tile weight data, while the depth change will not affect the latency. Correspondingly, we simplify the latency model of LUT-Core as: LUT = ( a bits , w L,bits , , , , a L,buf ) where such simplification also shrink the design space to be explored by reducing the variation of w L,buf .   As N 3 H-Core performs the DNN inference in a layer-wise synchronization manner, the total computation latency of target DNN with -layers can be expressed as: where the LUT and DSP denotes the computation latency taken by the LUT-core and DSP-core to complete the assigned workload for -th layer respectively. Since the instructions of our LUT-core and DSP-core are scheduled in static pipeline, the latency model formulated via accumulated execution time of schedule instruction is relatively accurate. To verify our latency model, we test 10000 design points with design knobs listed on Table 1 of original manuscript using randomly generated value, where the result is shown in Fig. 5 with prediction error less than 2%.

HYBRID QUANTIZATION WITH MIXED PRECISION
Suppose an -layer DNN as { , } =1 , ∈ R out × in × × and denotes the -th layer weight and input/activation tensor respectively, where is the kernel size 4 , in and out are number of input and output channels. The weight tensor can be viewed as a set , where ∈ R × × denotes the -th filter. Our work applies filter-wise quantization, which takes a finer-grained strategy with as its granularity. For different filters in one layer, 3-D tensor is assigned with different quantization bit-width. The hybrid quantization method integrates both uniform-and mixed-precision quantization in this work. The computation of one layer for DSP-and LUT-core is allocated in the granularity of filters. All the computations conducted by DSP-core are 4-bits quantized on weights. The rest of filters computed by LUT-core are quantized with w-L bits (2 ∼ 8-bits) which is determined by our optimization framework (Section 5). Note that, w-L varies in different layers. The filter allocation ratio will be discussed in Section 5.3, which determines the number of filters allocated to DSP-/LUT-core on each layer. Besides, we need to consider which filters should be allocated to DSP-/LUT-core. In this work, the filter allocation (to DSP/LUT) depends on the KL-divergence D KL of the original filter weight distribution and its quantized weight distribution (calibrated via one batch of images). The filters with greater D KL in one layer will be allocated to the computing core with higher bit-width, since the quantization bit-widths w-L of LUT filters is flexible. Now, the filter number and type allocation problems are settled.
We adopt the layer-wise quantization manner for activations shared by both LUT-and DSP-core. The activation and weight of first and last layer are quantized into 8-bits. In our work, we set the activation bit-widths range as 2-4 bits for all other layers, as the LUT-core enjoys much less latency with operands in low bit-width. Moreover, we linearly quantize the weights and activations of each layer using 4, w-L -bits and { a }-bits respectively. The linearly quantized model only requires fixed-point computing unit, which is more efficient in our heterogeneous architecture. The proposed quantization method of the th convolution layer is depicted in Fig. 6.

FRAMEWORK FOR OPTIMIZED SYSTEM 5.1 Framework Overview
This section describes our framework to automatically generate the optimized system configuration for N 3 H-Core to achieve the optimized performance across varying DNNs and FPGAs. The objects to be optimized by the framework come in threefold: 1) Resource Allocation. The framework configures the design parameters listed in Table 1 to achieve the optimized resource allocation for both LUTcore and DSP-core; 2) Workload Split. As the N 3 H-Core splits the computing task into LUT-and DSP-core in the layer-wise fashion, the framework is expected to split the computation task for workload balance properly; 3) Quantization bit-width. Since LUT-core supports operands with flexible bit-width while DSP-core only supports the fixed one (int4 in this work), the quantization bit-width affects the overall inference latency and accuracy.

Resource allocation & Quantization
As discussed in Section 3, we formulate the latency model for LUT-and DSP-core, and design factors influencing the latency and the corresponding ranges are tabulated in Table 2. Design factors , , , a L,buf , a D,buf , w D,buf are related to the resource allocation. The left design factors w-L , a =1 are the quantization bit-widths of LUT filters and activations of each DNN layer. We leverage the Reinforcement Learning (RL) with DDPG algorithm [25] for the design space exploration (can be viewed as sequential decision making) to identify the optimal system configuration with low latency but high accuracy. The change of design factors leads to different resource usage estimated by our cost model, which is constrained by the available resources on the target FPGA devices.

Neuron-based workload split
For each layer, the filter-based workload split ratio is defined as: ratio = Filter LUT Filter all (11) Where Filter all is the number of neurons/filters and Filter LUT is the number of filters computed by LUT-core. After the quantization bit-widths w-L , a of the -th layer are determined by the agent, the best workload split ratio ratio is computed by minimizing the latency of the -th layer, as defined: arg min ratio max LUT , , , a L,buf , w-L , a , ratio , The generated ratio is the optimal filter allocation strategy for inter-layer parallelism of N 3 H-Core. Fig. 7 shows an example of the relationship between the latency and ratio. From the figure, the optimized ratio is 0.75, which means 0.75 × 256 = 192 filters are computed by the LUT-core and the rest 256 − 192 = 64 filters are assigned to DSP-core.

RL implementation details
The overview of our DDPG based RL learning framework [17] to explore the both the architecture configuration and workload assignment between heterogeneous cores is depicted in Fig. 8. Details of action space, state space, reward and etc are given as follows.

Action
Space. The action space contains two tasks of actions: action-1 1 and action-2 2 in sequence. The action-1 decides the resource utilization design factors which is constrained by the available resources of given FPGA devices. Table 2 provides the range of resource utilization design factors on two FPGAs. In each episode, the RL agent takes 6+2N action steps, where N is the number of layers of given DNN. For action-1, the agent chooses the hardware-related factors , , , a L,buf , a D,buf , w D,buf sequentially at the first 6 time steps. For each time-step, the agent output one action applies on one aforementioned hardware factor, which can be described as: where ∈ [0, 5]. The continuous action (range: [0, 1]) of the DDPG agent are discretized by round functionround() into the range of 6 design factors in integer. Note that, , a L,buf , a D,buf , w D,buf are in the format of · where is a constant parameter and is a variable. For these 4 parameters, [value min , value max ] is the range of the parameter , so the actions 0 1 , 3 1 , 4 1 , 5 1 will be multiplied with their corresponding constant parameters which are shown in Table 2.
For action-2, the agent chooses the software-related factors { w-L , a } with 2N steps. The actions can be denoted as: where [ min , max ] is the bit-width range in Table 2.

State
Space. The state space for the -th time step is expressed as: where id func is the binary indicator for the current task (0 for 1 , 1 for 2 ) , is the layer index, NN is the DNN information of the -th layer, ratio is the calculated filter number allocation ratio in the -th layer, is the action of the current time step. , NN and ratio are set as 0 when id func =0. The DNN information of the -th layer NN is a group of parameters defined as: NN = in , out , kernel , stride , fmap , params , id sc&dw , id m/n (16) where in and out are #input/output channels, kernel , stride and fmap are the sizes of the kernel, stride and feature map. params denotes #parameters. id sc&dw is the binary index for shortcut layers of ResNet [16] and depthwise layers of MobileNet [26]. If shortcut or depthwise, it is set 1, otherwise 0. id m/n is the binary index for the type of the current action: 0 for quantization of LUT filters m-L , 1 for quantization of activations a . Thus, the entire state space can be unfolded as: = id func , , in , out , kernel , stride , fmap , params , In addition, we normalize each dimension of the state space into [0, 1] to make them in the same scale.

Reward.
After actions are taken in each episode, the model latency with current design configuration is estimated by our latency model, and the DNN is quantized w.r.t the identified quantization scheme. Then, we retrain the quantized model for one epoch to recover the accuracy, denoted as acc q . The reward function R is designed by combining the model latency m and accuracy acc q , which is formulated as: where t is the latency bound or target latency, acc b is the baseline accuracy of our model, = 0.01 is a scaling factor. Note that, the reward remains less than -1 when the model latency exceeds the latency bound, otherwise the reward is in the interval of (-1, 1).

EXPERIMENTS
In this section, we conduct experiments on various representative DNNs and FPGA devices to demonstrate the merits of hardware and software parts of N 3 H-Core.

Experimental Setup
Networks and Dataset. In this work, we uses two the classic image classification neural architectures, i.e., ResNet-18 [16] and MobileNet-V2 [26] with large-scale ImageNet dataset [8]. The data augmentation is identical as stated in [16]. The full-precision pretrained model utilized in the N 3 H-Core is obtained from the Pytorch model hub [24]. Hyper-parameters of software framework. The RL agent explores 900 episodes for each experiment setting. For each episode, if the model latency is within the latency bound, we will retrain the quantized model for one epoch to recover its accuracy for reward evaluation. We use SGD as optimizer with a fixed learning rate of 0.001 and momentum of 0.9. We randomly sample 100 categories of ImageNet to accelerate the exploration process. FPGA platforms. Two FPGA boards, i.e., XC7Z020 and XC7Z045, are chosen as the target hardware platforms. With the N 3 H-Core architecture deployed upon different FPGAs, the device-specific configuration scheme is given by the framework automatically, constrained by the available hardware resources (e.g., DSP, LUT and BRAM). The operating frequency is set to 100MHz.

Results and Analysis
We evaluate the framework under different target latency bounds, and generate 8 configuration combinations for the two DNNs and two FPGAs. The accuracy and latency of each configuration are tabulated in Table 5, where the configuration details are given in Table 3. For each configuration, we also provide the quantization bit-widths and the workload-split ratio of each layer in Figs. 9 to 12. Table 3, on D A we can find that is greater when the DNN is ResNet-18. denotes the computational parallelism of input channels of activation. A large allows more input channels to be computed in parallel, improving the computation efficiency. Compared with ResNet, MobileNet contains depth-wise layers which occupy one-third of the number of layers. Since depth-wise layers has a small number of input channels, the agent tends to assign a small to LUT-Core. It indicate that LUT-Core is more efficient in computing ResNet than MobileNet. Thus, for MobileNet, the agent allocates more BRAMs to DSP-Core, which allows DSP-Core compute faster to compensate for the inefficiency of LUT-Core. As shown in Fig. 11, for the same reason, the split ratios of many depthwise layers in MobileNet  are zero, which means that the workload of the depthwise layer is solely assigned to the DSP-Core for minimized latency.

Quantization
Analysis. In Fig. 9, it reveals that the RL agent assigns more bit-width in down-sampling layers (layer 8,13,18). To avoid the information bottleneck, higher bit-width are assigned to those layers (e.g., 7-bits for LUT-Core filters and 4-bits for shared activations). When the target latency bound is set to 30ms, the quantization bit-width becomes lower under the strict limitation. Consequently, the LUT-Core latency is reduced, and more workload is assigned to the LUT-Core to keep the latency below the bound. This proves that the agent can automatically analyze the latency and adjust the bit-width and ratio. As shown in Fig. 11, LUT-Core is not efficient to compute depth-wise layers, thus the workload split ratio of depth-wise layers is lower than point-wise layers. When the bound is set to 5ms, the bit-width across layers is further lowered compared to the 7ms counterpart. On D B , the configuration of quantization and ratio shows the same trend.

Comparison with the state-of-arts accelerators.
We selected the results of ResNet18 and MobileNet networks on two devices XC7Z020 and XC7Z045, as shown in Table 4, and compare the experimental results with the existing state-of-arts designs. When deploying ResNet18 on the given two FPGA devices, we obtain comparable accuracy while reducing latency by 1.32× and 1.24× compared to the Mix&Match design [3]. The Mix&Match is also a heterogeneous architecture utilizing DSP and LUT resources, but without exploring the design space, our latency-centric architecture outperforms it on resource utilization, quantization bit-width flexibility, and performance (e.g., latency, throughput and frame rate), even though they adopt more hardware-friendly power-of-2 [21] quantization scheme.  As can be found in Table 4, our proposed optimization framework makes better utilization of the on-chip resources, consuming more BRAMs and LUTs to boost the performance. In terms of the throughput per resource unit, the GOPS/DSP is higher on both devices. The GOPS/kLUT on XC7Z020 is less, but when more batches are computed simultaneously on XC7Z045, it outperforms the design in [3]. On MobileNet, we get the speedup by about 1.12× on both FPGA devices. The lower performance improvement on Mo-bileNet over ResNet is due to the low computation efficiency of LUT-Core on MobileNet. This is also reflected in the efficiency of hardware resource usage. Despite the performance improvement, the throughput per resource unit on both devices is lower than Mix&Match [3]. Therefore, the proposed heterogeneous architecture has a limited effect on depth-wise acceleration. Moreover, in terms of the frame-rate (FPS), our design outperforms the mix&match, and will outperforms design in [35] (2.48 ∼ 2.89× FPS) as well when it operates at 100MHz.

CONCLUSION
In this paper, we propose a heterogeneous architecture on FPGA with two cores (DSP/LUT-core) computing synchronously for efficient DNN inference. We also propose a hybrid quantization scheme which quantizes the NN weights according to the filter computation allocation. To rapid hardware-mapping evaluation and find the optimal resource allocation (BRAM, LUT) scheme for DSPand LUT-core, we build the latency and cost models for both that parameterized by architecture configuration knobs. Based on aforementioned models, we utilize the reinforcement learning agent to identify the optimal design considering both resource allocation and quantization in an end-to-end fashion, for low latency and high accuracy. The framework can adapt to different FPGAs by changing the available resources in the cost model. We utilize two FPGAs XC7Z020 and XC7Z045 as the test-beds to evaluate our framework, and the explored configuration can reduce latency by 1.12×-1.32× compared with the state-of-the-art Mix&Match design.

ACKNOWLEDGMENTS
This work was supported by Alibaba Group through Alibaba Innovative Research Program and National Natural Science Foundation of China (Grant No. 62102257).