{'fraction_non_alphanumeric': 0.03233889280400908, 'fraction_numerical': 0.008926474042753112, 'mean_word_length': 5.053080568720379, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluating complexity and resilience trade-offs in emerging memory inference machines

Neuromorphic-style inference only works well if limited hardware resources are maximized properly, e.g. accuracy continues to scale with parameters and complexity in the face of potential disturbance. In this work, we use realistic crossbar simulations to highlight that compact implementations of deep neural networks are unexpectedly susceptible to collapse from multiple system disturbances. Our work proposes a middle path towards high performance and strong resilience utilizing the Mosaics framework, and specifically by re-using synaptic connections in a recurrent neural network implementation that possesses a natural form of noise-immunity.


I. INTRODUCTION
A variety of neuromorphic systems have been implemented using emerging memory devices, but few perform at industrial levels due to the difficulties of implementing softwarequality synapses and gradients. These challenges are somewhat simplified in the inference-only application, where imported weights from a separately pre-trained network are used to accelerate predictions on a known task. Nevertheless, electrical issues such as parasitics, cycle-to-cycle read noise, and deviceto-device variability may limit the ultimate size and accuracy of inference-only neuromorphic accelerators [1], [2]. These issues can quickly degrade the performance of deep networks with hundreds of thousands of parameters. Accelerators have been proposed to leverage memristive crossbars such as PUMA [3], ISAAC, [4], but they typically discuss the implications of these issues at minimal length. We explore the idea that these systems may be unexpectedly fragile to combined perturbations, and seek mitigations.
Neural networks with an intrinsic temporal behavior may yield new sources of power and resilience [5]. In order to efficiently map standard networks to temporal ones, the efficiency of fundamental operations in a given graph must be considered. As illustrated in Fig. 1, while the overall size of an artificial neural network (ANN) graph may be very large, parts of the necessary computation are done repeatedly. Mosaics in this context refers to a temporal form of neuromorphic multiplexing, whereby certain computations (in this example, the neurons, which are often a limiting resource) can be reused for different stages of an algorithm; allowing larger scale algorithms to be implemented on a resource restricted neuromorphic platform. By exploring the partitioning of a neural graph into sub-graphs that enable the computation to be performed on a smaller subset of available computing nodes -trading space for time -we open a new avenue for failure and resilience analysis. Concretely, in the following sections we explore this contrast through three relatively well-known ANNs: a) a simple, low parameter multi-layer perceptron, b) a complex, medium parameter convolutional neural network (CNN), and c) a medium-parameter, medium complexity recurrent neural network (RNN) inspired by Mosaics re-use concepts. In general, RNNs are an attractive emerging option for the demonstration of on-chip learning or inference, as they are powerful general computational models [6] . Recently, long-short-term memory (LSTM) networks have been the most heavily considered for implementation with dense nonvolatile memory arrays [7]; however, such schemes involve complex crossbar partioning and hardware implementation of many transcendental functions (e.g. tanh) on the edge of the periphery. Feasibly, LSTM implementations can be done on-chip during inference mode, such as with phase-change memory (PCM) NVM synapses [8], yet the full overhead required to implement such systems has not been documented.
Drawing on previous work which shows that an RNN network using appropriate, simple (ReLU) activation functions can still perform competitively to an LSTM on certain tasks [9], in the following sections we demonstrate that a vanilla RNN system can perform competitively to a compact CNN on two state of the art machine learning (ML) tasks, at a lower energy budget.

II. METHODOLOGY
We trained CNN models using the Keras framework [10] on the iconic MNIST data task [11] as well as the newer fashion-MNIST data task [12]. We then imported these models into CrossSim, a crossbar simulation tool that helps model resistive memory crossbars as highly parameterizable neural cores [13], and which has recently been extended to perform physics-rich analysis of inference operations. Our imported models for CNNs contain 119,322 trainable parameters and contain both convolutional and fully connected (FC) layers, each followed by a bounded rectified linear function (ReLU, given as f (x) = min(max(0, x), 1)) as visible in Fig. 2(a); exact model is given in Table 1. [14] and as visible in Fig.  2(a). Our imported MLP models are standard consisting of one hidden layer (128 ReLU functions) and a logit of 10; this model has 101,770 trainable parameters. Our recurrent neural network design, newly developed for this work and visible in   The RNN core crossbar is connected to a read-out/logit crossbar of dimensions D o x L, where L is the number of classes (here L = 10); critically, the second core is only  activated at the final time step (every t steps). In our resilience testing strategy, we have considered two classes of pre-trained networks: noise-prepared or regularized networks, which train with jittered gaussian filters on every hidden neuron's ReLu function during training following the scheme given in [14] and standard/un-prepared networks which have been trained without noise. As first suggested in [15], the use of noise regularization provides a definitive improvement in inference (Test ) performance of models to internal and external noise or perturbation effects. During training phase, noise centered around 0 with a distribution width σ neu is injected into every ReLu neuron; during testing phase σ syn is then injected on a synapse-per-synase basis . This device-by-device synaptic dispersion most closely relates to intrinsic noise effects that would occur during the operation of large arrays (Johnson-Nyquist noise), but can approximate shot noise or devicespecific perturbations or weight variance as suggested in [1]. In addition, given the test set Y, we have also considered the addition of additive gaussian noise at dispersion σ te where Y noisy = Y + y(σ te ). We have perturbed all weight matrices in every system consistently based on the generation of random numbers with seeds different in every run.

A. Resilience to noise sources
First, we consider the raw resilience of MLP, RNN, and CNN networks to noise at physically plausible values. Noiseless NVM inference performance values for the MLP, RNN and CNN sit at 97.7%, 98.5%, and 99.1% for MNIST, and at 84.5%, 89.1%, and90.1% for f-MNIST. As demonstrated in Table II, all considered systems degrade least from these results given only internal noise, second best with just external noise, and suffer the most when the effects are compounded. There is a significant task-dependence, with CNN systems performing best on MNIST (easier task) even in the worstcase, and with RNN systems performing by far the best on the fashion-MNIST (harder task) case. Of particular concern/interest is the degradation of the CNN f-MNIST models to test-set noise degradation and combined noise degradation, which does even worse than the MLP system. Although the possibility of catastrophic responses of CNN networks to adversarial perturbations are welll-known, this effect can probably be at least partially lessened by a more complex CNN architecture with larger filter sizes [16].
Next, we consider broader sweeps of the internal noise parameter σ syn along with a consideration of the effect of noise-regularization during inference time for the two bestperforming systems overall (CNN, RNN). As visible in Figure  3, in both the MNIST (a,b) and f-MNIST (c,d) cases, the stochastic ReLU behavior during training helps the CNN systems far more than the RNN systems. For MNIST, the regularization helps the CNN achieve best overall performance at both the purely internal and combined noise source cases (blue improves to orange), while the regularization assists the RNN but not as dramatically (green to red). This same trend holds in fashion MNIST with only internal noise; however, in the combined noise case for fashion-MNIST, at σ syn < 0.075 the regularized and non-regularized models are broadly superior to CNN approaches. Overall, the current results support the argument that RNNs seems to have greater intrinsic noise immunity as compared to CNNs, while CNNs benefit far more from a standard neuron-level regularization approach.

B. Temporal stacking sensitivity of RNN
The considered RNN approach can treat the number of timesteps of input presentation , which are automatically concate-  nated at the output of the RNN core, as a free parameter. As noted in [9], extending t too far can cause problems in learning convergence since the temporal dependencies (trace) between the eventual output and input stream can become too weak. We find a similar result but with even greater sensitivity in the NVM system. As shown in Figure 4, when t = 26 a slight degradation is visible, and at t = 56 inference at the task even at no noise loses 8 − 10% depending on the task.

C. Energy overhead considerations
We combine elementary energy costs and system dimensions to estimate energy overhead. Two primary energy costs drive inference: the vector-matrix-multiply (VMM) operation, which requires signficant energy to charge and read out (given ADC costs) the memristive array, and the neuron activation energy costs. Using the methodology proposed in [17], the CMOS ReLU circuit given in [18], and considering two device candidates for inference -optimized filamentary resistive RAM (ReRAM) alongside charge-trap based ultra-low energy SONOS memory [19]-we find that emerging memory CNN systems are energy hungry relative to the other options (Table  III). Our proposed RNN system consumes only 7x more than the MLP system by exploiting the weight re-use or temporal encoding strategy, and saves between 13-38x energy compared to the competitor CNN system.

IV. DISCUSSION
One phenomenon suggested but not proven in our analysis is the idea that noise regularization is still helpful , but less critical than in more complex CNN models, due to the presence of attractor basins in recurrent architectures which help to provide some intrinsic level of noise resilience [20]. We have also discovered that binary stochastic neurons may not be sufficient for full RNN regularization; a more complex function or method may be required. One interesting method suggested in [21] would be the use of temporal skip connections. Another fruitful extension of this work could be the exploration of how recurrent , convolutional , or mixed architectures using effective regularization mtehods (tuned based on the timestep dynamics of networks) could provide defensive properties against adversarial noise or perturbation effects [22], [23]. Finally, while we have showed the Mosaics concept implemented in the temporal domain in this work, the efficient parallel or horizontal stacking of convolution operations may yield more efficient and/or resilient convolutional operations (Figure 1c).

V. CONCLUSION
Making emerging memory inference systems more reslient is an important goal for the neuromorphic engineering field, but so far analysis has focused almost exclusively on the limitations of CNN systems. In this work, we have put CNN resilience in conversation with other approaches and in particular our novel RNN approach. We have shown that an ostensibly simple recurrent neural network design efficiently implements the idea of temporal stacking or weight re-use and reduces energy costs by at least 13.5x while achieving results that are competitive with CNNs -at least on tasks of intermediate complexity. In the future, we plan to extend our physics-aware methods to analyse the cross-over points at which various deeper ANNs scale to state-of-the-art tasks given both temporally and physically sequential sub-systems.
ACKNOWLEDGMENT Sandia National Laboratories is a multimission laboratory managed and operated by NTESS, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energys National Nuclear Security Administration under contract de-na0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.