{'fraction_non_alphanumeric': 0.03018924926091366, 'fraction_numerical': 0.0072721779849377445, 'mean_word_length': 4.968061815840309, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Spiking Hyperdimensional Network: Neuromorphic Models Integrated with Memory-Inspired Framework

Recently, brain-inspired computing models have shown great potential to outperform today's deep learning solutions in terms of robustness and energy efficiency. Particularly, Spiking Neural Networks (SNNs) and HyperDimensional Computing (HDC) have shown promising results in enabling efficient and robust cognitive learning. Despite the success, these two brain-inspired models have different strengths. While SNN mimics the physical properties of the human brain, HDC models the brain on a more abstract and functional level. Their design philosophies demonstrate complementary patterns that motivate their combination. With the help of the classical psychological model on memory, we propose SpikeHD, the first framework that fundamentally combines Spiking neural network and hyperdimensional computing. SpikeHD generates a scalable and strong cognitive learning system that better mimics brain functionality. SpikeHD exploits spiking neural networks to extract low-level features by preserving the spatial and temporal correlation of raw event-based spike data. Then, it utilizes HDC to operate over SNN output by mapping the signal into high-dimensional space, learning the abstract information, and classifying the data. Our extensive evaluation on a set of benchmark classification problems shows that SpikeHD provides the following benefit compared to SNN architecture: (1) significantly enhance learning capability by exploiting two-stage information processing, (2) enables substantial robustness to noise and failure, and (3) reduces the network size and required parameters to learn complex information.


Introduction
Many applications run machine learning algorithms to assimilate the data collected in the swarm of devices on the Internet of Things (IoT). Sending all the data to the cloud for processing is not scalable, cannot guarantee a real-time response. However, the high computational complexity and memory requirement of existing DNNs hinder usability to a wide variety of real-life embedded applications where the device resources and power budget is limited [1][2][3][4]. Therefore, we need alternative learning methods to train on the less-powerful IoT devices while ensuring robustness and generalization.
System efficiency comes from sensing and data processing. Unlike classical vision systems, neuromorphic systems try to efficiently capture a notion of seeing motion [5][6][7][8][9]. Bio-inspired learning methods, i.e., spiking neural networks (SNNs), address issues related to energy efficiency [5,[10][11][12][13][14][15][16][17][18][19][20][21][22]. SNNs have been widely used in many areas of learning and signal processing [23][24][25][26]. These systems have yet to provide robustness and intelligence that matches that from embodied human cognition. For example, the existing bio-inspired method cannot integrate sensory perceptions with actions. SNN applications in machine learning have largely been limited to very shallow neural network architectures for simple problems. Using deep SNN architecture often does not improve learning accuracy and can result in a possible training divergence [9]. In addition, SNNs lack brain-like robustness and cognitive support.
On the other hand, Hyperdimensional Computing (HDC) is introduced as a promising brain-inspired solution for robust and efficient learning [27]. HDC is motivated by the understanding that the human brain operates on high-dimensional representations of data originated from the large size of brain circuits [28]. It thereby models the human memory using points of a high-dimensional space, that is, with hypervectors. HDC performs a learning task after mapping data into high-dimensional space. This encoding is performed using a set of pre-generated base vectors. HDC is well suited to address several learning tasks in IoT systems as: (i) HDC is computationally efficient and amenable to hardware level optimization [29][30][31], (ii) it supports single-pass training with no back-propagation or gradient computation, (ii) HDC offers an intuitive and human-interpretable model [32], (iii) it is a computational paradigm that can be applied to a wide range of learning and cognitive problems [32][33][34][35][36][37][38], and (iv) it provides strong robustness to noise -a key strength for IoT systems [39]. Despite the above-listed advantages, HDC encoding schemes are not designed for handling neuromorphic data. HDC lacks the behavioral resemblance to neurons to extract features from neuromorphic data effectively.
While SNN mimics the physical properties of the brain (how biological neurons are operating), HDC models the brain at a more abstract and functional level. This makes these two computational models complementary. Inspired by the classical and popular memory model, introduced by Atkinson-Shiffrin [40], we propose a novel framework that fundamentally combines Spiking neural network and hyperdimensional computing. Our framework, called SpikeHD, enables a scalable and strong cognitive learning system to better mimic brain functionality. SpikeHD creates a cross-layer brain-inspired system that captures information of sensory data from different perspectives: low-level neural activity and pattern-based neural representation. Since both SNN and HDC have memorization capability, they are powerful in preserving spatial and temporal information. Therefore, SpikeHD can ensure advanced learning capability with high accuracy.
• To the best of our knowledge, SpikeHD is the first framework that fundamentally combines SNN and HDC. SpikeHD first exploits a few layers of spiking neural network to extract low-level spatiotemporal information of raw event-based data. Then, it utilizes HDC to operate over SNN output, learn the abstract information, and classifying the data. To ensure robust, efficient, and accurate HDC learning, we present a non-linear neural encoder that transforms data into knowledge at a very low cost and with comparable accuracy to state-of-the-art methods for diverse applications.
• We develop an end-to-end framework that enables co-training of SNN and HDC models. Instead of using deep SNN architecture, we exploit a simple SNN architecture that updates based on gradient rule and connects it to an HDC module capable of fast and single-pass learning. Our framework trains SNN and HDC models simultaneously to ensure that the data generated by SNN is optimal for HDC learning.
• SpikeHD supports online learning from the data stream. In this configuration, we keep the SNN layer static while exploiting HDC single-pass training capability to update the model in real-time. This enables SpikeHD model to learn or update its functionality with very few samples and without paying the cost of storing large-scale train data for iterative learning.
We evaluate SpikeHD on multiple classification problems. Our evaluation shows that SpikeHD provides significant benefits compared to both HDC and SNN architectures: (1) enhance learning capability by exploiting two-stage information processing, and (2) significantly reduces the network size and required parameters to learn complex information. For example, our results indicate that SpikeHD can provide 6.1% and 3.8% higher classification accuracy on MNIST and DVS Gesture datasets.

Brain-Inspired Computing Models
The human brain remains the most sophisticated processing component that has ever existed. The ever-growing research in biological vision, cognitive psychology, and neuroscience has given rise to many concepts that have led to prolific advancement in artificial intelligent accomplishing cognitive tasks [41][42][43].

Analogy from the Brain
In this work, we enhance the machine learning method by exploring and translating the memory processing capability of the brain [40]. To maximize the synergy between anthropogenic concepts and a body in silico, we analyzed the distinct neuromorphic nature of SNN and Vector Symbolic Architecture (VSA) [44,45]. We found that the two studies approach neuromorphic computing from complementary philosophies: SNN embodies the sensory processing patterns of the brain from a biological standpoint, while the VSA approach processes data from the behavioral patterns. Finally, we evaluate prototypes in both fields using DECOLLE [46] (as SNN representative) and Hyperdimensional Computing [27] (as VSA representative).
Memory model: The wildly accepted memory model of Atkinson and Shiffrin [47] includes sensory registers, short-term store, and long-term store. In particular, short-term store (STS) consists of the memory in storage for a short amount of time (referred to as "short-term memory") that can be actively engaged in processing ("working memory"). Long-term store (LTS) refers to the memory maintained for long periods of time. Structures centered around the hippocampus serves to process and transfer memory between short-term and long-term storage [48,49].
SNN (DECOLLE) as STS: SNN mimics biological neural networks at the neuronal level, where the representation of the information is the collective state of the spiking neurons, including membrane potential and synaptic states. Given neuromorphic data that are either transformed from frame-based counterparts or captured directly by Dynamic Vision Sensors (DVS), SNN has the structural advantage in accomplishing simple cognitive tasks. In particular, the recurrent nature of DECOLLE renders it ideal for serving both as a sensory processing unit and as an STS.
VSA (HDC) as LTS: Hyperdimensional Computing (HDC) mimics the brain on a functional and behavioral level. Just like how the hippocampus represents long-term memory for the sake of efficient storage and retrieval, HDC represents information in hypervectors that can be efficiently and robustly stored in hardware, such as FPGA and GPU, when compared to both non-VSA and most VSA representations. Given informative pieces of data (in this case, the sensory data after being processed by SNN), HDC can efficiently extract higher-level concepts through the process of encoding (what the hippocampus does) and memorization (what the long-term storage does).

Hyperdimensional Computing
The brain's circuits are massive in terms of numbers of neurons and synapses, suggesting that large circuits are fundamental to the brain's computing. HDC [27] explores this idea by looking at computing with ultra-wide words -that is, with very high-dimensional vectors, or hypervectors. The fundamental units of computation in HDC are high dimensional representations of data known as "hypervectors", which are constructed from raw signals using an encoding procedure. There exist a huge number of different, nearly orthogonal hypervectors with the dimensionality in the thousands [50]. This lets us combine such hypervectors into a new hypervector using well-defined vector space operations while keeping the information of the two with high probability. Hypervectors are holographic and (pseudo)random with i.i.d. components. A hypervector contains all the information combined and spread across all its components in a full holistic representation so that no element is more responsible for storing any piece of information than another.
In recent years, HDC has been employed in a range of applications, such as text classification [51], activity recognition [52], biomedical signal processing [53], multimodal sensor fusion [54], and distributed sensors [55,56]. A key HDC advantage is its training capability in one or few shots, where object categories are learned from one or few examples and in a single pass over the training data instead of many iterations. HDC has achieved comparable to higher accuracy compared to support vector machines (SVMs) [57,58], gradient boosting [59], and convolutional neural networks (CNNs) [32], as well as lower execution energy on embedded processors compared to SVMs [60], CNNs and long short-term memory [58].

Spiking Neural Network
Spiking neural networks (SNNs) are brain-inspired solutions for fault-tolerant and energy-efficient signal processing. SNNs take inspiration from the biological functionality of neurons in the human brain to engineering more efficient computing architectures. In the area of machine learning, SNN shares common properties to Recurrent Neural Network (RNNs), such as similarity in general architecture, temporal dynamics, and learning through weight adjustments [61]. Several works are now establishing formal equivalences between RNNs and networks of spiking leaky integrate-and-fire (LIF) neurons which are widely used in computational neuroscience [62]. In the LIF model, the neuron's state is the momentary activation level that can be pushed higher or lower depending on the incoming spike value. The neuron state will be reset to a lower value after firing the state [63].
There are multiple existing update rules in SNN, such as Spike Time Dependent Plasticity (STDP) and gradient descent. STDP depends on pre-synaptic and post-synaptic information [64]. Several existing hardware solutions have focused on their implementation in the forms of unsupervised and semi-supervised learning [65,66]. However, these works are limited in learning static patterns or shallow networks. Recent breakthrough research shows that Deep Continuous Local Learning (DECOLLE) provides effective and efficient training with approximate loss that maintains synaptic plasticity [23,24]. However, even with a gradient, the ability to train deep SNNs (with several hidden layers) has remained a major obstacle: the inclination for deep SNNs to diverge during training limits its expected potential.

SpikeHD Overview
In this paper, we propose SpikeHD, the first hybrid solution that fundamentally combines spiking neural networks and hyperdimensional computing. Our framework exploits SNN and HDC in the following ways: • Spiking Neural network: SNN extracts low-level information of neuromorphic data. SNN is like a feature extractor that learns spatiotemporal information of noisy spikes and projects them into meaningful representation. SNN eliminates noisy events that are less frequent in a temporal manner and exploits redundancy to further strengthen spatially correlated information. DECOLLE, in particular, uses deep continuous local learning, where the network errors are computed within each layer, thus requiring little memory overhead for computing gradients.
• Hyperdimensional Encoding: HDC performs a higher level learning over spike data generated by SNN. As explained in Section 1, HDC consists of two layers: encoder and learner. The encoder maps SNN output spikes into high-dimensional space. HDC encoder is unsupervised and significantly efficient since it does not require any training process. Since the encoder is non-linear (Please refer to Section 1 in supplement), a single-layer HDC classifier can effectively learn the data. The training only operates over the HDC model to keep efficiency and does not propagate to the encoder module.
How to Combine SNN & HDC: A naive approach can use SNN and HDC in parallel or in series to make a prediction. In the parallel version, both SNN and HDC can make independent predictions, and we can make decisions by looking at the decisions along with their confidence. However, this approach has the following challenges: (1) high computational cost to train two separated models, and (2) decision making and trust in the model is a complex task and requires another learning model. Similarly, the serial connection of SNN and HDC has the following challenges: (1) an information flow that is limited between the models impedes the ability of both: the latter fails to make good predictions due to a lack of information, and the former fails to be updated due to a lack of loss inferred from the prediction. spike data and is trained using gradient-based rule, while HDC works using dense high-dimensional data representation. This makes HDC and SNN learning not compatible and their interactions non-trivial.
Our Contribution: To get simultaneous benefits from SNN and HDC, SpikeHD needs to combine them based on their strengths and capabilities. Figure 1 shows an overview of our hybrid SpikeHD operating over neuromorphic data. SpikeHD exploits two layers of information extraction from event-based sensors: (1) SNN layer to extract low-level information by preserving spatiotemporal correlation of data and (2) hyperdimensional computing to learn the abstract and high-level trend of data. SpikeHD developed a novel framework that co-trains SNN and HDC models. The co-training enables the interaction between SNN and HDC to ensure convergence towards an optimal model. As Figure 1 shows, hyperdimensional learning, which operates over the spike data, has two components: a non-linear neural encoder that maps SNN output to high-dimensional space and an HDC learning model that combines encoded hypervectors to generate a hypervector representing each class. The HDC model will always take the final prediction.
In Section 1, we explained the details of HDC learning. In the rest of the proposal, we present our framework that integrates SNN and HDC (Section 4).

SpikeHD: Integration of Brain Models
In this section, we propose a novel framework to combine spiking neural networks and hyperdimensional computing. Figure 2 shows an overview of our framework combining SNN and HDC.

Step I: SNN Training
Our first step aims to establish feature extraction and to fine-tune the short-term memory behavior of the model. The solution starts by training the original SNN model, implemented with DECOLLE, using an entire or a batch of train data. The SNN is a multi-layer network that gets spike data as an input and makes a learning decision on the output layer. Depending on the loss function defined on SNN output, the SNN uses a gradient-based rule to update the synapse's weights ( • A ). During this phase of the training, the SNN learns to extract information from noisy neuromorphic data. Because the synaptic plasticity rules are partially derived from the neural dynamics of the spiking neurons, it is capable of learning spatiotemporal patterns. Increasingly complex features can be learned by adding layers to the SNN, but it is costly and introduces difficulty in convergence. Thus, the number of layers is often limited.

Step II: HDC Training
After training the SNN to approximate convergence, the solution applies HDC injection: we split SNN from early layers (often a relatively deep layer) and connect it to the HDC module to extract more abstract information ( • B ). In this step, the SNN layers are considered to be static because no training happens on SNN. For each train data, we first pass data through SNN. Then, we will be given the SNN representation in the split layer as labeled data to the HDC module for training. HDC encoding serves as the "hippocampus" and maps the spike data into high-dimensional space. Due to the non-linearity of the encoder, an efficient HDC learning module can effectively learn the pattern of data ( • C ). One advantage of HDC is its capability in learning a one-pass classification model. This eliminates the necessity of a costly iterative method to train a model. As the learning heavily depends on memorization of the encoded hypervectors, the HDC model serves as the long-term memory of the model.

Step III: SNN and HDC Co-training
The current approach trains SNN and HDC sequentially, where the SNN training does not get any feedback from HDC module. The SNN module is trained based on the loss function defined at the SNN output layer. However, as we explained, our hybrid architecture performs the prediction using the HDC model. This results in sub-optimal training of SpikeHD architecture.
To address this issue, we propose a novel co-training method that enables SNN to be trained based on the HDC model prediction. To ensure the SNN layer is well trained for HDC prediction, SpikeHD retrains the SNN layers after HDC training. For every training data or batch of data, SpikeHD starts updating the SNN as follows: Firstly, the spike data passes through the split SNN layer ( • D ) up until the point of injection and generates vectors as input to the HDC module. Next, the HDC module encodes the input, which is then compared to the HDC model; the loss function is computed against the target label ( • E ).
After that, the loss is used to update the HDC model in a single-pass. The learning in HDC is patternbased and can perform with significantly higher computation efficiency ( • F ). Then, HDC back-propagates the loss through the HDC module back to the point of injection ( • G ). Finally, we update the SNN model using a gradient-based rule with the backpropagated loss ( • H ). This procedure continues iteratively over train data until finding a suitable SNN representation that can ensure maximum prediction accuracy in HDC output.
One thing to note is that the back-propagation in ( • F ) mainly concerns two matrix multiplications with no significant overhead during training. Passing through the HDC model requires multiplying the loss with the HDC model itself, and passing through the encoder requires the inverse of the activation function and the (Moore-Penrose) inverse of the encoding matrix, the latter of which can be pre-computed upon the initialization of the model. Since the HDC model has faster training than SNN, one can decide to update the HDC model less frequently during the co-training step. During co-training, the HDC loss function can be back-propagated and used to update the SNN layer while the HDC model stays constant. The HDC model update can happen less frequently to ensure lower training costs.

SpikeHD Online Learning
Despite the effectiveness of the proposed framework, the training hybrid SNN and HDC model can be costly for small embedded devices. Our framework requires iteratively repeat co-training phases, which often require a large number of data and iterations to update the quality of the SNN model. Embedded devices may have the following limitations to implement SpikeHD iterative learning: 1) embedded devices often do not have enough memory to keep all train data for iterative learning. As a result, to enable online learning, SpikeHD needs to be updated in one-pass with no need to store train data. 2) embedded devices have limited resources which may not be enough to support the costly gradient-based model

Static
Step I: SNN Training Step II: HDC Training Step III: SNN + HDC Co-Training Step I -SNN iterative training. An original instance of SNN is trained without the influence of HDC, in which initial feature extraction recurrent learning from neuromorphic data are established.
Step II -HDC single-pass training. The HDC module is first injected into a deep SNN layer, and train data is propagated from the SNN input layer forward through the HDC module to make a prediction, which leads to modification in the HDC memory component.
Step III -SNN and HDC co-training. the loss from the output of HDC module is used to update both HDC memory and SNN layers simultaneously.
update required by SNN. Here, we propose a solution that enables online SpikeHD learning. As explained in Section 1, HDC supports single-pass training by creating a model with one-time looking at train data. We exploit this feature to update SpikeHD model in real-time based on the data stream. In this configuration, the SNN model is assumed to be static and does not get updated. Instead, for each batch of data, SpikeHD only updates the HDC model based on the generated loss function. This is similar to transfer learning, where the SNN knowledge is used for new data or environments. This model results in much faster convergence and eliminates the necessity of storing train data.

SpikeHD Scalability & Robustness
SpikeHD hybrid architecture provides an advanced brain-inspired learning solution with multi-layer information processing. This architecture is significantly strong in preserving spatiotemporal information.
SpikeHD also provides the following advantages: • Scalability: Using deep SNN architecture often does not improve learning accuracy or results in a possible divergence. SpikeHD hybrid architecture enables SNNs to use effective shallow networks rather than deep non-scalable networks. HDC encoding is used as secondary information processing to provide a high quality of learning while ensuring fast and scalable SNN training.
• Robustness: HDC encoding is holographic and redundant, thus provides significant robustness to noise and failure. Our represents stores information of events as a pattern of neural activity in highdimensional space. Therefore, losing a single or series of dimensions would not remove the information of an event. We further explore on SpikeHD robustness in Section 5.5.

Evaluation
We evaluate the classification performance of SpikeHD on two benchmarks: DVS Gesture Dataset [67] and spike-trained MNIST [68]. DVS Gesture Dataset is obtained by Dynamic Vision Sensor (DVS) capturing 11 types of hand and arm gestures from 29 distinct subjects under 3 different lighting conditions. It is thus natively neuromorphic. Spike-trained MNIST, in contrast, is artificial. It comes from processing the original frame-based MNIST images to spike trains, where the serialized pixel values determine the firing rate of the simulated sensors. The proposed SpikeHD framework has been implemented with two co-designed modules: spiking neural network and hyperdimensional computing. For SNN, we use an existing open-source library [46] that trains a network using the gradient-based rule. For HDC, we have developed an in-house library compatible with PyTorch. Our library is an optimized version of PyTorch that better handles the HDC memory requirement for CPU and GPU. The default parameters of SpikeHD is as follows. The SNN component consists of 5 LIF layers with respectively 150, 120, 100, 120, and 150 neurons. Each LIF layer is associated with a readout layer and a dropout layer as made necessary by local learning. For the HDC component, a dimension of D = 4000 is used, and the HDC encoder utilizes the hyperbolic tangent function (Tanh) as the activation function. The injection depth that indicates the layer of SNN where HDC is injected is by default 4, which means that it is injected right before the last LIF layer. Finally, the default dataset for evaluation is spike-trained MNIST. Our evaluation shows that default SpikeHD outperforms both SNN and HDC in terms of quality of learning. HDC model alone provides the lowest classification accuracy, as the HDC encoder is weak in extracting spatial and temporal information from noisy spike data. In other words, HDC learning is abstract and cannot be well adapted to extract low-level information from neuromorphic data. In contrast, SNN naturally models the brain's visual systems, thus providing high classification accuracy. However, the SNN accuracy saturates with the increase in the number of layers. In contrast, SpikeHD is a powerful classifier that extracts multi-layer information from the neuromorphic data. Therefore, it eliminates the necessity of using deep SNN layers. Our evaluation shows that SpikeHD achieves, on average, 5.7% and 3.2% higher classification accuracy compared to the SNN model after co-training on MNIST and DVS Gesture, respectively.

Hyperdimensional Encoding
To show the impact of our HDC encoder, Figure 3 also compares SpikeHD accuracy when HDC is using different encoding modules: linear (Binary) [32,69], random projection (Uniform) [58,70], and our proposed non-linear encoder (Gaussian). Our evaluation shows that SpikeHD using a non-linear encoder provides significantly higher classification accuracy compared to the other encoders, independent of the base hypervectors. In particular, linear encoding (labeled as 'Binary' in Figure 3) provides the lowest accuracy among them due to limited HDC memory capacity -4000 bits per class -compared to the ones with non-binary base vectors. The higher accuracy of our encoding comes from: (1) SpikeHD capability in considering the interactions between the features, and (2) exploiting an activation function that makes the mapping non-linear. Our evaluation shows that SpikeHD utilizing non-linear encoder achieves, on average, 3.1% and 2.4% higher quality of learning compared to linear and random projection encoder.
Our evaluation also showed that there is progressive improvement in the learning accuracy as the model proceeds with training steps. In most cases, the test accuracy has significant improvement from step I to step II, and slightly less from step III. We will discuss the possible causes in the later evaluations.
During step I of the training (Figure 4a), the accuracy of both the training and testing data quickly increases and stabilizes. This implies the efficiency and effectiveness of DECOLLE in simulating and processing spike data, despite its limited generalization as indicated by the non-trivial difference between test accuracy and train accuracy.
When the model enters step II, the training accuracy experience a sharp drop at epoch 20 because we switch from the latter LIF layer to HDC module (Figure 4b). The cause is obvious: since the newly introduced HDC memory is initially a random model, it has no predictive power. That said, the train/test is then reduced and stabilized in about one epoch. In addition, test accuracy experience an improvement (Figure 4c) for the displayed depths of injection, which indicates that HDC modules further extracted and memorized features from the DECOLLE layers that are useful to the prediction. We omitted the result of the trivial model with HDC-injection at depth-5 because it achieves similar performance as the baseline (pure SNN). This is because the depth-5 HDC module connects to the last layer of SNN, which essentially takes the prediction produced by SNN and does learning based on that. Finally, the model enters step 3 at epoch 30. Notice that the training accuracy does not have any improvement while the test accuracy has its final improvement (Figure 4d) most accentuated for the depth-3 model. This indicates that the improvement in test prediction comes from the update to the SNN layer in conjunction with that to the HDC memory.
As results indicate, the original SNN network suffers from overfitting, as demonstrated by the difference in train and test accuracy due to the depth of the network. In comparison, SpikeHD mitigates this issue by introducing more explicit memory based on non-linear encoding and is less sensitive to the number of layers. Starting from depth-4, SpikeHD observes significant improvement on the prediction, and the best performance is at depth-4 with the parameters we have chosen; this is partly due to DECOLLE's capacity to extract more meaningful features in the deeper layers, and the long-term memory that the HDC model provides gives rise to increased testing accuracy and, if not eliminated, mitigated the overfitting of DECOLLE model (Figure 4d). Notice firstly that even with only 100 MNIST samples, 10 for each class, DECOLLE was able to

Epochs
Step II Step III Step I Step I Step II extract many meaningful features. This is implicated by the immediate increase in test accuracy in Step I (See Figure 5). In Step II, the epoch-wise and time-wise convergence results are reported for both offline and online methods. The time-wise graph is shown as one epoch has different times in offline and online techniques. Our results indicate that the online method reaches convergence quicker than the offline method, though the offline method may perform better upon convergence. We observe that the training time of the offline method is much longer than the online learning method. This is partly due to DECOLLE's local training. We see from the convergence speed and the accuracy improvement that (1) online training incorporates new samples quickly into HDC memory, and (2) the co-training succeeds in backpropagating the loss to the SNN module so that it gets updated for better performance. Our evaluation shows that SpikeHD-online can provide comparable accuracy to SpikeHD-offline learning method even when the initial training is very limited (100 samples). In other words, SpikeHD can ignore costly iterative training for a big portion of train data. Instead, it simply updates the HDC model at a minimal cost. Our evaluation shows that SpikeHD-online can significantly speedup the training process and also reduces the memory footprint required for training. For example, SpikeHD-online enables 4.6× faster and 3.1× lower trainable memory while ensuring the same quality as an online model.

SpikeHD Accuracy and Robustness vs. HDC Dimensionality
Dimensionality creates a trade-off between three SpikeHD parameters: accuracy, efficiency, and robustness. Figure 6 shows the impact of hypervector dimension on SpikeHD test accuracy (considering 0% error rate). SpikeHD in higher dimensionality is a powerful model that can effectively learn the SNN output patterns. SpikeHD provides maximum accuracy even when the dimension reduces from D = 8k to D = 4k. Further decreasing the dimensionality from D = 4k results in a minor effect on SpikeHD quality of learning. For example, SpikeHD in D = 2k and D = 1k provides only 2.3% and 3.5% quality loss compared to SpikeHD model in full dimensionality (D = 8k). SpikeHD efficiency also directly depends on the model dimensionality. A higher dimensionality increases the number of required computations in both train and test. However, because the accuracy of HDC modules resembles a sigmoid function along the dimension, to reduce the computation cost, one can decide to use SpikeHD in lower dimensionality. For example, reducing SpikeHD dimension from D = 4k to D = 2k (D = 1k) results in 1.7× (3.1×) faster computation. We compare SpikeHD computational robustness with SNN. Our evaluation shows that SpikeHD HDC module significantly improves SNN robustness to possible noise and failure. Figure 6 shows SpikeHD accuracy when losing a different proportion of random neurons in the model. The results are reported for SpikeHD using different dimensionality and using different size SNN networks. Our evaluation shows that SpikeHD, in general, provides higher robustness than existing SNN, especially when SpikeHD dimensionality increases. For example, under 10% random noise, SpikeHD and SNN maintain 94.0% and 87.1% quality. The ability to sustain prediction quality generally increases as the dimension of HDC memory increases, though it does generate a slight dip when the HDC dimension is low. This can be attributed to the fault tolerance of DECOLLE and the higher vulnerability of low-dimensional HDC modules. Our results indicate that test accuracy has only a slight advantage when the HDC dimension is high, and SNN is large. This advantage will be more accentuated in smaller SNN, or in more complex tasks.

Conclusion and Discussion
In this paper, we propose SpikeHD, a novel framework that combines Spiking neural network and hyperdimensional computing in order to design a scalable and strong cognitive learning system that better mimics brain functionality. SpikeHD exploits spiking neural networks to extract low-level features by preserving the spatial and temporal correlation of raw event-based spike data. Then, we utilize HDC to operate over SNN output by mapping the signal into high-dimensional space, learning the abstract information, and classifying the data. Our evaluation on a wide range of classification problems shows that SpikeHD provides significant benefit compared to both HDC and SNN architecture: (1) enhances learning capability by exploiting two-stage information processing, (2) significantly reduces the network size and required parameters to learn complex information. For the rest of this section, we highlight some  of the open challenges that our framework has yet to overcome and encourage exploration of the question along multiple axes (Figure 7). Loss Backpropagation: During step III of SpikeHD, a Moore-Penrose inverse of the HDC encoder is applied to backpropagate the loss from the HDC module to the SNN. Since HDC encoder maps vectors to hypervectors, the rank of the inverse is limited to the output dimension of the SNN at the point of injection, which may be way less than the dimension. A large amount of information may be lost from this transition. We experimented with several methods to solve this problem. One example is to continue training the original SNN, transfer the new weights to the SpikeHD, and then train the HDC module. This did not improve performance except in the case of the transfer learning task, where the context of the data or the task changes. One method that may be suggested is to introduce a regularization term in the loss function of the SNN layers such that it outputs an HDC-like vector as the representation of the data directly. This will avoid the explicit usage of the HDC non-linear encoder, and the loss will be optimally propagated up to the approximation introduced by the regularization term.
Component Choices: We have selected DECOLLE as SNN and HDC as VSA for our hybrid model for the reason we've discussed in section2.1. It is optimized for time and energy efficiency, and practicality, as both models are known for such traits. Readers interested in the exploration of other aspects may choose to adopt our memory framework to other components. such as Legendre Memory Units [71] and HDC, or BI-SNN and HRR [72].
Concept Interpretability: Our current usage of the memory framework is to directly operate on long term memory and derives decisions from its representation, which simulates what the cerebellum does. For the purpose of completing the analogy to the Atkinson-Shiffrin memory model, it has yet to be incorporated the decoding mechanisms of the HDC memory: we did not fetch the long term memory back to the hippocampus and decode it for operations in working memory. This subject is not the purpose of this paper, and the decoupling of encoding and decoding invites more possibilities, as the HDC memory may be used as heterogeneous storage such that multiple tasks may be performed in one memory model.
That said, the subject of interpretability remains interesting at two levels. Each entry in the HDC memory represents the concept of the class, which can be decoded to retrieve a representation of the concept at the SNN level. The representation in SNN can then be interpreted to infer knowledge on the original sensory data.
Supplementary material for "Spiking Hyperdimensional Network: Neuromorphic Models Integrated with Memory-Inspired Framework"

Hyperdimensional Classification
We present a robust and lightweight hyperdimensional classification. Figure S1a shows an overview of HDC learning. The first step in HDC is to encode data into high-dimensional space. Then, HDC performs a learning task over encoder data by performing a single-pass training. The result of training will be to generate a hypervector representing each class. The inference task can perform by checking the similarity of an encoded query to the class hypervector.

Non-Linear Encoding
The encoder is the most important component of hyperdimensional learning. The goal of the encoder is to map data into high-dimensional space such that we can extract knowledge from data at a lower cost. The encoder selection depends on the data type (structure of data) and the target learning task. There are multiple existing HDC-based encoding methods [32,73,74]. Although these methods have shown excellent classification accuracy for their application-specific problems, to the best of our knowledge, the existing encoding methods linearly combine the hypervectors corresponding to each feature, resulting in sub-optimal classification quality for general classification problems. To obtain the most informative hypervectors, the HDC encoding should consider the non-linear interactions between the feature values with different weights.
In this context, we propose a novel encoding method that exploits the kernel trick [75,76] to map data points into the high-dimensional space. The underlying idea of the kernel trick is that data, which is not linearly separable in original dimensions, might be linearly separable in higher dimensions. Let us consider certain functions K(x, y) which are equivalent to the dot product in a different space, such that K(x, y) = Φ(x) · Φ(y), where Φ(·) is often a function for high dimensional projection. The Radial Basis Function or Gaussian Kernel is the most popular kernel: We can take advantage of this implicit mapping by replacing their decision function with a weighted sum of kernels: where (x i , y i ) is the training data sample, and the c i s are constant weights. The study in [75] showed that the inner product can efficiently approximate Radial Basis Function (RBF) kernel, such that: The Gaussian kernel function can now be approximated by the dot product of two vectors, z(x) and z(y). The proposed encoding method is inspired by the RBF kernel trick method. Figure S1b shows our encoding procedure. Assume an input vector in original space F = {f 1 , f 2 , · · · , f n } and F ∈ R n . The encoding module maps this vector into high-dimensional vector, H = {h 1 , h 2 , · · · , h D } ∈ R D , where D n. The following equation shows an encoding method that maps input vector into high-dimensional space: where B k s are randomly chosen hence orthogonal base hypervectors of dimension D 10k to retain the spatial or temporal location of features in an input and b i ∼ U(0, 2π). That is, B kj ∼ N (0, 1) and δ( B k 1 , B k 2 ) 0, where δ denotes the cosine similarity. However, this activation is not a convex function, After this step, each element h i of a hypervector H n has a non-binary value. In the HDC, binary (bipolar) hypervectors are often used for computation efficiency. We thus obtain the final encoded hypervector by binarizing it with a sign function (H = sign(H n )) where the sign function assigns all positive hypervector dimensions to '1' and zero/negative dimensions to '-1'. The encoded hypervector stores the information of each original data point with D bits. In our example, the feature vector, F , is highly sparse event-based data. However, after mapping data through the above encoder, the generated high-dimensional data will get a dense binary representation.

Hyperdimensional Model Training
We exploit hyperdimensional learning to directly operate over encoded data. Figure S1a shows an overview of HDC classification. HDC identifies common patterns during learning and eliminates the saturation of the class hypervectors during single-pass training. Instead of naively combining all encoded data, our approach adds each encoded data to class hypervectors depending on how much new information the pattern adds to class hypervectors. If a data point already exists in a class hypervector, HDC will add no or a tiny portion of data to the model to prevent hypervector saturation. If the prediction matches the expected output, no update will be made to avoid overfitting. This adaptive update provides a higher chance and weight to non-common patterns to represent the final model. This method can eliminate the necessity of using costly iterative training.
Let us assume H as a new training data point. HDC computes the cosine similarity of H with all class hypervectors. We compute similarity of a data point with C i as: δ( H, C i ). Instead of naively adding a data point to the model, HDC updates the model based on the δ similarity. If an input data has label l and correctly matches with the class, the model updates as follows: where η is a learning rate. A large δ l indicates that the input is a common data point which is already exist in the model. Therefore, our update adds a very small portion of encoded query to model to eliminate model saturation (1 − δ l 0). If the input data get an incorrect label of l , the model updates as: where δ l − δ l determines the weight that the model needs to be updated. Small δ l − δ l indicates that the query is marginally mismatched while larger mismatch is updated with a larger factor (δ l − δ l 0).  Figure S2: SpikeHD performance with different SNN/HDC training parameter ratios. The lines describe the average training accuracy of SpikeHD varying on the amount of trainable parameters that is assigned to SNN.

Hyperdimensional Inference
In inference, HDC checks the similarity of each encoded test data with the class hypervector in two steps. The first step encodes the input (the same encoding used for training) to produce a query hypervector H. Then we compute the similarity (δ) of H and all class hypervectors. Query data gets the label of the class with the highest similarity. Figure S2 demonstrates the effect of trainable parameter ratio on the performance of the model. To stress-test our model, the shape of the SNN is limited to 3 LIF layers with size ratio 2:3:2, and the dimension of HDC is derived from however many trainable parameter is allowed in HDC Memory, up to residue. For small SpikeHD (blue line with 10k parameters), there is a fine spot where optimal accuracy is reached. This is due to the fact that the training data requires an SNN big enough to extract meaningfull features (which explain the left slope) and an HDC memory large enough to memorize these features (which explain the right slope). Due to the fact that HDC memory accuracy resembles a sigmoid function with respect to dimension in practice, the accuracy deteriorate rapidly upon lowering the dimension to a certain threshold. This is demonstrated in the rapid increase in error rate for SpikeHD with very high SNN proportion. In addition, due to the correlation of the dimension threshold to the output dimension of the SNN instead of the size of the SNN, large SpikeHD (black line with 40k parameters) requires smaller proportion of HDC memory to perform well. The demand for the size of HDC also comes the number of classes. For example, the Omniglot dataset, a few-shot, 1623-class classification task, demands high HDC memory.