{'fraction_non_alphanumeric': 0.031107347200132196, 'fraction_numerical': 0.010988784004296367, 'mean_word_length': 4.760142772159429, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
KCNet: An Insect-Inspired Single-Hidden-Layer Neural Network with Randomized Binary Weights for Prediction and Classification Tasks

Fruit flies are established model systems for studying olfactory learning as they will readily learn to associate odors with both electric shock or sugar rewards. The mechanisms of the insect brain apparently responsible for odor learning form a relatively shallow neuronal architecture. Olfactory inputs are received by the antennal lobe (AL) of the brain, which produces an encoding of each odor mixture across ~50 sub-units known as glomeruli. Each of these glomeruli then project its component of this feature vector to several of ~2000 so-called Kenyon Cells (KCs) in a region of the brain known as the mushroom body (MB). Fly responses to odors are generated by small downstream neuropils that decode the higher-order representation from the MB. Research has shown that there is no recognizable pattern in the glomeruli--KC connections (and thus the particular higher-order representations); they are akin to fingerprints~-- even isogenic flies have different projections. Leveraging insights from this architecture, we propose KCNet, a single-hidden-layer neural network that contains sparse, randomized, binary weights between the input layer and the hidden layer and analytically learned weights between the hidden layer and the output layer. Furthermore, we also propose a dynamic optimization algorithm that enables the KCNet to increase performance beyond its structural limits by searching a more efficient set of inputs. For odorant-perception tasks that predict perceptual properties of an odorant, we show that KCNet outperforms existing data-driven approaches, such as XGBoost. For image-classification tasks, KCNet achieves reasonable performance on benchmark datasets (MNIST, Fashion-MNIST, and EMNIST) without any data-augmentation methods or convolutional layers and shows particularly fast running time. Thus, neural networks inspired by the insect brain can be both economical and perform well.


Introduction
Connectomics [48] aims to produce a blueprint of the connections between all neurons. Such maps promise to help illuminate how the brain learns and how memories are formed. Active connectome research focuses not only on the human brain [49] but also on the brains of insects [22,50]. The fruit fly Drosophila melanogaster has shown excellent associative learning ability to link certain odorant stimuli (i.e., complex mixtures of odor compounds) to rewards or penalties [12]; the fly tends to recognize a mixture of stimuli as if they were a unitary object [19,20]. Studies of the fruit fly nervous system (as well as related studies in insects in general) have revealed that olfactory signals from the antenna terminate in a so-called antennal lobe (AL) of the brain, which is a bundle of ∼50 discrete subunits known as glomeruli. These glomeruli act as feature extractors, turning each complex odor mixture a 50-digit code. Projection neurons (PNs) connect each of these glomeruli code components to multiple of the ∼2000 so-called Kenyon Cells (KC), which are located in a remote area of the insect brain known as the mushroom bodies (MB's). Thus, the projection from the AL to the MB represents a massive representational expansion. These KCs in the MB are then decoded by smaller neuropils that lead to actuation of the fly behavior. From a deep neural network perspective, this architecture has the form of a single-hidden-layer network with an input layer (the AL), a hidden layer (the MB), and several parallel output layers (the downstream neuropils that control fly motion). However, research into the topology of the AL-MB connections has revealed that the inputs to each KC are randomized, where even isogenic flies appear to express different connections [13]. Furthermore, feedback inhibition within the MB regulates excitability of the KCs to small regions, thereby enforcing a sparse representation of each odor across the MBs that, when artificially disrupted, greatly reduces the ability of flies to discriminate between odors [39]. Thus, flies achieve neuronal function through the involvement of only a small fraction of their latent-variable space in the MBs [29], where each of these latent variables are effectively a randomized combination of a few components of each feature [12].
Meanwhile, Artificial Neural Networks (ANNs) and deep learning have shown excellent performance as tools applied in various fields such as computer vision, speech recognition, and natural language processing, and nonlinear predictive classifiers that can extract useful features from input data [11,45]. However, training ANNs typically requires optimizing a non-convex objective function with multiple local optima and saddle points [25], which is difficult and costly to optimize. For this reason, neural networks with relatively simple structures (i.e., single hidden layer) have gained renewed attention [11]. The architecture of these models is easy to implement and allows reasonable performance through simple optimization and fast learning processes. A promising technique for the development of such networks is Randomization -construction of a neural network with a subset of randomly generated and fixed weights so that the remaining weights can be learned in a straightforward linear optimization problem. By projecting inputs randomly into a high-dimensional space and then focusing training on finding weights that map this space to the output, learning can be solved as a highly tractable linear least square problem [45]. Furthermore, not only is the nature of the optimization problem simpler, but the number of weights that need to be trained is only a fraction of the total weights in the network. Such networks have universal approximation properties [31, 33,37], which has allows them to be broadly applied to classification and regression problems.
In this paper, we propose a novel approach for generating randomized, single-hidden-layer networks based on insights from the fly brain that allow for increasing the efficiency of these simple architectures. In particular, we introduce KCNet, a single-hidden-layer neural network with sparse and randomized binary weights that mimics how odor stimuli are encoded in the insect brain's high-dimensional KC representation. In addition, we also define a dynamic optimization method based on the straight-through gradient estimator [8], which allows the model to dynamically explore a more efficient set of inputs. We first apply our model to odorant-perception tasks, which determines how the perceptual characteristics of odorants can relate to the structural properties of the molecules. We demonstrate that our model achieves better performance compared to the existing data-driven approaches, such as XGBoost. Furthermore, to demonstrate the proposed model's versatility, we also address image-classification tasks on image benchmark datasets, including MNIST, Fashion-MNIST, and EMNIST. We show that without the need for data augmentation or convolutional layers, KCNet can achieve reasonable performance.

Related Work
Our work is strongly inspired by insect neuroscience, particularly the olfactory architecture of the fruit-fly nervous system. Olfactory sensory neurons located on the antennae and maxillary palps of fruit flies are responsible for detecting odors. The majority of these neurons only express one of the ∼50 odor-receptor proteins [15,17,24,52]. The olfactory sensory neurons are projected to the AL, where they are merged so that the neurons representing the same receptor gather in one of the ∼50 glomeruli [17,24]. Thus, this wiring diagram in which each odorant receptor is mapped to one glomerulus shows how odors are represented in the AL. When an odorant stimulus or a molecule binds to its cognate receptors, the sensory neurons expressing these receptors fire, and an odor-specific pattern of glomerular activity emerges [53]. PNs involved with each glomerulus pass olfactory information to two higher brain centers: the lateral horn and the MB. Specifically, the MB is responsible for the fly's ability to learn to approach or avoid odors, and an input neuropil of the MB, called the calyx, contains the axon terminals of the ∼50 types of PNs that connect with ∼2000 KCs (i.e., a 40× representational expansion). This paper mainly focuses on the transformation of odorant representation in the AL to the higher-order representation across the KCs of the MB and how to implement artificial KCs in our ANN model.
Our work is also influenced by Neural Network with Random Weights (NNRW), a single-hiddenlayer neural network with random weights from the input layer to the hidden layer. The weights between the input and the hidden layers are randomly generated in a suitable range and kept fixed during the training process. Compared with conventional learning methods with global tuning, such as deep learning with the back propagation, NNRW mitigates problems related to slow learning convergence and local optimality traps due to highly non-convex optimization. NNRW can achieve a much faster training speed with acceptable accuracy. Furthermore, NNRW is easy to implement, and is theoretically capable of universal approximation [31, 33,37]. Although many studies of NNRWs have been conducted under several names, such as Extreme Learning Machine (ELM) [32], Random Vector-Functional Link (RVFL) networks [43], and Schmidt's method [46], the underlying ideas and training mechanism of the three methods are very similar, and so we briefly introduce the formulation of ELM model as follows.
Given a generic input x ∈ R d , the basic ELM model can be represented by linearly combining B random nonlinear transformations of the original input, as in where ith transformation h i is parameterized by a vector w i of randomly drawn weights from a specific probabilistic distribution or a certain rule. In contrast, the linear coefficients β i are trained using training data. Thus, inputs are randomly transformed to hidden latent variables, which the output layer is trained to decode. In the addictive case, a hidden node h i (·) of the model applies a predefined nonlinearity on a random affine combination of input vector, as in: where random vector and g is a nonlinear function, such as a sigmoid. Given a set of N data samples, where each sample i ∈ {1, . . . , N } consists of an input vector x i and a one-hot-encoded target vector y i ∈ R c , a properly trained model will best approximate the target matrix Y with the linear system Hβ, where which can be written compactly as That is, H is the hidden layer output matrix of the network; the ith column of H indicates the ith hidden node's output vector with respect to x 1 , . . . , x N , and the jth row of H represents the output vector of the hidden layer with respect to input x j . Because the learning problem is equivalent to a linear regression of the output (Y) on the transformed versions of the inputs (embedded in H), the output weights β can be estimated as β * = H † Y where H † is the Moore-Penrose generalized inverse [6,7] of the hidden layer output matrix H. As we discuss herein, our model is loosely connected to a specific configuration of NNRWs and exploits the learning process of training the output weights β.
3 KCNet: An Insect-Inspired Single Hidden Layer Neural Network

Intuition
For bio-inspiration of our model, we focus our attention on: (i) how to form random connections between the AL glomeruli (input layer) and the KCs (hidden layer), and (ii) how to ensure hiddenlayer sparsity to best differentiate one odor stimulus from another. To define an arbitrary set of inputs for each KC in the model, the connections between glomeruli and KCs can be represented by sparse, binary, and random matrices [13]. The firing rates from ∼7 randomly selected glomeruli are passed and summed to each KC [13,34]. For example, Dasgupta et al.
[18] proposed a computational model of the fly olfactory system, including a sparse binary random projection matrix, to generate a tag for each odor composed of a set of neurons that are excited when that odor is presented. We also refer to that definition and take 7 random indices of the input vector to generate the input for each KC.
The adequate method of representing the KCs' output activity is critical because the output of KCs is responsible for learning behavioral responses to different odors [42].
where KC inj indicates the weighted sum of input from 7 random indices of the input vector 1 , φ is the ReLU, C is the strength of global inhibition, and N KC is the total number of KCs. Herein, we use C = 1.0 and N KC = 2000 matching the values best calibrated to real KC responses [4,23].

Model Architecture
We take input from the feature space X ⊆ R d and derive our model for a single output for simplicity. Our proposed model with B (∼ N KC ) hidden nodes is a parametric function f B : R d → R taking an input x = [x 1 , x 2 , . . . , x d ] T . To define our randomized encoder in the hidden nodes, we select N in ∈ [1, d) indices of the input at random without replacement for each hidden node 2 . In other words, the outputx j of each hidden node j ∈ {1, ..., B} is a weighted combination of all d inputs where only N in < d inputs have w j,i = 1 and all other inputs have w j,i = 0. So the inputs are effectively gated by the weights on each hidden node. Once the jth hidden node has produced weighted sumx j , the post-processing step to produce intermediate output is performed in the hidden layer. Mimicking Eq. (4) with C = 1, the jth intermediate outputx j is: where B is the number of hidden units. In the model's decoder definition, the proposed network is defined as follows in the same way that a linear model is applied as a decoder in NNRW:   where g is the ReLU function [26] that allows the model to produce sparse hidden output, which is more biologically plausible. Therefore, our model can be formulated in the same form as in NNRW with the hidden layer output matrix H and target matrix Y from Eq. (3). However, in contrast with NNRW, we estimate β with a regularized Moore-Penrose inverse: where λ is a specific regularization term; using a ridge regularization helps to improve generalizability and performance of the model. The overall architecture of the KCNet is depicted in Fig. 1. Our model has several noticeable characteristics that can be translated in terms of computer science and machine learning, and the details can be seen in Appendix A.2.

Dynamic Optimization Algorithm (DOA) For Feature Selection
Although KCNet can perform well by increasing the number of the nodes in the hidden layer (i.e., the hidden layer's width), we define the Dynamic Optimization Algorithm (DOA), an iterative method that allows the KCNet to explore a more efficient set of random inputs to obtain better performance than conventional ones. For a fully trained single-hidden-layer neural network, all the hidden weights and the output weights are optimized with respect to a dataset and a loss function using gradient descent, whereas the hidden weights of the KCNet are initially fixed to either 0 or 1, and training adapts only the output weights during its learning process. What if we can find efficient combinations of inputs that allow the model to achieve better performance? How can we find which connection is effective while turning on or off the connection between the input and hidden layers? Our algorithm never updates the weights directly to address the issue. Since it associates two options, 0 or 1, for each connection, we instead optimize the preference to select the weight between 0 and 1 to use.
Initialization Let w ji be the connection between the ith input node and the jth hidden node, and let s ji ∈ [−1, 1] be the preference score for w ji , indicating the degree to which it prefers to choose between 0 and 1. We initially draw s ji ∼ U(lb, ub) randomly from a uniform distribution between lb = −1 and ub = 1 after generating the randomized binary weight w ji ∈ {0, 1} on each input. Then, we can readily convert the weight w ji into the preference score s ji , and vice versa, by defining selection functions ρ w→s and ρ s→w as follows: With the above setting, we can start by iterating the following forward pass and the backward pass to update the preference score s ij instead of w ij using the gradient descent method. However, because neither selection functions ρ are differentiable, we cannot directly compute the gradients for s ji . We explain how to solve this in the part of the backward pass.
Forward pass: training output weight We split the dataset into two sets randomly; the training set for learning the output weight β and the validation set for searching a random collection of input. Then, we build a model using the weight w ji and train β using a least-squares method.
Backward pass: updating preference score In the backward pass, all the preference scores are updated with gradient descent method based on the straight-through gradient estimation [8] using the validation set by propagating to s ji the gradients that w ji receives. Specifically, the selection function ρ s→w is a kind of threshold function in Eq. (8), and we ignore the derivative of the function and pass on the gradients that w ji receives as if the function was an identity function. Therefore, we can bypass the selection function in the backward pass and compute the loss gradient with respect to s ji : where L is the objective function. Therefore, it is then back-propagated to obtain gradients on the parameters that influence s ji (see the detail of the full derivative in Appendix A.1). Similar to usual back-propagation method, the preference score s ji is updated by gradient descent with learning rate α > 0: s ji ← s ji − α∇ sji (10) and then the updated preference score is saturated to be between −1 and 1. Based on the new score, the weight w ji is also updated by applying the selection function ρ s→w to it, and the forward pass to train β of the model performs again. We call the pair of forward and backward passes an epoch, and by continually iterating epochs, the KCNet can automatically find a more efficient combination of inputs that distinguish the representation belonging to one category from others. To terminate the iterative process, we established two stopping criteria: i) if a process has run a certain number of times, and ii) if the performance of the current model exceeds the standard we set. Our experiments demonstrate that this simple algorithm allows the model to find convincing configurations of random inputs resulting in good results across different datasets (see details in Algorithm 1 in Appendix).
Ensemble DOA to assemble a weight matrix Although the above DOA can enhance the conventional KCNet, the computational burden drastically increases if the size of the dataset or the number of hidden nodes in the model is enormous due to, for example, the computation of an inverse matrix or an inner product between input matrix and the derivative of a loss matrix with respect to the input in the backward pass. Furthermore, we noticed that the larger the size of the weight matrix, the more perturbed the convergence of the performance of the DOA-applied KCNet and the more sensitive the learning rate (see Fig. 7 in Appendix). Thus, instead of directly applying the DOA to the KCNet with a large weight matrix, we introduce another use case of DOA, Ensemble DOA, to assemble a large weight matrix by concatenating several small weight matrices after each small weight matrix was adapted individually by the DOA. In our experiments, we empirically determined that it was most effective to divide a large weight matrix into more than 10 small matrices. In this condition, KCNet with the Ensemble DOA shows a much faster training speed than the KCNet with the DOA and outperforms the original KCNet and the KCNet directly applied by the DOA.

Odor Perception
The odorant perception task is the prediction of an identity of an odor from the physical, chemical, and structural properties of its odor molecules. Understanding odorant perception continues to be an active area of research due to the complexity of the task. First of all, the dimensionality of the feature space for odorant perception and the relationship between the structure and odor are still unknown [10,40,47]. Furthermore, odor molecules cannot be predicted to have the same odor properties by looking at similarities in structural properties [9,21]. Rather, odor molecules with completely different structural features may have similar scents [2]. And depending on how concentrated the odor stimulus is in the environment, it can be perceived in many different smells [2].
Chacko et al. [14] focused on a specialized form of the odorant perception task where odors could be classified by the presence or absence of two particular subjective properties referred to as Sweet and Musky. They used a psychophysical dataset including odor perception ratings of 55 human subjects. Based on this dataset, Chacko et al. used machine learning to predict these perceptual characteristics from an odor molecule based on its structural properties. We applied our models to the same dataset, and we found that our KCNet models outperform the other data-driven approaches, such as XGBoost, that were used by Chacko et al. [14].
Data pre-processing Here, we briefly summarize the data pre-processing step used by Chacko et al. [14] to define the odor-perception classification problem. For input labels, Keller and Vosshall [35] proposed a dataset that consists of odorant attribute data for 480 structurally distinct compounds at two concentrations. The odor properties included were 20 semantic descriptors, such as Sweet, Musky, Flower, and so on, associating with odor quality. 55 subjects evaluated each of these perceptual attributes in the range of 0 to 100. The degree of the evaluation indicates how well each semantic descriptor describes the perceptual characteristics of the odor. We assigned ground-truth labels for their odor quality to each odorant compound based on the frequency of using semantic descriptors assessed by the 55 subjects. To consider the complexity of odor perception and the possibility that an odor stimulus can be recognized to have multiple odor characteristics, we selected the top three semantic descriptors for each odorant compound. Thus, we defined each odor stimulus to be represented by three Odor Characters (OCs).
For input features, we utilized 196 two-dimensional RDKit molecular data as input features [28]. It consists of topological descriptors, connectivity descriptors, constitutional descriptors, molecular property, and MOE-type descriptors and provides quantitative information about the physical, chemical, and topological features measured in the 2D graph representations of the structures. After some zero values of the descriptors were removed and the dilution was converted into the one-hot encoding as a categorical feature, we have 960 data samples (480 compounds × 2 dilutions) with 159 input features. After data pre-processing, we noticed that other OCs showed a high imbalance in their distribution, whereas Sweet and Musky OCs had enough samples for the positive class (58.13%, 41.9%, respectively). Therefore, we formulated it as a binary classification task by creating two datasets: one for the Sweet OC and another for the Musky OC.
Methods We randomly split the processed dataset into a training and testing set with a 9:1 ratio after applying a standard normalization.
Only the training set was utilized for model parameter tuning using a 5-fold cross-validation. During cross-validation, the training set was split into five folds at every iteration, where one fold was kept as a validation set to test the performance with the trained parameters. For the classification of Musky OC, we applied an oversampling method to the training data, excluding the validation set, to reduce the imbalance in the data distribution. The weighted F1 score was used as an evaluation metric because of the imbalanced characteristic of the datasets. We configured the hyperparameter search space consistent with the models of Chacko et al. [14] and a specific range for the regularization term λ for our model. We utilized the Optuna framework [1] by repeating the cross-validation 100 times. After finding the best-performing number of hidden nodes of the KC-Net, we set it as the number of hidden nodes for both the DOA-applied model and the Ensemble-DOA-applied model, and we used the same value of the regularization term λ as was used for KCNet. Details of the methods, hyperparameter search spaces, and implementations are given in Appendix A.4.
Results Table 1 shows the performance comparison of test-weighted F1 score with 95% confidence interval averaged by five experiments among models. The performance variances of the KCNet and its variants are natural because their weight matrices are stochastically determined, while the performances of other models are stable after their hyperparameters are fixed. Our model and its variants usually outperforms other models, and the best performing algorithm was the KCNet with Ensemble DOA for Sweet OC and the original KCNet for Musky OC, with test-weighted F1-scores of 0.8193 and 0.7434, respectively. Applying the DOA to the KCNet deteriorated the performance of the KCNet for Musky OC, which may indicate that the sample size for this class is not large enough, and the oversampling techniques might disturb the performance of the DOA. As shown in Fig. 2, after Table 1: Test-weighted F1 score of all models for odor perception. "w/ DOA" is the DOA-applied KCNet and "w/ Ensemble DOA" is the Ensemble-DOA-applied KCNet. Mean ±SD (# of hidden units) for "KCNet" and "w/ DOA" and Mean ±SD (# of submodels × # of hidden units for each submodel) for "w/ Ensemble DOA". The best performance is displayed in bold.
Classification Tasks

Image Classification
To show the general applicability of our model, we applied it to the image-classification task using well-known image benchmark datasets, MNIST [36], Fashion-MNIST [54], and EMNIST [16].
Datasets MNIST is comprised of 10-class 28 × 28 pixel grayscale images of handwritten digits. It contains 60,000 images for training and 10,000 images for testing. Fashion-MNIST is a dataset including 28 × 28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training (test) set has 60,000 (10,000) images. The EMNIST dataset is a set of handwritten characters and digits derived from the NIST Special Database 19 and converted to a 28 × 28 image format. Six datasets that comprise the EMNIST dataset but we focus one subset, EMNIST-Balanced, which has 47 classes and 131,600 images so that 112,800 images for training and 18,800 images for testing can be used.
Methods We implemented the KCNet with PyTorch [44] to boost the training speed supported by GPUs; PyTorch supports the above three datasets so that they can be easily used. Therefore, we conducted our experiments after only applying a standard normalization to them without the use of any data-augmentation method. We also implement the ELM model and a Fully-trained Single-Hidden-layer Neural network (FSHN) and test them to compare the performance with our model. The ELM was introduced as a baseline classifier by Cohen et al. [16], and its characteristics are very similar to our model. We formulated the ELM model in a method that is often used in practice [11,32,45] as follows: i) its hidden weights w i = [a T i , b i ] T were defined by drawing uniform random variables a i ∼ U(−1, 1) and b i ∼ U(0, 1); ii) the sigmoid function is used as a nonlinearity function g; and iii) the output weight β was estimated using the Moore-Penrose inverse. FSHN was implemented using the linear models followed by the ReLU function and optimized using the stochastic gradient descent, supported by PyTorch. The number of hidden nodes in the FSHN was set to have the number of adaptable parameters similar to KCNet. For hyperparameter setting in the DOA-applied KCNet and the Ensemble-DOA-applied KCNet, the original training set was randomly split into a training set and a validation set with a 5:1 ratio every epoch. See Appendix A.4 for details of methods, hyperparameter search spaces, and implementations.
Results Table 2 shows the results of test accuracy with 95% confidence interval on the image benchmark datasets over five experiments. Our model outperforms ELM, and Fig. 3a (and Fig. 8 in Appendix) confirms that the KCNet performs better than the ELM model across all the image Table 2: Test accuracy of all models for image classification. Mean ±SD (# of hidden units) for "ELM", "KCNet" and "w/ DOA" and Mean ±SD (# of learnable parameters) for "FSHN". Mean ±SD (# of submodels × # of hidden units for each submodel) for "w/ Ensemble DOA". The best performance is displayed in bold.
Image  The best-performing model is the FSHN for EMNIST-Balanced, and the Ensemble-DOA-applied KCNet for MNIST and Fashion-MNIST. The direct application of DOA to KCNet seems rather performance-deprecating, which shows that the larger the weight matrix, the less fully exploitable the DOA. However, the Ensemble DOA is very effective at enhancing the KCNet because the DOA can be fully utilized to explore an effective set of inputs for relatively small weight matrices. Even though the performance of KCNet was worse than the FSHN, the running time of our model was much faster than FSHN (and ELM) across all the benchmark datasets as shown in Fig. 3b. Therefore, the KCNet allows for compromising performance slightly in order to increase speed. For more details, see Appendix A.4.

Conclusions and Broader Impact
We proposed a fast, insect-brain-inspired single-hidden-layer neural network and a dynamic optimization algorithm that allows the model to explore meaningful connections in hidden weights and to utilize an ensemble method of assembling weight matrices. Although we showed potential of our model through the above experimental results, there are several limitations yet to be addressed, as detailed in Appendix A.5. For future work, by using the discriminating power of our model, we will extend application to other tasks, such as one-shot learning or contrastive learning. Furthermore, we will construct a more brain-like model by combining our model with biologically plausible learning algorithms, such as target propagation [41].
Broader Impact This study integrates ideas from computational insect neuroscience with machine learning to show how neural networks can be made smaller and simpler to train with little compromise in performance. The net societal impacts of this research are primarily positive. By helping to reduce the resource footprint and increase the speed of neural networks used for common tasks, this research increases the sustainability of artificial neural networks. That said, reducing the barriers to the application of AI can lead to faster growth in the use of AI, which can be positive or negative. Beyond the technological impacts, this research has positive scientific value as it further illuminates the adaptive function underlying architectures in the insect brain, potentially aiding in understanding the evolution of these architectures. Despite the many qualitative large-scale differences between arthropod and vertebrate brains, a number of similar circuit motifs and smaller-scale brain structures have been found. Thus, better understanding how insects learn to discriminate among complex signals can eventually also aid in understanding the mechanisms and evolution of similar functions in the human brain.

A.1 The Derivative of ∇ sji
All of our experiments make use of classification tasks, with model outputs generated by applying the softmax function. Consequently, the objective function L that we use is the categorical cross-entropy loss function. The loss function for the odor-perception task is a binary cross-entropy loss, which is a special form of the categorical cross-entropy loss.
Given an input vector x ∈ R d and one-hot-encoded target vector y ∈ R c , consider the case where the output weight matrix β * of the KCNet was selected to best fit the linear system β T h = y where h is the hidden layer output vector of the network, as in: andx is the processed input in Eq. (5). Based on this, the predictionŷ of the KCNet is: In the backward pass, the batch gradient descent using validation set is performed. Given the validation set, the gradient ∇ sji can be computed as follows: Because ∂x j /∂x j is a constant and can be integrated with the learning rate α, we can write ∇ sji concisely as follows:

A.2 Detailed characteristics of the KCNet
Although the KCNet is devised from the underlying perspectives of biology and neuroscience, several apparent characteristics of the model can also be interpreted from the perspective of computer science and machine learning.
First of all, determining the weight between the input and the hidden layers in the KCNet can be seen as a special case of random weight generation in a NNRW, and the KCNet also uses a bounded nonlinear piecewise continuous activation function, which is the ReLU in our setting. Therefore, KCNet inherits the universal approximation property of NNRWs, which means that a single-hiddenlayer network with randomly generated hidden nodes and with a broad type of activation functions can universally approximate any continuous target functions in any compact subset X ⊆ R d [30-32]. So, theoretically, the more the number of hidden nodes in the KCNet (i.e., the wider the hidden layer), the better its performance (see the details of KCNet performance results in Section 4).
The second property of the proposed network is that from the point of view in computer science, the hidden-layer output can be viewed as the output of a hash function. Dasgupta et al. [18] took this perspective to interpret the Kenyon Cells (KCs) as locality-sensitive tags for odors; the more similar a pair of odors, the more similar their assigned tags. Based on this idea, they developed a brain-inspired class of Locality-Sensitive Hash (LSH) functions [3] for efficiently searching approximate nearest neighbors of high-dimensional points. Because our formulation is partly derived from those insights, our model also can generate tags that preserve the neighborhood structure of the input points using distance-preserving embedding of points, as a family of LSH. Whereas we have focused on the trade-off between conciseness and expressiveness in the width of the KC hidden layer in artificial neural network applications, Dasgupta et al. [18] provided computational theory to better understand the performance of architectures matching the scale of the fly brain. Moreover, in their application, a Hebbian-like update rule was used for unsupervised learning of whether an odor had been experienced before (in contrast to the supervised learning we employed). However, their analysis combined with ours implies that KCNet may be employed in unsupervised learning applications, and there will likely be a similar trade-off between the width of the hidden layer and the overall performance of the trained networks.
Finally, to mimic the KCs receiving global inhibition from each other, we subtract the mean of all hidden nodes outputs from each intermediate output in Eq. (5). That computation can be viewed as a form of the layer normalization in machine learning [5], which helps to maintain the contributions of each feature so that the network can be unbiased, reduce the internal covariance shift, and accelerate the optimization process. Thus, our model helps to import insights from machine learning into the interpretation of the adaptive function of global inhibition in neuronal structures like the mushroom body.

Details of Experiments
In this section, we describe the details of our experimental results. The following details apply in common to the two tasks we addressed, odor perception and image classification. Details specific to each task are described in subsections of each task. All the following results were averaged over five experimental replications. Hardware specification of the server The hardware specification of the server that we used to experiment is as follows: • CPU: Intel® Core TM i7-6950X CPU @ 3.00GHz (up to 3.50 GHz) • RAM: 128 GB (DDR4 2400MHz) • GPU: NVIDIA GeForce Titan Xp GP102 (Pascal architecture, 3840 CUDA Cores @ 1. 6 GHz, 384 bit bus width, 12 GB GDDR G5X memory) Source code All source codes are available at https://anonymous.4open.science/r/ KCNet-E858. They have been provided as electronic supplementary material as well.
Hyperparameter search space Table 3 shows the hyperparameter search space of all models for all tasks. To optimize over this hyperparameter space, we used the Optuna [1] framework with 100 iterations. It is possible to dynamically construct the parameter search space because the framework allows users to formulate hyperparameter optimization as the maximization/minimization process of an objective function that takes a set of hyperparameters as input and returns a validation score. Furthermore, it provides efficient sampling methods, such as relational sampling that exploits the correlations among the parameters.
The hyperparameter search space for other non-KCNet machine learning algorithms used as benchmarks were taken to match Chacko et al. [14] for odor perception tasks.
To set the number of submodels for the Ensemble-DOA-applied KCNet, we empirically determined that it was the most efficient to divide a large-sized weight matrix of the KCNet into at least 10 small matrices for the tasks that we addressed. Furthermore, we set a value such that the product of the number of submodels and the number of hidden nodes for each submodel in the Ensemble-DOAapplied KCNet equals the number of hidden nodes in the KCNet or DOA-applied KCNet. By doing so, they have the exact dimension of the output weight matrix.

A.4.1 Odor Perception
Methods and Implementations Figure 4 depicts the overall workflow for odor-perception tasks. We used the compound identification (CID) number of 480 odorants included in the original dataset to generate the SMILES notation of the molecules, which was utilized to compute the RDKit descriptors. We obtained the SMILES notations of the odorant compounds using the PubChemPy package of Python programming language. RDKit descriptors were calculated using ChemDes, a free web-based platform for the calculation of molecular descriptors. All the preprocessing steps and model training was performed using Python libraries: NumPy, pandas, imblearn, and scikit-learn. All models that we tested are provided by scikit-learn, and we also implemented our proposed model compatible with scikit-learn. The ADASYN class of the imblearn package was used to scale and oversample the training data during the five-fold cross-validation. In addition, we set the random_state to 1 for reproducing our results readily. Figures are generated using Matplotlib and Seaborn libraries supported by Python programming language.
Results Figure 5 shows the results about the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for Sweet/Non-sweet OC. Figures 5a and 5b represent the initial preference score and weight matrix, respectively, before applying the DOA to the KCNet. Figure 5c depicts the change in the validation-weighted F1 score during the DOA, and this resulted in the final weight matrix of the KCNet shown in Fig. 5d. Compared to the initial weight matrix in Fig. 5b, the final matrix was more densified to enable the exploitation of the combination of inputs. We applied the Ensemble DOA to the KCNet by dividing the weight matrix with 1,300 hidden nodes into 13 submodels with 100 hidden nodes for each submodel. Figure 5e demonstrates the change of the validation-weighted F1 score of 13 submodels during the Ensemble DOA and the final weight matrix assembled by 13 submodels was shown in Fig. 5f. Compared to the weight matrix of the DOA-applied KCNet in Fig. 5d, it appears to have a slightly more regular pattern, which seems to be due to the concatenation of multiple small matrices. The best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for Sweet OC as follows: • The KCNet : lambda = 100 • The DOA-applied KCNet : n_epoch = 100, learning_rate = 0.5, stop_metric = 0.87 • The Ensemble-DOA-applied KCNet : n_epoch = 100, learning_rate = 0.5, stop_metric = 0.90 Similar to the above description, Fig. 6 demonstrates the results about the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for Musky/Non-musky OC. The best hyperparameter settings for the KCNet, the DOA-applied KCNet and the Ensemble-DOA-applied KCNet for Musky OC as follows: •  Table 2. Because the FSHN exploits an usual linear model supported by PyTorch and it includes biases as parameters, we set them to match the number of parameters of the KCNet as closely as possible. The numbers of hidden nodes in the FSHN that we set can be seen at source codes that we provided.
For the hyperparameter setting for the FSHN, by referring to the basic tutorial of constructing a classifier using MNIST that PyTorch provides, we define the FSHN as follows: training_batch_size = 64, test_batch_size = 1000, epoch = 5, learning_rate = 0.1. This setting was shared for other image classification tasks. The reason why the number of epochs was set to 5 for the FSHN was the number that allows the FSHN to achieve better performance for MNIST than the KCNet.
Analysis of the DOA In this section, we explain the limitation of the DOA and how the limit resulted in the invention of the Ensemble DOA. To analyze the effectiveness of applying the DOA to the KCNet, we tested our model and optimization method using MNIST dataset. It was not appropriate to analyze the performance of the DOA using the odor-perception datasets because, as we mentioned, the number of data samples for odor perception was not enough (only 960 data samples), and the dataset had the inherited characteristic of imbalance. In contrast, the MNIST dataset is large enough and has a balanced distribution of all classes.   Figure 6: Results of the KCNet the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for Musky/Non-musky This corresponded to the substantial increase of test accuracy in Fig. 7g. However, the larger the weight matrix of the KCNet, the more perturbed the convergence of validation accuracy and the more sensitive the KCNet to the learning rate. This gave rise to a marginal improvement of the DOAapplied KCNet as its weight matrix is too large to properly deal with the complexity of exploring a set of inputs in Fig. 7g. Furthermore, the larger the weight matrix, the more burdensome the computation of the backward pass of the DOA.
Thus, to reduce the computational burden of the backward pass and to exploit the full power of the DOA, we devised the Ensemble DOA, the ensemble method of assembling a weight matrix by concatenating several smaller matrices after applying the DOA to each small matrix. Figure 7h demonstrates the running time, including the model configuration time, training time, and evaluation time, of the KCNet, the DOA-applied KCNet. The running time refers to the time spent running on a single CPU and RAM because we did not want to include any redundant time such as the time to exchange the data between CPU and GPU. As shown in Fig. 7h, the running time of the Ensemble-DOA-applied KCNet is much faster than the DOA-applied KCNet, and the application of the Ensemble DOA allows the KCNet to achieve better performance than the DOA-applied KCNet.
The results can be shown in Tables 1 and 2 for all tasks.
Results Figure 3b in the main text shows the running time, including model configuration, training, and evaluation time of the KCNet, ELM, and FSHN using a single CPU and RAM only (i.e., not making use of a GPU). This is because we wanted to exclude any redundant time such as the data transmission time between GPU and GPU. We observed that it had a tendency that the DOA couldn't improve the KCNet if its weight matrix was relatively large. As shown in Fig. 9c indicating the change of validation accuracy while applying the DOA to the KCNet, the validation accuracy gradually decreased as the DOA progressed. The difference between the weight matrix before (Fig. 9b) and after the DOA (Fig. 9d) was very marginal, which might be the cause of degrading the performance of the DOA-applied KCNet compared to the original KCNet shown in Table 2. However, the validation accuracy of 10 submodels during the Ensemble DOA in Fig. 9e was converged well, and Fig. 9f represents a noticeable improvement of the weight matrix. Therefore, we were able to validate the hyperparameter setting for the Ensemble-DOA-applied KCNet based on the increasing trend of its submodels' validation accuracy, which was applied for other image benchmark datasets. The best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for MNIST as follows: • KCNet: lambda = 13 • DOA-applied KCNet: n_epoch = 5, learning_rate = 1e-4, stop_metric = 0.96 • Ensemble-DOA-applied KCNet : n_epoch = 100, learning_rate = 0.01, stop_metric = 0.87 The additional experimental results for Fashion-MNIST (Fig. 10) and EMNIST-Balanced (Fig. 11) also tend to be similar to those for MNIST. We specify the best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for Fashion-MNIST as follows: • KCNet: lambda = 5 • DOA-applied KCNet: n_epoch = 5, learning_rate = 1e-5, stop_metric = 0.90 • Ensemble-DOA-applied KCNet: n_epoch = 20, learning_rate = 5e-5, stop_metric = 0.84 The best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the Ensemble-DOAapplied KCNet for EMNIST-Balanced are as follows:

A.5 Limitations
We showed the potential for our model through the experimental results. However, there are several limitations that we should handle in the future.
In our experiments, we trained and tested our model in supervised batch learning, which has the limitation of scaling the size of the network. This problem has been dealt with in NNRWs, and there are several works about adapting the output weight using online learning [38,51], which is more practical for large datasets. Online learning also is more congruent with the biological systems that inspired KCNet.
Vulnerability to noisy inputs is the second limit of our model. The main motivation of the KCNet is the transformation of odor stimuli between glomeruli in the antennal lobe and KCs in the mushroom bodies. So our current approach implicitly assumes that, as in the pre-processing of odors in the actual insect's brain, all input features are already pre-processed sufficiently. The image benchmark datasets that we tested satisfied this assumption by applying standard normalization to them, but these conditions are often not applicable to datasets of real-world problems. For example, we tested the KCNet on CIFAR10, an image benchmark dataset with more input features due to the large input and three RGB channels. However, KCNet achieved about 51% accuracy because the feature dimensions are relatively large, and the purpose of the KCNet is not to distinguish between critical and background features in the images. It instead is a model that generates unique latent representations with combinations of input values. So, directly feeding the real-world images as input into the KCNet may not be a good strategy to take advantage of KCNet's strengths. Just as the central complex (CX) in the insect's brain is responsible for integrating and elaborating visual information, we anticipate combining an additional feature extractor as an artificial CX, such as pre-trained models, with the KCNet would solve this problem. The KCNet can be a good ad-hoc model for other models, and it will show excellent performance in real-world image tasks if it receives sanitized features processed by a pre-trained model or a simple convolutional network.