{'fraction_non_alphanumeric': 0.027091955663384234, 'fraction_numerical': 0.019718376861234004, 'mean_word_length': 4.86373779637378, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explainable and Optimally Configured Artificial Neural Networks for Attack Detection in Smart Homes

In recent years cybersecurity has become a major concern in adaptation of smart applications. Specially, in smart homes where a large number of IoT devices are used having a secure and trusted mechanisms can provide peace of mind for users. Accurate detection of cyber attacks is crucial, however precise identification of the type of attacks plays a huge role if devising the countermeasure for protecting the system. Artificial Neural Networks (ANN) have provided promising results for detecting any security attacks for smart applications. However, due to complex nature of the model used for this technique it is not easy for normal users to trust ANN based security solutions. Also, selection of right hyperparameters for ANN architecture plays a crucial role in the accurate detection of security attacks, especially when it come to identifying the subcategories of attacks. In this paper, we propose a model that considers both the issues of explainability of ANN model and the hyperparameter selection for this approach to be easily trusted and adapted by users of smart home applications. Also, our approach considers a subset of the dataset for optimal selection of hyperparamters to reduce the overhead of the process of ANN architecture design. Distinctively this paper focuses on configuration, performance and evaluation of ANN architecture for identification of five categorical attacks and nine subcategorical attacks. Using a very recent IoT dataset our approach showed high performance for intrusion detection with 99.9%, 99.7%, and 97.7% accuracy for Binary, Category, and Subcategory level classification of attacks.

: Smart home scenario

Introduction
The Internet of things (IoT) has been recognised as a disruptive technology with a huge impact in our everyday life. Some widely used applications are smart home, smart health, smart city, smart grid, smart agriculture, smart cars, logistic tracking to name a few [1]. IoT devices are connected via the Internet that collect and exchange data with each other for a particular purpose [2]. Due to a huge demand for IoT based applications, the IoT devices are continuously evolving in terms of low cost, easy handling, small form factor and sensor precision. However, security and privacy has not been given a priority while designing these devices [3]. Therefore a huge amount of research has focused on securing IoT based applications; however, a lot more needs to be done to protect IoT networks from intruders.
Smart home is one of the applications that has played an important role in massive growth of IoT usage. Smart home allows individuals to fully connect their home appliances/devices with each other and also enable the owner to control the devices remotely, as shown in Figure 1. That provides unprecedented control and comfort. But this benefit comes with a huge security risk, if adequate security measures are not taken into account. In a smart home environment, one of the biggest challenges is in terms of users inability to understand, appreciate and take security precautions. Most IoT devices used in smart homes are designed with an extended set of sensors and actuators. For example, smart light may come with a microphone that may leave the IoT system vulnerable to security and privacy intrusions. A common household resident may not have knowledge and skills to interpret and safeguard the IoT system [4]. Hence, there is a huge need for an automated mechanism to protect smart homes from security and privacy intrusions.
In recent years, many research efforts are focused on automated intrusion detection systems using machine learning. The essence of a machine learning based system is to train the activity model based on the signature, pattern or characteristics of attacks which will be compared against the new traffic behaviour. A number of different supervised and unsupervised machine learning algorithms such as random forest, decision tree, support vector machine, long short-term memory (a type of neuron network), Artificial Neural Networks (ANN) could be used for this purpose. ANN is one of the machine learning algorithms that has been widely used for classification tasks across many domains, due to its black box nature of adapting to the underlying system [5]. This property makes ANN highly suitable for network traffic intrusion detection when a high dimensional data set is involved due to its robust ability of model building when the pattern is buried [6]. A very important aspect of ANN's high performance lies in optimal selection of its hyperparameters for that dataset [7]. Considerable effort is required for finding the optimal setup, which is a non-trivial task as it not only affects performance but can be a very time consuming exercise due to availability of a large number of options for each parameter involved in the setup.
In this paper we propose a novel multi-layer ANN based intrusion detection approach for smart homes. This approach is capable of identifying network attacks at sub-categorical level, which has not been explored extensively in the existing literature. The strength of the approach is its dynamic ability to fine tune ANN hyperparameters to detect attacks with high precision. We would like to emphasise that this selection of hyper parameters is based on detection accuracy, computational overhead and time constraint. The accuracy of prediction is important as missing any attack in case of wrong detection can be very damaging. Computation overhead plays an important role as finding the most optimal solution using extensive computational resources may not be feasible and affordable. Time is one of the constraints as such detection has to be real time and any model that takes a long time to train or detect is not suitable for intrusion detection in this scenario. We also evaluate the selection of the hyperparameters to analyse efficiency of the proposed model. An important thing to mention is that we evaluate our approach on a new IoT dataset, IoTID20 [8] as the dataset shows recent network traffic characteristics (collected in 2020).
The main contribution of this paper is: • We are providing an optimised ANN model with low overhead.
• Our approach shows high performance for security attack detection at Subcategory level for a recently captured IoT dataset which shows applicability of the approach for detection of new attacks.
• We analyse the explainability of the ANN model which can be helpful for user understanding and trust building.
The paper is organised as follows: section 2 discusses related literature in ANN based intrusion detection system and also the research carried out using IOTID20 dataset. Section 3 provides complete details of our proposed multi tiered ANN model for intrusion detection. The hyper parameter selection is given in section 4. Section 5 provides the categorisation results and analysis for intrusion detection purpose. We conclude our paper in section 6 and identify future work options.

Related Work
There has been a huge amount of research actively going on in the area of Cyber Security where machine learning algorithms are used for anomaly and attack detection. We provide a brief review of some of the literature that is relevant to our work in the following subsections.

ANN Based Intrusion Detection
As mentioned above, machine learning based anomaly and intrusion detection is an active research area; some of the commonly used algorithms are Random Forest (RF),K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Multi-Layered Perceptions (MLP), Artificial neural network (ANN) etc. [9]. Among all these different machine learning algorithms, ANN has been considered as one of the most efficient algorithms for anomaly or intrusion detection [10] [7].
Taher et al. [11] compared the performance of SVM and ANN in detection network intrusion and found the ANN performs better than SVM. Authors used NSL-KDD data-set in this work.
Choras et al. [7] proposed an ANN based system for intrusion detection for multi-class classification. In this work, the authors tried various possible options of different hyperparameters to come up with the most efficient system. The authors argue that selection of hyperparameters plays a very important role in classification results and a little change in hyperparameter setup has a major impact on accuracy. The hyperparameters that were fine tuned in this work were activation function, the optimiser, number of epochs, number of neurons and batch size. The performance of the proposed model was evaluated using two popular datasets (NSL-KDD and CICIDS2017). Experimental results show that the most effective model achieves an accuracy of 99.909% for multi-class attack detection.
Hodo et al. [12] presented an ANN based network intrusion detection system for IoT networks. This work focused on binary classification -detecting normal and attack traffic (DDoS/DoS). The performance of the proposed system was tested in a simulated IoT network and the result shows 99.4% accuracy in detecting Distributed Denial of Service (DDoS) and Denial of Service (DoS) attacks.
Subba et al. [13] proposed an ANN based intrusion detection system that could be easily deployed for real-time intrusion detection. The authors argued that some of the existing machine learning based systems may achieve high performance in intrusion detection, but suffer from high complexity and computational overhead. So in this work, authors proposed a three layer ANN model (with one hidden layer) to minimize the computational overhead for training and execution of the system. The performance of the model has been analysed using NSL-KDD dataset. The results from performance evaluation show that this proposed system performs better than Naive Bayes based model and similar as SVM and C4.5 based IDS models.
From the above discussion, it is apparent that ANN is a good choice for intrusion detection purpose but faces a challenge of being complex for optimal selection of hyperparameters.

IoT Dataset
When machine learning algorithms are used for attack detection then datasets become very crucial as training and testing of these algorithms cannot be carried out without a dataset. Recently, a number of datasets have been used by researchers for this purpose, like UNSW-NB15 [14] and Bot-IoT [15]. We have chosen the IoTID20 dataset for training and testing our ANN model as this is one of the newest dataset collected in the IoT environment and it may depict more realistic and up to date network traffic characteristics. Another important aspect to mention is that as this dataset was collected and made available for use very recently, there are a handful of research utilising this dataset for experimentation. Moreover, none of the existing research (until June 2021) was able to achieve Subcategory classification accuracy of more than 90% for this dataset. We have considered this dataset to analyse our IoT intrusion detection model as there is a scope of improvement. Ullah and Mahmoud [8] collected and experimented with this dataset while considering a number of machine learning approaches. Anomalous activities are detected at binary, category and subcategory level. A number of machine learning algorithms (SVM, Gaussian NB, LDA, Logistic Regression, Decision Tree, and Random Forest) and an ensemble method are used for IoT attack detection with varying results. For binary and category classification Decision Tree, Random Forest and Ensemble methods produced very good results with F-Score of more than 95% in most cases. However, when it comes to subcategory classification none of the algorithms was able to get F-Score and Accuracy more than 88%. In security scenarios, getting high accuracy for attack detection is highly recommended and the intrusion detection system with 88% accuracy may not be relied upon completely. Hence, it can be deduced that there is a room for improvement to the machine learning model in order to achieve high accuracy and precision for subcategory based attack detection using this dataset.
The importance of feature selection has been analysed for anomaly based intrusion detection approaches and the IoTID20 dataset is used for testing the hybrid feature selection approach using Random Forest algorithm [16]. The authors only discussed partial results for three attack categories indicating more than 99.9% accuracy. The accuracy for detection of some attack categories is very high which shows the strength of the hybrid feature selection approach that only uses a subset of features which are common among the features selected using Information Gain and Gain Ration methods. However, the applicability of such feature selection approach cannot be justified unless complete results are provided and analysed for all category and subcategory based attacks.
Qaddoura et al. [17] have proposed a multistage approach for anomaly detection for imbalanced datasets like IoTID20. The authors used approaches like oversampling and K-means++ clustering in order to mitigate the impact of imbalance among the examples for different categories. The results showed good accuracy for binary classification only. The classification of categories and sub categories was not undertaken by this research work and hence, the effectiveness of this approach for multi-class classification in the IoTID20 dataset cannot be assessed. Similarly, Anjum [18] has discussed cross dataset evaluation considering IoTID20 and Bot-IoT datasets. However, all the attack detection was only for binary or category level and subcategory level classification was not considered by the author.

Multi-tiered ANN Model for Intrusion Detection (MAMID)
In this section we describe our system that uses ANN for intrusion detection by selecting hyperparameters for high prediction accuracy, low computational overhead and real time detection, the proposed model is shown in Figure 2. The dataset is preprocessed in order to eliminate any redundant data and missing values. After the preprocessing, feature selection is performed in order to find the minimum set of features that can be effectively used for classification. The data is then passed to the ANN optimiser that finds the optimal neural network topology and hyperparameters for configuring the neural network for training and testing. Not only the prediction accuracy but also the computational overhead and time complexity is considered in order to apply this approach to smart home scenario that requires real time detection and may not have access to high end servers. The following subsections discuss the main components of our system in detail.

ANN
The ANN model consists of three types of layers [19], input layer, hidden layer and output layer, as shown in Fig. 3. In each layer, there are many nodes called neurons. The input layer contains the input data feeding into the model. The number of neurons is the same as the number of input features. Each neuron

Input layer
Hidden layers Output layer Figure 3: A structure of a classic artificial neural network, including one input layer, multiple hidden layers, and one output layer from the input layer interconnects with one or more neurons in the second layer with a weight. The output of the second layer will go to the next layer until the output layer. Given a three-layer ANN, the weights and a bias term between the input layer and the hidden layer are W 1 (x 1 , x 2 , x 3 ) and b 1 , respectively. The inner product between the weight matrix and the input data is transformed by an activation function g(x) (e.g. the Sigmoid activation function in Eq. 2). The mapped values are fed into the next layer until the output layer. For binary classification, there is only one neurons in the output layer. The feed-forwarded process can be formulated as given in Eq. (1).
where g(·) is a Sigmoid activation function, it can also be Softmax, Tanh, ReLU activation functions etc, W 1 and W 2 are weight matrices for the input and hidden layers, respectively, and b 1 and b 2 are bias terms added to the hidden and output layers, respectively.

ANN Optimiser
The optimisation of ANN setup is the most important aspect when it comes to the accuracy of the prediction and the computational overhead involved in the process [7]. With the correct selection of topology and hyperparameters, the model can achieve the highest accuracy of the ANN classifiers. Figure 4 shows hyperparameter selection process by ANN optimiser. A general description of the hyperparameters is given below. The actual options for each hyperparameter that we experimented with are given in Section 4. Optimisation methods are crucial when it comes to the effective training of the neural networks. Optimisation methods are used to reduce losses and hence, improving performance by changing weights and the learning rate of Figure 4: ANN Optimiser the neural network. Finding the optimisation method suitable for the dataset can play a vital role when it comes to prediction accuracy. Some commonly used optimisers are gradient descent, stochastic gradient descent, momentum, nesterov accelerated gradient and adaptive moment estimation (ADAM). Every optimiser has its own strengths and weaknesses, for example, gradient descent is computationally inexpensive but requires large memory. On the other hand, ADAM converges rapidly but is computationally expensive [20].
Epoch is when a complete training data has once passed through the neural network and updated the internal model parameters. Most of the time all samples of the training data cannot pass in one go and hence, each epoch is divided into several batches. Single epoch may result in underfitting and multiple epochs are required as the internal parameters need to be changed multiple times for better generalisation. Once the number of epochs starts increasing the model starts fitting better until it reaches optimal fitting and if the number of epochs are still increased then the overfitting can happen [21].
Batch size defines the number of samples that ANN algorithm must walk through before updating internal parameters. Batch size can affect the performance of ANN algorithm in terms of prediction accuracy and training time. In general, smaller batch size is better in generalising to the test set than large batch training [22].
Activation function lets the neural network learn complex patterns in the data by adding non linearity into the model. Various non linear activation functions are available such as Sigmoid, Softmax, Tanh, ReLU etc. Choice of activation function can impact the performance of the training process [23]. The activation function for input layer, hidden layer and output layer may be different based on the dataset.
Based on the domain knowledge, resource restrictions and dataset characteristics, a number of hyperparameter options are chosen. ANN optimiser tests the prediction accuracy of the neural network using these optimisers with a small subset of data to find the most suitable set of options.

Explainability of our Model
Advanced predictive models like ANN achieve good performance by using complex models which are difficult to explain. However, assessing how a particular prediction is made is vital for understanding and trusting the model by the end user. We assess the explainability of our model using SHapley Additive exPlanations (SHAP) by identifying feature relevance on the basis of each sample [24]. SHAP explains what the model is doing by finding the important relationships between the input features and the predicted outcome, which allows the users to interpret the predictions of our ANN model. SHAP uses an explainer method to combine the inputs to evaluate the effects on the predictive model and produces a locally interpretable model for every prediction [24]. The explainer illustrates the SHAP value for each feature that illustrates the importance of each feature for particular prediction.
After training the ANN model with a training subset of dataset, SHAP explainer analyses the testing subset. The explainer generates the output of the ANN model and assigns a SHAP value to each feature for prediction. These SHAP values indicate which feature increases or decreases the influence on the prediction to belong to a particular class [25]. The summary plot is used to visualise the effects of the indicated features on prediction of the attack subcategory. We have used the force plot to show how features effect the prediction for a single value.

Hyperparameter Selection by ANN Optimiser
In this section we discuss the hyperparameter selection by ANN optimiser for IoTID20 dataset for attack detection at binary, category and subcategory level.

ANN Setup
The optimal ANN setup is found by using a grid search, which completes an all-encompassing search over the hyperparameter's space. The grid search parameters included: • Number of epochs It is worth mentioning that the experiments were conducted with one, two, or three hidden layers. The results were not significantly different. Therefore, in this study, only one hidden layer was chosen for hyperparameter selections.
The full cycle of learning and adapting the weights of the network is called epoch. In this model, 100 epochs and 200 epochs were used in the grid search. The particular count of samples utilised in one iteration is called batch size. 10 and 100 batches were used in this grid search. The units or nodes in the input and hidden layers are called neurons. In this model, 100 neurons and 200 neurons were used in the grid search.
The optimiser functions evaluated in this model are: • Adaptive Moment Estimation (Adam) -a stochastic optimisation method that uses a randomly selected data subset to create a stochastic approximation, instead of using the entire data set to calculate the actual gradient.
• Root Mean Square Propagation (rmsprop) -an extension of gradient descent and the AdaGrad version of gradient descent that uses a decaying average of partial gradients in the adaptation of the step size for each parameter.
• Stochastic gradient descent (SGD) -an iterative method for optimising an objective function with suitable smoothness properties. It can be regarded as a stochastic approximation of gradient descent optimisation, since it replaces the actual gradient by an estimate thereof.
• AdaMax (AdaMax) -an optimisation method that implements the AdaMax algorithm. It is a variant of Adam based on the infinity norm.
• AdaGrad (Adagrad) -a stochastic optimisation method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring features, and larger updates for parameters associated with infrequently occurring features.
In the input and hidden layer, and in the output layer, the activation functions were evaluated independently. The Activation functions evaluated in this model are: • Rectified Linear Unit (ReLU) -the range of the activation value is from 0 to infinity.
• Hyperbolic Tangent (tanh) -the range of the activation value is from -1 to 1.
• Sigmoid (sigmoid) -the range of the activation value is from 0 to 1.
• Softplus (softplus) -it is the softer (smoother) version of ReLU and softplus. The range of the activation value is from 0 to infinity.
• Softmax (softmax) -it is used for multiclass classification by converting a vector of values to a probability distribution. The elements of the output vector are in range (0, 1) and sum to 1.
The evaluations were conducted for three types of classifications: • Binary classification of normal and attack classes • Category level attack classification • Subcategory level attack classification For each classification model, it required 1,000 experiments to evaluate all the pre-defined hyperparameter options. That is Epochs (2) × Batch (2) × Neurons (2) × optimiser (5) × Activation I (5) × Activation II (5) = 2 × 2 × 2 × 5 × 5 × 5 = 1000 experiments. The aim of the study is to show that the results of subset dataset can be effectively used to classify the full dataset. Therefore, this study required 6,000 experiments to include subset and full dataset experiments. The first 3,000 experiments were conducted over a subset of the full dataset to find the optimal hyperparameter selection. and the second 3,000 experiments were conducted to validate the optimal selection with the full dataset of 625,783 records.

Dataset
In this work we used IOTID20 dataset [8] as this dataset was collected in smart home environment which consists of multiple smart home devices. The

Performance Metrics
In this work, the accuracy, precision, recall, F1-score, and support are used as performance measures. It is worth noting that the macro and weighted versions of the measures are considered for imbalanced multi-class classification. The aim is to have higher accuracy, precision, recall, F1-score. The support is the number of true instances for each class.
where n class is the number of classes, T P i , T N , F P and F N are the number of true positives of the ith class, the number of true negatives of the ith class, the number of false positives of the ith class and the number of false negatives of the ith class, respectively.

Optimal Hyperparameter Selection using Subset
Because of the mismatch data structure between the data and optimiser activation functions, some experiments were not successful. The analysis is based on the successful experiments over the subset with 10,000 records. 464 out of 1000 binary experiments, 474 out of 1000 category experiments, and 437 out of 1000 subcategory experiments returned successful results. The results over three different classifications were distributed on scatter plots in Figure 5. By observing this figure, it is easy to see that, with the increasing number of target types, the accuracy rates become lower.
To understand the distribution of the high accuracy rates over 90%, the summary of the results using the predefined hyperparameters for binary classification, category classification, and subcategory classification are listed in Table  1. As we can see from the table, it is much more challenging to reach 99% with the number of prediction targets increases in the experiments. 57% of binary experiments, 0.5% of category experiments, and 0% subcategory experiments reached over 99% accuracy rates.
To select the optimal hyperparameters, two approaches were taken in this study. The first one is to investigate the hyperparameter settings from the experiments with highest accuracy results (Approach I). The second approach is to investigate each hyperparameter setting together with its accuracy levels from all experiments (Approach II). Therefore, Approach I is to investigate the top 10 results of three classifications (Table 2). Table 2 shows the range of top 10 accuracy for binary is between 99.82-99.88%, Category is between 98.89-99.04%, and Subcategory is between 95.49-95.65%. It is obvious to see the optimal options for Optimiser, Activation I, and Activation II. The optimal optimiser for top 10 classification results is adam, which ran 21 (7+6+6) out of 30 experiments. The optimal Activation I function for top 10 classification results is tanh, which conducted 24 (10+7+7) out of 30 experiments. The optimal Activation II function for top 10 classification results is softmax, which was used in 22 (2+10+10) out of 30 experiments. However, it is difficult to identify the optimal options for the number of Neuron, Batch, and Epochs. For each hyperparameter, a similar number of either option was used for top 10 classifications.
Approach II is to investigate how well the hyperparameters were used in all experiments in terms of accuracy. Six bar charts were generated to show the accuracy levels of each hyperparameter option for three classification types in all successful experiments.
The bar charts of epochs, batch and neurons show not much difference in either hyperparameter selections as shown in Figure 6. The results from this approach are similar to Approach I. Combining the results from both approaches, the optimal selections were made. 200 was selected as the optimal number of neurons, 100 was selected as the batch size, and 200 was selected as the optimal number of epochs.
Using Approach I, adam was selected for optimiser, tanh was selected for Activation I, and softmax was selected for Activation II. This selection was supported by the results from Approach II. In Fig. 6, adam, tanh, and softmax generated the highest accuracy as optimiser, Activation I, and Activation II. The final selection of the hyperparameters is shown in (Table 3).

Validation with Full Dataset
Although the full dataset was used for validation purposes, full experiments were conducted on this dataset to support the optimal hyperparameter selection from the subset dataset. The procedures are explained in the following steps. Through the grid search, 3,000 experiments were conducted on the full dataset with 625,783 records.

Binary Classification of Normal and Attack Classes
Among the 1000 experiments based on the grid search technique, 571 results successfully returned using the predefined hyperparameters. 74% of experiments (421 out of 571) returned the results above 90% accuracy. 63% of experiments (361 out of 571) returned the accuracy above 95%. 58% of experiments 334 out of 571) returned above 99%. The accuracy results from the 459 experiments were presented as a scatter plot shown in Figure 7a. The top 10 accuracy results with detailed hyperparameter setups on this binary classification are presented in Table 4. For all top 10 results, the accuracy rates stay between 99.951% and 99.956%. It is no surprising that all the Activation II (the output layer) is sigmoid because this is a classification problem. The best accuracy is 99.9559%, with a mix of 200 neurons, AdaMax optimiser, ReLU activation function in hidden layers, batch size of 10, and 200 epochs. The activation function is ReLU in the top 5 accuracy results. The batch size of 10 is suitable for SGD optimiser while 100 for AdaMax.

Category Level Attack Classification
Using the predefined hyperparameters, the grid search returned 414 results out of 1000 experiments. 53% of experiments (221 out of 414) returned the results above 90% accuracy. 47% of experiments (193 out of 414) returned the accuracy above 95%. 25% of experiments (103 out of 414) returned the accuracy above 99%. The accuracy results from the 449 experiments were presented as a scatter plot in Fig. 7b. The top 10 accuracy results with detailed hyperparameter setups on this category classification (5 classes) are presented in Table 5. For all top 10 results, the accuracy rates stay between 99.7% and 99.72%, which are slightly worse than the binary classification (compared to Table 4). This is because with the increase of classes, it is more difficult to accurately predict the correct class. It is expected that for multi-class classification, the Activation II in the output layer is softmax activation function. The best accuracy is 99.7238%, with a mix of 100 epochs, 200 neurons, adam optimiser, tanh activation function in hidden layers and batch size of 100. Among all the optimisers, adam has the best performance in the top 10 accuracy results.

Subcategory Level Attack Classification
Using the predefined hyperparameters, the grid search returned 563 results out of 1000 experiments. 46% of experiments (259 out of 563) returned the results above 90% accuracy. 31% of experiments (174 out of 563) returned the accuracy above 95%. No experiments returned above 99%. The accuracy results from the 563 experiments were presented as a scatter plot in Fig. 7c. The top 10 accuracy results with detailed hyperparameter setups on this subcategory classification (9 classes) are presented in Table 6. For all top 10 results, the accuracy rates stay between 97.75% and 97.84%. The top 10 results are worse than the category and binary classification (compared to Tables 4 and 5) because with the increase of classes, it is more difficult to accurately predict the correct class. For multi-class classification, the Activation II in the output layer is softmax activation function. The best accuracy is 99.84%, with a mix of 200 epochs, 200 neurons, adam optimiser, sigmoid activation function in hidden layers and batch size of 100. The epoch for all the top 10 accuracy is 200. Among all the optimisers, adam has the best performance in the top 4 accuracy results.

Optimal hyperparameter selection
The accuracy results of different hyperparameter settings are compared and presented in Figure 8. The accuracy results are grouped by the classification types: Label (binary classification), Category(5-class classification), and Subcategory (9-class classification).

Number of Epochs
The accuracy rates of different predefined number of epochs are presented on a bar chart in Figure 8. Observing the results, there was almost no difference in   (Table 4), 7 out of 10 in category classification (Table 5), and 10 out of 10 in subcategory classification (Table 6).

Batch Size
The accuracy rates obtained by different batch size are presented on a bar chart in Figure 8. The results from this figure show that 100 as batch size provided high accuracy in all prediction experiments. The top 10 accuracy results listed in Tables 4 -6 also present that 100 epochs provides better results for all three classification types. There were 6 out of 10 top 10 best hyperparameter settings used 200 epochs in label classification (Table 4), 7 out of 10 in category classification (Table 5), and 7 out of 10 in subcategory classification (Table 6).

Number of Neurons
The accuracy rates by the number of neurons are presented on a bar chart  (Table 4), 6 out of 10 in category classification (Table 5), and 8 out of 10 in subcategory classification (Table 6).

Optimiser
The accuracy rates by five optimiser functions are presented on a bar chart Figure 8. By observing this figure, SGD could be the best hyperparameter option for all three classification types. In the top 10 accuracy results, AdaMax, adam, and SGD were used in the experiments. In label classification (Table 4), 3 out of top 10 accuracy results used AdaMax, 3 used adam, and 4 used SGD.

Activation for Input and Hidden Layers
The accuracy rates by five activation functions for input and hidden layers are presented on a bar chart Figure 8. No particular activation function can be observed as the best hyperparameter option for all three classification types. Relu, Tahn, and Sigmoid were selected by the top 10 accuracy results as the activation function for the input and hidden layers. In label classification (Table  4), 6 out of 10 top accuracy results used ReLU, 2 used tanh, and 2 used sigmoid. In category classification (Table 5), 4 used ReLU, 4 used tanh, and 2 used sigmoid. In sub category classification (Table 6), 3 used ReLU, 6 used tanh, and 1 used sigmoid. tanh may not be the best option for all classifications. It definitely shows the advantages in subcategory classification.

Activation for the Output Layer
The accuracy rates by five activation functions for output layers are presented on a bar chart Figure 8. As we can see from this figure, Tanh and Softplus are not the suitable algorithm for the data struction from this full dataset. Only ReLU, Sigmoid, and Softmax completed the computation and provided results. Based on this figure, Softmax could be the best hyperparameter option for the accuracy function for the output layer. By observing the top 10 accuracy results tables, softmax was used in 10 out of 10 experiments in Table 5 and Table 6. Although Sigmoid was used as the activation function for the output layer in the top 10 accuracy results for Label classification in Table 4, softmax is the optimal activation function for category and subcategory classification.

Results
In this section we provide complete experimental results for subset dataset and full dataset for all three type of classification using the optimal set of hyperparameters. Tables 7 and 8 show the binary classification based on the subset dataset and full dataset, respectively. In Table 7, the f1-scores for macro f1-score, weighted f1score, anomaly f1-score, and normal f1-score are 92.3214%, 98.313%, 99.1304%, and 85.5124%, respectively for subset of dataset. In Table 10, the f1-scores for macro f1-score, weighted f1-score, anomaly f1-score, and normal f1-score are 95.7921%, 99.0317%, 99.4988%, and 92.0855%, respectively for the full dataset. In these two tables, the results for anomaly are higher than normal because there are more data for anomaly than normal and the ANN could fit better to the majority class. In addition, the accuracy is 99.7942% for the subset while 99.9326% for the full dataset. This is expected because with more data used for training, ANN has better performance in terms of precision, recall, and f1-score. This is because there are many parameters (weights between neurons) in ANN, it can better fit the problem with more data used.   Tables 9 and 10 show the category classification based on the subset dataset and full dataset, respectively. In Table 9, the f1-scores for macro f1-score, weighted f1-score, and the f1-scores for DoS, MITM, ARP, Spoofing, Mirai, Normal and Scan are 78.9534%, 88.8707%, 99.5851%, 34.1463%, 94.0527%, 90%, and 76.9831%, respectively. In Table 10 Table 9 is 98.8722% for the subset while 99.7283% in Table 10 for the full dataset. This is because ANN could extract more hidden knowledge with more data used for training.  Tables 11 and 12 show the Subcategory classification based on the subset dataset and full dataset, respectively. As we can see from the tables, the accuracy in Table 11 is 95.4654% while 97.767% in Table 12. In Table 11 Table 12 are better than that in Table 11 is because there are many parameters (weights between neurons) in ANN, it can better fit the problem with more data used.

Subcategory Level Attack Classification
In summary, with more data used for training, ANN achieves better results based on the full dataset in Tables 8-10 reinforces the optimal hyperparameter selection from the subset in Tables 7-11.

Comparison of Our System with Other Related Work
We have compared performance of our approach with recent existing work using the same data and the summary is presented in Table 13. It is evident from the table that only Ullah and Machmoud [8] has considered subcategory level of attack classification for this dataset and the accuracy and F score is 88% with decision tree method. Most of the work that considered IoTID20 dataset did

Explainability Analysis
As discussed previously, explainability is an important factor when it comes to adaptation of complex machine learning models by end users. MAMID is designed for smart home scenario where users trust in the system is crucial for users to adapt this security system in their homes. In this section we briefly discuss the explainability of the models based on the subset dataset. The SHAP plots shown in Figure 9 show the feature importance and summary plot for binary classification based on subset dataset. In both the plots, the features are placed on the Y-Axis according to their importance.
The feature importance plot shows how each feature impacts the detection of the classes, in this case for binary classification Class 1 is normal (blue) and Class 2 is attack (red). The impact of every feature on the detection of the   attacks show that Flow Duration has the highest impact on detection of these classes.
The summary plot shows the positive and negative relationships of the predictors with the target variable. The value on the X-Axis in summary plot shows whether the effect of that value is associated with a higher or lower prediction, the color shows high (in red) or low (in blue) for that observation. The low level of the Flow Duration has a low and negative impact on the attack detection. Similarly, the feature importance plot and summary plot for category classification are shown in Figure10.
The force plot shown in Figure 11 shows the effect of different features on a particular predicted value for binary classification based on subset dataset. The given plot shows the baseline and the average predicted probability for a particular value to be classified as an attack is 0.07. The data sample has low predicted value as the increasing effect of TotLen Fwd pkts field is offset by decreasing effect of ACK Flag Cnt field.

Conclusion
Accurate detection of intrusion is very important in cybersecurity. In this study, we used ANN for intrusion detection based on hyperparameter optimisation. To examine the effectiveness of different hyperparameters for ANN, experiments were conducted, including changing the number of epochs, batch size, number of neurons, optimiser, activation for input and hidden layers, and activation for the output layer. To validate our proposed model, a dataset with three different cases was considered: (1) Label classification (binary), (2) Category classification (5 classes), and (3) Subcategory classification (9 classes). The experimental results revealed that our proposed model with hyperparameter optimisation has high performance for intrusion detection with 99.9%, 99.7%, and 97.7% accuracy for Label, Category, and Subcategory classification. Although we only tested our proposed model in intrusion detection, it can also be applied to other cybersecurity problems (e.g. malicious website domain detection) and also other areas (e.g. disease detection and anomaly detection). In the future work, we will investigate more about the ANN structure, including automated selection of the right hyperparameters to prevent over-fitting problems. We will also explore different feature selection or feature extraction options for our proposed model.