{'fraction_non_alphanumeric': 0.017892124853075616, 'fraction_numerical': 0.004113882721692569, 'mean_word_length': 4.947572815533981, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fuzzy Based Diagnostics System for Identifying Network Traffic Flow Anomalies

This paper discusses the usage of network traffic properties in passive network monitoring which are used in recognizing and identifying anomaly.


Introduction
The idea of developing a fuzzy based diagnostics for identifying network traffic anomalies in a network operation has been a vital step to overcome problems as well as to serve as an aid in the field of network monitoring. The need for proper network monitoring tools is essential to simulate worst case or stressed network performance scenarios in order to assist in network capacity planning.
Early experiments with traffic measurement tools such as NetMon [1] and Spade [2] developed for monitoring network traffic, and MRTG [3], a popular tool used by many network operators for network traffic measurement, confirmed some similarities which had some drawbacks. Some of the known features of these tools which are also part of the reporting engine are namely to provide visual data and/or create log files which require tedious inspection of the log files. The other constraint would include indication of false alarm and/or excessive notification of an anomaly at a particular timestamp. Lastly, all the alarming and reporting facilities have a large monolithic application which is difficult to manage and configure.
The steps involved in the process of network traffic inspection are to classify and identify precise characterization of anomalous network traffic behavior hence representing it in a set of rules which make up the heart of Fuzzy System. The main benefit of this system is to reduce the volume of data that the network administrators need to analyze, thus enabling them to spend more time on tasks that require higher skill levels, such as diagnosing the cause of the problem, and fixing the fault [16].
Three classified major anomalous traffic behaviors are network operation anomalies, network abuse anomalies and flash crowd anomalies [4]. This paper focuses mainly on developing a tool to monitor data network passively in the field of network operation anomalies which include plateau behavior, network device outages and significant differences due to configuration changes. Flash crowd behavior [5] has also been looked into. The main aim of this research is to experimentally reduce some of the possible constraints of the current existing tools by introducing a fuzzy based diagnostics system in for identifying network traffic anomalies.

The Main Structure of the System
The main architecture of the fuzzy system is featured in Figure 1. This is a general overview of the whole system. All the fuzzy set operations depicted in the shaded boxes are defined in the following subsections of this paper.  Referring to Figure 2, the system consists of two major parts which are Fuzzy Logic and Neural Network. The former is used in the decision making processes and the latter for learning processes. The implementation of both parts mentioned forms a hybrid system which has the capability of learning, adaptation and identification. However, in this paper, the focus is only given on constructing a knowledge-based Fuzzy system by manipulating the data packet in order to identify anomalies. The characteristics of the fuzzy system application in this case would be as customary, with decoded data packet as inputs.
The system consists of two modes, the 'Survey' mode and the 'Ready to Alert' mode. Each mode plays its respective function as stated in Figure 2. The functions are important since different data network conveys different pattern in packet movements in the wire. Likewise, the same data network with different segment has different characteristics. Each network has its respective peculiar network traffic curve that does not change significantly over the time [13]. In this case, Neural Network has been implemented as a tuner for fuzzy system.

Inputs from network adapter
The Packet Capture Driver is manipulated in order to have the capability to capture raw data packets from Ethernet/IEEE 802.3 technologies [6]. It sniffs all the packets in the wire, not just the packets [7] which are intended for the particular node. The received frames are decapsulated and are filtered in accord with the parameters to be set for the system.

The system variables and fuzzy parameters
By capturing the essentials of the problem, the design of the process is constructed, leaving aside all the factors that could be arbitrary. In general, the simpler parameters which are kept contribute to the more understandable behavior of the system and will be more efficient in the sense of computation power consumption [8], [9], [10], [11] .In this paper, much attention is given to the aspects of selecting a suitable fuzzy set operation and tuning it, taking into consideration the reliability of the computed result.
In this system, Takagi and Sugeno's fuzzy model [10] has been implemented. This fuzzy model can be formulated as the following form:  used for the four modules stated in Figure 3. Since time is an invariant element that plays an important role in gaining pattern, it is used against type of packets, number of packets and size of packets captured respectively for a fixed timestamp.

Figure 4 -The input parameters for data network traffic diagnostics
Out of the different fuzzy variable membership functions, the type used for this research is the triangular function. Triangular functions are used because piecewise linear functions are easy to handle with computer in the sense of storage and computations [10].
The crisp input for the first two modules represent the number of packets according to type transmitted at a particular timestamp. For the purpose of monitoring, the Internetwork Packet (IP packets) and Internetwork Packet Exchange (IPX packets) which are the majority type of packets transmitted through the Netware server in the School Of Computer Science, USM (NRG Lab) are experimented with.
NetWare is used in the lab to provide services such as transparent remote file access and distributed network services namely printer sharing [13]. Hence, it is important to watch over the IPX type of packets. The advantage about Netware is that, it is designed with IPX RIP and IPX SAP which broadcasts every 60 second for updates.
For the case of IP packets, it is used to encapsulate different type of packets such as UDP, TCP, ICMP, OSPF and IGMP. It captures a broader perspective while retaining its unique pattern for a particular data network segment.
Modules for Network Utilization and Bytes per Sec. are constructed based on the standardized antecedent parameters as in Figure 4. For rules construction, data collected from the selected network segments are investigated for patterns. The data concerning abnormal network traffic flow behaviors are analyzed thoroughly. Utilizing the available data together with knowledge expertise, the finer rules are developed producing a precise alert type with minimum overhead.

Linguistic rules and evaluation
The control rules are defined using the linguistic terms associated with fuzzy sets that appear in the fuzzy partitions of the domains. Figure 5   Accurate characterizing of important classes of anomalies greatly facilitates their identification which depends on robust and timely data [13]. Some of the current best practices for identifying and diagnosing traffic anomalies consist mainly of visualizing traffic from different perspectives and from prior experience [1], [2], [3]. In general, automating the anomaly identification process has been difficult in the sense of generating precise alerts to facilitate a respective abnormal condition. There has been vast amount of research on identifying network traffic anomalies utilizing tools such as Integrated Measurement Analysis Platform for Internet Traffic (IMAPIT) which has been able to provide substantial information to detect anomalies [13]. IMAPIT includes signal analysis utility which enables network traffic data to be decomposed into its frequency components using wavelet and framelet systems. Wavelet has been used to provide means for isolating characteristics of signals via a combined time-frequency representation. This is mainly to determine isolation of short and long-lived traffic anomalies. Deviation score [13] has been used to effectively isolate anomalies and has been generalized for threshold based alerts. However, less focus has been given on the alert modes, methods of investigating and facilitating various networks.
In order to overcome this, methods of embedding intelligence in network diagnosis tool has been experimented with in this research. Fuzzy logic has been implemented to accommodate this purpose by constructing rules derived from daily and weekly traffic cycle data.

FL based Diagnostics
Two main groups of anomalies separated based on observed duration are long-lived and short-lived event [13]. The first group consists of flash crowd events which are long-lived event. Example of flash crowd event is the occurrence of heavy traffic due to services demand in the most extreme.
Referring to the Figure 6, the shaded area refers to flash crowd behavior of a segment in USM network due to a software release. During the flash crowd event it should be noted that both the average of packet size increment and the usage of module Byte Per Sec. enables easy detection. The height of peak of the graph in Figure 6 determines the appropriate decision to be taken based on initial rules stated in Figure 5. The four decisions in this case are ignore under normal circumstances, log when the peak deviates slightly higher than normal, email when the deviation of the peak is more and Short Message Services (SMS) when deviation is acutely high. For the case of Figure 6, the decision of periodic email would be taken. Module. Figure 7 shows the comparison between the normal and abnormal network traffic flow in the NRG segment. The abnormality of the flow was monitored on a different day in which power failure had occurred. This is an example of the short lived anomaly which was detected. Taking into consideration of the scenario which had a sudden and rapid fluctuation of the spike, the typical response was to alert via SMS. In the case of other factors contributing towards the abnormality of the flow, the pattern of the graph will show deviation from the standard pattern. This is clearly shown in Figure 8 whereby a network device, for instance a hub, caused a network segment to go down.  Another example of a different short-lived anomaly caused by loss of the router's connectivity is shown in Figure 9. In this instance, the response of alert chosen was to SMS due to the reason that the pattern showed extremely low in packet count.

Figure 9 -Failure of a router detected based on packet count
The figure below shows the packet count obtained upon filtering IPX packet from Novell Netware operating system. The graphs show a comparison between the normal and abnormal pattern. The deviation of the abnormal pattern was caused by hardware failure in this case a network adapter.

Discussion
Optimizing with Neural Network The fuzzy system is unable to completely function intelligently without a tuner of the parameter. Taking this into consideration, there has been means to tune the system ad hoc which is time consuming. As a step to overcome this problem, Neural Network has been introduced into the system to act as an intelligent tuner [14]. Neural network modifies the membership function when the survey mode is activated. It serves as an important function in producing the '∆x', displacement of the vertex of fuzzy set (with the highest fuzzy strength) for the membership function in order to obtain a better control of the alerts which act as consequent parameter. By this process, it has the ability to learn a unique pattern of a particular network segment and alert accordingly.
There have been researches involving ways of optimizing neural network to define universe of discourse along with adding controlling rules for best accuracy in selecting the suitable alert type.

Future Work
The action of the system currently works based on a simple counter in order to avoid spamming or creating large number of alerts for a particular anomaly. In order to overcome this, the research is looking into means to stabilize the action of the alerts. A proposed method for this is a function which works as a feedback fuzzy controller system [15]. This will not only lead to controlling the type of action to be taken but is also able to control the number of the action taken.
Another alternative way to tackle the problem would be to exchange the initial control rules.
The other aspect which can be explored further is to enhance the readability of anomalies by applying a variety of time-frequency analysis techniques, particularly wavelet and deviation score. These techniques consider signal variation in both high and medium frequency bands and are found to be extremely effective at isolating anomalies [13]. Utilization of techniques as such enables the achievement of two goals. One, a finer characteristics of anomalies which leads to accurate type of action and secondly, informative alerts can be generated for more efficient troubleshooting to minimize network downtime.

Conclusion
In this research, aspects compromising of a Fuzzy based system together with Neural Network has been implemented to develop a tool which takes action intelligently. All the fuzzy operators which are fixed in order for the system to be optimized and reducing false alerts have been discussed in detail. Behavioral learning of the patterns by neural network has been explored as well to obtain more accurate alerts. The results obtained shows that number of anomalies can be detected particularly network operation anomalies. Filters at the level of network layer are implemented in order to scrutinize in accord to a particular type of packet. In this case, the IPX and IP type of packets are looked into and discussed in elaborate.
Some constraints evidently arise during this research. Due to uncontrollable network operations, some difficulty was faced when taking the statistical reading of the network traffic flow data. The other constraint arises during the process of defining a standard pattern for a particular network segment. Measures to tackle this problem were done by obtaining reading from various network segments rather than from one particular segment. Since there lacks a standardized definition for network anomalies, hence the process of characterizing the anomaly is carried out in a purely empirical approach. The proper advancement of tool for pattern gathering will give a breakthrough in the area of anomaly identification.
Finally the exploration of ideas in this paper augments the existing threshold based alerts. It is hoped that a system incorporating intelligence in software based tools is developed to help administrators [17]. Since data network has become a vital aspect in the era of information technology, the downtime of a network should be minimized. In this sense, these tools are helpful enough to passively notify network administrator regarding the health of a network from time to time.