{'fraction_non_alphanumeric': 0.026650091294044522, 'fraction_numerical': 0.009556738277456199, 'mean_word_length': 4.616844861444469, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
INTELLIGENT SYSTEM FOR AUTOMATIC CLASSIFICATION OF FRUIT DEFECT USING FASTER REGION-BASED CONVOLUTIONAL NEURAL NETWORK (FASTER R-CNN)

In 2018, the Indonesian fruit exports increased by 24% from the previous year. The surge in demand for tropical fruits from non-tropical countries is one of the contributing factors for this trend. Some of these countries have strict quality requirements – the poor level of quality control of fruit is an obstacle in achieving greater export yield. This is because some exporters still use manual sorting processes performed by workers, hence the quality standard varies depending on the individual perception of the workers. Therefore, we need an intelligent system that is capable of automatic sorting according to the standard set. In this research, we propose a system that can classify fruit defects automatically. Faster RCNN (FRCNN) architecture proposed as a solution to detect the level of defect on the surface of the fruit. There are three types of fruit that we research, its mangoes (sweet fragrant), lime, and pitaya fruit. Each fruit divided into three categories (i) Super, (ii) middle, (iii) and fruit defects. We exploit join detection and video tracking to calculate and determine the quality of fruit in real-time. The datasets are taken in the field, then trained using the FRCNN Framework using the tensor flow platform. We demonstrated that this system can classify fruit with an accuracy level of 88% (mango), 83% (lime), and 99% (pitaya), with an average computation cost of 0.0131 m/s. We can track and calculate fruit sequentially without using additional sensors and check the defect rate on fruit using the video streaming camera more accurately and with greater ease.


INTRODUCTION
The fruit is widely cultivated in Indonesia. National fruit production has significantly increased in the past years. In 2016, national fruit production amounted to 17.711,548 tons, which subsequently increased by 7.4% to 19.021,099 tons in 2017. This growth causes Indonesia to become one of the top tropical fruit exporters in several export destinations. in 2017, Indonesia exported 33.68 thousand tons of fruit worth of 19.95 million dollars in the US [1]. Mangoes, pitayas, and limes are some of the fruits exported from Indonesia to several countries. Besides, we want to focus on researching these fruits because mangoes, pitayas, and limes have become a commodity in recent years due to the increased production of said fruits in Indonesia and high demand from the domestic and foreign markets.
The type of fruits planted by farmers is very dependent on the harvest season and some are produced throughout the year. Sundry varieties of fruits produced in Indonesia are mango, pineapple, malacca, pitaya and limes which are scattered throughout different provinces. Additionally, each region has a different commodity value for different fruits. Harvesting mangoes is carried out if it has a maturity level of a minimum of 80% in a periodic range of 110 to 120 days after sprouting mango blossoms with green color and red stalk [2]. Characteristics of ripe mangoes can be observed from the surface of the skin, where green spots and yellowish speckles are present. Furthermore, a pitaya fruit is ripe when the fruit shows red spots that were initially green. Farmers can harvest it when the fruit appears 32 days after flowering [3]. Ripe limes have the characteristics of green, shiny skin. They must be free of defects or have very slight defects at most, provided these do not affect the general appearance, quality, keeping quality and presentation of the produce. Limes of diameter below 42 mm are excluded if the fruit has a fundamental difference between small and large fruits shown in diameter sizes not exceeding 7 mm [4]. The limes that have fulfilled the above characters are ready for harvest. The quality of fruit after harvest must be maintained throughout the process of sorting, packaging, storing, and fruit distributed systems to the supermarket.
The majority of Indonesian farmers are currently sorted fruit using manual methods. However manual sorting involves workers having to perform sensory tasks in large capacity and for longer working hours, which renders such method as unreliable. Besides, this process also requires a long time and a greater number of employees. A large number of employees will increase the cost of production and can lead to uncertainty and inaccuracy because individual judgments are subjective and inconsistent with fruit objects and work done repeatedly can cause saturation [5]. To overcome these limitations, it requires the help of computer image sensors to be more effective and efficient. Therefore, we propose a method for detecting multi-fruit using the AI-based smart cameras approach that can classify several varieties of fruit at one time based on color characteristics to make it easier for farmers in the process of fruit sorting. Artificial intelligent technologies have been using research to classify mangos teen defects on the surface fruit [6]. However, the method used in the traditional deep learning approach required 15 seconds to detect one fruit. It would be inefficient if this method was to be used in processing very large amounts of data. The same problem is acknowledged by K.N. Rajit et al. In identifying and classification fruit diseases [7].
More recently, Deep Learning is implemented by [8] to accurately detect fruit. They found that the application of a faster R-CNN framework (FRCNN) in this research could detect paprika fruits and achieved impressive accurate results. Susovan Jana et al. conducted fruit classification using the method of color segmentation and edge detection [9]- [12]. The proposed approach is carried out using the image acquisition process. However, they did not consider that the approach taken was not only detection-estimating the quality of fruit was also important in this case. Quality and quantity of fruit are the two main factors of farmers' success in distributing fruit to several countries.
Furthermore, they developed a system that can calculate the amount of harvest obtained by farmers. Their approach uses the radial symmetry transformation method to detect grapes. The detection results are then processed to produce estimates [13], [14]. However, this method only detects grapes that appear in front of them, whereas the blocked fruit cannot be detected. L Agilan Deswari also researched determine the quality of mangoes before reaching the mango market [15]. The author's approach was not optimal because algorithm optimization was not used in the selection of features to improve the accuracy obtained.
Other researchers have designed an application for image processing and color transformation. The method used in this study was color transformation to detect and classify the maturity level of bananas. Images of bananas in extraction are based on the values of red, green, and blue and are then converted to this [16]. From the experiment, the authors show that the approach taken can obtain 85% accuracy. Mc.Cool proposed of pepper detection and segmentation [17]. they used the LBP (Local Binary Pattern) as a method for extracting feature series [18], histogram [19] HSV color gradient and auto-encoder features. Then to segmentation, it is necessary to process random data to create vector features. This method has a good performance similar to human image processing, but its detection performance is surpassed by Deep Fruits.
Recently, Deep Fruit was proposed by Sa et al. This approach uses the FRCNN object detection framework [20] which uses deep learning to enter the proposal region and classify the proposed region. They achieved impressive results by applying this technique to several plants and showing that such an approach can be trained quickly and used. Another deep learning approach was suggested by Chen et al [21] to perform segmentation and detection using DCNN. From here they estimate results only from the image. Two aspects that are not considered from one of his works are the potential for estimating maturity and how to calculate of fruit the conveyor runway using video Tracking (or real-time data). This case is our inspiration to develop a system that can be to evaluate the quality and quantity of fruit in the field.
Three necessary steps should be done to develop an automatic classification system of fruit defect. The first stage is to divide each fruit into three categories; a) super, b) middle, c) defect. Secondly, using two network architectures and joint detection as a method to detect the quality of fruit and defects simultaneously (i) multi-class detection and (ii) layer parallel task. Third, ensure that the fruit will only be counted once using the tracking-via-detection approach. It can track and calculate fruit sequentially without using additional sensors and makes it easier for us to check the defect level on the fruit accurately using a video streaming camera

MATERIAL AND METHODS
In this research, we propose a system that can automatically classify mangoes, pitaya, and limes and detect defects in the fruit's surface. There are three approaches that we propose in this study, first by classifying each fruit as one of the three classes, "a", "b" and "c". Second, we create a system capable of estimating the level of maturity and defects based on the class of fruit. Third, we use video tracking. Methods for estimation of maturity and detection of fruit defects were inspired by Deep Fruits [8], however, we made an additional improvement by modifying previously trained network structures [14] (based on VGG-16 Architecture) to estimate fruit maturity and detect defects found on fruit surfaces. In the video tracking system, we count the number of fruits using the camera and simultaneously classify the quality of the fruit as good, medium, or defect. We also create a simple framework to ensure that the camera can classify fruit accurately and count the number of fruits in sequence without using a threedimensional (3D) camera.

Design System
We develop a system that can classify and estimate fruit quantity on the conveyor runway. The design system of the sorting machine is shown in the following Fig.   Fig 1. The architecture of sorting machine proposed Fig. 1 shows a master plan of the system that will be created. the conveyor uses a camera to detect fruit. The results from the camera will be stored on the database server. Next, we create a real-time analysis chart that can be accessed using web applications and android app. At an early stage, the focus of our discussion is the detection system and fruit quantity estimation. The system design that we propose can be seen in the picture below.

Datasets
In this study, we collect data real from farmers located in Gresik regency with 11,820 images that are further divided into 3 categories, A, B, and C or defect categories. Category A is characterized by a fruit that is shiny, green and has no defects. The B category fruit has the characteristics of a dull, green fruit and its surface has a rather yellowish color, while the defect fruit is yellow. Mangoes with category A, are those that are green and have white faded spots. A fruit in category B is a fruit that is too mature, making its shelf life in the market shorter. A Category B mango is a fruit that has a dull green color mixed with yellow and when held feels soft. A Category C mango is a fruit that has a defect on the surface of its fruit. This fruit is mostly used for bird feed so the price of the fruit in the market is very cheap. The three fruit categories above also have different prices, a fruit in Category A has the most expensive price and the fruit in the reject category has the cheapest price. Categories from all the datasets that we have collected can be seen in the table 1.

Image Labelling
The next stage after collecting images is an annotation process to label the segmentation process. We crop the image to 300x300 pixels and apply a label from one of 9 classes (limes A, limes B, limes defect, mango A, mango B, mango C, pitaya A, pitaya B, and pitaya defects). The output of this process will be the Region of Interest (ROI). We use Imglabeling tools to annotate images by labeling them one by one. The form of labeling can be seen in the picture below.

Fig 4. Sample of Fruits datasets
After the image is labeled one by one, labeled the data is stored in the form of an XML file. This XML file contains information on a fixed spatial distance from (h = high; w = width) while h and w are fixed hyper-parameters of each particular RoI. To easily process data, the contents of the XML file are converted into files that have the extension .csv.

Fruit Detection Module Using Faster RCNN
The framework that we propose is similar to the approach used by Deep Fruits, but we propose two new models to estimate the quality of the fruit and its detection system. Our initial stage detects each fruit based on their class sequences to determine super, medium, and defect fruit categories using multiclass Faster R-CNN. The next stage uses parallel layers (Parallel FRCNN) as one method to detect fruit and group fruit qualities together. More details can be seen in Fig. 5.
In this step, we carried out the detection of fruits using the FRCNN architecture by utilizing the Network Region proposal as a method to detect objects in real-time. We modified it using 8 convolution layers with a 300x300 image size and filter window 3x3 tread 2 on each filter. This technique is a refinement of the VGG-16 architecture [14] previously trained by Image Net which then produces a fully-connected layer output that contains a collection of aggregate scores. We executed this because the Image Net has 1000 classes, in the VGG16 architecture it consists of 4096-4096-4096 neurons in the hidden layer and 1000 neurons in the output layer, whereas in our case, we have 9 classes (limes-A, limes-B, limesdefect, mango-A, mango-B, mango-defect, Pitaya-A, pitaya-B, and pitaya defect). This is the main reason we made modifications to the FC layer of the VGG-16. Subsequently, in the Multi Class FRCNN and parallel-FRCNN, the same parameters are used to make it easier for us to adjust the initial weight on the 8 convolution layers that have been made. In this case, a forward propagation and backpropagation process is carried out in each layer, where each layer will be trained one by one until it is stable. When a layer is stable, we move on to the next layer and repeat the entire process using the same method Once all the layers have been trained, the output will produce a Map feature. Produce optimal performances, the optimization of each layer is required using a momentum optimizer with several 0.9, the weight of 0.0006, and a learning rate of 0.00001. Next, we rotated, flipped and cropped in the image to multiply the dataset without losing the core or essence of the data.

 Multi-Class Faster R-CNN used Multiple
Object class for detection and estimation quality of fruits: fruit quality estimation is given the symbol N where the fruit can be classified according to fruit type (Sf) and quality of fruit (Sd) is given the formula N+1. For more details, see in Fig. 2 (a). This is a simple strategy to provide information on fruits that have defects, based on type. However, if the number of fruit datasets in each type is different, it is difficult to estimate the quality of fruit that has defects. Parallel (layer) Faster R-CNN: in this case, we divide into two stages of the process. The first layer serves to classify the variety of fruit. The second layer estimates the quality of fruits that have defects, where the estimated quality of the fruit is given a symbol of class N. The layer will be activated when there is fruit on the conveyor runway. Each layer will back-propagation and forward propagation to produce a stable feature. When the fruit is moving on a conveyor runway, the first and second layers will produce a crossentropy class or loss (fruit detection layer) and (defect detection layer). Then, we calculated the total loss of both using formula: . If the fruit is found detected. However, this strategy does not apply when there is no fruit running on the conveyor runway.

Object Tracking System
After the fruit is picked by the farmer, it is then stored in the basket and put into the conveyor. Fruit that moves on the conveyor runway will be classified according to the type and quality of the fruit. However, we must ensure that the fruit that runs on the conveyor is detected only once and is activated periodically to prevent duplicate amounts of data. This approach greatly facilitates farmers because they only use cheap cameras to carry out classification and detection. The tracking system uses a detector to obtain information about the latest scale and pose of the object detected. IOU used to take sample videos at a ratio of 40 frames per second to objects running on the conveyor runway.
Two steps must be done. In the first frame, we initialize the fruit to be detected, the fruit image taken from each frame is unique and sequential. Then the image is saved and initialized as the initial track. This requires a basic knowledge of computers to activate the camera when they look at the fruit moving on the conveyor runway. The steps we take are as follows:  We calculate the intersection over Union (IOU) to determine which activates track (3).  If the camera detects a new fruit, it will be recording as a new fruit track by:  Stages to compare threshold value from IOU, y_merge that activated and  Boundary threshold y-boundary, compared to the active track to determine whether the object is a new track based on points (1).
Stage 1: data from frame -th then compared to new detection with the currently active track . The IOU of each active track (K) and new detection in each frame will be calculated to search the highest value between the two ,and . When IoU threshold value is higher than the value: , and , then is considered the same as . Consequently, the position of and will activate the track and update simultaneously. detection ,k, and active track from are then deleted from the upcoming calculation. This process (1) will be repeated until no track is detected, or until IOU that best matches less than y_d t , if a track is not considered active on three frames then it will be set to inactive and removed from the inactive tracklist.
Stage 2: remaining detection values J will be counted as the candidate who made the new track. However, to solve it if detection must be calculated as a new track. First, detection will be compiled with active tracks and their IOU will be measured. This IOU is above the , detection will be removed from further consideration. Second, the limit measurement of the next detection will be compared with the active track, ( ) and is calculated (See picture 2). Therefore, if the limit size is greater than the threshold , then the detection will be removed from further consideration. After step (2) is completed, the new track will be initialized using the rest of the previous detection. During our experiment, IoU was unable to resolve several cases. One of them is if an image has a large size and is on a small track. This causes many IOU values from these images to not proceed to the next step. Therefore, we determine the upper limit by measuring the limit by calculating how manyth, contained in the th track.
( ) (1) This method makes it easy for us to track new objects and add track based on detection results. Each track stores the following regression information output a region boundary with location coordinates (x, y, h, w), the results of the classification on the area containing the object or not will determine the probability of 0 or 1 [22]. (2) here are with, height and center of anchor and are the ground truth bounding box width, height, center.

RESULT AND DISCUSSION
The data presented herein were collected from one of the fruit farmer groups in the district of Gresik, East Java Province. The images we collected consist of 3 plant varieties, namely mango, pitaya, and lime. Each of the fruit varieties had a different harvest period. The image was captured using a Canon camera with a size of 300x300 pixels. Next, we annotate the images individually using labeling tools and store them in the form of data distribution.

Experiment 1: detection and quality estimation
Making a model that can recognize various classes is challenging, especially when large amounts of data and different variants are involved. Therefore, we try to use transfer learning to perform feature extraction in our case use transfer learning, we use the architectural model VGG-16 which had been trained before by Image Net. However, we made an architecture identical to VGG-16 without using the fully-connectedlayer and downloading the weight. All weights from VGG-16 have been trained using the Image Net dataset and can recognize colors, shapes, textures, etc. Hence it is possible to use this model to perform feature extraction processes from all the images we have. Forward propagation and backpropagation processes in each layer will be trained individually until stable. When it is stable, we proceed to the next layer and repeat the training process using the same method. If all the layers have been retrained, then their output will produce a feature Map. This feature map is stored in file form and given the name train-feature.npy and Val-feature.npy. The file can be used to flatten or reshape the feature map into a vector, hence it can be used as input from the fully-connected layer.
Tensor board Graph Visualization makes it possible for us to see the learning process and see the debug of the scenario we are doing. Additionally, the Tensor board can graph up to thousands of nodes with a simple display. After getting the Feature Map from Feature extraction using the VGG-16 model, the feature map is then By using the number of epoch 500 and a learning rate of 0.0001 the accuracy obtained in this scenario is 93.5% with loss cross-entropy of 0.02985. According to this scenario, we conclude that the feature extraction process is very influential on the accuracy of each learning and testing. Therefore, to produce maximum accuracy, more careful observation is needed to determine the model used in feature extraction. However, not all models that have good accuracy are suitable for use in the case of detection objects in real-time.
We have used this model for object detection, but many objects are not even undetectable. However, many objects are incorrectly detected, for example, a mango-A, instead of being labeled as limes-C.
The accuracy obtained in the training process using the CNN Region-Based algorithm is very good. However, the task of solving the case of object detection in real-time is very slow. The experiment above is one of the implementations using Region-Based CNN(RCNN). From the observation process, the time needed to detect one image is 2 seconds. For this reason, the experiment applies the Faster R-CNN algorithm by utilizing a new method, namely the Region Proposal Network. The experiment also utilizes the same CNN, which requires very fast operating time. From our observations, this algorithm can detect video streaming objects by 5 frames per second. The following is a comparison table of the computational processes of several different methods. Several methods even applied in our works to compare the accuracy of each experiment carried out. The results of the accuracy of several experiments that have been carried out can be seen in Fig. 11. Fig. 10 shows the optimal accuracy obtained used the Faster R-CNN Framework.

Experiment 2: Video Tracking
The table shows the result of a screenshot of our experiment in conducting a fruit detection process in real-time. From the results above, we conclude the process for object detection is remarkably constant, but there are shortcomings. When the fruit is inserted individually, the results of the detection area by the actual label. However, when fruit are entered simultaneously, some fruits do not match the correct label. This algorithm offers a solution for resolving cases of fruit quality detection in real-time managing many classes. In our future research, we aim to modify the architecture on feature extraction, so that it becomes capable of producing the best model and minimize errors in fruit quality detection.

CONCLUSION
In this paper, we present a system that classifies fruit with an accuracy level of 88% (mango), 83% (limes), 99% (pitaya) and with an average computation cost of 0.0131 m/s. We can track and calculate fruit sequentially without using additional sensors. additionally, we can also check the defect rate on fruit accurately using the video streaming camera. the datasets are taken in the field areas then trained using the frcnn framework using the tensor flow platform. Faster R-CNN is one model that proved possible in solving complex computer vision problems with the same principle and shows incredible results at the beginning deep learning revolution. This algorithm provides a solution for solving cases of fruit quality detection in real-time using multi-classes. In our future research, we experiment need to be done and make android apps integrated with an analytics system. we also used the tracking video system to calculate the number of fruits on the conveyor runway. we can track and calculate fruit sequentially without using additional sensors and check the defect rate on fruit using the video streaming camera more accurately and with greater ease.