{'fraction_non_alphanumeric': 0.034364033635701514, 'fraction_numerical': 0.022313447659405416, 'mean_word_length': 4.4409221902017295, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fruit sorting robot based on color and size for an agricultural product packaging system

Received Dec 29, 2019 Revised Feb 16, 2020 Accepted Mar 23, 2020 Indonesia's location in the equator gives an ideal condition for agriculture. However, agriculture suffers the issue of old farming due to a lack of youth interest working in this sector. This problem can be overcome by applying digital farming methods, in which one of them is by employing robots. Robotics technology is suitable for handling the harvested product, such as a sorting robot. This paper presents the application of a 4DOF fruit sorting robot based on color and size in a packaging system. The sorting is made possible by image processing where color is recognized by HSV analysis, and the diameter is known in the grayscale image and setting the thresholding. The fruit to be sorted is red and green tomatoes and red and green grapes. The experiments were conducted to show the effectiveness of the proposed method. The time requires for the robot to accomplish the task is 11.91s for red tomatoes, 11.76s for green tomatoes, 12.56s for red grapes, and 12.92s for green grapes. The time difference is due to the position of the boxes for the sorted fruit. The experimental results show that the arm robot manipulator is applicable for a sorting robot using the proposed method.


INTRODUCTION
Indonesia's location in equator gives advantages for developing the agriculture sector in which unfortunately still relies heavily on conventional agriculture. The agriculture sector suffers the issue of old farming. This term refers to a lack of youth motivation to work in agriculture. The youth in the village would instead pursue their fortune in the city rather than continue their parents' footsteps to become farmers. This issue is not happening in Indonesia only. The lack of workers in the agriculture industry can be overcome by applying automation or digital farming. Robot technology is implemented to support farmers from seeding, farm maintenance, harvesting, and packing the agriculture product [1][2][3][4][5][6][7]. Automation or digital farming promises better and more efficient handling and treatment of the product [8].
The current application of robots for agriculture products is during harvesting [1][2][3][4][5][6][7]. This robotics application can be extended to sorting the agriculture product during packaging [8], where the robot is designed to grade the fruits and sort them based on color and size. This system can be realized by employing a camera functioning as the eye for the robot. The current camera comes in small size with enough pixels and quality to be the input for the robot system [1][2][3][4][5][6][7]. The small size camera can be attached to the robot as an eye in hand robot and attached somewhere else within the system [9][10][11] or as an eye to hand robot.
The current application of the automatic sorting system is using an automatic belt conveyor or detection system by employing image processing for object detection; the detection can be a simulation [9] [10]. The objective of image processing is to separate the object from its background [11,12]. Image processing is a powerful technique for detection in terms of grading and sorting agriculture production [9,[13][14][15][16][17][18]. Grading and sorting are similar in term deciding which one belongs to which [9,[13][14][15][16][17][18]. The common image processing techniques used are HIS [18] and HSV [19][20][21], L*a*b* histogram [22,23], Otsu thresholding [20][21][22], edge detection [22,24], image segmentation [6,25], Blob analysis [9,26,27], random forest [28,29], contour [29,30], and convex surfaces [31]. The image processing is used to determine size [9,10,13,18,19], maturity [22,29,30,[32][33][34], bruising [35], color [9,30] and pattern of the fruit [9,17,26,30,36]. The sorting process requires an additional sensor, such as a weight sensor to classify products based on the weight [10]. The researchers even include artificial intelligence to improve the effectiveness of the detection system [19,32]. This paper presents the fruit sorting robot based on size and color applied for an agricultural product packaging system. The objective of this study is to design a sorting robot as the improvement of the current application of a belt conveyor or manual system for sorting the agricultural product. The main contribution and novelty of this project are creating a robot imitating the human's arm for the sorting system. The application of a robot is expected to create a more efficient working environment. The camera applied is an IP camera to detect the fruit, and the image processing is kept simple to facilitate the limited source of microcontroller available in the market. The fruits to be sorted in this study are red and green tomatoes and purple/reddish and green grapes. The fruits are classified into small green and red, and big green and red. Experiments are conducted to show the effectiveness of the proposed method.

RESEARCH METHOD
This paper discusses the application of an arm robot manipulator as a fruit sorting robot based on color and size for an agricultural product packaging system. The object to be sorted considered in this study are red and green tomatoes and red and green grapes. The fruits are chosen since those fruits are a standard agricultural product. The proposed sorting robot system is shown in Figure 1.

Mechanical and electrical design
The robot applied in this study is a 4DOF arm robot manipulator where each joint is moved by a servo motor. The electrical and mechanical design is shown in Figure 2. This articulated robot is considered an ideal type to work as the sorting robot in which workspace is a crescent-shaped cross. The sorting robot system is equipped with a camera attached near to the initial fruit position and a proximity sensor attached to the end-effector to sense the distance between the end-effector and the fruit. The end-effector designed is a gripper to pick and place the fruit according to its size and color. The sensors applied are proximity and vision sensors. The proximity sensor attached to the gripper is the low-cost ultrasonic sensor (effectively from 2-400 cm). The vision sensor is an IP camera functioning as an eye to hand, which advantage is not suffering any occlusion while completing its task. The main controller is an ATMega2560 responsible for moving the servo motors, and the processor for image processing is Raspberry Pi.

Fruit sorting algorithm design
This study discusses the application of robotics to sort fruits based on size and colors. This application relies heavily on image processing to decide which fruit belongs to which container. The image processing is conducted in two steps, deciding the colors and recognizing the size. The color recognition is using HSV analysis, and the size recognition is conducted by calculating the diameter of the object/fruit in the grayscale image and setting the thresholding.
The fruit color and size is conducted by converting the raw image captured by IP camera into a grayscale image, and isolate the object from the background by converting to a binary image, HSV analysis, setting the thresholding, and finally detecting edge to find the diameter of fruit by counting from the center to the edge. The first step of image processing is detecting the color of the fruit. The color analysis is by converting the RGB image to HSV. The process is started by retrieving the red (R), green (G), and blue (B) on a scale of 0-1. To achieve scaling 0-1 in the binary image, the scale of each component is decided by taking 255; therefore, each for R, for G, and for B is 255. The new value of R, G, and B are given by: (1) The first step to decide HSV analysis is to get H (Hue) value, which represents the color type. The H is taken by taking the largest ( ) and the smallest ( ) of R, G, and B values. The difference between the largest and the smallest value is called ∆. The H is normalized by adding either 0, 2, or 4 to get any real number. Any arbitrary number below 0 and above 6 is considered redundant. The H value is decided by: where = max(R, G, B), = min(R, G, B), and ∆= − . The brightness of the assigned object is defined by the V (Value) value, which ranges from 0-255, where 0 is the darkest, and 255 is the brightest. S (saturation) gives the vibrancy to the object color. The value range from 0 to 255. The V value is given by: where scale v is 255. The lower the saturation value, the more fading the image is. S is also defined as the difference between the most significant and smallest channel values and divided by V, as shown below: where scale s is 255, and if V is 0, then S is 0. Therefore, as the IP camera in the packaging system captures the RGB images from the online video, the Raspberry PI converted those captured images into HSV value, as calculated above. The HSV values are used to determine the x-y coordinates position of the object and decide the color of the object between red and green. The red detection is for reddish color, starting from orange to very dark red.
The next step of fruit detection is the determine the size of the fruit, to differentiate between red tomato and red grape, and green tomato and green grape. The size is decided by using ellipsoidal shape achieved from the grayscale image where the 4 ROIs (region of interest) of the detected fruit are identified. The thresholding value is set, and the edge of the object is detected. This process is as shown in Figure 3.
where and is the center of detected fruit, and is the geometric moment ( , = 0,1,2, …). The geometric moments are calculated by the summation extends over all the elements in the detected fruit ( ), and given by: while the diameter of the fruit is: = √ 2( 20 + 02 +√( 20 − 02 ) 2 +4 11  2 ) 00 .
The diameter of the fruit gives the size of the fruit. As the image processing process is complete, the color and size of the fruit are known. The data is inputted to the microcontroller to sort the fruit by moving it to the box of red/green tomato or red/green grape. The result of image processing is shown in Figure 3, where the detected fruit is classified as a red tomato.

RESULTS AND DISCUSSION
This paper studies the image processing for fruit sorting robot based on the color and size. The applied robot is a 4DOF articulated robot with gripper as the end-effector. The HSV analysis decides the color, and thresholding and edge detection give the size of the fruit. The considered fruits are red and green tomatoes and red and green grapes. An experimental testbed is set, as shown in Figure 4. The experiments are conducted to show the effectiveness of the proposed method. The arm robot sorts and places the fruit in box 1 to box 4 according to the color and size. The first box is for red tomatoes, the second box is for green tomatoes, the third box is for red grapes, and the fourth box is for green grapes.

Fruit detection
The first experiment is setting the fruit in x and y coordinates, and the processed image is shown in Tables 1 and 2, where Table 1 is the data for tomatoes detection, and Table 2 is for grapes detection. The detection shown in Table 1 is only for red tomato, and Table 2 is green grape. The image plane considered in this study is 640×480 pixels. The fruit position is divided into the middle, right, and left area of the image plane, as shown in Tables 1 and 2. These x-y coordinates are necessary to tell the robot where to pick the fruit, and the size is necessary for the robot to place the fruit in the right assigned box, as shown in Figure 4. The circle is to show the viewer of the detected fruit, and it is achieved from the edge detection method.  In the left area of the image plane

Arm robot motion in sorting the fruit
The arm robot moves to pick and place the fruit in its assigned box according to the location information shown in 3.1. The process of picking and placing is conducted by controlling the angle of the servo motors' motion. Table 3 shows the angle of each servo motors during a particular motion. The screenshots of robot motion while sorting the tomatoes and grapes are shown in Figure 5, according to the experimental setup in Figure 4. The fruits are placed accordingly to the assigned boxes. The experiment in Figure 5 is conducted ten times for each fruit, as shown in Table 4, and Table 5 shows the time requires to pick, place the fruit, and the arm robot returns to stand by position in three experiments for each fruit. The robot is powered by a 2200 mAh battery to enable the robot working for 2 hours. The picking and placing fruit process is prone to illumination that can affect the performance of the robot. The "√" in Table 4 is when the robot was able to sort the fruit, and "X" is when the robot failed. The effectiveness of the proposed method in sorting, picking and placing the fruit is 80% for red tomatoes, 90% for green tomatoes, 70% for red grape, and 60% for green grapes. The failures are caused by the size of the fruit and illumination during the experiment. In experiment 4 and 6 of red tomatoes, the tomatoes are too big for the gripper, and experiment 1, 4, 7, and 8, the grapes are too small for the gripper. Experiment 2 for green tomato is due to its changing color from green to red; therefore, the robot cannot decide whether the fruit is green or red. Experiment 4, 6, and 8 for red grapes, the color of grapes is too dark. The average time for the robot to accomplish sorting red tomatoes is 11.91 s, green tomatoes is 11.76s, red grapes is 12.56 s, and green grapes is 12.92 s. This is the time requires for the robot from detection in standby position (initial position), picking up the fruit, and placing it to the assigned boxes. The time requires is not much different in real-time.

CONCLUSION
This paper discusses the application of an arm robot as a sorting robot based on color and size. The image processing conducted by setting HSV value analysis to recognize the color of the fruit and the size is achieved by calculating the diameter of the fruit in the grayscale image and setting the thresholding. The fruits to be sorted are red and green tomatoes and red and green grapes. The experiment was conducted ten times for each fruit. The effectiveness of sorting, picking, and placing the fruit is 80% for red and 90% for green tomatoes, 70% for red, and 60% for green grapes. The failures are due to the size of fruits and illumination during the experiment. This drawback can be overcome by applying flexible gripper that can adjust to the size of the fruit, and improving the image processing algorithm to be less prone to illumination. The time requires to accomplish the fruit sorting is noted for three experiments for each fruit, and the average time for the robot to sort red tomatoes is 11.91 s, green tomatoes is 11.76 s, red grapes is 12.56 s, and green grapes is 12.92 s. The experimental result shows that the arm robot is applicable for a fruit sorting robot based on the proposed method in this study.