{'fraction_non_alphanumeric': 0.030922120600229638, 'fraction_numerical': 0.019242190283881695, 'mean_word_length': 4.957075471698113, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Impurities Detection in Intensity Inhomogeneous Edible Bird’s Nest (EBN) Using a U-Net Deep Learning Model

As an important export, cleanliness control on edible bird’s nest (EBN) is paramount. Automatic impurities detection is in urgent need to replace manual practices. However, effective impurities detection algorithm is yet to be developed due to the unresolved inhomogeneous optical properties of EBN. The objective of this work is to develop a novel U-net based algorithm for accurate impurities detection. The algorithm leveraged the convolution mechanisms of U-net for precise and localized features extraction. Output probability tensors were then generated from the deconvolution layers for impurities detection and positioning. The U-net based algorithm outperformed previous image processing-based methods with a higher impurities detection rate of 96.69% and a lower misclassification rate of 10.08%. The applicability of the algorithm was further confirmed with a reasonably high dice coefficient of more than 0.8. In conclusion, the developed U-net based algorithm successfully mitigated intensity inhomogeneity in EBN and improved the impurities detection rate.


Introduction
Edible bird's nest (EBN) extract has been recognized as a supplementary food of high nutritional value, especially with its glycoproteins and an abundance of sialic acid. Its known benefits include boosting immunity, improving brain function, and preventing virus/influenza infections. Additionally, it has shown potent activity in skin aging therapies [1][2][3]. Owing to its rarity, perceived nutritional value and labor intensive production, raw EBN has become a high value and expensively-priced exotic delicacy for export. To be safe for human consumption and fulfill stringent export requirements, the cleanliness of EBN is paramount, i.e., it needs to be free from any impurities.
Several researches [4][5][6] have attempted to investigate automatic cleanliness inspection; however, the results were not satisfactory. Subramaniam et al. [5] performed EBN impurities detection on raw EBN soaked in water. The authors managed to detect 68.18% to 87.93% of the impurities from the raw EBN. In Goh et al. work [4], the K-Means and Fuzzy C-Means algorithms were used to detect impurities in solid raw EBN. The method achieved average accuracy ranging from 84% to 86% for large impurities with more than 350 pixels. However, the method was almost not able to detect less than 10 pixels impurities. The misclassification rate was recorded at 57%. With proposed optical segmentation and image fusion techniques, Yee et al. [6] demonstrated that the recall/true positive rate of the impurities detection fetched as high as 93% out of 980 impurities. Even though the result was encouraging, a high misclassification rate was recorded at 32.25%. The impurities detection remained unsatisfactory. Dark and thick EBN strands and other artifacts such as shadow could be easily misclassified as impurities, resulting in a high false-positive rate. Generally, the performance achieved by previous methods was yet to fulfill the stringent requirement in food industry [7]. These methods were unable to perform efficiently due to intensity inhomogeneity occurred in EBN images.
EBN imaging suffers from a severe form of intensity inhomogeneity, as there are abrupt spatial changes in intensities besides the smooth phase, due to the entangled salivary strand structures of EBN. The problem is further complicated by the natural color and translucency of the salivary strands. The natural translucency of the EBN allows illuminated light to be transmitted, absorbed and reflected at different degrees, thus resulting in varying and highly inconsistent intensities of EBN [8] across the EBN region. The irregularities in shape, density, and thickness [6] of the entangled EBN strands further complicates the EBN image processing and analysis. Although researchers have developed various image processing and analysis methods [8][9][10] to reduce the effects of intensity inhomogeneity in other domains, these methods are application-specific and can only be used in certain domains of problems.
Due to the convolutional operations of feature extraction, deep learning models are able to extract deep features or the feature representations [11] from images with intensity inhomogeneity, which could not be accomplished by traditional image processing algorithms. Several successful cases have been presented in the past such as the use of convolutional neural network (CNN) [12][13], autoencoder [14][15], fully convolutional network (FCN) [16], and U-net [17][18], especially in biomedical medical imaging [12,14,16] and other domains [13,15,[17][18]. With its encoder and decoder network architecture, U-net can be considered as a better model when dealing with intensity inhomogeneity [19]. During the encoding operation, the spatial information is reduced through a series of convolutions and pooling operations while the feature information of the targeted object is extracted. The latter decoding operation then combines the feature extracted and spatial information through a sequence of deconvolutions and concatenations with high resolution features from the encoding operation [19]. Through supervised learning, a denser feature information of the targeted objects can be extracted with minimum spatial information.
Compared to other deep learning models, U-net achieves a higher pixel-wise localization accuracy at a cheaper computational cost [19]. U-net also outperforms FCN [17] with more convolutional layers in decoder. Therefore, U-net can be trained for image segmentation with intensity inhomogeneity in a more precise and efficient way.
This work attempts to improve impurities detection in EBN by incorporating the state-of-art U-net deep learning model for impurities detection. The EBN impurities detection algorithm development involves essential steps such as image acquisition and dataset preparation, U-net model training, and algorithm performance evaluation. These are further explained in the following sections. The algorithm performance evaluation metrics include precision, recall, F 1 score, average distance between the centroids of the annotated and predicted impurities, and dice coefficient for statistical similarity in predictions.
The results collected are analyzed both generally and based on impurity size. Finally, the findings of this study are summarized and recommendations on future investigations are provided.

Image acquisition and dataset preparation
In this work, a Basler acA2500-14uc camera with a resolution of 1,942 pixels × 2,590 pixels was used for EBN image acquisition. The camera was fixed at a height of 25 cm from the sample. With this setting, the size of one pixel in an image was equivalent to 0.019 mm × 0.019 mm. This image resolution was enough to capture the smallest impurity observable by a human with a minor axis length of 0.073 mm at the same distance and at a minimum human observable visual angle [20]. In this work, the minor axis length of impurity was used to describe the size of impurities and defined as the minor axis length of an ellipse enclosing an impurity. This phenomenon was attributed to the difference in the penetration power of the lights with various wavelengths and optical phenomena in different mediums [21]. Therefore, different information could be retrieved from the images captured under different lights to facilitate the EBN impurities detection [6]. To retrieve this information, the EBN sample images were captured under various colored lights, namely red, green, blue, and white lights provided by TMS Lite LBRQ 3 Rows Series. In order to produce a low angle direct illumination and reduce light reflection, the height of the light source from the EBN sample was set to a low value of 10 cm, and the direction of the lighting was adjusted to achieve an angle of 50° from the horizontal plane [21]. The EBN image dataset for training and testing purposes consisted of a total of 50 EBN image samples prepared according to standard practice in the EBN industry [22]. A total of 40 out of 50 EBN images were selected randomly as training dataset images while the other ten images were used as test dataset. A total of 18,170 impurities were identified from these images and annotated. Fig. 1(e) depicts an example of the annotated image based on Fig. 1(d). Ground truth binary masks of the annotated impurities were prepared as shown in Fig. 1(f). Data augmentation was carried out on these images for generalization purposes [23] due to the limited number of samples. The procedure of data augmentation is shown in Fig. 2.
Affine transformation and elastic deformation [24] were applied to the training dataset to ensure that the trained model was invariant to small deformation of impurities. Rotational transformation of 0°, 120°, and 240° was then applied to the training dataset to train the model to be rotationally invariant. Finally, the training dataset images were randomly cropped into smaller image patches. For the testing dataset, only the random cropping technique was applied for augmentation in order to maintain the originality of images for the testing. As a result, a total of 30,000 image patches were prepared as a training dataset while 2,000 images patches were prepared for testing and contained a total of 3,210 annotated impurities. All of these image patches were prepared with a size of 128 pixels × 128 pixels in order to fit the input dimensions of the U-net.

Model training and hyperparameters
In this work, the architecture of the U-net model proposed by Ronneberger et al. [19] as shown in Fig. 3 was used. The model was reinitiated and trained with improved hyperparameters for EBN impurities detection, as explained in this section.
The training of the U-net model was performed using TensorFlow Core r2.0 software with Python 3.6.8. Python was used as it Categorical cross-entropy loss function with two classes, impurity and non-impurity, was applied in the U-net training.
The pixel-wise losses in channels of different classes were multiplied with different loss weightages before they were summed up. These loss weightages aimed to control the tendency of the U-net classification [19]. The L 2 norm regularization was performed with a weight decay rate of 0.00001, to ensure that the U-net weightages would not be indefinitely tuned to a very large value [23]. The Adam optimizer [25] was used as it offered the benefits of momentum [26] and RMSprop [27] simultaneously. The hyperparameters in the Adam optimizer were set to default values [25], except for the learning rate which was selected to be 0.0001. Both the weight decay rate and learning rate values were determined based on heuristic based iterative approach using values from the recommended ranges in the previous work [28][29] with other hyperparameters being fixed as constant.
A k-fold cross-validation [23,30], where k = 5, was carried out to analyze the loss and the accuracy of the U-net model with limited samples. The loss weightages for both classes were set to be equal at 0.5 during the k-fold cross-validation. The cross-validation was carried out for 100 epochs, with a batch size of 30 in consideration to the limitation of the GPU memory.
The stopping criterion was chosen as the number of epochs when the difference between the lowest and current validation losses exceeded 5% of the lowest validation loss value [29]. To reduce the effects of validation loss noise, a polynomial function was fitted to the validation loss [31]. The modified stopping criterion is shown in Eq. (1).
where f poly is the polynomial function used to fit validation loss, t is the epoch number when model training should be stopped, and t min is the epoch number when minimum validation loss is obtained. The average number of epochs, when early stopping was applied, was found to be 51 using Eq. (1). After the k-fold cross-validation procedure was conducted, a new set of hyperparameters for the U-net model was determined. As loss weightages could affect the performance of the U-net in detecting EBN impurities, the U-net model was investigated by applying different channel loss weightages in training for further performance investigation. In this investigation, the impurity channel loss weightages varied from 0.30 to 0.70, with intervals of 0.05. The summation of impurity and non-impurity region channel loss weightages was constrained to one.
where i is the row coordinate of EBN image patch, j is the column coordinate, p i,j,imp is the probability of pixel at coordinate (i, j) being impurity, and p i,j,nonImp is the probability of pixel at coordinate (i, j) being non-impurity. The values of p i,j,imp and p i,j,nonImp were obtained from the probability tensor generated by U-net. The probability tensors were rearranged to form a dimension of 1,942 pixels × 2,590 pixels before the binary mask was generated from the probability tensors based on Eq. (2) for impurities classification purposes. Pixels with intensity values of 255 on the mask indicated the presence of impurities, while pixels with zero intensity represented the non-impurity regions. As the U-net model was trained to detect all impurities, including those with intensity values close to their adjacent non-impurity regions, there was a possibility for the U-net model to become over sensitive and classify non-impurity regions as impurities. To remove any misclassifications due to the sensitivity of the U-net model, an opening operation was carried out on the binary mask generated by the U-net model. This operation was carried out using a 3 × 3 cross-shaped structuring element. The size of the structuring element was selected to minimize isolated non-impurities or noises. The opening operation improved the precision of the EBN impurities detection algorithm, but at the expense of algorithm performance to detect smaller impurities.

Performance evaluation
The output binary masks generated by the EBN impurities detection algorithm discussed in Section 3.3 were compared to the ground truth binary masks obtained in Section 3.1. Precision, recall, and F 1 scores were used to evaluate the performance of the EBN impurities detection algorithm. Precision indicates the probability of actual impurities detected out of all detections made. A high precision is desirable; however, a lower precision might indicate an oversensitive algorithm for impurities detection. Recall measures the proportion of the number of actual impurities detected out of all impurities annotated. A high recall means the algorithm is able to perform the impurities detection as intended. F 1 score is the harmonic mean of precision and recall [32]. A high F 1 score indicates that the algorithm has achieved high precision and recall at the same time. The precision and recall values were calculated based on the number of successful and unsuccessful detections of the impurities.
Multiple impurities that were predicted and overlapped with the same annotated impurities in the ground truth masks were considered as one single detection. A true-positive detection was an impurity predicted by the algorithm in the output binary mask, which overlapped with an annotated impurity in the ground truth binary mask. A false-positive detection was a predicted impurity that did not have any overlap with an annotated impurity. A false-negative unit occurred when there was no prediction of an impurity hitting on the annotated region. To ensure a fair comparison between human vision and the EBN impurities detection algorithm performance, the false-negative units with minor axis lengths smaller than or equal to 4 pixels (0.076 mm) were excluded from the recall calculation.
The performance of the impurities detection algorithm to locate the impurity was confirmed by computing the Euclidean distance between the centroids of the overlapping annotated and predicted impurities. Dice coefficient [33] was calculated pixel-wise to observe the statistical similarity between the actual and predicted impurities. Dice coefficient of value higher than 0.7 [34] indicated a high statistical similarity in terms of size and shape of a prediction. It should be noted that the dice coefficient in this context is not equivalent to the F 1 score as the latter was calculated based on the count of impurities predicted.
A more detailed analysis of the distance between centroids and dice coefficient was carried out by grouping the annotated impurities into minor axis lengths of less than 0.1 mm, between 0.1 mm and 0.5 mm, between 0.5 mm and 1 mm, and larger than 1 mm. The ratio of the distance between the centroids to minor axis length was calculated to observe the relative distance between the centroids compared to impurity size.

Results and Discussion
The performance of the EBN impurities detection algorithms by using the test dataset as input is summarized in Table 1.
This dataset comprises of EBN images that were not presented during the training and validation stages. The performance of the proposed U-net model based impurities detection algorithm could be considered as good, with an average precision value of 0.8404 ± 0.0422, recall or true positive rate of 0.9758 ± 0.0115, and F 1 score 0.9016 ± 0.0214 at 95% level of confidence.
The size, shape as well as the location of the impurities predicted by the algorithm could also be considered good with a sufficiently small distance of 0.0370 mm ± 0.0040 mm (95% level of confidence) between the predicted and annotated impurity centroids, as well as a high dice coefficient of 0.7646 ± 0.0236 (95% level of confidence). The model trained with the impurity channel loss weightage of 0.45 showed the optimal performance for EBN impurities detection based on the F 1 score of 0.9302. This model yielded considerably high precision and recall values of 0.8961 and 0.9669, respectively. Detailed analysis of the impurities detection revealed that the optimal model was able to detect up to 96.69% impurities (true-positive rate) out of 3,210 annotated impurities in the testing dataset, but with a 2.97% false-negative detection. The remaining 10.08% of the detection were false-positives. This can be translated to an accuracy of 86.95%, a ratio of true-positive detection to all detections made. The misclassification rate therefore could be calculated as 13.05%, a summation of false-negative and false-positive. The true-positive rate recorded in this study (96.69%) was higher than that achieved in Subramaniam's [5] work (68.18% to 87.93%) and Yee's [6] work (93.39%). In addition, the U-net based EBN impurities detection algorithm was able to score an improved misclassification rate (13.05%), far lower than the recorded values of 57% in Goh's [4] work and 32.25% in Yee's work. A lower false-positive detection recorded in this work indicated that the intensity inhomogeneity had been mitigated to a greater extent. However, it should be noted that this study utilized different method, camera resolution, setup of apparatus, and lights to capture the EBN sample images.
Careful investigation on the ground truth annotated impurities on EBN images, such as a sample image shown in Fig. 5(a) with the annotated version in Fig. 5(b), was performed to understand the root causes of misclassification and inaccurate segmentation. Figs. 5(c)-(f) respectively show the example of false-positive detection, branch structured impurity, large impurity, and small impurity, which are cropped from Figs. 5(a)-(b). In general, the U-net model based detection algorithm did not perform well in distinguishing impurities and low intensity artifacts which have similar intensity values. These low intensity artifacts were normally formed by high density or thick EBN strands that prevented light penetration and produced low intensity regions within EBN images. These high-density or thick EBN strands could be seen in Fig. 5(c). The low intensity artifacts would confuse the algorithm and be misclassified as impurities. Meanwhile, low intensity shadows within EBN could be other forms of low intensity artifacts. False-negative detection normally occurred when impurities have high intensity similarities to their neighborhood pixels or regions, including impurities which had been brightened up due to the internal reflection and refraction phenomena within translucent EBN [21].  6 shows the performance of the impurities detection algorithm in terms of distance between the centroids (Fig. 6(a)), ratio of the distance between the centroids to minor axis length ( Fig. 6(b)), and dice coefficient of the predicted impurities in relation to annotated impurities (Fig. 6(c)). Table 2 summarizes the numerical values in Fig. 6. The average distance between the centroids ranged from 0.03733 mm to 0.04532 mm for different impurity minor axis length values. These values indicated that those predicted impurities were approximately two pixels, or 0.04 mm away from the annotated ground truth impurities.
The location deviations predicted were mainly subjected to the irregular sizes and shapes of the impurities such as elongated structures, branches, and star-like shapes, as shown in Fig. 5(d). As it can be observed in Fig. 5(d), the feather impurity has a branched structure, and can be predicted. However, the predicted shape and size were not exactly the same as the annotated ground truth. Deviations in location therefore were expected. From Fig. 6(b), the average ratio of the distance between the centroids to minor axis length was reduced from more than 0.5 to less than 0.03. This reducing trend indicated that the deviation in distance of the two centroids became insignificant when the sizes of the impurities increased. In consideration of the two-pixel deviation, the locations of those predicted impurities could be deemed accurate and close to the annotated impurities.
The statistical similarity of those predicted impurities to annotated impurities could be observed by analyzing dice coefficients recorded in Fig. 6(c) and Table 2. Average dice coefficients for impurities with minor axis lengths more than 0.1 mm were higher than 0.8, thus indicating a high statistical similarity. Average dice coefficient for impurities with minor axis length less than 0.1 mm was recorded as 0.6700 ± 0.0168. This particular phenomenon can be better understood by observing Fig. 5(e) and Fig. 5(f). For small impurities, the overlapping area of annotated and predicted impurities was relatively small and fewer pixels were involved, compared to the total area covered by both impurities. Incomplete prediction and deviations also contributed to the lower true-positive pixels count and this could be significant for small impurities. As a result, dice coefficients for small impurities were relatively lower compared to large impurities.
(a) Average distance between centroids (b) Average distance between centroids to minor axis length ratio (c) Average dice coefficient Fig. 6 Graphs of different evaluation metrics versus minor axis length of impurity In general, the U-net based impurities detection algorithm proposed in this work demonstrated reasonably high accuracy in EBN impurities detection, especially for impurities with minor axis lengths larger than 0.1 mm. The size, shape and location of the predicted impurities could be considered to be close to the annotated ground truth impurities. Detection of smaller impurities could possibly be improved by using a higher resolution camera to capture images of small impurities in a more detailed manner.

Conclusions
In this work, a U-net based EBN impurities detection algorithm was developed and tested. The incorporation of the U-net model to the algorithm had successfully mitigated the intensity inhomogeneity issues and improved the impurities detection performance. Based on the results obtained, conclusions can be drawn as follows: (1) The algorithm proposed in this work recorded a higher true positive rate and a far lower misclassification rate compared to the results reported in previous literatures of relevant studies.
(2) For true-positive detections, a high statistical similarity had been achieved for all human visible impurities, and the location of impurities predicted were close to actual impurities annotated within a two-pixels accuracy.
Future studies should therefore focus on the reduction of misclassification rate caused by the false-positive and false-negative detections. The scopes of future study are proposed as: (1) The exploration of the lighting effects [21] towards the model performance on classification of impurities, non-impurity regions as well as artifacts in order to reduce the misclassification rate as reported in this work.
(2) Meticulous investigation and performance comparison on other deep learning architectures in search of a better model compared to the U-net model as reported in this work.
(3) The development of an improved version or a completely redesign model shall be regarded as a part of the future work for practical deployment on physical system.

Conflicts of Interest
The authors declare no conflict of interest.