{'fraction_non_alphanumeric': 0.03113373800665753, 'fraction_numerical': 0.011308008615625613, 'mean_word_length': 4.882234379498993, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Segmentation of Weakly Visible Environmental Microorganism Images Using Pair-wise Deep Learning Features

The use of Environmental Microorganisms (EMs) offers a highly efficient, low cost and harmless remedy to environmental pollution, by monitoring and decomposing of pollutants. This relies on how the EMs are correctly segmented and identified. With the aim of enhancing the segmentation of weakly visible EM images which are transparent, noisy and have low contrast, a Pairwise Deep Learning Feature Network (PDLF-Net) is proposed in this study. The use of PDLFs enables the network to focus more on the foreground (EMs) by concatenating the pairwise deep learning features of each image to different blocks of the base model SegNet. Leveraging the Shi and Tomas descriptors, we extract each image's deep features on the patches, which are centered at each descriptor using the VGG-16 model. Then, to learn the intermediate characteristics between the descriptors, pairing of the features is performed based on the Delaunay triangulation theorem to form pairwise deep learning features. In this experiment, the PDLF-Net achieves outstanding segmentation results of 89.24%, 63.20%, 77.27%, 35.15%, 89.72%, 91.44% and 89.30% on the accuracy, IoU, Dice, VOE, sensitivity, precision and specificity, respectively.


Introduction
The large scale of industrialization and urbanization is providing good living conditions for human beings. However, it has brought serious environmental pollution, including water, air and soil pollution [1], which raises the risk of diseases such as lung cancer. To eliminate such pollution (pollutants), the use of environmental microbiological method offers higher efficiency, lower cost and harmless compared to the use of chemical methods. It involves the use of Environmental Microorganisms (EMs) for monitoring, controlling and decomposing pollutants. For example, Epistylis is employed as a sign of poor quality of water and Actinophrys is used for decomposition of organic wastes in sludges [2].
Thus, identification of proper EMs and their corresponding physiological characteristics is necessary. Generally, there are four methods used for identification of EMs. First is the chemical method, which is accurate, but it creates secondary pollution of chemical reagents [3]. Second is the physical method, which requires expensive equipment [3]. The third is the molecular biological method, which distinguishes EMs by sequence analysis of genome [4]. This method needs expensive equipment, is time consuming and requires professional researchers.
Fourth is the morphological method, which needs an experienced operator to observe EMs under a microscope and give identification by shape characteristics [5], [2]. This approach is laborious, time-consuming, inconsistent, and subject to the moods of the operator.
In order to eliminate such drawbacks, automatic image processing techniques are used for the identification of EMs. Image segmentation is a crucial stage in feature extraction [6] and classification [7], so we develop a system for segmentation of EM images. The majority of EM samples are obtained from complex environments where large amount of impurities like rubbish is present, which leads to noisy image problems. Moreover, some essential EMs have transparent like body features such as Ceratium and Actinophrys. This renders less information of the foreground for segmentation tasks, which leads to undersegmentation and poor segmentation results. Furthermore, some EM images suffer from low contrast between the foreground and background, such as Codosiga and Vorticella, which leads to poor segmentation results. To jointly overcome all segmentation challenges above, we use Pairwise Deep Learning The steps shown in Figure 1 from (a) to (f) respectively, are described below.
(a) Weakly visible classes: In this study we use an in-house dataset which is is also publicly available in [8] and published in [9]. It contains 21 classes of EMs.
Therefore, from 21 classes, the eight most weakly visible classes are selected. This paper is organized as follows: Section 2 gives a review of related works on microorganisms image segmentation methods (particularly in subsection 2.1) while the review on feature extraction and pairwise feature methods are given in subsection 2.2. Section 3 describes in detail our proposed methods and different key points of our contributions. Experimental results and analysis are discussed in section 4. Lastly, conclusion and future works are given in section 5.

Related Works
In this section, different works related to our work are reviewed. Section 2.1 gives a review on segmentation of microorganisms images. Due to the importance of feature extraction in our work, different related works on feature extraction and pairwise features are reviewed in section 2.2. Finally, the contributions of our work are given at the end of section 2.2.

Microorganims Image Segmentation
Different techniques are implemented to enhance good segmentation performances of microorganisms. These techniques can be categorized into classical and machine learning based techniques [1]. Table 1, gives a summary of the categories and subcategories of microorganisms image segmentation methods.  [15], [16], [17] Classical methods are the traditional techniques which have found broad applications. For instance, in [10] outstanding results are achieved by applying Otsu thresholding in the segmentation of floc and filament. In order to enhance shape feature extraction, an active contour method is used in [11] for segmentation of Rotavirus-A. A seed watershed algorithm is applied in [12] for segmentation of Bacillus subtilis bacteria in clustered biofilm. Generally, classical methods are associated with challenges such as, they can not work direct on colour images, they need pre-processing like denoising and colour conversion and they cannot perform well on images which have uneven background colours. To overcome above challenges machine learning based methods have been adopted for segmentation.
Machine learning based methods can be categorised into unsupervised and supervised [1], as shown in table 1. Unsupervised machine learning (ML) techniques build their mathematical models from a set of data that contain only input without target output labels (segmentation can be referred to as pixel level classification, in that context the target labels are the individual pixel values/ranges in the ground truth mask images. Where, for the case of of unsupervised ML they are not required. An example of unsupervised ML algorithms is the k-means clustering). These techniques unsupervisely discover the data pattern and cluster them into segments [18]. For instance, in order to automate the detection of pulmonary tuberculosis (TB) which is caused by Mycobacterium tuberculosis, k-means and self organizing map (SOM) clustering were proposed in the segmentation of the basilli from Ziehl-Neelsen sputum smears [13] and [14]. While in [19], a modified fuzzy divergence clustering method which is based on Cauchy membership function is leveraged in the segmentation of Plasmodium vivax from C channel CMYk color model of images containing the parasites in blood smears. Although unsupervised methods are simple to apply, their ability to learn the pattern of data is inadequate in transparent images, which is the case for the weakly visible EM.
In recent years the use of supervised methods has shown promising results in segmentation tasks. Supervised machine learning algorithms build mathemat-ical models from a set of labeled data. Example of supervised techniques are convolution neural networks (CNN), support vector machine (SVM) and naive Bayes model. Due to the ability of CNN to capture pattern of data in challenging datasets, they have been used in many works. For instance, [15] increases the receptive field by applying 7 × 7 filter size on fully convolutional network (FCN), this results in an outstanding segmentation performance of 99.7% accuracy on feline calicivirus images. In [16], [17], in order to tackle the challenge of imbalance between the foreground and background, a dice coefficient is ap- as SegNet [20] and U-net [21] which are capable of working in small number of datasets, gives us a suitable option for our dataset. Moreover, SegNet shows more superiority for having few parameters and hence faster to train, because it passes pooling indeces to the upsampling layers and does not use the heavy deconvolution layers. U-net has been applied in many works for segmentation of EM. Nevertheless, to the best of our knowledge no any work has been done on segmentation of EMs using SegNet, except for one work which uses SegNet directly without any network changes from the original one on sementation of yeast cells [22]. Thus, in this paper we attempt to leverage SegNet for segmentation of weakly visible microorganisms.

Feature Extraction and Pairing of Features
Feature extraction is an important stage in the image processing pipeline. In most cases features are used in image classification and object matching works such as [23], [24] and [25]. Mainly there are two categories of feature extraction methods, hand crafted and feature learning [26], as indicated in table 2.
Hand crafted features are manual features which are extracted based on prior knowledge. For example, color (ie. RGB, HSV, LAB, HUE color modes), texture which is defined by the spatial distribution of pixels in the neighbourhood  [27], [28], [29], [30] Feature Learning Deep learning (VGG-16, AlexNet, ResNet), BoVW [31], [32], [33], [34] of an image (ie. energy, entropy, homogeneity, correlation, and contrast [35] [42], because they replicate the feature extraction capability of visual cortex in human brain [43]. However, because of the orderless representation of local features in it, it does not achieve maximum performance. To remedy that and improve the performance of BoVW, some studies have considered spatial arrangement of features to discover higher order in BoVW for object matching and classification [44], [45]. Among the methods of arranging spatial features is by pairing of close visual words [46]. For instance, in [34] and [47] pairing is done on visual words (where Prior to pairing, feature descriptions are mapped to the visual words, and then pairing is carried out on the visual words). Yet, the underlying distribution of pairs of neighboring local feature descriptors appears to be ignored by the pairing of visual words. To address that, [32] and [33] suggested that the pairing of spatial close local descriptors (such as SIFT) can be done before the building of BoVW. This seem to achieve maximum improvement on classification of challenging dataset. Motivated by the concept of pairing features and to the best of the authors' knowledge, there is no any work which has been done on pairing of deep learning features for segmentation task, thus in this study we pair deep learning features generated from corner interest points' locations and concatenate them to the base model for segmentation of weakly visible EMs.

Methods
This section desctribes in details the novel techniques used in this paper.
The main focus being on tackling the segmentation challenges on weakly visible EMs. These are EMs which show poor segmentation results in our initial tests using the original base model SegNet. Example of weakly visible EMs are shown As observed in figure 2, weakly visible EMs suffer from low contrast, transparency and indistinct boundary between background and foreground. To be able to achieve better segmentation results, the following techniques are applied.

SegNet
SegNet is one of the powerful models in computer vision for semantic segmentation [20]. It consists of the encoder and decoder, as shown in figure 3.

Feature Extraction
Due to the challenges on weakly visible EMs dataset, the base model misses fine information from images during training, which gives poor segmentation results when using SegNet alone. Therefore, we use external pairwise features to enhance the performance of the base model by combining the advantage of interest points' locations (hand crafted features) and deep learning features.
Specific techniques are describes below;

Shi and Tomas Intest Points' Location
In order to enhance the segmentation results, we choose to use corner interest points, because from test/initial experiments the base model misses tiny outer corners and boundaries on the weakly visible EMs due to low contrast and transparency on images. A corner is as a place or point in the image where a small change in location causes a significant change in intensity in both the horizontal (X) and vertical (Y) axes. It can also be described as the intersection of points on an object's contour edges that preserve significant object's features [49]. Shi and Tomas corners theorem is one the most superior corner theorems [50]. Simply the Shi and Tomas theorem operates on three steps; Firstly, it is to find the window which produces high variation in intensity with a small change in the X and Y -axis. Numerically, to find a window that can produce large variation, let the window be centred at (x, y) and an intensity at this point be I(x, y). I(x, y) is an individual intensity at a position which varies from 0 to 255 for gray level image. When the window is shifted by (u, v), the intensity at the new location will be I(x + u, y + v) and [I( is the difference in intensity due to shift. For a corner, this difference must be high. Therefore we maximize this term by differentiating it with respect to x and y. Letting w(x, y) be the weights of pixels over the rectangular or a Gaussian window, Then, E(u, v) which is the difference between the original and the shifted window, is defined as : Applying the Taylor series with only the first order, which is Rewritting the shifted intensity using the above formula: Let: d(x,y) dx = Ix , and, d(x,y) dy = Iy I x and I y are image derivatives in X and Y directions respectively. Then, Expanding the above equation, Taking u,v out and rewritting in matrix notation, the equation becomes; where, Where, M is a symmetric 2 × 2 matrix whose eigenvalues are used to determine whether the scanned window contains a corner. Secondly, Calculating the score value S associated with scanned window [50]. It is given by; where, λ 1 and λ 2 are eigenvalues of the matrix M .
Thirdly, is to determine points along the shift of the window that can be considered as corners. For the point to be considered as corner, the score value S must be greater than the specified value (if both the λ 1 and λ 2 are greater than the minimum threshold values respectively).     Figure 6 summarizes the process of deep learning features extraction.

Feature Pairing
To pair feature maps which have been extracted from the interest points' coordinates, we use the Delaunay triangulation theorem.

Delaunay Triangulation (DT) Theorem
DT theorem is one of the most robust graphical theorems for the representation of data. It is the triangulation theorem which forms triangles (Delaunay triangles) by connecting each data (coordinates) to its nearest neighbour, such that the circumcircle associated with each triangle does not contain a point in and (X 2 , Y 2 ). The corresponding feature maps of the patches centred at these two points be F 1 and F 2 .
The middle point coordinate X m , Y m is given by; The pairwise feature map F m which corresponds to middle point X m , Y m is given by; In average 36 to 43 pairwise features (F m) are formed from 10 original features for each image.

Joint Pairwise Feature Formation
At this stage, we join the features formed on the interest points' coordinates

Training, Validation and Testing Dataset
The dataset is divided into training, validation and testing in ratio 1:1:2 respectively. However, in order to overcome the overfitting due to small dataset and improve the performance of our segmentation models, we applying augmentation on all original weakly visible EM and their corresponding GT images. We

Experimental Environments
To conduct the experiments, we use a work station with Intel (R) Core(TM) i7-7700 CPU with speed of 3.60Hz. RAM of 32GB and NVIDIA GeForceGTX 1080 8GB. For implementation of the networks, we use python 3 and Keras framework with Tensorfow as backend.

Evaluation Metrics
In  Table 3, summaries definition of these metrics.  only. We found that the performances are better when the concatenation is only at one block either block 1, block 2, block 3, block 4 and block 5 only. The increase in the number of concatenation blocks simultaneously leads to oversegmentation. Thus, we focus our research on concatenation at one block for all other experiments which we present in this paper. We compare and examine the performance of the PDLF-Net on small dataset of weakly visible classes by treating each class alone. Table 4 shows the performance of the PDLF-Net with concatenation at different blocks and the original SegNet.     From the general figure 9, even though the number of dataset for training

Evaluation of the PDLF-Net on Augmented Dataset
In order to enhance the performance of PDLF-Net, we augment all the weakly visible EMs and their GT images. Then joint pairwise features are extracted from each image and concatenated to different blocks. Each block is trained and tested independently for each dataset class.   Comparing the observation performance from figure 11, the PDLF-Net shows better segmentation results. For instance, in data class DC3 and DC4 (2 nd and 3 rd rows from the top) SegNet in (c) has not been able to show the foreground while there is a good segmented output of the same image by PDLF-Net in (e) and (f). In DC8 (last row), SegNet over-segments the image while good visual results are observed by PDLF-Net when concatenation of pairwise feature is at block 2, 3 and 5. Generally, the visual results show great improvement of segmentation results when using PDLF-Net.

Evaluation of the PDLF-Net on Test Dataset
To evaluate more the effectiveness of the PDLF-Net, we examine it on the test dataset. The test dataset contains 480 images, which are twice in number to the training and validation sets. The average segmentation performance of the PDLF-Net on test dataset for all classes is shown in figure 12. The graph shows a comparison of both the test set and validation set for each block performance.   It can be observed from the graphs on figure 13 that the PDLF-Net performs better on all challenging images with an average improved performance of more than 2.00% against SegNet on each metric. This justify the capability of the PDLF-Net to capture more spatial features in noisy, transparent and low contrast images.

Training and Testing Time Evaluation
In this section we compare the training and testing time of the PDLF-Net against other well-known CNN based segmentation models as shown in Table 6.

Method's Limitations
It should be noted that, although the proposed method has shown potential on EMs, it focuses only on segmentation of one (single) or two microorganisms on the image and not biofilms. Example of the segmentation of two EMs on the same image can be seen in Fig. 11 in classes DC3 and DC4, while other classes contain only one EM on every image. In the future work, we will extend our scope to testing our novel method on microorganim dataset with more than two microorganisms, clusters, and biofilms. The results show that the proposed network upon improving the performance of the base mode SegNet, it can focus more on the foreground which can overcome the segmentation challenges on image such as noise and low contrast. Apart from being useful in segmentation of EMs, the proposed network can find more application in segmentation of brain tumor and breast cancer images.

Conclusion and Future Work
During initial experiments we tested the pairwise deep learning features on binary classification of two EMs classes using SVM. Promising results were obtained. Therefore, the pairwise features can also be suitable not only in segmentation tasks but also in classification and image matching works.
In the future work, we plan to use other superior convolution neural networks such as Inception, Xception and DenseNet for extraction of deep learning features to improve more our segmentation results.