{'fraction_non_alphanumeric': 0.0353047336165733, 'fraction_numerical': 0.012369816144677971, 'mean_word_length': 4.690629011553273, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Joint Triplet Autoencoder for Histopathological Colon Cancer Nuclei Retrieval

Deep learning has shown a great improvement in the performance of visual tasks. Image retrieval is the task of extracting the visually similar images from a database for a query image. The feature matching is performed to rank the images. Various hand-designed features have been derived in past to represent the images. Nowadays, the power of deep learning is being utilized for automatic feature learning from data in the field of biomedical image analysis. Autoencoder and Siamese networks are two deep learning models to learn the latent space (i.e., features or embedding). Autoencoder works based on the reconstruction of the image from latent space. Siamese network utilizes the triplets to learn the intra-class similarity and inter-class dissimilarity. Moreover, Autoencoder is unsupervised, whereas Siamese network is supervised. We propose a Joint Triplet Autoencoder Network (JTANet) by facilitating the triplet learning in autoencoder framework. A joint supervised learning for Siamese network and unsupervised learning for Autoencoder is performed. Moreover, the Encoder network of Autoencoder is shared with Siamese network and referred as the Siamcoder network. The features are extracted by using the trained Siamcoder network for retrieval purpose. The experiments are performed over Histopathological Routine Colon Cancer dataset. We have observed the promising performance using the proposed JTANet model against the Autoencoder and Siamese models for colon cancer nuclei retrieval in histopathological images.


Introduction
Image retrieval is one of the important problems of computer vision to retrieve the visually matching images from a dataset for a given query image [1]. The ranking of the images is generally carried out by matching the features of a query image with the features of dataset images. Thus, the performance of retrieval depends upon the quality of the features extracted from the images which should be discriminative, robust and low dimensional [2]. Several hand-designed features have been explored in the recent past, such as Local Binary Pattern (LBP) [3], Local Tetra Pattern (LTrP) [4], Multichannel Decoded LBP (mdLBP) [5], Scale Invariant Feature Transform (SIFT) [6], Interleaved Order-based Local Descriptor (IOLD) [7], etc. The hand-designed features have been also explored for biomedical image retrieval such as Local Wavelet Pattern (LWP) [8], Local Mesh Patterns (LMeP) [9], Local Bit-plane Decoded Pattern (LBDP) [10], Local Ternary Co-occurrence Patterns (LTCoP) [11], Local Diagonal Extrema Pattern (LDEP) [12], etc.
In the recent past, a paradigm shift has been observed from the hand-designed feature extraction to the data driven feature learning. Thanks to the recently emerged Deep learning [13] which facilitates the feature learning automatically from the data. Convolutional Neural Network (CNN) is a type of Neural Network designed to deal with the image and video data. AlexNet was the first CNN model developed for image classification problem in 2012. The imagenet visual recognition challenge was won by AlexNet in 2012 with a great margin as compared to the best performing hand-designed features. Since 2012, various CNN models have been proposed for different applications such as image classification [14], object detection [15], image segmentation [16], face recognition/retrieval [17], [18], face anti-spoofing [19], [20], facial micro-expression recognition [21], [22], hyperspectral image classification [23], [24], image-to-image translation [25], [26], [27], and many more. Sirinukunwattana et al. have introduced a Spatially Constrained Convolutional Neural Network (SC-CNN) for histopathological routine colon cancer (RCC) nuclei detection and recognition [28]. Moreover, they have also collected the RCC nuclei dataset which is used in this paper for the experiments in retrieval framework. Recently, Basha et al. have developed a RCCNet CNN model having less number of parameters for the classification of RCC Nuclei patches [29]. Rajpurkar et al. have developed a ChexNet CNN model over chest X-rays data for pneumonia detection [30]. Wang et al. have proposed a Text-image embedding network (TieNet) for recognising the thorax disease in chest X-rays [31]. Recently, the deep learning based methods are also proposed for biomedical image retrieval [32], [33], [34]. Gu and Yang have used the dense connection for multi-magnification hashing applied over histopathological images [35]. A very recently, Sun et al. have used the adversarial learning for lesion detection [36]. Li et al. have developed a dual-channel deep neural network to identify the antiviral peptides [37]. Autoencoder is one of the type of Neural Network which tries to learn the latent space through reconstruction process [38]. Basically, first the input image is projected to a latent space (i.e., feature space) using an Encoder network and then it is reconstructed from that latent space using a Decoder network. The loss between the original image and the reconstructed image is minimized using Stochastic Gradient Descent (SGD) Optimization to learn the Encoder and the Decoder networks. Krizhevsky and Hinton have used a deep autoencoder for content-based image retrieval [39]. Zhang et al. have used the stack of sparse autoencoder and fused its features for histopathology image analysis [40]. Leng et al. have utilised the autoencoder with CNN for 3D object retrieval [41]. Zhu et al. have also exploited the features learnt through autoencoder for 3D shape retrieval [42]. The autoencoder has been also used in medical area such as autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction [43], detection of interacting protein pairs via ensemble of autoencoder and LightGBM [44], Convolutional Autoencoders to study the Alzheimer's Disease [45] and Optimizing autoencoders based network to analyze health data [46]. The major drawback of such models is that they are completely unsupervised and not able to learn the discriminative features.
Siamese network is a supervised learning framework to learn the features from triplets [47]. A triplet contains three images with two from same class and one from different class. Siamese network tries to minimize the intra-class distance and maximize the inter-class distance. A pair-wise cosine loss and quantization loss is used by Cao et al. to learn the feature for image retrieval [48]. Further, they have introduced a HashNet model for retrieval [49]. A triplet ranking loss is used by Yao [54]. An asymmetric deep supervised hashing is proposed by Jiang and Li by incorporating asymmetric pairwise loss [55]. Recently, Wu et al. have proposed a deep incremental hashing network to learn the hash codes of new images without changing the hash code of existing images [56]. Siamese neural networks are also used for spinal metastasis detection [57]. The major problem with Siamese network is that the learnt feature is very specific to that dataset. Moreover, it is derived only from one-way mapping (i.e., no mapping from feature to image), thus the discriminative ability might be compromized over the unseen data.
It is observed from the literature that Siamese networks and Autoencoder networks have its limitation in terms of the generalizability and discriminativeness. We propose a Joint triplet Autoencoder Network (JTANet) to learn the Encoder network of Autoencoder as a Siamese network which is termed as a Siamcoder network in this paper. Basically the triplets are used for learning of Siamcoder network in a joint fashion. The main contributions of this paper are as follows: • A semi-supervised Joint Triplet Autoencoder Network (JTANet) is proposed by utilizing the Siamese and Autoencoder networks.
• A Siamcoder network is used as a common CNN for Siamese network as well as Encoder network of Autoencoder.
• A joint training is performed for supervised Siamese network and unsupervised Autoencoder network.
• The latent space output of Siamcoder network is used as the feature vector/embedding for the retrieval purpose.
• The joint training enhances the generalizability and discriminativeness of the latent space.
• The histopathological colon cancer nuclei retrieval experiments are performed by using the features derived from the learnt Siamcoder network of the JTANet model.
• The effect of different losses is analyzed through experiments.
The organization of the paper is as follows. Section II describes the proposed JTANet model; Section III presents the histopathological colon cancer nuclei retrieval framework using JTANet model; Section IV describes the experimental settings; Section V illustrates the experimental results and analysis; and finally, Section VI concludes the paper.

Proposed Joint Triplet Autoencoder Network
A Joint Triplet Autoencoder Network (JTANet) is proposed in this paper which combines the power of Siamese and Autoencoder networks. The proposed JTANet model is illustrated in Fig. 1. A Siamcoder network (S CN) (basically the encoder network of Autoencoder) is used to transform an image patch (I) into the feature/latent space/embedding (F). Note that the Siamcoder network is shared between the Siamese network (S N) and Autoencoder network (AN). The Siamcoder network is a Convolutional Neural Network (CNN) having different layers such as convolution, batch normalization, activation function, and max pooling. Basically, the non-linear transformation function of Siamcoder network is denoted as f S CN : R m×m×3 → R dim which converts the two-dimensional data of size m × m having three channels into an one-dimensional feature vector of size dim. Thus, f S CN can be seen as a hierarchy of some linear and non-linear functions. The Decoder network (DN) of Autoencoder is an Up-Convolutional Neural Network to reconstruct the output image from the latent space which is corresponding to the original image. It is also a non-linear function having different layers such as transpose-convolution, batch normalization, activation function, and upsampling. The transformation function for the Decoder network is denoted as f DN : R dim → R m×m×3 which transforms the one-dimensional feature F of size dim into an image patch R of size m × m with three channels. Note that R is the reconstructed image w.r.t. the original image I.
Consider {I A , I P , I N } as a triplet of patches with C I A = C I P and C I A = C I N where C I k represents the class label for image patch I k for k ∈ {A, P, N}. The image patches I A , I P , and I N are referred as the Anchor, Positive and Negative samples, respectively. A pictorial representation of triplet is shown in Fig. 2. The proposed JTANet network generates the triplets of image patches for a batch of input from its latent space using online triplet mining as portrayed in Fig. 1 at the training time. However, it only needs a single image at test time to extract the features. Thus, the features used by online triplet mining for a batch of images are computed by the same Siamcoder network. The feature vectors are normalized before online triplet mining.

Objective Function
Three losses, namely Autoencoder loss, Siamese loss and Feature Regularization loss are used as the objective function to train the JTANet model. Mathematically, the objective function of the proposed JTANet model is given as, where L JT A is the final loss function, L AE is the Autoencoder loss, L S M is the Siamese loss, L FR is the Feature Regularization loss and {λ AE , λ S M , λ FR } are the hyper-parameters as the weights for the Autoencoder, Siamese, and Feature Regularization losses, respectively.

Autoencoder Loss
The Autoencoder loss (L AE ) is computed between the original images I k and its corresponding regenerated images R k for a batch of input. The purpose of Autoencoder loss is to make sure that the relevant features are being learnt by Siamcoder network. It is ensured by reconstructing the image from feature/latent space. Thus, it makes sure that Siamcoder network should not learn the random features. Mathematically, L AE is given as, where B is the batch size (i.e., the number of samples in a batch) and L AE i is the Autoencoder loss between the i th input image patch I i and its corresponding reconstructed image patch R i for a batch. The L AE i is computed as the mean square error (MSE) between I i and R i and given as, where I i (u, v, c) and R i (u, v, c) denote the values in the image patches I i and R i , respectively at image co-ordinate (u, v) in c th channel and ||.|| 2 avg represents the mean square error (MSE).

Siamese Loss
The online triplet mining uses the features derived from Siamcoder network to generate the triplets of the Anchor, Positive and Negative samples. Consider, {F A , F P , F N } as the Anchor, Positive and Negative triplets of Siamcoder network generated features corresponding to the Anchor, Positive and Negative image patch triplet {I A , I P , I N }, respectively. The Siamese loss (L S M ) is computed by using the {F A , F P , F N } features corresponding to the Anchor, Positive and Negative samples. The purpose of Siamese loss is to decrease the distance between the features of Anchor and Positive samples and to increase the distance between the features of Anchor and Negative samples. By doing so, it forces the Siamcoder network to learn the class specific features such that the features for samples of same class are closer to each other and the features for samples of different class are apart from each other. The L S M is given as, where L S is defined as, where L S is the Siamese loss, L intra S is the Siamese intra-class similarity loss, L inter S is the Siamese inter-class similarity loss, and λ m is a margin hyper-parameter. The Siamese intra-class similarity loss is computed between the features of the samples of the same class (i.e., F A and F P ). Similarly, the Siamese interclass similarity loss is computed between the features of the samples of the different classes (i.e., F A and F N ). These losses are computed as, and where F i A , F i P , and F i N are the feature vectors of size dim derived using Siamcoder network for i th triplet having Anchor (I i A ), Positive (I i P ), and Negative (I i N ) image patches, respectively, nb is the number of triplets generated by online triplet mining for a batch of input and ||.|| 2 represents the sum of square distances (SSD).

Feature Regularization Loss
The Feature Regularization loss (L FR ) is computed from the Siamcoder network's output features (F i | 1≤i≤B ). The purpose of Feature Regularization loss is to increase the generalization ability of features being produced by the the Siamcoder network. It is computed as, where L i FR | 1≤i≤B is the Feature Regularization loss over the feature vector F i and defined as, by calculating the L 2 -Norm of feature vector F i for i ∈ [1, B].
We use Stochastic Gradient Descent (SGD) based optimization techniques called as Adam to train the Siamcoder and Decoder networks of the proposed JTANet model. The weights of any network in deep learning are generally trained by updating it during backpropagation of training using SGD based update rules [58], [59].

Routine Colon Cancer Nuclei Retrieval using Proposed JTANet Model
The proposed JTANet model is trained using the Routine Colon Cancer (RCC) Nuclei triplets. The training is performed for Siamcoder and Decoder networks in joint fashion using Autoencoder, Siamsese, and Regularization losses. Once the JTANet model is trained, only Siamcoder network is required to extract the features from any image patch as depicted in Fig.  3. A training feature database is created as follows, where N train is the total number of images in the training set, I train i,C i is the i th image in the training set having C i as the class label, f S CN is the Siamcoder network function to transform an image patch into a feature vector, and F train i,C i is the extracted features for image patch I train i,C i . Consider I q as a query image patch for which we want to retrieve the best δ number of images from training set. The features F q are also extracted for input query I q using the same trained Siamcoder network to facilitate the feature matching as, Note that for the experimentation purpose, the query image patch I q is taken from the test set of the Routine Colon Cancer Nuclei database. Thus, q = 1, 2, 3, ..., N test where N test is the number of images in the test set. The distance between features of query image F q and training images F train i,C i for i = 1, 2, ..., N train is computed to retrieve the best δ images from training set based on the distance ranking in increasing order as illustrated in Fig. 3. The Euclidean distance is computed between the features as follows:   The performance of the model is computed by considering each image patch in test as the query image one by one. Consider I test j,C j | j=1,2,...,N test as the j th sample in the test set as the query patch q from C th j class where N test is the total number of RCC image patches in the test set. The feature extracted using trained Siamcoder network for test image patch I test j,C j is F test j,C j . The mean precision is computed as, where Pr j is the precision when j th test sample is considered as the query and computed as, where #CR j and #T R j are the number of correctly retrieved and total retrieved images for a query.

Experimental Settings
This section is devoted for the acrchitecture details, dataset description, triplet generation and hyper-parameter settings.

Architecture for Siamcoder and Decoder Networks
The proposed model uses Siamcoder and Decoder networks for feature extraction and image patch reconstruction, respectively as depicted in Fig. 4. The Siamcoder network consists of seven convolution layers. The Decoder network also consists of seven deconvolutional layers. The layer-wise details of the proposed JTANet model is summarized in Table 1.

Routine Colon Cancer Dataset
A CRCHistoPhenotypes -Labeled Routine Colon Cancer (RCC) Cell Nuclei Dataset 1 is used in this paper for the experiments [28]. This dataset contains 100 H&E stained histology images of colorectal adenocarcinomas. The co-ordinates of 22,444 RCC nuclei is also provided with this dataset along with its associated labels. Basically, this RCC dataset has 4 classes which are named as, epithelial, fibroblast, inflammatory and others. We use the co-ordinates of nuclei cells to extract the 32 × 32 patches around the cells. The image patches 1 https://warwick.ac.uk/fac/sci/dcs/research/tia/data/crchistolabelednucleihe/ are distributed unevenly among four classes (i.e., epithelial-7, 722 patches, fibroblast-5, 712 patches, inflammatory-6, 971 patches, others-2, 039 pactches). The dimension of the RCC image patches is 32 × 32 having 3 channels. The sample image patches of RCC dataset is shown in Fig. 5. The dataset is split into train and test sets having 20, 444 and 2, 000 image patches, respectively.

Triplet Generation
Online triplet mining [60] is used in this work to generate the triplets. Only training set is used to generate the triplets as it is needed only at training time. First, the Anchor-Positive pairs are generated using the distance matrix computed from the embeddings extracted by the siamcoder network. Next, a suitable negative sample for each Anchor-Positive pair is selected to form the triplet. The negative sample is taken from any other class than anchor class using the negative selection function. A triplet score is passed as the input to the • Hard Negative: The difference between the distances should be greater than zero. The negatives sorted in descending order of the score values and the negative with largest score value is selected for an anchor-positive pair.
• Semi-hard Negative: The difference between the distances should be greater than zero and at the same time it should not exceed the margin. All such negatives satisfying the condition are considered and one of them is selected as the suitable negative for a given anchor-positive pair.
• Random-hard Negative: A combination of both hard and semi-hard triplets is considered. The negatives are randomly selected which satisfy a common condition that the difference in the distances should be greater than zero.

Hyper-parameter Settings
The experiments are conducted using the PyTorch framework. The Adam optimiser is used with 0.001 learning rate for 50 epochs. The batch size is set to 256. The input patch dimension is resized to 64×64×3. In random-hard and semi-hard negative selections, the margin threshold value (λ m ) used is 0.5. The default weight factors/coefficients for all the loss functions are used as 1 untill or otherwise specified.

Experimental Results and Analysis
In this section, the performance of the proposed JTANet model is reported and analyzed for different settings under retrieval framework. First, we report the experimental results for Hard, Random Hard and Semi Hard triplet selection strategies. Then, we conduct the convergence analysis and finally, the loss weight coefficient analysis is done.

Results using Different Embedding Lenghts
The experimental results in terms of the mean retrieval precision is demonstrated in Fig. 7. The mean precision values are plotted against the number of images retrieved from 5 to 100. The embedding lengths (i.e., the latent space feature dimension) used are 2,4,8,16,32,64,128,256, 512 and 1024. The plots in Fig. 7a, Fig. 7b, and Fig. 7c correspond to the Hard, Random Hard, and Semi Hard negative selection strategies of triplet generation, respectively. Generally, the mean precision decreases with increase in the number of retrieved images.
The best performance is achieved for 1024 embedding length in case of Hard selection. Whereas, the best performance is observed for 512 embedding length in case of Random Hard and Semi Hard selections. The possible reason of such behaviour is associated with the difficulty of the triplets which is more in case of Hard selection based strategy. It is also noticed that the mean precision is lowest for embedding length 2 across all the plots. Moreover, the perfromance is lower with smaller embedding lengths in most of the cases due to the limited discriminative power of less features. The mean precision for Randon Hard and Semi Hard strategies is better than Hard strategy. The mean precision values for 5 number of retrieved images using the proposed JTANet method is also reported in Table 2 using Hard, Random Hard and Semi Hard strategies. It shows the suitability of the proposed model for routine colon cancer patch retrieval task.

The Convergence Analysis
In order to understand the training behaviour of the proposed JTANet model, the convergence analysis is done in this paper. Fig. 8 shows the loss values for autoencoder loss, siamese loss, feature regularization loss, and total loss in terms of the number  of the training iterations. The loss curves in Fig. 8a, Fig. 8b, and Fig. 8c

The Loss Weight Coefficient Analysis
The high precision using the proposed JTANet model is observed due the different losses used, such as autoencoder loss, siamese loss and feature regularization loss. The total loss is computed as the weighted combination of above mentioned three losses. In earlier experiments, all the weight coefficients are set as 1. In this experiment, the peformance comparison is done by varying the weight coefficients for different losses in order to compute the total loss. The mean precision (%) using the proposed JTANet model by varying the weight settings of different losses for 5 number of retrieved images is reported in Table 3. The feature lengths used for Hard, Random Hard and Semi Hard triplet selection strategies are 1024, 512 and 512, respectively in this experiment.
The different weight coefficient combinations for Autoencoder Loss (AE), siamese loss (SM) and feature regularization loss (FR) (i.e., AE:SM:FR) are 1:1:1, 5:1:1, 1:5:1, 1:1:5, 10:1:1, 1:10:1, 1:1:10, 0:1:1, 1:0:1 and 1:1:0, respectively. From Table. 3, we notice that when the autoencoder loss is missing (i.e., the weight values are 0:1:1), the mean precision is best for Hard and Random Hard triplets. However, the autoencoder loss is influential with Semi Hard triplets. When the weight values are 1:5:1 (i.e., more weight given to siamese loss), the mean precision is best for Semi-Hard triplets. The biased weighted highest mean precision is improved as compared to the ideal weighted (1:1:1) mean precision by 11.65%, 0.60%, and 0.97% for Hard, Random Hard and Semi Hard selection strategies, respectively. It is also observed that very high weight for siamese loss (i.e., 1:10:1 setting) lowers the performance for Hard selection. Whereas, it is evident from 1:0:1 setting that the siamese loss is the most important loss for Random Hard and Semi Hard triplet selection strategies. We can say from this experiment that following is the relevancy of different losses: siamese loss > feature regularization loss > autoencoder loss in the proposed JTANet model for routine colon cancer patch retrieval.

Conclusion
This paper proposes a joint triplet autoencoder network (JTANet) for histopathological colon cancer nuclei retrieval. The proposed JTANet is a joint venture of the autoencoder and siamese networks. The encoder network of autoencoder is shared with the siamese network. The main aim of the proposed model is to learn more discriminative, robust and efficient feature embeddings for the retrieval task. In order to achieve it, three losses, namely autoencoder loss, siamese loss and feature regularization loss are used. The siamese loss is computed from the triplets which is generated from the embeddings itself using Hard, Random Hard and Semi Hard strategies. The image retrieval experiments are conducted over histopathological colon cancer nuclei dataset. The experimental results suggest that the Semi Hard triplet selection method with 512 embedding length is the most suitable for JTANet with 1:5:1 weighting between autoencoder, siamese and regularization losses. It is also observed that the proposed JTANet model exhibits the faster convergence property. The autoencoder loss is not important with Hard triplet selection strategy. It is also noticed that the siamese loss dominates over other losses in Random Hard and Semi Hard strategies. The experimental results confirm the suitability of the proposed JTANet model for RCC patch retrieval.