{'fraction_non_alphanumeric': 0.03545265796999256, 'fraction_numerical': 0.01739334821953226, 'mean_word_length': 4.710961968680089, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives

The volume of available data has grown dramatically in recent years in many applications. Furthermore, the age of networks that used multiple modalities separately has practically ended. Therefore, enabling bidirectional cross-modality data retrieval capable of processing has become a requirement for many domains and disciplines of research. This is especially true in the medical field, as data comes in a multitude of types, including various types of images and reports as well as molecular data. Most contemporary works apply cross attention to highlight the essential elements of an image or text in relation to the other modalities and try to match them together. However, regardless of their importance in their own modality, these approaches usually consider features of each modality equally. In this study, self-attention as an additional loss term will be proposed to enrich the internal representation provided into the cross attention module. This work suggests a novel architecture with a new loss term to help represent images and texts in the joint latent space. Experiment results on two benchmark datasets, i.e. MS-COCO and ARCH, show the effectiveness of the proposed method.


Introduction
A plethora of real-world applications is emerging due to the rapid expansion in the volume and variety of data on the one hand and the recent success of deep learning on the other hand. However, the majority of existing models can only work with a single modality of data, e.g., only analyzing images or text. This restriction on the data source prevents the models from having a more generalizable and robust problem representation for downstream tasks. The absence of a model that can be used across several modalities and that can use information from multiple sources of data is in great demand and has become increasingly common in both academia and industry. One task that has gained popularity over the years is retrieving the relation between one of the data modalities; a task called "cross-modality retrieval". This task seeks to bring a pair of data from two distinct sources together by recognizing their intrinsic relationship. Because such cross-modal topologies need to learn the inter-modal correspondence and modality representation separately, their design and training can be rather challenging. Image and text are the most extensively utilized and commonly observed modalities in real-world data. However, The diversity in information sources can be observed more commonly in the medical field. Images, reports and molecular data in patients' medical records are different and indispensable sources of information. A model able to retrieve a source of data when a different one is available can help provide complete information about the patient.
The common solution to approach to explore the relationship between image and text is to map the visual semantic embeddings (Karpathy and Fei-Fei, 2015;Frome et al.) of an image and the corresponding words/phrases/sentences into a common latent embedding space (Barnard et al., 2003;Berg et al., 2004;Socher and Fei-Fei, 2010;Chong et al., 2009). In these methods, the goal is generally to find a common space where the corresponding representations of images and text are as close as possible, hence making recognizing their relationship easier. Recent studies investigate the use of the attention mechanism to understand the similarity between the two modalities. The majority of studies in this category have employed cross-attention mechanisms, which allow the model to selectively attend to the parts of an instance that are relevant to the context from the other modal (Lee et al., 2018;Huang et al., 2017). Other methods attempt to refine their representation regarding the information of the other modal (Chen et al., 2020). Nonetheless, due to the gap between the representation of images and texts, these methods may not find the optimal representation. Moreover, ambiguity in text documents is a common challenge posing a learning obstacle to the model if it uses an injective embedding. Aside from that, humans use a hierarchical structure to organize and store diverse semantic concepts. However, the majority of the currently available approaches group semantics together in a consistent manner (Lee et al., 2018;Chen et al., 2020;Song and Soleymani, 2019).
In this study, the mentioned issues are addressed by providing an iterative regime to capture related information in the other modality and extract the most significant features by looking into its context. It can help the model capture the information related to the other modality and itself simultaneously. Accordingly, such an approach can help extract richer latent embeddings for each instance. Moreover, training the model in an iterative regime can help the model to capture higher-level features gradually based on the features acquired in the previous steps. Furthermore, refining the extracted features based on the intersection between modalities can help the model to enrich its representations.

Literature Review
Many works on cross-modality retrieval have been looking for a similar pattern in the feature representation of both modalities (Karpathy and Fei-Fei, 2015;Lee et al., 2018;Huang et al., 2018;Lu et al., 2020;Radford et al., 2021). These studies can generally be categorized into global and local feature matching approaches.
Global feature matching methods were among first attempts to solve the image-text retrieval task. Kiros et al. (Kiros et al., 2014) proposed to use a CNN and a RNN to encode given images and text data. For the loss function, they proposed an idea to implement a pairwise ranking loss objective to rank images and correspondence descriptions. Faghri et al. (Faghri et al., 2017) modified the loss function with hard negatives in the triplet loss function and applied on a similar architecture to (Kiros et al., 2014). Furthermore, by improving the quality of generative models (Zhu et al., 2017;Karras et al., 2019), some studies have attempted to exploit the capability of generative models to enhance the performance of proposed methods (Patrick et al., 2020;Peng and Qi, 2019;Gu et al., 2018).
Local feature matching methods consider the correspondence between image and text feature representations at the level of image regions and words. Karpathy et al. (Karpathy and Fei-Fei, 2015) proposed a method to extract features for image regions at object level using Faster R- CNN (Ren et al., 2015) and text words, and then align them into a common space. However, as the application of the attention mechanism in a variety of tasks has become more common, several approaches in image-text retrieval seek to leverage the attention mechanism (Chen et al., 2020;Lee et al., 2018;Song and Soleymani, 2019;Thomas and Kovashka, 2020;Lee et al., 2018). Yale et al. (Song and Soleymani, 2019) proposed a method to combine global features with locally-guided features via a multi-head self-attention module. Christopher et al. (Thomas and Kovashka, 2020) in a similar network proposed a novel within-modality loss function that drives feature representation toward more coherence. The majority of these methods rely on the premise that vision and text are mutually exclusive and equally important. However, grounding (or base) representation of each modality derived from another modality to finer details is critical for bridging the gap between vision and text modality. Kuang-Huei (Lee et al., 2018) was one of the first studies that investigated the use of cross-attention to explore full latent matching using image regions and words in the text as context.
Furthermore, Hui et al. in IMRAM paper (Chen et al., 2020) presented an iterative alignment method that captures the fine-grained correspondence between image and text progressively using a cross-attention module. By the development of Transformer models, serious approaches were proposed for cross-modality retrieval task using Transformers (Lu et al., 2019;Tan and Bansal, 2019). Authors in (Lu et al., 2019) proposed a network architecture that consists of two single-modal Transformer networks applied on input images and texts data respectively.
Recent studies have shown that training on large-scale datasets could help models to achieve significant improvement in image-text retrieval task (Li et al., 2020;Radford et al., 2021;Jia et al., 2021). Jia et al. in ALIGN paper (Jia et al., 2021) leveraged a dataset of over one billion paired image and alt-text data and claimed that a simple dual-encoder network could learn to align image and text representation using a contrastive loss. Similar to ALIGN paper, Xiujun et al. in Oscar paper (Li et al., 2020) collected a dataset over 6.5 million paired of image and text to train their network. Another example for boosting the performance using a large amount of data can be CLIP (Radford et al., 2021) paper. Authors in CLIP paper also demonstrated that the task of image-text retrieval could be outperformed using a massive dataset, 400 million in this case. However, a bottleneck of these large-dataset-dependent approaches emerges in those domains which are unlikely to be included in their collected dataset, such as histopathology domain. Moreover, training on that amount of data requires huge computational resources.
Cross-Modality Retrieval is new to the field of histopathology. There are not many studies that explore retrieving a description correspondence to an image or vice versa. As a result, a few works were trying to retrieve or utilize image-text modalities in their suggested methods. Zhang et al. (Zhang et al., 2017) was among the first authors who proposed a model and dataset for image-text in pathology. Authors proposed a new framework, namely MDNet, to establish a direct multi-modal mapping between images and diagnostic reports. Moreover, they developed a private patch-based dataset based on pathology bladder cancer images. For descriptions, they asked pathologists to write 5 short sentences for each patch.
The proposed method is capable of generating short diagnostic reports and retrieving images based on symptom description. Yet the developed dataset was too small, private and limited to only H&E images. Recently, Gamper et al. (Gamper and Rajpoot, 2021) developed a new publicly available dataset, ARCH, that contains more than 7,000 patch-based images with the corresponded captions. The authors of the paper used PubMed medical articles (Pub) and academic pathology textbooks to collect the ARCH dataset.

Methodology
The proposed model, as demonstrated in Figure 1 includes different components. LILE, a dual attention network using Transformers, takes images and texts as inputs and extracts feature representations for each of those using Transformers. Then, a self-attention module (Vaswani et al., 2017), which is applied for extracting the most significant parts of each modality with respect to itself, is applied. For the next step, a cross-attention module and gated memory block are applied to help the model refine the representation of each instance with respect to the outputs of attention modules. Additionally, an iterative matching scheme using a gated memory block is applied to refine the extracted features for each modality. Data representation, or representation learning (Bengio et al., 2013), is a collection of approaches in ML that allows a model to automatically discover the representations required from the given data. Recently, Transforms become more prevalent for representation learning. the Transformers' modular architecture enables the processing of different modalities (e.g., images, videos, text, and voice) leveraging similar processing blocks. It scales efficiently to huge capacity networks for complex tasks and performs well with massive datasets. Transformer architecture is chosen for text and image feature extractors in this study due to these advantages. Image Representation: In this study, a pre-trained ViT (Dosovitskiy et al., 2020) is implemented to encode input images . Each image is split into a sequence of fixed-size, nonoverlapping image patches before being fed to the ViT. Utilizing object detection models is another method for extracting visual information from an image. These models are trained on the input image in order to extract the objects. As a result, the input image can be represented as a set of extracted feature maps for all of the objects in the image (Lee et al., 2018;Chen et al., 2020). Depending on the dataset and the availability of annotations for the object detection task, one of these approaches is used in this study.
Text Representation: Text representation can be defined as a method for encoding sentences into vectors. Transformers have recently emerged as the top-performing solution for the majority of NLP tasks and outperform many other approaches (Liu et al., 2019;Radford et al., 2019;Usi). In this study, a pre-trained "Roberta" architecture (Liu et al., 2019) which is a Transform-based model is deployed to encode the input text. It receives a text description as an input and returns a feature map that represents the given data.

Attention and Gated Memory Blocks
After the representation for each modality instance has been extracted, a multi-head selfattention module is applied to obtain m enhanced feature maps for extracted features from the previous stage similar to the (Wei et al., 2020). These m representations highlight the most significant features of each modality in relation to itself.
A cross-attention mechanism is utilized to attend to distinct parts of one modality given the context of another modality as proposed by authors in (Chen et al., 2020;Wei et al., 2020). The application of cross-attention in the proposed approach is to attend to image regions regarding the text input tokens and vice versa. The gated memory block seeks to refine the extracted features from each modality considering the cross-attention feature maps. The input modalities are denoted as 1, m], y i ∈ R d } which X and Y can be either image or text features, where m is the number of attention heads in the multi-head self-attention module in the previous step. The cross-attention module helps to have the highlighted information for modality X related to modality Y . To achieve this goal, a suitable measurement is needed to quantify the similarity between each feature map and feature maps in another modality.
In the suggested solution, the cosine similarity is applied (Ji et al., 2019;Jia et al., 2021;Lee et al., 2018). The similarity between each instance in modality X and Y is determined as Eq.1, where s ij denotes the similarity between the i-th feature map for modality X and the j-th from modality Y. Furthermore, it is beneficial to threshold the similarity at 0 and normalize it (Karpathy et al., 2014;Lee et al., 2018).
To attend on set Y with respect to a given feature x i in X, a weighted combination of y j is defined. The definition of attention function in Eq.1 is a variant of the "dot product attention" that is commonly used in other studies (Luong et al., 2015): . ( In Eq.2, the parameter λ is the inverse temperature of the softmax function (Chorowski et al., 2015). A more smooth attention function can be achieved by adjusting λ. A x is defined as {a x i ∈ [1, m], a x i ∈ R d } where each element of A x captures significant parts of each x i given the whole Y set as context. To refine the extracted features of X regarding the important parts of each x i given Y , a memory unit has been used. It would dynamically update and refine the feature maps of X by looking to both A x and X as Eq.3, where f (·) can be defined differently (Kalra et al., 2020;Weston et al., 2014;Burtsev et al., 2020). In this study, a gated mechanism has been adopted for f (·) as follows:

Iterative Matching
As stated in section 3.2, gated memory can assist the model in refining the feature representation regarding the shared information between two modalities as proposed by authors in (Chen et al., 2020). Having the gated memory in an iterative scheme can be beneficial as each iteration step, with the help of A x feature representation of X can be re-calibrated. The iterative scheme can be summarized as Eq.4, where k is the iteration step that will be performed to refine the alignment for the next iteration: At iteration step k, the similarity score between image I and text T is calculated as follows: where α is a learnable scalar weight parameter that balances the influence of the similarity score terms. S (v,v− →T ) (v i , T ) and S (w,w− →I) (I, w i ) are defined as similarity score between image regions and text T and text tokens and image I respectively proposed by authors in (Chen et al., 2020) . These similarity scores are derived as The similarity score can be boosted by including directly the similarity between image and text as S (v,T ) (v i , T ) = sim(v i , T ) and S (w,I) (I, w i ) = sim(I, w i ).
This can assist the model in preserving the semantic meaning of each instance while it attempts to bring paired instances closer together. To put all k steps together, the similarity score between image I and text T will be derived as Eq.7, where k is the number of matching steps that will be set as a hyper-parameter. See appendix D.

Loss Function
In this study, N-pairs bi-directional triplet-loss is implemented.
In Eq.8, ∆ is a margin value and [x] + = max(0, x). The term S(I, T ) is defined in Eq.7 and measures the similarity between image I and text T . This similarity score forms a similarity score matrix S of size n × n, where S is symmetric, See appendix A. In the training phase, n is the size of mini-batch. Images and text with a same subscript are paired instances which means the diagonal of the matrix should have the largest value compared to the other indices. This regime will be trained in an end-to-end manner.

Experiments
We used two benchmark datasets to evaluate the effectiveness of the proposed method: • MS-COCO (Lin et al., 2014), a large-scale object detection, segmentation, key-point detection, and captioning dataset, and • ARCH (Gamper and Rajpoot, 2021), a computational pathology multiple-instance captioning dataset, see appendix B.
To compare the proposed method with other approaches, Recall at K (R@K) is used, which measures the fraction of queries retrieved correctly among top K search results (Chen et al., 2020;Lee et al., 2018). To show the efficiency of the proposed method, the reported results also includes "R@sum" which is the summation of evaluation at different K, i.e., R@1+R@5+R@10 as in (Huang et al., 2017). The results are shown in Table 1 and Table 2 for MS-COCO and ARCH datasets respectively. The proposed method, called LILE, outperformed the previous best model, i.e., IMRAM (Chen et al., 2020), by a large margin of 6.3 and 8.3 in terms of overall performance R@sum in MS-COCO(1K) and MS-COCO(5K) datasets, respectively. Because there is no previously reported results on the ARCH dataset, publicly available methods like IMRAM (Chen et al., 2020), PVSE (Song and Soleymani, 2019) and CLIP (Radford et al., 2021) were selected and modified to evaluate them on the ARCH dataset. LILE significantly outperformed the best previous best model, i.e., IMRAM, with a large margin of 28.1 in terms of overall performance R@sum. These results demonstrate the effectiveness of the proposed approach for the cross-modality retrieval task. The implementation details described in appendix C.
The cross-modal retrieval task can benefit greatly from the use of large amounts of collected data and powerful computational resources. Consequently, methods trained on large-scale datasets (i.e., CLIP (Radford et al., 2019), ALIGN (Jia et al., 2021), and OSCAR (Li et al., 2020)) can not be directly compared to methods other methods and LILE which trained exclusively on smaller datasets such as MS-COCO. The values for those methods trained on massive datasets are reported to highlight latter point.
Additionally, the results on ARCH dataset show even more improvement compared to MS-COCO results. One of the reasons that may have influenced this enhancement could be the use of Transformer for encoding the input data as IMRAM (Song and Soleymani, 2019) and PVSE (Song and Soleymani, 2019) applied GRU for their text encoding. Moreover, analyzing the results on ARCH dataset can prove that methods like CLIP (Radford et al., 2021) which have been trained on massively large datasets cannot perform very well on tasks like image-text retrieval in pathology. This type of tasks are specialized and apparently need more sophisticated methods like the proposed framework.

Conclusion
The proposed cross-modality approach can simultaneously capture the most salient features of each modality in relation to itself and other modalities. A multi-head self-attention module was used to aid the network in gaining a better understanding of each modality. Meanwhile, to assist the model in identifying all potential alignments between two modalities, the multi-head self-attention module output is fed into a cross-attention module. This approach can aid the model in adjusting retrieved features from one modality considering the effect of the other one. Additionally, the suggested novel loss objective can assist the model in determining the optimal weight to balance both implicit and explicit sources of information used to match paired instances in different modalities. Experiments on the MS-COCO and ARCH datasets demonstrated the efficiency of LILE in terms of recall, a typical metric for retrieval tasks in both general-purpose and histopathology contexts. Papillary intralymphatic angioendothelioma. This is a dermal or subcutaneous lesion of children or adults, with intravascular growth of cells that have a lymphatic endothelial immunophenotype. Some have adjacent lymphangiomas or clusters of lymphatic vessels.The earlier terminology was malignant endovascular papillary angioendothelioma of childhood , but in a more recent series, reported cases did not recur or metastasize High PR expression (3+) in a leiomyoma,(PR x400) Cortex in a renal biopsy specimen from a man of 38 known to have membranous nephropathy, shown by a biopsy nearly 2 years before this one. The nephrotic syndrome had persisted since the first biopsy, and acute renal failure had developed 1 month before this biopsy. There is now widespread uniform tubular atrophy, which suggests longstanding renal vein thrombosis. This was confirmed by radiologic investigation. Renal function did not recover Figure 3: Three samples from the ARCH dataset. As it can be seen, images are from different staining and and normalizations with varied description structure.
the last two blocks of image encoder were trainable and the rest of the network were frozen.
For the text encoder, the network weights are initialized from the BioMed-RoBERTa-base (Gururangan et al., 2020) which had been trained on 2.68 million scientific papers from Semantic scholar corpus (Sem). This model is a language model based on RoBERTa-base architecture (Liu et al., 2019) which includes 12, 768 hidden dimensions with 110M learnable parameters. Same as image encoder, only 2 last layers of text encoder were trainable and the rest of the network were frozen. The challenging part for ARCH dataset as it shown in Figure 4 is related to the descriptions length. The descriptions are mostly non-uniform, i.e., ranging from 2 tokens to 484 tokens and much longer compared to MS-COCO dataset. To address this problem, each description is split into sentences. Then, concatenation of the first sentence with every other sentence are considered as new individual descriptions. By doing this, each image can have one or more than one description which help to augment the data and uniform the descriptions. For experiments using MS-COCO dataset, the same architectures are implemented for both image encoder and text encoder. The only differences are the weight initialization applied for these networks and the number of layers that set to be trainable. The image encoder weights are initialized from a pre-trained model that trained on a classification task on ImageNet-1k dataset (Deng et al., 2009), which has 1 million images with 1,000 classes. Moreover, the input data for image encoder is the feature maps that had been extracted using Faster R-CNN as explained in section method section. The last 6 layers of the image encoder were trainable and the rest were frozen. The text encoder weights are initialized from a pre-trained Roberta-base network that had been trained on the reunion of five general datasets 1 . The last 6 layers of the text encoder were trainable and the rest were frozen.
All the experiments are implemented in Pytorch V1.3 and run with two NVIDIA V100 GPUs. For the conducted experiments, images were resized to 224 × 224 pixels and normal-  ized before feeding into the model. The hyper-parameter m, which specifies the number of self-attention heads, is set as 16 for optimal performance across all experiments. The Adam optimizer (Kingma and Ba, 2014) is used with an initial learning rate of 1e−4. The learning rate is decreased when the metric has stopped improving. The margine for contrastive loss function set to be 0.2.