{'fraction_non_alphanumeric': 0.03811056297770532, 'fraction_numerical': 0.014100908301750969, 'mean_word_length': 4.564561734213006, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR

The efficacy of zero-shot sketch-based image retrieval (ZS-SBIR) models is governed by two challenges. The immense distributions-gap between the sketches and the images requires a proper domain alignment. Moreover, the fine-grained nature of the task and the high intra-class variance of many categories necessitates a class-wise discriminative mapping among the sketch, image, and the semantic spaces. Under this premise, we propose BDA-SketRet, a novel ZS-SBIR framework performing a bi-level domain adaptation for aligning the spatial and semantic features of the visual data pairs progressively. In order to highlight the shared features and reduce the effects of any sketch or image-specific artifacts, we propose a novel symmetric loss function based on the notion of information bottleneck for aligning the semantic features while a cross-entropy-based adversarial loss is introduced to align the spatial feature maps. Finally, our CNN-based model confirms the discriminativeness of the shared latent space through a novel topology-preserving semantic projection network. Experimental results on the extended Sketchy, TU-Berlin, and QuickDraw datasets exhibit sharp improvements over the literature.


Introduction
Sketches are the most minimalistic representation of visual data. With the advancement in sensor technology, a quick handdrawn sketch can be used as a query image to retrieve similar category samples from the visual domain. This is especially useful when there is a lack of available visual query sample at hand, while it is only at the mind of the user at a vague picture (Xu et al., 2020a). In traditional sketch-based image retrieval (SBIR) (Eitz et al., 2010), the training and the test classes are the same. However, this is unrealistic as the model may encounter novel classes during inference in many on-the-fly retrieval applications. The zero-shot learning (ZSL) (Xian et al., 2017(Xian et al., , 2018Romera-Paredes and Torr, 2015) aims to bridge the gap between the non-overlapping sets of training classes (seen) and test classes (unseen) using semantic side information for visual recognition. To this end, attempts have been made to integrate ZSL with SBIR to get the zero-shot sketchbased image retrieval (ZS-SBIR) problem (Yelamarthi et al., 2018), even without constraining the search space to contain only the unseen classes at test time, e.g., generalized ZS-SBIR (GZS-SBIR) . This essentially requires aligning the visual and semantic features (which is used for describing the class) in the embedding space. In this paper, we aim to solve the (G)ZS-SBIR problem by looking into some of the existing problems that are explained in the following subsections and propose improved solutions to this end. Figure 1: Given a cluttered feature space for images and sketches, a suboptimal domain alignment fails to produce a discriminative latent space which is affected by problems like hubness and negative transfer. Feature adaptation at multiple feature scales together with a discriminative feature space learning ensures that zero-shot testing can be performed well in BDA-SketRet.

Observations
A key challenge in solving ZS-SBIR is because of the fact that sketches are usually drawn by various artists. ZS-SBIR (Yelamarthi et al., 2018;Dutta and Akata, 2020) models need to overcome a substantial within-category variance in addition to the domain gap between sketches and images given their disparity in spectral, spatial, and texture properties (Xian et al., 2017(Xian et al., , 2018Romera-Paredes and Torr, 2015). Moreover, natural images have random background effects which are completely absent in the sketches. In this regard, the majority of the existing ZS-SBIR approaches learn independent mappings to align the visual domains either in a semantic space (Dutta and Akata, 2019;Pandey et al., 2020) or in a semantics influ-enced latent space (Dey et al., 2019). In both cases, the representations are extracted from the entire image / sketch data while largely neglecting the complex distributions of different local regions. This results in sub-optimal domain alignment as not all the information is equally transferable. Another challenge is that the performance on the unseen classes in generalized (G)ZS-SBIR is often compromised since the model is biased towards the seen classes. A discriminative visual-semantic mapping is generally preferred to alleviate this issue.
Further, images and sketches that are significantly dissimilar across domains in the feature space should not be forcefully aligned; otherwise, the model becomes vulnerable to the negative transfer of irrelevant knowledge. This dilemma can be curtailed if the global feature extraction is guided by highly domain-independent mid-level regional properties. The existing work in the ZS-SBIR literature that use domain adaptation primarily tries to align the final features of the sketch and images (Dutta and Akata, 2019;Dey et al., 2019;. This often leads to a sub-optimal domain alignment due to variations in the sketch domain. These methods are non-trivial to use in the ZS-SBIR setup given the disjoint nature of the training and test classes. As the quality of the spatial feature maps obtained from the intermediate layers affects global image-level features extracted from deeper layers in a CNN, it becomes hard to forcefully align the global features without ensuring equivalence of the preceding feature maps. Apparently, the feature maps look substantially different for the sketch and the image modalities considering several issues like the variations in the sketch drawings and background effects in case of images. We argue that the well-aligned midlevel feature maps would supplement the global feature adaptation process. Hence, we introduce a ZS-SBIR model called BDA-SketRet which inculcates a rigorous multi-level modality adaptation policy between sketches and images both at the mid and high level feature embeddings to ensure a highly domainagnostic, discriminative, and task-specific representation learning. Fig. 1 illustrates in briefs the main idea behind the proposed BDA-SketRet.

Our Approach
In BDA-SketRet, we propose to apply cross-modal visual adaptation separately at the local and global levels. The local adaptation is essential as the mid-level feature maps are very dissimilar between the domains and hence making them indistinguishable at this level is challenging. The major problem in aligning the feature maps arises from the vagueness of the decision boundary between the domains. Hence, we propose an cross-entropy based loss measure where the ground truth probability is selected to have high entropy. In the global adaptation, we aim to make the high-level feature embeddings more discriminative and devoid of any irrelevant domain-specific information which may hamper the alignment. For this, we introduce a novel metric learning based information theoretic alignment loss based on symmetric KL divergence. Finally, since the sketch and image modalities are vastly different, we include two cross-modal mid-to-high level feature reconstruction modules which further boosts the domain invariance of the back-bone networks. This is different from the traditional crossreconstruction of the high-level features typically followed in the multi-modal learning tasks. On the other hand, we propose a semantic projection network for the class prototypes which preserves the neighborhood topology of the original semantic space into the embedding space. Generally, the ZSL models are biased towards to training classes which may partially be solved with neighborhood preservation. In order to accomplish the same, we introduce a network consisting of MLP and graph CNN (GCN).
Our major contributions are as follows: i) We introduce BDA-SketRet, a discriminative multi-level domain adaptation framework for solving ZS-SBIR and GZS-SBIR. ii) We showcase the effectiveness of aligning both the domains in two different feature levels (mid and high level) and propose a novel strategy for the same. A novel symmetric KL divergence based formulation is introduced for information bottleneck loss to be used for semantic feature alignment as opposed to the traditional asymmetric KL based VIB loss measure (Tishby and Zaslavsky, 2015). iii) An intuitive neighborhood-preserving semantic projection module is proposed which is seen to reduce the model bias towards the seen classes specifically for GZS-SBIR. iv) We conduct extensive experiments on the extended Sketchy, TU-Berlin, and QuickDraw datasets and rigorously ablate the model. The efficacy of the proposed model is verified experimentally as the proposed network beats the existing state-of-the-art models by a margin of ≈ 3 -4% in all the evaluation metrices.

Related Works
In this section, we review prior related works on ZS-SBIR, semantic projection and GCN that are related to ours. We also show how is our work different from the existing state-of-theart models.

State-of-the-Art ZS-SBIR Algorithms
Prior to solving ZS-SBIR tasks, the simple SBIR problem received a lot of attention (Lei et al., 2019). As already stated, the major obstacle in solving the SBIR task stems from the fact that the distributions difference between sketch and image data is exceedingly large. Early works in this area include conventional pattern recognition methods for retrieval by engineering hand-crafted visual features (Hu and Collomosse, 2013;Saavedra, 2014). The proposition behind such approaches is to solve the task at hand by obtaining the edge-map of the natural images and to further match them with sketches arising from the same categories. As expected, the low-level SIFT (Lowe, 1999), SURF (Bay et al., 2006), or HoG (Dalal and Triggs, 2005) based descriptors are unable to properly encode the regional variations of the sketch data, resulting in an inferior cross-modal matching. The performance measures of deep CNN based SBIR models have witnessed a massive enhancement lately, thanks to the data-driven feature learning capabilities of CNN. Since the retrieval performance benefits from a discriminative feature space, several endeavors rely on distance metric learning strategies like contrastive-loss (Chopra et al., 2005), triplet-loss (Sangkloy et al., 2016), and HOLEF-based loss , to name a few. (Bui et al., 2017) proposed an efficient representation for sketch and image features using triplet loss, while on the other hand (Federici et al., 2020) provides an information-theoretic approach. Likewise, Qi et al., 2016;Yu et al., 2017;Sangkloy et al., 2016;Wang et al., 2015;Liu et al., 2017;Zhang et al., 2018) aims to solve a similar task.
The ZS-SBIR literature consists of both the discriminative and generative deep learning based techniques. Under the generative umbrella,  proposes a hashing network for the semantic knowledge reconstruction (ZSIH). Similarly, (Yelamarthi et al., 2018) introduces a conditional generative model for ZS-SBIR based on variational learning. The stacked auto-encoder (SAN) method (Pandey et al., 2020) deploys a generative framework based on stacked-adversarial networks within a Siamese architecture. The paired cyclic consistency loss proposed in SEM-PCYC Akata, 2019, 2020) helps in aligning the sketches and images in an encoded semantic space using adversarial training. Inspired by style transfer, (Dutta and Biswas, 2019; develops a styleguided image to image translation model for ZS-SBIR, while (Dey et al., 2019) uses a triplet-based network to solve the task at hand.  highlights the implications of data and class imbalance in ZS-SBIR and introduces an adaptive margin diversity regularizer (AMD-reg) to combat the same. As opposed to the real-valued feature embedding, hash-code based representations are also considered in this regard which offers a trade-off between performance and storage . The generative models for cross-modal style transfer are also explored (Dutta and Biswas, 2019) in this regard. While all the techniques showcase their performance on ZS-SBIR, a few works Akata, 2019, 2020;Pandey et al., 2020) also demonstrate their experiments for the GZS-SBIR setting.

Semantic projection and GCN.
The semantic information has an important role to play for attaining an improved ZS-SBIR inference. Generally, the visual modalities are aligned to a fixed semantic space. While (Dutta and Akata, 2019) uses an encoded semantic space obtained via a semantic auto-encoder, methods of (Dey et al., 2019;Dutta and Biswas, 2019) reconstruct the original semantic vectors from the visual information. Generating visual information from semantic prototypes has been dealt with in this regard (Yelamarthi et al., 2018). The graph CNN (Kipf and Welling, 2016) can encode the structural similarity among several objects and have been successfully applied for vision tasks like ZSL, image captioning, and visual question answering (Kampffmeyer et al., 2019;Yang et al., 2019;Narasimhan et al., 2018). The graph CNN has been considered in ZS-SBIR recently (Zhang et al., 2020) where the modality alignment takes place by restricting the domain-specific graphs to be isomorphic. How are we different? Contrary to the existing literature, we choose to look into the ZS-SBIR problem mainly as a domain adaptation task wherein we propose to do a multi-level adaptation by tackling both spatial and semantic domain drifts. The notion of multi-level domain adaptation is new in this regard, to the best of our knowledge. From the theoretical point of view, we define a new symmetric loss function for the information bottleneck principle which is found to be well-suited for multimodal labeled data pairs, something that the existing asymmetric divergence based functions fail to model. Finally, our semantic projection network offers topology preservation within minimal overhead and combats hubness effectively where all the existing techniques overlook this aspect completely.

Problem definition and preliminaries
Let Z s = {A s , B s , C s , W s } be a multi-modal training dataset consisting of images A s and sketches B s obtained from the |C s | seen visual categories. Additionally, we have access to semantic side information W s which typically corresponds to the distributed word-vector embeddings of the individual category names. During inference, image and sketch data Z u = {A u , B u } from a non-overlapping set of previously unseen classes C u are considered (C u ∩ C s = ∅) in the zero shot SBIR setup. We deal with the unpaired dataset setting in Z s where the number of sketch and image instances in A s and B s are different: The model is trained to reduce the distribution mismatch between A s and B s and subsequently to transfer the knowledge from Z s to Z u with the help of the semantic information W s . The testing phase concerns the retrieval of images with similar semantic categories from A u given the sketch queries from B u . In contrast to ZS-SBIR, GZS-SBIR assumes the presence of images from A s ∪ A u during testing for unseen-class sketch queries coming from B u .

BDA-SketRet
The goal of BDA-SketRet is to align the images and sketches from the same class in a semantically meaningful shared latent space. It is composed of cross-modal triplets where the sketch data from B s serves as the anchor (α) while the positive (p) and negative (n) counterparts are selected from A s (Fig. 2). The feature networks for A s and B s are defined by φ(·) and ψ(·) which are convolutional neural networks with integrated attention sub-networks φ att (·) and ψ att (·).The attention block outputs are simultaneously projected to the local adversarial domain classifier l(·) to highlight spatially indistinct features of the same-class samples from A s and B s and to the shared latent space. We further introduce two cross-modal feature reconstruction modules (V α (·), V p (·)) which aim to reconstruct φ(p) from ψ att (α) and ψ(α) from φ att (p) through variational bottlenecks. On the other hand, the outputs of φ(·) and ψ(·) need to be synchronized for defining the shared embedding space. In this regard, the global domain adaptation on φ(p/n) and ψ(α) is carried out considering a combination of the domain classifier f (·) and a multi-class category classifier h(·). A semantic subnetwork g(·, ·) comprising of an MLP g 1 (·) and a graph CNN g 2 (·, ·) is used to non-linearly project the semantic vectors into the shared space. The outputs of g 1 (·) and g 2 (·, ·) are concatenated and projected onto the latent space by another MLP g 3 (·). Figure 2: A depiction of our BDA-SketRet architecture.The images and sketches undergo a bi-level domain adaptation at the outputs of (φ att , ψ att ) and (φ, ψ), respectively. The important learning objectives: (i) domain losses, (ii) cross-modal multi-level reconstruction loss, and (iii) neighborhood preserving semantic projection loss are shown and how they are evaluated in the network. In the semantic graph, thicker edges means more discriminativeness. In the semantic graph, thicker edges means more discriminativeness. 0/1 represent the domain labels.
There are mainly three learning objectives that together train BDA-SketRet, namely, (i) spatial and semantic domain losses (ii) cross-modal multi-level reconstruction loss, and (iii) neighborhood preserving semantic projection loss. They are detailed in the following subsections.

Multi-level Sketch Image Alignment
The semantic adaptation works on the final features obtained from the last dense layer (φ and ψ) of the backbone, while the spatial adaptation is carried out on the mid-level feature maps obtained from the final convolutional layer (φ att and ψ att ). (i) Semantic Adaptation. We aim to maximise the information between the probability distributions P α = P(ψ(α)|α) and P p = P(φ(p)|p), and minimise the information they share with P n = P(φ(n)|n). We employ a posterior distribution matching method to generate a latent space in which α and p are as aligned as possible, while α and n are sufficiently disjoint. We model the probability distributions P α , P p , and P n by appropriate normal distributions.
We follow the deep variational information bottleneck framework (Alemi et al., 2017;Federici et al., 2020) where the classification task is considered for data from both the visual modalities with the combined cross-entropy loss as L class . The aim is to maximise information between latent variables and output space by minimising cross entropy loss between the distributions, and minimises information between source and latent variables by using the standard normal distribution as a variational approximation to their marginal. We restrain this latent space to be a standard normal distribution. We minimize the symmetric KL divergence between P α and P p is to reduce their relative entropy. Apart from reducing the distance between P α and P p , our framework also requires that P α and P n be disjoint in the feature space. For this, we define L skl (P α , P p/n ) as 1 2 D KL (P α P p/n )+ D KL (P p/n P α ) . Hence, we introduce a novel symmetric KL divergence triplet loss L t-skl as, Here β and λ are hyper-parameters which denote the weight of SKL divergence and the margin respectively. By definition of the loss function, our goal is to bring P α and P p closer while maintaining a difference of at least λ from the distance between P α and P n .
Along with L t-skl , we also use instance-based triplet loss L triplet here so that the cross-modal data become class-wise dense clusters in the embedding space. They complement each other in obtaining a dense feature space highlighting only the domain-shared features while neglecting any artifact. While L t-skl maximizes the information between latent variables and output space, L triplet makes the cross-modalities dense in the embedding space. The cross-modal triplet loss aims to bring the same class sample φ(p) from the image modality closer to a given sketch anchor ψ(α) while pushing the negative image sample φ(n) far from ψ(α) at least by a margin of µ as per Eq. 2 using the Euclidean distance metric D.
(2) Therefore, the combined semantic adaptation loss is: (ii) Spatial Adaptation encourages the learning of abstract local spatial concepts common to A s and B s . This is done by adversarially adapting the spatially average-pooled feature-map outputs of φ att (p) and ψ att (α). Most adversarial domain adaptation frameworks employ a binary domain classifier l(·). We observed that sketches and images are extremely disparate in terms of spatial and spectral properties. Therefore, it is fairly unchallenging for a discriminator to learn distributions corresponding to domains promptly, defeating the purpose of adversarial training. The root of this problem is the hard decision boundaries we set for the domain classifier in the min-max optimisation. We argue that the margin between the decision boundaries is still substantially large for the generator to overcome, given the inherently huge domain gap. Therefore, we introduce a pseudo-decision boundary between spatial features of α and p that helps the generator to surpass the margin.
This decision boundary introduces a certain degree of domain invariance before projecting them onto the latent space, leading to stable training, i.e. l(x) = 0.5, if x ∈ A s or x ∈ B s . In an adversarial setup, the generator competes with the discriminator to increase P(x ∈ A s ) to be greater than 0.5 and P(x ∈ B s ) to be less than 0.5. The decision boundary is symmetrical to both the domains in the feature space, rendering 0.5 as a good value for a pseudo decision boundary. So the local domain adaptation is given as: (3) (iii) Cross-modal multi-level reconstruction. Although L 2 adapts the intermediate feature maps of A s and B s , it does not guarantee that the feature maps of a given modality are aware of the final latent feature distributions for the other modality. In this regard, it is essential that the high-level feature distributions P α and P p characterised by domain-independence do not completely lose out class-related information about their conjugate modality i.e. α and p. To better equip the feature maps with the cross-modal information, we introduce the classwise cross-modal reconstruction loss using generative modelling. Specifically, the cross-modal encoder-decoder modules reconstruct the latent feature embedding of sketch anchor ψ(α) given the outcome of φ att (p) and vice-versa. Both V e α and V e p are designed to be stochastic encoders and their outputs follow the standard normal distributions as per the principles of variational learning. Let D KL be the Kullback-Leibler divergence, referred to as equation 4.
The cross-modal reconstruction loss is given as follows.
The overall domain loss is given by equation 6.

Neighborhood Preserving Semantic Projection Loss
The semantic side information is obtained by taking various combinations of text-based and hierarchical word embeddings for the category names. For the distributed word-vector models, we consider the pre-trained Word2Vec (Mikolov et al., 2013) and fasttext (Bojanowski et al., 2017) while the Jiang-Conrath (Jiang and Conrath, 1997) and path similarity are used for the latter.
As aforementioned, the neighborhood information is severely affected when the semantic prototypes are directly projected onto the latent space using non-linear dense layers, limiting the overall performance. Hence, we desire to ensure the latent space to mimic the neighborhood information of the original semantic space, in addition to being discriminative. We project the semantic prototypes W s together with the topology information of the original semantic space to the shared latent space. The topology information, which is found to bring in a regularization effect into the latent space is encapsulated in the weighted semantic adjacency matrix Γ |C s |×|C s | defined by the pairwise cosine dissimilarity among the semantic prototypes of the seen classes. Ideally, the outputs of g 1 (W s ) and g 2 (Γ, W s ) are concatenated and subsequently projected onto the latent space by another MLP g 3 (·): g(W s , Γ) = g 3 ([g 1 (W s ), g 2 (Γ, W s )]) where [·, ·] defines the vector concatenation operation. The semantic reconstruction loss brings φ(p) and ψ(α) closer to the projected class embedding g(w + , Γ) while maximizing the divergence between φ(n) and g(w + , Γ). This is accomplished through the graph regularized semantic loss L semantic as follows, where the distance S between the vectors (x, y) is defined in terms of the cosine distance for a given threshold t as, S (x, y, t) = 1 2 (t − xy T x y ). The overall objective function for the BDA-SketRet framework can now be put forward as L total = L domain + L semantic .

Theoretical proof for IB-based alignment for tighter generalization bound
In this section, we provide a theoretical proof to show that the proposed semantic adaptation and information bottleneck framework generate a latent space with two properties: the projections of α and p are coalesced, while those of α and n are disjoint. To prove this, we show that the distance between probability distributions of α and n is always greater than the distance between α and p using the theory of learning from different domains.
Adopting the notation from (Ben-David et al., 2009), we denote error given by a hypothesis h ∈ H on a source domain S and target domain T by S and T respectively, where H defines the hypothesis space. The error is defined as the probability according to the distribution D that a hypothesis h disagrees with a true labeling function f .
Let us first consider the objective of generating fused P α and P p . Without any loss of generality, we fix the source domain to be D α = {x ∈ A s } and target domain to D p = {x ∈ B s | x belongs to class w + }. Translating our notations to the theoretical bound obtained in (Ben-David et al., 2009), we provide the bound for the error on target domain as The constraints on the latent distributions to be the standard normal distribution as per the variational upper bound principle, results in the following inequality by using the triangular Table 1: Comparing our BDA-SketRet with SOTA on ZS-SBIR (top) and GZS-SBIR (bottom) on both the splits of Sketchy-extended (S1 and S2), TU Berlinextended and Quickdraw-extended datasets. All models use VGG-16 feature backbone. The '-' represents the evaluation metrices which were not mentioned in the respective papers.

Sketchy-ext (S2)
Sketchy-ext (S1) TU Similarly, by virtue of the information bottleneck, we can again use the triangle inequality for d H (P α , P p ) as By Subtracting Eq. 8 from 9, we get d H (P α , P n ) − d H (P α , P n ) ≤ d H (P p , N(0, I))− d H (P n , N(0, I)) We would like to point out that the RHS consists of terms that are minimized by the variational bottleneck framework. Therefore from Eq. 5 , under optimal conditions on L t-skl , d H (P α , P p ) − d H (P α , P n ) ≤ 0. Therefore, we can conclude that the generated latent space probability distributions follow d H (P α , P n ) ≥ d H (P α , P p ).

Experiments
Datasets. We validate the efficacy of the BDA-SketRet by performing experiments on the benchmark Sketchyextended (Sangkloy et al., 2016), TU Berlin-extended (TUB) (Eitz et al., 2012), and the newly introduced QuickDraw-extended (Dey et al., 2019) datasets. Sketchy consists of 125 categories of unpaired sketch and photo images. We use the two conventional train-test splits. In split 1 (S1) we randomly select 25 classes as the unseen test data, while in split 2 (S2) we use |C s | : |C u | = 104 : 21 as mentioned in (Yelamarthi et al., 2018) where the 21 unseen classes are carefully chosen not to be part of ImageNet (Deng et al., 2009). For the remaining datasets, we follow the same protocol as .
Training and Evaluation Protocol. We select around 5000 triplets in each training iteration based on the aforementioned triplet-mining protocol. µ (Eq. 2) and λ (Eq. 1) for Sketchy: (0.1, 0.1), Tu-Berlin: (1, 1), and QuickDraw: (1, 1), while β is set to 0.0001 for all the datasets. These parameters were estimated using grid search with cross-validation (Yelamarthi et al., 2018). We use batch-normalization and leaky-ReLU nonlinearity after each of the layers to ensure a stable training. L total is optimized using the stochastic gradient descent (SGD) with momentum as the optimizer with a mini-batch size of 32. An initial learning rate of 0.0001 and a momentum of 0.9 are set. We find that L total converges for all the datasets within 50 epochs. We report the performance of BDA-SketRet in terms of mAP@all (mean average precision), mAP@200, P@100, and P@200, respectively, where P stands for precision. Implementation Details. The feature backbone networks φ(·) and ψ(·) are the ImageNet pre-trained VGG-16 model (Simonyan and Zisserman, 2014). Two modality specific spatial attention learning modules consisting of convolution kernels with sigmoid non-linearity are applied on the outputs of the final convolution layer (conv-5) of φ and ψ, and the network upto the attention blocks each producing 512 feature maps of size 7×7. The attention blocks are followed by three new dense layers which project the attended feature maps onto the final latent space with dimensions R 256 . Besides, a spatial average pooling across the channels is applied on the outputs of (φ att , ψ att ) to obtain a single channel feature map of resolution 7 × 7. The encoder and decoder modules of V α and V p are single dense layers of 128-d. The local and global domain classifiers l(·) and f (·), and the multi-class category classifier h(·) are also one dense layer each. For the semantic projection network g(·), g 1 and g 3 are three dense layers each. g 2 is a graph convolution layer, followed by the pooling and flattening layers. During training, VGG-16 layers prior to conv-5 are frozen while the proposed layers are updated.

Results
We choose the following state of the art methods, ZSIH , SEM-PCYC (Dutta and Akata, 2019), Doodle2search (Dey et al., 2019), Style-guide (Dutta and Biswas, 2019), AMD regularizer with SEM-PCYC and Style-guide , respectively, for analyzing our performance. While SEM-PCYC, and Style-guide are based on adversarial training, ZS-SBIR utilizes variational encoderdecoder networks. AMD regularizer helps in tackling the data/class imbalance between the training and test sets. The performance of our full BDA-SketRet model on the Sketchy dataset with its two splits, the Tu-Berlin dataset and Quickdrawextended datasets in comparison to the state of the art (SOTA) is reported in table 1. The mAP value for ZS-SBIR in Sketchy S2 is 43.5. Similar to BDA-SketRet, these techniques report their performances using the VGG-16 (Simonyan and Zisserman, 2014) based feature backbone networks. We have not compared our results with (Xu et al., 2020b;Chaudhuri et al., 2020) as the subset of test data considered is different from ours. For Sketchy, split 2 is considered to be more difficult than split 1 as it consists of test classes which are unseen to the ImageNet pre-trained networks.
Amongst the other competing techniques, we find the inclusion of AMDreg boosts the performance of the baseline ZS-SBIR systems. In spite of this, BDA-SketRet beats SEM-PCYC + AMD by a considerable margin of 3 − 5%. The TU-Berlin dataset is challenging mainly due to the presence of class-wise as well as domain-wise data imbalance. The performance of other competing techniques are extremely low. In contrast, we achieve a boost of 5% over the existing literature. The Quick-Draw dataset is excessively large consisting of highly ambiguous sketches and is by far the most challenging dataset for ZS-SBIR. BDA-SketRet beats the P@100 value over the SOTA with a margin of 10%, while falls marginally to SEM-PCYC in the mAP value.
Similar to ZS-SBIR, BDA-SketRet showcase overall improved performance measures for GZS-SBIR for all the datasets. In particular, BDA-SketRet produces high P@100 values of 41.3 for Sketchy (S1), 35.7 for TU-Berlin, and 28.6 for QuickDraw which are at least 2.5 % more than the literary works. No prior approach report the GZS-SBIR score for S2 for Sketchy yet. However, we find that our performance in this case is substantially high.
We note that the existing techniques only report P@100 or P@200 but not both. However, we feel that an effective ZS-SBIR system should produce high scores for both the metrics together. In this section, we report the performance of all the datasets, including the two splits of the Sketchy dataset with all the evaluation metrics in table 2. The top part report the ZS:SBIR results, while the bottom part reports the GZS:SBIR.

Evaluating Effect Of Input Modalities.
In the ZS-SBIR setup, the significance of the semantic information is imperative in maneuvering the alignment of the  multi-modal data in the latent space. Different models yield different topological alignment of the classes in the latent space, which effectively causes the similar classes to cluster in a short range, while pushing apart the faraway classes. We consider the individual textual (300-d) and hierarchical embeddings as well as their concatenations and report the mAP values in table 3 for both Sketchy and TU-Berlin. We observe that there is a variation of up to 4 − 6% in the performance of BDA-SketRet by using different semantic information. It is found that the individual semantic spaces provide superior performance than their pairwise combinations as neighborhood topology may not be consistent in different semantic spaces. We obtain the best performance of Sketchy is using the fasttext model, while it is Jiang-Conrath for TU-Berlin produces the best performance with a mAP of 37.4.

Evaluating Effect Of Feature Backbones.
Further, different backbone networks have been utilized by a few existing techniques for ZS-SBIR. It is unjust to directly compare them with the rest of the literary works which exploit  Table 5: Ablation of loss functions and model components in terms of mAP for Sketchy (S2) and TU-Berlin. F denotes the full model. (L tri = L triplet , L sem = L semantic ).

Experimental set up Sketchy TUB
Losses L sem + L tri 27.6 20.1 L sem + L tri + L t-skl 30.3 23.6 L sem + L tri + L t-skl + L class 33.4 27.7 L sem + L tri + L t-skl + L class + L 2 36.9 33.5 L sem + L tri + L t-skl + L class + L 3 40.1 34.8 L sem + L tri + L 2 31.5 29.2 L sem + L tri + L 3 33.2 27.0 L sem + L tri + L 2 + L 3 36.5 32.9 L sem + L tri + L t-skl + L class + L 2 + L 3 43.  (He et al., 2016), both pre-trained on the Imagenet dataset. We also test the performance of our framework using the trending SE-ResNet-50 feature extractor. SAKE (Liu et al., 2019) uses a conditional-SE-ResNet 50 (Hu et al., 2018) architecture, while using an auxiliary task to approximately map each image in the training set to the ImageNet semantic space. SE-ResNet is different from CSE-ResNet as it does not use any conditional variable. Similarly, (Thong et al., 2020) follow a different evaluation protocol from the remaining literature. While the other works use the entire seen classes along with the unseen classes for the GZS-SBIR experiments, in this paper the authors claim that they just use 20% of the samples from the seen classes for evaluation. Hence, a direct comparison of results with (Thong et al., 2020;Liu et al., 2019) may not be fair. Apart from these two, overall it can be observed that BDA-SketRet beats the concerned techniques consistently when adopting the respective visual feature extractors.

Ablation Studies And Qualitative Results
Ablation of model components: The full model consists of a group of sub-modules, each contributing in its own way to enhance the performance. In the ablation analysis, the baseline network comprises of L triplet + L semantic . The global adaptation is performed on the latent features to reduce the domaingap between the data from the two modalities by increasing the domain confusion. This improves the performance marginally, as seen from table 5 . This is expected to yield a class-wise overlapping embedding space for sketches and images. Simply adding the binary domain classifier without the label classifier leads to mode collapse. To avoid this and ensure class-wise discriminativeness, we add the full L 1 loss and observe an increase in the overall performance. We then append the network with the local adaptation module applied on the intermediate (c) Grad-CAM plots highlighting the ROI on the model without L t-skl , L 2 and L 3 losses.
(d) Grad-CAM plots highlighting the ROI on the model without L 2 and L 3 losses.
(e) Grad-CAM plots highlighting the ROI when trained with the full model. Figure 3: Ablation of domain adaptation stages using Grad-CAM plots to highlight the region of interest (ROI). feature maps to highlight important local constructs common to both the modalities. When we add the cross-modal reconstruction modules, we observe significant improvements in the results (43.5 / 37.4). To study the individual contributions of L 2 and L 3 , we use them individually and in conjunction with each other to the baseline model. A clear fall of about 7 − 10% is observed, highlighting the contribution of the semantic adaptation module. As evident from table 5, the full model incurs a boost of 12 − 13% on the mAP values for both the datasets than the baseline, ranging from 27.6 to 43.5 in the Sketchy and 20.1 to 37.4 in the TU-Berlin.
Further, we study the effects of the GCN module in g(·) and the spatial attention layers in φ att and ψ att , respectively. We observe a marginal performance drop of 1 − 2% when the GCN layer is removed from the full BDA-SketRet. Similarly, the attention module is crucial in highlighting the domain-invariant mid-level features and BDA-SketRet without the attention layers is found to marginally degrade the performance. We also look into the effect of removing the GCN module for the GZS-SBIR experiments and notice a drop of 10% in mAP. Figure 4: Top-8 retrieval instances for a few sketch queries from the Sketchy dataset without using the local adaptation on the left and BDA-SketRet on the right. The green checks denote correctly retrieved classes, while the red crosses denote incorrect class images. The blue stars denote the hub instances.
Qualitative analysis of hubness.
Hubness problem (Radovanovic et al., 2010) occurs when a model has a training bias and retrieves images only from a subset of the available categories. This occurs as some embedding vectors of images (also called as "hubs") appear in the nearest neighborhood of many test query sketches. An ineffective feature alignment between the visual modalities may trigger the generation of hubs. In order to show the role of domain losses in retrieving more discriminative image samples, we first train the model with just the adversarial global adaptation (without L 2 and L 3 ), followed by training the entire BDA-SketRet model. In the first case, we notice the presence of hubs. Precisely, the left column of Fig. 4 shows a scenario where the instances of rifle class are retrieved for nearly many query samples as its embeddings are cluttered in the feature space with multiple classes. This adversely affects the overall performance. In the second case, no particular class is visibly found to clutter the retrieval results in the latent space. It depicts that by jointly using the global and local adaptations and using cross-modal reconstruction modules, we achieve a hub-free retrieval results for the same set of queries.
Preservation of semantic space using GCN In this section, we aim to show that the leveraging a graph convolution network (GCN), we preserve the semantic topology in the embedding feature space. The semantic information has an important role to play for attaining an improved ZS-SBIR inference. Generally, the visual modalities are aligned to a fixed semantic space. While (Dutta and Akata, 2019) uses an encoded semantic space obtained via a semantic auto-encoder, methods of (Dey et al., 2019;Dutta and Biswas, 2019) reconstruct the original semantic vectors from the visual information. These methods could lead to a drastic change of the semantic topology in the embedding feature space. Fig. 5 (a) from the main manuscript shows the co-variance matrix of the 250 classes of the TU-Berlin dataset, while Fig. 5 (b) shows the co-variance matrix of the same after passing through a semantic auto-encoder. It can be seen that the semantic topology is visibly distorted, which would lead to improper mapping of visual and semantic space for the unseen classes.
In the proposed BDA-SketRet framework, we utilize graph CNN (Kipf and Welling, 2016) to encode the structural similarity among the different classes. We model an independent latent (a) Original word embeddings.
(b) Auto-encoded word embeddings in latent space.
(c) GCN preserved word embeddings in latent space. space given the original modalities but constraint the space to be influenced by the characteristics of the original semantic space. Fig. 5 (c) shows the co-variance matrix of the 250 classes of the TU-Berlin dataset. It can be seen that the semantic topology is visibly preserved in the embedding space. In effect, this helps us in obtaining a better mapping of the visual and semantic space for the unseen classes.

Grad-CAM Visualization
Gradient-weighted class activation mapping (Selvaraju et al., 2017) (Grad-CAM) primarily uses the gradients of the target class at the final convolution layer to synthesize an intermediate localization map which highlights the most important regions in the image. It effectively helps in displaying the region which gets the most importance for any particular target-class. Effect of bi-level adaptation. We ablate among the various domain adaptation modules in the network and provide a visualization of the same in Fig. 3 and show the Grad-CAM plots of both the models for a few sketch and photo images.
In Fig. 3 (b), we train the model without the L global dom , L t-skl , L 2 and L 3 losses. Notice that this causes improper distribution of weights especially in the sketch images. The network primarily learns from the background of the sketch images, while for the photo images, the region of importance is scattered all over the image. Adding the L global dom , relatively constricts the scattered region of importance to a more localized form. When we further go ahead and add the L t-skl loss to the model, from Fig. 3 (d) we start noticing that the alignment of important region in the sketch images start improving. We can see that the full model produces better highlight to the local constructs.
This establishes our claim that the notion of fine-grained domain adaptation helps in obtaining a more discriminative latent space to combat hubness and negative knowledge transfer judiciously. The Grad-CAM plots displays the region which gets the most importance (ROI) when we train the model. We can see that the full model produces better highlight to the local constructs. Notice in the butterfly image, without the bi-level adaptation, the attention is splattered across the image and this leads to incorrect class assignment during retrieval. Using the full model of BDA-SketRet with bi-level adaptation, although the foreground consists of both the butterfly and the flower, the importance is layed properly on the concerned butterfly class.
Qualitative results. In Fig. 6 we show the GradCAM plots of a few sample photo and sketch images. From the photo images it can be seen that query class is properly highlighted in each image, keeping the remaining image as a background. For example, it can be seen from the Alarmclock images that the attended region is carefully just the clock, leaving out the human as the background. Similarly, the dendritic-structure of the cactus receives the most attention as they contribute more towards the overall recognition of its class. It can also be seen that out of the two power vehicles, the network correctly chooses the bulldozer and puts more weight on it.
Similarly, for the sketch classes it is seen that the highlighted region is the most important characteristic part of the sketch. For example, for frog, the webbed feet are the most important region, while for a skull, the cross-bones commonly drawn under skulls along with the hollow eye sockets conveyed the most information. It can be noted that since CNNs have a tendency to learn from image textures (Geirhos et al., 2018), the network provides more weights towards the image boundaries (as seen from Fig. 3 (b)). This is exactly the reason why sketch-based learning tasks are challenging. The proposed BDA-SketRet helps the network to learn the most important regions and their corresponding weights properly even from sketch images as shown in Fig. 6. In Fig. 7, we show the ZS-SBIR results for a few sample sketches from the Sketchyextended dataset.

Sensitivity To Hyper-parameters
In the proposed work, we have primarily 3 hyper-parameters, i.e., λ, β, and µ from the equations 1 and 2 of the main manuscript. For tuning the hyper-parameters, we follow the standard protocol of the zero-shot learning community by splitting the training data for cross-validation into training and validation data (psuedo-seen and psudeo-unseen split). The overall performance of the proposed framework by varying these parameters with some of the settings are shown in Table 6 for both Sketchy and TU-Berlin in terms of mAP values. We experiment the model performance by choosing different values in the range of 0.01 to 1 for µ and λ, while 0.00001 to 0.01 for β, as the L t-skl loss is found to converge the fastest amongst the rest. We choose the combination of µ, λ, and β values that result in the best model performance.

Conclusions
We introduce a novel ZS-SBIR framework called BDA-SketRet in this paper. The main premise of our model is to perform improved alignment between the image and sketch features based on both the mid-level and high-level CNN based feature embeddings. Together, we introduce two generative cross-modal reconstruction modules to ensure the learning of robust modality-independent features. We further propose to project the semantic information into the shared latent space through a two-stream fusion network by jointly exploiting both the prototypes and the semantic class neighborhood. Overall, BDA-SketRet learns a discriminative and compact latent space and wisely tackles both the negative transfer and the hubness Figure 7: Top-15 retrieval instances for a few sketch queries from the Sketchy dataset using the full model. The green checks denote correctly retrieved classes, while the red crosses denote images from incorrect class. Notice that there are no hub instances generated here.
issues of domain adaptation and ZSL, respectively. Experimentally, we outperform the recent techniques in all the performance metrics on all the existing datasets. We are currently interested in extending BDA-SketRet to support the paradigm of lifelong learning.