{'fraction_non_alphanumeric': 0.0324361475700603, 'fraction_numerical': 0.010442532813054274, 'mean_word_length': 4.6331959535406515, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GlideNet: Global, Local and Intrinsic based Dense Embedding NETwork for Multi-category Attributes Prediction

Attaching attributes (such as color, shape, state, action) to object categories is an important computer vision problem. Attribute prediction has seen exciting recent progress and is often formulated as a multi-label classification problem. Yet significant challenges remain in: 1) predicting diverse attributes over multiple categories, 2) modeling attributes-category dependency, 3) capturing both global and local scene context, and 4) predicting attributes of objects with low pixel-count. To address these issues, we propose a novel multi-category attribute prediction deep architecture named GlideNet, which contains three distinct feature extractors. A global feature extractor recognizes what objects are present in a scene, whereas a local one focuses on the area surrounding the object of interest. Meanwhile, an intrinsic feature extractor uses an extension of standard convolution dubbed Informed Convolution to retrieve features of objects with low pixel-count. GlideNet uses gating mechanisms with binary masks and its self-learned category embedding to combine the dense embeddings. Collectively, the Global-Local-Intrinsic blocks comprehend the scene's global context while attending to the characteristics of the local object of interest. Finally, using the combined features, an interpreter predicts the attributes, and the length of the output is determined by the category, thereby removing unnecessary attributes. GlideNet can achieve compelling results on two recent and challenging datasets -- VAW and CAR -- for large-scale attribute prediction. For instance, it obtains more than 5\% gain over state of the art in the mean recall (mR) metric. GlideNet's advantages are especially apparent when predicting attributes of objects with low pixel counts as well as attributes that demand global context understanding. Finally, we show that GlideNet excels in training starved real-world scenarios.


Introduction
To fully comprehend a scene, one should not only be able to detect the objects in the scene but also understand the attributes (properties) of each object detected. Even if two objects belong to the same category, their behavior might vary depending on their attributes. For example, we can't predict the route of a driving vehicle based on a still 2D image alone, unless we know the vehicle's heading/direction and if the vehicle is parked or not. Accurate classification of objects and their attributes is critical in numerous applications of computer vision and pattern recognition such as autonomous driving where a thorough grasp of the surroundings is essential for safe driving decisions. In order to drive safely, a driver must be able to predict numerous crucial aspects. They include, among other things, the activities of other drivers and pedestrians, the slipperiness of the road surface, the weather, traffic signs and their contents, and pedestrian behavior.
Attributes are often defined as semantic (visual) descriptions of objects in a scene. An object's semantic information includes how it looks (color, size, shape, etc.), interacts with surroundings, and behaviors. The category of an object, in general, determines the set of possible attributes that it can have. For instance, a table might have attributes related to shape, color, and material. However, a human will have a more complicated set of attributes related to age, gender, and activity status (sitting, standing, walking, etc.). Some properties, such as the visible proportion of an object, may exist across multiple categories. Therefore, to accurately predict an object's attributes, we must consider the following: 1) some attributes are unique to certain categories, 2) some categories may share the same attribute, 3) some attributes require a global understanding of the entire scene and 4) some attributes are inherent to the object of interest. In this paper, we present a new algorithm -Global, Local and Intrinsic based Dense Embedding Network (GlideNet) -to tackle the attribute prediction problem. GlideNet is capable of addressing the aforementioned listed concerns while also predicting a variety of categories. To appear in the IEEE/CVF CVPR 2022 Proceedings -New Orleans, Louisiana, USA Earlier methods for object detection and classification relied heavily on tailored or customized features that are either generated by ORB [59], SIFT [43], HOG [11] or other descriptors. Then, the extracted features pass through a statistical or learning module -such as CRF [29] -to find the relation between the extracted features from the descriptor and the desired output. Recently, Convolutional Neural Networks (CNN) have proven their capability in extracting better features that ease the following step of classification and detection. This has been empirically proven in various fields, such as in object classification [33,19], object detection [16,55] and inverse image problems such as dehazing [45,74], denoising [40,57], HDR estimation [41,46,9], etc. Deep learning with CNN typically requires a large amount of data for training and regularization [5,71,2,54,6]. Classical methods [4,14] for predicting attributes may require less data, however they perform worse than deep learning based techniques.
In this work, we present a new deep learning approach GlideNet for attributes prediction that is capable of incorporating problem (dataset) specific characteristics. Our main contributions can be summarized as follows: • We employ three distinct feature-extractors; each has a specific purpose. Global Feature Extractor (GFE) captures global information, which encapsulates information about different objects in the image (their locations and category type). Local Feature Extractor (LFE) captures local information, which encapsulates information related to attributes of the object as well as its category and binary mask. Lastly, Instance Feature Extractor (IFE) encapsulates information about the intrinsic attributes of objects. It ensures that we estimate characteristics solely from the object's pixels, excluding contributions from other pixels.
• We use a novel convolution layer (named Informed Convolution) in the IFE to focus on intrinsic information of the object related to the attributes prediction.
• To learn appropriate weights for each Feature Extractor (FE), we employ a self-attention technique. Utilizing binary mask and a self-learned category embedding, we generate a "Description" Then we use a gating mechanism to fine-tune each feature layer's spatial contributions.
• We employ a multi-head technique for the final classification stage for two reasons. First, it ensures that the final classification step's weights are determined by the category. Second, the length of the final output can vary depending on the category. This is significant since not every category has the same set of attributes.
The term "class" can be confusing because it can refer to the object's type (vehicle, pedestrian, etc.) or the value of one of the object's attributes (parked, red, etc). As a result, we avoid using the term "class" throughout the work. We use the word "category" to refer to the object's type and the word "attribute" for one of the semantic descriptions of that object. In addition, we use uppercase letters X to denote images or 2D spatial features, lowercase bold letters x for 1D features, and lowercase non-bold letters x for scalars, a hat accent over a letterx to denote an estimated value and calligraphic letters X to denote either a mathematical operation or a building block in GlideNet's architecture.

Related Work
Attributes prediction shares common background with other popular topics in research such as object detection [66,26], image segmentation [22,30] and classification [39,61]. However, visual attributes recognition has its unique characteristics and challenges that distinguish it from other vision problems such as multi-class classification [56] or multi-label classification [13,10].
Examples of these challenges are the possibly large number of attributes to predict, the dependency of attributes on the category type, and the necessity of incorporating both global and local information effectively. This has motivated several past studies to investigate how we could tailor a recognition algorithm that can predict the attributes.
So far, the majority of relevant research has concentrated on a small number of generic attributes. [27,67,64,58,34,68] or a targeted set of categories [17,49,69,63,31,1]. For instance, [21,62] predict the attributes related to the vehicles. [62] have proposed a vehicle attributes prediction algorithm. The proposed method uses two branches one to predict the brand of the vehicle and another to predict the color of the vehicle. They use a combined learning schedule to train the model on both types of attributes. Huo et al. [21] use a convolution stage first to extract important features, then they use a multi-task stage which consists of a fully connected layer per an attribute. The output of each fully connected layer is a value describing that particular attribute. For more details about recent work in vehicle attribute prediction, Ni and Huttunen [48] have a good survey of recent work, and some existing vehicle datasets for vehicle attributes recognition (e.g. color, type, make, license plate, and model) can be found in [70,38].
On the other hand, [1,24,63] tackle the prediction of attributes related to pedestrians or humans. Jahandideh et al. [23] attempts to predict physical attributes such as age and weight. They use a residual neural network and train it on two datasets; CelebA [42] and a self-developed one [42]. Abdulnabi et al. [1] learns semantic attributes through a multi-task CNN model, each CNN generates attribute-specific feature representations and shares knowledge through multi-tasking. They use a group of CNN networks that extract features and concatenate them to form a  Figure 1: GlideNet -the inputs are the image, the binary mask and the category of an object. The output is the attributes of the object. Note that, the category embedding is self-learned from the extracted features of LFE using the category estimator. All shown images are taken from Visual Attributes in the Wild (VAW) Dataset.
matrix that is later decomposed into a shared features matrix and attribute-specific features matrix. [73] attempt to focus on datasets with missing labels and attempt to solve it with "background replication loss". Multiple datasets focus on attributes of humans, but the majority target facial attributes such as eye color, whether the human is wearing glasses or not, · · · , etc. Examples of datasets for humans with attributes are CelebA [42] and IMDB-WIKI [58]. Li et al. [32] propose a framework that contains a spatial graph and a directed semantic graph. By performing reasoning using the Graph Convolutional Network (GCN), one graph captures spatial relations between regions, and the other learns potential semantic relations between attributes. Only a handful of published work tackled a large set of attributes from a large set of categories [53,20,60,69]. Sarafianos et al. [60] proposed a new method that targeted the issue of class imbalance. Although they focused on human attributes, their method can be extended to other categories as well. Pham et al. [53] proposed a new dataset VAW that is rich with different categories where each object in an image has three sets of positive, negative, and unlabeled attributes. They use GloVe [52] word embedding to generate a vector representing the object's category.

Proposed Model
Universal semantic (visual) attribute prediction is a challenging problem as some attributes may require a global understanding of the whole scene, while other attributes may only need to focus on the close vicinity of the object of interest or even intrinsically in the object regardless of other objects in the scene. We also aspire to estimate the possible attributes of various types of categories. This necessitates a hierarchical structure where the set of predicted attributes depends on the category of the object of interest. In this section, we discuss the details of GlideNet and the training procedure to guide each FE to achieve its purpose.  Fig. 1 shows an example where the object of interest is the small portion of the floor below the bed. The output is a vector of the attributes of the floor. We can decompose the information flow in GlideNet into three consecutive steps; feature extraction, feature composition, and finally interpretation.

GlideNet's Architecture
In the next few subsections, we discuss the details of each step. However, the reader can refer to Appendix B for exact numerical values of the parameters of the architecture.

Feature Extraction
Feature extraction generates valuable features for the final classification step. It is of utmost importance to extract features that help in predicting attributes accurately. Some of which require an understanding of the whole image while others are intrinsic to the object. In addition, we are interested in the multi-category case. Thus, we need to strengthen the feature extraction process to deal with arbitrary shapes for the object of interest. For these reasons, we have three FEs; namely Global Feature Extractor (GFE), Local Feature Extractor (LFE) and Instance Feature Extractor (IFE). Each FE has a specific purpose so that collectively we have a complete understanding of the scene while giving attention to the object of interest. GFE generates features related to the entire image I. It produces features that are used for the identification of the most prominent objects in the image. Specifically, the generated features from GFE describe objects detected in the image (their center coordinates, their height and width, and their category). We use the backbone of ResNet-50 [18] network here. We extract features at three different levels of the backbone network to enrich the feature extraction process and for enhanced detection of objects at multiple scales. We denote the extracted features by GFE as F 1 G , F 2 G , F 3 G and collectively by F G . Since the extracted features will have different spatial dimensions, we upsample F 2 G , F 3 G to the spatial size of F 1 G ; which is denoted by h × w for the height and width, respectively. Let U(X, Y ) represent a function that upsamples X to the spatial size of Y and S be a concatenation layer, then LFE generates features related to the object of interest, but it also considers the object's edges as well as its vicinity. The extracted features from LFE are used for the identification of the object's binary mask as well as its category and attributes. LFE should be capable of estimating a significant portion of attributes as it focuses on the object of interest in contrast to GFE. However, GFE is still necessary for some attributes, which require an understanding of other objects in the scene as well. To illustrate, consider a vehicle towing another one. We cannot recognize the attribute "towing" without recognizing the existence of another vehicle and their mutual interaction. That is why we employ GFE in features extraction. Similar to GFE, we use ResNet-50 as the backbone for LFE. The extracted features are denoted by F 1 L , F 2 L , F 3 L and collectively by F L . F 2 L , F 3 L are up-sampled to the spatial size of F 1 L .
IFE generates intrinsic features of the object of interest, utilizing its binary mask using a novel convolutional layer dubbed as Informed Convolution. It is of great importance to differentiate and distinguish between the objectives of LFE and IFE. Both of them attempt to extract features that predict the object's attributes. However, IFE generates features related to the intrinsic properties of the object (its texture as an example). On the other hand, LFE generates features associated with its neighborhood and the boundaries of the object. To clarify, assume we want to predict the attributes of a pole in an image. LFE cannot estimate its color, as typically poles have low aspect ratios; its height is much larger than its width. Thus, the number of pixels contributing to the pole's color is small compared to the total number of pixels in the cropped image I C . Therefore, any typical FE will obscure the pole's pixels with other pixels in the cropped image, even if we use an attention scheme to the output features. On the other hand, IFE cannot understand the interaction of an object with its vicinity, as it only considers the object's pixels while extracting features. As an example, consider an object's exposure to light. IFE cannot predict the exposure to light accurately; as that requires   comparison with other objects in the vicinity of the pole (a dark-red object may be dark due to its low exposure to light or that it intrinsically has that color). Therefore, LFE and IFE supplement each other for a better estimation of attributes. The structure of IFE resembles the backbone of ResNet-50 where we replace each convolutional layer with an informed-convolutional one (see Section 3.3). The extracted features are denoted by F 1 I , F 2 I , F 3 I and collectively by F I . F 2 I , F 3 I are also up-sampled to the spatial size of F 1 I .
Therefore, we have three different sets of features at the end of the feature extraction step; F G , F L , F I . Each of them contains features from three levels (dense embeddings) that are all up-sampled to the same spatial size h × w, which we set to 28 × 28 in our implementation.

Feature Composition
Feature composition amalgamates the generated dense embeddings from different feature extractors. A diligent feature composition is indispensable here, as a weak one will impair the extracted features and give all of the attention to only one of the FEs. Therefore, we leverage the binary mask of the object of interest besides a self-generated and learnable "category embedding" to produce a description D for the composition mechanism. Details about how we generate the "category embedding" can be found in Section 3.2.2. After generating the description D, it passes by spatial gating mechanisms G G , G L , G I to generate spatial attention weights denoted by A G , A L , A I in Fig. 1. Later, we use these weights to reduce the 2D spatial extracted features That effectively generates spatial attention maps to each feature level of each FE based on the shape and category of the object. In other words, GlideNet learns to focus on different spatial locations per each FE individually.
The structure of the Object Descriptor (D), Fig. 3, is as follows. First, the binary mask M passes through a convolution block to learn spatial attention based on the object's shape. Meanwhile, the Category Embeddingĉ passes by a fully connected block to learn an attention vector based on the category. Then the category attention vector is broadcasted and multiplied by the mask attention as follows.
where [m, n] represents a spatial location and i represents the channel number. This leads to a composed description for the attention based on the object's shape and category. Finally, a convolution block is used to refine the output description and generates D. The exact structure of D can be found in Appendix B.
Then, D passes by three different gates G G , G L , G I each has a final Sigmoid activation layer to assert that the output is between 0 and 1. Each gate generates a three channels spatial attention map A for each FE. Then, δ reduces the 2D extracted features from FE to 1D features by multiplying each with its corresponding spatial attention map as follows. where

Interpretation
The interpreter translates the final feature vector to meaningful attributes. Its design depends on the final desired attributes outputs. In Section 4, we experiment with two datasets VAW and Cityscapes Attributes Recognition (CAR). Both datasets are very recent and focus on a large set of categories with various possible attributes. However, there are some differences between them. Specifically, VAW has three different labels (positive, negative, and unlabeled). On the other hand, CAR doesn't have unlabeled attributes; it has a complex taxonomy where each category has its own set of attributes, and each attribute has a set of possible values it may take. This obligates the interpreter to depend on the training dataset and the final desired output. Therefore, two models are provided in Fig. 4. In both cases, we first start with a dimensional reduction fully connected layer from R l to R m ; m < l. That enables us to create multiple heads for each category without increasing the memory size drastically. Then, the reduced features f A passes by a single head corresponding to the category of the object of interest. For CAR in Fig. 4a, the output size n c varies from one head to another depending on the taxonomy of category c. While for VAW in Fig. 4b, the output size is the same n = 620. The other difference between the two interpreters is in the possible values the output can take. In VAW, the output ranges from 0 to 1, where 0 represents (a) Input Mask (b) Partial Convolution (c) Informed Convolution Figure 5: An input mask and its propagated mask after 5 layers for different update rules -kernel size = 5.
negative attributes and 1 represents positive ones (unlabeled attributes are disregarded in training). In CAR, the output is not binary as some attributes have more than two possible values. Therefore, we encode each attribute as one hot encoder. For example, the "Vehicle Form" attribute can take one of 11 values such as "sedan", "Van", etc. Thus, we have a vector of 11 values where ideally we want the value 1 at the correct form type and 0 elsewhere. It's noteworthy to mention that CAR has an "unclear" value for all attributes. We skip attributes with unclear values during training.

Training
Since GlideNet has a complex architecture, tailored training of the model is necessary to lead each FE to its objective. Therefore, we have developed a customized training scenario for GlideNet which divides the training into two stages. In Stage I, we focus on guiding FEs to a reasonable good status of their objective by adding some temporary decoders to guide the feature extraction process. The objective in Stage I is to have powerful and representative FEs. Therefore, we do not train D nor I in Stage I. In Stage II, we focus on the actual objective of GlideNet, which is predicting the attributes accurately. Therefore, we remove the temporary decoders, and we train the whole network structure as in Fig. 1

Global Feature Extractor (GFE)
GFE is trained as in Fig. 2a by having a temporary objects decoder that attempts to detect the objects in the input image I, their categories and their bounding boxes center locations (c x , c y ), widths w and heights h.Ô G has c + 5 channels; c of which are a one-hot representation of the category (P), 4 values for the bounding box, and the remaining value is the probability of having the center of an object in that pixel (P 0 ). The training loss term for GFE is as follows.
where L BCE is the Binary Cross Entropy loss, L CE is the multi-class Cross Entropy and L MSE is the Mean-Square-Error loss. λ gp0 , λ gp , λ gd , λ gc are hyperparameters used to tune the importance of each term. Fig. 2b shows the training of LFE. Here, we use three decoders; two temporary decoders for the binary mask M and attributes, and one decoder for the category embedding C. The training loss term for LFE is as follows.
The category embedding encapsulates visual similarities between different categories unlike a word embedding [52], which was previously used in [53]. We reason that learnable vectors, rather than static pre-trained word embedding, capture greater visual similarities between objects depending on their attributes; a teddy-bear is visually similar (attributewise) to a toy more than to an actual real bear.

Instance Feature Extractor (IFE)
where λ ia is a hyperparameter. Therefore, the complete training loss function in Stage I is as follows.

Stage II
In Stage II, the following loss function focuses on generating the final attributes vector correctly from the interpreter while maintaining accurate category embeddingĉ.
Therefore, the main goal is to predict the desired attributes. However, we keep the term for the category embedding to ensure the convergence of the category embedding during training in Stage II.

Informed Convolution
The utilization of the binary mask in the feature extraction process has been previously applied in image inpainting problems in [37,72,8]. [72,8] used learnable gates to find the best mask-update rule, which is not suitable here as we want IFE to only focus on intrinsic attributes of the object. Therefore, a learnable update rule does not guarantee the convergence to a physically meaningful updated mask. Inspired by [37] we perform a mask-update rule as follows. where k is the kernel size of the convolution layer, X (i) , M (i) are the input features and input binary mask for convolution layer i that is only visible for the kernel and represents element-wise multiplication. It is important to notice the difference between our mask-update rule and the one provided in [37]. Fig. 5c shows an output example based on the update rule. In our case, each pixel contributes to the new mask by a soft value that depends on the contribution of the object of interest at that spatial location. Furthermore, Informed Convolution can be reduced to a regular convolution if the binary mask was all ones. In this case, the object of interest completely fills the image and the intrinsic features would be any feature that we can extract from the image. It is also noteworthy to recognize the difference between Informed Convolution and Masked Convolution presented in [65], where the authors are interested in generating an image from a caption by using a mask to ensure the generation of a pixel depends only on the already generated pixels. Their purpose and approach is entirely different.

Experiments and Results
In this section, we validate the effectiveness of GlideNet and provide results of extensive experiments to compare it with existing state-of-the-art methods. Specifically, we provide results on two challenging datasets for attributes prediction -VAW [53] and CAR [44]. In addition, we perform several ablation studies to show the importance of various components of GlideNet. While we can consider other datasets such as [51,28], they lack diversity in either categories or attributes. However, VAW has 260, 895 instances; each with 620 positive, negative and unlabeled attributes. On the other hand, CAR [7] has 32, 729 instances focusing on self-driving. Unlike VAW, CAR has a complex hierarchical structure for attributes, where each category has its own set of possible attributes. Some attributes may exist over several categories (such as visibility) and some other are specific to the category (such as walking for pedestrian). Experiment setup: the model is implemented using PyTorch framework [50].
Evaluation Metrics: mean balanced Accuracy (mA), mean Recall (mR), F 1 -score and mean Average Precision (mAP) are used for evaluation. They are unanimously used for classification and detection problems. Specifically, they have been used in existing work for attributes prediction such as [53,13,60,35,25,3]. Excluding mAP, we calculate these metrics over each category then compute the mean over all categories. Therefore, the metrics are balanced; a frequent category contributes as much as a less-frequent one (no category dominates any metric). However for mAP, the mean is computed over the attributes similar to [53,15]. We compute the mean over attributes in case of mAP to ensure diversity in metrics used in evaluation. As in this case, we ensure having balance between different attributes. All metrics are defined as follows.
where c and n are the numbers of categories and attributes respectively. TP i , TN i , P i , N i and PP i are the number of true-positive, true-negative, positive samples, negative samples and predicted-positive samples for category i. AP j is the average of the precision-recall curve of attribute j [36].
Since some attributes are unlabeled in VAW, we disregard them in the evaluation as [53] did. Conversely, CAR does not contain unlabeled attributes. It has, however, a complex hierarchical taxonomy of attributes that requires modification in the metrics used. For instance, most attributes are not binary. They can take more than two values; a "visibility" attribute may take one of five values. Therefore, we define TP and TN per attribute per category. Then we compute the mean over all attributes of all categories. For example, mA would be as follows.
where n i is the number of attributes of category i. TP i,j is the positive samples of attribute j of category i. Similarly, we can extend the definition of other metrics to suit the taxonomy of CAR. For further details, the reader is encouraged to check Appendix D.  Results on VAW and CAR: Table 1 shows the results of GlideNet in comparison with four state-of-the-art method over VAW and CAR. In VAW, GlideNet obtained better values in all metrics. More prominently, it was able to gain 5% in mR metric than the closest method [53]. This is mainly due to GlideNet's usage of IFE and GFE to detect attributes requiring global and intrinsic understanding. In CAR, Gli-deNet was capable of achieving even a higher gain (∼ 8% mR). GlideNet can be trained directly with CAR dataset due to its varying output length. However, we had to slightly modify the architecture of other method to work with CAR.

Ablation Study
Several ablation studies are presented here to demonstrate the importance of the unique components in GlideNet. Only ablations from the VAW dataset are shown here, however, similar behavior was noticed in CAR as well.
Dense Embedding: Table 2 shows the results of GlideNet with different combinations of FEs. We achieve best results by using all FEs. Notice that the gain from using IFE is higher than GFE. This is expected given most attributes in VAW focus on the object of interest itself and do not require a lot of global context. However, GFE is still valuable when global understanding of the scene is necessary, as in CAR.
Informed Convolution: We retrained the model with a restricted dataset comprising objects with low pixel counts to demonstrate the usefulness of Informed Convolution layers. We specifically identified examples with a lower than 0.35 ratio between their binary mask and their corresponding bounding boxes. This reflects the goal of Informed Convolution layers, which is to give low-pixel-count objects special attention.
Because the only architectural difference between IFE and LFE is in the usage of Informed Convolution layers, we test two scenarios: one with and one without IFE. In all measures, GlideNet obtains the best performance, as seen in Table 3 by meaningful margin.
Object Descriptor: Table 4 shows a comparison between GlideNet with and without the Object Descriptor D. Despite the fact that the results without D are less than ideal, they are still meaningfully higher than [53]. This suggests that the generated dense embeddings are helping in better attributes recognition. The feature composition of D, on the other hand, is superior. Semantic Embedding: GlideNet uses a self-learned category embedding that encapsulate semantic similarities between objects. If the category embedding confuses two categories, it is most likely owing to their visual similarities. In prior studies [53], word embeddings [52] were used to capture the semantic but a word embedding alone would not be sufficient to capture the visual similarities. Table 5 shows a comparison of GlideNet by swapping the Category Embedding (CE) with GloVe [52] -a word embedding. Limited Training Scenario: We also perform a limited training data size comparison between GlideNet and other methods in Fig. 6. The training data size is limited to 60% and 40% of the original training data size of VAW while keeping the validation set as it is. Although all methods suffer in the limited data size scenario, GlideNet shows a much more graceful decay in comparison to other methods.

Conclusion
Global, Local, and Intrinsic based Dense Embedding Network (GlideNet) is a novel attributes prediction model that can work with a variety of datasets and taxonomies of categories and attributes. It surpasses existing state-of-theart approaches, and we believe this is due to the use of a variety of Feature Extractors (FEs), each with its distinct goal. A two-stage training program establishes their objectives.
Furthermore, the self-attention method, which combines a binary mask and a self-learned category embedding, fuses dense embeddings based on the object's category and shape and achieves richer composed features. The suggested Informed Convolution-based module estimates attributes for objects in the cropped image that have a very low pixel contribution. A rigorous ablation study and comparisons with other SOTA methods demonstrated the advantages of Gli-deNet's unique blocks empirically.

A. Introduction of the Supplementary
This is a supplementary that contains some useful information for accurately reproducing findings, as well as the reasoning for using VAW and CAR datasets for experiments in the GlideNet paper.
The complete specifications and setup of the proposed network -GlideNet -architecture are stored in Appendix B.
Appendix C discusses the two datasets, VAW and CAR datasets, used in the evaluation and why we chose those two datasets in particular.
All of the configurations for training GlideNet are contained in Appendix D.

B. Network Architecture Details
In this section, we discuss the exact details of each building block of GlideNet. The reader is encouraged to read Section 3.1 first to understand the purpose of each building block. Here, we only show the configurations without a detailed description of the purpose. In our training algorithms (Section 3.2), we have two stages. The first stage uses 'temporary' decoders that are removed later in both Stage II of training and the inference stage. The details of these temporary decoders are found in Appendices B.6 to B.8

B.1. Feature Extractors (FEs)
For all FEs, we use the backbone of ResNet-50 [18]. Specifically, we take the output after layers 2, 3, &4 as our features. The inputs to the FEs are always resized to 224 × 224. Since the output features don't match in spatial dimensions, we upsample them to the size of the largest, which is 28 × 28. The total number of the output channels for each FE in this case is 128 + 128 + 512 = 768. In the case of IFE, we replace each convolution layer with the novel layer proposed in Section 3.3. However, other than the usage of the (Mask-)Informed Convolution concept, it has the same exact structure as other FEs. The output of GFE, LFE and IFE is denoted by F G , F L and F I respectively.
We initialized the weights of the FEs with the pretrained model found in PyTorch framework [50] that is initially trained for classification problem with the ImageNet dataset [12]. For IFE, we initialize the weights of all Informed Convolution layers with their corresponding 'normal' convolution layers found in the pretrained model. The reason is that a 'normal' convolution layer is a special case of the Informed Convolution layer and it can be a good initialization for the weights of IFE. However, we found that the difference in performance is insignificant between with and without the pretrained initialization of the IFE.

B.2. Object Descriptor
The Object Descriptor, Table 6, is depicted in Fig. 3. It has two input branches for the category embedding and the binary mask of the object. The category embedding branch D.C consists of two fully connected layers while the binary mask branch D.M consists of two 2D-Convolution layers. The output of D.C is broadcasted and multiplied with D.M as in Eq. (5).
We have investigated the usage of the cropped image I C in the generation of the description D. However, we noticed that it actually increases the complexity while not increasing the performance. Since all important features are extracted from I&I C by our strong FEs. The object descriptor needs only to 'learn' where to give attention. This can mainly be obtained by information about 1) where the object is located in the input image I C which is provided through the binary mask M , and 2) what category this object belongs to which is provided by the self-learned category embeddingĉ.

B.3. Gating Mechanism
We have three gates G G , G L &G I for the three different extracted features F G , F L &F I and their corresponding outputs are A G , A L &A I , respectively. Table 7 shows the architecture of each gate. The input is D (the description of the object), which is the output of D (the Object Descriptor). We use a final Sigmoid activation function to ensure that the output range of the learned attention is between 0 and 1. This guarantees the numerical stability of the network, as later the produced values A G , A L &A I are multiplied with the features F G , F L &F I . Without bounding the range of the learned attention maps, the values may explode.

B.4. Interpreter
The interpreter, Table 8, consists of two stages as shown in Fig. 4. The first part I.E reduces the length of the feature vector to 256. Then for each category, we have its own I.H i , ∀i ∈ {1, 2, · · · c}, where c is the number of categories. The output length in the case of VAW is 620 as the set of attributes is fixed over all categories. However, the output length in the case of CAR varies depending on the set of possible attributes of each category. For example, the output length for a Pedestrian object would be 38. While the output length for a Mid-to-Large Vehicle is 41. Please refer to Section 4 and appendix C for more details and discussion about the datasets.

B.5. Category Estimator
The first column of Table 9 shows the structure of the category estimator, which is a single fully connected layer with an output length equal to the number of categories. Ideally, the Category Estimator should produce a one-hot encoding  representation of the category of the object. However, practically it produces a Probability Mass Function (PMF) of the object's category. If the PMF has two or more peaks, then that is primarily due to the visual similarity between their corresponding categories. In other words, the generated embedding captures visual similarities between different categories based on the shape of the object of interest.
As we have argued in Table 5, this is better than a fixed pretrained word embedding such as GloVe [52] in Pham et al. [53].

B.6. Multi-Object Detection Head
The multi-object-detection head (MODH), Table 10, detects different objects in the whole given image. The output should not change by using different objects in the same image as the input of GFE is the whole input image I. The input to the MODH is the upsampled and concatenated features (F G ) Eq. (1). The output has 2265 channels in case of using VAW dataset and 17 in case of using CAR, since the number of categories is different in each dataset. The final output passes by a custom activation function γ that splits the channels of the input features into sets and passes each set to a different activation function depending on the purpose of this set. γ is defined as follows. For the first channel that represents the confidence, the activation is a Sigmoid as well as for the bounding box center coordinates. On the other hand, the activation is an exponential for the dimensions of the bounding box (to ensure non-negativity) while it is a softmax for the multi-category channels (to ensure PMF axioms -non-negativity and summation to one).

B.7. Mask Estimator
The Mask Estimator (M) is a temporary decoder that takes the output features of LFE (F L ) Eq. (2). Its structure is depicted in Table 11. The output is a single channel representing the estimation of the binary mask. The output is restricted to be between 0 and 1 through a final Sigmoid layer -one indicates the pixel belongs to the object while zero indicates that this pixel does not belong to the object of interest.

B.8. Attribute Predictors
The second and third column of Table 9 shows the structure of the local and intrinsic temporary decoders respectively. They both have the same structure, however, the inputs and outputs of each are distinct. In the case of LAE, the inputs and outputs are F L Eq. (2) andâ l respectively. While for IAE, the inputs and outputs are F I Eq. (3) andâ i respectively.

C. Discussion of Datasets
In Section 4, we have trained GlideNet using two new and challenging datasets VAW [53] and CAR [44]. The following are a few of the reasons why these two datasets were chosen in particular: 1. the number and diversity of categories in both datasets are high enough to evaluate the validity and effectiveness of the proposed multi-category architecture.
2. VAW has already been used in attributes prediction as in [53].   1 focuses on the application of attributes prediction for self-driving vehicles. Therefore, CAR focus on attributes such as the activity of a pedestrian, visibility of a vehicle, the color of a traffic light, the speed limit of a traffic sign. On the other hand, VAW 2 is pretty generic and has a wider variety of categories but it has the same set of attributes for all. Most of those attributes are unlabeled; since the majority of them are not meaningful to a certain category. All attributes in VAW can take one of three different labels; positive, negative, and unlabeled.
C.1. Objects with low pixel-count: Fig. 9 depicts two examples that demonstrate the importance of IFE. Let's look at the first example (a), (b) & (c), a narrow vertical pole. To predict its attributes without using IFE, we can either crop while keeping the aspect ratio (no distortion) or stretch it (distorting the image). And both techniques influence the prediction of other attributes. For example, image (c) no longer looks like a pole. Additionally, (d) displays another example (from the VAW dataset) where cropping and stretching are difficult due to the toilet

D. Experimental Setup
In our Experiments and Results Section 4, we have compared GlideNet with four different state-of-the-art methods [13,25,60,53]. We also performed ablation study to prove the importance and effectiveness of the different FEs as well as the novel convolution layer -Informed Convolution.
In our training, we have trained the network at Stage I for 15 epochs, then we switched to Stage II for 10 epochs. We have noticed that changing the number of epochs slightly for each stage did not have a noticeable change in performance. The temporary decoders are removed during Stage II and inference stage; they are only used in Stage I to guide the FEs for their supposed objectives. As mentioned in Section 4, we set the values of the hyperparameters of the training loss function to be as follow: λ gp0 = 1, λ gp = 0.01, λ gd = 0.5, λ gc = 0.5, λ lm = 0.1, λ lc = 0.01, λ la = 1, λ ia = 1 and λ lc2 = 0.01. We have attempted training with different values. What we noticed the most is that it is important to keep the values of λ gp , λ lc &λ lc2 lower than other values significantly. Otherwise the training diverges and the training focuses more on estimating the correct category than actually ensuring decent performance for other decoders.
We have developed our code based on the PyTorch framework [50]. We used GPUs to speed up the training, specifically, we used NVIDIA Tesla V100 ®GPUs. We distributed the code over 4 GPUs to speed up the training. The training time was less than one day and the average infer-  ence time per an entire image was ∼ 0.05 second. This is a very reasonable time for a real-time application. Typically, it is not required to predict attributes with a frequency greater than 20 Hz in most applications such as autonomous vehicles. For an average speed of 60 MPH (88 feet/second), a vehicle will on average predict attributes of the scene every 4 to 5 feet (88/20). This is a small traveling distance for the scene to change significantly. In other words, 20 Hz is sufficient to estimate attributes of all objects in a scene and make fast real-time decisions based on the predicted attributes. It is worth noting that this time can significantly be reduced per scene if we kept the produced General Features F G from one instance to another in the same scene (image); as they all share the same scene. In addition, the time can be reduced by tracking objects with predicted attributes. This will help in decreasing the number of objects that require attributes prediction in each scene, which in turn decreases the computation time per scene.