{'fraction_non_alphanumeric': 0.033278441059420884, 'fraction_numerical': 0.017170989433237274, 'mean_word_length': 5.28674646824113, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lexical semantics enhanced neural word embeddings

Current breakthroughs in natural language processing have benefited dramatically from neural language models, through which distributional semantics can leverage neural data representations to facilitate downstream applications. Since neural embeddings use context prediction on word co-occurrences to yield dense vectors, they are inevitably prone to capture more semantic association than semantic similarity. To improve vector space models in deriving semantic similarity, we post-process neural word embeddings through deep metric learning, through which we can inject lexical-semantic relations, including syn/antonymy and hypo/hypernymy, into a distributional space. We introduce hierarchy-fitting, a novel semantic specialization approach to modelling semantic similarity nuances inherently stored in the IS-A hierarchies. Hierarchy-fitting attains state-of-the-art results on the common- and rare-word benchmark datasets for deriving semantic similarity from neural word embeddings. It also incorporates an asymmetric distance function to specialize hypernymy's directionality explicitly, through which it significantly improves vanilla embeddings in multiple evaluation tasks of detecting hypernymy and directionality without negative impacts on semantic similarity judgement. The results demonstrate the efficacy of hierarchy-fitting in specializing neural embeddings with semantic relations in late fusion, potentially expanding its applicability to aggregating heterogeneous data and various knowledge resources for learning multimodal semantic spaces.


Introduction
Neural language models employ context-predicting patterns rather than the traditional context-counting statistics to yield continuous word embeddings for distributional semantics. Neural word embeddings (NNEs), working either on the character level (Bojanowski et al. 2017) or on the unified (Mikolov et al. 2013a, Mikolov et al. 2013b) vs contextualized (Devlin et al. 2018, Peters et al. 2018) word level, have become a new paradigm for achieving state-of-the-art performances in the benchmark evaluations such as GLUE (Wang et al. 2018) and SuperGLUE (Wang et al. 2019a). Notably, in a broad set of lexical-semantic tasks such as synonym and analogy detection , NNEs have significantly improved distributional semantics compared to the traditional cooccurrence counting. For example, after linear vector arithmetic on word2vec (Mikolov et al. 2013a), queen was found distributionally close to the composition result of kingman + woman in a distributional space.
However, calculating distributional similarity in NNEs usually yields semantic association or relatedness rather than semantic similarity (Hill et al. 2015), inevitably caused by sharing co-occurrence patterns in a context window during self-supervised learning. For example, after calculation of the cosine similarity on word embeddings such as the word2vec Skip-gram with Negative Sampling (SGNS) (Mikolov et al. 2013a, Mikolov et al. 2013b, GloVe (Pennington et al. 2014), and fastText (Bojanowski et al. 2017), we find that the most distributionally similar word to man is woman, and vice versa. In SGNS, queen is one of the top 10 similar words to king, and vice versa; in GloVe and fastText, king is one of the top 10 similar words to queen. Although man vs woman or king vs queen belongs to antonymy, each pair in the embeddings is scored as highly similar. Semantic relatedness contains various semantic relationships, whereas semantic similarity usually manifests lexical entailment or the IS-A relationship. As hand-crafted knowledge bases (KBs) such as WordNet (Miller 1995, Fellbaum 1998 and BabelNet (Navigli and Ponzetto 2012) mainly consist of IS-A taxonomies, along with synonymy and antonymy, they are often used for computing semantic similarity (Pedersen et al. 2004, Yang andYin 2021). Distributional semantics needs to fuse semantic relations in KBs to enhance the semantic content in NNEs, which is necessary for improving the generalization of neural language models.
The current study often employs joint-training and post-processing methods to harvest word usage knowledge from distributional semantics and human-curated concept relations from KBs. Most joint-training methods directly impose semantic constraints on their loss functions while jointly optimizing the weighting parameters of neural language models (Yu and Dredze 2014, Nguyen et al. 2017, Alsuhaibani et al. 2018. Another way of joint training is to revise the architecture of neural networks either through training Graph Convolutional Networks with syntactic dependencies and semantic relationships (Vashishth et al. 2019) or by introducing attention mechanisms Mitchell 2017, Peters et al. 2019). Joint training can tailor NNEs to specific needs of applications, albeit with an excessive training workload in early fusion. In contrast, the post-processing methods such as retrofitting (Faruqui et al. 2015), counter-fitting (Mrkšić et al. 2016) and LEAR (Vulic and Mrkšić 2018) can avoid such burdensome training processes, semantically specializing NNEs via optimizing a distance metric in late fusion. Semantically enhanced NNEs can facilitate downstream applications, e.g. lexical entailment detection (Nguyen et al. 2017, Vulic andMrkšić 2018), sentiment analysis (Faruqui et al. 2015, Arora et al. 2020, and dialogue state tracking (Mrkšić et al. 2016, Mrkšić et al. 2017.
Inspired by previous works (Faruqui et al. 2015, Mrkšić et al. 2016, Vulic and Mrkšić 2018 on semantically specializing NNEs in late fusion, we investigate how to post-process NNEs through merging symmetric syn/antonymy and asymmetric hypo/hypernymy. We seek to leverage the IS-A hierarchies' multi-level semantic constraints to augment distributional semantics. By learning distance metrics in a distributional space, we can effectively inject lexical-semantic information into NNEs, pulling similar words closer and pushing dissimilar words further. Consistent results on lexical-semantic tasks show that our novel specialization method can significantly improve distributional semantics in deriving semantic similarity and detecting hypernymy and its directionality. This paper is organized as follows: Section 2 introduces deep metric learning and examines typical post-processing approaches to injecting semantic relations into neural word embeddings; Section 3 describes hierarchy-fitting, our new late fusion methodology of specializing a distributional space under different semantic constraints; Section 4 outlines our experiments on evaluating hierarchy-fitting, and other popular post-processing approaches in calculating distributional semantics; Section 5 and 6 investigate the efficacy of hierarchy-fitting in refining neural word embeddings through deriving semantic similarity and recognizing hypernymy and its directionality on the benchmark datasets, respectively; Section 7 concludes with several observations and future work.

Metric learning
The self-supervised training objective of neural language models (NLMs) is to maximize the prediction probability of a token given an input of its context, where cross-entropy is often employed as a cost function for backpropagation to produce NNEs, e.g. word2vec (Mikolov et al. 2013a, Mikolov et al. 2013b) in a simple feedforward network and BERT (Devlin et al. 2018) in a deep transformer network. The joint-training approaches to semantic specialization can directly refine the original training objective with hand-crafted relations (Fried and Duh 2014, Yu and Dredze 2014, Nguyen et al. 2017, Alsuhaibani et al. 2018. To impose semantic constraints on generating neural embeddings, they can also modify the attention mechanisms in recurrent neural networks (Yang and Mitchell 2017) and transformers (Peters et al. 2019). Since the joint-training approaches often produce task-specific NNEs, which are computationally demanding when learning from scratch with massive corpora, we only investigate post-processing approaches that can work on any distributional space.
As with semantic specialization on pre-trained NNEs, instead of cross-entropy loss, ranking loss in deep metric learning (Kaya and BİLge 2019) is often used to learn a Euclidean distance in a latent space under the constraints of semantic relations in KBs. Deep metric learning has broad applications from computer vision (Schroff et al. 2015, Lu et al. 2017 to natural language processing (Mueller and Thyagarajan 2016, Ein Dor et al. 2018, Zhu et al. 2018 to audio speech processing (Narayanaswamy et al. 2019, Wang et al. 2019b. Given two tokens: 1 and 2 in the original vector space of NNEs with a weighting function ′ : ∈ , metric learning constructs a distance-based loss function ( ( 1 ), ( 2 )) to yield the augmented embeddings with : ∈ . With the help of KBs that specify the relationship between 1 and 2 using a similar or dissimilar tag , metric learning continuously updates ′ to pull similar tokens closer or push dissimilar ones farther, until ( ( 1 ), ( 2 )) finally arrives at minimum for similar tokens and maximum for dissimilar ones.
In deep metric learning, data sampling for computing ranking loss, either in Siamese (Bromley et al. 1993) or Triplet (Hoffer and Ailon 2015) networks, plays a crucial role in specializing neural embeddings. Correspondingly, contrastive or pairwise loss (Chopra et al. 2005) and triplet loss (Schroff et al. 2015) are two popular cost functions in metric learning, followed by many of their variants, such as Quadruple Loss (Ni et al. 2017) and N-Pair Loss (Sohn 2016).

Contrastive loss
Contrastive or pairwise loss (Chopra et al. 2005) was first used for face recognition on the hypothesis that similar faces from the same person should be positioned at a smaller distance in a Euclidean space and different faces from different ones with a larger one. It can be applied in post-processing NNEs as follows: ( 1 , 2 , ) = ( ( 1 ), ( 2 )) + (1 − ) (0, − ( ( 1 ), ( 2 ))) Here, for the similar tokens: 1 and 2 with the tag = 1, contrastive loss regards them as a positive sample and seeks to decrease their distance ( ( 1 ), ( 2 )); and for the dissimilar tokens with = 0, it considers them as a negative sample and recommends a distance margin to regularize ( ( 1 ), ( 2 )). That is to say if ( ( 1 ), ( 2 )) > , no backpropagation is needed; otherwise, metric learning has to increase their distance. Contrastive loss only works on two token inputs in computing loss every time.

Triplet loss
Triplet loss (Schroff et al. 2015) simultaneously takes three inputs in computing rank loss, which can be defined as follows: For an anchor token , and denote its positive and negative samples in a triplet input, respectively. Here, works as a margin gap to distinguish an easy negative sample from a hard negative one (Kaya and BİLge 2019), and it also serves as a distance boundary between ( ( ), ( )) and ( ( ), ( )) when selecting a triplet in metric learning. If ( ( ), ( )) > + ( ( ), ( )), is an easy negative sample as no loss is generated, and it is not necessary to push farther from ; and if ( ( ), ( )) < ( ( ), ( )) , is a hard negative sample as is distributionally closer to than , indicating that backpropagate is needed to update ; and any negative token located between ( ( ), ( ))and + ( ( ), ( )), is categorized as semi-hard to push away.
In specializing NNEs with prior knowledge, most methods use contrastive loss and triplet loss with different negative-sample selection policies, among which we list some typical ones in the following sections. Faruqui et al. (2015) proposed to retrofit word embeddings with semantic lexicons, including PPDB (Ganitkevitch et al. 2013), WordNet (Miller 1995, Fellbaum 1998, and FrameNet (Baker et al. 1998). The positive pair: and should bear a corresponding semantic relationship extracted from lexicons, including lexical paraphrasing in PPDB, synonymy and hypo/pernymy in WordNet, along with words association in FrameNet. These relations were organized into different graphs, in which word embeddings can be altered through belief propagation. The loss function for retrofitting can be articulated as: where is often set to 1 to control the specialization strength for ; and , , equal to ( ) −1 , is another regularizing factor for in propagation. Since ( , ) is a convex function, minimizing its derivative to ( ) can be denoted as:

Retrofitting
Retrofitting works similarly to contrastive loss. Although it extracts multiple positive samples for , retrofitting pulls only similar or related tokens closer. Srinivasan et al. (2019) adapted the retrofitting method by introducing a WordNet-based similarity score to better account for the closeness between and its neighbours that are located within a 2link distance in an IS-A hierarchy, and achieved competitive results in intrinsic and extrinsic evaluations.

Counter-fitting.
Inspired by retrofitting, Mrkšić et al. (2016) incorporated synonymy and antonymy in semantically enhancing word embeddings. They linearly assembled the loss functions from different semantic constraints while preserving distributional semantics, which are: and are margins for synonymy and antonymy, respectively. Besides semantic specialization on NNEs, they also preserved distributional semantics using ( , ), where is one of the top distributionally similar words to in ( ). In place of the whole vocabulary with ( ), also acts as a pseudo-negative word in ( , ) for efficient backpropagation. Note that the Euclidean distance is often converted with the cosine similarity. Counter-fitting bears a close resemblance to contrastive loss as synonymy and antonymy constraints specialize NNEs conversely. Mrkšić et al. (2017) further improved counter-fitting with semantic constraints from monoand cross-lingual resources. They used triplet loss rather than contrastive loss to refine a distributional space, i.e. attracting synonyms and repelling antonyms, therefore termed ATTRACT-REPEL. The loss functions in ATTRACT-REPEL are listed as follows: While injecting semantic constraints into a vector space, ATTRACT-REPEL retrieved online samples from each mini-batch to calculate the Euclidean distance D. Specifically for in the triplet ( , , ), to pull synonyms closer, the negative sample, , is one of the remaining tokens in a mini-batch that holds the shortest distance to . Likewise, to push antonyms farther, the positive sample, , is one of the remaining tokens in a mini-batch that holds the longest distance to . Since synonymy and antonymy are both semantically symmetrical, ATTRACT-REPEL also takes and as anchor nodes to to recalculate corresponding triplet losses, respectively. Except for the positive and negative samples ( and ) to , there is no other online selection of samples in preserving distributional semantics.

ATTRACT-REPEL.
After ATTRACT-REPEL, the specialized NNEs achieved better outcomes than counter-fitting in similarity judgement and dialogue state tracking tasks. Moreover, thanks to cross-lingual links in the multilingual KBs such as PPDB (Ganitkevitch et al. 2013) and BabelNet (Navigli and Ponzetto 2012), ATTRACT-REPEL can enhance word embeddings in other languages that lack lexical resources.

LEAR.
Apart from syn/antonymy in ATTRACT-REPEL, LEAR (Vulic and Mrkšić 2018) also incorporated lexical entailment or hypernymy to refine NNEs. Given that lexical entailment in WordNet is organized into a hierarchy, semantic constraints in LEAR consist of direct or immediate hypernymy and indirect one that holds more than a two-link distance. LEAR established the same distance margin for hypernymy and synonymy to pull any hyponym-hypernym pair closer. Moreover, LEAR employed asymmetric distance metrics to learn hypernymy's directionality, assuming that a concept's vector magnitude should be less than its hypernym's after semantic specialization. As a variant of ATTRACT-REPEL, LEAR firstly defined the symmetric ATTRACT cost on hypernymy constraints as follows: It then quantified an asymmetric distance metric to encode hypernymy's directionality. LEAR converted the dot product of D in ATTRACT-REPEL with the cosine similarity. Since LEAR singled out the directionality of lexical entailment as a separate learning objective, it achieved state-of-the-art results in recognizing hypernymy.

LexSub
Instead of fusing various semantic constraints to specialize a unified distributional space, e.g. in ATTRACT-REPEL (Mrkšić et al. 2017) and LEAR (Vulic and Mrkšić 2018), LexSub (Arora et al. 2020) learned a separate projection matrix to construct a subspace for each semantic constraint. Semantic constraints in LexSub consist of symmetric syn/antonymy, asymmetric IS-A relations or hypo/hypernymy, and asymmetric PART-OF relations or mero/holonymy. Arora et al. (2020) claimed that the main advantage of LexSub over other post-processing methods was that it might avoid the interaction of different semantic relationships during specialization and be particularly helpful for some domainspecific applications. To learn a projected subspace for each semantic constraint, LexSub employed contrastive loss rather than the triplet loss in ATTRACT-REPEL and LEAR, which can be defined as follows: 1. Symmetric ATTRACT of synonymy: ∈ ℎ 2. Symmetric REPEL of antonymy: 3. Asymmetric ATTRACT of hypernymy or meronymy: Here, µ and γ are two tunable hyperparameters, and each loss function works in a separate subspace permutated by a projection matrix f. For example, for the symmetric ATTRACT of synonymy, transpose a data point in the original neural embeddings of ′ : ∈ into : ∈ , using a learned d by n matrix , i.e.

( ) = ′ ( ).
A key difference of LexSub from the triplet-loss based methods, including ATTRACT-REPEL and LEAR, is negative sample selection. LexSub chose a group of negative samples in metric learning under the condition that ( As for the asymmetric relations, LexSub also applied affine transformation on to learn an asymmetric distance metric between hyponym and hypernym in determining their directionality. LexSub significantly improved the performance of hypernym detection and directionality in comparison with other post-processing methods such as retrofitting, counter-fitting, and LEAR. Note that LexSub also retrofitted VSMs with asymmetric mero/holonymy, which indicates that the refined vector space may be suitable for computing semantic relatedness.

Summary
Overall, apart from enhancing NNEs with different semantic constraints, the main differences among the late fusion approaches lie in choosing loss functions and data sampling, as shown in Table 1.
Post-processing methods mainly focus on metric learning to semantically specialize a distributional space, but coupling positive and negative samples in metric learning is also widely employed in self-supervised training. For example, to maximize the likelihood of word prediction in word2vec (Mikolov et al. 2013a), except for the use of hierarchical softmax, negative sampling (Mikolov et al. 2013b) simplified the optimization process through constructing a contrastive loss, preventing it from learning identical embeddings , which can be formulated as: Here, w and c denote a word and its neighbour in context, respectively; n is one of the negative samples randomly retrieved in a vocabulary for loss calculation. and ′ stand for the weight matrices learned for the hidden and output layers in SGNS, respectively. denotes the sigmoid function to derive the prediction probability of c occurrence given the target word w in skip-gram, and the dot product of vectorized words indicates their distributional relatedness. Hence, the objective of the loss function in SGNS is two-fold: maximizing the probabilities between w and its contextual words (positive) inside a sliding window and simultaneously minimizing the probabilities between w and a negative sample randomly selected from the outside of the window. It can prevent Skipgram or CBOW from updating all the weights in the output layer and significantly lower the loss calculation complexity incurred by softmax. Therefore, the objective of the negative sampling in the above loss function is in line with it in the contrastive loss of metric learning, which aims to attract related words while repelling unrelated ones in a Euclidean space. Note that in learning hierarchical embeddings via joint training in early fusion, Nguyen et al. (2017) employed a triplet distance metric and selected distinctive contexts for hypo/hypernym to minimize their distributional difference, which works equivalently to injecting lexical entailment into a distributional space.
Moreover, the log probability in ( , , ) that inherently captures distributional similarity plays a similar role as a distance metric in late fusion. In measuring taxonomic similarity, Resnik (1995) proposed to compute information content instead of simple edge-counting in an IS-A hierarchy, given the semantic variation of a single link in the hierarchy. The information content of a conceptual node is equal to the negative log probability derived from summing up the occurrences of its hyponymy children (both direct and indirect) in a corpus. Therefore, as a conceptual node ascends in a hierarchy, its probability increases and its information content decreases correspondingly, indicating that it provides less helpful information for computing semantic similarity because it becomes more abstract or general. So given a pair of words: w1 and w2, their taxonomic similarity can be defined as Sim(w1, w2) = −log(ncn(w1, w2)), where ncn is the nearest common node for w1 and w2 in an IS-A hierarchy, and information content can replace standard edge-counting methods with corpus statistics in calculating taxonomic similarity. Likewise, the log probability in ( , , ) is practically the same as information content, which can derive semantic similarity through calculating distributional similarity.

Hierarchy-fitting
Under the hypothesis of similar words sharing similar contexts in vector semantics (Harris 1954, Firth 1957, words that frequently co-occur in a context window will result in similar vectors. For example, the training objective of SGNS is to maximize the similarity of a word with its contextual words and minimize it with negative samples, which will inevitably yield similar embeddings for highly associated words in context . Therefore, distributional semantics derived from neural embeddings may mix semantic relatedness with semantic similarity (Hill et al. 2015, Lê andFokkens 2015), in which synonyms are hardly distinguishable from antonyms.
We propose post-processing NNEs to enhance distributional semantics in deriving semantic similarity rather than relatedness. We collect relationships primarily used to compute taxonomic similarity, including syn/antonymy and hypo/hypernymy, to refine NNEs. Instead of learning a separate subspace for each type of relationship in LexSub (Arora et al. 2020), we follow the same procedure as ATTRACT-REPEL (Mrkšić et al. 2017) and LEAR (Vulic and Mrkšić 2018) in combining different relationships to construct a unified VSM space. The main drawbacks of LexSub exist in its extra cost of learning another layer of weights for each semantic constraint to project an original distributional space into a dedicated one. We construct a separate loss function for each semantic constraint and correspondingly run AdaGrad with a single mini-batch for backpropagation. It can effectively lower the interaction of specialization with different semantic constraints.

Injecting Synonymy and antonymy
We impose specialization on a distributional space with two key semantic constraints in computing semantic similarity: synonymy and antonymy. Their cost functions can be outlined as follows: 1. ATTRACT with synonymy For the synonymy specialization in the triplet loss ( , , ), the margin msyn imposes a constraint on the range of distributional distance among the anchor , its synonym , and its negative sample , namely, pulling and closer and pushing and farther. As for ( , , ) for antonymy, mant serves in constraining , its positive sample , and its antonym . Given the symmetric feature of synonymy and antonymy, we also apply the same hinge loss function on the triplets of ( , , ) and ( , , ), respectively, in each mini-batch. To better regularize the Euclidean distance between and its synonym in computing loss gradient, we mainly retrieve informative negative samples from a remaining mini-batch, which are distributionally closest tokens to or . Likewise, the positive samples for antonymy consist of tokens that are distributionally farthest to or in a remaining mini-batch. We convert the cosine similarity into the vector distance D.

Retrofitting hypernymy
Apart from injecting syn/antonymy into distributional semantics, we also leverage lexical entailment or the IS-A hierarchical relationship in specialization, which works as: Since hyponym-hypernym pairs are less similar than synonymous ones in weighting lexical-semantic relations (Hirst and St-Onge 1997), we set up a different margin of mhyp from msyn to regularize the Euclidean distances of to its direct hypernym ℎ and negative sample . Note that we only employ the direct hypernym of in computing the cost function, which is different from LEAR that retrieves both direct and indirect hypernyms of . We conjecture that imposing a unanimous distance margin on the hyponym-hypernym pairs that are located on different levels of IS-A hierarchy may impose inaccurate semantic constraints on specialization, causing miscalculating the loss gradient.
Assuming that a concept's vector norm or magnitude should be shorter than its hypernym's, LEAR also complemented its specialization objective with an asymmetric distance function to detect hypernymy and directionality in distributional semantics. We suggest that encoding the directionality of lexical entailment through vector norms would further interfere with the triplet loss function. Since our primary goal of specializing neural embeddings in late fusion is to improve their capability of deriving semantic similarity, we neglect the asymmetric distance function at the current phrase. We only use it for detecting hypernymy and directionality in a distributional space.

Adjusting hierarchy relationship
Apart from using the triplet loss functions to inject semantic constraints, we define a quadruplet loss (Chen et al. 2017) to regularize further semantic variation in the IS-A hierarchy, hence termed hierarchy-fitting, which is defined as: Here mhie-syn and mhie-hyp are two margins assigned for synonymy and hypernymy, respectively, to fine-tune their semantic nuances. Unlike msyn or mhyp in the triplet loss, mhiesyn attempts to differentiate synonymy from hypernymy using a tiny distance margin to show their nuances in measuring semantic similarity. Given an anchor word , its synonym and hypernym ℎ bears a high semantic resemblance to but having some degree of semantic variations, namely, is semantically closer to than to ℎ . In the quadruplet loss, mhie-syn imposes a constraint on the range of distributional distances among , , and ℎ , namely, attracting and closer and repelling and ℎ farther. After that, mhie-hyp further regularizes distributional distances among the quadruplet of , , ℎ , and , where is a negative sample for optimizing loss computation. In line with the synonymy specialization, we retrieve that is distributionally closest to or in the remaining minibatch for backpropagation. Correspondingly, we enforce an identical procedure to calculate another quadruplet loss for .
Unlike LEAR and LexSub, which treat synonymy and hypernymy identically on the semantic specialization of embeddings, we propose hierarchy-fitting to differentiate them, as illustrated in Figure 1. Hierarchy-fitting can fine-tune similarity variation between synonyms and hyponym-hypernyms via explicitly weighting hierarchical information in a quadruplet loss. Intuitively, the two margins: mhie-syn and mhie-hyp are supposed to regularize distributional distances of to , ℎ , and , namely It can preserve hierarchical information between concepts. Since not every word can find its antonyms, we replace them with negative samples retrieved in a remaining minibatch during training. Sample selection is critical in speeding up convergence and reducing the odd of local minima (Kaya and BİLge 2019). As depicted in Figure 1, as with the Euclidean distance to , negative samples in hierarchy-fitting include: 1. Hard ones with their distances less than D(fθ (xa), fθ (xsyn)); 2. Semi-hard ones with their distances between D(fθ (xa), fθ (xsyn)) and mhie-syn+D(fθ (xa), fθ (xsyn)); 3. Easy ones with their distances larger than surpassing mhie-syn+D(fθ (xa), fθ (xsyn)).
Hard negative samples may cause a high variance in computing the loss function gradient, whereas easy ones may contribute little to refining models. Hard, semi-hard, and easy sampling have applications in deep metric learning (Kaya and BİLge 2019), with hard and semi-hard ones more informative for training. As for sampling strategies, we found that retrieving two negative samples in LEAR: one is randomly selected, and the other is distributionally closest to the anchor works effectively in our experiments. Figure 1. A hypothetical illustration of semantic specialization with a quadruplet loss. Given an IS-A hierarchy of WordNet starting from the root node on the right, each node depicts a synset that usually contains a group of synonyms. A synset of (happiness, felicity) has an antonym node of (unhappiness), a direct hypernym node of (emotional state, spirit), and a coordinate node of (embarrassment, ecstasy). The injection process of these semantic relations into neural embeddings is depicted on the left. The quadruplet loss first fine-tunes the distances of happiness to felicity and emotional state with the margin mhie-syn, and then to emotional state and unhappiness with the margin mhie-hyp.

Preserving vector space
Besides injecting semantic relationships into a distributional model, we also preserve distributional semantics learned from co-occurrence counts in context, which can be defined as: We set up a regularizing factor, , to fine-tune embeddings in fusing semantic relations hand-crafted in KBs and distributional semantics harvested from text corpora.

Word embeddings
To thoroughly investigate hierarchy-fitting along with other post-processing methods, we incorporated a group of popular word embeddings in the experiments, including Skip-gram with Negative Sampling (SGNS) (Mikolov et al. 2013a), GloVe (Pennington et al. 2014), and fastText (Bojanowski et al. 2017). These embeddings usually assume a bag-of-word (SGNS and GloVe) or bag-of-character (fastText) context to collect co-occurrence statistics. Note that GloVe explicitly factorizes a word by context matrix to reduce its dimensionality in yielding condensed embeddings, whereas SGNS implicitly factorises a co-occurrence counts matrix . They may be envisioned as prediction-based or neural embeddings, as Levy et al. (2015) suggested.
We collected the publicly available English embeddings for SGNS, GloVe, and fastText, with the same dimensionality of 300 to avoid the bias caused by different training methods and corpora (Fares et al. 2017). These embeddings were trained using the same corpora: English Wikipedia Dump (February 2017) and Gigaword 5 th Editio), containing about 6.8B tokens. The pre-processing steps on the corpora were identical with 10-word window size, lemmatization, and a vocabulary of 260K words.
Apart from the embeddings solely trained using contextual words, we also incorporated PARAGRAM (Wieting et al. 2015) in the experiment, which was initialized with Skip-gram embeddings and specialized with the paraphrases in PPDB (Ganitkevitch et al. 2013). In comparison with other popular embeddings (Yang and Yin 2021), PARAGRAM has achieved state-of-the-art results in lexical-semantic tasks. Note that PARAGRAM performed post-processing on SGNS using a triplet loss, which was equivalent to ATTRACT-REPEL except for exclusively using paraphrasable relationships in PPDB. PARAGRAM was trained using Wikipedia (about 1.8B tokens) with 300 dimensions.

Semantic constraints
To investigate the effects of semantic knowledge on enhancing distributional semantics, we selected two commonly used lexical resources: WordNet (Miller 1995, Fellbaum 1998 and Roget's thesaurus (Kipfer 2009). They contain about 909K and 555K synonymous pairs, respectively, along with 465K and 39K antonymous pairs. The dataset of direct hypernymy constraints on nouns and verbs consists of about 321K pairs, extracted only from WordNet's IS-A hierarchies, as there are no such relationships available in Roget's thesaurus.

Training setup
In line with ATTRACT-REPEL and PARAGRAM, we applied AdaGrad to optimize the cost functions in the above sections, through which specialization with each category of semantic constraint reached convergence after 20 epochs. For example, as with hierarchyfitting, we ran a grid search to fine-tune the hyperparameter values, including learning rates, the margins of mhyp, msyn, and mant, and the size of mini-batch. Using the gold-standard data sets of WordSim-353-similarity (Agirre et al. 2009) on nouns (201 pairs) and the training part of SimVerb-3500 (Gerz et al. 2016) on verbs (500 pairs), we finalized the optimal hyper-parameter values: the learning rate of 0.03; the margins of mhyp=0.6, msyn=0.9, and mant=0.3; the regularizing factor of = 0.001; and the mini-batch size of 128. In the quadruplet loss of hierarchy-fitting, the optimal values for mhie-syn and mhie-hyp were 0.001 and 0.6, respectively.

Evaluation tasks
We evaluated the specialized neural embeddings mainly on deriving semantic similarity from distributional semantics. Since we validated the specialized embeddings with semantic similarity datasets, and our semantic constraints included syn/antonymy and hypo/hypernymy, we employed two intrinsic tasks: judging semantic similarity and recognizing hypernymy to assess the post-processing methods.

Semantic similarity calculation
We validated hierarchy-fitting by reporting Spearman's rank correlation coefficient (ρ) on SimLex-999 (Hill et al. 2015), which contains nouns, verbs, and adjectives, as well as on the test part of SimVerb-3500 (Gerz et al. 2016) (denoted as SimVerb-3000), which is composed of 3,000 pairs of verbs. These gold standard datasets are often used to evaluate various methods in deriving semantic similarity (Yang and Yin 2021), mainly consisting of common words that frequently occur in generic domains. Moreover, we assessed hierarchy-fitting and other post-processing methods on the datasets collecting human similarity ratings on infrequent or rare words. We selected two benchmark datasets: RW-2034 (Luong et al. 2013) and CARD-660 (Pilehvar et al. 2018), for the evaluation.
Subject to the vocabulary size of the four embeddings, we tweaked a simple but effective back-off policy (Speer et al. 2017) to handle the issue of out-of-vocabulary (OOV) words. We deleted the last letter of an OOV word until its remaining string matched one of the tokens in the vocabulary. The simplified back-off policy can address the OOV issue in the evaluation with a full coverage rate on the datasets.

Hypernymy detection and directionality
Lexical entailment is a primary component of organizing concepts and inferring concept relationships in semantic memory (Collins and Quillian 1969). WordNet employed lexical entailment to interlink synsets or concepts into the IS-A hierarchies (Fellbaum and Miller 1990). Mining lexical entailment plays a critical role in knowledge engineering Powers 2010, Navigli et al. 2011), e.g. automating taxonomy construction, enriching handcrafted lexical resources, and augmenting downstream applications (Ido et al. 2013). Since hierarchy-fitting employs lexical entailment for semantic specialization, we evaluated it on detecting hypernymy and directionality, which proceeds in the same way as LEAR and LexSub.
Hypernymy identification in distributional semantics (Kiela et al. 2015, Nguyen et al. 2017) is often framed as an unsupervised learning task under two hypotheses: distributional inclusion Weir 2003, Geffet andDagan 2005), claiming that a hypernym should share a substantial number of context features of its hyponyms; and distributional informativeness or generality (Santus et al. 2014), suggesting that a hypernym holds more generic or less informative features than its hyponyms. Unlike the symmetric synonymy and antonymy, hypernymy or its counterpart hyponymy is asymmetric and holds directionality in semantics. The hypotheses often work equivalently in identifying lexical entailment's directionality, essentially identical to information content (Resnik 1995). Information content supposed that a hypernym should be less informative or specific than its hyponyms for computing taxonomic similarity. In contrast to unsupervised learning on hypernymy identification, Levy et al. (2015) demonstrated that supervised learning might fail to recognize lexical entailment because it just tried to memorize distinctive contextual features of hyponyms or hypernyms rather than their underlying connections. Such lexical memorization could not deduce inherent relationships between a hyponym-hypernym pair because there was little connection information encoded in a distributional space. We, therefore, treated hypernymy detection and directionality as an unsupervised learning task, through which we can compare hierarchyfitting with other post-processing methods.
LEAR defined a few metrics for encoding hypernymy directionality (xhypo to xhyper), among which the following worked best: We, therefore, supplemented hierarchy-fitting NNEs with L(xhypo, xhyper) to distinguish the directionality of a hyponym-hypernym pair.
To thoroughly compare hierarchy-fitting with the specialization methods in Section 2, we used the HyperVec package (Nguyen et al. 2017) to evaluate them and report the average precision. The package defined an unsupervised metric of HyperScore for hypernymy identification, which was quantified as the product of a distributional similarity (cosine) between xhypo and xhyper and the ratio of Euclidean norms xhypo to xhyper. HyperScore could indicate two critical aspects in recognizing lexical entailment: xhypo and xhyper should be distributionally similar at first; the Euclidean norm of xhypo should be less than it of xhyper. The HyperVec package contains three benchmark datasets (Kiela et al. 2015) for detecting hypernymy and its directionality, which include: 1. BLESS. Word pairs in BLESS were annotated with various semantic relationships, from which 1,337 hyponym-hypernym pairs (xhypo and xhyper) were extracted. The task for detecting hypernymy directionality in BLESS is to identify which word is a hypernym or superordinate simply by comparing the Euclidean norms of xhypo and xhyper in a ratio of ||xhypo|| to ||xhyper||. Supposing the ||xhypo|| should be shorter than ||xhyper|| on the premise that xhyper is more abstract or general than xhypo, we can predict the directionality in a pair of xhypo and xhyper without setting up a predefined threshold.
2. WBLESS. WBLESS was designed for hypernymy detection, which contains 1,668 word pairs that were sourced from BLESS. A half of WBLESS was categorized as hypernymy or lexical entailment, and the other half with other semantic relationships such as hyponymy and meronymy. Following Nguyen et al. (2017), we formulated hypernymy detection in WBLESS as a binary classification task and fixed a threshold beforehand using 2% of WBLESS data, randomly extracted in 1,000 iterations.
3. BIBLESS. BIBLESS was a by-product of WBLESS, in which 834 hypernymy pairs were preserved, and the remaining 834 pairs were further grouped into two parts: 208 hyponymy pairs and 626 pairs of other relationships. BIBLESS is suitable for detecting both hypernymy and directionality, which is more complicated than detecting hypernymy on WBLESS and directionality on BLESS. We first attempted to locate a threshold to distinguish hypo/hypernymy from the other relationships and then searched for a second threshold to detect their directionality. We used the same procedure on WBLESS (Nguyen et al. 2017) to fix the thresholds.

Graded lexical entailment
Instead of considering hypernymy recognition as a binary classification task,  treated it as a regression task and created the HyperLex dataset for recognizing graded lexical entailment. They measured hypernymy or lexical entailment using a liker scale of similarity scores, assuming that lexical entailment should be appreciated gradually rather than binarily in human semantic memory (Collins and Quillian 1969). Apart from detecting hypernymy and directionality in the unsupervised binary classification tasks, we further validated hierarchy-fitting on HyperLex in deriving graded strength scores on lexical entailment.
There were 2,616 pairs of words in HyperLex, which were tagged with seven categories of semantic relations, such as hypernymy and meronymy. Each pair was rated with its association strength score, collected on a crowdsourcing platform. We first predicted HyperScore for each pair in HyperLex and then calculated Spearman's rank correlation coefficient to demonstrate how well each specialization method can approach humans on grading lexical entailment. The upper bound on HyperLex, a mean interannotator agreement, is 0.864. Note that among the seven semantic relations in HyperLex, hypernymy, containing different link distances in the IS-A hierarchies of WordNet, accounts for about 57.3%, whereas its counterpart, hyponymy, only holds about 11.3%.

Computing semantic similarity
Subject to its use of co-occurrence patterns, distributional semantics yield more semantic relatedness than semantic similarity. We aim to inject semantic relations to improve the efficacy of distributional semantics in deriving semantic similarity. We assess the specialization methods using the gold standard similarity datasets on common and rare words.

Semantic similarity on common and rare words
We first calculated distributional similarity between a pair of words and then measured Spearman's correlation between distributional similarity and average human ratings across each dataset. Table 2 shows the correlational results of using hierarchy-fitting and other post-processing methods in Section 2 to enhance neural embeddings. Given the Zipfian distribution (Zipf 1965) of word usage, we systematically compared these methods, using two classes of similarity datasets: SimLex-999 and SimVerb-3000 for frequent or common words and CARD-660 and RW-2034 for infrequent or rare words. As with SGNS, GloVe, and fastText, hierarchy-fitting almost achieved the best results on the four datasets. It improved the vanilla NNEs by 72.7% and 11.5% on average on the common-and rareword datasets, respectively, whereas the state-of-the-art LEAR only attained 67.2% and 5.8%. Even on PARAGRAM, already specialized with the paraphrases in PPDB, hierarchy-fitting consistently outperformed other post-processing methods except on RW-2034. Table 2: A comparison of measuring semantic similarity using different specialization measures on word embeddings. PG, SN, GV, and FT denote four embeddings: PARAGRAM, SGNS, GloVe, and fastText, respectively. Spearman's rank correlation coefficient (ρ) indicates the agreement between distributional similarity (cosine) and human similarity ratings. The score in the bracket stands for the upper bound of interannotator agreement (pairwise) on a dataset. The bold values indicate the best performance while imposing different semantic specializations.  SimVerb-3000 ( Note that LexSub only provided the specialized GloVe for evaluation, and its vanilla GloVe (with 300 dimensions) (Pennington et al. 2014) was trained with 6 billion tokens extracted from Wikipedia 2014 and Gigaword 5 th Edition corpora, with a vocabulary size of 400k. However, the vanilla GloVe in our experiment was trained with Wikipedia 2017 rather than 2014, with a vocabulary size of 260K. Given the potential impacts of using different corpora and vocabularies on yielding embeddings, we did not compare the LexSub results on GloVe with ours and only listed them for reference: 0.51 on SimLex-999; 0.40 on SimVerb-3000; 0.21 on CARD-660; 0.41 on RW-2034.

Salience of semantic relations
We also conducted an additive study on our hierarchy-fitting model to discriminate the validity of different semantic constraints, as shown in Table 3. Among mono-relations injected, all the semantic constraints substantially improved the vanilla embeddings on SimLex-999 and SimVerb-3000, but on the two rare-word datasets, they only showed a slight advance to the vanilla embeddings with 2.7% on average. As for the hybrid effect of adding semantic constraints incrementally on deriving semantic similarity, after injection of synonymy and antonymy, our model improved the vanilla embeddings by 57.4% on average on SimLex-999 and SimVerb-3000. Table 3: A comparison of evaluating semantic constraints on specializing NNEs in the order of synonymy, antonymy, hypernymy, and hierarchy-fitting, along with additive evaluation on the relations. As for the two rare-word datasets, except for PARAGRAM, synonymy specialization augmented the three remaining vanilla embeddings by 11.7% on average, but continuously injecting antonymy was detrimental to the correlational results. Hypernymy specialization further enhanced NNEs in deriving semantic similarity, which reached their peak performances on SimLex-999 and SimVerb-3000 but not reliably on CARD-660 and RW-2034. When hierarchy-fitting the embeddings to further regularize hierarchical relationships, our model kept improving SGNS, GloVe, and fastText on CARD-660 and showed no significant change on RW-2034, although it deteriorated somewhat on SimLex-999 and remained nearly identical on SimVerb-3000.

Summary
The results of semantic specialization on NNEs emphasize the validity of hierarchy-fitting, which bears a close resemblance to ATTRACT-REPEL and LEAR in leveraging semantic relations to post-process distributional vectors. Hierarchy-fitting achieves state-of-the-art results in measuring semantic similarity. One of the key differences of our model from LEAR is that we exclusively employ direct rather than indirect hypernymy for semantic specialization. Semantic transitivity (Marathe and Hirst 2010) for indirect hypernymy may increase semantic distance, whereas semantic distance under direct hypernymy is relatively short and reliable to function as semantic constraints. Moreover, the hypernymy margin, a variable in the triplet loss, serves as the least similarity score for an authentic pair of direct hyponym-hypernym. If we keep it constant for any lexical entailment, mining indirect hypernymy to specialize NNEs may mistakenly attract and repel words in a distributional space. Hierarchy-fitting can enhance NNEs in deriving semantic similarity. It has further strengthened our hypothesis that collaborating multiple semantic relationships into one cost function may yield hierarchy embeddings.

Detecting lexical entailment
Specializing a distributional space with lexical-semantic relations can significantly enhance distributional semantics in deriving semantic similarity. We further evaluate different specialization approaches to recognizing lexical entailment in four tasks.

Unsupervised detection of hypernymy and directionality
We classified the specialization methods into two groups to investigate to what degree asymmetric distance (AD) metrics in late fusion could recognize hypernymy. In Table 4, Group 1 includes retrofitting, counter-fitting, LexSub, and hierarchy-fitting that employ no AD metrics. LexSub was placed in Group 1 as it only used affine transformation to distinguish hypo/hypernyms. Group 2 applies the AD metrics in Section 4.4.2 to hierarchyfitting mining two variants of hypernyms: one only contains direct hypernymy with the size of 0.32M, denoted as AD_dir; and the other consists of multiple-level or indirect hypernymy with the size of 1.55M, denoted as AD_indir. Note that the size of AD_indir is the same as it in LEAR. Apart from the post-processing methods in Section 2, Table 4 includes HyperVec (Nguyen et al. 2017), a state-of-the-art joint training method in early fusion for detecting hypernymy. We evaluated hierarchy-fitting in Group 2 with AD_dir and AD_indir, as shown in Tabel 4.
In Table 4, no specialization methods in Group 1 show particular advantages over others, with retrofitting scoring about the best precision of 0.57 (fastText) on BLESS, and counter-fitting about 0.54 and 0.38 (PARAGRAM) on WBLESS and BIBLESS, respectively. Hierarchy-fitting performed competitively across the three datasets. After specialization on NNEs with the asymmetric distance metric, Hierarchy-fitting+AD_dir improved hierarchy-fitting by 0.35, 0.13, and 0.08 on BLESS, WBLESS, and BIBLESS, respectively; and Hierarchy-fitting+AD_indir further boosted Hierarchy-fitting+AD_dir by 0.32, 0.26, and 0.35. Hierarchy-fitting+AD_indir was barely distinguishable from stateof-the-art methods, including LEAR and HyperVec (Nguyen et al. 2017). It significantly surpassed the initial results of Kiela et al. (2015) on the three datasets, although Kiela et al. (2015) employed image generality rather than linguistic generality of hypernyms in the evaluation. Note that in learning a separate space for hypernym detection, LexSub defined an asymmetric distance function through the affine transformation of embeddings. However, LexSub only achieved 0.34 (BLESS), 0.47 (WBLESS), and 0.33 (BIBLESS) in our experiments, which were deviated from its initial results of 0.21, 0.60, and 0.50 achieved using the cosine similarity only. Our results reflect that leveraging asymmetric distance to encode the directionality of lexical entailment can enhance distributional semantics in hypernymy identification, which corroborates well with LEAR (Vulic and Mrkšić 2018). It also confirms the previous findings of Levy et al. (2015) on the limitations of unaltered distributional semantics. Our study provides further evidence that calculating distributional similarity with cosine is appropriate for detecting symmetric relations or deriving semantic similarity but not for detecting asymmetric relations such as IS-A and PART-OF relations. In the same vein of LEAR, the directionality of hypernymy can also be recognized on the hypotheses of distributional inclusion Weir 2003, Geffet andDagan 2005) and generality (Santus et al. 2014). For example, distributional inclusion leveraged the intersection of contextual features sharing between hypo/hypernyms for identifying lexical entailment (Weeds and Weir 2003, Weeds et al. 2004, Kotlerman et al. 2010 or for joint training in late fusion to yield hierarchical embeddings (Nguyen et al. 2017); and distributional generality used information entropy to detect the less informative contextual features of hypernyms. Without imposing directionality specialization on distributional semantics, there may be little tangible information that can be captured to represent lexical entailment.
Further analysis indicated that the remarkable gain of Hierarchy-fitting+AD_indir and LEAR on recognizing hypernymy might be at the expense of their deteriorating outcomes on measuring semantic similarity, as shown in Figure 2. Hierarchy-fitting+AD_dir only used direct IS-A relations to enhance distributional semantics on measuring semantic similarity and distinguishing lexical entailment, and the results indicated that it matched well with hierarchy-fitting in balancing different semantic constraints. Injecting more indirect IS-A relations was detrimental to hierarchy-fitting+AD_indir and LEAR in deriving semantic similarity, adversely affecting semantic specialization on distributional semantics. This result has further supported our hypothesis that semantic transitivity caused by introducing multiple-level of lexical entailment may impose improper constraints on regularizing a distributional space.

Graded lexical entailment
We assessed hierarchy-fitting in identifying hypernymy and directionality as a binary classification task. On the other hand, as  has highlighted in HyperLex, apprehension of lexical entailment might exist as a gradual process in human semantic memory, and lexical entailment should be quantified with a graded or continuous value. We, therefore, further compare the semantic specialization methods in measuring graded lexical entailment in HyperLex. Table 5, without using any asymmetric distance to specify hypernymy's directionality, hierarchy-fitting attained best on PG, SN, GV, and FT in Group 1, whereas other methods, including retrofitting, counter-fitting, and LexSub, almost showed no improvement on each vanilla embedding. After factoring in the asymmetric character of hypernymy, hierarchy-fitting+AD_dir attained moderate improvements compared to hierarchy-fitting; and with the injection of more indirect IS-A relations, hierarchy-fitting+AD_indir further advanced hierarchy-fitting+AD_dir by 0.30 on average.

As shown in
Among the state-of-the-art methods in refining distributional spaces with asymmetric hypernymy, LEAR best measured graded lexical entailment but fell behind hierarchyfitting in deriving semantic similarity, as shown in Figure 2. HyperVec employed both direct and indirect IS-A relations in joint-training NNEs and was inferior to LEAR and hierarchy-fitting+AD_indir. LexSub created a separate subspace for hypernymy detection but gained a worsening result in HyperLex. These results indicated that adding asymmetric distance to generate the directionality of lexical entailment in late fusion might be more effective than both joint training in early fusion (Nguyen et al. 2017) and singling out a distributional sub-space dedicated to hypernymy (Arora et al. 2020). Table 5：Spearman correlation results of different specialization methods in computing graded lexical entailment on HyperLex.

Summary
Our semantic specialization method, hierarchy-fitting, can achieve state-of-the-art results in recognizing hypernymy in both binary and gradual evaluations. Since the evaluation datasets contain direct and indirect IS-A relations, we inject them to enhance NNEs through learning an asymmetric distance function to encode hypernymy's directionality. Overall, the advantage of hierarchy-fitting over other specialization methods such as LEAR and LexSub is that it can leverage multiple semantic constraints in detecting hypernymy and directionality without compromising semantic similarity computation.

Conclusion
Semantically specializing neural word embeddings aims to aggregate lexical and distributional semantics in early or late fusion. We have proposed to train a distance metric in late fusion to refine a distributional space to pursue the goal. Together, in a specialized space, metric learning can attract synonyms closer, repel antonyms farther, and pull hypo/hypernyms closer while maintaining hypernymy's directionality. Moreover, distributional semantics can be maximumly preserved when merging semantic constraints' contributions in loss functions. Our hierarchy-fitting method defined a quadruplet loss to regularize semantic difference among symmetric syn/antonymy and asymmetric IS-A relations in specializing neural embeddings, which can significantly improve distributional semantics in deriving semantic similarity and recognizing lexical entailment. This investigation corroborates those of earlier studies on semantic specialization on distributional vectors. It is scalable to other tasks, e.g. training a vector space model for measuring semantic relatedness and detecting mero/holonymy.
Post-processing methods can significantly reduce the computational workload caused by unsupervised joint-training processes to yield semantics-enhanced neural embeddings. To further improve such late fusion methods and enrich the semantic content of distributional vectors, we recommend that the follow-up phase of the study should concentrate on the following: 1. Training a unified distance metric while injecting different linguistic constraints.
An integrated loss function can simplify hyperparameters' tuning process in metric learning and improve the generalization of semantics-enhanced distributional vectors. The margins in the metric should be adjustable according to types of semantic relations. Hierarchy-fitting can apply a different margin to indirect hypernymy, distinguishing it from direct hypernymy to inherently reflect semantic nuances when taxonomic links of lexical entailment vary. A fixed margin will incur pushing in/direct hypernymy within the same distance range, inevitably miscalculating their semantic distance. A more flexible way of setting up margins is to dynamically adjust their values according to taxonomic distances on different semantic relations.
2. Refining a distributional space through multimodal semantic constraints. The present studies mainly focus on examining the linguistic aspect of semantic constraints, such as lexical semantics from WordNet, Roget's Thesaurus, and PPDB. Cross-lingual concept relations from BabelNet is also helpful to yield a unified embedding for different languages, given the unbalanced distribution of linguistic resources. Besides linguistic constraints, semantic content can be grounded on other perceptual sources such as vision and sound, e.g. hypernym detection using image generality (Kiela et al. 2015). Such heterogeneous data are complementary to constructing a robust and unified data representation in a multimodal semantic space (Bruni et al. 2014). Deep metric learning via late fusion can enrich a distributional space while lowering the complexity of early information fusion on symbolic and signal information. Doing so can harvest the synergy from mining hand-crafted knowledge resources, multimodal data, and the combination thereof.