{'fraction_non_alphanumeric': 0.03312737061054692, 'fraction_numerical': 0.006576532485011623, 'mean_word_length': 4.848479427549195, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval

Candidate generation is the first stage in recommendation systems, where a light-weight system is used to retrieve potentially relevant items for an input user. These candidate items are then ranked and pruned in later stages of recommender systems using a more complex ranking model. Since candidate generation is the top of the recommendation funnel, it is important to retrieve a high-recall candidate set to feed into downstream ranking models. A common approach for candidate generation is to leverage approximate nearest neighbor (ANN) search from a single dense query embedding; however, this approach this can yield a low-diversity result set with many near duplicates. As users often have multiple interests, candidate retrieval should ideally return a diverse set of candidates reflective of the user's multiple interests. To this end, we introduce kNN-Embed, a general approach to improving diversity in dense ANN-based retrieval. kNN-Embed represents each user as a smoothed mixture over learned item clusters that represent distinct `interests' of the user. By querying each of a user's mixture component in proportion to their mixture weights, we retrieve a high-diversity set of candidates reflecting elements from each of a user's interests. We experimentally compare kNN-Embed to standard ANN candidate retrieval, and show significant improvements in overall recall and improved diversity across three datasets. Accompanying this work, we open source a large Twitter follow-graph dataset, to spur further research in graph-mining and representation learning for recommender systems.

is to embed users and items in the same vector space, and then use approximate nearest-neighbor (ANN) search to retrieve candidates close to the user [7,20]. However, ANN search will often return candidate pools that are highly intra-similar (e.g., all candidates pertain to one 'topic' only) [37]. A side effect of training embeddings to place users close to relevant items, is that similar items are also placed close to each other. During ANN-based candidate retrieval, this unfortunately leads to similar candidates that may not reflect a user's diverse multi-topic interests, and hence low recall; we demonstrate this in Section 7.
In this paper, we introduce kNN-Embed, a new strategy for retrieving a high-recall, diverse set of candidates reflecting a user's multiple interests. kNN-Embed captures multiple user interests by representing user preferences with a smoothed, mixture distribution. Our technique provides a turn-key way to increase recall and diversity while maintaining user relevance in any ANN-based candidate generation scheme. It does not require retraining the underlying user and item embeddings; instead, we build directly on top of pre-existing ANN systems. The underlying idea is to exploit the similarity of neighboring users to represent per-user interests as a mixture over learned high level clusters of item embeddings. Since user-item relevance signal is typically sparse, estimating the mixture weights introduces significance variance. Thus, we smooth the mixture weights with information from similar users. At retrieval time, we simply sample candidates from each cluster according to mixture weights. Within each cluster, we perform ANN search using a smoothed per-user per-cluster embedding.
Our contributions in this paper are (1) a principled method to retrieve a high-recall, diverse candidate set in ANN-based candidate generation systems and (2) a large-scale open-source dataset for studying graph-mining, recommendation, and retrieval over real-world graphs.
After related work, in Section 2, and preliminaries, in Section 3, we describe a simple embedding framework that learns unimodal embeddings by co-embedding users and items in the same space in Section 4. In Section 5 we outline kNN-Embed, our proposed approach to transform unimodal embeddings into smoothed mixtures of embeddings. In Section 6 we introduce Twitter-Follow, a large graph dataset where users follow Twitter content producers (items) on Twitter that we curate and open-source to the community. In Section 7, we thoroughly evaluate the proposed solution along multiple axes, including recall, diversity, and goodness-of-fit; additionally, we perform a careful hyper-parameter study. Finally, we conclude and discuss future work in Section 8.

RELATED WORKS
Traditionally, techniques for candidate generation have focused on large sparse vectors, and have relied on fast, scalable approaches to search for similar sparse vectors from large target collections [1,4]. These approaches often apply innovative indexing and optimization strategies to scale similarity search.
One family of such approaches are cluster-based retrieval. These strategies generally build a static clustering of the entire collection of items; clusters are retrieved based on how well their centroids match the query [35]. These approaches have been applied to document and image retrieval [6,27]. On a high level, these methods either (1) match the query against clusters of items and rank clusters based on similarity to query or (2) utilize clusters as a form of item smoothing. Both of these approaches differ from kNN-Embed which represents the query as a mixture of clusters and selects candidate items from each cluster -ensuring a diverse retrieved set.
With the proliferation of embedding-based methods for recommender systems such as content-based recommendation [8] and collaborative-filtering recommender systems [40], there have been approaches developed for performing similarity search in large collections of dense targets. Initial approaches for candidate generation in dense-spaces have proposed hashing-based techniques such as mapping input and targets to discrete partitions and selecting targets from Manuscript submitted to ACM the same partitions as inputs [36]. With the advent of fast approximate nearest-neighbor search [19,29,33], dense nearest neighbor has been applied by recommender systems for candidate generation [7].
The application of scalable graph-based embedding methods has significantly improved recommender systems [12,38].
With this improvement, there have been methods that have attempted to extend the embeddings used from single-mode to multiple mode representations [32]. These, and other [13], methods apply clustering over user actions to represent users with multiple embeddings. Our method extends upon this idea by incorporating nearest neighbor smoothing such that engagements of neighboring users can enhance the overall user representation. We believe this is crucial to address the sparsity problem of generating mixtures of embeddings for users with few engagements.
Smoothing via k-nearest-neighbor search has been applied for better language modeling [22] and machine translation [21]. Our approaches adopts a similar insight to smooth our mixture embeddings by leveraging engagements from similar users. We posit this is especially helpful for better representations for users with sparse engagement.

PRELIMINARIES
Let U = { 1 , 2 , . . . } be the set of source entities (i.e., users in a recommender system) and I = { 1 , 2 , . . . } be the set of target entities (i.e., items in a recommender system). Let G constitute a bipartite graph representing the engagements between users (U) and items (I) as shown in Figure 1. For each user and item, we define a 'relevance' variable in {0, 1} indicating an item's relevance to a particular user. An item is considered relevant to a particular user if a user, presented with an item, will engage with said item.
Based on the engagements within G, for each user, , is associated with a -dimensional embedding vector u j ∈ R ; similarly each target item is associated with an embedding vector i k ∈ R . We will call these the unimodal embeddings, and assume that they model source-target relevance (relevance| , ) = (u j , i k ) for some suitable function .
Given the input user-item engagement graph, our goal is to learn mixtures of embeddings representations of users.
These mixture embeddings should better capture the multiple interests of a user as evidenced by higher recall in a candidate retrieval task.

LEARNING UNIMODAL USER AND ITEM REPRESENTATIONS
We apply a simple approach to co-embed users and items into the same space to learn our initial unimodal embeddings for each. To perform this embedding, we construct a bipartite graph G consisting of users and items, where an edge constitutes a form of engagement (e.g., citing paper cites cited paper or user follows content producer). We seek to learn an embedding vector (i.e., vector of learnable parameters) for each user ( ) and item ( ) in this bipartite graph; Manuscript submitted to ACM we denote these learnable embeddings for users and items as u j and i k respectively. A user-item pair is scored with a scoring function of the form (u j , i k ). Our training objective seeks to learn u and i parameters that maximize a log-likelihood constructed from the scoring function for ( , ) ∈ G and minimize for ( , ) ∉ G.
For model simplicity, we apply a simple dot product comparator between user and item representations. For a user-item pair = ( , ), this operation is defined by: As seen in Equation 1, we co-embed users and items by scoring their respective embedded representations via dot product. The task is then formulated as an edge (or link) prediction task. We consume the input bipartite graph G as a set of user-item pairs of the form ( , ) which represent positive engagements between a user and item. The training objective of the embedding model is to find user and item representations that are useful for predicting which users and items are linked via an engagement. While a softmax is a natural formulation to predict a user-item engagement, it is impractical because of the prohibitive cost of computing the normalization over a large vocabulary of items. As such, following previous methods [15,30], negative sampling, a simplification of noise-contrastive estimation, is used to learn the parameters u and i. We maximize the following negative sampling objective: represents the log-likelihood of predicting a binary "real" or "fake" label for the set of edges in the network (real) along with a set of the "fake" negatively sampled edges. To maximize the objective, we learn u and i parameters to differentiate positive edges from negative, unobserved edges.
Negative edges are sampled by corrupting positive edges via replacing either the user or item in an edge pair with a negatively sampled user or item. As user-item interaction graphs are very sparse, randomly corrupting an edge in the graph is very likely to be a 'negative' edge absent from the graph. Following previous approaches, negative sampling is performed both uniformly and proportional to node prevalence in the training graph [5,24]. Optimization is performed via Adagrad [10].

SMOOTHED MIXTURE OF EMBEDDINGS
In Section 4, we described a simple approach to co-embedding users and items to learn unimodal embeddings. In this section, we describe how kNN-Embed can be applied to transform unimodal embeddings into smoothed mixtures of embeddings. We then demonstrate an approach to candidate retrieval using these smoothed mixtures that leads to diverse, higher-recall candidate items.

Proposed Method: kNN-Embed
To use embeddings for candidate generation, we need a method of selecting relevant items given the input user. Ideally, we would like to construct a full distribution over all items for each user ( | ) and draw samples from it. The sheer number of items makes this difficult to do efficiently, especially when candidate generation strategies are meant to be light-weight. In practice, the most common method is to greedily select the top few most relevant items using an ANN search with the unimodal user embedding as query. A significant weakness of this greedy selection is that, by its nature, ANN search will return items that are similar not only to the user embedding, but also to each other; this drastically reduces the diversity of the returned items. This reduction in diversity is a side-effect of the way embeddings are trained -typically, the goal of training embeddings is to put users and relevant items close in Euclidean space; however, this also places similar users close in space, as well as similar items. We will repeatedly exploit this 'locality implies similarity' property of embeddings in this paper to resolve this diversity issue.
We summarize our approach before providing a richer description. We model the distribution of user preferences for items, ( | ), using a mixture over learned item clusters denoted by . In order to generate candidates for a user , we take the following steps: • Using Equation 5, compute cluster distribution smoothed ( | ).
• Sample times with replacement from smoothed ( | ) to obtain cluster counts for each cluster.
• For each item cluster , compute the cluster-specific query embedding u using Equation 7 and retrieve items using ANN. The union of these cluster samples is the candidate pool.
Clustering Items Since neighboring items are similar in the embedding space, if we apply a distance-based clustering to items, we can arrive at groupings that represent individual user preferences well. As such, we first cluster items using spherical k-means [9]. Under this scheme, each embedding vector is first normalized to unit norm, and, after each epoch, cluster centroids are similarly normalized. As such, all cluster centroids are placed on a high-dimensional sphere with radius one.
Given these item clusters, instead of immediately collapsing the distribution ( | ) to a few items as ANN search does, we can write the full distribution ( | ) as a mixture over item clusters: where in each cluster, we now want to learn a separate distribution over the items in the cluster ( | , ). Thus, we are modeling each user's higher level interests ( | ), and then within each interest , we can apply an efficient ANN-search strategy as before. In effect, we are interpolating between sampling the full preference distribution ( | ) and greedily selecting a few items in an ANN.

Mixture of Embeddings via Cluster Engagements
After clustering target entities, we need to learn ( | ). The natural method is the maximum likelihood estimator (MLE) given by: where, count( , ) is the number of times has a relevant item in cluster . For computational efficiency, we take M to be 's top most relevant clusters. We normalize these counts to obtain a proper cluster-relevance distribution.
Nearest Neighbor Smoothing Unfortunately, we typically have little user-item engagement data on a per-user basis; thus, while the MLE is unbiased and asymptotically efficient, it can also be high variance. To this end, we introduce a smoothing technique that once again exploits locality in the ANN search, this time for users.
In Figure 2, we illustrate our scheme whereby we identify nearest-neighbors to the query user, and leverage information from the neighbors' cluster engagements to augment the user's cluster relevance. To do this, we query the nearest-neighbor index to retrieve 's -nearest user neighbors K . We then compute a distribution over item clusters  Fig. 2. Example of retrieving two candidates. In an ANN, items 4 and 5 would be deterministically returned for user 1. In our proposed kNN-Embed, even though the distances to cluster 2 are larger, smoothing means that we will sometimes return items from that cluster, yielding more diverse items. Note in this case, we don't even require that user 1 has previously relevant items in cluster 2.
by averaging the MLE probability for each nearest neighbor (item clusters that are not engaged with by a retrieved neighbor have zero probability).
While Dirichlet smoothing is a natural Bayesian approach to smoothing our multinomials, the lack of a closed form estimate makes the approach impractical at scale [31,39]. As an alternative, we apply Jelinik-Mercer smoothing to interpolate between a user's MLE distribution with the aggregated nearest neighbor distribution [16,18,22].
where ∈ [0, 1] represents how much smoothing is applied. It can be manually set or tuned on a downstream extrinsic task.
Sampling within Clusters Within each cluster there are many ways to retrieve items on a per user basis. A simple, but appealing, strategy is to represent each user as a normalized centroid of their relevant items in that cluster: where ( , ) is the set of relevant items for user in cluster . However, since we are applying smoothing to the cluster probabilities ( | ), it may be case that has zero relevant items in a given cluster. Hence, we smooth the user centroid by using information from the neighbors to obtain the final user representation u c j : Equation 7 shows the -smoothed user-specific embedding for cluster . This embedding takes the user-specific cluster representations from Equation 6, and performs a weighted averaging proportionate to each user's contribution to smoothed ( | ). The final centroid vector is once again normalized to unit norm.

EVALUATION DATASETS AND METRICS
In this section we describe the datasets we evaluate on, including Twitter-Follow, a Twitter Follow graph that we curated, and are open-sourcing alongside our work. Additionally, we introduce three metrics that we use to evaluate the quality of embedding-based candidate retrieval methodologies.

Datasets
We evaluate kNN-Embed on three datasets with their associated candidate retrieval tasks, which we describe below: HEP-TH Citation Graph: Our first dataset we evaluate on is arXiv HEP-TH -a high energy physics theory academic paper citation graph [14]. This paper citation network is collected from Arxiv preprints from the High Energy Physics category. The dataset consists of: 34,546 papers and 421,578 citations. The task is to retrieve candidate papers that a user may cite for a given source paper.

DBLP Citation Graph:
The second dataset we evaluate on is the DBLP citation network -a paper citation network extracted from DBLP [34]. The dataset consists of 5,354,309 papers and 48,227,950 citation relationships. Once again, the task is to retrieve candidate papers that a user may cite for a given source paper.
Twitter Follow Graph: As an accompanying resource to this paper, we created this dataset by querying Twitter follows for each user (available via API) and heavily sub-sampling this user follows user graph. Each edge is directed, and the presence of symmetric edges indicates that both users follow one another. We construct this graph by first selecting a number of 'highly-followed' users that we refer to as 'content producers'; these content producers serve as 'items' in our recommender systems terminology. We then sampled users that follow these content producer accounts. All users are anonymized with no other personally identifiable information (e.g., demographic features) present. Additionally, the timestamp of each follow edge was mapped to an integer that respects date ordering, but does not provide any information about the date that follow occurred. In total, we have 261 edges and 15.5 vertices, with a max-degree of 900 and a min-degree of 5. We hope that this dataset will be of useful to the community as a test-bed for large-scale retrieval and embedding research.

Metrics
We evaluate kNN-Embed on three aspects: (1) the recall of retrieved candidates as measured on a held-out set of items (2) the diversity of retrieved candidates and (3) the goodness of fit, namely how well a multi-interest mixture of embeddings models the user's interests as evidenced through a held-out set of items. Below, we formalize these metrics.

Recall@K:
The most natural (and perhaps most important) metric for computing the efficacy of various candidate retrieval strategies is @ . This metric is given by considering a fixed number of top candidates yield by a retrieval system (up to size ) and measuring what percent of these candidates are held-out relevant candidates. The purpose of most candidate retrieval systems is to collect a high-recall pool of items for further ranking, and thus recall is a relevant metric to consider. Additionally, recall provides an indirect way to measure diversity -to achieve high recall, one is obliged to return a large fraction of all relevant documents, which simple greedy ANN searches can struggle with.
Diversity: To evaluate the diversity among the retrieved candidates, we measure the spread in the embeddings of the retrieved candidates by calculating the average distance retrieved candidates are from their centroid. The underlying idea is that in for embeddings, 'locality implies similarity'; as a corollary, if candidates are further in Euclidean distance, then they are likely to be different. As such, for a given set of candidates C, we compute diversity as follows: where C denotes the set of retrieved candidates andî = ∈ C i k /|C| is the mean of the unimodal embeddings of the retrieved candidates. We could also have used the square norm here, which gives largely similar results. Since our embeddings are normalized, square norms tend to be very small and quite difficult to interpret.

Goodness of Fit:
In addition to diversity of retrieved items, we need to ensure that a user's mixture representation is an accurate model of their interests -that is the mixture of embeddings identifies points in the embedding space where relevant items lie. Thus, we compare held out relevant items to the user's mixture representation we use to query. We measure this "goodness of fit" by computing the Earth Mover's Distance (EMD) [25] between a uniform distribution over a user's relevant items and the user's mixture distribution. The EMD is a metric that measures the distance between two probability distributions over a metric space. In our case, we measure the distance between a user's cluster distribution (e.g., Equation 3 and Equation 4), to a uniform distribution over a held-out set of relevant items: ( | ). Following previous works, distance is measured over the Euclidean embedding space [11,23]. In our case, the EMD is computed by soft assigning all held-out relevant items of a user to clusters, minimizing the sum of item-cluster distances, with the constraint that the sum over soft assignments matches ( | ). As seen in Figure 3, with standard unimodal representations, a single embedding vector is compared to the held-out items and the goodness of fit is the distance between the item embeddings and the singular user embedding.
In comparison, Figure 4, shows our mixture embeddings, under this model, we model each user with multiple user embeddings who each have fractional probability mass that in total sums to 1. The goodness of fit is then the distance achieved by allocating the mass in each item to the closest user embedding cluster with available probability mass.
Observing Fig. 3, a single unimodal embedding is situated in the embedding space and compared to held-out relevant items. As shown, some held-out items are close to the unimodal embedding, while others are further away. In contrast, Fig. 4, multiple user-embeddings represent each user where each embedding lies close to a cluster of relevant items.
The intuition is that if a user has multiple item clusters they are interested in, multiple embeddings can better capture these interests.
We compute this goodness of fit EMD between held-out items and a user representation by solving the linear program: Where ∈ | |× | I | is a non-negative matrix, where each , denotes how much of cluster for user is assigned to item for user , and constraints ensure the flow of the probability mass for an item or cluster cannot exceed its allocated mass. Specifically, this formulation ensures the the entire outgoing flow from cluster equals ( | ), i.e. , = ( | ). Additionally, the amount of incoming flow to item must match ( | ), i.e., , = ( | ).

EXPERIMENTS
In this section, we evaluate the performance of kNN-Embed and compare it against baseline unimodal embeddings upon which it's built. Additionally, we perform ablation experiments to isolate the improvement due to smoothing.

Experimental Setup
For our underlying ANN-based candidate generation system, we start by creating a bipartite graph between source entities and target entities for each dataset, with each edge representing explicit relevance between items (e.g., citing paper cites cited paper or user follows content producer). Given each bipartite graph, we learn unimodal embeddings for users and items by follows the steps as described in Section 4. We learn 100-dimensional user and item embeddings and train our embeddings over 20 epochs. For the clustering algorithm, we apply the spherical -means algorithm for 20 epochs to cluster items based on their unimodal embedding vectors [3,28].
Evaluation Task: We evaluate three candidate generation strategies -baseline ANN with unimodal embeddings (which is how most ANN-based candidate generation systems work), mixture of embeddings with no smoothing (as is done in [32]), and mixture of embeddings with smoothing (as described in Section 5). For each strategy, we compute the @ , diversity, and fit in a link prediction task i.e., predicting which additional papers to cite, or which accounts to follow, given the training set. Recall provides a natural way to measure diversity -to achieve high recall, one is obliged to return a large fraction of all relevant documents, which simple greedy ANN searches are unable to achieve.
In addition, explicitly measuring diversity via average deviation of retrieved candidates and goodness of fit via EMD metrics shows that a mixture of embeddings is naturally a superior paradigm to model user-item engagements for the purpose of multi-interest candidate generation.
Research Hypotheses: We aim to explore two research hypotheses (as well as achieve some understanding of the hyperparameters): (1) Unimodal embeddings miss many relevant items due to the similarity of retrieved items. Mixture representations can yield more diverse and higher recall candidates.
(2) Smoothing, by using information from neighboring users, further improves the recall of retrieved items. In Table 1 and Table 2, we report results when evaluating recall on citation prediction tasks for the HEP-TH and DBLP citation networks. Results support the first hypothesis that unimodal embeddings may miss relevant items if they don't lie close to the source entity in the shared embedding space. When we utilize a mixture of embeddings with no smoothing, we see a 14% relative improvement in @10 for for HEP-TH, and 16% relative improvement for DBLP.

Recall
Our second hypothesis (2) posits that data sparsity can lead to sub-optimal mixtures of embeddings, and that nearestneighbor smoothing can mitigate this. Our experiments support this hypothesis, as we see a 25% relative improvement for HEP-TH in @10, and 35% for DBLP and when using kNN-Embed. We see similar significant improvements over baselines in @20 and @50.
In Table 3, we report results for kNN-Embed and baselines for our new Twitter follow prediction task (see Section 6).
In this case, the improvements in recall are dramatic -534% in relative terms going from unimodal embeddings to a mixture of embeddings in @10. We suspect this significant improvement is because Twitter-Follows simulatenously has a much higher average degree than HEP-TH and DBLP and the number of unique nodes is much larger. It is a more difficult task to embed so many items, from many different interest clusters, in close proximity to a user. As such, we see a massive improvement by explicitly querying from each user's interest clusters. Applying smoothing provides an additional 74% in relative terms, and similar behaviours are observed in @20 and @50.

Diversity
We apply Equation 8 to retrieved candidates, and measure the spread of the distances between each retrieved candidate's embedding vector to the centroid of the retrieved candidates. As seen in Tables 4, 5, and 6, we notice that candidates generated via unimodal retrieval are less diverse than candidates generated via multi-interest. This verifies our first research hypothesis that unimodal embeddings may retrieve many items that are clustered closely together as a by-product of ANN retrieval (i.e., diversity is low, and so is recall). However, querying multiple times from mixtures of Interestingly we notice that kNN-Embed (i.e., smooth mixture retrieval) results in slightly less diverse candidates than unsmoothed mixture retrieval. We posit that this is due to the high-variance of the maximum likelihood estimator of the mle ( | ) multinomial (Equation 3). While this high-variance may yield more diverse candidates, we believe variance results in less accurate candidates retrieved; this is verified by cross-referencing diversity results with recall as seen in Tables 1, 2, and 3 where kNN-Embed consistently yields better recall than unsmoothed mixture retrieval. While high diversity is necessary for high recall, it is insufficient on its own (e.g., returning uniformly random candidates is high diversity, and low recall), and is not unconditionally desirable in candidate generation.

Goodness of Fit
In addition to measuring the diversity of retrieved candidates, we also seek to evaluate how well modeling users via unimodal, mixture, and smoothed mixture embeddings fits a user's interest. To facilitate this, we evaluate the goodness of fit between user representations used for retrieval to a set of held-out relevant items for each user. The main idea is that the better fit a user representation is, the closer it will be to the distribution of held out relevant items for that user.

Hyper-parameter Ablation
To better understand how hyper-parameters affect the performance of kNN-Embed, we perform a study on the HEP-TH dataset. In this section, we focus on recall as the sine qua non of candidate retrieval problems. In Figure 5a, we vary the smoothing parameter -for simplicity, we use the same parameter for smoothing the both the mixture probabilities and the cluster centroids. We see that for our HEP-TH dataset, using heavy smoothing improves performance significantly.
This likely stems from the sparsity of HEP-TH where most papers have only a few citations.
Next, in Figure 5b, we vary the number of embeddings (i.e., the mixture size). We find that performance improves markedly until saturating at six mixture components. Out of all the hyperparameters, this seems to be the critical one in achieving high recall. In practice, latency constraints can be considered when selecting the number of embeddings per user, explicitly making the trade-off between diversity and latency. One interesting aspect is that kNN-Embed with a single embedding performs only slightly worse than the original unimodal embedding. The comparable performance supports our first hypothesis that a unimodal user embedding is often placed next to the single item cluster it has the most engagement to, leading to the lack of diversity in candidates.
Finally, in Figure 5c, we vary the number of clusters in the NN clustering of items. For this particular dataset, recall peaks at = 2500 and then decreases. HEP-TH is a small dataset with only 34,546 items; it is likely that generating a very large number of clusters leads to excessively fine-grained and noisy sub-divisions of the items.

CONCLUSIONS AND FUTURE WORK
In this work, we present kNN-Embed, a method of transforming a single user dense embedding, into a mixture of embeddings, with the goal of increasing retrieval recall and diversity. This multi-embedding scheme effectively represents a source entity with multiple distinct topical affinities by globally clustering items and aggregating the source entity's engagements with clusters. Recognizing that user-item engagements may often be sparse, we propose a novel nearest-neighbor smoothing to enrich an entities mixture representation from similar entities. Our proposed smoothed mixture representation better models user preferences retrieving a diverse set of candidate items reflective of a user's multiple interests. This significantly improves recall on a candidate generation retrieval task on three datasets: DBLP citation prediction, ArXiv HEP-TH citation prediction, and Twitter user account suggestion. In conjunction with this work, we open-source our curated Twitter Follow Graph dataset as a resource to the information retrieval community.