{'fraction_non_alphanumeric': 0.031135855881414234, 'fraction_numerical': 0.008901024859037555, 'mean_word_length': 4.8786741713570985, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning User Representations with Hypercuboids for Recommender Systems

Modeling user interests is crucial in real-world recommender systems. In this paper, we present a new user interest representation model for personalized recommendation. Specifically, the key novelty behind our model is that it explicitly models user interests as a hypercuboid instead of a point in the space. In our approach, the recommendation score is learned by calculating a compositional distance between the user hypercuboid and the item. This helps to alleviate the potential geometric inflexibility of existing collaborative filtering approaches, enabling a greater extent of modeling capability. Furthermore, we present two variants of hypercuboids to enhance the capability in capturing the diversities of user interests. A neural architecture is also proposed to facilitate user hypercuboid learning by capturing the activity sequences (e.g., buy and rate) of users. We demonstrate the effectiveness of our proposed model via extensive experiments on both public and commercial datasets. Empirical results show that our approach achieves very promising results, outperforming existing state-of-the-art.


INTRODUCTION
We propose a novel recommendation approach with hypercuboids. The hypercuboid (also known as hyperrectangle) is a generalization of a three-dimensional cuboid to four or more dimensions. More specifically, we use the expressive hypercuboids to represent users in order to capture their complex and diverse interests. Recommendations are made based on the relationships between user hypercuboids and items. This work is motivated by the recent popularity and growing importance of recommender systems in real-world applications (e.g., e-commerce and online entertainment platforms). As an effective tool to ameliorate information overload and generate income, its importance cannot been overestimated. A major step in recommender systems is finding the right items that suit users' preferences and interests. However, obtaining informative user representations is a challenging task as users can have a wide range of interests and different tastes concerning different categories of items. The massive amount and diversity of item sets in real world applications make the case even worse.
Common user modeling approaches aim to represent users as points in vector space. For example, the standard factorization model [17] factorizes the large interaction matrix into user and item latent embeddings in low-dimensional space. As a result, recommendation is reduced to identifying items which are near or similar to the user in that space. Despite its popularity and wide applications, this paradigm faces weaknesses. On one hand, embedding the disparate and diverse interests into single points is problematic [42]. Intuitively, users are likely to have different tastes arXiv:2011.05742v1 [cs.IR] 11 Nov 2020 and considerations for items from different categories or domains. On the other hand, for most factors, users have preferred ranges spread. For example, consumers usually have their acceptable price range. It is inexpressive to model such range with a single value. Clearly, such systems are geometrically restrictive.
Thus, we need a much richer latent representation system to overcome these limitations. To this end, we propose a new paradigm that represents users as hypercuboids. We go beyond point embeddings, adopting the more expressive hypercuboids to effectively capture the intricate and complex interests of users. A hypercuboid is a subspace that encapsulates innumerable points, enabling more representation capacities than a single point. Moreover, the preference range of each factor (e.g., accepted price) can be naturally modelled with the edge of hypercuboids.
Overall, the basic idea is that: for each user, we learn single or multiple hypercuboids to represent the user's interests. The relationship between the user hypercuboids and item points is captured by a re-designed distance measure, where both distances outside and inside the hypercuboid are considered. Ideally, the user hypercuboid will allow for a greater degree of flexibility and expressiveness.
The main contributions of this work are as follows: • We present a novel user representation method for personalized recommendation. For the first time, we adopt hypercuboids to represent users' complicated user preferences or interests. Two variants are proposed to further enhance the representation capacity in capturing the diversity of user interests. • We design a distance function to measure the distance between items and user hypercuboids. This measure provides improvements in terms of flexibility and model expressiveness of the algorithms. • We conduct experiments on six datasets. Our model achieves the best performances on all the benchmark datasets. We investigate the inner workings of our model to gain a better intuition pertaining to the model's high performance. We also provide case study on real-world datasets to show the effectiveness of the proposed approach.

THE PROPOSED METHOD
In this section, we formulate the hypercuboid representation for user interest modeling, introduce two variants of hypercuboid, and present the overall architecture for hypercuboid learning. Our work is concerned with the recommendation task with implicit feedback. In a standard setting, we suppose that there are a set of users, ∈ {1, ..., |U|} and a set of items, ∈ {1, ..., |I|}. The goal is to generate a set of recommendable items for a user given existing feedback, typically, implicit feedback such as clicks, buys, likes. We set the interaction between user and item , to 1 if the implicit feedback exists, otherwise, = 0, where is an entry of the interaction matrix Y ∈ R | U |× | I | . In addition, time steps will also be considered in the following subsections.

Representing Users as Hypercuboids
Hypercuboid is an analogue of a cuboid in four or more dimensions. For the convenience of computation, we define a hypercuboid with are not distinguishable when user is a point. Representing the user as a hypercuboid can effectively address this problem.
a center and an offset, both with the same dimensions. Formally, we operate on R . Let c ∈ R represent the center of a hypercuboid and f ∈ R 0+ denote the non-negative offset. The offset is used to determine the lengths of the edges of the hypercuboid. A hypercuboid is defined as: where p denotes the point which is inside the hypercuboid. In our framework, every user is modeled with such a hypercuboid. We use bold uppercase letters C ∈ R |U |× and F ∈ R |U |× 0+ to denote the centers and offsets respectively for all the users. As such, a user is represented by a hypercuboid user (C , F ). Items are points and are embedded by V ∈ R |I |× . Obviously, an item point can be outside, inside, or on the surface of the user hypercuboid. A proper method is needed to measure the relationships between users and items. Meanwhile, the characteristics of hypercuboid should be retained.
To achieve this, a composition points-to-hypercuboid distance is adopted. It consists of an outside distance and an inside distance. Firstly, we find the nearest point on the surface of the hypercuboid to the item, which is performed with the following expression: where p max and p min denote the upper-right corner and lower-left corner of the user hypercuboid (shown in Figure 2). Their formal definitions are: It is clear that p is relevant to both the user hypercuboid and the item embedding. The movement of either will result in transformation of this point.
Then, the outside distance is the distance between p and V , and the inside distance indicates the distance between p and the center of the hypercuboid. So we have: where euclidean distance is used. Then, we compose these two distances as follows to measure the user-item relationships.
where the coefficient is employed to control the contribution of the inside distance. Theoretically, if we set = 0, we presume that the user is interested in an item regardless of its exact coordinate as long as this item is located in the user hypercuboid. This might alleviate the congestive problem of common approaches which map users and her interacted items onto the same points [35]. This paradigm may suggest two benefits. On one hand, a user hypercuboid can be regarded as a set that contains countless item points, which means that it could enwrap as many items as the user likes without much loss of information, enabling higher capacities. On the other hand, the composition distance in (3) could overcome an obvious impediment of the direct distance adopted in the existing body of literature. To explain the latter, suppose that we have two items located at the circumference of the circle with center C . With commonly used euclidean distance, the system will conclude that this user likes both items equally. However, consider the case in Figure 2. As we can see, this might not be true when the edges of the hypercuboid is taken into account: it is clear that the distances to the hypercuboid of item 1 and item 2 are not equal. This is provable with the cosine theorem, or by directly setting the hyper-parameter to 0. In this example, we can clearly conclude that ℓ ( , item 2 ) > ℓ ( , item 1 ) so this user is more interested in the item 1 . One might think of the offset as a range for user preference in each dimension. If the -axis indicates prices and the -axis represents styles, it is evident that this user has less tolerance in style than that in the price factor. As such, even though item 2 has a more acceptable price than item 1 for this user, she still likes item 1 more because the item 2 does not suit her taste in style at all. To ensure the items in a hypercuboid to be higher ranked than items outside the hypercuboid, we can add an additional distance to ℓ ( , ), where is set to a value greater than 100 and is the sigmoid function. This distance equals to zero when item is in the hypercuboid. Otherwise, it is approximately greater than the half of the maximum diagonal length of the hypercuboid.

Variants of Hypercuboids
To further enhance the capability, we devise the following two variants of hypercuboid. The rational is to generate multiple hypercuboids, with each one providing a different view for user interests. This could improve the capability in capturing the diversities of user interests. Figure 3 illustrates these two variants.

Concentric Hypercuboids.
One solution is to use multiple concentric hypercuboids to represent each user. In this case, we add several offset embeddings and keep the center unchanged. With concentric hypercuboid we can not only extend the regular hypercuboid into irregular space, but also mitigate the above problem. Suppose we have offsets, there are two distances for each hypercuboid. We integrate them into the following final distance: All the outside distances are summed up while only the minimum inside distance is kept. The motivation is to weaken the effects of the inside distances.

Multiple Independent Hypercuboids.
To further enhance the capability in modeling diverse interests, we can also use multiple hypercuboids with different centers and offsets. The final distance function is defined as: As for the addition distance, we can take the maximum additional distance from all hypercuboids.

Learning the Centers and Offsets
An important step is to learn the center and offset of the user hypercuboid. Generally, user interests can be derived from users' historical behaviors, e.g., buy, watch, and click. So it is natural to learn the centers and offsets from the past interaction log. In most practical applications, user actions are often ordered and timestamped, which also records users' interests evolution. Formally, we make use of the last actions of users, and aim to predict the next recommendable item(s). At the embedding layer, we get an embedding matrix S ( ) ∈ R × , to represent these items at time-step , by looking up the item embedding matrix V with the index of the items.
To capture the shifts of user interests, we employ a stacked neural network consisting of a bidirectional LSTM [28] and a self-attention module over the sequential actions.
The bidirectional LSTM is used to enhance the sequence representation capability. Then, a self-attention network is applied.
where a ∈ R × is transformed representation for the current sequence; is nonlinear layer. It is then passed into a pooling layer to get a dimensional vector.
where the pooling function can be average-pooling, max-pooling, min-pooling, or sum-pooling. We can use s ( ) to represent the center, the offset, or both. In addition, we notice that the offset of hypercuboid usually needs to memorize mass amount of information. For example, to get the price interval, we need to store the prices of all items the user bought to accurately infer her acceptable price range. To this end, we adopt a key-value memory network to perform the memorization process. Let M ∈ R × denote the memory and K ∈ R × denote the key matrix, where controls the capacity of the memory.
The input of this memory module is s ( ) , which is used to retrieve the key from the key matrix.
With this attentive key vector k, we select relevant pieces of information from the memory.
We wrap all the offsets with the function [5] to ensure all the values are non-negative. Figure 4 illustrates the overall architecture of the proposed model.
For multiple hypercuboids, we can add dense layers over the center/offset and get multiple centers and offsets.

Optimization
In this section, we introduce the objective function and training scheme. We adopt the pairwise ranking loss for the optimizing objective, which is defined as.
where T + and T − are the positive user-item pairs and negative useritem pairs respectively, is the margin that separates the two items. For regularization, standard 2 normalization and dropout [31] are viable. We use the same negative sampling strategy as that of HGN [20]. Our model is end-to-end differentiable and an adaptive gradient descent optimizer is used. It is noteworthy that other losses such as Bayesian personalized ranking loss or cross entropy loss are also applicable.

EXPERIMENTS
In this section, we present our experimental setup and empirical evaluation to verify the effectiveness of our method.

Datasets
We conduct our empirical evaluation based on four public wellstudied benchmark datasets and two commercial datasets. The statistics of the datasets are reported in Table 1. 3.1.1 Public Datasets. Four public datasets from different domains are used, including Amazon-Books, Amazon-Movies&TVs, Amazon-CDs [8,20,23] and an E-commerce dataset [19]. The feedback in the Amazon datasets is ratings from 1 to 5, we treated those with no less than 4 as positive feedback. For the E-commerce dataset, we use the version provided by Lv et al. [19] which consists of user-item interaction logs of four weeks. To filter noisy data, we discard users with fewer than ten actions and items with fewer than 5 associated actions for each dataset [20].

Commercial
Datasets. This dataset is from an online retailer platform, Alibaba. It is collected from the interaction logs generated in seven days. We sampled two versions for two groups of users. The first version consists of users with 100 to 500 interactions and the second version consists of users with 50 to 100 interactions.
We target at the next item prediction task where the aim is to predict a user's next interaction. As such, each dataset is split into three subsets based on the chronological order of interactions, with 70% of the user sequence as the training set, the next 10% as the validation set and the latest 20% as the test set. Same as [20], the validation set is used during the inference stage.

Evaluation Measures
Three commonplace evaluation measures [29] including Recall@k, Normalized Discounted Cumulative Gain (NDCG@k), and Mean Average Precision (MAP@k) are adopted. For each user, we rank all the items (exclude items the user has rated) and select the highest ranked k items for recommendation. Recall at indicates the proportion of relevant items found in the top-recommendations. NDCG at is a measure of ranking quality and the position of correctly recommended item is taken into account. MAP at is the mean of the average precision scores of all users and is a precision measure that gives larger credit to correctly recommended items. Definitions are omitted for brevity.

Compared Methods
We compare our method with two groups of well-established and competitive baselines. The first group contains two non-neural recommendation methods.
• Matrix Factorization with Bayesian Personalized Ranking (BPRMF) [26] is a classical method with pairwise Bayesian ranking loss. It models user-item interactions with matrix factorization. • Translational Recommender Systems (TransRec) [6,7] is a sequential recommendation model where users and items are embedded into a transition space following the prev. item + user ≈ next item translation operation.
Five neural networks based recommendation models are included in the second group.
• YouTubeDNN [2] is a deep neural networks based recommendation model proposed by YouTube 1 . User vectors, vectors of the latest interacted items and the target item are concatenated and are fed into multiple dense layers. • Gated Recurrent Unit for Recommendation (GRU4Rec) [10] is a recurrent neural network based model for session-based recommendation (a user is a session).

• Convolutional Sequence Embedding Recommendation
Model (Caser) [34] is a convolutional neural networks based sequence-aware recommendation model. The proposed framework captures the sequential patterns at both point-level and union-level. • Self-Attentive Sequential Recommendation (SASRec) [15] is a self-attention [36] based recommendation model, where user sequences are modelled with a transformer like network. • Hierarchical Gating Networks (HGN) [20] is a state of the art sequential recommendation model which is based on a novel gating mechanism and includes item-item relations.

Implementation Details
We implemented our model and baselines with tensorflow 2 . For all models, the embedding size is set to 100 for fair comparison, the sequence length and the target length are set to 5 and 3, respectively. Adagrad optimizer [3] is used for model parameters updates and the learning rate is set to 0.05. The ℓ 2 regularization rate is tuned amongst {0.  Table 2 and Figure 5 present the recommendation performance of all baselines and our model with a single hypercuboid on the six datasets. The best performer on each row is highlighted in boldface.

Experimental Results
We observe that our model achieves the best performance across all datasets, outperforming a myriad of complex architectures. Our model consistently performs better than HGN, SASRec, Caser, and TransRec, which are all recent competitive sequence-aware recommendation methods. Notably, we obtain a clear performance gain over the second best baseline. Furthermore, our method performs much better than the inner product model (BPRMF) and the Euclidean distance based model (TransRec), which ascertains the effectiveness of the proposed distance function. Among all the neural networks based baselines, HGN outperforms self-attention, RNN, and CNN based approaches. The second best neural networks based model often switches between YouTubeDNN and SASRec. We also note that non-neural models such as BPRMF and TransRec achieve very competitive performance on the three Amazon datasets and the E-commerce dataset.
Performance of top 5, 10, 20, 30, 50 on four public datasets are presented in Figure 5. Our model consistently outperforms all the other models, indicating that our model can generate not only more accurate topmost recommendations but also higher quality recommendation lists on the whole.

Adding More Hypercuboids
We conduct experiments to verify the effectiveness of the proposed hypercuboid variants. Table 3 shows the performance with varying number of hypercuboids. Notably, adding more hypercuboid can further enhance the model performance, hitting a new state-ofthe-art, which reaffirms the importance of modeling the diversity of user interests. Generally, multiple independent hypercuboids usually marginally outperforms multiple concentric hypercuboids. Using four to six hypercuboids can usually lead to a satisfying improvement. We also tried eight and ten hypercuboids but performance degradation is observed. Note that these two variants will not incur large increases in the number of parameters as increasing the embedding size. We can choose the number of hypercuboids base on the degree of diversity of users' interests in general.

Using Hypercuboids Only
We study the impacts of the neural architecture on the model performance. To this end, we report the performance by replacing all the neural architectures with a simple average pooling over the

NDCG@k
NDCG@k on E-commerce sequences in Table 4, Firstly, we observe that removing the neural architectures will decrease the performance, especially on the Amazon Books dataset and company dataset V1/V2, which confirms the importance of the neural architectures. Next, we note that, even without the neural architectures, the pure hypercuboid architecture still get higher performance than the second best baselines (on Books, E-commerce, and Company Dataset V1), which signifies that the hypercuboid is essential for the improvement.

Sensitivity of Hyper-parameters
We conduct hyper-parameters analysis on three public datasets.  Table 4: Ablation analysis of the neural architectures on three datasets. Δ Defaut and Δ 2ndbest are the relative improvement(%) over the default version (in Table 2) and the second best baseline, respectively. Here, we analyze the impact of the embedding size in Figure 6 (a), (b), and (c). It is clear that increasing the embedding size improves the performance. Unlike models such as Caser, SASRec, and HGN (checking the embedding size analysis in the corresponding papers.), our model is less likely to be over-fitting with larger dimension size. It suggests that setting a dimension size no less than 100 to be a reasonable choice for our model.

Sequence length.
By varying the sequence length , we get Figure 6 (1), (2), and (3). We observe that the model gets better performance with shorter sequence length on CDs and Movies&TVs, while it is more stable on the Books datasets. Overall, it is more preferable to set to a value around 5 as longer sequence might introduce too many irrelevant activities in the sequences. Table 5 shows the results with different pooling functions. We find that all the three choices (sum, min, and max pooling) lead to deprovements over the mean pooling on the three datasets. In addition, there is no clear winner between the three pooling functions.

Visualization of User Hypercuboids and Item Embeddings
In order to check what the proposed model is learning, we visualize the user hypercuboids on the Amazon books dataset. We project 3,000 items and 3,000 user centers into a 3-dimensional space with T-SNE [21] (Perplexity 9, learning rate 10). Then, we randomly choose eight users and, for each of the eight users, we sample 100 points in the hypercuboid uniformly and project them into the same space. Figure 7 shows the learned hypercuboids. Clearly, our model successfully learned a hypercuboid with different sizes and  We randomly show the hypercuboids of eight users. Blue points represent items and red points represent the user centers and the orange points are randomly sampled points in the user hypercuboids. We manually circle each hypercuboid with an irregular polygon.
positions for each user. Items may be located inside or outside the user hypercuboid. Items which are inside the user hypercuboid are more likely to be recommended to this user. Intersections between users' hypercuboids can also be observed. Moreover, our model can also learn effective item representations. To demonstrate this, we project the item embeddings we learned from for the company dataset (V1) into 2-D space. As shown in Figure 8, we observe that items from different categories (e.g., snack, clothes, toy, furniture, etc. ) are roughly separated. Related categories tend to be closer to each other (e.g., bedding and furniture).

Case Study with Multiple Hypercuboids
To demonstrate the efficacy of using multiple hypercuboids, we show the recommendations list for a randomly selected user in Figure 9. In this case, we learn three independent hypercuboids for this user. Several observations can be made: (1). The recalled items are strongly correlated with the corresponding user sequence; (2) Different hypercuboid can capture different aspect of the interests in the historical sequences, i.e, hypercuboid one recalls T-shirts, hypercuboid two recalls sport footwear; (3). A single hypercuboid can also recall a diverse set of items from closely related categories.  For example, hypercuboid three recalls both socks and basketballs. Clearly, our model is suitable for capturing user's diverse interests.

RELATED WORK
Recommender systems have been widely studied in the literature [1,16,17,27,30,32,40]. Conventional recommendation methods usually represent users and items as points/vectors in a latent space. The interactions are modelled with vectors similarity [13,17,49] or points closeness [12,39]. In the recent years, we can easily observe the increasing numbers of neural network models for recommendation [46,48]. A number of works [2,9,43,44] proposed to model the interactions with multilayered perceptrons. There is also an emerging line of works focusing on capturing the sequence patterns in user activities to enrich user representations [4,24,33]. Hidasi et al. [10,11] proposed a recurrent neural network based model for session-based recommendation. Convolutional neural networks based sequence modeling approaches show promising results on sequential recommendation [34,45]. Self-attention mechanism [14,15,50] can also utilized to enhance the performance by emphasizing important activities in user behaviors. The recent model, hierarchical gating networks [20], moving beyond RNN, CNN, and self-attention, is a gating network that consists of a feature gating module, an instance gating module and an item-item product module. Different from previous works, our model takes the advantages of hypercuboid representations to overcome the limitations of traditional user representation methods. A customized sequence modeling block is also designed to boost the effectiveness of the hypercuboid by incorporating information from user sequential behaviors.
Hypercuboid is an extension of cuboid in higher dimensions. It has attracted attention in the field of machine learning recently [22,25,41,47]. Maji et al. [22] proposed a rough hypercuboid method for relevant and significant features selection. Vilnis et al. [38] proposed a probabilistic embedding method with hypercuboid lattice Measures for knowledge graphs. In the followed-up work, Li et al. [18] improved this model to enhance the robustness in the disjoint case while keeping the desirable properties. Ren et al. [25] proposed a hypercuboid representation approaches for reasoning over knowledge graphs. Our work is also related to Venn Diagrams [37] in the sense that we can treat users as a set of entities they preferred.

CONCLUSION
In this paper, we propose a novel recommendation method with hypercuboids. In our model, users are represented as hypercuboids and the relationship between users and items are modeled with the distance between points and hypercuboids. It demonstrates the state-of-the-art performance on four publicly available and two commercial datasets for personalized recommendation. To capture the diversity of user interests, we proposed two variants of hypercuboids which could further improve recalling performance. We also studied the impacts of effects of hyperparameters and provided case analysis on real-world dataset.