{'fraction_non_alphanumeric': 0.040814858766204574, 'fraction_numerical': 0.008810255857022133, 'mean_word_length': 4.768903640701172, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting Human Mobility via Self-supervised Disentanglement Learning

Deep neural networks have recently achieved considerable improvements in learning human behavioral patterns and individual preferences from massive spatial-temporal trajectories data. However, most of the existing research concentrates on fusing different semantics underlying sequential trajectories for mobility pattern learning which, in turn, yields a narrow perspective on comprehending human intrinsic motions. In addition, the inherent sparsity and under-explored heterogeneous collaborative items pertaining to human check-ins hinder the potential exploitation of human diverse periodic regularities as well as common interests. Motivated by recent advances in disentanglement learning, in this study we propose a novel disentangled solution called SSDL for tackling the next POI prediction problem. SSDL primarily seeks to disentangle the potential time-invariant and time-varying factors into different latent spaces from massive trajectories data, providing an interpretable view to understand the intricate semantics underlying human diverse mobility representations. To address the data sparsity issue, we present two realistic trajectory augmentation approaches to enhance the understanding of both the human intrinsic periodicity and constantly-changing intents. In addition, we devise a POI-centric graph structure to explore heterogeneous collaborative signals underlying historical check-ins. Extensive experiments conducted on four real-world datasets demonstrate that our proposed SSDL significantly outperforms the state-of-the-art approaches -- for example, it yields up to 8.57% improvements on ACC@1.


INTRODUCTION
The proliferation of geo-tagged social media (GTSM) such as Foursquare and WeChat have enabled numerous users to post interesting places, report daily activities, and make like-minded friends, resulting in the accumulation of massive amounts of contextual data (e.g., check-ins). This, in turn, offers unprecedented opportunities to explore human diverse life experiences (e.g., mobility patterns) and facilitate the development of various user-centric downstream applications such as trajectory identification [1], POI recommendation/prediction [2], itinerary prediction [3] -to name a few. As a fundamental task in mining check-in data, predicting human mobility (often exemplified as next POI prediction/recommendation) is critical for researchers and practitioners to explore the informative semantics and mutual interactions behind human check-ins [4], [5]. For instance, it enables one to precisely ascertain users' future intentions and draw in more potential customers for new ventures [6], [7].
Spatio-temporal check-in sequences (i.e., trajectories) reflect human daily activities upon a set of POIs, which may include certain (periodic) regularities. The majority of the pioneering works in human mobility prediction aimed at Fan Zhou is the corresponding author.
• Qiang Gao  modeling human sequential behaviors taking into account spatio-temporal preferences. For instance, in order to predict where a certain user will go in the near future, conventional approaches such as Markov Chain [8] and Tensor-based Factorization [9] that rely on data-driven paradigms, attempt to incorporate individual visiting preferences and exploresequential patterns. However, these approaches depend heavily on hand-crafted characteristics and face the challenge of comprehending the diverse semantics underlying massive volumes of human trajectories. This, in turn, leads to narrow solutions in disclosing human implicit interactive hints/signals regarding historical check-ins. More recent deep learning techniques such as recurrent neural networks (RNNs) have brought about encouraging achievements of learning informative check-ins (including POIs) from human trajectories and become a widespread and popular methodology in tackling miscellaneous mobility learning tasks [4], [10], [11]. For example, Wu et al. [10] present a PLSPL model, which leverages a Long-Short Term Memory (LSTM) neural network to model human shortterm sequential preferences while learning contextual features of POIs behind human historical check-ins via attention mechanism. To consider the spatial and temporal influences for next POI recommendation, Kong et al. [12] incorporate the spatial and temporal intervals between two successive check-ins into recurrent hidden states to mitigate the data sparsity of human trajectories. Due to the higher model efficiency and the ability to quantify the contribution of each check-in in a given trajectory, several attention mechanisms like, for example, self-attention and vanilla attention emerged for handling long human historical trajectories [2], [6], [13].
Several state-of-the-art methods have employed graph arXiv:2211.09625v1 [cs.
LG] 17 Nov 2022 structure learning to explicitly uncover spatial correlations or collaborative signals to understand individual human interests. More concretely, they attempt to acquire expressive POI representations by considering the rich contexts of highly correlated POIs. For instance, conventional methods such as word2vec-based [14], [15] and deepwalk-based [16], [17] have successfully uncovered the higher-order correlations between consecutive check-ins and offered contextual POI representations. Other schemes using popular graph neural networks (GNNs), such as graph convolutional networks [11] and graph attention networks [18], primarily seek to incorporate the POI-to-POI correlations (e.g., geographical proximity) behind massive human trajectories. Despite the recent achievements in deep human mobility learning, we observe that existing solutions still have three significant drawbacks: (a) Implicit semantic entanglement. Although there is a large body of work on human mobility representation learning, the most common scheme is to take a past check-in sequence as input, and either that sequence or the user's next POI is used as the supervision signal. The former can be framed as self-supervised learning, while the latter is standard supervised learning. Nevertheless, both ultimately focus on fusing multiple semantics behind sequential trajectories to predict the user's next POI, which could lead to a myopic perspective and produce a non-diverse recommendation result. We call this phenomenon semantic entanglement. In practice, human trajectories, as typical sequential data, contain rich user mobility patterns that reflect diverse periodic regularities or behavioral habits of humans. More importantly, the intrinsic individual patterns/habits of humans are difficult to change over time, but their near-term intentions/behaviors are prone to be influenced/dictated by certain time instants. Thus, we consider that human mobility patterns can be implicitly disentangled into two aspects: time-independent and time-dependent behaviors. Existing solutions rely on data-driven models to understand limited mobility patterns, which fail to reveal the nature of human visiting intents. As a result, they only provide a narrow scope to become familiar with human future behaviors, which usually carries the risk of prediction bias due to the limited scale of trajectory data available.
(b) Sparsity in representation learning. Only when a user decides or is willing to check in via location-based applications can a POI be recorded, which inevitably leads to the sparsity problem when gathering historical human footprints. As a result, the sparsity problem hinders the model from learning a good representation of human mobility. Existing methods either use the next POI as the sole supervisor or complement the representation learning in a semi-supervised manner with unlabeled trajectories. These paradigms mostly follow the merit of text representation in the field of natural language processing (NLP) which, however, easily fails in capturing the innate rules underlying human trajectories such as individual periodic regularity.
(c) Heterogeneous collaborative signals. Most existing efforts concentrate on learning POI-to-POI relationships (a.k.a. the connectivity of POIs) from a large number of trajectories, such as consecutive correlation and geographic proximity [17], [18], [19]. Despite the successful collaboration of individual human interests via these homogeneous graph structure learning, a notable limitation is that heterogeneous semantics affiliated with the POIs are not investigated well, yielding a limit in the exploration of affluent common preferences behind human diverse trajectories. For example, people may have similar visit time preferences for certain POIs, such as going to a café after lunch. In addition, each POI is associated with a textual description (e.g., POI category), reflecting the underlying human activity interest. We conjecture that incorporating the heterogeneous correlations between POI and their category can provide us with a coarse-grained view of the higher-order connective between POIs. For example, people often go to several fashion stores to buy clothes at a time.
To address the aforementioned limitations, we present a novel solution called SSDL, a self-supervised disentanglement learning framework for understanding human mobility. Rather than previous data-driven representation learning, SSDL performing self-supervision in the latent space aims at seeking a clean separation of the time-independent and time-dependent vectors for diverse human trajectories, which is inspired by the recent advances of variational inference and contrastive learning. Specifically, SSDL operates the sequential variational autoencoder (VAE) with a mutual information regularization to guide the training of evidence lower bound (ELBO), aiming at promoting the disentanglement of human mobility-related representations. In particular, we provide two realistic trajectory augmentation strategies to alleviate the sparsity issue in representation learning, which can further help us enhance the understanding of human intrinsic periodicity and constantly-changing intents. In addition, we also present a POI-centric graph structure to explore human common interests underlying diverse check-ins, which primarily seeks human consecutive, geospatial, temporal-aspect, and activity-aspect interests. In sum, our contributions can be summarized as follows: • We introduce a novel disentangled representation learning framework to understand human timeindependent and time-dependent behaviors of their individual mobility patterns. To the best of our knowledge, this study is the first work to disentangle human mobility and investigate how it can be used for the prediction of the next POI. • We propose two practical trajectory augmentation methods, guided by the inherent characteristics of individual human mobility patterns, to promote disentanglement learning. • To capture heterogeneous collaborative signals behind historical check-ins, We devise a flexible POI-centric network structure to explore rich human interests in trajectories, which enhances the performance of downstream next POI prediction task. • We conduct extensive experiments on four real-world datasets to evaluate the performance of our proposed SSDL. The results demonstrate that our approach outperforms state-of-the-art methods.

Next POI Prediction in Deep Learning
Recent deep learning solutions have stimulated many researchers and practitioners to learn human periodic regu-larities from massive historical check-ins. Especially, deep (recurrent) neural networks such as LSTM [20] and GRU [21] have received widespread interest in the next POI prediction task as they are able to capture the sequential dependencies for mobility pattern understanding. For instance, [22] extends the vanilla RNN model and integrates the spatialtemporal impacts into each RNN cell, yielding promising results on the next location prediction. Zhao et al. propose a novel ST-LSTM that implements time gates and distance gates into standard LSTM, aiming at capturing the spatiotemporal relation between consecutive check-ins [4]. To learn more contextual information, Wu et al. [10] propose a personalized long-and short-term preference learning scheme to learn the specific user context, where the different influences of locations and categories of POIs are considered. While most of endeavors focus on pruning or modifying the RNN-based modules [23], [24], [25], researches also tried to adopt other popular deep neural networks for next POI prediction, e.g., attention-based neural networks [2], [26] and convolutional neural networks [6], [27]. Xue et al. [2] build the Transformer architecture as the mobility feature extractor in which it regards the historical trajectory and semantic contexts as the input to handle multiple factors such as temporal and geographic contexts.

Mobility Representation Learning
POI embedding and trajectory embedding, as two core components in mobility representation learning, have been investigated in recent studies. For POI embeddings, the earlier studies such as [22] and [28] set a fixed or learnable matrix as the initial representations of POIs, primarily seeking to alleviate the "Curse of Dimensionality" concern. However, any semantic information between POIs is under-explored. As word embeddings, especially word2vec-based [29], have achieved great performance in NLP, recent studies also proposed various word2vec-based solutions aimed at capturing the proximity semantics of POIs from human check-in sequences (or real-world trajectories). For instance, [30] and [31] regard each POI as a "word" while each human trajectory as a "sequence", and use word2vec to obtain a low-dimensional vector for each POI. POI2Vec is a latent representation model that incorporates geographic influence when using word2vec method for POI embedding [14]. However, training sparse trajectories to obtain POI representations often confronts the problem of poor capability of POI semantics. More recently, the extraordinary success of graph neural networks (GNNs) has inspired tremendous researchers to turn to devise graph-based models to facilitate the learning of human trajectories [11], [18]. For instance, [18] proposes a graph-based model to explore the spatial, temporal, and preference factors behind the POIs. However, it only considers homogeneous interactions among the POIs and ignores heterogeneous interactions with other key entities such as activity and check-in time.
Regarding trajectory representation learning, the majority of existing research concentrates on taking the historical trajectory as input and using the next POI as the sole supervision signal [4], [7], [22], [32]. To address the narrow scale of trajectory data, some efforts attempt to employ the unlabeled trajectories as supplements and train them with the labeled trajectories jointly in an unsupervised or selfsupervised manner to acquire a good representation for each trajectory [6], [33], [34]. Especially, to operate the trajectories in a latent space, recent studies employ generative models such as variational inference or adversarial models to learn the intrinsic distribution underlying massive trajectory data and then turn to fine-tune the model for the next POI prediction tasks. For instance, VANext extended the variational autoencoder (VAE) to consider the uncertainty of user preferences for regularized representation of historical trajectories [6]. A meta-learning technique called METAODE also employed variational Bayes to encode past human movement patterns into latent space [35]. In essence, these approaches principally rely on integrating numerous semantics including sequential information into a unified space while omitting the possibility of disentangling it to expose the characteristics of human mobility patterns.

Disentanglement Learning
The privilege of disentanglement learning is that it enables an interpretable perspective to understand the multiple inherent motions/factors behind the intricate data representations in addition to notable expressiveness. To disentangle the learned representations, most recent studies developed VAEs such as β-VAE to optimize the mutual interaction between different latent factors [36], [37], [38]. For example, β-VAE [36] is a simple but effective variant of the ordinary VAE that severely penalizes the Kullback-Leibler (KL) divergence term for disentanglement learning. Li et al. presented a Disentangled Sequential Autoencoder (DSVAE) approach for sequential data (e.g., video), aiming at factorizing the latent variables into static and dynamic parts [39]. To make the latent variables interpretable and controllable, a latent variable guidance-based generative model called Guided-VAE makes an effort to utilize VAE to learn a transparent representation [40]. Bai et al. presents a sequential VAE to learn disentangled representations in a self-supervised manner [41]. Bai et al. also extend the sequential VAE with a self-supervised learning approach to facilitate the factorization of video representations [38]. In addition, The newly developed self-supervised learning offers a new avenue to drive the acquisition of semantic representations [42]. For example, Ma et al. employ the ideas of latent self-supervision and intention disentanglement to boost the convergence of representation learning and utilize it in sequential recommendation tasks [43]. In sum, the success of these approaches suggests that, in addition to facilitating the understanding of rich semantics underlying data, disentangling the representation into distinct parts can make the representation more transparent and interpretable.

Problem Definition
Definition 1 (POI). Let l ∈ L denotes a POI tagged by the location-based systems, and each POI corresponds to a geographic coordinate (e.g., longitude lo and latitude la) and a category ca (e.g., restaurant, museum, or park).
sequence of n POIs ordered by visiting time, where l u τ means a user u visit POI l at time t τ (τ ∈ {1, 2, · · · , n}). Let T u = {T 1 u , T 2 u , · · · , T m u } denote m historical trajectories of user u, where each trajectory T i u contains a sequence of POIs ordered by visiting time, e.g., , · · · , l m,u n } and entire historical trajectory T u , our goal is to predict a POI l m,u n+1 for user u to visit next. Notably, we mainly target disentangled representation learning for users' recently visited POI sequences. For simplicity, we will omit user identity (i.e. u) and trajectory index (i.e. m) in the following sections.

Variational Bayes
Variational Autoencoder (VAE) [44] containing an encoder and a decoder operates the input data x into a latent space, where the latent variables are denoted by z. Thus, the marginal likelihood log p(x) can be obtained by maximizing the Evidence Lower BOund (ELBO), which is defined as: Herein, q φ (z|x) is an approximate posterior distribution, parameterized by φ, p θ (x|z) with parameters θ is a likelihood function, and p(z) is a prior (e.g., Gaussian prior) over the latent variables.

Contrastive Estimation
In recent self-supervised learning paradigms [33], [45], mutual information (MI) is a common measure of the mutual dependence or compatibility between two variables. Specifically, they usually employ the noise contrastive estimation (NCE) [46], [47] to maximize the lower bound on the mutual information, which can be denoted as follows: , where x, x + , and x − respectively denote the anchor, positive, and negative instances. Besides, exp g(·) g(·) is a similarity measure (e.g., cosine similarity) between two instances.

ARCHITECTURE DESIGN
We make an overview of our proposed framework SSDL in Fig. 1, which mainly comprises three components. First, we build a POI-centric Graph (PGraph) to explore the common interests from the entire user trajectories and make interest aggregation to obtain both homogeneous and heterogeneous semantics underlying each POI. Then, our Selfsupervised Disentanglement Learning component attempts to produce the time-invariant and time-varying variables for each trajectory. At last, SSDL uses the disentangled representations as well as the user's long-term preference modeled by an attentive network to predict the next POI.

Common Interest Distillation
To distill multiple correlations behind the POIs and their affiliated context, we build a POI-centric graph (PGraph).

Graph Structure and Building Process
Incorporating prior correlations and multiple common interests are critical to obtain a good POI representation and understand human diverse mobility patterns. As several elements are recorded by LBSN, e.g., POI identity, geographical coordinate, visiting time, and POI category, we concentrate on exploring four contextual semantics to build our PGraph, including consecutive, geospatial, time-aspect, and activity-aspect interests. Let G = (V, E) denotes our PGraph that models the human common interests, where V = (V l ∪ V t ∪ V a ) is the set of nodes, and E = (E c ∪E g ∪E t ∪E a ) is the set of edges. Here V l = L represents a collection of different POIs, V t is the set of time bins, V a denotes the set of POI categories, and E c , E g , E t , E a indicate the above four contextual semantics, respectively. That is to say, G contains four sub-graphs, each of which represents an important user interest. Four subgraphs are described in the following four paragraphs. Consecutive Interest. According to [31], among millions of POIs in location-based systems, (1) people typically visit only a small subset of POIs that appeal to them; and (2) some POIs are visited more frequently than others. This phenomenon demonstrates that human mobility contains some common transitional regularities behind their past check-ins. Therefore, we consider that it is necessary to capture the consecutive correlations between distinct POIs to reveal human motion-based interests. Correspondingly, we formulate a weighted sub-graph G c = (V l , E c , A c ) to describe such diverse correlations, where V l is the set of distinct POIs, E c is the edge set, and A c ∈ R |L|×|L| refers to the adjacency matrix. Given two POIs (e.g., POI l i and POI l j ) that are successively visited, we create an edge between them and then calculate the edge weight (i.e., entry A ij c ∈ A c ) using the corresponding transitional probability. Formally, such an edge weight can be defined as: where f ij c refers to the frequency of edge l i → l j appeared in the check-in data, and f i c denotes the frequency of POI l i appeared in the check-in data. As such, we are able to acquire the matrix A c to preserve the consecutive interests underlying the trajectories. Geographical Interest. People are more likely to visit nearby POIs than distant ones [17]. Motivated by this, we formulate an undirected sub-graph G g = (V l , E g , A g ) to describe such interactions, where E g is the set of edges and A g (∈ R |L|×|L| ) denotes the adjacency matrix regarding geographical interests. Given POI l i and l j , the edge weight A ij g (∈ A g ) can be calculated as: Herein, g(l i , l j ) is the great-circle distance function, ∆g is a predefined threshold to restrict the impact of geographical noise. In this paper, we set ∆g = 3 km. Time-aspect Interest. For each check-in, it is associated with a visiting timestamp, reflecting the human temporal semantics. As it is a key factor for understanding human periodic regularity, we propose to investigate the mutual interactions between POI and visiting time to obtain the time-aspect interest. However, each visiting timestamp is actually a continuous value, we follow previous studies and aggregate all of the visiting timestamps into the hour-level time bins [48], [49]. Meanwhile, people may respectively show different preferences on weekday and weekend, we thus assign 48 time bins to replace the original visiting timestamps, where the weekday and weekend are specified. We thus formulate a weighted sub- where A t maintains human time-aspect interest. Similar to the above graph G c , we can also calculate the time-aspect interest between the POI l i and time bin t τ by: where f iτ t denotes the frequency of visiting POI l i at time t τ , and f i t is the total number that POI l i has been visited. Activity-aspect Interest. A user who wants to post a checkin to LBSNs indicates that he/she is engaged in a specific type of activity that appeals to him/her. In practice, each POI has a contextual description (i.e., POI category) that reflects a real-world activity, we consider that taking into account such contextual interactions is an essential addition to understanding human preferences. Notably, the number of POI categories is much smaller than the number of POIs. As a result, linking a POI to its category can offer a coarse-grained perspective on the higher-order interactions between various POIs. To this end, we build an undirected graph G a (V l ∪ V a , E t , A a ) to describe the activity-aspect interest. To be more precise, we explicitly build an edge between a POI and the contextual category it belongs to, and then we treat each category as a regular node in G a .

Interest Aggregation
To extract the semnatic contexts underlying POIs from the PGraph, we propose to adopt graph neural networks (GNNs) which have been widely applied in numerous graph-based tasks and obtained remarkable success. Homogeneous Semantic Aggregation (HoSA). According to the structure of the built PGraph, we can find that consecutive interest and geospatial interest that belong to the homogeneous semantics as they only contain the nodes of POI identities. Thus, HoSA attempts to aggregate the underlying information from the nodes of the same type, i.e., POI identity. First, the consecutive correlation matrix A c reflects human real-world transitional preferences, we can naturally regard each POI's transitional distribution as its prior feature to describe the relationship between a specific POI and its neighbors. To this end, we set each A i c as the initial feature of the POI node V i p . Besides, the geospatial correlation matrix A g preserves the geographical closeness between different POIs, providing the weak signal of human potential transitional tendencies. Hence, A g can be regard as an augmentation of the consecutive correlation matrix A c . Therefore, we merge these two matrices into a unified matrix A h to reveal observed and unobserved preferences of transitional dependencies. Specifically, given two distinct POI nodes V i l and V j l , its correlation score A ij h is defined as: For any POI node V i l , we embed each POI node to a unified representation: where W l ∈ R |L|×d and b l ∈ R d are trainable matrices. The dimension of s l i is d. Afterwards, each POI has its unique initial representation. To bridge the correlation between POI V i l and each of its neighbor V j l ∈ Ω(V i l ), we devise a scoring function to evaluate the different contributions of neighboring nodes. For instance, given POI node V i l and its neighbor V j l , we define contribution measure as: where ⊕ is the concatenation operation and b a ∈ R 2d is a learnable vector. Then, we follow the standard GAT [50] and use softmax function to normalize the attention scores across all neighbors of POI V i l , where each attention score regarding its neighbour V j l can be formulated as: In the end, we obtain the aggregated representation e l i of POI node V i l by a sum operation: where σ is the sigmoid activation function and W e ∈ R d×d is a set of trainable parameters. Heterogeneous Semantic Aggregation (HeSA). HeSA is to aggregate the associated information of POIs from the neighboring nodes with different types. In our PGraph, there are two correlations that describe the heterogeneous semantics between different types of nodes, i.e., the timeaspect and activity-aspect interests. In contrast to HoSA, we do not involve the attention mechanism to quantify the different contributions of POI's heterogeneous neighbors.
The reason is that the number of them are extremely smaller than that of the POIs, we thus attempt to capture all of the possible heterogeneous neighbors of a given POI directly to enhance the semantic information.
(1) For time-aspect interest, each POI p i is associated with a probability distribution A i t (∈ A t ) that describes the preference strengths between POI and time bins. We leverage the message-passing neural network inspired by [11] to incorporate the time-aspect preference of each POI, which can be formulated as: where tanh is the activation function and W t is a trainable matrix. Finally, we can obtain each POI's temporal context.
(2) For activity-aspect interest, we obtain each POI's activityaware semantic by: where W a is a trainable matrix. Finally, the homogeneous and heterogeneous semantics behind each POI are acquired by HoSA and HeSA, respectively. In the following mobility encoding procedures, we will use these contextual representations as the embeddings of POIs in user trajectories. And these embeddings can be jointly optimized during selfsupervised learning and task learning.

Context-aware Mobility Encoding
Existing studies usually choose the recurrent neural networks such as Long-short Term Memory (LSTM) or Gated Recurrent Unit (GRU) to capture human transitional regularities. Since the complex stacked gate operations in LSTM typically struggle with the gradient vanishing problem, we select GRU as the kernel of our mobility encoder. For each l τ in a given trajectory T = {l 1 , l 2 , · · · , l n }, we have collected the homogeneous and heterogeneous semantics behind it. In this way, they can be viewed as reflections of different interest in different domains. Therefore, we extend the GRU cell to capture the sequential information as well as the contextual information behind each POI. Correspondingly, the recursive process with GRU can be formulated as follows: where h τ and h τ −1 are the hidden states of the current POI l τ and the last visited POI l τ −1 , respectively. Herein, c τ is the contextual embedding of POI l τ , which is a unified representation that integrates the homogeneous and heterogeneous semantics of POI l τ (they include e l τ , e t τ , and e a τ ). In addition, W f and b f are trainable parameters.

Self-supervised Disentanglement Learning
Now we describe in detail the self-supervised disentanglement learning in SSDL.

Disentanglement via Variational Inference
Given any recent trajectory T = {l 1 , l 2 , · · · , l n }, we attempt to learn a set of time-varying variables z r 1:n = {z r 1 , z r 2 , · · · , z r n } and a time-invariant variable z s , where z r 1:n aims at exploring the dynamics of human time-dependent interests while z s undertakes the role of learning human inherent time-independent periodicity (habits). Formally, let z τ be the entangled latent code of check-in l τ , and we have z τ = (z r τ , z s ). For consistency, let l 1:n denote the check-in sequence {l 1 , l 2 , · · · , l n }. As people's future movements are affected by their previous check-in behaviors, we assume that each z τ depends on its previous states z <τ = {z 1 , z 2 , · · · , z τ −1 }. In addition, as user's longstanding interests will not be changed dramatically by recent activities, we assume that z r 1:n and z s are independent from each other, i.e., p(z 1:n ) = p(z r 1:n )p(z s ). Hence, we formulate our probabilistic generative model as follows: Prior: p(l 1:n , z 1:n ) = p(z 1:n )p(l 1:n |z 1:n ) where p(z 1:n ) is a prior. Herein, we choose the Gaussian distribution as the prior p(z s ), i.e., p(z s ) ∼ N (0, 1). We follow the rule of standard variational Bayes and set N (µ(z <τ ), σ 2 (z <τ )) as p(z r τ |z r <τ ), where µ(·) and σ 2 (·) can be modeled by popular recursive networks. In practice, we also use GRU to obtain z r τ as follows: where ∼ N (0, 1) and is element-wise multiplication. Subsequently, we expect to produce a posterior distribution q(z 1:n |l 1:n ) to cater to the learning manner of variational inference. Thus, we define the posterior as follows: Posterior: q(z 1:n |l 1:n ) = q (z s , z r 1:n | l 1:n ) (17) = q(z r 1:n |l 1:n )q(z s |l 1:n ) = q (z s | l 1:n ) n τ =1 q (z r τ | z r <τ , l ≤τ ) .
We note that the above process is also operated in an auto-regressive manner. We use another GRU cell that has the same architecture as the Mobility Encoding network to generate posterior distributions. At last, we obtain the Evidence Lower BOund (ELBO) as follows: ELBO: max p,q E l1:n∼p D E q(z1:n|l1:n) [log p (l 1:n | z 1:n ) where p D is the empirical trajectory distribution. As z 1:n is comprised of mutually independent z s and z r 1:n , the second term of KL-divergence can be disentangled as: KL [q (z 1:n | l 1:n ) p(z 1:n )] = (19) KL [q (z s | l 1:n ) p(z s )] + KL [q (z r 1:n | l 1:n ) ||p (z r 1:n )] . Following the principle of VAE [38], [44], we present a theoretical proof to illustrate above modeling processes. Proof 1.
log p (l 1:n ) ≥ − KL [q (z 1:n | l 1:n ) p (z 1:n | l 1:n )] + log p (l 1:n ) = − KL [q (z s , z r 1:n | l 1:n ) p (z s , z r 1:n | l 1:n )] + log p (l 1:n ) =E q(z s ,z r 1:n |l1:n) [log p (l 1:n | z s , z r 1:n ) − log q (z s , z r 1:n | l 1:n ) + log p (l 1:n )] =E q(z s ,z1:n|l1:n) [log p (l 1:n | z s , z r 1:n ) − log q (z s , z r 1:n | l 1:n ) + log p (z s , z r 1:n )] =E q(z s ,z r 1:n |l1:n) [log p (l 1:n | z s , z r 1:n ) − log q (z s | l 1:n ) − log p (z r 1:n | l 1:n ) + log p(z s ) + log p (z r 1:n )] =E q(z s ,z r 1:n |l1:n) [log p (l 1:n | z s , z r 1: Recall that the results of the above proof are similar to the results in standard VAE, which usually confront the agnostic prior distribution that causes posterior collapse problem and leaves the learned latent space still entangled [51]. This phenomenon has been revealed in recent studies, e.g., β-VAE [36] and β-TCVAE [37]. Additionally, [52] provides us with a clearer perspective that reveals the challenges with disentangled representation in variational inference. Thus, we conjecture that the last two terms regularized by KL-divergence in Eq. (20) are hard to close to their corresponding prior, which would make each posterior become non-informative. For the purpose of receiving clean disentanglement of z s and z r 1:n , we are inspired by recent selfsupervised learning and enforce disentanglement mobility learning from the perspective of mutual information.

Mutual Information Regularization
We now turn to detail on how to combine contrastive learning with disentangled mobility learning. We first introduce variational mobility learning from the perspective of Mutual Information (MI). The goal of MI is a measure of the mutual dependence between two variables. Since both z s and z r 1:n are derived from the original trajectory, we thus add three additional MI terms to regularize the latent space of them, which can be defined as follows: − α(KL [q (z s | l 1:n ) p(z s )] + KL [q (z r 1:n | l 1:n ) ||p (z r 1:n )]) + β(M I (z s ; l 1:n ) + M I (z r 1:n ; l 1:n )) − γM I (z r 1:n ; z s ) , where α, β and γ are weight coefficients. M I(·, ·) refers to MI term. For instance, M I (z s ; l 1:n ) is defined as: We note that other MI terms have the similar formulation. Now our goal become enforcing the posteriors matching with their corresponding priors while ensuring that z s and z r 1:n are disentangled from each other. Note that the complete proof of Eq.(21) is provided in Appendix part. To estimate the MI terms, we follow most of recent studies [33], [45], [46] and employ the NCE loss to make contrastive estimation. For instance, a contrastive estimation of M I(z s ; l 1:n ) can be defined as follows, where ψ(·, ·) = exp(sim(·, ·)/η), sim(·, ·) denotes the cosine similarity function, m is the number of negative trajectories, and η = 0.5 is a temperature parameter. Notably, we treat l 1:n as the positive trajectory sequence regarding z s and specify it using l + 1:n . Besides,l j 1:n refers to a negative sample (trajectory), which is generated from other users.
Augmentation for time-invariant factor. However, due to the limited scale of positive samples, we try to generate more realistic trajectories to augment the original samples. As z s reveals human intrinsic periodicity and should not be affected by recent moving behaviors, i.e., time-independent, we can thus randomly change the order of a given trajectory l 1:n and formulate several augmentation versions w.r.t l 1:n . We claim that it is a simple but efficient strategy to obtain rich augmented samples. Correspondingly, the contrastive estimation regarding these augmented samples can be denoted as follows: where l # 1:n indicates it is an augmentation version of l 1:n . For time-invariant factor z s , we can use the collected samples including augmented samples to make a final estimates as follows: Augmentation for time-varying factor. As for z r 1:n is a set of latent variables regarding l 1:n , showing human human time-dependent interests. Similar to Eq. (23), we can obtain the contrastive estimation of M I(z r 1:n ; l 1:n ) as C z r 1:n . Furthermore, we provide another data augmentation method to enhance the optimization of M I(z r 1:n ; l 1:n ). The intuition is that z r 1:n is a set of time-dependent variables. In practice, real-world check-in data could be subject to noise and uncertainty due to the presence of collective POIs [53]. Hence, a user usually posts an fuzzy POI to replace her accurate position, which could weaken human mobility pattern learning and even result in inaccurate predictions. Motivated by [53], [54], it is encouraging that we can use any member of related collective POI to replace the original POI in a trajectory to obtain an augmentation trajectory for time-varying factor training, which will not change any temporal semantics. In addition, another potential benefit of such a practice is to alleviate the uncertainty issue behind diverse human checkin behaviors. In our implementation, we use neighbors of the same category within 300m of a given POI as members of its collective POI and replace about 30% of the POIs in a given trajectory with their related collective POIs, which does not heavily affect the full semantics behind the original trajectory. As a result, we can obtain a large number of synthetic trajectories that provide multiple views of a given trajectory. Similar to Eq.(25), we can get the final estimation regarding z r 1:n as follows: M I (z r 1:n ; l 1:n ) ≈ where C * z r 1:n is the contrastive estimation of the augmented trajectories regarding time-varying factors. As for the final term M I(z r 1:n ; z s ), the variables in it are all in the latent space, we thus can directly choose the standard mini-batch weighted sampling (MWS) [37] for comparative estimation.

Task Learning
So far, we have obtained a set of time-varying variables and a time-invariant variable for each trajectory. We turn to use our task learning network to predict the next POI. For each user, we actually own her entire historical trajectory. Inspired by [6], [55] modeling such a long trajectory would boost the capture of human long-term transitional preferences. It is natural to adopt the RNN to encode the transitional regularity underlying human historical trajectory. But in practice, there are massive time-ordered POIs in their historical trajectories, which usually result in a serious time cost problem. Therefore, we employ a self-attention layer with position encoding to capture the taste of the transitional behavior of a user as well as the long-distance dependencies. Given a user's entire historical trajectory T 1:K containing K ordered POIs. We first reuse the linear layer (cf. Eq.13) to obtain the dense representation of each POI in T 1:K . Correspondingly, we use T 1:K to denote the trajectory with embedded POIs. To determine the order of POIs in T 1:K , we follow [56] and use the sine/cosine function-based position embedding to formulate the final representation of each POI, which can be denoted as: where Φ(T i ) is the position embedding of POI l i in T 1:K . Then, we employ one-layer self-attention network to receive a set of hidden states regarding T , as follows: In our study, we use the last state H K to represent T 1:K and regard it as one of the inputs for task prediction. Now we take z r 1:n , z s , and H 1:K as the input and employ a one-layer fully-connected network with softmax function to obtain the predict POI. the process can be expressed as: ) (29) Correspondingly, the loss function for trajectory T can be expressed as: To minimize the above cross-entropy loss, we employ Adam algorithm to optimize the parameters. We outline the complete pipeline of training SSDL in Algorithm 1. Make trajectory augmentation regarding z s ;

11
Make trajectory augmentation regarding z r 1:n ;

EXPERIMENTS
We now conduct experiments to evaluate the performance of our proposed SSDL on four real-world datasets.

Datasets
To facilitate reproducible results, we conduct all experiments on two publicly available LBS applications: Foursquare [57] and Gowalla [58].

Baselines
We compare our SSDL with several representative approaches for next POI prediction task.
• GRU [21] is a common approach for sequential data learning as its superiority in incorporating the semantics of long-term dependencies. • ST-RNN [22] is an RNN-based method that incorporates spatio-temporal contexts when predicting the next POI. • HST-LSTM [12] employs sequence-to-sequence learning scheme to include spatial-temporal influence in LSTM and makes use of contextual information to enhance model performance for sparse data prediction. • Flashback [59] models sparse user mobility footprints by doing flashbacks on hidden states in RNNs. Especially, it explicitly employs the spatio-temporal contexts to search past hidden states with high predictive power. In our experiments, we take the GRU cell as the recurrent component in Flashback for a fair comparison. • DeepMove [55] presents an attention-based RNN to encode human recent trajectories. Furthermore, it employs another RNN to learn user long-term preferences from historical trajectories. • VANext [6] proposes a novel variational attention mechanism to explore human periodic regularities. In addition, it employs a simple convolutional neural network rather than RNN to capture human long-term interests. • PLSPL [10] is a unified framework that jointly learns users' long-and short-term interests for next POI prediction. • MobTCast [2] is a Transformer-based approach that considers multiple semantic contexts behind check-ins to enhance the understanding of human mobility. Note that we remove the Social Context Extractor in MobCast as the social relationships are not available in our context. • β-VAE [36] is a widely used representation learning method that is able to separate latent factors into different space by using an adjustable hyperparameter β to the original VAE objective. In this study, we use GRU as the encoder and decoder network structure in β-VAE to model the temporal semantics. • SML [33] attempts to understand human mobility in a self-supervised learning manner. Especially, it leverages heuristic strategy to enumerate massive different views of original sparse trajectories for contrastive estimation.

Metrics
To evaluate the performance of our proposed SSDL, we follow most of the previous studies [6], [10], [33] and select three commonly used metrics to compare with the baselines. We first use the ACC@K to evaluate the recommendation performance. In this paper, we report the different testing results of K = 1, 5, 10. Additionally, we report area under the ROC curve (AUC) and mean average precision (MAP) metrics that are frequently used in classification tasks.

Implementation Details
We implement our SSDL and baselines in Python. All methods are based on the Torch library and accelerated by one NVIDIA GTX 1080 GPU. Besides, we choose Adam [60] to train all deep learning methods. In disentanglement learning, the learning rate is initialized as 0.01. We set the coefficient α of KL terms to 1. Besides, β and γ are fixed to be 1 and 0.1. In task learning, the learning rate is initialized with 5e-4. The dropout rate is set as 0.5, and the batch size is 32. The hidden size of the self-attention network is set to 300. In addition, we set dimension of z s to 256, while z r 1:n to 32. Table 2 reports the performance of different approaches on the datasets of four cities, where the best achievement is highlighted with bold and the second best is marked with underline. Specifically, we have the following observations.

Performance Comparisons
We can find that ST-RNN does not provide us with competitive achievements compared to GRU although it considers the spatial and temporal constraints. The plausible reason is that the sparsity issue of check-in data heavily affects the distillation of semantics contexts such as geographical distance. Meanwhile, relying on simple spatio-temporal features and regarding the next POI as the solo supervision usually results in an inference bias or uncertainty problem due to the boundary of available training datasets. To mitigate the data sparsity issue, HST-LSTM which combines spatial and temporal factors with a gate mechanism is able to boost the capture of human mobility patterns by a large margin. Furthermore, HST-LSTM models the periodicity of consecutive check-ins in an end-to-end manner, which brings an encouraging prospect for us to learn the complex distribution behind historical trajectories. Compared with HST-LSTM, which directly adds spatio-temporal factors to hidden states, Flashback achieves competitive performance because it explicitly uses a rich spatio-temporal context to search for past hidden states with high predictive power to predict the next POI.
As for DeepMove and VANext, they both attempt to correlate a certain user's recent trajectory and historical trajectory to accurately discover individual periodicity. Our experiments show that they achieve higher gains than models (e.g., ST-RNN) that only consider the past few checkins. Furthermore, VANext, the first variational inference approach to model human trajectories using a prior assumption, outperforms DeepMove due to the relief of the inherent uncertainty of user mobility. The paradigm of PLSPL is similar to DeepMove, but it operates an attention mechanism to evaluate the importance of each POI in a user's historical check-ins, aiming at exploring the tastes of different users. We can find that PLSPL performs better than DeepMove. In addition, MobTCast is a Transformer-based approach that uses self-attention to study the interactive signals between POIs in a given trajectory, as well as multiple semantic contexts, such as category and temporal semantics. We obtain similar performance results compared to PLSPL, indicating that considering multiple semantic contexts does help to discover users' future check-in intentions.
β-VAE is a popular disentanglement learning method that also obtains promising results, which suggests that employing the latent variables produced by variational Bayesian does help in understanding the inherent generative factors underlying human mobility. As for SML, it is the first self-supervised learning solution for the next POI prediction, achieving the best gains on AUC among the baselines. The reason is that it primarily seeks to produce massive synthetic trajectories for data augmentation and leverage contrastive learning to study the diversity of human moving intents behind existing historical check-ins.
In general, our proposed SSDL significantly outperforms the compared approaches by a relatively large margin across the four cities. For instance, SSDL respectively yields 8.57% and 11.94% averaged improvement over the best baseline regarding ACC@1 and MAP. This observation demonstrates the superiority of the self-supervised disentanglement learning paradigm in our SSDL.

Ablation Study
To evaluate the contribution of different components of SSDL, we devise several variants regarding SSDL from two aspects, i.e., POI embedding and trajectory embedding. First, we select 7 popular embedding methods to scrutinize the efficacy of our POI embedding.
• One-Hot [55] is the simplest method that maps each POI to a unique vector without any semantic information. • Random uses a dense matrix sampled from a Gaussian distribution to represent the POIs. • Word2vec is a popular embedding technique in NLP, aiming at exploring the surrounding context of a given word. Also, it has successfully applied in POI embedding [14], [31]. We implement the skip-gram model for POI embedding. • Causal [6] is a variant of word2vec that treats the previous footprints of the current POI as its semantic context to incorporate human practical transitional behaviors. • Deepwalk [17]is a data augmentation method that builds a POI graph to integrate users' historical visiting interests and geographical proximity and then leverages the skipgram technique for POI embedding. • GraphAE is popular method for node embedding. In this paper, we treat each POI as a node and build the same graph as Deepwalk for POI embedding. • GraphVAE is a variant of GraphAE, taking the advantage of VAE for POI embedding. Fig. 2 reports the performance of SSDL using different POI embedding methods. We can observe that our embedding method achieves the best results on the vast majority of metrics across the four cities, which indicates its higher effectiveness in capturing multiple human interests behind historical trajectory data. Next, we turn to investigate the effectiveness of devised components in SSDL. Herein, we conduct the experiments with three SSDL variants. The details are shown as follows: • SSDL-Base is a basic model that removes both graphbased embedding and mutual information regularization of SSDL. Instead, we use the word2vec technique for POI embedding. • SSDL w/o G only removes the graph-based embedding and use the word2vec for POI embedding. • SSDL w/o H only removes the mutual information regularization of SSDL. Fig. 3 illustrates the performance of variants on four cities. First, we can find that removing any modules would bring significant performance degradation, suggesting that both modules in our SSDL benefit to enhance POI prediction. Second, SSDL-Base performs worse than SSDL w/o H across all cities, demonstrating that considering multiple common interests behind historical check-in data are useful to discover human mobility patterns. Third, SSDL w/o G outperforming SSDL-Base proves that our self-supervised disentanglement learning is an effective module to provide promising representations for task inference.

Disentanglement Interpretability
In this part, we focus on studying the disentangled representations from the interpretability aspect. We first investigate whether z s and z r 1:n can be well extracted from original trajectories and reflect human time-invariant periodicity/habits and time-varying interests, respectively. To this end, we randomly sample eight different users' trajectories and change their orders to generate several groups of trajectories. Then, we use the TSNE toolkit [61]  the representations of z s produced by SSDL are grouped well, demonstrating that it can successfully separate the time-invariant factors to uncover the inherent preference of users that are not influenced by temporal factors. For z r 1:n , we visualize the distribution of the last states z r n for simplicity, we can find they are entangled, indicating that they are really affected by the temporal factors. Therefore, we conclude that z s and z r 1:n indeed play well the roles of timeinvariant and time-varying representations, respectively.
(a) z s in SSDL.
(b) z r n in SSDL. Besides, we also study the impact of our data augmentation approaches from a visualization perspective. We visualize the z s distribution of randomly sampled trajectories of eight different users after task training. As shown in Fig. 5(a), we can find β-VAE can only separate the representations with a small margin. Fig. 5(b) presents the results of SSDL without any data augmentations, and Fig. 5(c) shows the results of SSDL that has no augmentation for time-invariant factors. Compare to Fig. 5(d), we can clearly find that both augmentations used in SSDL can significantly help us distinguish the trajectory representations of different users. This observation further suggests that different user movement patterns can be well refined by our SSDL.

Sensitivity Analysis
Finally, we investigate the impact of significant hyperparameters in our SSDL to evaluate the model's robustness.
• Weight coefficients. The objective of our representation learning (cf. 21) contains three coefficients, which would determine the optimization procedure of each relative term. To this end, we generate different combinations of coefficients to investigate their impacts. The results of ACC@1 are shown in Fig. 6. We observe that γ=0.1 obtains better performance than γ=1 in general. We also find that the larger β helps to improve the accuracy of prediction since β represents the importance of mutual information between latent variables and trajectories. Finally, the weight coefficient α cannot be too large, otherwise it would constrain the performance. • Dimension of z s . Fig. 7 shows the performance variations of SSDL at different sizes of z s . We find that the larger dimension of z s does not give us promising results. Hence, for efficiency reasons, we set its dimensionality to 256. • Dimension of z r 1:n . Fig. 8 shows how different dimension of z r 1:n would influence the performance of SSDL. The performance decreases when the dimension of z r 1:n in each time step is larger than 32 and stays stable when the dimension increases. To obtain best performance, we set the dimension of z r 1:n to 32 in our experiments. • Embedding size. Embedding size is one of the critical factors affecting task prediction performance. Fig. 9 presents the effect of the embedding size. We can observe that the performance of SSDL climbs as the embedding size increases, and degrades or stays stable when the embedding size is larger than 256. In our experiments, we set the embedding size to 256.

CONCLUSION
In this paper, we present a self-supervised disentanglement learning framework, namely SSDL, to understand human mobility for tackling the next POI prediction problem. In contrast to existing sequential dynamics learning paradigms, SSDL mainly concentrates on disentangling the time-invariant and time-varying factors underlying massive sequential trajectories, which provides us an interpretable perspective to become familiar with human complex mobility patterns. Meanwhile, we present two practical trajectory augmentation strategies to relieve the sparsity issue of check-in data, which also enables the disentanglement of latent representations. Besides, we introduce a flexible graph structure learning method to incorporate multiple heterogeneous collaborative signals from historical checkins. We believe that several other associated contexts such as social relations and textual data are also easily incorporated into our graph learning. Finally, our extensive experiments on four datasets demonstrate the superiority of SSDL compared to state-of-the-art baselines. As our future work, we plan to investigate the possible more intricate prior assumption during representation learning.