{'fraction_non_alphanumeric': 0.031256917696019834, 'fraction_numerical': 0.009385929959711338, 'mean_word_length': 4.812660833762223, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Joint Learning with both Classification and Regression Models for Age Prediction

Age classification and regression are two main approaches to age prediction in social media, and these two approaches have their own characteristics and strength. For instance, the classification model can flexibly utilize distinguished models in machine learning, while the regression model can capture the connections between different ages. In order to exploit the advantages of both age classification and regression models, a novel approach to age prediction is proposed, namely joint learning for age prediction. Specifically, an auxiliary Long-Short Term Memory (LSTM) layer is employed to learn the auxiliary representation from the classification setting, and simultaneously join the auxiliary representation into the main LSTM layer for the age regression setting. In the learning process, the auxiliary classification LSTM model and the main regression LSTM model are jointly learned. Empirical studies demonstrate that our joint learning approach significantly improves the performance of age prediction using either individual classification or regression model.

Conventional approaches to age prediction are mainly based on supervised learning approaches where sufficient labeled data is essential for training the model. For ex-ample, extracting textual features from the messages posted by the user in Figure 1 to train an age prediction model. Generally, age classification and age regression are two implementation models in age prediction. The former model of age classification aims to classify the users into several different age groups [5], while the later model of age regression focuses on predicting the user's age with a discrete variable representing an exact age number [6][7][8]. These two different prediction models have their own distinctive characteristics. For instances, the classification model could flexibly take advantage of the discriminative models in machine learning, thus obtaining better performance on age prediction; the regression model is able to capture connections between different ages, meanwhile, has a better modeling for labels of age classification.
In order to exploit advantages of both the classification and regression models, we propose a novel approach, namely joint learning for user age prediction, which learns better representation by integrating both regression and classification representations generated from the classification and regression models respectively. Specifically, we firstly separate the age prediction task into a main task (age regression) and an auxiliary task (age classification), and then propose a joint learning approach to boost the performance of the main task with the help of the auxiliary task. A neural network architecture, namely Aux-LSTM, is proposed for the joint learning, which first learns an auxiliary representation from the auxiliary task with an auxiliary Long Short-Term Memory (LSTM) layer, and then integrate the auxiliary representation into the main task for joint learning.

Age Classification
In the last decade, there are various studies which focus their researches on age prediction. Most of previous studies model age prediction as a classification problem and conduct their empirical studies on two main domains, i.e., blog [9][10][11] and social media [12].
In the blog domain, Schler et al. [11] focus on textual features extracted from the blog text, such as word context features and parts-of-speech (POS) stylistic features for age classification. Burger and Henderson [10] explore some social features, such as location, time, and friend features, related to blogger ages. Ikeda et al. [13] is one exception which proposes a semi-supervised approach to age prediction by training multiple sub-classifiers with textual features. Some other studies, such as Rosenthal and McKeown [14] and Goswami et al. [15], explore both the textual and social features in automatic age classification.
In the social media domain, Mackinnon and Warren [12] explore some kinds of social features, i.e., the relationship between users to predict a user's age and country of residence in a social network. Peersman et al. [5] apply a text categorization approach to age classification with textual features only. More recently, Marquardt et al. [16] propose a multi-label classification approach to predict both the gender and age of authors from texts.

Age Regression
In contrast to age classification, much few studies model age prediction as a regression problem.
Nguyen et al. [6] explore textual features, such as word unigrams, POS unigrams and bigrams, together with gender features in age regression via a linear regres-sion model. Their empirical studies find that word unigrams can achieve reasonable performance and that POS patterns are strong indicators of the old age. Nguyen et al. [7] further explore age prediction of Twitter users with a linear regression model. They find that an automatic system can achieve better performance than a human being. Chen et al. [8] apply active learning approach to age regression for better exploiting the unlabeled data to improve the performance. Chen et al. [9] explore deep learning approach to age regression and combine results of age regression and classification linearly to improve the performance of age prediction. 3 Different from the above research, we employ a deep learning architecture to model age prediction. More importantly, we propose a joint learning approach with both age classification and regression models. To the best of our knowledge, this is the first attempt to address the age prediction when the two different models are both involved.

3.Background
In this section, we give the background on data collection and features (including textual features and social features) utilized for age prediction.

Data Collection
Our data is collected from Sina Micro-blog (http://weibo.com/), a famous Micro-blogging platform in China. The information of users can be acquired through analyzing their profiles. From the website, we crawl each user's homepage which contains user information (e.g., name, age, gender, Followers, Followings), and their posted messages. The data collection process starts from some randomly selected users, and iteratively gets the data of their followers and followings. We remove those unsuitable users who are verified as organizations because the age attributes of these users make no sense. Besides, although the posted messages are the basic and major factors to predict user age [6], some users post very few messages. To guarantee the reliability of the data, we remove those somehow non-active users who post less than 50 messages. In total, we collect the homepages of about 12000 users, together with their posted messages.  Figure 2 shows the user distribution in different ages. From this figure, we can see that the data distribution of user ages is rather imbalanced. Most users are young whose ages are in the range of 19 to 28.

Textual and Social features
Each user is represented by a feature vector, i.e., d i x R  as the input in prediction models. In the literature, various features, such as word unigrams, and social behaviors, have been successfully adopted on age prediction [14]. In this study, we categorize these features into two main groups, textual and social features. The former contains the features generated from the user-generated messages, e.g., word unigrams, while the latter contains the features generated from the user social behaviors, e.g., follower list and following list. Among textual features, BOW features are most popular in age prediction, and proven very effective due to the fact that word features reflect concerning topics, which can distinguish users of different ages. POS patterns are also popular textual features to capture the writing styles of the users.
Among social features, four statistical features, i.e., those with # of, capture the social behaviors of a user. For example, # of Messages indicate the numbers of the messages posted by a user and # of Comments indicate the numbers of messages commented from other users. The Time features capture the user habits on posting messages. For example, users of 20-24 ages might be more likely to post their messages very late at night. Followings and followers reflect the interests of users which provide an effective window to infer users' ages.

Basic LSTM Network
Long short-term memory network (LSTM) is proposed by Hochreiter and Schmidhuber [17] to specifically address this issue of learning long-term dependencies. The LSTM maintains a separate memory cell inside it that updates and exposes its content only when deemed necessary. A number of minor modifications to the standard LSTM unit have been made. In this study, we apply the implementation used by Graves [18] to map the input sequence of main task to a fixed-sized vector. Figure 3. A long short-term memory unit Figure 3 demonstrates the architecture of a LSTM unit which consists of an input gate i , an output gate o , a forget gate f , a hidden state h , and a memory cell c . At each time step t , the LSTM unit is updated as follows: Where t x denotes the input at time step t ,  denotes the logistic sigmoid function,  denotes elementwise point multiplication.W ,U andV represent the corresponding weight matrices connecting them to the gates. Intuitively, the forget gate controls the amount of which each unit of the memory cell is erased, the input gate controls how much each unit is updated, and the output gate controls the exposure of the internal memory state. Figure 4 illustrates the model architecture of age classification with a LSTM layer. We utilize input T to represent the input, and the input propagates through the LSTM layer, yielding the high-dimensional vector, i.e.,

LSTM Model for Age Classification
Where h is the output from the LSTM layer. Figure 4. Age Classification and Regression with LSTM Subsequently, the fully-connected layer which is similar to a hidden layer in the conventional multi-layer perceptron, accepts the output from the previous layer, weighting them and passing through a normally activation function as follows: Where  is the non-linear activation function, employed "relu" in our model. h  is the output from the fully-connected layer.
The dropout layer has been very successful on feed-forward networks [19]. By randomly omitting feature detectors from network during training, it can obtain less interdependent network units and achieve better performance, which is used as hidden layer in our framework, i.e., Where D denotes the dropout operator, p  denotes a tune-able hyper parameter (the probability of retaining a hidden unit in the network), and d h denotes the output from the dropout layer. The softmax output layer is used for a classification task. The output from the previous layer are then fed into the output layer to get the prediction probabilities, i.e.,

Classification Output
Softmax Layer

LSTM Layer
Dropout Layer

Regression Output
Linear Layer

LSTM Layer
Dropout Layer Where p is the set of predicted probabilities of the age classification, d W is the weight vector to be learned, and the d b is the bias term. Our age classification model for age prediction is trained to minimize a categorical cross-entropy loss function. Specially, the loss function is defined as follows: Where C loss is the loss function of the age classification, m is the total number of samples, l is the number of age categories, ij y indicates whether i-th sample truly belongs to the j-th category, ij p refers to the predicted probability. Figure 4 illustrates the model architecture of age regression with a LSTM layer simultaneously. Compare to the model architecture of age classification, we can see that most layers including the LSTM layer, the fully-connected layer, the dropout layer are the same as those in the age classification model, which have been described in subsection 4.2.

LSTM Model for Age Regression
Different from age classification, age regression utilizes a linear output layer instead of a softmax layer, i.e., Where d W and d b take the same meaning to age classification above. f is the predicted age value, which is a discrete variable.
For age regression, we employ "mean squared error" for loss function. Specially, the loss function is defined as follows: Where R loss is the loss function of age regression, and i y is the ground truth label of the i-th sample, and i f indicates the predicted age value of i-th sample, and m is the total number of the training samples. Figure 5 delineates the overall architecture of Aux-LSTM which contains a main LSTM layer and an auxiliary LSTM layer. In our study, we consider the age regression task as the main task and the age classification task as the auxiliary task. The goal of the approach is to employ the auxiliary representation to assist the regression performance of the main task. The main idea of our Aux-LSTM approach lies in that the auxiliary LSTM layer is shared by both the main and auxiliary tasks so as to leverage the learning knowledge from both the classification and regression models. Then we concatenate the two regression representation as the input of the hidden layer in the main task:

5.Joint Learning for Age Prediction
Where d main h denote the outputs of fully-connected layer (dense layers) in the main task, and  denotes the concatenate operator.

(2) The Auxiliary Task:
The auxiliary classification representation is also generated by the auxiliary LSTM layer, which is a reused LSTM layer and is employed to bridge across the classification and regression models. The reused LSTM layer encodes both the same input sequence with the same weights: Where aux h represents the representation for the classification model via the reused LSTM layer. Then a fully-connected layer is utilized to obtain a feature vector for classification, which is the same as the hidden layer in the main task: h denote the output of fully-connected layer (dense layer) in the auxiliary task. Other layers including a softmax layer and a linear layer, as shown in Figure 5, are the same as those which have been described in Section 4.2. Finally, we define our joint cost function for Aux-LSTM as a weighted linear combination of the cost functions of both the main task (i.e., the regression task) and auxiliary task (i.e., the classification task) as follows: In the above equation,  is the weight parameter, R loss is the loss function of age regression and C loss is the loss function of age classification. We take RMSprop [20] as the optimizing algorithm. All the matrix and vector parameters in neural network are initialized with uniform samples in , where r and c are the numbers of rows and columns in the matrices [21].

6.Experimentation
In this section, we systematically evaluate the performance of our joint learning for user age prediction with both the classification and regression models.

Experimental Settings Data Settings:
The data collection has been introduced in Section 3.1. We extract a balanced data set from the collected data by selecting 200 samples in each age and the age is limited in the range of 19 to 28, totally 10 age categories. We use 80% of the data in each age category as the training data and the remaining 20% data as the test data. We also set aside 10% users from the training as the validation data which are used to tune learning algorithm parameters.
Representations: The features employed in our experiment are separated into three groups, i.e., textual features, social features and joint features, where joint features mean the mixture of textual features and social features. To get the word, POS features, we use the public toolkit ICTCLAS 1 to perform word segmentation, POS tagging on the Chinese text. Each user is represented by a bag-offeature model [22]. Basic Prediction Algorithms: (1) Maximum entropy (ME), one popular shallow-learning algorithm, is implemented with the Mallet Toolkit 2 . (2) Support vector machine (SVM), another popular shallow-learning algorithm, is implemented with the libSVM 3 Toolkit. What's more, we implement SVM regression algorithm with the linear kernel, namely SVR for age regression. (3) MLP, includes fully-connected layer, activation layer and dropout layer which is implemented with the tool Keras 4 . (4) CNN, a popular deep-learning algorithm is used for performance comparison. We implement classifier and regressor via CNN respectively and the parameter setting follows the work by Johnson and Zhang [20]. (5) LSTM, as the basic prediction algorithm in our approach, is implemented with the tool Keras 4 . It is used in both the classifier and regressor for age prediction.
Parameters Setting: (1) The parameters of SVM, ME and SVR are set defaults. (2) The hyper parameters of LSTM are well tuned on the validation data by the grid search method, and most important hyper parameters are shown in Table 2. The hyper parameters of MLP is the same to LSTM. Dropout probability for classification 0.5

Epochs of iteration 15
Evaluation Metric: We employ the coefficient of determination 2 R to measure the regression and classification performance. Coefficient of determination 2 R is used in the context of statistical models with the main purpose to predict the future outcomes on the basis of other related information. 2 R is a number between 0 and 1. 2 R nearing 1.0 indicates that a regression line fits the data well [23]. Significance Test: T-test is used to evaluate the significance of the performance difference between two approaches [24].

Experimental results on individual age classification and regression
For thorough comparison, we implement several approaches to age prediction for both the classification and regression tasks. These approaches are introduced as follows.
Age   Table 3 shows the results of the four age classification approaches with three groups of features. From the table, we can see that C_CNN performs better than SVM, but worse than ME no matter what features are explored. Among all these approaches, C_LSTM performs best. Compared to SVM, ME and C_CNN, C_LSTM achieves the improvements of 7.5%, 4.1% and 6.1% with textual features, 9.8%, 4.4% and 7.8% with social features, and 6.1%, 0.6% and 3.4% with joint features. These results imply that the LSTM model is more appropriate for the age classification task than the other models. Significance test with t-test shows that our approach C_LSTM significantly outperforms the other approaches (p-value <0.05). Table 4 shows the results of the four age regression approaches with three groups of features. From the table, we can see R_CNN performs rather poorly when social and joint features are employed. Among all these approaches, R_LSTM preforms best. Compared to SVR, MLP and R_CNN, R_LSTM achieves the improvements of 9.2%, 5.3% and 6.5% with textual features, 6.5%, 2.1% and 8.2% with social features; 6.4%, 3.9%, and 7.8% with joint features. These results imply that LSTM is also more appropriate for the age regression task than the other models. Significance test with t-test shows that our approach R_LSTM significantly outperforms the other approaches (p-value <0.05).
Overall speaking, our empirical studies demonstrate that the LSTM model is a good choice to be the basic classification or regression algorithm in age prediction.

Experimental results of Joint Learning
For comparison, we implement following two approaches to joint learning on age prediction: Hybrid model：This approach focuses on combing the result of age classifier and regressor linearly. Specifically, this approach can be divided into two stages, in the first stage, we train age classifier and regressor respectively based on the same features. In the second stage, we simply combine the results from the classifier and regressor by averaging them. For instance, if the result of the classifier is 18 years old and the result of the regressor is 20 years old, the combining result is the average of them, i.e., (18+20)/2=19 years old. This approach is a straightforward strategy to take advantage of classification and regression model simultaneously.
Aux-LSTM：This is our approach which learns an auxiliary representation for joint learning, which is also described in section 4.4 in detail. In our Aux-LSTM model,  is set to 0.75.  Table 5 shows the performance comparison of four approaches to age prediction where C_LSTM and R_LSTM are two baseline approaches which are introduced in the above subsection and Hybrid model and Aux-LSTM are two joint learning approaches. From the table, we can see that, Hybrid model performs even worse than R_LSTM when textual features and joint features are employed, and it gets a slight improvement of 0.3% over R_LSTM with social features. Among all these approaches, Our Aux-LSTM performs best, achieving 0.481, 0.437 and 0.573 performance in 2 R when textual features, social features and joint features are employed respectively. Compared to R_LSTM, Aux-LSTM results in an improvement of 1.3% with textual features, 2.1% with social features and 2.0% with joint features. The experimental results demonstrate that our joint learning is consistently effective for performance improvement of age prediction. Significance test with t-test shows the improvement of our approach over Hybrid model is significant (p−value < 0.05).

7.Conclusion
In this paper, a novel approach is proposed, namely joint learning for age prediction, to exploit advantages of both the age classification and regression models. In our approach, an auxiliary LSTM layer is employed to learn the auxiliary representation in the age classification task (as the auxiliary task) and employ it in the age regression task (as the main task). To achieve this, a neural network based model, namely Aux-LSTM, is employed to bridge across the classification and regression models via a shared LSTM layer. Empirical studies demonstrate that the LSTM model is appropriate for both the age classification and regression task. Moreover, the results show that our joint learning approach significantly boost the performance of the main regression task with the help of the auxiliary classification task.
In our future work, we would like to seek better modification on Aux-LSTM for further improvement. Moreover, we would like to apply our proposed Aux-LSTM model in other NLP applications which involves both the classification and regression tasks.