{'fraction_non_alphanumeric': 0.04131780123771387, 'fraction_numerical': 0.008312098046353598, 'mean_word_length': 5.1320684523809526, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Modal Subjective Context Modelling and Recognition

Applications like personal assistants need to be aware ofthe user's context, e.g., where they are, what they are doing, and with whom. Context information is usually inferred from sensor data, like GPS sensors and accelerometers on the user's smartphone. This prediction task is known as context recognition. A well-defined context model is fundamental for successful recognition. Existing models, however, have two major limitations. First, they focus on few aspects, like location or activity, meaning that recognition methods based onthem can only compute and leverage few inter-aspect correlations. Second, existing models typically assume that context is objective, whereas in most applications context is best viewed from the user's perspective. Neglecting these factors limits the usefulness of the context model and hinders recognition. We present a novel ontological context model that captures five dimensions, namely time, location, activity, social relations and object. Moreover, our model defines three levels of description(objective context, machine context and subjective context) that naturally support subjective annotations and reasoning.An initial context recognition experiment on real-world data hints at the promise of our model.


INTRODUCTION
The term "context" refers to any kind of information necessary to describe the situation that an individual is in [2]. Automatic recognition of personal context is the key in applications like personal assistants, smart environments, and health monitoring apps, because it enables intelligent agents to respond proactively and appropriately based on (an estimate of) their user's context. For instance, a personal assistant aware that its user is at home, alone, doing housework, could suggest him or her to order a take-away lunch. Since context information is usually not available, the machine has to infer it from sensor data, like GPS coordinates, acceleration, and nearby Bluetooth devices measured by the user's smartphone. The standard approach to context recognition is to train a machine learning model on a large set of sensor readings and corresponding context annotations to predict the latter from the former. Existing implementations are quite diverse, and range from shallow models like logistic regression [14] to deep neural networks like feed-forward networks [15], LSTMs [7], and CNNs [12].
A context model defines how context data are structured. A good context model should capture all kinds of situational information relevant to the application at hand [2] and use the right level of abstraction [1]. Ontology is a widely accepted tool for formalizing con-text information [10], and several context ontologies have been proposed. Typical examples include CONON [16] and CaCONT [17]. CONON focuses on modeling locations by providing an upper ontology and lower domain-specific ontologies organized into a hierarchy. CaCONT defines several types of entities, and provides different levels of abstraction for specifying location of entities, e.g., GPS and location hierarchies. Focusing on semantic information of place, the work in [18] proposed a place-oriented ontology model representing different levels of place and related activities and improve the performance of place recognition. In [9], they proposed an ontology model involving social situation and the interaction between people.
These models, however, suffer from two main limitations. First, in order to support context recognition, the model should account for subjectivity of context descriptions. For instance, the objective location "hospital" plays different roles for different people: for patients it is a "place for recovering", while for nurses it is a "work place". This makes all the difference for personal assistants because the services that a user needs strongly depend on his or her subjective viewpoint. Most context models ignore this fact, with few exceptions, cf. [8]. Second, arguably answers to five basic questions -"what time is it?", "where are you?", "what are you doing?", "who are you with?", and "what are you with?"-are necessary to define human contexts. Correlations between these aspects are also fundamental in recognition and reasoning: if the user is in her room, a personal assistant should be more likely to guess that she is "studying" or "resting", rather than "swimming". In stark contrast, most models are restricted to one or few of the above five aspects and therefore fail to capture important correlations, like those between activity and location or between time and social context.
As a remedy, we introduce a novel ontological context model that supports both reasoning and recognition from a subjective perspective, that captures time, location, activity, social relations and object, and and that enables downstream context recognition tools to leverage correlations between these five fundamental dimensions. Our model also incorporates three levels of description for each aspect, namely objective, machine-level, and subjective, which naturally support different kinds of annotations. We apply and test our approach by collaborating with sociology experts within the SmartUnitn-One project [6]. We validate empirically our model by evaluating context recognition performance on the SmartUnitn-One context and sensor annotation data set [6], which was annotated consistently with our context model. Our initial results shows that handling correlations across aspects substantially improves recognition performance and makes it possible to predict activities that are otherwise very hard to recognize. arXiv:2011.09671v1 [cs.AI] 19 Nov 2020

CONTEXT MODELLING
Context is a theory of the world that encodes an individual' subjective perspective about it [3]. Individuals have a limited and partial view of the world at all times in their everyday life. For instance, consider a classroom with a teacher and a few students. Despite all the commonalities, each person in the room has a different context because they focus on different elements of their personal experience (the students focus on the teacher while the teacher focuses on the students) and ignore others (like the sound of the projector, the weather outside, and so on.) Given the diversity and complexity of individual experiences, formalizing the notion of context in its entirety is essentially impossible. For this reason, simpler but useful application-specific solutions are necessary.
Previous work has observed that reasoning in terms of questions like "what time is it?", "where are you?", "what are you doing?", "who are you with?", "what are you with?" is fundamental for describing and collecting the behavior of individuals [3]. Motivated by this observation and our previous work [4,5,11] , we designed an ontology-based context model organized according to the aforementioned dimensions of the world: time, location, activity, social relations and object. Formally, context is defined as a tuple: Context = TIME, WE, WA, WO, WI where: TIME captures the exact time of context, e.g., "morning". We refer to it as the temporal context. Informally, it answers the question "When did this context occur?". WE captures the exact location of context, e.g., "classroom". We refer to it as the endurant context. Informally, it answers the question "Where are you?". WA captures the activity of context, e.g., "studying". We refer to it as the perdurant context. Informally, it answers the question "What are you doing?". WO captures the social relations of context, e.g., "friend". We refer to it as the social context. Informally, it answers the question "Who are you with?". WI captures the materiality of context, e.g., "smartphone". We refer to it as the object context. Informally, it answers the question "What are you with?". Figure 1 shows a scenario as a knowledge graph representing the personal context of an individual in the class. For instance, attributes of WO are "Class", "Name", and "Role", and their values are "Person", "Shen", and "PhD student", respectively. Edges represent relations between entities, e.g., "Shen" is in relation "Attend" with "Lesson". The example in Figure 1 is presented in objective terms, that is, facts are stated as if they were independent of personal conscious experiences. However, each person interprets the world and her surroundings from her personal privileged point of view, which accounts for her personal knowledge, mental characteristics, states, etc. For instance, while in Figure 1 "Shen" has an objective role of Ph.D student, for other people "Shen" plays the roles of a "friend" or a "classmate" subjectively. The subjective context which is related to personal consciousness, knowledge, etc. can provide more information for applications such as personal assistant in order to give more intelligent services.
Notice that a person's view of her context is radically different from what her handheld personal assistant observes. In fact, machines interpret the world via sensors, while humans do not only interpret the world via their perceptions but with their knowledge as  well. For instance, while a machine views location (e.g., a building) as a set of coordinates, humans interpret it based on its function (e.g., whether the building is their home or office).
To model context precisely and completely, in addition to considering five dimensions, as discussed above, we also model three perspectives: objective context, subjective context and machine context. Table 1 shows the above example viewed through three types of perspective. The objective context captures the fact that at the University of Trento, Italy, at 11:00 AM, a person is attending a class together with Shen. When moving from objective to subjective, things change dramatically. From the perspective of the machine, the temporal context "11:00 AM" is viewed as a timestamp timestamp "1581938718026", and in subjective terms it becomes "morning"; similarly, "University of Trento" becomes coordinates "46 • 04'N,11 • 09'E" for the machine and "classroom" from a subjective perspective. For the perdurant context, the activity of taking lesson can be subjectively annotated as "study" by user, but it can be described as "connecting WIFI of classroom, sensors such as gyroscope, accelerometer are sensed as static". For the social context, "Shen" is described as friend subjectively by the user and the machine senses "Shen" is in the contact list of the user.

EMPIRICAL EVALUATION
In order to evaluate the proposed context model, we carried out a context recognition experiment using the SmartUnitn-One data set [6], and studied whether recognition of subjective context is feasible and whether taking inter-aspect correlations into account helps recognition performance.
Data Collection. The SmartUnitn-One data set consists of sensor readings and context annotations obtained from 72 volunteers (university students) for a period of two weeks. All participants were required to install the i-Log app [19], which simultaneously records sensor data from several sensors (cf. Table 2) and context annotations. During the first week, students were asked to report their own context every 30 minutes by administering them questionnaires comprising three questions about location, activity, and social relations. The i-Log app collected sensor data at the same time.
During the second week, the participants were only required to have the application running for the sensor data collection. All records were timestamped automatically. The questions were designed according to our context model and possible answers were modelled following the America Time Use Survey (ATUS) [13], leading to an ontology with over 80 candidate labels, see Figure 2 for the full list. Object context (WI) information was not collected as it is too hard to track without disrupting the volunteer's routines. All records were processed as in [20]. This resulted in 23309 records, each comprising 122 sensor readings (henceforth, features) and self-reported annotations about location, activity, and social context.
Experimental Setup. For every aspect in {WA, WE, WO}, we trained a random forest to predict that aspect from sensor measurements. We randomly split the dataset into training (75% of the records) and validation (25% of the records) subsets and then selected the maximum depth of the forest using the validation set only. The classifier performance was evaluated using a rigorous 5-fold cross validation procedure. The data set was randomly partitioned into 5 folds. We hold out the selected fold as the test set to train a classifier on the remaining folds and compute the performance on the held out (test) fold. Then, we compared this model to another random forests (with the same maximum depth) that was supplied both sensor data and annotations for (a subset of) the other aspects as inputs. In order to account for label skew (e.g., some locations and activities are much more frequent than others), performance was measured using the micro-average F1 score to account for class imbalance.
Results and Discussion. The average F1 score across users are reported in Figure 3. The plots show very clearly that knowledge of other aspects substantially improves recognition performance regardless of the aspect being predicted: supplying the other aspects as inputs increases the F1 score of predicting WA and WE by more than 10% and for WO by more than 5%. A breakdown of performance increase can be viewed in  Table 3. Improvement in F 1 score when using other aspects as inputs to the recognition model. Columns indicate the aspect being predicted. Figure 4 shows F1 scores (again, averaged across users) for each label. For WO, some labels are clearly easier to predict than others. The performance improvement is usually in the 5-10% range, with the notable exception of "other", which improves by about 20%. It seems that location information always facilitates recognition of WO, while activity does not. Their combination, however, is always beneficial. For WE, looking at either WO and WA helps recognition performance in all cases, and providing both WO and WA gives a larger improvement than than providing them separately. The exceptions are "library", "study room", and "shop", for which knowing WA improves more than knowing both WO and WA. This is somewhat surprising, as we expect social context to be moderately indicative of location, and deserves further investigation. Some locations ("canteen", "on foot", "auto", "shop", and "workplace") receive a major increase in recognition performance, from 25% to 40% approximately. This is partly due to the rarity of these classes in the data set, which shows that inter-aspect correlations supply to the lack of supervision. Finally for WA, some activities (like "housework",  "cultural activities", and "hobbies") are very hard to predict, as their F1 score is below 30%, while others ("work", "moving", and "lesson") are much easier to predict, with more than 80% F1 score. This mostly shows that rare activities are harder to predict, understandably, although other factors might play a role. Using the full context (with WE and WO) always improves performance, except for "housework". For all the other activities, the improvement is from 5% to 20%, and even larger for "Shopping", "Sport" and "Traveling", for which the improvement is up to 30%. This analysis provides ample support for our context model: correlations between different aspects improve context recognition performance for most users and, even more importantly, some values (like "Canteen") that are essentially impossible to recognize suddenly become much easier when full context information is provided.

CONCLUSION
We designed a novel context model that captures situational information about time, location, activity, and social relations of individuals using subjective-rather than objective-terms. An initial context recognition experiments on real-world data showed that machine learning models built using our context model produce higher quality predictions than models based on less complete context models. As for future work, we plan to study the effects of subjectivity more in detail, to migrate our architecture to more refined learning approaches (e.g., deep neural nets), and to carry out an extensive comparison against the state-of-the-art in context recognition.