{'fraction_non_alphanumeric': 0.035793940895855275, 'fraction_numerical': 0.009494455114305186, 'mean_word_length': 5.125711574952562, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multimodal Analytics to Understand Self-Regulation Process of Cognitive and Behavioral Strategies in Real-World Learning

Recent studies suggest that learning “how to learn” is important because learners must be self-regulated to take more responsibility for their own learning processes, meta-cognitive control, and other generative learning thoughts and behaviors. The mechanism that enables a learner to self-regulate his/her learning strategies has been actively studied in classroom settings, but has seldom been studied in the area of real-world learning in out-of-school settings (e.g., environmental learning in nature). A feature of real-world learning is that a learner’s cognition of the world is updated by his/her behavior to investigate the world, and vice versa. This paper models the mechanism of real-world learning for executing and selfregulating a learner’s cognitive and behavioral strategies to self-organize his/her internal knowledge space. Furthermore, this paper proposes multimodal analytics to integrate heterogeneous data resources of the cognitive and behavioral features of real-world learning, to structure and archive the time series of strategies occurring through learner-environment interactions, and to assess how learning should be self-regulated for better understanding of the world. Our analysis showed that (1) intellectual achievements are built by self-regulating learning to chain the execution of cognitive and behavioral strategies, and (2) a clue to predict learning outcomes in the world is analyzing the quantity and frequency of strategies that a learner uses and self-regulates. Assessment based on these findings can encourage a learner to reflect and improve his/her way of learning in the world. key words: real-world learning, learning analytics, self-regulation, multimodal behavior sensing, knowledge representation


Introduction
Among the various types of learning supports, recent research in learning science acknowledges the importance of encouraging a learner to learn "how to learn" [1]. Regarding this point, Weinstein stated that learners must be selfregulated so that they can take more responsibility for their own learning processes, meta-cognitive control, and other generative learning thoughts and behaviors [1].
"Learning how to learn" means that a learner learns how to self-direct and self-regulate his/her learning strategies by self-assessing the state of his/her learning for the purpose of effectively acquiring knowledge in his/her situation. In various educational settings (e.g., mathematical learning [2], [3], language acquisition [4]- [6], Manuscript  engineering education [7], [8], learning from others [9], and inquiry-based learning in a classroom [10]), a learner uses strategies (e.g., ways of inference, problem solving, surveying, and collaboration) to solve a problem or to achieve a particular intellectual achievement. When a learner cannot obtain a new idea or finding by a certain strategy of learning, the learner should self-regulate his/her learning by assessing and flexibly changing the old strategy.
Recently, the mechanism of executing or selfregulating strategies in classroom learning has become one of the important topics in learning science [1]- [7], [9], [10]. However, not much is known about the self-regulation mechanism of experience-based learning in out-of-school and real-world settings. Therefore, we model, assess, and understand how to learn by physical behaviors in a realworld situation, i.e., by the process of executing and selfregulating strategies in order to behave for better understanding of the world.

Self-Directed Learning in a Real-World Field
As an alternative viewpoint of traditional classroom learning [11], the theory of situated cognition suggests that knowledge can be better acquired when a learner is involved in a social situation and context in which the knowledge is actually used [12]. Our research investigates the importance of a self-directed style of real-world learning by which a learner is involved in diverse out-of-school situations, autonomously thinks and behaves in such real-world situations, and acquires knowledge through learner-world interactions [13]- [15].
As opposed to "a laboratory setting" or "a learning setting driven by artificially pre-programmed stimuli", our research defines the term "real world" as "the world where human activities occur in a natural setting as they really are without being artificially or strictly controlled". An example of a real-world field is a natural environment where various phenomena spontaneously occur in a symbiotic relationship among various living things, e.g., plants and animals. A model case of real-world learning is a self-directed style of environmental learning by autonomously walking and exploring in a natural environment.
Learning in the real world is done by a wide range of people including adults [12]. Our ICT-based studies of real-world learning from 1999 to the present included Copyright c 2020 The Institute of Electronics, Information and Communication Engineers several trials of field observations of elementary schoolchildren, but this paper focuses on real-world learning by adults in their twenties and over who have higher abilities of cognition, communication, interaction, and self-directed learning. The photograph in Fig. 1 shows adult learners in their twenties who participated in our experiment of environmental learning.
2.2 Self-Directed Task: "Learner as Scientist" Our real-world learning is designed to realize the concept of "learner as scientist" [16] and to encourage a learner to behave as a forest ecologist in a natural field. To be specific, a natural environment contains diverse information that is not artificially created and illustrates various features and functions of the self-organizational order of an ecosystem. An ecologist generally conducts scientific investigations to observe phenomena in a natural environment, finds and extracts essential information from the environment, and interprets this information to build a hypothesis or knowledge that can well explain the nature of an ecosystem [17]. We consider this to be an ideal example of the self-directed inquiry process to acquire knowledge about the world.
Based on the "learner as scientist" model, we designed a self-directed style of learning task for problem solving in a real-world situation, i.e., a hypothesis verification task [15]. This task was designed to encourage learners in a group to self-direct exploratory activities in a natural field and to cooperatively consider, build, examine, and verify hypotheses about the nature of phenomena and ecosystems in their field. Unlike conventional education, this task does not rigidly control learning activities; instead, this task fosters a learner's self-directed process of selecting, using, and modifying strategies to achieve intellectual achievements. As social constructivism suggests [18], interaction among learners is important for knowledge creation, and our task was designed to include group-style collaboration.

Learning by Cognition and Behavior
Generally, the learning resources of classroom learning are the information in textbooks, electronic documents, or computer databases. In contrast, the resources in real-world learning are information existing in the world [14], [15]. Therefore, even if a real-world learner is situated in a natural environment, no information or no knowledge can be obtained in the world by just thinking without any interactions with the world. Figure 1 shows that a learner internally constructs knowledge about the world by the self-directing behavior of (1) searching and observing a real-world phenomenon to extract information from the world, and (2) examining and interpreting the information found in the world.
In the process of acquiring knowledge, a learner's cognition of the world is updated by his/her behavior while investigating the world, and vice versa. This means that a learner's cognition and behavior are coordinated during realworld learning.

Research Issues
If analytics can trace and assess the states of self-regulation of strategies during real-world learning, we can obtain clues for encouraging a learner to flexibly switch or refine strategies for better understanding of the world. By integrating the viewpoints of learning science, behavior informatics, and embodied cognitive science, we model the process that a learner self-organizes and activates his/her knowledge space by self-regulating his/her strategies to interact with the world. Then, we propose multimodal analytics to structure, code, archive, and assess this strategy-based process by which a learner in the world self-regulates different human functions from cognition inside a brain to behavior via a body.

External and Internal Control of Learning
We consider a learner to be simultaneously regulated by two types of controls: external control and internal control.

External control given from outside a learner
External control is explicitly designed or implicitly given by a learning environment, for example, (a) an educator's instruction to navigate or direct a learner's activities, (b) a learning task to explicitly determine the way of thinking, and (c) a hidden constraint given by a learning environment.

Internal control self-organized inside a learner
Internal control is applied by strategies that a learner self-directs based on the given learning task.
We consider that between these two types of controls, the dominant control is determined by the design policy of the learning environment. When a learning environment is designed to adopt strong regulation from outside a learner, the dominant control is external (e.g., traditional programmed instruction in a school [11]). When the learning environment is designed to adopt weak regulation from outside a learner, the dominant control is internal (e.g., selfregulated learning in a classroom [1], [19] and real-world learning in a self-directed style [13]- [15]).

Self-Organizational Dynamics of Knowledge Spaces
Knowledge is an aggregate of memory information that is semantically structured. Thus, this research considers "knowledge acquisition" inside a learner as constructing a semantic network of different information [20]. In selfdirected learning, a learner needs to think and decide "what and how he/she should learn" and to autonomously construct strategy and knowledge. The range of knowledge space (i.e., what a learner can learn) in self-directed learning is not strictly limited by an educator's external control. Possible intellectual achievements are decided by a learner's internal control of searching, selecting, executing, self-assessing, and modifying his/her internal strategies. A learner's reflective process of assessing, proposing, and directing his/her way of learning is a key of self-regulation of learning.
We consider that knowledge space in self-directed learning is an open system with chaotic and selforganizational dynamics. From a mathematical viewpoint, a chaotic and complex system having diverse interactions displays nonlinear behavior, such as the emergence of unpredicted functional properties that have not been previously observed in the system. Therefore, a learner should execute and self-regulate diverse strategies not only to well extract information from the world but also to foster the changeability of the dynamics of his/her internal knowledge space.

Real-World Inquiry Behavior as Situated Action
For constructing an internal representation of knowledge about the world, our past studies found that a learner performs real-world inquiry behavior to search, discover, examine, and understand information in the world [14], [15]. As an example, learners frequently display a particular stay behavior (called stable stay) when they are engaged in observation, knowledge exchange, and intellectual investigation in the world [15]. A learner can acquire a higher level of knowledge (e.g., a scientific explanation of a real-world phenomenon) when the learner carefully pays attention to the world by physically moving his/her head and body to multiple observation targets in a natural field [14]. Through these real-world inquiry behaviors, a learner internally makes a semantic network of the found information. Furthermore, motion patterns of a learner's body accompanying this stay behavior and the attention distribution behavior can be defined and estimated with sensor data for orientation, speed, angular velocity, and acceleration of the head and body [15]. Our important findings from these results were as follows: (1) a learner's body-based behavior (e.g., real-world inquiry behavior) and his/her intellectual situation (e.g., the content of interest, the level and state of knowledge acquisition) are coordinated in the world, and (2) (1)'s internal situation can be estimated to some extent by observing a learner's external situation, externalized as the learner's body-based behavior. Figure 2, which is the detailed process model of Fig. 1, shows that a learner's knowledge space is self-organized by circulating real-world information under his/her situation. The environment surrounding a learner is a part of his/her cognitive system (i.e., embodied cognition [21]), and his/her behavior generation in the world is adapted to the situation where the learner is involved (i.e., situated action [22]). Situated learning is well explained by ethnomethodological framing, in which behavior is free flowing, habituated, and structured by learners' interactions with others and the environment [12]. However, as Ext. 0 on the right-hand side in Fig. 2 shows, we find a hidden constraint given by the learning environment (1(c) in Sect. 3.1). For example, if a learner is located in an interesting place where strange and unidentifiable objects exist, the learner can obtain information about the nature of the objects by engaging in the appropriate behavior [14], [15]. If a learner exists at a location that does not include important information, the behavior for inquiring knowledge is not effectively generated. This means that real-world inquiry behavior is situated action made under external control tacitly given by the real-world situation surrounding a learner (i.e., spatial context of the world [23]).

Internal Process to Drive Inquiry Behavior
We quantitatively studied how the external situation of a learner influences his/her real-world inquiry behavior, as shown in Sect. 3.3.1. However, conventional models have not been proposed to explain how the internal situation of a learner generates and regulates real-world inquiry behavior. Therefore, as shown in Fig. 2, the current paper models the process in which a learner's internal strategies drive the system of behavior generation, information processing, and knowledge acquisition in the world.
In our model, double and asynchronous loops consist of not only the cycle of real-world inquiry driven by behavior (Ext.1, Ext.2) but also the cycle of internal control driven by sensed information and strategy regulation (Int.0 -Int.3). A learner uses his/her body as an interface to actuate the world and generates behavior to investigate a phenomenon of interest to him/her (Ext.1). Through this behavior, the learner selectively senses and cognizes information of a specific target in his/her surroundings and obtains focused real-world information to be examined (Ext.2). Ext.1 is the behavior for cognitive change, and Ext.2 is the cognition for the next behavior; the behavior and cognition illustrate that the actuator and sensor of a learner's body are coordinated in the world.
Real-world information obtained from body-based interaction with the world is inputted into a learner's knowledge space so that the learner can carefully examine, interpret, and understand the information (Int.1). As parallel processing, a learner self-regulates his/her information circulation system by self-assessing the state of the system and his/her survey method, and, if necessary, by changing how to learn in the world (Int.0). Through the information processing of Int.1 and the learning regulation of Int.0, internal strategies are generated and self-regulated for adapting to the learner's internal and external situations (Int.2). The self-regulated strategies trigger the coordination between the actuator and sensor of a learner's body function in order to generate new behavior for making effective inquiries in the world (Int.3). A learner's behavior at a certain time point is generated within the range that his/her strategies allow [13]. To be concrete, a learner's strategies function as internal constraints for generating behavior in the world, i.e., a restrainer of one behavior and a promotor of another behavior. Again, the functions of this real-world behavior newly generated from the learner's strategies are to actuate the world and obtain new cognitive resources from the surroundings (Ext.1, Ext.2), and to change and expand his/her internal knowledge space (Int.1).
It is expected that the new structure of the knowledge space is self-organized through the cyclic process in which a learner's strategies in the world drive both the cognition for behavior and the behavior for cognition. Therefore, the possibility of acquiring new knowledge decreases when learning reaches the saturation state, which means that real-world information, strategies, and behavior are not updated. In such a case, learning should be self-regulated by executing a strategy to break the saturation state of the information circulation system. Figure 2 shows that a learner's cognition and behavior are coordinated by executing and self-regulating strategies in the world. This research considers three types of strategies:

Real-World Learning Strategies
Regulation strategy Strategies to self-regulate cognitive and behavioral strategies. For example, strategies of (1) self-assessing the state of the current knowledge acquisition, (2) self-assessing the effects of the performed behavior, and (3) self-assessing, proposing, directing, and switching a learner's way of learning in the world. Cognitive strategy Strategies to manipulate a learner's own or other learners' knowledge spaces, to examine real-world information, and to internally develop an understanding of the world. For example, strategies of using and processing real-world information, such as inference (deductive/inductive), prediction, classification, interpretation, and reasoning. Behavioral strategy Strategies to perform physical behaviors for executing cognitive strategies. For example, (1) interaction strategies with the world for the purpose of searching, observing, examining, and interpreting information embedded in the world, (2) communication strategies for finding, identifying, defining, and sharing a real-world problem, and (3) discussion strategies for constructing and verifying a hypothesis.
Cognitive strategies completely closed inside a learner without any interactions with others or the environment (e.g., solo thinking without communications or field observations) are not necessarily accompanied by behavioral strategies. However, in the case of group-style real-world learning, inquiries are usually made through learner-learner communications and learner-environment interactions [14], [15], and therefore a learner's cognitive strategies are often executed by behavioral strategies.

Analytical Framework
Based on the strategy-based learning model shown in Sect. 3, this section proposes multimodal analytics to observe a learner's internal and external situations and to trace, code, archive, and examine the time-series self-regulation process of cognitive and behavioral strategies for better understanding of the world (Fig. 3). As mentioned in Sect. 2.1, this analytics is developed to be applied to a self-directed style of real-world learning by adult learners.

Integrating Heterogeneous Data Resources
To multidirectionally examine strategy executions during one learning event, we collect and integrate two types of qualitatively different data resources. In Fig. 3, data resource 1 includes the features of behavior and data resource 2 includes the features of cognitive activities. When a learner's data from one device include multidirectional information, we use the data to extract both behavioral features and cognitive features.

Data 1: External and Behavioral Process
To examine a learner's external situation during strategy execution, this research proposes the extended use of our multimodal "behavior" observation technologies (e.g., utterance, vision, walking, and head and body movement, attention) by wearable sensors (e.g., eye camera, microphone, local positioning system, 3-axis accelerometer, and 3-axis gyroscope) of each learner and the cameras held by experimenters.
Time-series data from each learner's eye camera (firstperson view video) and each experimenter's handheld camera (third-person view video) are used in this research to observe and identify each learner's inquiry behavior at each time point, including his/her address of strategy execution (e.g., place of interest, focused real-world information), and his/her surrounding information. Time-series utterance data are used to identify the information on which a learner focuses and operates in the world at each time point. This research identifies the target of each learner's focus from location data and motion data (e.g., head and body orientation).

Data 2: Internal and Cognitive Process
To examine a learner's internal situation, this research extends our multimodal "knowledge" observation techniques [24]. In the current research, we collect on-site utterance data with a wearable microphone to read and identify a learner's strategy of thinking during his/her knowledge exchange. This is because utterance data externalized during communication are reliable for reading the learner's internal thoughts from the outside [25].
Furthermore, our activity map [24] is adopted to this study for collecting off-site data that identify not only each learner's knowledge acquisition but also his/her thinking process and cognitive functions. Specifically, at an off-site setting in a classroom after finishing real-world learning, each learner draws an activity map in network style to externalize his/her internal process during real-world inquiry behavior. A node in the map shows the concept acquired by a learner, and an arc shows the relationship among multiple concepts found by a learner. A learner clarifies the role of each activity during real-world learning by adding the following attribute information to each arc: (1) example of a concept, (2) general knowledge (a learner's background knowledge), (3) question, (4) hypothesis to explain the relation between concepts, (5) observation of a phenomenon in the world carefully watched for a period of time, (6) verification of behavior to examine and verify the hypothesis, and (7) discovery (new knowledge obtained through observation, discussion, and hypothesis verification). Free software † was used as a tool for drawing our activity map.

Expression of Multiplex Strategy Execution
Cognitive functions of a learner can be objectively annotated with strategy codes such as "infer", "explain", and "assess" [26]- [28], but we should create domain-oriented strategy codes that cover cognitive, behavioral, and regulation strategies in a real-world learning setting. Our codes should be designed at the abstraction level that engineering techniques can objectively trace. Our codes should be also parameterized to be systematically computed for data mining.
As a preliminary study for clarifying potential strategies that can occur during the process of real-world learning, we (multiple labelers) listed the contents, targets, and operations of strategies that were given by approximately 250 learners who participated in our past experiments of environmental learning. Specifically, we examined not only learners' behaviors to inquire about the world but also their utterances to externalize and share the process of information processing or to verbally regulate the state of learning. Then we investigated the contents, targets, and operations of their inquiries, information processing, and learning regulation. We compiled a list by both participation observation in on-site real-time settings [29] and video-based observations in an off-site setting.
This list enabled us to develop a structured expression of the state of strategy execution at an arbitrary time point with the three parameters of executor, operation, and target, as shown in Table 1. The state that a person (executor) executes an operation (operation) on a certain target (target) at a time point (t) is coded as S t (executor, operation, target). When a person (learner ID = L1) was making an inquiry by assessing the difference among phenomena in the world and inferring the background reason for a phenomenon on which he/she focused, the state of his/her strategy execution is coded as the simultaneous execution of ("L1", "assess", "difference among phenomena") and ("L1", "infer", "reason for phenomenon"). When learner L1 self-assessed the effectiveness of the survey method that he/she used, the state of strategy execution is coded as ("L1", "assess", "survey method"). If this self-assessment encouraged L1 to be engaged in a subsequent state of strategy execution of ("L1", "propose", "survey method"), we (i.e., the analysts) can understand that learner L1 was trying to change how to learn in the world.
With the above expression, a strategy is expressed by combining an operation code and a target code. We defined 52 operation codes and 44 target codes in total. As shown in Table 1, target codes include a physical object in the world (e.g., observation target), information in the world (e.g., difference among phenomena), a learner's own or other learners' cognitive resource (e.g., an idea or a question), a survey method, and the state of information circulation. Operation codes express cognitive, behavioral, and regulation operations on these targets.
Many of the general operation codes in the table are widely used in traditional learning sciences (e.g., "infer", "explain", and "assess" [26]- [28]), but our target codes are extended to express the information processing derived from real-world phenomena. By combining the parameters of operation and target, a wide range of cognitive, behavioral, and regulation strategies during real-world learning can be expressed even if we exclude logically incompatible combinations of parameters. Each code of target and operation Table 1 State of strategy execution: S t (executor, operation, target).
Showing all codes for constructing learning strategies in Table 1 encourages readers to adopt a similar data-mining approach for analyzing real-world learning. Our strategy codes can be used by several annotation tools, which include our developed software (SyncPlay † ) and other free software (ELAN † † ).

Annotating Strategy Executions
Based on our multi-parameter expression of strategy execution, we depict the audio-visual data at each time point as several kinds of annotation codes of the used strategies. ure 4 is a schematic depiction of structured time-series data of strategy executions, given certain annotation data of realworld learning. This figure can be created by integrating the data of P1 and P2 in the following procedure: P1 A learner internally constructs new meanings by examining real-world phenomena. Thus, we divide the time sequence of each learner's activities into chunks of meaning. A chunk of meaning is the basic unit of our analysis. For example, the period from time t i to t i+1 in Fig. 4 is when a learner L1 examined a chunk of meaning C i , i.e., "rotten branches at the waterside".
A chunk of meaning is extracted by carefully manually classifying the contents of nodes (i.e., concepts) and arcs (i.e., relationships among concepts) in each learner's activity map. This is because a learner's activity map is a network-style expression of knowledge that he/she acquired in the world, and the map information is the aggregate of chunks of meaning. In the example of Fig. 5, a chunk of meaning C i is extracted from the node information of N1 and N5 and the arc information of A4 and A10 in the activity map of learner L1.
An activity map has content information to show what a learner examined in the world, but the map does not have time information to show when the content of each node and arc was examined by the learner as his/her on-site activities. For making a precise analysis, we manually examine multimodal time-series data (e.g., wearable cameras, microphones, handheld cameras) to identify which on-site utterance and behavior were related to the examination of the content of each node and arc in a learner's activity map. By referring to the time information synchronized among all devices (e.g., sensors, video cameras), we identify and annotate the start and end time that the content of each chunk of meaning was examined in the world.
Time t i+1 to t i+2 is the period that no data of chunks of meaning are given (chunk of meaning "N/A"). Since a chunk of meaning is extracted by the above technique, a N/A period is time spent on on-site activities that are not related to any knowledge that a learner finally externalized in arcs or nodes of his/her activity map.
P2 As a process independent of P1, multiplex strategy execution in the world is expressed by providing 0-5 strategy codes for each time point. Since our preliminary surveys found that in our case of environmental learning, the number of multiplex strategy executions did not exceed five, this research set the upper limit of simultaneous strategy annotations at five.
By complementarily using multimodal reference data (Sect. 4.1) to monitor a learner's internal and external situations, we identify (1) target of focus and inquiry,  Table 1, and construct timeseries data of each learner's strategy executions.
By integrating the data of P1 and that of P2, we can find strategies used for examining a certain content of learning. For example, Fig. 4 shows that learner L1 examined different contents of learning (e.g., C i , N/A, C k ) by differently executing strategies. Furthermore, our analytics is designed so that a learner's activity at a certain time point can correspond to two or more chunks of meaning when the activity is an extensive investigation by simultaneously examining different chunks of meaning of his/her activity map.

Database System of Knowledge and Strategies
We developed a database system to structure and archive the relationships among the contents of each learner's knowledge acquisitions and the result of his/her strategy execution, as shown in Fig. 6. The system automatically analyzes annotation data to extract time-series information of the multiplex strategy executions by each learner, and converts the data into the data format of our database. Time information (i.e., start and end time) of each chunk of meaning is stored in our database by being associated with used strategies. Our system automatically stores table values including the following:

Assessment information of knowledge acquisition
The content of each chunk of meaning, that is, the contents of nodes and the attributes of arcs in an activity map.

Assessment information of strategy executions
The state of strategy executions (executor, operation, target) for examining each chunk of meaning.
By combining these two types of assessment information, our system calculates and stores numerical metrics that show how a learner's or a learner group's executions of strategies are related to knowledge acquisitions in the world. Without knowing the detailed structure of our database, an analyst can use our GUI tool (Fig. 7) to interactively generate SQL inquiries to obtain metrics prepared in the system or to calculate new metrics. Newly generated metrics are archived in the database for sharing with other analysts.

Related Works
A paper [30] examined the development of a particular infant by creating a repository of long-period audio-visual data of his lifetime experience from 9 to 24 months old, and traced longitudinal changes of his language acquisition at home. This idea was recently extended to assess delayed language development of a child who had been diagnosed with autism spectrum disorder (ASD) [31]. Similar to these prior studies [30], [31], we consider that development or learning is achieved by being involved in the world, and thus practical understanding of development or learning can be acquired from data capturing real-world experiences. However, these prior studies [30], [31] did not focus on how an infant acquired knowledge by executing his internal strategies to interact in and with a real-world environment. The audio-visual recording techniques of these prior studies are extended in our current study to assess the knowledge acquired through the process of collaborative learners self-directing the executions of their strategies in the world. Our extension is created not only by probing the self-organizational and networked structure of a learner's internal knowledge but also by coding, structuring, and tracing his/her internal strategy changes in the world.
A recent study [32] aimed at understanding the complex system of sensor-motor behaviors that underlies the establishment of joint attention between parents and their children (i.e., 12-or 18-month-old infants), by using headmounted eye-tracking systems and frame-by-frame coding of mutual actions. Similar to our study, this prior study was interested in assessing the sensor and motor functions of a person by using the data of attention and actions [32]. However, that study [32] did not focus on how the behavior generation system of an infant was related to his/her internal strategy regulations for interacting with others or the surroundings. Our current study models and traces a learner's self-directed process of internal strategy executions that coordinate his/her sensor and motor functions, and the real world. Our dataset includes the multisensory information used in [32] (e.g., human attention, conversation content, and hand movement) but extends that dataset to trace timeseries strategy executions by additionally capturing the data of real-world experiences such as a learner's total body movement to interact with others or the environment, and a learner's internal observation data of his/her intellectual interactions.
Unlike conventional studies [30]- [32], we are interested in experience-based learning by adults. More importantly, our analytics is not just capturing a learner's realworld behavior as external data from multimodal sensors. Essential features of our analytics are (1) modeling the dynamics that a learner's knowledge space is activated, fluctuated, and self-organized in the world by different human functions from cognition inside a brain to behavior via a body, (2) structuring a coding scheme of the time-series and multiplex execution of strategies related to a learner's on-site cognition, behavior, and learning regulation, and (3) integrating a learner's external data and his internal data at a high abstraction level of activity information for the purpose of assessing the processes by which learners autonomously decide, execute, and self-regulate "what and how to learn in the world".

Objective
Because not much is known about the mechanism of selfdirected learning through interaction with the world, this paper performs a data analysis to accumulate the basic understanding of how learner strategies are executed and self-regulated during different levels of real-world learning activities.

Method
Our analytics was used to qualitatively and quantitatively assess the data obtained from our field experiments with 15 adult learners who participated in group-style learning (three learners in a group) for one hour in an area (about 130 × 50 m) of Kamigamo Experimental Forest, Kyoto University, Japan. Our field is a forest environment that is close to human's living areas, and maintains a symbiotic relationship among humans and the nature. In general, this type of forest environment is often used for the purposes of nature observations, ecological surveys, nature games, and environmental learning.
The 15 learners consisted of eight men and seven women in their twenties, who were applicants for our public call for participants. This age range was selected based on our research focus described in Sect. 2.1. Our prequestionnaire confirmed that the learners were general ones who did not have special experiences of environmental learning. The learners were randomly assigned to a group of two men and a woman, or a group of a man and two women. Three members of each group had not known each other before the experiments. We had not been acquainted with any of the 15 learners before the experiments.
Our hypothesis verification task (Sect. 2.2) was given to the learners, because the task encourages a learner to be involved in a learning environment with ecological validity so that his/her strategy executions can be self-directed without being externally controlled by experimenters' interventions, educators' instructions, or pre-defined navigations. This task was used in our experiments by considering adult learners' ability of cognition, communication, interaction, and self-directed learning.
The total amount of our annotated data was 50,475 s (14.0 hours), because we excluded data irrelevant to learning (e.g., the time required for experimenters' instructions or experimental operations). Although our annotation data include time codes with the millisecond order (Table 2), the time precision of this analysis was the second order by considering the temporal accuracy of the manual annotation data.

Data Annotation
For an accurate analysis, the content of knowledge acquisition and strategy executions at every time point was manually identified using multimodal data (e.g., audio-visual evidence and activity maps) and was carefully annotated with structured strategy codes (Sect. 4.2, Table 1) by an expert who has studied environmental learning for about 20 years and his assistants, who were well educated in the field of environmental learning. This strategy annotation was performed after multiple labelers had carefully determined the criterion for strategy annotation and mutually checked the consistency of their judgements. Because one of the 15 learners had some grammatical errors in his activity map, we slightly modified the map to correct the errors without changing the meaning of the contents. Figure 4 is a graphical representation of the structure of multiplex strategy annotation, given certain annotation data. Table 2 is given as an example of the manual annotation data of multiplex strategy executions at the experiments that this study analyzed. The format of a line of our annotation data is as follows: identifier, start time, end time, duration, description. Identifier consists of a learner's ID and the attribute of data line (e.g., operation code, target code, speech content, chunk of meaning, or experimenter's operation to start and end the experiment). When identifier shows a meaning chunk, description shows the IDs of nodes and arcs constituting the meaning chunk † . Description also includes a learner's conversation content, his/her strategy execution (i.e., operation code, target code), and the experimenter's calls of the start and end of the experiment. The fourth and fifth lines of the table show timeseries strategy executions that three learners of the group G1 performed during a chunk of meaning (00:34:43.399 -00:34:53.561) consisting of nodes N1, N4, and N7 and arcs A3 and A6 in L3's activity map. A learner's simultaneous execution of multiple strategies can be found in this table, e.g., L3 Operation1, L3 Target1, L3 Operation2, and L3 Target2 during 00:34:43.399 to 00:34:45:224. Systematically structuring the execution states of real-world strategies (Fig. 6) was performed by automatically analyzing our hand-labelled annotation data.

Mining for Relationships between Strategy Executions and Learning States
We consider that a learning activity at every time point does not necessarily make the same contribution toward acquiring real-world knowledge. We expect that some activities contribute toward a better understanding of the world, but some activities do not. Thus, we extracted chunks of meaning (Sect. 4.3) from the semantics of the summative knowledge that a learner finally externalized in his/her activity map. Then, we classified an activity at an arbitrary time point into type A or B by judging whether the activity was included in the examination of at least one chunk of meaning. (

1) Type A activity
These are on-site learning activities that examined one or more chunks of meaning in a learner's activity map, e.g., periods C i and C k in Fig. 4. This activity type is considered to comprise the activities that directly influenced the summative knowledge that a learner finally acquired.
(2) Type B activity These are on-site learning activities that did not examine any chunks of meaning in a learner's activity map, i.e., the N/A period in Fig. 4. This activity type is considered to comprise † A meaning chunk "N/A" is discriminated by assigning a z# code to description. the activities that did not directly influence the summative knowledge that a learner finally acquired.
According to the definitions, we consider type A activities to be more effective than type B activities.
As shown in Fig. 8, time-series data of type A or B activities and time-series data of strategy executions were separately coded and independent. The data of type A or B activities (Sect. 4.3; P1) were created without using the data of strategy executions (Sect. 4.3; P2), and vice versa. This is one of our techniques to exclude the possibility that annotation labelers could be biased by their prior expectation of the relationships between learning states and strategy executions. As Fig. 8 shows, our analytics mined for hidden relationships between these independent time-series data of strategy executions and learning states.

Strategies Regulated to be Chained
Learners autonomously executed diverse intellectual activities in the world without any explicit intervention by the educators. We observed that type A activities were achieved by chaining qualitatively different activities so that learners could raise their level of understanding from the empirical level to the abstract level. An example is shown below.
(1) Hypothesis Construction Learners generally feel a sense of strangeness when they encounter an unknown phenomenon in the world or find a difference between newly inputted information and their pre-existing knowledge. Thus, we often found that construction of a hypothesis was first triggered by addressing a question raised by visiting a certain place. Then, the learners executed a strategy to accumulate empirical evidence of an individual phenomenon (e.g., shape and features of a plant in a field). This is because induction for discovering general rules and principles from particular facts is a basic method for hypothesis construction. They gave an answer with consideration of the background reason for why the phenomenon occurred. The answer occasionally led to a new question that triggered the next activity for concretizing or abstracting their hypothesis. When learners experienced difficulty in constructing a hypothesis, their way of selecting or observing empirical phenomena was reconsidered or changed.
(2) Hypothesis Verification The learners proposed a hypothesis that they constructed to theoretically explain a real-world phenomenon, set the hypothesis for the learner group to examine, discussed effective ways to investigate the hypothesis, and obtained observational results by paying close attention to a certain realworld phenomenon. Learners verbally confirmed the content of a hypothesis and its verification results, searched for a contradiction or an inconsistency between the hypothesis and the results, reconsidered and modified the hypothesis by getting an idea from the contradiction or the inconsistency, revised their method for hypothesis verification, and finally abstracted the learners' achievements as the conclusion.
Note that having a question was a trigger for constructing a hypothesis, and having a hypothesis was a trigger for verifying the hypothesis. During type A activities, diverse information is examined and interpreted in the world one piece after another. This is the self-regulation of real-world learning: a learner's current inquiry could clarify a next issue to be examined. Likewise, his/her subsequent inquiry of this issue could clarify a further issue to be examined. This self-regulation chained the execution of cognitive and behavioral strategies by assessing the learners' own achievements and their survey method and by proposing a new survey method.
On the other hand, during type B activities, cognitive and behavioral strategies were not well self-regulated to be chained for acquiring knowledge. For example, each activity of questioning, observation, discussion, hypothesis construction, and hypothesis verification was carried out separately and the results of these different activities were not integrated, or the activities were discontinued before assessing or improving these activities.

Frequency and Variety of Strategy Executions
The unique strategy codes (i.e., combination of operation codes and target codes) used in all experiments numbered 115 kinds and consisted of 32 operation codes and 25 target codes. Table 3 shows strategy executions during our experiments. Among all activities, 36.0% (5.04 hours) of the time was spent on type A activities for acquiring knowledge. Type B activities not for acquiring knowledge used 64.0% of the time (8.98 hours).
During type B activities, the frequency of strategy execution was low (1.51 times/min) and the variety of executed strategies was small (62 kinds of strategies). On the other hand, type A activities came from learners more actively executing a wider variety of strategies. Specifically, during type A activities, 95 kinds of strategies were executed, and the frequency of strategy execution was 2.46 times/min.
Regulation strategies (i.e., strategies to self-regulate learning for assessing, deciding, and changing how to learn in the world) included strategy codes ("propose"/"assess"/"direct"/"plan", "survey method") and strategy codes ("assess", "saturation state of information circulation"). By focusing on the frequency of executing these regulation strategies, we found that learning was frequently self-regulated during type A activities (once per 10.4 min), but learning was seldom self-regulated during type B activities (once per 49.0 min). This means that type A and B activities were performed under different conditions of selfregulation. Table 4 shows strategy codes frequently used during type A activities for acquiring knowledge in the world. The codes are sorted according to the number of times a strategy is executed. Among the 95 kinds of strategies, 21 kinds of strategies (22.1%) occupied 83.4% of the total number of strategy executions (743 executions). This means that a small fraction of strategies performed the dominant function to acquire knowledge in the world. This result is consistent with the Pareto principle, which states that roughly 80% of effects come from 20% of causes. According to our definition of strategy codes, this table includes cognitive strategies (#2, 4, 8, 12, 17, and 21), behavioral strategies (#1, 3, 5-7, 9-11, 14-16, and 18), and regulation strategies (#13, 19, and 20).

Dominant Strategies Frequently Used
The learning task tacitly requires the execution of some basic strategies (e.g., #1-4, 6, 7, 8, 10, 11, 16, and 18), and these strategies were also included in 12 kinds (19.4%) of dominant strategies that occupied 82.1% of strategy executions during type B activities (814 executions). However, the dominant strategies during type A activities included regulation strategies (#13, 19, and 20), but the dominant strategies during type B activities did not include any regulation strategies. This is an important difference of strategy execution between the two types of activities.
In the subsequent sections, we focus on Table 4 to examine strategy executions during type A activities, in detail.

Self-Directedness and Group Collaboration
All learners verbally explained the state of their observation target (#1 in Table 4), which illustrates that all learners participated in learning for acquiring knowledge in the world. Of learners, 86.7% tried to resolve their own question; this action illustrates that they spontaneously behaved based on their own question (#2). Learners did not ignore others' ideas or questions, and they maintained the state of learner-learner collaboration by approving, complementing, and assessing other's ideas (#3, 9, and 16), and by collaboratively resolving others' questions (#11).
Self-directed learning in a group is achieved not only autonomously by each learner but also in a collaboration state that the learners co-create. Our analytics enabled us to assess how individual-level self-directedness and grouplevel collaboration were balanced.

Open Learning System
Real-world learning is formed by each learner's cognitive function, learner-learner collaboration, and learnerenvironment interaction. Our analysis showed that during type A activities for acquiring knowledge, learners executed strategies to observe and operate not only a learner's own knowledge space (e.g., #2, 7, 14, and 17 in Table 4) but also other learners' knowledge spaces (e.g., #3, 9, 11, and 16) and the real world (e.g., #1, 4-6, 8, 10, 12, 15, 18, and 21). This result means that the information processing system of a real-world learner was not closed inside a learner's brain but functioned as an open system that was updated by external information not only of the co-learners' knowledge space but also of the world.

Constructing and Verifying a Hypothesis
As explained in Sect. 6.3.1, learners obtained observational results of a phenomenon, interpreted the results by considering the background ecology, and internally constructed the learners' own meaning. Through this process, clarifying the difference among phenomena (#12 in Table 4) was a cognitive strategy to find and examine the relationships among phenomena, and the strategy was executed to infer the background reason for a phenomenon and the nature of the observation target (#4 and 8). Learners identified, proposed, and explained their own ideas (#7, 14, and 17) through such an inference, summarized the observation result (#10), and constructed and examined a hypothesis. Among the strategies of hypothesis verification, an important strategy is self-assessing and critically examining the observation result (#21). However, this strategy, which requires a high level of cognitive processing, was executed by only 20% of the learners (Table 4).

Strategies to Self-Regulate Learning in the World
Difficult but important strategies were the regulation strategies, for example: • #13: strategy ("propose", "survey method") • #19: strategy ("assess", "survey method") • #20: strategy ("assess", "saturation state of information circulation") The above strategies were included among the top 21 dominant strategies during type A activities, and 40.0% of learners executed the strategy ("propose", "survey method"; #13) to propose how to learn in the world. However, only 20.0% of learners executed the strategy ("assess", "survey method"; #19) to critically self-assess the effectiveness of the survey methods that they used, although this strategy is the basis of re-examining, reconsidering, and improving how to learn in the world.
Another assessment was how learners executed the regulation strategy ("assess", "saturation state of information circulation"; #20). When a learner's information circulation system reaches the saturation state such that the knowledge space and the strategies are not updated, the possibility of drastic changes of behavior decreases (Sect. 3.3.2). This saturation state includes the state that learners did not find or did not create a new answer to a question, a new idea, a new hypothesis, new observational evidence, or a new conclusion through a certain way of real-world inquiry. To break out of this saturation state, a learner should have first selfassessed whether the state of their learning situation fell into such a saturation state. However, only 26.7% of learners were able to execute the strategy ("assess", "saturation state of information circulation"; #20) to make such an assessment for triggering a drastic change of how to learn in the world. Table 5 shows the group-level metrics of type A activities. The metrics from the left column to the right column are the following:

Learning Regulation and Intellectual Achievements
• Learning triggers.
-The total number of questions and hypotheses included in the chunks of meaning that each group member examined.
• The total number of times that each group member executed a strategy. • The number of unique strategy codes executed by each group and its ratio to the 95 total strategies. • The total number of times that regulation strategies were executed by each group and its ratio to the total number of strategy executions of the group. • The width of the space of knowledge that learners acquired in the world.
-The total number of nodes and arcs included in the chunks of meaning that each group member examined. Table 5 shows that the learners' knowledge space widened according to the number of questions and hypotheses for each group (G5, G4, G2, G1, and G3, in descending order). This result is consistent with Sect. 6.3.1, which shows that having a question or a hypothesis is the beginning point of scientific inquiries.
From the viewpoint of the width of knowledge acquisition, G5 was the best group, whose score (255) was 1.5 times that of the second group (172 of G4). G5 had slightly larger numbers of questions and hypotheses than did G4, but this difference was not very large. An interesting finding came from the difference of strategy regulation by groups G4 and G5. Group G4 was the top not only from the viewpoint of number of strategy executions (231) but also from the variety of unique strategies used in the group (44 kinds). However, this group did not often execute regulation strategies through learning (only 2.2% of strategy executions by this group). This result means that group G4 roughly executed various activities without properly self-regulating the activities. On the other hand, group G5 frequently executed regulation strategies (8.8% of the strategy executions by this group), although groups G1 -G4 did not (about 2% -3% of the total strategy executions of each group). The ratio of regulation strategies of group G5 (8.8%) was 4.0 times that of group G4 (2.2%).
In short, group G5 achieved the highest knowledge achievements among the five groups by having the largest number of questions and hypotheses as process triggers of learning, and by most frequently self-regulating the state of group learning.

Our Findings
Our analytics is an evidence-based approach for extracting findings on the mechanism of better understanding of the world by multimodally capturing, structuring, and coding time series of learning experiences in the world. Our results show that a clue to predict learning outcomes in the world is analyzing the quantity and frequency of learning strategies that a learner uses and self-regulates. Our findings are listed below. F3 Executing diverse strategies (F2) only was not sufficient to acquire the highest effects of learning. An important action to achieve high intellectual achievements was to self-assess the content and way of real-world learning and to orchestrate the procedure of learning to foster changeability of strategy execution. We considered that under insufficient self-regulation of learning, a learner is not able to adaptively switch strategies to search, examine, and discuss real-world information even if his/her old strategies are not effective. The following are examples of effective learning regulations: (1) revising an ineffective strategy by rethinking strategy execution results, (2) adjusting the achievement level by raising or reducing the abstraction level of understanding, and (3)  This research showed that the effect of acquiring knowledge about the world differentiates according to the self-directed process of executing and regulating a learner's internal strategies in his/her situation. We consider that the self-regulation of strategies in the world functions to (1) change the coordination of the actuator (i.e., information probing) and the sensor (i.e., information acquisition) of a learner's body, (2) expand the boundary of real-world information examined in his/her knowledge processing system, (3) temporally create instability of the system, and (4) establish a new regulation state of the system to raise the possibility of acquiring better intellectual achievements.

Toward Advanced Learning Support
As shown in our results, not all real-world learners or not all learner groups can effectively execute or self-regulate diverse strategies. Our assessment information of strategy executions clarifies the strengths and weaknesses of the style of real-world learning, and so it is useful for estimating the possibility of achieving discovery knowledge. Our assessment information can be used as a basis of on-site or off-site support for encouraging a learner to find a better way of learning regulation instead of persisting in using ineffective strategies.
Time-series data of type A and B activities are generated by supplementing the data of formative assessment of learning processes with the data of summative assessment of learning results. For finding relationships between processes and results for a problem-solving type of collaborative learning in a classroom, a similar definition can be made by matching time-series process data of a learner's activities (e.g., conversation data during problem solving) and summative data resulting from his/her activities (e.g., a learner's structured report of found solutions). It is also expected that our analytics can be a basis to clarify requirements for selfregulating the dynamics of a strategy-based control system in different settings of problem solving in the world (e.g., on-the-job training or collaborative work in an office).

Limitations
This paper discussed the initial results of our analytics for a specific experimental setting of real-world learning by adult learners in their twenties. Since this paper did not focus on real-world learning by elementary schoolchildren or the elderly, our future work should examine the applicability of our analytics to those people. However, we expect that our analytics and hypothesis verification task are applicable to young adults in a relatively close age group because they are expected to have a similar learning mechanism that this paper assumed.
Our experiments based on our designed task were done at a real-world field where environmental learning is generally practiced. Our analytics does not require a learning field to install special facilities such as sensor devices embedded in the field since the analytics works with the data of learners' wearable sensors, their activity maps, and experimenters' video cameras. Thus, we expect that our analytics and task can be applied to a similar environment where a learner can interactively find and examine various real-world phenomena to consider a symbiotic relationship among humans and the nature.
For confirming the above expectations, our future work should accumulate more evidence for the applicability of our analytics and its experimental design to a wider age range of learners (e.g., elementary schoolchildren, the elderly), a wide variety of learning fields (e.g., a virgin forest), other learning settings (e.g., learning subjects, tasks), or a larger data set. Moreover, analyses from the viewpoint of simultaneous strategy executions among collaborative learners were not covered by the current paper, and will be our future topic.
Our analysis was carried out to find relationships between short-term learning effects and strategy executions in the world. Analyzing how the process of strategy executions of a learner influences long-term learning effects (i.e., longitudinal changes not only of learning effects but also of ways of learning) is also future work for us.
The feature extraction of annotation data is automated by our database system, but the creation of annotation data is a manual operation using multimodal labelling software. We can reduce the time cost of strategy annotation if we code only the dominant strategies found during type A activities.
In the process to generate the time-series data of chunks of meaning, our system does not automatically find how the nodes and arcs of a learner's activity map correspond with his/her time-series interaction data, and vice versa. For reducing the analysis cost, our future work should consider a hybrid use of conversation or interaction analysis to automate or semiautomate the synchronization process that our current research manually matched nodes and arcs of activity maps with time-series interaction data.

Conclusions
Knowledge can be better acquired when a learner is involved in the social situation and context in which the knowledge is actually used. However, conventional learning analytics have not been developed to assess how a learner autonomously thinks and behaves in diverse real-world situations and self-directs inquiries to find and understand the semantics of real-world information.
Our past studies multidirectionally examined how the external real-world situation of a learner affects his/her inquiry behavior [14], [15]. The present research thus focused on the internal driving force of a learner. We modeled the dynamics that a learner's knowledge space is selforganized through his/her strategy-based interaction with the world. Then, we developed new analytics to observe, trace, and examine a learner's self-regulation process of his/her cognitive and behavioral strategies during realworld learning by integrating heterogeneous observation techniques of multimodal behavior sensing and structured representation of knowledge. We created multi-parameter expressions to structure time-series and multiplex executions of strategies. We also developed a database system to bridge the relationship between the assessment information of knowledge acquisition and the assessment information of strategy execution, and to compute and store metrics that show how knowledge is acquired by executing and selfregulating strategies in the world.
Our analysis showed that a clue to predict learning outcomes in the world is analyzing the quantity and frequency of learning strategies that a learner uses and self-regulates. We showed that a learner's cognitive and behavioral strategies were chained through real-world learning, for example, during the process of empirically or abstractly understanding the world. We found that the range of knowledge space was widened by executing various strategies driven by learners' questions and hypotheses. We also found that learners achieved high intellectual achievements in the world by self-assessing the content and way of real-world learning, and by self-regulating learning to fluctuate the dynamics of a strategy-based system of information circulations.
By integrating the viewpoints of learning science, behavior informatics, and embodied cognitive science, we studied the activation process of human intelligence emerging through behavior-based information processing in the world. The research will contribute toward the development of an AI-based learning support system that considers the internal strategies of people, enhances their intellectual activities, and creates the self-organizational dynamics of generating new knowledge.