{'fraction_non_alphanumeric': 0.029508640527004467, 'fraction_numerical': 0.004105942336326309, 'mean_word_length': 5.10074318744839, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Next Generation of Pedestrian and Connected Vehicle In-the-loop Research: A Digital Twin Simulation Framework

Digital Twin is an emerging technology that replicates real-world entities into a digital space. It has attracted increasing attention in the transportation field and many researchers are exploring its future applications in the development of Intelligent Transportation System (ITS) technologies. Connected vehicles (CVs) and pedestrians are among the major traffic participants in ITS. However, the usage of Digital Twin in research involving both CV and pedestrian remains largely unexplored. In this study, a Digital Twin framework for CV and pedestrian in-the-loop simulation is proposed. The proposed framework consists of the physical world, the digital world, and data transmission in between. The features for the entities (CV and pedestrian) that need digital twined are divided into external state and internal state, and the attributes in each state are described. We also demonstrate a sample architecture under the proposed Digital Twin framework, which is based on Carla-Sumo Co-simulation and Cave automatic virtual environment (CAVE). The proposed framework is expected to provide guidance to the future Digital Twin research, and the architecture we build can serve as the testbed for further research and development of ITS applications on CV and pedestrian.

accidents. Many ITS applications have been developed and investigated to enhance pedestrian safety in a CV environment. Due to safety concerns, most of the ITS technologies are developed and tested in a simulation environment, which is based on the assumption that the simulation environment is well-calibrated to represent real-world scenarios. However, this is not always true, and in simulation there is no interaction between simulated entities and real-world traffic participants, causing a reduction in the validity of testing. Besides, a relatively long cycle is needed from theoretical research to testing and evaluation, and finally to field implementation.
Digital Twin, as an emerging technology in the transportation field, provides unprecedented opportunities to support the development of ITS applications and address the above issues. By definition, a Digital Twin is a digital replica of a physical entity in the real world [2]. In transportation, it could be the technology that projects all the traffic participants into a digital road network in a real-time manner. The effects of ITS applications on CV or pedestrian can be investigated through simulation in the digital world. As the digital entities are projections of real traffic participants represents real-world traffic, the experiment results could be as convincing as field experiments.
In this work, we propose a Digital Twin framework to support ITS research on CV and pedestrian. A sample architecture under the framework is demonstrated, which incorporates a Sumo-Carla co-simulation platform for CV and a Cave automatic virtual environment (CAVE) for pedestrian.

B. Literature Review
There are numerous research focused on CV technology, including vehicle connectivity, cyber security, driving assistance, automotive control, and cooperative driving, etc. Most of the studies were conducted in a simulation environment as field test is expensive and time-consuming. Microscopic traffic simulation is widely used to study CV applications' benefits on safety [3]- [5], mobility [6]- [8], and energy consumption [9]- [11]. Virtual simulators, powered by game engine, have become increasingly popular for CV research, as they can simulate environment perception, vehicle dynamics, and control algorithms [12]. Driving simulator is also an effective tool to test CV technology including safety warning system [13], [14], traveler information [15], and cooperative driving [16], [17]. Recently, an increasing number of research adopted co-simulation technics for CV research. The cosimulation incorporates multiple simulation tools with different functionalities such as autonomous driving, traffic flow simulation, vehicular network and sensing technology.
The research on applying intelligent transportation system applications to enhance pedestrian safety heavily relies on simulation. Various collision warning or avoidance systems have been proposed that powered by vehicle-to-pedestrian communication [18], [19], and tested in micro-simulation by modeling the vehicle and pedestrian behavior. The advent of virtual reality (VR) technology brings opportunities to create a high-fidelity environment to involve real pedestrians for research and testing. VR headset is the most adopted equipment to display a virtual environment because of reasonable price and comprehensive technical support. However, the motion sickness issue is still a major problem that affects user experience [20]. Recently, some researchers used the Cave automatic virtual environment (CAVE) for pedestrian research [21]- [23]. CAVE is a VR environment where projectors are directed to between three and six of the walls of a room-sized cube [24]. It has much less motion sickness compared to headsets and allows people to walk in the cube space freely.
Pedestrian motion: the way of simulating walking in the virtual environment, is another hard task for pedestrian-in-theloop research. The traditional method to simulate pedestrian crossing behavior is to perform "shout test": participants shout their crossing decision when they intend to cross the street. Walking simulators such as VR treadmills are also used in some research [25]. When conducting experiments, the participant is wearing a VR goggles and walking in-place on a bowl-like treadmill, while tied with a band on his/her waist to prevent falling down. It could simulate unlimited walking distance, but the walking experience is inconsistent and still quite different from real-world walking. Recent studies have tried walking inhouse with a VR headset [26]. The issue with it is that the participant is easy to lose balance and prone to fall down. Locomotion, that pedestrian mark time, and use leg a band to detect pedestrian motion, is another method to simulate walking [25].
The above-mentioned efforts are CV or pedestrian research based on simulation in designed experiments. The emerging Digital Twin technology allows researchers to develop new ITS applications in a real-virtual hybrid environment. There are already some research that explored the applications of Digital Twin on connected vehicles and drivers. A Mobility Digital Twin framework is proposed in [27], which consists of physical and digital space of three components: human, vehicle, and traffic. A concept of Driver Digital Twin is introduced in [28] with the aim of bridging the gap between existing automated driving system and driver digitization. Also, deep learning for security in Digital Twin of cooperative ITS is presented in [29]. To the best of our knowledge, there is no Digital Twin framework that considers both CV and pedestrian in-the-loop. As they are among the two most important traffic participants on the road, it is necessary to build a Digital Twin architecture to facilitate the research and development of the next generation of ITS technology.

C. Organization of this Study
The organization of the remaining paragraphs is as follows: Section II introduces the proposed Digital Twin framework; Section III presents a sample architecture using Carla-Sumo cosimulation and CAVE; Section IV summarized this study.

II. DIGITAL TWIN FRAMEWORK
The proposed Digital Twin framework consists of the physical world, the digital world, and the data transmission between all the modules. The framework architecture is shown in Fig. 1. The elements inside the brown rectangle on the left are the digital replica of CV and pedestrians. The physical world is represented in the green rectangle on the right, in which the real drivers and pedestrians are involved. The blue rectangles are the attributes of CV and pedestrian that need to be digitally twined. The grey rectangle at the center contains the CV and pedestrian-related ITS technologies that will be applied.

A. Physical world
The Physical world is the space where the real traffic participants (vehicle, driver, pedestrian, signal, and traffic sensors, etc.) interact with the environment. The challenge of digital twinning associated with the physical world is the state detection of the physical entities and the extraction of key features that send to the digital world. The traffic participants could be divided into living entities and non-living entities, and they may need different digital twining technics.
For non-living entities, they only have external states, which are the physical states instead of psychological states, and their states can be fully captured and modeled through sensing or detection. For vehicles, the states include the kinematic features (e.g., position, speed, and acceleration) and ego characteristics (e.g., vehicle model and color). These features can be captured in real-time through onboard sensing (e.g., CAN BUS, GNSS) or external sensor detection (e.g., roadside cameras, radar). The digital twinning of traffic flow can also be challenging. The traditional method like Microwave Vehicle Detection Sensor (MVDS) provides reliable information about traffic flow in terms of volume, speed, and occupancy. However, the MVDS data is still at a macro-level and the micro-level, or vehicle group level information is not captured. The increase in vehicle connectivity makes it easier for precise information collection of a vehicle group. In a mixed traffic environment, which may be a long-term state in the foreseeable future, the connected vehicles could build up a Vehicular ad hoc network (VANET) that collectively sense and share the information of the surrounding traffic.
The digital twinning for living entities is more complex than non-living entities, as both external state and internal state need to be detected or modeled. The internal state refers to the psychological status of a traffic participant including intention, emotion, personality, etc. The internal state normally cannot be measured directly and relies on identification or prediction through external behaviors. For drivers, their external state could be a set of human characteristics (e.g., gender, age) and wellness indicators (e.g., distraction, fatigue), while the internal state includes driving intention and driving style preference. For pedestrians, motion is the external state such as walking/running/standing and body gestures, and it could be measured by wearable sensing suites and device, or through external sensing like video cameras. Pedestrians' internal state mainly includes crossing intention and desired path, and it also needs to be predicted.

B. Digital world
The digital world is a virtual environment that parallels the real-world, where all the traffic participants are the projection of real entities in a real-time manner. Under the proposed Digital Twin framework, the digital world is the place where the vehicle and pedestrian-related ITS applications are deployed and simulated.
For connected vehicle digital twinning, the primary task is the projection of the vehicle states from the real world to the digital world. The vehicle's external state, including localization, kinematic features, and vehicle characteristics, are the features that could be directly replicated in the digital world. If considering the CV and drivers as an entity, the vehicle could thus have internal state due to the drivers' psychological attributes. The CVs share the information about its ego states and the surrounding traffic, and thus they can conduct ITS applications such as safety warning, collaborative sensing, decision making, and vehicle control. Assisted by the computing power in the digital world, the effects of these ITS technologies can be simulated. Afterwards, the optimal driving strategy can be derived and feedback to the real-world CVs.
The digital pedestrian should also have the external state and internal state. The external state is relatively easy to be represented in the virtual space. It only requires the pedestrians' motion captured from the real-world to be digitally twined such as localization, moving speed and direction, and body gesture. The internal state, referring to pedestrian's psychological condition, needs to be modeled and predicted. As the behavior of the digital avatar is the same as real-world pedestrian, their internal state (e.g., crossing intention, desired path) could be modeled and predicted through probabilistic models or machine learning methods.
Once the digital avatar of vehicles and pedestrians are replicated in the digital space, ITS applications could be implemented virtually. For a traffic scenario with both CV and pedestrian presence, various ITS technologies can be applied. Based on the communication method, it can be vided into vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), vehicleto-infrastructure (V2I), or vehicle-to-everything (V2X) in general. In terms of functionality, these ITS technologies include driving cooperation, V2V or V2P collision warning, eco-driving guidance, etc. Normally, the digital world has stronger computing power (e.g., using cloud computing), and it can execute fast simulation and prediction to investigate the effects of the ITS technology and to generate optimal driving or crossing suggestions for CV and pedestrian respectively. Afterwards, the suggestions are forwarded to the real world entities and form a data transmission closed loop.

III. FRAMEWORK REALIZATION
In this section, a sample architecture is presented to demonstrate a possible realization of the proposed Digital Twin framework, as shown in Fig. 2. The architecture is built based on various simulation and physical environment that incorporates both CV and pedestrian.

A. Connected Vehicle Digital Twin
Since it is technically challenging and cost demanding to use a real vehicle for field tests, driving simulator could be a good alternative to represent a high-fidelity driving environment. Also, for experiments with safety concerns (e.g., a pedestrian crossing in an occlusion condition), simulator is the primary choice. In this sample architecture, a multi-driver co-simulation platform is adopted. The simulator is based on the cosimulation between Carla and Sumo, while multiple participants can drive simultaneously in the Carla virtual environment [30]. Fig. 3 demonstrates the Carla-Sumo cosimulation based on the location ExpresswayA from CitySim dataset [31]. Carla is an advanced autonomous driving simulator that is built on Unreal 4 game engine, and manual and autonomous driving can be conducted in Carla [12]. For driving simulator experiments, Carla is capable of rendering highfidelity graphic displays and simulate vehicle dynamics. Sumo is an open-source microscopic traffic simulator [32]. It is powered by vehicle behavior models including driving behavior models and route choice models. After fine model calibration, Sumo can generate traffic flow that represents the real traffic and vehicle behaviors [33].
To reproduce the real-world driving scene, high-fidelity 3D maps of the studying location are built using the road scene designer RoadRunner (3D maps are available at CitySim dataset homepage: https://github.com/ozheng1993/UCF-SST-CitySim-Dataset). The maps are built using GIS data to guarantee the quality, including arial images, elevation data, and point cloud data. By such means, the driving behaviors of real drivers at real locations can thus be collected. Benefiting from using the simulator, the vehicle state can be directly accessed through Carla's Python API, instead of being detected or sensed as in real world. The drivers' state can be captured through detecting or sensing the same methods as real-world incabin technologies such as eye-tracking and fatigue identification. Once the driver and vehicle data are obtained, they are uploaded to the cloud server (Redis in the architecture) and stored. In addition, the connected and automated vehicle (CAV) and human-driven vehicle (HDV) are also included in the sample architecture in Fig. 2 to be able to simulate a mixed traffic environment.

B. Pedestrian Digital Twin
Safety is always the primary concern for pedestrian-related studies. When the pedestrian is exposed to potential risk in an experiment, it must be held in simulation instead of field testing. Due to this reason, the objective of pedestrian digital twinning is to create an immersive and close-to-real simulation environment for pedestrians.
In this architecture, we used a Cave automatic virtual environment (CAVE) to serve as the testbed for pedestrians. The CAVE equipment used in this study is called "VDen", which is a lightweight, portable version of CAVE that can be run by a single PC, with native SDK support for Unity game Fig. 2. Sample architecture of CV and pedestrian in-the-loop digital twin framework engine. The system projects 3D videos on 4 screens (left, front, right and bottom), and provides a 3*3-meter movable physical space. The users are wearing a 3D glasses and a tracker for localization in the CAVE. Also, an additional tracker is used to track the eye position in order to get the correct perspective of view.
Compared to the other VR device Head Mounted Displays (HMDs), CAVE has several evident advantages, especially for pedestrian simulation. First, it feels more natural and comfortable for the user only to wear a pair of lightweight 3D glasses and a tracker instead of a heavy headset. Second, CAVE brings less cybersickness because CAVE displays multiple images on the walls simultaneously instead of generating a new image when the user is moving his head. In addition, when experiencing VR in CAVE, users could naturally be aware of the physical presence of their own and each other's entire bodies, which is crucial for multi-user and social VR scenarios. Users can freely communicate with one another as if they were in the real world. In HMD setups, this is very challenging to achieve.
In order to replicate the pedestrian's pose in the digital world, the real body's keypoints are captured and matched to the digital avatar. Keypoint detection is carried out by Google MediaPipe Posekeypoint detector using pre-trained weights from BlazePose GHUM 3D. Unlike most state-of-the-art approaches, which depend on a robust server environment for inference, the MediaPipe achieves real-time pose detection on the 2022 MacBook pro with M1 Pro Max GPU. The MediaPipe outputs are uploaded to a Redis Pub/Sub server in JSON format, which includes 33 human key points' names and the bone transform in 3D space. This JSON file will be pushed into CARLA Client "WalkerBoneControl" class every tick to modify the CARLA walker agent's skeleton. To simulate the pedestrian's walking movement, locomotion method is used. It allows users to move forward or back by lifting their legs to simulate a real walking pose. The implementation is based on binding two additional lighthouse trackers to the user's legs and monitoring the range of movement of the sensors in the vertical direction to trigger a movement event. The horizontal orientation of the movement of the sensor also determines the direction of motion. The methods of pedestrian pose matching and achieving locomotion are shown in Fig. 4.

C. Closed-loop Data Transmission
Since the presented architecture uses two major platforms: Carla-Sumo (vehicles) and Unity (pedestrian), the two digital space needs to be synchronized. Fig. 5 shows the data transmission structure between the two platforms. The data transmitted from Carla-Sumo to Unity includes the vehicle state and driver state; while the data sent from Unity/CAVE to Carla is the pedestrian's location, motion and pose.

Fig. 5. Data transmission between each module
The idea of Digital Twin is to form a closed loop from physical entities to digital avatars and then generate feedback for physical entities. Hence, the proposed architecture follows this concept and keeps a closed-loop data transmission structure. When conducting an experiment, the driver and vehicle information is read from the driving simulator and is uploaded to Redis server. The pedestrians' location, motion and gestures are also tracked in the CAVE and are uploaded to the server simultaneously. At each simulation step, the system accesses the data server to download the vehicle or pedestrian data, and projects the avatar into the digital space. Afterwards, ITS technologies are implemented in the virtual world. For example, V2P collision warning system predicts crashing likelihood and triggers warning messages. Finally, suggestions for CV drivers or pedestrians are generated and feedback to the physical space (e.g., display warning messages on vehicle onboard unit).

IV. CONCLUSION
A Digital Twin framework that incorporates CV and pedestrian is proposed. The framework consists of the physical world, the digital world, and the data transmission between realdigital spaces. The attributes of the CV and pedestrian, including their external state and internal state are specified. A sample architecture of the framework is also built. The architecture combines two platforms: Carla-Sumo and Unity for the digital twinning of CV and pedestrian, respectively. The Carla-Sumo platform serves as the driving simulator to simulate a CV environment, while a CAVE powered by Unity is used to provide the simulation environment for the pedestrian. The data transmission between different platforms is defined and the method of forming a closed-loop data transmission structure is discussed. The proposed Digital Twin framework is expected to serve as a powerful testbed for future research on ITS technology to enhance CV and pedestrian safety and mobility. The future work of this research is to expand the framework to incorporate other traffic participants such as cyclists and motorists, and conduct a case study using the presented Digital Twin architecture.