{'fraction_non_alphanumeric': 0.02934816187828236, 'fraction_numerical': 0.007167130058696324, 'mean_word_length': 4.974898486526394, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tactical Decision Making for Emergency Vehicles based on a Combinational Learning Method

Increasing response time of emergency vehicles (EVs) could lead to an immensurable loss of property and life. On this account, tactical decision making for EV's microscopic control remains an indispensable issue to be improved. Our approach verifies that deep reinforcement learning could complement rule-based methods in generalization. It reveals that deterministic avoidance strategy for common vehicles at a low speed benefits EVs a lot, nevertheless, when at a high velocity, DQN breaks the deadlock of reduced safe distance and brings boldness to EVs in lane changing. Besides, a novel DQN method with speed-adaptive compact state space (SC-DQN) is put forward to fit in EVs' high-speed feature and generalize in various road topologies. All Above is implemented in SUMO emulator, where common vehicles are modeled rule-based whereas EVs are intelligently controlled.


I. INTRODUCTION
Emergency Technology Company RapidSOS mentioned a set of data:"In medical emergencies such as cardiac arrest, every one-minute delay in response time causes mortality rate to increase by 1% and imposes additional $1542 in hospital costs, leading to 7 billion dollars increase in healthcare expenditure per year only in USA. Similarly, one minute of reduced response time leads to a decrease in healthcare costs by 326,000 baht ($10,190) in Thailand." [1] from which we consider cutting down response time of EVs an unavoidable issue.
Most researches focus on the route optimization and traffic signal preemption, which address the problem from a macroscopic perspective. However, real-time traffic data are not taken full advantages of. Besides, hardly has the impact on normal traffic been considered. Moreover, these deterministic strategies are harder to generalize to various traffic scenarios than policies acquired by Deep Reinforcement Learning. Even so, It is not to say that we could totally abandon rule-based policies since they outperform other methods in outstanding stability. Consequently, we combine rule-based strategy and intelligent control together. For common vehicles, we devise a priority zone where they have to give way to EVs if under safe constraints. For EVs, DQN method enables them to change lane actively on their own when common vehicles could not execute the avoidance strategy on account of minimum longitudinal gap requirement.
Our departure point is to shorten the response time of emergency vehicles in a straightway environment, where limited research has been done. We tend to propose a time- saving, real-time and data-efficient tactical decision-making method for emergency vehicles, with rule-based common vehicles that adhere to avoidance strategy surrounding by.
The remainder of this paper is organized as follows: In Section II we give a brief overview of the research on different 1 aspects to cut down the response time of EVs. A detailed description about how we model our method and how it performs follows in Section III and Section IV respectively. Last but not the least, we draw our conclusion and put forward considerable future work in Section V.

II. RELATED WORK
Prior researches on shortening EV's response time on straight way mostly concentrate on intelligent sensing, traffic signal control, route optimization and deterministic decisionmaking strategies.

A. Intelligent Perception
Plenty of researchers throw themselves into enhancing the precision and rate of position and velocity perception, like new GIS-based navigation approach [2] and RFID and GPS based automatic lane clearance system [3].
An Arduino-based intelligent traffic signal controller prototype that automatically detect EVs and length of queue so that it can communicate with adjacent intersection to clear the traffic flow for EVs is designed in [4], using wireless sensor network to interact with their algorithm, and it can be easily integrated into existing traffic signal system.

B. Traffic Signal Control
Tactical control at signalized intersections can assist EVs to pass with a priority, which is also known as Routepreemption.
The most commonly used method is controlling traffic signals at the intersection where EV is upcoming, to halt lower-prioritised traffic [5] or disperse the occluded traffic in advance [6], and eventually reduce the delay of EVs. In [7], a method that takes coordination of intersections into consideration is mentioned, with less negative impact on the normal traffic streams. 1

C. Route Optimization
It's very significant to dispatch EVs in optimal routes to keep away from traffic congestion. Dynamic route guidance system [8] and vehicles evacuation algorithm [9] based on Dijkstra have been devised, and other path optimization algorithms like A* and Ant Colony Algorithm are also summarized in an overview [10]. In [11], A two-stage shortest path algorithm composed of K-paths algorithm and shuffled frog leaping algorithm is proposed. [12] demonstrates emergency vehicle route guidance by utilizing additional road features.
All above is about optimizing the response time of EV in a macroscopic perspective, so real-time data is usually not leveraged to the utmost like what microscopic control tactic does.

D. Deterministic Decision-making Policy
Limited researches have shown concern to microscopic control of EVs. A series of EV traversal model are mentioned in [13], incorporating Fixed Lane Strategy, Best Lane Strategy, Lane Change Algorithm and Risk Assessment Algorithm. From Fixed Lane Strategy, they construct a priority zone and common vehicles have to comply with an avoiding policy if they block the way of EV within the priority distance.
That is not to say deterministic policies cannot perform well in certain scenarios, but they do not have much advantages in exploration for lane changes and adaptation for changeable road topologies against Deep Reinforcement Learning methods.
In conclusion, most technologies and strategies for EVs are macroscopic and traffic-signal-based while straightwaydomain simulations are few and far between, let alone intelligent control for EVs on the straight. Response time spent on straight way should not be neglected, especially on highway where the distance between adjacent junctions also contributes crucially to the total travel time. Deterministic avoiding strategy could hardly generalize to complex traffic scenarios, but they could guarantee the stability of EVs' performance. Therefore, we fit Deep Reinforcement Learning method in semantic environment for EVs with common vehicles following avoidance strategy, trying to acquire the optimal method in this combinational circumstance.

III. PROBLEM MODELING
Our framework is implemented in SUMO [14] traffic emulator, with RL method for EVs and rule-based avoidance strategy for common vehicles. First and foremost, we set up our speed-adaptive compact state space and discrete action space of RL agent. Afterwards, EV-specific reward functions are devised. Finally, introduction about how we initialize the input of environment and how we structure the Speed-adaptive Compact Deep Q-Network (SC-DQN: an ad-hoc DQN method for autonomous EV) is going to be discussed. A. Speed-adaptive Compact State Space Other previous work about compact state space [15] has referred to a semantic method to structure a relational grid about neighboring vehicles. To make the learning process more data-efficient, we refine the state space in a more compact way (Fig.2), which originally contains 6 neighbors of EV (ego car). Moreover, this kind of compact structure could fit in variant road topologies, including but not limited to curve, ramp and merge. We extract three features of each neighbor. Presence is whether its neighbor exists. Velocity and Distance of this neighbor is zero when Presence is assigned 'True'. It is noteworthy that Fig.2 only demonstrates the directional relation of EV and its neighbors, but does not depict that of EVs' neighbors, i.e. Left Leader may not be the Left Follower of Leader Car (Fig.2).
Intuitively, auxiliary speed-adaptive features should be additive to the EV compact observation for more forwardlooking view if velocity of EV is more than half of its maximum velocity. These incorporates leader of leader car, left leader and right leader, called speed-adaptive observation in Fig.1.
Speed-adaptive observation is concatenated with normal compact observation, together with the execution effect of avoiding strategy (mentioned in [13], which denotes that leader car ought to avoid the EV within the priority distance). Whether the leader car execute the avoiding strategy successfully is a key feedback feature that plays an important role in our model, and also a crucial innovation point for combining DRL algorithm with deterministic strategy organically, whose simulation result will be presented and analyzed in the Section Ⅳ. R make a right lane-change;

C. Reward Designing
To evaluate the reasonability of chosen actions and guide the training agent into a convergence state, reward designing for autonomous driving basically focuses on three aspects as follows.
 Safety Unlike the ordinary way that gives a negative reward when collision occurs, we implement continuous reward function proportional to survival distance, serving as a real-time stimulus: Efficiency reward is also proportional to the current velocity, following the continuous reward designing concept as what we do in 'Safety'.
RL-based autonomous vehicles have always been denounced for their undesirable lane-changing frequency. Hence, we consider lane-changing at EVs' maximum velocity to be a meaningless behavior because lane-changing is intuitively an action for pursuing higher velocity, but it could not obtain higher velocity by lane-changing at that time. Thus, (= −1) will be given if changing lane at its maximum velocity, to smooth the trajectory and alleviate the impact of frequent lane-changing to normal traffic, and surely assist learning process more convergent.

D. Mask of Prior Knowledge
A multitude of researchers adopt prior knowledge in their RL environments, addressing the problem that RL methods have a natural instability against some proven policies. Additionally, as a preprocessing, prior knowledge initializes the agent at the very beginning, so less episodes are spent on learning hard constraints and basic rules.
Consequently, we set up two masks to filter the output action out. One is Rule Mask, to check whether the action that maximizes Q value complies with the traffic rules and road topology constraints. For instance, if EV takes a right lane change on the rightmost lane or accelerates to a velocity that surpasses the maximum velocity, the invalid action will be rejected and the suboptimal action according to Q values will be selected. This is similar with the Q-masking technique in [16]. Another is Safety Mask. RL-based methods are wellknown for lane-changing initiative, whereas safety issue is always the drawback that hardly converges. In SUMO, we initialize the EVs with the parameter laneChangeMode=512 (disables all autonomous changing but still handle safety checks in the simulation, either one of the modes 256 (collision avoidance) or 512 (collision avoidance and safety-gap enforcement) may be used. [17]). Thus, the lane change mode blocks the safety-challenging actions and chooses one from its experience. If TraCI, the configurable Python interface for SUMO, could not execute what SC-DQN commands, this step will not be learnt by our network.

E. Network Architecture
Deep Q-Network [18] is one of the most practical RL-based network architectures that has been implemented in autonomous driving. In our model, transformed DQN utilizes speed-adaptive state space and the execution of avoiding strategy as its current observation(O ). Our target network contains an input layer of 32 neurons, two hidden layers with 20 and 10 neurons respectively, and the output layer with 5 neurons for each action.
With the probability of exploration rate ϵ (initialized as 0.9 and anneals 4 × 10 − each step until reaching 0.1), Action(A ) is selected by Q-values, output of the target network, otherwise, action will be chosen randomly as an exploration. Every 5000 iterations, parameters of target network are assigned to those of evaluate network, a network with same structure, which is a crucial step leading to the final convergence.
Successor state O + and reward follow the interaction between and the environment. ( , , , + ) is a transition and it will be stored in the memory pool with the capacity of 2000 transitions, if is determined by RL-agent instead of prior knowledge.
Hereafter, our SC-DQN(Speed-adaptive Compact Deep Q-Network) will be trained with experience replay technique [19] and update the mini-batch with 32 transitions every learning step. Besides, Adam Optimizer is applied with the learning rate decaying exponentially. Last but not the least, to prevent the phenomenon of gradient explosion, Deep Q-Network learns its parameters to minimize Huber Loss ( ( ) 5) instead of Mean-Square Loss: = + + ( + , + ; − ) (4)

IV. EVALUATION
On a 2000-meter straight way for simulation test, we implement our simulation with random normal traffic flow and parameter-setup EVs training car by car. Table 1   From the preliminary work came a suite of discoveries leading to the drawbacks of Avoiding Strategy in [13] and SC-DQN method respectively. It reveals that Avoidance Strategy for CVs at a low velocity benefits EVs a lot, however, it is proved deadlock-prone when at a high velocity because CVs are more likely to address their own safety than EVs' efficiency. Intuitively, SC-DQN is no more advantages if leading CVs are bound to avoid for EVs while it could mitigate the conflict between the concern for reduced safe distance and the desire for higher speed, so it outperforms in high-speed traffic flow.
Based on the above discussion, we compared baseline, SC-DQN and our method in our simulation. Our baseline is a brain with prior knowledge based on lane-keeping strategy in SUMO. SC-DQN is speed-adaptive compact DQN, an ad-hoc DQN method that could fit well in environment with EVs and different road topologies. Our method is an organic connection of SC-DQN for EVs and Avoiding Strategy for CVs in which CVs' execution of Avoiding Strategy serves as a feedback in the state space of RL agent.
As shown in Fig.4, SC-DQN and SC-DQN with avoiding strategy feedback both converge to better performance than the baseline. From Table.3, we evaluate these algorithms in three main aspects: efficiency, safety and smoothness. Our method facilitates the agent to pursue higher velocity and lower collision rate, but high lane-changing rate is incomparable to that of human drivers.   Figure 5. EVs learn to take action when CVs disobey avoiding strategy Fig.6 denotes that EVs gradually learn how to break the deadlock when CVs could not comply with the deterministic avoiding strategy during the training process: instead of waiting the leader car to avoid, they could take actions like lane-changing so quickly that the blue bars become finer and thinner.

V. CONCLUSION
We have devised a framework about how autonomous EVs in self-driving traffic flow make tactical decisions based on DQN with a speed-adaptive and data-efficient observation. For the first time, the idea of combining deterministic avoiding strategy with RL algorithms organically is proposed, which allows the execution of avoiding strategy flowing back to feed the RL observation. It bridges the gap between deterministic policies and RL-based algorithms in a complementary way. But it is not to say that we have solved the instability of DQN, because we are still exploring how to ameliorate the trajectory smoothness where our auto-EVs underperform, by methods like intrinsic reward [20] [21], Model-based RL [22] and Attention-based Hierarchical RL [23]. Additionally, congestions are neglected in our modeling hypothesis and we try to find out a Sublane-changing Model based on microscopic simulation framework [24][25] [26] in the future work.