{'fraction_non_alphanumeric': 0.029934384806618575, 'fraction_numerical': 0.005624159432693484, 'mean_word_length': 5.100062150403978, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prioritized Experience-based Reinforcement Learning with Human Guidance: Methodology and Application to Autonomous Driving

Reinforcement learning requires skillful definition and remarkable computational efforts to solve optimization and control problems, which could impair its prospect. Introducing human guidance into reinforcement learning is a promising way to improve learning performance. In this paper, a comprehensive human guidance-based reinforcement learning framework is established. A novel prioritized experience replay mechanism that adapts to human guidance in the reinforcement learning process is proposed to boost the efficiency and performance of the reinforcement learning algorithm. To relieve the heavy workload on human participants, a behavior model is established based on an incremental online learning method to mimic human actions. We design two challenging autonomous driving tasks for evaluating the proposed algorithm. Experiments are conducted to access the training and testing performance and learning mechanism of the proposed algorithm. Comparative results against the state-of-the-arts suggest the advantages of our algorithm in terms of learning efficiency, performance, and robustness.


I. INTRODUCTION
EINFORCEMENT learning (RL) has substantially contributed to numerous fields [1][2][3][4] by solving optimization and control problems. As a branch of machine learning methods, RL improves the capability of controlling agents in black-box environments through the exploratory trial-and-error principle [5]. Recent popular RL algorithms, e.g., rainbow deep Q-learning [6], proximal policy optimization (PPO) [7], and soft actor-critic (SAC) [8], have shown greatability in handling high-dimensional environmental representation and generalization, thanks to the introduction of deep neural networks. Albeit RL can achieve good performance in complex tasks, its drawback emerges that their interactions with the environment are very inefficient [9]. Thus, using RL to solve a problem needs skillful definitions and settings and consumes remarkable computational resources [10].
Combining human guidance with RL can be a promising way to mitigate the above drawback [11]. There are three categories of human guidance that have been integrated into RL. 1) Human feedback. The human expert's prior knowledge about the task could be used to qualitatively or quantitatively score the RL behaviors [12]. In this manner, people successfully guide an RL-based robotic vehicle to run through a maze [13]. 2) Human intervention. Intervention is a more direct manifestation of human knowledge than merely giving feedback. RL agent is devised to reduce their confidence in its adopted action if intervention occurs [14], which is generally achieved through the reward shaping technique [15]. Human intervention has been validated to improve RL performance in playing Atari games [16] and manipulating drones [17]. 3) Human demonstration. Enabling the RL agent to learn from demonstration is another intuitive way to enhance performance [18]. The objective functions are generally reshaped compatible with supervised learning in this context [19]. Dexterous manipulations of a high degree-of-freedom robotic arm [20][21][22] and human-level game operation [14] have been achieved through combining RL algorithm and human demonstration.
Despite introducing human guidance, RL needs to process numerous data from its self-explorations. The above-discussed approaches do not particularly optimize the utilization of human guidance data, consequently, they still need great human workloads to avoid submersion of guidance in exploratory data. Additionally, human guidance, which is variant to proficiency, mental and physical status of participants, should not be equally treated since some low-quality guidance can even impair the RL performance.
We propose a priority-based experience replay method on human guidance and put forward the associated human guidance-based RL algorithm to bridge the abovementioned gap. Our approach is off-policy, which leverages the experience replay mechanism [23] to maximize the utilization efficiency of self-exploratory data. The proposed priority replay mechanism can further improve the utilization efficiency of human guidance data by quantifying their values and weighing their utilized probability, which ultimately augments the RL performance. As a result, the efficiency can be improved by over seven times under the adopted task. The schematic diagram of our algorithm is depicted in Fig. 1. To evaluate the training and testing performance of our proposed method, we design two challenging autonomous driving scenarios. The experimental results suggest the advance of the proposed algorithm compared to state-of-the-art baselines in learning efficiency, practical performance, and robustness.
The contribution of this paper can be summarized into four aspects. 1) we propose a novel prioritized experience utilization mechanism regarding human guidance in the RL process and an associated RL algorithm to improve performance. 2) we establish a comprehensive and holistic framework of human guidance-based RL by integrating human-RL action switch system, human-related objective function shaping technique, human-demonstration replay method, and human-intervention avoidance mechanism. 3) we put forward an online incremental learning model to mimic human behavior in the human-in-theloop RL process which relieves the human workload. 4) we validate the superior performance of the proposed algorithm in solving challenging autonomous driving tasks.
The remainder is organized as follows: Preliminaries for the proposed algorithm is introduced in Section II, the core content, human guidance-based reinforcement learning is elaborated in Section III, the human behavior model substituting real human participant is established in Section IV. Section V presents the problem formulation for the adopted autonomous driving tasks, Section VI provides the experimental results, and the conclusion is drawn in Section VII. Fig. 1. The framework of the proposed human-guidance-based reinforcement learning algorithm. The RL algorithm in this paper is shaped in multiple aspects to adapt to human guidance. In the proposed human guidancebased priority experience replay mechanism, TDQA represents the proposed priority calculation scheme, and the number 1-4 indicates the flow sequence of data. The dotted line of the action signal represents that the framework allows intermittent human-in-the-loop guidance.

II. PRELIMINARIES
In this section, we first introduce the notation and concept of off-policy actor-critic RL, then we illustrate the prioritized experience replay mechanism. All three parts in this section are the base for the proposed prioritized human guidance-based RL algorithm.

A. Notation
We consider a standard reinforcement learning setting where an RL agent interacts with the controlled environment. Such an interaction can be formulated as a discrete-time Markov decision process (MDP), defined by the tuple ( , , , ). The state-space consists of continuous state variables and the action space constitutes continuous action variables . (⋅ | , ): × → is a reward function mapping the stateaction pair ( , ) to a deterministic reward value . The environment dynamics generates state transition probability (⋅ | , ): × → ( ′ ) mapping the state-action pair ( , ) to the probability distribution over the next state ′ .
At each time step , the agent observes the state ∈ and sends the action ∈ to the environment, receiving the feedbacks of a scalar reward and next state +1 . The agent's behavior is determined by a policy ( | ): → ( ), which maps a state to the probability distribution over candidate actions. We utilize to represent the state-action distribution induced by the policy .

B. Off-policy actor-critic architecture
The goal of reinforcement learning is to optimize the policy which maximizes the expected value over the environment dynamics [5]. In the standard RL setting, the objective value under a policy is calculated from the accumulated future rewards as: where ∈ (0,1) is the discount factor.
A Bellman value function (also called critic) can be established to estimate in a bootstrapping way. This value function is usually called . Under an arbitrary policy , is defined as: Then the policy function (also called actor) can be obtained concerning maximized , represented as: Above two components concrete the actor-critic RL framework. In the practical implementations, the Bellman value function pursues the evaluation regarding only the optimal policy ⋆ , regardless of the policy executing the interaction. Therefore, RL decouples the policy evaluation process and the policy's behavior, which makes the agent update in an offpolicy manner.
We use neural networks as the function approximator to formulate the actor and critic, the objectives can then be reached through the loss functions. Specifically, the loss function of the critic ℒ can be expressed as: where (⋅; ) represents the parameterized critic function and represents the parameters of the critic network, (⋅; ) represents the parameterized actor function and represents the parameters of the actor network.
And the loss function of the actor ℒ can be expressed as: By minimizing the value of the loss functions, the critic and actor can reach the objectives in Eqs. (2) and (3), respectively.

C. Prioritized experience replay mechanism
The experience replay mechanism establishes an experience buffer to store the data at each interaction. Accordingly, the RL agent can retrieve data generated by previous policies from the buffer for policy evaluation and improvement.
Given an arbitrary time step , the interaction between the RL agent and the environment generates a transition tuple, which is stored into the experience replay buffer as: Conventionally, the experience in the buffer is subject to uniform distribution , and we obtain batched experience from the buffer for RL training using uniform random sampling. The loss functions of the actor and critic can then be expressed as: where is the size of the stored experience.
Instead, the experience is subjected to a nonuniform distribution ℐ in the prioritized experience replay mechanism [24]. Specifically, the probability mass function of the experience distribution ℐ~ℐ can be expressed as: where ∈ [0,1] is the scaling coefficient, represents the priority of each tuple , which is determined by the temporal difference (TD) error and expressed as: where ∈ ℝ + is a small positive constant to guarantee the probability larger than zero.
A larger TD error indicates an experience worth learning to a higher extent. Thus, the TD error-based prioritized experience replay mechanism can improve the RL training efficiency.
With the prioritized experience replay mechanism, the loss functions in (7) and (8) can be respectively expressed as:  In this section, we first summarize the human behaviors in the RL training process which can be leveraged in the algorithm design. Based on that, we establish an actor-critic framework adapting to human guidance. Then, two modules are proposed to further improve RL in the context of human guidance: a novel prioritized experience replay mechanism concerning human demonstration, and a reward shaping technique concerning human intervention. Finally, a holistic human-in-the-loop RL algorithm is instantiated using the above components.

A. Human guidance behavior in the RL training process
We define two useful human guidance behaviors in the RL training process: intervention, and demonstration.
Intervention. Human participants recognize RL interaction scenes and identify whether a guidance behavior should be conducted based on their priori knowledge and reasoning abilities. If human participants decide to intervene, they can manipulate the equipment to get the control authority (partially or totally) from the RL agent. The intervention generally happens when the RL agent conducts catastrophic actions or is stuck in local optima traps. Thus, RL could learn to avoid unfavorable situations from the intervention.
Demonstration. Human participants perform their actions when an intervention event happens, which generates the corresponded reward signal and next-step state. The generated transition tuple can be seen as a piece of demonstration data since it is induced by human policy instead of the RL's behavior policy. RL algorithm could learn human behavior from the demonstration.
State-of-the-art human-guidance-based RL algorithms have been integrating learning from intervention (LfI) [16], and learning from demonstration (LfD) [25]. In this paper, both LfI and LfD will be employed in the proposed architecture. Specifically, LfI based on the reward shaping technique is utilized in the reward function definition, while LfD plays its role in the underlying principles of the algorithm.

B. Human-guidance-based actor-critic framework
In this subsection, we elaborate on the interaction mechanism and learning objective of the proposed human-guidance-based actor-critic RL algorithm.
First, we focus on the interaction mechanism. In the standard interaction between the RL agent and the environment, RL's behavior policy will output actions to explore the environment. Given an off-policy actor-critic RL, above process is shown as: where ∈ ℝ dim( ) is a training-dependent variable that scales the exploration noise, ⨀ represents the Hadamard product and ~(0, dim( ) ).
We give full authority to human participants whenever they decided to intervene in the training loop of RL. Thus, the eventual action is filtered by a mask as: where represents the action from the human participant's policy, ∈ ℝ dim( ) is an intervention discriminator: it is an identity matrix when human intervention happening and a zero matrix in the non-intervened step.
The interaction transition tuple will be recorded and stored into the experience replay buffer once the action is sent to the environment. In particular, actions from the human policy and the RL policy are stored in the same buffer. For this context, the new transition tuple is defined to discriminate human demonstrations from normal RL experiences as: Then, we focus on the learning objective. Given a batch of transition tuples with batch size , there could exist data 1 from the RL policy and 2= − 1 from the human policy. The critic's objective should consider both experiences, therefore, we devise the loss function of the critic network as: Given the data from the human policy, the actor should learn from these demonstrations in addition to maximizing the critic's value. Hence, we devise the loss function of the actor network as: where is a manually determined constant that weighs the importance of learning-from-demonstration. It should be noted that the mean squared error (MSE) losses involved in the above formulas are for exemplified calculation, meaning that they can be alternated by any loss functions.

C. Prioritized human-demonstration replay mechanism
In this subsection, we put forward a novel prioritized experience replay (PER) mechanism for human demonstration.
Human demonstrations are generally more critical than most exploration from RL's behavior policy due to priori knowledge and reasoning ability. Thus, a more effective method is needed to weigh human demonstrations among the buffer. We propose an advantage-based metric instead of the TD-error of the conventional PER to establish the prioritized humandemonstration replay mechanism.
First, we define an advantage measure regarding the human demonstration against the RL's behavior policy. Since the critic can evaluate the policy, we calculate the difference between the value of the human action and that of the RL action. Given a human-demonstration transition tuple ( , = , , +1 ), the priority level is defined as: where exp is the exponential function to guarantee the nonnegative advantage value.
We call the last term of the above equation as the Qadvantage term, which evaluates to what extent should a specific human-demonstration tuple be retrieved except the TDerror metric. Through the RL training process, the RL agent's ability varies and the priority level of one human-demonstration tuple changes accordingly, which gives rise to a dynamic priority mechanism. We abbreviate Q-advantage as QA and call the above mechanism TDQA to illustrate it combines two metrics as the measurement of human guidance. The QA term is removed for non-demonstration tuples when calculating the above equation, therefore, the priority levels of nondemonstration data are aligned with those in the conventional PER mechanism.
In this manner, the experience in the buffer ℬ subjects to a distribution ℐ ′ , and the probability mass function of the experience distribution ℐ ′~ℐ ′ can be expressed as: The priority mechanism introduces the bias to the estimation of the expectation of the value function since it changes the experience distribution in the buffer. Biased value network could have little impact on the RL asymptotic performance, yet it may affect the stability and robustness of the mature policy in some situations. As an optional operation, we can anneal the bias by introducing the importance-sampling weight to the loss function of the value network. The importance-sampling weight of a transition is calculated as: where ∈ [0,1] is a coefficient: the fully non-uniform sampling happens if = 1 , and fully uniform sampling happens if = 0. will gradually decrease to zero along with the training process.
The importance-sampling weight can be added to the loss function of the value network, expressed as:

D. Human-intervention-based reward shaping mechanism
In this subsection, we introduce the human-interventionbased reward shaping technique. Consider human participants' behavior in the RL training loop: they can intervene in the process to acquire control authority and substitute RL actions with their actions. Due to the sustainedly repeated training episodes and unmatured RL policy, it is tedious for human participants to continually give demonstrations in the training process, whereby intermittent intervention becomes a more feasible solution. In this context, the human participants would only intervene in those crucial scenes (catastrophic behaviors or trapped into the local optima) to save the RL agent and prolong the training episode. Thus, we can shape the vanilla reward function with an additional penalized term: where rt shape is the reward after shaping, rpen is a scalar that weighs the intervention penalty. The above reward scheme prevents the RL from being stuck in those human-intervention states. However, it triggers the penalty only at the first timestep of one human intervention event. The rationale behind this is that demonstrations of human participants would inertially last for a while once participants acquire control authorities, whereas only the initial scene can be confirmed as the crucial one.

E. Prioritized human-in-the-loop RL algorithm
In this subsection, we integrate all the components and propose an RL algorithm considering human guidance, which is named Prioritized Human-In-the-Loop (PHIL) RL.
Specifically, we obtain the holistic human-in-the-loop RL configuration through equipping the human-guidance-based actor-critic framework with prioritized human-demonstration replay and intervention-based reward shaping mechanisms. We instantiate the PHIL algorithm based on one of the state-of-theart off-policy RLs, i.e., twin delayed deep deterministic policy gradient (TD3) [26]. We also remind the above components are adaptive to various off-policy actor-critic RL algorithms.
In TD3, the target networks, namely, the target critic ′ with parameter ′ and target actor ′ with parameter ′ are utilized to stabilize the algorithm update. And the actor's output becomes a deterministic value instead of a sample from the probability distribution.
Considering the role of human participants in the RL interaction process, the eventual action in the time step can be expressed as: where is the noise coefficient vector dependent on the training proceed, is the bounding of the exploratory action, is the covariance matrix of the Gaussian distribution . A transition tuple is obtained through the above interaction step and stored into the proposed human-demonstration experience buffer as: Stored experience tuples will be retrieved for the training of the value and policy networks. An arbitrary transition tuple with index would be retrieved by the probability , which is calculated by: wherein the priority level is: It should be noted that Q-advantage is calculated by the target critic network ′ to avoid unstable updates.
Supposing the batched tuples with size contains 1 amount of non-demonstration transition tuples and 2 = − 1 human-demonstration ones, the loss function of the critic can be expressed as: where = 1,2 represents the index of two networks. Note the double network trick, which utilizes the smaller value of two networks ( = 1,2), is introduced here to eliminate the value overestimation effect. The loss function of the actor is calculated as: It should be noted that the training of the policy network can be delayed stabilizing the algorithm, that is, the actor would be updated once given the critic updating times.
Lumping all factors, the final algorithm is provided in Algorithm 1.

IV. HUMAN POLICY MODEL
In this section, a human policy model is established in conjunction with PHIL-RL. The model can relieve human workload in the human-in-the-loop RL process by imitating the behavior policy of actual human participants.
Albeit that employing human participants to conduct PHIL-RL is most helpful to improve the performance, the excessive participation would fatigue human beings. We train a regression model to imitate human policy simultaneously with RL, and this policy model can substitute humans when necessary.
Recall human behaviors in the RL training process: human interventions are intermittently imposed to the loop and human demonstrations are incrementally supplemented into the training set (buffer). Considering this, we train the human policy model leveraging an online-and incremental-based imitation learning algorithm, i.e., the Data Aggregation (DAgger) [27], which is free from offline large-scale collection of the demonstration data.
Denoting the human policy model with ℋ, the objective is to find a policy ℋ minimizing its difference with the human policy : We initialize model ℋ by replicating an untrained RL policy network. After the first human-intervention event, model ℋ is established as: In subsequent episodes, we retrieve human demonstrations to conduct incremental learning with the loss function: and update the model with the gradient method as: where is the episode number of the RL process. Through the above update, model ℋ would gradually be competent to accurately mimic human policy, and accordingly, substitute human participants to assist RL. It should be noted that if using this human policy model to cooperate with PHIL, the activation conditions of model ℋ shall be manually defined varying to specific environments.

V. PROBLEM FORMULATION
The proposed PHIL-TD3 algorithm, like most RLs, is universally adapt to any continuous-action decision and control tasks. Here we choose the end-to-end autonomous driving problem as the validation object, evaluating our algorithm in two challenging driving scenarios. It should be noted that the RL-based autonomous driving problem can be solved by numerous reasonable settings, while the problem formulation in this section is to provide a fair environment for algorithm evaluation and comparison. In this section, two challenging autonomous driving scenarios are introduced to evaluate the control and optimization performance of the proposed algorithm, then the standard optimization setting is established.

A. Autonomous Driving Scenarios
RL is better suited to the challenging driving tasks compared to rule-based or model optimization-based approaches due to its high representational and generalization capabilities. We choose two scenarios, shown in Fig. 2, to evaluate the RL performance where conventional autonomous driving strategies can be incompetent due to the complex combinatorial relationships.
Unprotected T-intersection left-turn: This scenario is illustrated in Figs. 2(a-b). The ego vehicle in the side road is trying to make a left turn and merge into the main road. No traffic signals guide the vehicles in the intersection. We assume the lateral path of the ego vehicle is planned by other techniques, while the longitudinal control is assigned to the RL agent. Surrounding vehicles are initialized with varying random velocities ranging from [4,6] m/s and controlled by the intelligent driver model (IDM) [28] to execute lane-keeping behaviors. All surrounding drivers are set with aggressive characteristics, meaning that they would not yield to the ego vehicle. The control interval for all vehicles is set as 0.1 seconds. Highway congestion: This scenario is illustrated in Figs. 2(cd). The ego vehicle is stuck in severe congestion and tightly surrounded by other vehicles; thus, it is trying to shrink the gap with its leading vehicles and conduct the car-following task with the target velocity. We assume the longitudinal control is completed by IDM with a target velocity of 6 m/s, while the lateral control is assigned to the RL agent. Surrounding vehicles are initialized with the velocity ranging from [4,6] m/s and controlled by IDM to execute car-following behaviors. The B. RL-based autonomous driving problem definition 1) State: The bird-view graphs are taken as the state information for the RL agent. Two consecutive frame images are used to constitute one state variable to enable temporal perception. We scale the camera-captured image to a smaller size to relieve the computational burden. The state variable can be expressed as: where ∈ ℝ 45×80 is a pixel matrix of which the elements are normalized.
2) Action: The action variable can be either lateral or longitudinal commands adaptive to different requirements. For the lateral control task in the congestion scenario, we choose the angle of the steering wheel as the action, expressed as: where ∈ ℝ 1 is the continuous steering command, of which the negative value indicates a left-turn command and the positive value corresponds to a right-turn command, and is the scaling factor that limits the steering range.
For the longitudinal control task in the left-turn scenario, we choose the accelerating/braking pedal aperture, expressed as: where ∈ ℝ 1 is the continuous pedal aperture, of which the negative value indicates a braking command and the positive value corresponds to an accelerating command.
3) Reward: The goal of an autonomous vehicle is to rapidly complete traffic scenarios through safe and smooth driving behaviors. RL-based driving strategy achieves this by an appropriate reward function definition. The reward schemes of the two tasks in Fig.2 can be respectively defined as: where rgoal = 10 and Sgoal is the set of goal states where the ego vehicle successfully completes the scenario; rfail = -10 and Sfail is the set of failure states where the collision occurs; while rspeed = -||vego -vtarget|| is the reward that encourages the target speed, i.e., 5m/s set in this paper; rsteer = ||δt -δt-1|| is the reward that discourages frequent steering behaviors. It should be noted that both rspeed and rsteer can implicitly play a role in promoting smooth driving. Additionally, we set the penalty term rpen in Eq. (22) the same as rfail and incremented it to the above reward when human intervention occurs.

4) Function approximator:
The function approximators of the value and policy functions are concrete by deep convolutional networks, as shown in Fig. 3.

5) Auxiliary functions:
We define some auxiliary control functions independent of the RL action to achieve a complete control suit. When RL manipulates the steering wheel, the longitudinal control is achieved by an IDM. When RL manipulates the pedal aperture, the lateral motion target is to track the planned waypoints by a proportional-integral (PI) controller. Fig. 3. a, The policy function architecture achieved by the neural network, where the target value network owning the same structure is omitted for brevity. b, The value network architecture achieved by the neural network, where the target value network owning the same structure is omitted for brevity.

A. Baseline algorithms
We employ state-of-the-art in the domain of human-involved RL algorithms as baselines and compare their performance against the proposed algorithm.
IA-TD3. This baseline is derived from Intervention Aided Reinforcement Learning (IARL) [17], which is a representative combination of a continuous-action RL algorithm and human demonstration. The RL's policy network is modified to adapt to human demonstrated actions by introducing the behavior cloning objective. Once human intervention happens, the human demonstration will substitute the RL's exploratory action, and a penalty signal will impose on the reward value. In this study, we devise a modified IARL by replacing the onpolicy base algorithm (PPO) with TD3, which essentially augmented the algorithm performance by improving the sample efficiency. We also implement the prioritized experience replay (PER) in this baseline for a fair comparison.
HI-TD3. This baseline is derived from Human Intervention Reinforcement Learning (HIRL) [16], which is a combination of a discrete-action RL algorithm and human demonstration. Once human intervention happens, the human demonstration will substitute the RL's exploratory action, and a penalty signal

Embedding layer
Action: steer/pedal signals Flatten will take on the reward signal. In this study, we devise a modified HIRL by replacing the discrete-action base algorithm (DQN) with TD3, which augmented the algorithm performance by improving the representation and control precision. We also implement the PER in this baseline for a fair comparison. RD2-TD3. This baseline is derived from Recurrent Replay Distributed Demonstration-based DQN (R2D3) [29], which is a representative combination of PER mechanism and human demonstration. In this study, we devise a modified algorithm by replacing DQN with TD3. The original R2D3 utilizes the recurrent neural network to augment performance, which is not the concerned technique in the context of this paper, thus, we remove the recurrent network structure and only focus on its replay distributed character regarding human demonstrations. Thus, we devise a Replay Distributed Demonstration-based (RD2) TD3 algorithm, which distributes human demonstration and RL exploratory experience into two experience buffers respectively and retrieves experiences by PER. The probability of utilizing human guidance instead of RL exploratory experience is aligned with the ratio of human guidance amount and total data amount.
Additionally, we employ the vanilla PER+TD3 that is shielded from human guidance as an ablated baseline.

B. Experiment setting
Multiple experiments are to evaluate the comprehensive performance of PHIL-TD3 against baselines. First, the training efforts of involved algorithms are comparatively evaluated in the two autonomous driving scenarios. Then the well-trained autonomous driving strategies are tested regarding control performance with several metrics. Last, a series of experiments involving both training and testing stages are conducted to analyze the mechanism of PHIL-TD3. The training hardware comprises a driving simulator and a high-performance workstation. The driving simulator is utilized to collect human data to train the human policy model complying with Section IV, and the workstation is dedicated to processing RL training. A high-fidelity autonomous driving simulation platform, CARLA [30], is employed to implement driving scenarios and generate RL-environment interaction information. The schematic diagram of the RL training stage is illustrated in Fig. 4(a).
The testing hardware is a robotic vehicle. The post-trained RL policy is implemented on the computation platform of the vehicle, which can communicate with the CARLA server through the wireless network. The on-board RL policy receives state information from CARLA and sends its control command back to remotely complete autonomous driving tasks. The robotic vehicle aims to test whether the RL policy is wellworked under the current onboard calculation and communication situations. The schematic diagram of the RL testing stage is demonstrated in Fig. 4(b).
The detailed configuration of the above experimental platform is provided in Appendix Table I. The algorithms are concreted based on neural networks, of which the architecture is illustrated in Appendix Table II. And the hyperparameters of the algorithms are given in Appendix Table III.

C. Evaluation of training efforts of RL algorithms
In this section, we explore whether human guidance can indeed improve the RL training, and further, which algorithm can achieve the best learning performance given the same human guidance. Additionally, we also investigate the effects of human guidance in dealing with RL tasks of different difficulties.
To eliminate the deviation brought by participant randomness and obtain repeatable results, we use the identical human model (see Section IV) to mimic human guidance behaviors in RL training processes. We fixate the sequence of random seeds and make the triggering conditions of human interventions invariant in all training attempts, which achieves a fair comparison across different algorithms. Two metrics are employed: the average reward of the training episode (excluding intervention-based shaping term), and the surviving distance of the ego vehicle in the training episode before a goal state or failure state in Eq. (33) occurs. A higher value of both metrics indicates a better learning performance. Fig.5 visualizes the learning performance through curves, represented with a solid line of the mean value and an error band of the standard deviation. We run each algorithm five times in the unprotected left-turn scenario and demonstrate their learning processes in Figs. 5(a-b). The vanilla TD3 is struggling to improve its policy, while the other three algorithms achieve higher rewards and surviving distances in a much shorter time, which indicates the effectiveness of human guidance. Among the human-involved algorithms, HI-TD3 performs the slowest learning process suggested by either reward or surviving distance, and IA-TD3 exhibits a faster convergence but with limited asymptotic performance. In opposite, PHIL rapidly

Real-time onboard environment High-fidelity scenario server
Action command

Being-trained RL driving policy
Pretreat human model before RL

Human policy
Interaction information (tuple) Update strategy seizes the opportunity of human guidance and learns the best asymptotic policy. It should be noted that PHIL-TD3 achieves the best asymptotic average reward of the baselines in less than 50 episodes, improving the learning efficiency by over 700%. We also run the congestion scenario five times for each algorithm and plot the learning curves in Figs. 5(c-d). The comparable PHIL and IA-TD3 perform better than the other two baselines when considering the reward. While the metric of surviving distance further confirms this advantage and profitably differentiates the algorithm abilities. Specifically, PHIL wins the highest eventual score. IA-TD3 and HI-TD3 manifest comparable levels of asymptotic performance while IA-TD3 has an advantage in learning efficiency. In this scenario, PHIL-TD3 achieves the best asymptotic average surviving distance of the baselines in 220 episodes, improving the learning efficiency by over 120%. Overall, the results in this training session highlight the significant superiority of the proposed algorithm in learning performance.  We further explore the learning performance of RLs with different task difficulties, which gives rise to Fig. 6. The normal setting complies with the problem definition in Section V, which is adopted throughout the paper, while the tough setting changes consecutive-frame input of Eq. (33) into a single frame input, impairing the temporal perception ability of RL agents. At the high level, the statistical results of the normal setting are aligned with the trends of Figs. 5(a-d). And it is indicated that the tough setting does not change the performance ranking of algorithms despite the degradations in different degrees. At the detail level, the performance difference between the normal and the tough settings, i.e., the ratios in Fig. 6, can manifest more algorithmic characteristics. Specifically, PHIL-TD3 and IA-TD3, which own the behavior-cloning objective, are less affected by the incomplete problem definition of the tough setting, whereas HI-TD3, and vanilla TD3, which less or not rely on human guidance, are significantly degraded in the same condition. Despite the single-frame state input in the autonomous driving task is not fairly reasonable, the findings through this comparison are useful. Since numerous complex real-world tasks are intractable to be well-defined or are only partially observable, the strong integration of human guidance into RL, e.g., behavior-cloning, can play a more remarkable role than pure RL algorithms.

D. Evaluation of testing performance of RL-based driving strategies
In this section, the post-trained driving strategies are tested in terms of autonomous driving performance, adaptiveness, and robustness, which can further evaluate the practicality of the above algorithms. Fig. 7. The driving performance of different RL strategies under six autonomous driving scenarios. The two noise-injected scenarios and two variant scenarios are different with the two training scenarios, which can examine the robustness and adaptiveness, respectively. "C" and "L" refer to congestion scene and left-turn scene, respectively, while "N" and "V" denote noise-injected and variant scene, respectively. a, The heatmap of success rate. b, The barplot of surviving distance in the left-turn scenarios. The theoretical maximum surviving distance of the scenario is 21 meters. The error bar describes the standard deviation. c, The barplot of surviving distance in the congestion scenarios. The theoretical maximum surviving distance of the scenario is 80 meters. The error bar describes the standard deviation. The paired t-test is adopted for the statistical test.
The zero-mean Gaussian noises, of which the standard deviation is 5% of the whole control domain, are injected to output commands of the driving strategies to test the robustness. More types and amounts of surrounding vehicles are added to construct variant scenarios to test the adaptiveness. We conduct 50 runs with the same sequence of random seeds for each post- trained strategy in each scenario. The success rate, which is defined as the number of completed runs divided by the total attempts in the same scenario, is taken as the metric for evaluating the safety performance in Fig. 7(a). Our PHIL-TD3 achieves the highest success rate in all scenarios, showing its superior task-completeness abilities. The vanilla TD3, albeit with its unstable training performance, performs competitively like IA-TD3 and HI-TD3 in the testing stage. Considering the two trained scenes (rows 1, 4) and noise-injected scenes (rows 2, 5), three baseline strategies behave acceptably, nevertheless, the scenario variants (rows 3, 6) significantly degrade their safety. Our PHIL, instead, maintains the highest ability regardless of varying testing conditions, manifesting itself with good robustness and adaptiveness. In Figs. 7(b-c), PHIL-TD3 once again shows its superiority in safety by the highest average surviving distance, and importantly, its performance stability is confirmed due to the lowest variance. Fig. 8 can further evaluate the detailed performance of driving strategies. Time consumption of the episode is the secondary target of RL optimization in the left-turn task, which is implied in the reward function of Eq. (36); thus, the related boxplot is illustrated in Fig. 8(a) to access this objective. It is found that the proposed strategy enjoys minimal time consumption, which is significantly different from other candidates. In congestion tasks, smoothness is the secondary target of the reward function of Eq. (37); thus, we choose the lateral acceleration as the smoothness measure and provide the associated boxplot in Fig. 8(b). The comparable humaninvolved strategies show their superior smoothness to vanilla TD3 in the training and noise-injected scenes, while the variant congestion scenario profitably validates the advantage of PHIL-TD3. Additionally, we compare the performance of three human guidance-based RL algorithms to the human guidance itself. Specifically, the surviving distance of these human-involved RLs are compared with the human policy model, and the results provided in Tables I. The results suggest the superiority of the proposed PHIL-TD3 over the human policy model.
Overall, our PHIL-TD3 perpetuates its predominance of training performance and takes the top spot in the testing stage.

E. Discussion on prioritized human experience utilization mechanism
In this subsection, we explore the effect of PHIL-RL from three aspects: the performance improvement by the TDQA mechanism, the merit of the single-buffer experience replay structure, and the algorithmic robustness to bad demonstrations.
TDQA, as the crucial innovation of PHIL-TD3, can improve learning performance in the context of human guidance-based RLs, as suggested in the above two sections. More specifically, it establishes a novel priority indicator to deal with various human guidance. Thus, we evaluate TDQA by comparing different priority schemes. "Q-adv" represents the scheme in which the priority of human guidance is calculated based only on Q-advantage. "TD", i.e., temporal difference, the scheme is inherited from the original PER method, but the TD weights of human demonstrations in it are doubled to highlight the human guidance in the replay buffer. Five learning attempts are conducted with the same sequence of random seeds for each candidate, and the corresponding learning curves are in Fig. 9. We find scheme comparison in two training scenarios shows similar trends when observing results in Figs. 9(a-b) and Figs. 9(c-d). The pure TD scheme learns faster than pure Q-advantage in both scenarios, yet its asymptotic scores (both reward and surviving distance) are significantly lower than those of the Q-advantage scheme. Essentially, these two schemes score human guidance based on different indicators, and a better indicator can provide RL with more high-quality guidance to improve learning efficiency. Thus, we find TD indicator, as proved in conventional PER, is indeed beneficial to rapidly improve performance, nonetheless, the Q-advantage indicator is superior to the TD indicator in the later stage of the training process. The delayed superiority of Qadvantage complies with intuition since unlike the direct indicator as TD, the evaluation ability of the Q network, i.e., the source of Q-advantage, also needs to be trained. The proposed PHIL, which smartly combines both indicators, achieves the most favorable performance in the two scenarios, showing the effectiveness of the TDQA mechanism. PHIL puts the human guidance and exploratory experience of RL into the same experience replay buffer. This structure differs from the double distributed scheme which is represented by R2D3. To evaluate the performance of these two schemes under the devised autonomous driving tasks, RD2-TD3 is developed which utilizes TD as the indicator to respectively retrieve data from two buffers. Additionally, the TDQA priority mechanism is ported to the RD2-TD3 setting forming the other variant, RD2-PHIL. Five learning attempts with the same sequence of random seeds are conducted by RD2-TD3 and RD2-PHIL. Through learning curves in Figs. 10(a-d), it is found that the double distributed buffer scheme, i.e., RD2-PHIL, fails to achieve the same level of learning efficiency as the proposed PHIL. A possible reason behind this is that human guidance can only be utilized in a chunk way under the doublebuffer setting, whereas the single buffer scheme of PHIL is more flexible and friendly to small-scale human guidance data. The conventional RD2-TD3 is least favorable, which is within expectation due to the lack of the TDQA mechanism. To sum up, the results in Fig. 10 support the single-buffer structure utilized in the PHIL-TD3, and profitably suggest the effectiveness of the proposed TDQA mechanism. A general situation occurs that human guidance is not perfect, a non-proficient human participant can sometimes conduct actions that are harmful to the task. We test if the unfavorable guidance would impair the learning process, that is, evaluating the robustness to harmful guidance. It should be noted that the robustness discussed here is distinguished from that in Section VI-D: we discuss how the algorithms are affected by poor guidance instead of the anti-noise ability of post-trained driving strategies. The human intervention condition of the training stage keeps the same as foregoing experiments, while one-third of the demonstrations from the human model are replaced with random actions to simulate nonproficient human behaviors.
Post-trained driving strategies under poor guidance are tested to conduct 50 runs in each scenario and are compared with those under the good guidance of Fig. 7. The stacked barplots in Fig.  11 provide the adversarial testing performance of three humanguidance-based RL algorithms under good and poor guidance.
We take the average surviving distance as the metric and the less performance deterioration by poor guidance suggests better robustness. Our PHIL-TD3 exhibits good performance since a nearly 50:50 situation occurs in all six scenarios. IA-TD3 falls behind with a 2.1% degradation on average in poor guidance context, while HI-TD3 is even improved by an average of 3.6% extent given poor guidance. Intuitively, poor guidance would remarkably degrade PHIL and IA-TD3 since they utilize the behavior-cloning objective to learn from human guidance, while HI-TD3, which only substitutes partial RL explorations with human guidance, can be less affected. The not-degraded HI-TD3 and most-degraded IA-TD3 support the above idea. Our PHIL defeating IA-TD3 is attributed to the TDQA mechanism: Q-advantage well access the quality of human demonstrations and feed more high-quality demonstrations to the RL agent; accordingly, the agent learns greater from good guidance than negative guidance. The secondary optimization target of RL, i.e., driving smoothness, is evaluated in Fig. 12 by acceleration distribution. The proposed PHIL-TD3 wins all scenarios by the most favorable smoothness which further confirms the abovementioned superiority.   Overall, the TDQA mechanism, as the core innovation of the PHIL-RL algorithm, contributes to the preponderant learning performance through its unique discriminatory power on the quality of human guidance. It also improves the robustness to poor guidance, which can relieve the requirement on the proficiency of human guidance. Additionally, the single buffer setting is more favorable than the double distributed buffer scheme under autonomous driving tasks of this paper.

V. CONCLUSION
In this paper, we establish a human-guidance-based reinforcement learning framework and propose a novel experience utilization mechanism of human guidance. Based on that, we put forward an algorithm, PHIL-TD3, aiming at improving algorithmic abilities in the context of human-in-theloop RL. We also introduce a human behavior modeling mechanism to relieve the human workload. PHIL-TD3 is employed to solve two challenging autonomous driving tasks, and its performance is comparatively evaluated against state-ofthe-art human-guidance-based RLs as well as the non-guidance baseline. Three main points are obtained through experimental results: 1) The proposed PHIL-TD3 can improve the learning efficiency by over 700% and 120% under the adopted two situations, respectively, and achieve remarkably higher asymptotic performance compared to state-of-the-art humanguidance-based RLs.
2) The proposed PHIL-TD3 achieves the most favorable performance, robustness, and adaptiveness in a series of metrics under the adopted two types of challenging autonomous driving tasks.
3) The proposed TDQA mechanism prominently contributes to the advance of PHIL-TD3 and can well discriminate the quality of various human guidance to relieve humans by less requiring on human proficiency.