{'fraction_non_alphanumeric': 0.02933282813930725, 'fraction_numerical': 0.008114679043920306, 'mean_word_length': 4.934420086190744, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Integrated Longitudinal Speed Decision-Making and Energy Efficiency Control for Connected Electrified Vehicles

To improve the driving mobility and energy efficiency of connected autonomous electrified vehicles, this paper presents an integrated longitudinal speed decision-making and energy efficiency control strategy. The proposed approach is a hierarchical control architecture, which is assumed to consist of higher-level and lower-level controls. As the core of this study, model predictive control and reinforcement learning are combined to improve the powertrain mobility and fuel economy for a group of automated vehicles. The higher-level exploits the signal phase and timing and state information of connected autonomous vehicles via vehicle to infrastructure and vehicle to vehicle communication to reduce stopping at red lights. The higher-level outputs the optimal vehicle velocity using model predictive control technique and receives the power split control from the lower-level con-troller. These two levels communicate with each other via a controller area network in the real vehicle. The lower-level utilizes a model-free reinforcement learning method to improve the fuel economy for each connected autonomous vehicle. Numerical tests illustrate that vehicle mobility can be noticeably improved (traveling time reduced by 30%) by reducing red-light idling. The effectiveness and performance of the proposed method are validated via comparison analysis among different energy efficiency controls (fuel economy promoted by 13%).


I. INTRODUCTION
N recent years, connected autonomous vehicle technology has aroused interest in the automobile companies as well as in the academic community [1]. Researchers are exploring ways to achieve mobility, safety and environmental impact goals via communication between vehicle and vehicle or infrastructure (V2V or V2I). Vehicle information (e.g. position, velocity, and acceleration) and traffic information (congestion and light timing) are able to be shared among the connected vehicles to reduce or eliminate crashes and enhance current operational practices of drivers [2].
To promote powertrain mobility and energy efficiency, reducing the red light idling and minimizing the braking times are the promising solutions for connected autonomous vehicles (CAVs) [3]. For example, Asadi et al. proposed a strategy using model predictive control (MPC) to decrease idle time at red lights and fuel consumption by signal phase and timing (SPAT) information [4]. A predictive optimal velocity planning algorithm was proposed in Ref. [5] by using the probabilistic traffic SPAT information. The relevant simulation results indicate that the presented method could increase energy efficiency via probabilistic timing and real-time phase data. Furthermore, Jin et al. proposed a power-based longitudinal control algorithm which can be applied in a connected eco-driving system [6]. The roadway grade and vehicle's brake specific fuel consumption are considered in this work when calculating an optimal speed profile with respect to energy savings and emissions reduction.
In current academic or industrial communities, CAVs are usually founded on or refitted from electrified vehicles, such as the hybrid electric vehicles (HEVs) and battery electric vehicles (BEVs) [7]. These vehicles have the potential to improve fuel economy and reduce pollutant emissions by embedding the electric storage systems into the powertrain [8]. Energy management is a significant technology for HEV and it means searching the optimal power distribution for a hybrid powertrain [9]. In general energy management problem, HEVs designers aim to optimize a pre-selected cost function (e.g. fuel consumption, harmful emissions, and running cost) by optimizing the power split controls of multiple energy storage sources while satisfying the driving power demand [10].
Many kinds of approaches have been utilized to formulate energy management strategies for HEVs, and the representative one is the optimization control theory-enabled method. For example, as the specific driving cycle is known a priori, dynamic programming (DP) algorithm [11] and equivalent consumption minimization strategy (ECMS) [12] have been developed to obtain the globally optimal power split decision for HEVs. The authors in [13] treated DP-based results as a benchmark to estimate the learning-based method in an energy management problem for a hybrid tracked vehicle. By computing the feasible range of optimal equivalent factor in ECMS, Ref. [14] presented an adaptive EMCS-enabled power split policy for a particular HEV configuration. Also, convex programming (CP) is another approach to derive the globally optimal power split control for HEVs [15]. The carbon reductions arising from renewable energy integration and the CP framework are quantified in [16], and the authors stated the proposed method could efficiently improve battery lifetime and reduce carbon dioxide emissions. Furthermore, when the full driving cycle is not given prior, stochastic dynamic programming (SDP) [17], model predictive control (MPC) [18] and reinforcement learning (RL) [19] are studied to achieve real-time optimal controls. For example, Stellato combined approximate DP and direct MPC to handle integer optimal control problems over long prediction horizons [20]. It is able to reduce the computational burden and enable sampling times below 25 μs. Ref. [21] discussed the control performance of different RL algorithms, Q-learning and Sarsa, in an online energy management problem. The pros and cons of these algorithms are analyzed and compared in the same hybrid powertrain. However, most literature did not involve the energy efficiency and fuel economic controls of autonomous electrified vehicles in the connected environments.
Overall, an integrated combination of CAV and HEV attracted more and more attention in recent years. How to improve powertrain mobility and fuel economy of this synthetical architecture is an important and comprehensive research hotspot [22]. However, few literature has considered and discussed the automation function of CAV and energy utilization of HEV together [23]. In order to fill that literature gap, this paper aims to optimize an integrated objective for a combination of CAV and HEV, which consists of the powertrain mobility and fuel economy. The proposed system is a hierarchical control architecture, which is assumed to consist of higher-level and lower-level controls, see Fig. 1 as an illustration. The entire goal of this work is promoting driving mobility and reducing fuel consumption for each automated electrified vehicle in the connected environment (formulated in (2) and (8)). The presented system aims to achieve this objective in two steps, the vehicle speed is optimized by the higher-level control first and then the energy management policy is derived in the lower-level.
Especially, MPC is utilized to solve the longitudinal speed decision-making problem in higher-level and RL is employed to research the energy management problem in lower-level. The higher-level exploits the SPAT and state information provided by V2I and V2V communication to reduce stopping at red lights. By doing this, the waiting time at the signal lights can be availably shortened in order to promote efficiency and mobility. The higher-level outputs the optimal vehicle velocity for each vehicle and receives the power split control from the lower-level controller. The lower-level aims to improve fuel economy for each vehicle, which indicates it would manage the power distribution between different energy storage systems according to the obtained speed profile.
Three perspectives are contributed to this paper: (1) a novel hierarchical control architecture is presented to provide a holistic method for improving mobility and energy efficiency for connected autonomous electrified vehicles; (2) MPC approach is applied to make a decision that focuses on improving powertrain mobility and enhancing vehicle safety via decreasing red light idling; (3) model-free RL technique is applied to obtain optimal power split control between the engine and battery for each electrified vehicle. Numerical tests illustrate that vehicle mobility can be noticeably promoted by reducing red light idling. Comparison of different energy efficiency control strategies demonstrates the effectiveness and control performance of the presented method.
The rest sections of our paper are organized as follows. Section II describes the higher-level architecture of MPC framework for longitudinal speed decision-making in con-gested road conditions; Section III illustrates the formulation of a power split control problem in electrified powertrain, and the structure of the lower-level model-free RL algorithm is also introduced; In Section IV, the vehicle and experiment parameters are first elaborated, and experiment results of higher-level and lower-level performance comparison are presented; Section V summarizes the key takeaways.

II. HIGHER-LEVEL PROBLEM FORMULATION AND MODEL PREDICTIVE CONTROLLER
In the higher-level, we focus on optimizing the traction or braking force to minimize the red light idling and energy consumption for traction of each connected autonomous vehicle. The position and velocity information of each vehicle can be obtained by V2V communication for their near neighborhood. The SPAT information of traffic lights is sent to each vehicle by roadside units through the wireless network. The proposed controller consists of two parts: First, a target velocity is calculated to avoid stopping at red lights, and second, MPC strategy is formulated to minimize the pre-defined cost function.

A. Higher-Level Problem Formulation
For longitudinal control, the dynamics of the vehicle i is described as [24]: where xi = [si, vi], si and vi denote the position and velocity of vehicle i. The control action ui is the traction or braking force per unit mass. The symbols Mi, Cd, ρa, Ai, μ and θ denote vehicle mass, drag coefficient, air density, vehicle frontal area, rolling friction coefficient, and road gradient, respectively.
In the higher-level, the optimization objective can be computed by dividing the total distance from fuel consumption, which is denoted by where η i tr is the transmission efficiency connecting the fuel tank and powertrain, H i LHV represents the fuel lower heating value for vehicle i, P i w is the traction power depending on the traction force, and η i rec is the recuperation efficiency acquired from the lower-level controller [22]. vmin, vmax, u i min and u i max depict the upper and lower threshold values for the allowable speed and possible acceleration, respectively. Hence, the higher-level controller generates the optimal velocity profile for each connected vehicle and inputs them into the lower-level for energy efficiency control.

B. Target Velocity Computation
The upcoming SPAT information is obtained a priori via the V2I communication for each connected vehicle. Based on this, the target velocity of each vehicle can be decided wisely to avoid idling at the red light. The target velocity at time instant k for the vehicle i is defined as following [ if Lig green and otherwise where dia(k) is the distance between traffic signal and the vehicle location (si(k)), tr , tg and tcy denote the red, green light and total cycle durations, respectively. Kw represents the cycle number for the traffic light, and mod denotes a function to reserve the remainder for division k by tcy. Eq. (3) indicates that when the equation dia(k) / (Kwtcy -k) ≤vmax is satisfied and the status of the traffic light is green, the target velocity is selected as the maximum allowable speed. Violation of this condition implies that the vehicle cannot pass through the current green light cycle although using the maximum speed. Thus, the vehicle needs to slow down and wait for the next green light window. Sometimes, the vehicle has to stop as there is no feasible velocity in the consecutive green light windows [23]. Overall, target velocity means the available speed to help each vehicle pass through the traffic signal at different green light windows.
Assuming the vehicle i could go through the traffic signal at one of the green light windows, and thus there is a speed range for the target velocity because the green light window is a time interval. Assume v i l(k) denotes the lower threshold velocity and v i u(k) depicts the upper value, they are computed as [22]: Fig. 2 depicts the schematic of the target velocity and velocity range calculation process [22]. t1 and t2 describe the time interval between the current time instant and the next green instant or the next red instant, respectively. Note that, the upper value of the speed range is the target velocity. Based on the speed range, the limits on the control action can be calculated as follow: Thus, the control action range (traction or braking force) is evaluated as: This limit implies that the vehicle velocity will fall into the velocity range as in Eq. (4), if the control action is bounded as in Eq. (6). Therefore, the vehicle can avoid stopping at a red light.

C. Model Predictive Control
In the previous section, we have discussed the optimization objective in the higher-level problem and the definition of the target velocity based on the communication between each vehicle and traffic lights via V2I information exchange. Note that, the optimal velocity trajectory for energy efficiency control and the target velocity in the higher-level may be different because the latter does not consider energy efficiency. Hence, we compute the optimum velocity profile by considering the extra position information of the front vehicle j over a finite future time horizon T via V2V information exchange. The discussed driving scenario is depicted in Fig. 3, wherein the driving mobility of each vehicle can be interpreted as the traveling time for the same scenario and the energy efficiency could be quantified as the fuel consumption of each HEV [25,26].
As the position information of the front vehicle j and the target velocity of the current vehicle i are available, the MPC approach is used to generate the best velocity profile. It can be formulated as a nonlinear optimization problem at time instant k as follow [27]: where S0 and thw represent the predefined critical distance and headway time, and the two inequality constraints are derived from Eqs. (4) and (6) respectively. In the cost function, the first term depicts the division of total fuel consumption by travel distance, the second items denote the desirable distance deviation between the vehicles i and j, the third term restrains the velocity of vehicle i close to the computed target velocity, and the last term aims to optimize control action [22]. The optimal energy efficiency velocity profile highly depends on the selection of the weights φ1 to φ4. The weights φ1 and φ3 are defined as a function of (v i u(k)-v i l(k)) to ensure when the deviation is large, more emphasis is focused on energy efficiency rather than velocity tracking (φ1 high and φ3 low), and as the deviation is small, more attention is paid to the target velocity tracking (φ1 low and φ3 high). The weight φ2 is relevant with the distance deviation (s t j-s t i), and it increases as the deviation is small and it decreases as the deviation is large, finally the weight φ4 is a constant [23]. In conclusion, the speed range serves two goals, first it helps choose some weights in the cost function, and second, it forces the constraints on the velocity deviation to avoid red light idling.
Three key processes are involved in MPC algorithm: 1) search the optimization controls to minimize the pre-defined cost function during a prediction horizon; 2) obtain the first portion of the optimization controls and apply it into the physical plant; 3) moves forward the entire prediction horizon and repeat the first step [28]. Here we apply sequential quadratic programming to solve the nonlinear optimization problem (see Eq. (7)). Especially, the optimal control actions are computed using the command fmincon in Matlab/Simulink and the optimal solution for time k is applied to formulate the initialization

III. LOWER-LEVEL PROBLEM FORMULATION AND REINFORCEMENT LEARNING
In the lower-level, we aim to improve the energy efficiency of each connected autonomous electrified vehicle by deriving the optimal power split control between multiple energy sources. The sole state variable is the state of charge (SOC) in the battery and control action is the engine throttle percentage th(t). After the higher-level gave the optimal speed profile, the relevant power demand can be calculated via the powertrain modeling. Then, the transition probability matrix (TPM) of the power demand can be calculated by the maximum likelihood estimate. Finally, the Q-learning algorithm in the RL framework is employed to evaluate the optimal control problem in the lower-level.

A. Lower-level problem
In the minimization problem of the energy efficiency for a parallel electrified vehicle, the optimization objective is the sum of the charge sustenance and fuel consumption. The problem can be mathematically established as follow [20,21] where χ is a positive weighting factor (equal to 1000 in this work) to guarantee the initial and final SOC values stay the same. SOCmin, SOCmax, thmin, and thmax are the boundary values of the SOC and throttle percentage. Ten and nen are the engine torque and speed, nen, min, nen, max, Ten, min, and Ten, max Modeling the battery with the equivalent circuit model [29], the state equation of the SOC is given by: where Qb denotes the battery rated capacity. Voc represents the open circuit voltage and rin describes the battery internal resistance. Voc and rin are relevant to the SOC.

B. Transition Probability Matrix
As the velocity profile is known in advance from the higher-level controller, the power demand to impel each connected autonomous electrified vehicle is computed as follow: 3

21.15
where δ is the mass factor, ai is the i-th vehicle acceleration. For the purpose of keeping power balance in the powertrain, the power demand is satisfied by both of the engine and battery: where η i m is the traction motor efficiency for vehicle i. As the engine output power can be evaluated by Eq. (9), and then the battery power can be estimated from Eq. (12). The transition probability of the power demand is depicted via maximum likelihood estimator as: , 1, 2,...
where N i nk,m denotes the number of times for the transition from P i dem,n to P i dem,m at vehicle speed v i k, N i nk is the total transition counts initiated from P i dem,n at vehicle speed v i k, k is the discrete time step, and M is the amount of discrete power demand index. Fig. 4 describes the calculational schematic for the TPM of the power demand.

C. Reinforcement Learning Control
Based on the previous TPMs of power demand and vehicle modeling, RL technique is employed for energy efficiency control derivation. Especially, the Q-learning algorithm is implemented in Matlab with the Markov decision process (MDP) toolbox presented in [30], and the computer has an Intel quad-core CPU of 2.70 GHz a memory of 3.8 GB.
In the RL framework, a learning agent interacts with a stochastic environment. Five key variables are exploited to model the interaction, wherein S is the state variables set, A is the control actions set, P denotes the TPM for power request, r(s, a) ∈R represents the reward function and ∈(0, 1) means a discount factor.
The sequence of control actions constitutes the control policy π. The optimal value function can be computed as follows [31]: where psa,s' denotes the transition probability. As the optimal value function is decided at each state variable, the relevant optimal control policy is determined as: The computation process of optimal control action can be interpreted as the matching design of state variable and control action. Hence, the (s, a) is called state-action pair and the related action-value function Q(s, a) is given as follows:   (16) Note that, variable V * (s) means the optimal value function that taking an optimal action a at state s. Thus, V * (s) =Q * (s, a) and π * (s)=arg mina Q * (s, a). Finally, the updated criterion for the optimal action-value function of the Q-learning algorithm is indicated by: where γ∈[0, 1] is a decaying factor of the Q-learning algorithm.

finish when s is the terminal
To solve the optimal energy efficiency control problem, the state variables and control actions are discretized as follow: S={(SOC(t))|0.4≤SOC(t)≤0. 8} and A={th(t)| 0 ≤th(t)≤1}. The set of reward function is R={ f m (s, a)+ χ (SOC(t)-SOC(0)) 2 }. The initial and final values of SOC are selected to avoid overcharge and over-discharge in the battery. Table I shows the calculation process of optimal control actions in Q-learning algorithm. First, the matrix of action-value function Q(s, a) and the state variable s are initialized. Then, in each episode, the control action is decided by ε-greedy policy at each step, and the action value function is updated by (17). Finally, one episode ends when the state variable is the terminal, and the whole process ends when the number of episodes reaches the setting value. The obtained matrix Q(s, a) can be used to derive the control action in particular driving situation.
In this paper, the discount factor τ is taken as 0.96 to balance the importance of current and future rewards, the decaying factor γ is equal to 1 / 2 k + to accelerate the convergence rate and the value of ε is 0.2*0.99 k (ε decreases with time steps) to select the control action at each step. The iterative times NK is 10000, and the sample time △t is 0.5 second. The effectiveness of the MPC and RL approaches for the higher-level, and lower-level problems are validated in the next section. A signal lane road with eight vehicles constructs the simulation scenario, wherein the traffic lights are settled at every 500 m. Each parallel electrified vehicle has the same configuration and powertrain parameters, see Fig. 5 as an illustration. The primary parameters of them are listed in Table 2. The rated power for the diesel engine is 100 kW, and the maximum torque of diesel engine is 200 Nm as the rotational speed change from 110 rpm to 390 rpm. The maximum power, maximum torque and maximum rotational speed of the traction motor are 78 kW, 130 Nm, and 600 rpm, respectively. The battery pack is 6 Ah capacity with a nominal voltage of 250 V. For the SPAT parameters, the time intervals for the red and green light are 30 and 15 seconds (tr=30 and tg=15), respectively. The minimum and maximum allowable speeds for each connected vehicle are unified as 0 m/s and 20 m/s. Every vehicle's initial position is given by random, and the initial relative distance between any two vehicles is also from 10 m to 20 m by random. The initial velocity of each connected autonomous electrified vehicle is also stochastic and selected from 10 m/s to 15 m/s.

IV. RESULTS AND DISCUSSION
The proposed MPC-enabled and RL-based longitudinal speed decision-making and energy efficiency control strategy for connected autonomous electrified vehicles is evaluated in this section. First, the performance of the MPC-enabled higher-level controller is examined by comparing with a baseline method. Then, the presented hierarchical system is compared with a benchmark system to assess its availability. The benchmark system employes MPC method in both higher and lower levels. Simulation results demonstrate that the proposed longitudinal speed decision-making and energy efficiency control strategy can highly reduce stopping at red lights and improve fuel efficiency.

A. Higher-Level Controller Evaluation
In this section, the MPC-based longitudinal speed decision-making control is compared with a baseline method to verify its effectiveness. Here, the baseline method means the connected vehicle apply V2I communication to calculate the target velocity in Eq. (3), and then executes the connected cruise control strategy [32]. The vehicles can only go through the traffic signal during the green bars period. The zoom-in plots in these figures indicate that connected autonomous vehicles move together until the traffic red lights split them. In the baseline method, the vehicles split into two groups at 100 second, however, the vehicles in the MPC-based method split until 400 second. These results attribute to the limitation of road speed (the rest of vehicles cannot violate the limitation) and indicate that the MPC-based technique can significantly improve the system mobility. To avoid chaotic and unclear exhibition, four random vehicles from eight vehicles are selected to display their running speeds in the defined scenario. Fig. 8 describes the velocity profiles for these vehicles (their order: 1, 3, 5, 6, a total of 8 vehicles). It is apparent that the vehicle velocities in the MPC-based method always stay positive which indicates that all the vehicles would not stop at every traffic signal light. Oppositely, the connected vehicles controlled by the baseline method have to stop at all traffic signals. Thus, the traveling time of each vehicle (pass through the same driving scenario) could be reduced by 30% (shown in Table 3) in the proposed method. It also suggests that longitudinal speed decision-making control strategy presented in the higher-level controller could effectively avoid stopping at red signal lights so that the system mobility can be highly promoted. In this section, the proposed whole system is compared with a benchmark system to assess its effectiveness and merits. In this benchmark (baseline) system, the higher-level controller is the same as our proposed MPC-based longitudinal speed decision-making control strategy, while the lower-level still utilizes the MPC method [28] instead of the RL algorithm. Similarly, to prevent tedious and chaotic demonstrations of relevant results, two random vehicles are chosen to compare their energy management strategies. The order of these two vehicles is 2 and 7 (totally eight vehicles).

B. Evaluation of Integrated System
The velocity tracking in both vehicles using the RL algorithm in the lower-level controller are shown in Fig. 9. It illustrates that the lower-level controller is able to track the optimal velocity trajectory provided by the higher-level controller almost perfectly. Under this situation, the proposed method can avoid stopping at red lights and guarantee improving energy efficiency simultaneously. Fig. 10 shows the state variable SOC evolution for these two vehicles in different approaches. For the same vehicle, the derived velocity trajectories are different in these two methods.
Thus, the related power demand and control actions are different, which results in the disparate variation of SOC curves. Due to the charge sustenance constraint in the lower-level controller, the final SOC value is close to the initial value. This implies that the battery power used by the electrified vehicle, leading to a decrease of SOC, which is compensated by charging with the power from the engine and regenerative braking power. Hence, we can evaluate the different methods' control performance by comparing the power split and fuel consumption. The power split controls in these two vehicles are described in Fig. 11. As Eqs. (9)-(12) stated, the battery and engine powers are finally decided by the throttle percentage. From Fig.  10, it is obvious that the power split controls in the proposed system are different from those of the benchmark system, which implies these two systems could achieve different control performance although the input speed profiles are the same. The working points of the engine in the different control strategies are given in Fig. 12. It is apparent that the engine working area under the proposed RL-based control locates in the lower fuel consumption region more frequently, compared to the MPC-based control. This implies that the RL method leads to higher energy efficiency than MPC technique. Hence, the proposed system could achieve lower fuel consumption compared with the baseline system. To quantify the performance in driving mobility and energy efficiency [33,34], Table 3 shows the corresponding fuel consumption after SOC-correction and traveling time in these two systems for two selected vehicles (Vehicle 2 and 7). Note that, the corrected fuel consumption in the proposed system is lower than that of the baseline system for both vehicles. These results illustrate that our proposed energy efficiency control can achieve higher control performance than MPC-based control in the lower-level. For the same vehicle, it is obvious that the traveling time in the proposed system is shorter than that of benchmark system. It means the vehicle could reduce red light idling and improve the average running speed, and thus promote the driving mobility by using the proposed system. Thus, the proposed MPC and RL system can improve both driving mobility and energy efficiency. Furthermore, to evaluate the possible application in re-al-world, Table 4 displays the computation time of the pro-posed system and benchmark system for the chosen two ve-hicles. It can be discerned that the proposed system is shorter than the baseline one and it is feasible to be applied online for automated HEV. The future work includes extending the MPC and RL-enabled system into multiple driving scenarios, and thus it can help the vehicle improve the mobility and fuel economy in real-world environments.   Overall, our proposed longitudinal speed decision-making and energy efficiency system can not only improve the system mobility for a group of connected autonomous vehicles but also reduce the fuel consumption for each internal electrified vehicle. The reasons for these results are theoretically because the speed decision-making policy could efficiently reduce the red idling and the RL-based energy management strategy could reasonably allocate the output powers for multiple energy sources. As the computation time of the presented whole system is short and the RL algorithm could obtain model-free controls, the proposed hierarchical control architecture is promising to be applied in real-time.

V. CONCLUSION
A hierarchical control architecture to address the longitudinal speed decision-making and energy efficiency control problem for connected autonomous electrified vehicles is presented in this paper. The higher-level controller can use the model predictive control approach to handle the signal phase and timing and state information of vehicles via vehicle and vehicle or infrastructure communication to improve the system mobility. The lower-level controller exploits the reinforcement learning algorithm to manage the optimal velocity profile from the higher-level to reduce the fuel consumption for each electrified vehicle.
It is verified in the tests that the proposed longitudinal speed decision-making and energy efficiency control strategy is effective and available. Connected autonomous vehicles can avoid stopping at red signal lights by the model predictive control method, and each can highly improve fuel economy via the reinforcement learning approach. This methodology can be applied to both full and semi-autonomous vehicles. The future research direction involves predicting the driver behavior using the higher-level controller and addressing complex decision-making tasks in the lower-level controller.