{'fraction_non_alphanumeric': 0.02895104284841609, 'fraction_numerical': 0.008530412775983943, 'mean_word_length': 4.67219403539166, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Self-adaptive SAC-PID Control Approach based on Reinforcement Learning for Mobile Robots

Proportional-integral-derivative (PID) control is the most widely used in industrial control, robot control and other fields. However, traditional PID control is not competent when the system cannot be accurately modeled and the operating environment is variable in real time. To tackle these problems, we propose a self-adaptive model-free SAC-PID control approach based on reinforcement learning for automatic control of mobile robots. A new hierarchical structure is developed, which includes the upper controller based on soft actor-critic (SAC), one of the most competitive continuous control algorithms, and the lower controller based on incremental PID controller. Soft actor-critic receives the dynamic information of the mobile robot as input, and simultaneously outputs the optimal parameters of incremental PID controllers to compensate for the error between the path and the mobile robot in real time. In addition, the combination of 24-neighborhood method and polynomial fitting is developed to improve the adaptability of SAC-PID control method to complex environments. The effectiveness of the SAC-PID control method is verified with several different difficulty paths both on Gazebo and real mecanum mobile robot. Futhermore, compared with fuzzy PID control, the SAC-PID method has merits of strong robustness, generalization and real-time performance.


INTRODUCTION
Proportional integral derivative (PID) control is the most widely used control method in industrial and robotic control because of its simple structure, strong robustness, strong adaptability and the fact that the parameter adjustment and tuning rarely depends on the specific model of the controlled object. Actually, traditional PID control often uses the experience tuning method to obtain the appropriate PID parameters. Therefore, modern control theory has contributed to many tuning methods. Hernández-Diez 1 proposed a parameter adjusting approach for PID controllers which partition the parameter space in to regions, defined by separating hyperplane, with constant number of unstable roots. Onat 2 developed a graphical method for tuning the PI-PD controller parameters based on the stability boundary locus by equating the real and the imaginary parts of the characteristic equation to zero.
However, the above tuning methods often need accurate mathematical model and complex calculation. The PID tuning techniques tend to have a poor performance requiring an exhaustive expert knowledge about the system behavior and its variant arXiv:2103.10686v1 [cs.RO] 19 Mar 2021 operational environments. In this sense, the classical PID control cannot be modified in real time during the control process of the robot and make it difficult to realize the real-time control and optimal control. Yang 3 designed a new adaptive motion controller for robot manipulators subject to unmeasurable or uncertain plant parameters. Besides, Yang 4 proposed the controller based on adaptive fuzzy control approach to solve the control problem for a class of MIMO underactuated systems with plant uncertainties and actuator deadzones. These intelligent control methods have the ability to adapt the changes of environment or the performance of robots, which seems to be suitable for solving these types of problems. Futhermore, adaptive parameter tuning control algorithm for PID controllers began to arise. For example, Papadopoulos 5 proposed an automatic PID tuning method based on amplitude optimization principle for single-input and single-output control problem of a linear dynamic model. However, traditional adaptive PID control methods usually fail to achieve optimal control performance, especially for mobile robots operating in unknown environments, such as automatic line-following robots with unknown paths.
In recent years, automatic line-following robots have been increasingly used in industrial scenes, such as material escorting robots in automatic production lines and fire-fighting inspection robots in fire-prone areas 6,7 , because they can complete some high-workload or high-risk tasks in complex and harsh applications. In addition, the line-following robot has been widely used in civil, military and medical fields 8,9 . Line-following robot is a kind of robot that can move autonomously according to a given route mark which is usually represented by black line on a white surface or other color combinations. The sensors are required for automatic line-following robots to extract the external route information and transmit it to the control system. These sensors can be grayscale sensors, electromagnetic sensors and cameras. According to the obtained route information, the control system manipulates the robot to maintain the route, and continuously corrects the deviation from the route which it is following at the same time. Engin 10 successfully realized the tracking of straight, circular, sharp turning and S-shaped line by using PID control algorithm, and improved the navigation accuracy. Juang 11 proposed a slope matching algorithm for the route tracking of humanoid robots. The camera was employed to capture the route information, and the curvature and slope estimation parameters of the route obtained from image processing were applied as the input of PID controller to reduce the tracking error. However, due to the problems such as unknown route and low modeling accuracy of mobile robots, it is difficult to obtain the optimal parameters of PID controllers in real time only by using traditional adaptive parameter tuning control methods.
For adaptive control formulations of difficult modeling controlled objects in complex and variant environments, such as linefollowing robots, reinforcement learning(RL) is one of the most powerful learning paradigms. Different from the traditional control methods, reinforcement learning 12 can not only interact with the environment autonomously and learn the optimal policy through repeated trials, which has certain robustness and real-time performance, but also solve complex problems through modelfree learning algorithm 13 for some control objects that are difficult to model, which has certain generalization. The previous reinforcement learning method 14 is based on the optimal control theory in which the sequence decision problem is considered as an adaptive dynamic programming problem. An optimization policy algorithm based on the value function was further proposed as evaluation criterion 15 . However, traditional reinforcement learning is limited to low-dimensional and simple scenarios due to the lack of good state and policy representation ability. It is difficult to converge to an optimal policy for high-dimensional state. Mnih 16 proposed a deep Q network model DQN that combines the advantages of feature representation in deep learning and real-time optimization policy for reinforcement learning. Since then, deep reinforcement learning has been widely used in game development 17,18 , natural language processing 19,20 , autonomous driving 21 , recommended search systems 22 , robotic skills learning 23,24 and other fields.
Reinforcement learning has been incerasingly used in the field of PID control to realize the adaptive adjustment of PID parameters. Carlucho 25 proposed an incremental Q-learning algorithm to learn and tune the parameters of the PID controller online, and the effectiveness of this method was proved on a real mobile robot. Based on this, Carlucho 26 also developed the double Qlearning algorithm for on-line autonomous adaptation of low-level PID controllers of mobile robots. The Q-learning algorithm based on value function has strong ability of policy evaluation, but it is difficult to obtain effective solutions for the problems of continuous action space or high-dimensional action space. Policy-based method is another way to optimize policy, compared with the value-based method, which has better convergence and can deal with the problems of continuous action space and stochastic policy, but it is easy to fall into local optimum. Therefore, Konda 27 proposed an actor-critic algorithm which combines the value-based method and the policy-based method. Akbarimajd 28 used the adaptive PID controller based on the actor-critic algorithm and drove the two-degree-of-freedom of the robotic manipulator. Carlucho 29 presented a model free goal-driven deep RL method based on the deep deterministic policy gradient (DDPG) algorithm 30 for self-tuning of the low-level PID controllers of mobile robots. The proposed hybrid control policy adopts an actor-critic structure and a gradient inversion scheme to constrain the actor outputs along the training phase. However, because of the combined effect of the deterministic actor network and Q-function, DDPG also has some disadvantages, such as the difficulty in stabilization and sensitivity to hyperparameters. In order to solve the above problems, Haarnoja 31 proposed an off-policy maximum entropy actor-critic algorithm with a stochastic actor, that is soft actor-critic (SAC), which ensures the effective learning of samples and the system stability. The above methods of combining RL and PID controller are all applied to specified target velocity for tracking, but the tracking situation when the target velocity is constantly changing is not discussed, such as line-following task. As an alternative, Saadatrnand 32 proposed a MIMO simulated annealing (SA) based on Q-learning method to control a line following robot. However, this method requires precise physical modeling and the application scenarios are more limited. Matczak 33 proposed tracking algorithm based on deep learning for the line-following robot, which requires large training image and high-performance computing equipment.
Inspired by RL and the automatic line-following method based on PID controller, we propose a model-free adaptive SAC-PID control method based on soft actor-critic for the line-following robot with unknown and complex routes including forks and sharp turns. Compared with above method, our proposed method can adjust optimal parameters of PID controller to track the changing target velocity in real time without accurate physical models of mobile robots and large training data. The proposed SAC-PID control method uses a new hierarchical structure including soft actor-critic layer and incremental PID layer. Soft actor-critic receives the information of the mobile robot as input, and simultaneously outputs the optimal parameters of the incremental PID controllers at the current time, which compensates for the error between the path and the mobile robot in real time. In addition, we also proposed a combination of 24-neighborhood method and polynomial fitting, which effectively extracts the correct boundary of the fork and reduces the disturbance caused by the change of environment. Compared with fuzzy PID control, the strong robustness, generalization and real-time performance of the SAC-PID control approach are verified through simulations of different difficulty paths in Gazebo. Furthermore, we also demonstrate the effectiveness of the SAC-PID control method on a mecanum mobile Robot in real world environments. In our knowledge this is among, if not, we are the first to implement model-free adaptive PID based on reinforcement learning to control line-following robots. Qualitative results are available at https://youtu.be/GaWI_T6etUM.
The remainder of this paper is organized as follows: In section 2, the problem statement is presented. In section 3 we indicate the details of the SAC-PID control approach for line-following robot. The simulation and physical experiment results of this SAC-PID control method presenting in section 4. Finally, we draw a conclusion in section 5.

PROBLEM STATEMENT AND ANALYSIS
Traditional PID control is difficult to complete the complex control task in unknown and variant environment, and the parameters of PID controllers need to be adjusted manually with expert knowledge. To deal with the complex control environment and avoid the tedious process of tuning PID parameters, a model-free adaptive SAC-PID control method will be studied in this paper to solve the automatic adjustment problem of the PID control parameters. In this section, we introduce the main elements that support our proposal.

Incremental PID controllers in mobile robots
For a computer-controlled line-following robot, the control algorithm should be discrete rather than continuous. Different from the typical continuous PID control law, the digital PID control law is discrete which can generally be divided into positional PID control and incremental PID control. Once the control output ( ) is wrong, using positional PID controllers will cause a large change of the system because of the accumulation of all the past errors. Consequently, the positional PID control has poor robustness and heavy calculation workload, compared with the incremental PID control which is only related to the error sampling value of the last three moments. In addition, the output of incremental PID control is the increment of controller output ▵ ( ), so the incremental PID control has less influence on the system and strong robustness compared with the positional PID control with direct output ( ). The incremental PID control law is as follows: where is the discrete sampling time, , and are the proportional, integral and differential coefficients of the PID controller respectively, ▵ ( ) is the output of the incremental PID controller at the current time , that is, the difference between the control output ( ) at the current time and the control output ( − 1) at the previous time − 1, and ( ), ( − 1) and ( − 2) are the system errors of the time , − 1 and − 2 respectively.
Generally, in the application of mobile robot, the dynamic performance of the robot is not completely predictable in advance and may change with the external environment, so it is difficult to establish an accurate mathematical model of the mobile robot. What's more, the route is often unknown, so it is necessary to consider the adaptive PID control. The adaptive PID control can select the optimal PID parameters in real-time according to the dynamic performance and operating environment of the mobile robot and this is where artificial intelligence can be of aid. Compared with traditional adaptive PID control methods such as fuzzy-PID, model-free self-adaptive PID controller based on reinforcement learning can adjust PID parameters better, especially when encountering complex unknown routes, such as route line crossing, sharp turning. In the next section, the RL formulation to obtain the gains of the PID controller is introduced.

Deep reinforcement learning for PID tuning
The basic process of reinforcement learning that is a Markov decision process (MDP) represented by a tuple: Agent observes the state at time , and then selects an action . The state transfers from to +1 according to probability ( +1 | , ). At the same time of state transitions, agent receives reward ( , ), where ∈ , ∈ , The cumulative reward obtained at the end of the process is given by where is discount factor. The goal of reinforcement learning is to obtain the optimal policy * , that is, to maximize the cumulative reward while reaching the target state. Compared with standard RL, the SAC introduced a more general maximum entropy objective, which favors stochastic policies by augmenting the objective with the expected of the policy over ( ) where is the temperature parameter to determine the relative importance of entropy term against the reward and control the stochasticity of the optimal policy. The maximum entropy objective not only enables the agent to balance exploration and exploitation intelligently, but also enables the agent to capture multiple modes of near-optimal behavior. SAC is derived starting from a maximum entropy variant of the soft policy iteration method based on a tabular setting, which consists of policy evaluation and policy improvement. In the policy evaluation step of soft policy iteration, the soft Q-value ( , ) of a policy can be computed according to the maximum entropy objective in Eq. 3. Therefore, the modified Bellman backup operator  is given by where is the soft state value function.

Lemma 1. Consider the soft bellman backup operator  in Eq. 4 and a mapping
Then the sequence will converge to the soft Q-value of as → ∞.
Soft Q-function for policy can be obtained by repeatedly applying  as Lemma 1.
In the policy improvement step, the policy is updated according to minimize the Kullback-Leibler (KL) divergence of the policy and Q-value policy. The new policy can be obtained as follows where ( ) is the partition function. Moreover, the policies are restricted to some set of policies Π just like Gaussians in order to make it tractable.

Lemma 2. Let
∈ Π and let be the optimizer of the minimization problem defined in Eq. 6. Then According to Lemma 2, the new policy has a higher value than the old policy with respected to the maximum entropy objective. Iterate soft policy evaluation and soft policy improvement repeatedly from any ∈ Π converges to a policy * such that ( , ) ≥ ( , ) for all ∈ Π and ( , ) ∈  × ,|| < ∞. In the vast majority of mobile robot applications, the performance of the mobile robot and the operating environment are not entirely predictable in advance. In order to maintain optimal robot performance, the parameters of its control module must be continuously adjusted. Therefore, the design of an adaptive PID controller that combines the real-time performance and optimality of RL with the strong stability of traditional PID control is considered. The design principle is that the RL agent adjusts the parameters of the lower level PID controller according to various states. It is obvious that this design principle can solve the problem that mobile robots cannot maintain optimal performance due to unpredictable environmental changes. Specifically, the RL agent makes a decision based on the state observed in real-time and outputs an action to adjust the parameters of the lower-level controller. In this case, the lower-level controller is the PID controller, so the actions are the parameters of the PID controller, i.e.
= . Then the PID controller controls the mobile robot to maintain its optimal performance based on the changing parameters and the measurement error resulting from the interaction between the mobile robot and the environment. The mobile robot follows the path which is unknown and changeable, that is, the target velocity is changing at every moment. Therefore, the designed control method must have good real-time performance and stability. In addition, the optimality of the controller also needs to be considered because of some paths with large curvatures. Hence, a new hierarchy structure is required to control the line-following robot such that soft actor-critic is the upper controller and the PID controller is the lower controller. Soft actor-critic receives the error of the mobile robot as input and simultaneously outputs the optimal parameters of incremental PID controllers to compensate for the error in real-time.

SAC-PID CONTROL APPROACH BASED ON ADAPTIVE DEEP RL
When the mobile robot follows the line, the error between the path and the mobile robot is related to the current time and changes in real time, which satisfies the Markov property. We can regard the line-following task as a Markov decision process: given a state , the agent selects and executes an action according to the policy ( ). After changing the velocities of the mobile robot, the new state +1 and corresponding reward ( , +1 ) will be obtained, which constitutes a set of tuples ( , , +1 , ). The goal of reinforcement learning is to find a optimal policy * ( ) by the maximization of the expected cumulative reward .
Combined with the idea of adaptive PID control, we investigate the use of off-policy maximum entropy soft actor-critic to train a stochastic policy ( ) that adjust the parameters of the PID controller to ensure the completion of line-following task. Based on previous soft actor-critic algorithm and incremental PID controller, a new hierarchical structure is designed for adaptive SAC-PID control approach. The overall structure is shown in Fig. 1. In this structure, the RL agent plays the role of the upper controller, which can adjust the parameters of the lower controllers in real time according to the current state . The lower controller compensates the error by giving the output to the mobile robot. In this sense, the actions selected by the RL agent are the parameters of the PID controllers, such that the actor produces = ( ) and the critic provides information for training the actor.

FIGURE 1
The overall structure of the SAC-PID control system.

The upper controller design in the SAC-PID control system
Formally, we describe the details of the upper controller manipulating the mobile robot to complete line-following task from the aspects of policy optimization, state representations and reward function.
In the policy optimization phase, function approximators are used for the Q-function,the value function and the policy because of the continuous line-following task. Specifically, function approximators for the Q-function, the value function and the policy are employed instead of running evaluation and improvement to convergence like soft policy iteration, alternated between optimizing both networks with stochastic gradient descent. The parameterized state value function ( ) and soft Q-function ( , ) can be modeled as two expressive neural networks, i.e., value network and critic network. The parameters of these networks are and . Besides, a tractable policy ( | ) can be modeled as a Gaussian distribution with mean and covariance given by actor neural network, where is the parameter of the actor network. Each of the value network, critic network and actor network, in this context, includes an input layer, two hidden layers and an output layer. The rectified linear unit (ReLU) in the hidden layer is adopted as the activation function which maps the input to the output signal. In addition, Replay buffer is constructed to record tuples ( , , , +1 ) and to update the parameters of value network, critic network and policy network until RL agent learns the optimal policy * .
Value network approximates the soft value, which can stabilize training and is convenient to train simultaneously with other networks. Value network approximator is related to critic network and policy according to Eq. 5, and it can be trained to minimize the squared residual error. The objective function of value network is given by where  is the Replay buffer. Actions are sampled from the current policy ( | ). The role of the critic network is to evaluate the current actor and guide the actor to converge to * , which can be trained to minimize the soft bellman residual. The objective function of critic network is presented as where is the exponentially moving average of the value network weights 34 . In particular, two Q-functions are parameterized, with parameters to be trained independently to optimize ( ), where ∈ {1, 2}. In addition, the minimum of the Q-functions is served for value gradient and policy gradient, as proposed by Fujimoto 35 . The actor network, which represents a Gaussian distribution, can be learned by directly minimizing the expected KLdivergence in Eq. 6. Because the sampling actions from the distribution is discrete, and it is impossible to derive and backpropagate to update the network parameters, the reparameterization trick 36 is adopted to obtain actions : where is a noise vector which is sampled from some fixed distribution, in this context, normal distribution is used. ( ) is the mean of Gaussian distribution, and ( ) is the variance of Gaussian distribution. Thus the objective function can be rewritten: For state representations, each state is modeled as normalized representation the coordinates of five pixels, curvature error and the velocities of the mobile robot of the scene at time . The process of calculating the normalized coordinates of five pixels includes image preprocessing, boundary point extraction and centerline calculation.
Firstly, we capture RGB image from fixed-mount camera, resize the raw image to 128 × 72 and then convert it to a binary image. Then the region growing method is used to extract all coordinates of black line boundaries in the binary image. And all the pixel coordinates of the black line are obtained. Specifically, the five pixels are located at the minimum, 1∕4, 2∕4 ,3∕4 and the maximum vertical coordinate values of black line ( 1 , 1 ),( 2 , 2 ), ( 3 , 3 ), ( 4 , 4 ) and ( 5 , 5 ) as shown in Fig. 2(a). Finally, the five pixel coordinate values are normalized to [−1, 1] as the first 10 dimensions of the state given by The value is the deviation value between the black line and the front of the mobile robot at the current moment, where ∈ {1, ⋯ , 5}. For instance, when the point ( 4 , 4 ) is on the same longitudinal horizontal line as the center point of the front of the mobile robot, 4 = 0. The value represents the distance between its corresponding deviation value and the front of (a) Five pixels of black line (b) The fork pixels of intersection FIGURE 2 Path 4 and corresponding performance parameters curves the mobile robot at the current moment, which is convenient for the agent to determine the importance of according to global information during training. However, the mobile robot may encounter the black line crossing in the process of following the line. The traditional region growing algorithm extracts the pixel coordinates of all the boundaries of forks, which causes the black line to deviate from the true value and the mobile robot will turn quickly and lose its stability. In this work, 24-neighborhood threshold method is used to determine whether the boundary pixels are fork pixels. As shown in Fig. 2(b), there are four boundary fork pixels in an intersection, which are 1 , 2 , 3 and 4 .
There are more black pixels in the 24-neighborhood with the fork pixel as the center than the ordinary boundary pixels. Therefore, when extracting the boundary pixels, it is necessary to calculate the sum of all pixel values in the 24-neighborhood of each boundary pixel separately to determine whether it is a fork pixel. The rule of judgment is as follows: Since the region grows from the bottom of the binary image, only 1 and 2 can be judged first. If it is the fork pixel 1 or 2 , the polynomial fits the left and right boundaries to grow to 3 and 4 , and then use 3 and 4 as the starting point for region growth so as to extract the complete left and right boundaries of the black line.
In order to make the mobile robot more stable at the curve line, the curvature error between the center line of the black line and the mobile robot itself is also added as one of the dimensions of the state. To compute this error , we transform the projection of the RGB image to the top view, resize the image to 72 × 72, generate binary image, use region growth and 24neighborhood threshold method to find the center line of the black line, and finally find the curvature of the center line. Thus, can be calculated is given by where is the curvature of the center line of the black line, is the curvature of the mobile robot itself. In addition, considering that the mobile robot changes its velocity with time delay, the linear velocity and angular velocity of the mobile robot at time t are also part of the state. Thus the entire can be expressed as an 13-dimensional vector.
The idea of reward function is designed to incorporate key objectives of this study, which is to develop an automated linefollowing policy centered around stability and efficiency. More specifically, these evaluation indicators are explained as follows: 1) Stability: The mobile robot must always be on line, not out of line.
2) Efficiency: Evaluation of the velocity at which the mobile robot completes a full circle to complete the line-following task.
The reward function ( ) is divided into three items corresponding on the three scenarios where the mobile robot reaches the goal, exceeds the boundary of the line and maintains on the line but not reaches the goal. Specifically, ( ) is defined as is the punishment item. , and are the weights of three items. The stability of the mobile robot is related to the error at each moment. The third item of ( ) is introduced to ensure the center of the mobile robot exactly on the line. Specifically, the closer the center of the mobile robot is to the center of the line, the greater the reward, and vice versa. In terms of efficiency of evaluation indicators, the line-following robot should manage to move to the goal as soon as possible. So when the mobile robot goes out the line or reaches the goal, we will evaluate its average linear velocity as an efficiency indicator. Therefore, the physical quantity of average linear velocity is added to encourage the mobile robot to complete the line-following task at the fastest velocity while ensuring good error correction. In addition, we also added a penalty item to encourage the mobile robot to complete the task. When the mobile robot reaches the goal, we give it positive feedback + , and if it fails in the route, we give it a negative feedback − .

The lower controller design in the SAC-PID control system
The function of the lower controller is to receive the output of actor network in real time, and control the angular velocity of the mobile robot according to the control law of the incremental PID controllers. In order to make the mobile robot more stable when following the line, two parallel PID controllers, namely the main controller and the auxiliary controller, are used to jointly control the angular velocity of the robot. The control law of the main controller is indicated as below where ( ), ( − 1) and ( − 2) can be considered respectively as the 4 value of the , −1 and −2 , ▵ is the output of main controller and , and are the coefficients of the first three dimensions of actor network . The principles for choosing the error are as follows (1) There exits the phenomenon that controlling the mobile robot by the computing device will cause communication lag.
(2) The controller needs to be forward-looking to make the mobile robot follows the path with large curvature accurately.
(3) The center of the mobile robot is encouraged to be at the center of the path as much as possible.
where ( ), ( − 1) and ( − 2) can be considered respectively as the value of , −1 , −2 , ▵ is the output of auxiliary controller and , , are the coefficient of the last three dimensions of actor network . Thus the angular velocity of mobile robot is as follow where is the angular velocity sent to the mobile robot at time , −1 is the angular velocity of the mobile robot at the time − 1, and is the proportional coefficient.
can be reconstructed as follows In the line-following task, the mobile robot has two degrees of freedom, namely linear velocity and angular velocity. In order to make the mobile robot more stable when following the line, the two-degree-of-freedom mobile robot can be decoupled, and the SAC-PID control system controls the angular velocity in real time according to the current state as shown in Eq. 17. In order to ensure the stability of the mobile robot, a slope function can be constructed as below so that the linear velocity of the mobile robot changes with the change of error 4 = − | 4 | + (19) where and are the coefficients that control the linear velocity range of the mobile robot. When | 4 | = 1, the linear velocity of the mobile robot changes to the minimum value. Thus the mobile robot can slowly pass a sharp turn or stabilize in time when the vibration occurs. When | 4 | = 0, the linear velocity of the mobile robot changes to the maximum value. Hence the mobile robot can quickly follow the line when it is stable, saving time and cost. for step=1 to N do 6: Select actions based on current actor network 7: Compute angular velocity according to and action 8: Compute linear velocity according to 9: Apply velocities ( , ) to environment and observe next state +1

Algorithm Statement
The entire algorithm process is divided into three stages, namely initialization, interaction and optimization. In the initialization phase of the algorithm, the parameters of the critic network ( , ), value network ( ) and actor network ( | ) are initialized. At this stage, the previous parameter files of actor network and critic network can be loaded to continue training. In addition, an empty replay buffer to store tuple ( , , , +1 ) can be initialized.
For interaction phase, each episode contains a series of steps, agent interacts with environment from the first step to the last step and then enters into the next episode. Once the agent enters a new episode, it needs to observe the state as the initial state 1 . Then the agent samples an action from current actor network according to the state . The angular velocity and linear velocity of mobile robot can be calculated according to action and state in Eqs. 17 and 19. After it finished changing these velocities, agent will observe the current state as +1 . Then the dynamic transition ( , , , +1 ) is stored in the replay buffer R, which will be used to train the agent.
The optimization phase is the core of the entire algorithm, which updates the network parameters. In the training process, agent randomly samples transitions of the replay buffer to form a minibatch, which can be used for the network update. Firstly, the parameters of the value network can be updated by using the minimum of the Q-functions for value gradient in Eq. 7. Next, two Q-functions are trained independently to optimize ( ). Thirdly, the parameters of the actor network can be updated by using the minimum of Q-functions for policy gradient in Eq. 10. At last, is updated according to exponentially moving average of the current value network weights. , and are the learning rates, and is the target smoothing coefficient. Algorithm ends once all the training episodes have been accomplished. The output of the whole algorithm is the critic network and the actor network obtained after training, and the replay buffer can also be used as the output if necessary.
In order to facilitate readers understanding, the pseudo-code of the proposed SAC-PID control approach is outlined in Algorithm 1 and the more specific structure is given in Fig. 3.

FIGURE 3
The specific structure of the SAC-PID control system.

EXPERIMENTAL RESULTS
In order to verify the robustness and real-time performance of the SAC-PID control method, we trained the policy of the SAC-PID control and tested it on Gazebo simulation platforms. As for generalization, we randomly choose one model trained on the corresponding path to test other paths. Furthermore, the effectiveness of the method was further verified in hardware experiments.

Simulation setup
Before presenting the results of the experiment, some common aspects of all trials are introduced. Firstly the hyperparameters of SAC-PID we employed in the line-following task are listed in Appendix 6.1. Secondly, the coefficients of reward function must provide the agent with enough information and can be implemented in different scenes. Considering that there are time delay of communication when the velocity of the line-following robot changes, the error coefficients 1 , 2 and 3 can be set as 0.7, 0.2 and 0.1. In terms of the overall evaluation, stability is put in the first place, followed by distance traveled and finally average velocity. Thus the overall evaluation coefficients , and are set as 0.5, 0.3 and 0.2. In addition, the penalty score is set to be 20. Finally, for the threshold of the fork pixel, 1 and 2 are set to be 41 and 82. The training process and final test results are implemented by using Pioneer 3-AT on Gazebo simulation platform which is shown in Fig. 4. Pioneer 3-AT has 2 degrees of freedom in total, namely angular velocity and linear velocity . A camera is installed in the front of Pioneer 3-AT as a sensor to collect route information. Specifically, angular velocity and linear velocity of the mobile robot can be calculated in Eqs. 17

Simulation experiments
The whole process of the experiment is divided into two stages: the training phase and the testing phase. We randomly select four different difficulty paths, namely playground circle (Path 1) shown in Fig. 5(a), circle with one fork (Path 2) shown in Fig.  6(a), complex circle with some forks (Path 3) shown in Fig. 7(a) and complex circle with many forks (Path 4) shown in Fig.  8(a). The start and goal points of each path are the same location, which forms a loop. In the training phase, the line-following robot was initialized to the starting point at the beginning of each episode. If the mobile robot reaches the goal or exits the line, it enters the next episode. The success rate is employed as an evaluation index, the number of success times that the robot reached its goal without out of line over the total testing numbers. If the success rates of multiple episodes can stabilize to the same level, the training process is completed. After the training of each path is finished, the error data 4 is served as the evaluation index of the SAC-PID control system stability. In addition, average reward of each episode can also evaluate the stability of the algorithm. In order to make it easier for readers to observe the experimental results, we smoothed raw data of error and success rate respectively to generate smooth error and smooth success rate, which was shown in Figs. 5-8. As shown in Fig. 5(b), the success rate of training on the Path 1 reaches 75% and converges from about 1250th episode. As shown in Fig. 8(b), although the convergence starts from 1400th episode, the success rate reaches 85% for the complex Path 4. The success rates of training on Paths 2 and 3 respectively reach 72% and 73% as shown in Figs. 6(b) and 7(b). The results shows that the SAC-PID control method can converge to the optimal policy in line-following tasks.
As shown in Figs. 5(c)-8(c), the average reward curve of each path appears a steady upward trend without obvious decline and fluctuation, which means the training process is very stable.
The error for the mobile robot is taken as an index to measure the stability of the proposed SAC-PID control method. And the error changes with the movement of the line-following robot. Specifically, the range of error is defined as [−1, 1]. When the error of the mobile robot is not in the above range [−1, 1], the mobile robot is out of the line and the line-following task fails. After   the mobile robot completes the training process, we randomly sampled one of the episodes which successfully completes the line-following task. Then the error of each step in the sampled episode is drawn, as shown in Figs. 5(d)-8(c). Since the error range of each path is concentrated in [−0.6, 0.6] and the oscillation is small, the proposed SAC-PID control approach has good stability.
In the testing phase, we used the trained model. Specifically, the parameters of the trained model cannot be changed and the RL agent does not explore in the testing process. Firstly, we tested four different trained models on their corresponding paths. As shown in Table 1, the success rates are 100% in the simpler routes such as Paths 1 and 2. In complex routes such as Paths 3 and 4, the success rates are more than 85%. Then, in order to verify the generalization of the SAC-PID control method, we selected Model 3 trained in Path 3 to test all other paths with different difficulties. Furthermore, we also employed Model 3 to test the performance of SAC-PID control approach in Test Path, as shown in Fig. 9. As shown in Table 1, the success rates of the test results are above 80%, even reach 100%, which shows that the proposed SAC-PID control method has good generalization and strong robustness. At last, we visualised the original path and the SAC-PID tracking trajectory in Figs. 5(a)-9(a) to observe the effectiveness of the SAC-PID control method. It can be said with certainty that the tracking trajectory of SAC-PID fits the paths except for some paths with larger curvature.

Comparison with fuzzy PID control
To the author's best knowledge, the previous work used traditional PID controllers to complete the line-following task on the specific path. However, because of traditional PID control without the feature of adaptivity, we chose fuzzy PID control as the baseline to compare with the SAC-PID control method. All the details of fuzzy PID control are described in Appendix 6.2. We chose Model 4 as the policy of SAC-PID control to compare with fuzzy PID control on the most difficult Path 4 on Gazebo. The error data is generated by the mobile robot completing the line-following task under the control of SAC-PID and fuzzy PID respectively. To facilitate comparison, the curves shown in Fig. 10 are smooth error curves. The error range of the SAC-PID control is between [ − 0.25, 0.25], which is much smaller than that of fuzzy PID control [ − 1, 0.25]. It is obvious that the stability of SAC-PID control is significantly better than fuzzy PID control. The average linear velocity of the mobile robot controlled by the SAC-PID control is also faster than the velocity under the control of fuzzy PID control, as shown in Table  2. The average velocity gap is caused by the fact that the real-time performance of SAC-PID control is much better than that of fuzzy PID control. In particular, on a computer with an i5-9400 CPU, a Python implementation of SAC-PID (Algorithm 1) takes about 65ms per iteration on line-following tasks. Under the same configuration, fuzzy PID control method needs 670ms per iteration. In order to compare the effectiveness of SAC-PID control and fuzzy PID control, the trajectories of the SAC-PID control and fuzzy PID control are also displayed in Fig. 8(a). It is seen that the performance of the SAC-PID control is much better than that of fuzzy PID control.

FIGURE 10
The error for the SAC-PID control and the fuzzy PID control

Real world tests
We also conducted evaluations on a Mecanum mobile robot equipped with NVIDIA Jetson TX2 and the RGB camera. We employed Model 4 trained on gazebo as the policy parameters of the SAC-PID control to test two paths of different difficulty, as shown in Fig. 11(a) and Fig. 11(c). The quantitative evaluation results are shown in Fig. 11(b), Fig. 11(d) and Table 3. In the experiment we observed the SAC-PID control method has achieved good results in the real environment. The success rate of SAC-PID is as high as 80% in both simple paths without forks and complex paths with forks, which represents SAC-PID control has a certain generalization from simulation to reality. Although the error compensation performance of SAC-PID control method has a certain gap between simulation and reality due to the real mobile robot has a larger time lag than simulation, the SAC-PID control is also qualified for the line-following robot on real mobile robot.

FIGURE 11
The error curves on two real path

CONCLUSION
In this paper, a novel model-free self-adaptive SAC-PID control approach based on reinforcement learning for automatic control of mobile robots is proposed. The new hierarchical structure was developed, which includes the soft actor-critic layer as the upper controller and the incremental PID control layer as the lower controller. The proposed model-free SAC-PID control architecture can adaptively drive the parameter selections of the incremental PID controllers for mobile robots without tedious tuning task. This particular point makes the SAC-PID control method robust for implementations in different systems with different variant, unknown and complex environment. Using the presented approach for line-following robots, the parameters of incremental PID controllers can be adjusted in real time to achieve optimal control. In this sense, the errors between the center of the mobile robot and the following line were compensated in real time. Compared with fuzzy PID control, the strong robustness, generalization and real-time performance of the SAC-PID control approach are verified through simulations of different difficulty paths on Gazebo. Furthermore, the effectiveness of SAC-PID control method is also demonstrated on a mecanum mobile Robot in real world environments. In a more broad sense, since the proposed method is able to control different systems, it can also extend the SAC-PID control approach to different robotic platforms, such as robotic manipulators.

ACKNOWLEDGMENTS
This paper was supported by National Key Research and Development Plan Intelligent Robot Key Project(2018YFB1308402). Table 4 lists the hyperparameters SAC-PID employed in training phase.

Appendix B
The design of the overall structure of the fuzzy PID controller is shown in the Fig. 12. To be specific, fuzzy PID control is composed of two fuzzy controllers with the same control rules and two incremental PID controllers with the same structure as SAC-PID controller. The inputs of the fuzzy controller are ▵ 4 , 4 ( ) and ▵ , ( ) and the outputs are ▵ and ▵ , where ▵ 4 = 4 ( ) − 4 ( − 1), ▵ = ( ) − ( − 1), and are the parameters of two incremental PID controllers that are adjusted by the fuzzy controllers in real time. In addition, the specific parameters of fuzzy PID are shown in Table 5.

FIGURE 12
The structure of fuzzy PID controller.