{'fraction_non_alphanumeric': 0.026679148139480458, 'fraction_numerical': 0.006143224900538264, 'mean_word_length': 4.849760438056126, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Application of Deep Reinforcement Learning in Control of Drawing Robots

In this paper, the drawing robot studied is after the system extracts the edge of the image, convert its pixels to actual physical coordinates, then control the robot to track these coordinate values and realize image drawing. In many cases, the edges extracted by the edge detection algorithm have a lot of unnecessary details and edge lines of many branch points. Here, some techniques are used to remove these redundant and branch points. When the robot is tracking the trajectory, the quality of its control algorithm directly determines its performance. In this paper, the control system is designed using the deep deterministic strategy gradient (DDPG) algorithm in reinforcement learning, focusing on the setting of its reward function. Here we propose a reward function. After simulation testing, the reward function setting meets the system requirements, the controller can achieve high-precision and high-speed trajectory tracking effect, and it can also suppress chattering caused by external interference and model errors.


Introduction
At present, there are a lot of research on humanoid robots. These robots not only increase the human-computer interaction, but also have a series of entertainment purpose, such as educational robots [1] and portrait drawing robots [2]. These robot application scenarios are inseparable from the precise control of the robot. More and more researchers are beginning to pay attention to the control of robots, especially their trajectory planning. In reference [3], the kinematic model of the two link manipulator is established by using the rotating coordinate system, and the trajectory planning and tracking are carried out. In reference [4], a hybrid fuzzy adaptive controller is designed to ensure the performance of robot trajectory tracking. However, in the design of actual robot controllers, it is difficult to design a good controller because the robot, as a dynamic system, has problems such as nonlinearity and parameter uncertainty, so there is a lot of literature about advanced control methods. In reference [5] for the limitation of traditional Jacobian Matrix Pseudo-Inverse (JMPI) method, a Robot Tracking Control Method based on Jacobian Matrix Adaptive (JMA) is proposed. Continuous control strategy based on nonlinear model proposed for flexible robots [6], response control strategy designed for human-machine interaction [7], in high-precision application scenarios, a synovial observer [8] and an ultra-high-order synovial controller [9] have been proposed. For the control scenario of uncertain kinematics and dynamics, an observer-based controller is proposed to achieve exponential tracking control [10]. In recent years, the wave of artificial intelligence has arrived. Artificial intelligence has been applied in various industries and has achieved good results. As a branch of artificial intelligence, reinforcement learning (RL) has also received great attention from researchers. Many researchers have applied reinforcement learning to the design of robot controllers, and have achieved good results. For example, a reinforcement learning control scheme based on neural network model is proposed in reference [11] for unknown parameters and nonlinear robot models，the RL algorithm based on interval fuzzy logic control proposed in reference [12] for rotary steering drilling overcomes the control problems caused by the complexity and uncertainty of control caused by the actual environment. A new policy iteration Q-learning algorithm is proposed in [13]. Literature [14] proposed a tracking algorithm for WMR system based on partial reinforcement learning neural network (PRLNN), in the nonlinear discrete dynamic robot system with slip, literature [15] proposed an adaptive neural tracking algorithm based on reinforcement learning.
Reinforcement learning has a very adaptable nature. After acquiring the system observations, it learns through rewards and punishments to make the robot learn how to perform the designed tasks correctly. When using reinforcement learning to track the robot's trajectory, use an appropriate strategy to map the state to the action, and when training the strategy, consider its convergence and convergence speed. Improper state representation may lead to the decrease of learning speed, and the system may not converge [16]. Another issue worth noting is the "dimensional disaster" problem, which exists in continuous control tasks. In [17], offline training is used to avoid this problem.
The main work done in this paper is as follows. First, extract the edge of the image, perform the preprocessing of branch removal, and then design the controller so that the robot can accurately draw the given image with minimum control force. In order to achieve the above goals, this paper proposes a reward function to optimize the strategy. On this basis, actor network and critic network are designed to train the DDPG agent, and this agent and reward function are used as the robot controller. The simulation results show that the method has good convergence speed and tracking performance, which indicates that the reward function can cooperate with the DDPG agent effectively, and the design of the controller is effective and feasible.

Reinforcement learning
The process of reinforcement learning [18] is that the agent obtains observations from the environment, performs an action according to the strategy, and then gets a reward.
The goal is to design a strategy that maximizes the rewards it receives when it executes an action. The initial state is randomly sampled from the initial state distribution, and the action represents the initial action obtained by executing the strategy. The trajectory is described by the state-action pair, which is , , , ⋯ , in state Go to state can be deterministic， , Where f(•) is the transfer function and P(•) represents the sampling probability. The agent performs actions according to its strategy and exerts influence on the environment.
In addition, the setting of the reward function is particularly important in reinforcement learning. The goal of the agent is to maximize the cumulative reward on a trajectory .
One kind of return is finite-horizon undiscounted return, and its reward accumulation value R (τ) is calculated as follows: represents the reward value obtained at time t. Another reward is infinite-horizon discounted return, ∑ (4) Where 0,1 represents the discount factor. As far as common sense is concerned, the current gain must be better than the past and the future, so it will not directly add up all the rewards. Whatever the desired measurement and strategy, there is only one purpose to maximize the reward received. When discussing the expected return of the trajectory, first consider the probability distribution on the action trajectory. In the case where both the state transition and the strategy are random, the probability of a trajectory of length T is: then the expected return is: The best strategy is the one that maximizes the expected return.

Deep Deterministic Policy Gradient (DDPG)
Deep deterministic Policy gradient [19] is an algorithm for learning Q function and strategy at the same time. Assuming the optimal Q-value function is * , and the optimal action function is * , you need to learn * , and * approximators in DDPG. We construct a main neural network with ∅ parameter in DDPG to approximate the * , value, and then update the parameters of the secondary network with the trained parameters of this network to approximate * .
The Bellman equation of the optimal action value function is as follows: Where ~ is shorthand for saying that the next state, s', is sampled by the environment from a distribution P (·| s, a). This Bellman equation is the starting point of the approximator, in other words, it is the initial state of the neural network we constructed. Now suppose that a set D of state transitions is obtained, and each state transition is represented as (s, a, r, d, s'), where d is the end flag. Then the loss function of the neural network is as follows: The parameter update formula for the target network is as follows: The loss function is： Since the action space of the drawing robot is continuous, and we assume the Q-function is differentiable with respect to action, we can just perform gradient ascent (with respect to policy parameters only) to solve. Another optimization technique of DDPG is to use a circular experience buffer to store past experience, and then the agent randomly obtains a small batch of experience from the buffer to update the critic and actor network.

Image processing
When we try to make the robot draw a picture, we must first find the important feature points or edges of the image. Edge detection technology is an important image processing technology, and many edge detection algorithms [20] are available. We tried various algorithms, and finally found that the image edges extracted by the canny algorithm [21] meet the design requirements.

Location generation
After acquiring the image edge points, the branch points need to be processed. In order for the arm to move smoothly from one end point to another, it is required that there should be no branch points on the edge line, so the branch points need to be removed as follows: (1) After setting the threshold, use the corner detection algorithm [22] to detect the intersection between the image contours，change the pixel of the detected intersection from 1 to 0. (2) Delete the edge lines that cover fewer areas and have fewer pixel points, which reduces the noise impact, prevents the robot from entering the singularity, and removes some unimportant feature points to make the image look cleaner. The next step is to determine the starting pose and the scale factor from which the pixels are converted to actual physical points. In addition, to simplify the control difficulty, the posture is fixed to (0, , 0) when the posture points are generated, and then the pixel points processed by the scale factor are combined with the posture points to form a posture matrix, so the posture points are generated.

Controller design based on reinforcement learning
In the design of the drawing robot controller, the parts outside the controller are treated as a black box. The controller obtains the desired joint position, the actual joint position and the torque from the black box, and then the controller outputs a torque to the black box.
The main purpose of this paper is to train a six-axis robot to use DDPG agent to complete image drawing. It is necessary to construct a critical network and an actor network here. The input of the critic network is observation and action, and the output is a gradient parameter. In this network, it mainly provides an internal estimate of the network performance. In an actor network, the input is observation and the output is action, its update method does not use weighted gradient update, but uses gradient ascent. The relationship between these two networks is like this: First, the environment gives an observation, the agent generates an action value (with noisy) according to the decision made by the actor network, and the environment will give a reward R after receiving this action. Then update the observations, use this reward R to update the critic network, and then update the actor network according to the direction suggested by the critic.   The reward function is the core of the control device. The setting of reward function directly affects the performance of the controller. Set and is the error of the i axis of the robot arm in the x direction and its derivative, and is the error of the i axis of the robot arm in the y direction and its derivative, is the i axis of the robot arm in the x direction speed, is the i axis of the robot arm in the y direction speed, and is the sampling time and the simulation deadline, is the action value of the joint i at the previous time step. Considering that the main trajectory space of the drawing robot is two-dimensional, the end executor of the drawing robot can get a reward as soon as it reaches the specified position and posture point. Because we fixed the pose angle earlier, only the x-axis and y-axis values are considered when designing the reward function. According to continuous testing and parameter adjustment, the following reward function formula is given.
This reward function enables the end effector to reach the desired position by providing a positive incentive for the x-and Y-direction speeds of the arm. In addition, to avoid premature termination of training, a fixed profit value(50T s / T f )is given in the reward function. Why add a decay rate to the cumulative value of past actions here? Intuitively speaking, past actions certainly have less influence on the current state than current actions. Mathematically, the sum of an infinite number of rewards will most likely not converge to a finite value. With decay rates and appropriate constraints, the value of an infinite sum will converge. Eliminate the negative effects of past actions on the current state by subtracting the accumulated amount of past actions. The rest of the formula is undesirable states, such as high errors and trends toward high errors.

Trajectory generation
We use standardized uniform B-spline [23] to fit the trajectory. Here is a simple definition of B-spline, there are n+1 vertices , ⋯ , then the k-order (k-1 degree) B-spline curve is: It's here i, Set three 0 position points in front of the track point, and three repeated position points behind, indicating initialization and end. It takes the form of [0,0,0,0,1,2,3,……m-1,m,m,m,m]. The first three points and the last three points are denoted as

Simulation
In the actual simulation test, the built neural network does not know the dynamics of the robot and the corresponding parameters. In this system, the controller treats the robot as a black box, and the agent interacts with the black box to find an optimal strategy to maximize the profits obtained.
This paper uses the PROBOT Anno robot model for simulation test. It is a 6-axis arm product based on ROS development. The physical parameters are as follows：  6. 6-axis robotic manipulator. Before conducting agent training, some necessary settings need to be made, such as setting the maximum training episode to 6000, 10 iterations per episode, setting the initial weight randomly between (0,1), and setting the initial and end positions of the end executor randomly. Stop training if the average reward in consecutive 100 episodes exceeds 19, and save a copy of the agent when the average cumulative reward in the training set exceeds 20. After every 50 steps, each worker sends experience to the host. DDPG agents require workers to send 'Experiences' to the host, and perform a verification.
We give the loss function graphs of the two networks after the training episode number is 100 and the number of iterations is 1000. It can be seen that actor and critic networks converge rapidly after 100 iterations and stabilize after 400 iterations. In addition, the reward graph after 12,000 iterations after training 1200 sets is shown. It can be seen from the graph that as the number of training episode increases, the reward value is also increasing, and its Q value is also increasing, indicating that we set the reward function It meets the control requirements.   After successful learning, the agent can quickly control the robot arm to reach the desired position. Figure 10 shows the variation of the error between the actual position of the six axes and the expected position. As can be seen from the diagram, the agent can quickly reduce the error, adjust time is less, rise time is fast, and its control performance is stable. Finally, the working effect diagram of the drawing robot is given in Figure 11.

Conclusion
For the design of high-precision controller in drawing robot, this paper uses reinforcement learning to design the controller, which is different from traditional control methods. It can also design a good controller to complete the specified tasks without the control designer knowing the control algorithm and detailed parameters of the robot. In image processing, this paper uses some small tricks to process the input edges, such as removing branch points by transforming the pixel values, and automatically ignoring some edge lines with low coverage. On the controller, all modules except the controller are treated as black boxes, and the DDPG agent is trained to find the optimal strategy by getting the relevant observations from the black boxes. The action produced by this strategy is the optimal action. In addition, the setting of the reward function is one of the keys, which directly affects the performance of the controller. In this paper, with the goal of reducing the speed and position errors of six axes, a reward function is proposed. After simulation testing, it is found that it meets the control requirements.