{'fraction_non_alphanumeric': 0.037437253324136874, 'fraction_numerical': 0.012836724527723492, 'mean_word_length': 4.758605472197705, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Policy Gradient from Demonstration and Curiosity

With reinforcement learning, an agent could learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remained challenging for existing methods, especially in scenarios where the extrinsic feedback was sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations were usually required. In this work, an integrated policy gradient algorithm was proposed to boost exploration and facilitate intrinsic reward learning from only limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated on a range of simulated tasks with sparse extrinsic reward signals where only one single demonstrated trajectory was provided to each task, superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.


Introduction
Over the last decade, reinforcement learning (RL) [20] has achieved impressive success in various applications. Based on experiences collected through interaction with the environment, an agent learned a decision making strategy by means of trial and error. Mnih et al., 2015 [23] trained an agent with Deep Q Networks (DQN) to play Atari games and achieved professional human level performance across a set of 49 games. In 2016, by introducing human knowledge, Monte-Carlo Tree Search (MCTS), and self-play, Silver et al., [18] built the very first agent, AlphaGo, to defeat a professional human Go player. Recently, with the help of large-scale distributed training infrastructure, reinforcement learning has been applied to real-time strategy multiplayer video games, which were thought to be very challenging due to issues like long time horizons, partially observable environment, and high-dimensional state and action spaces. The OpenAI Five (Berner et al., 2019) defeated the Dota 2 world champion in 2019 [3]. And Vinyals et al., 2019 proposed the AlphaStar agent to master the game of StarCraft II and was rated at Grandmaster level [13].
Alongside the tremendous success of reinforcement learning, exploration [11] and reward shaping [19] remained challenging for existing algorithms. The agent struggled to learn especially when the extrinsic reward signals were sparse or the exploration spaces were huge. Recently, reinforcement learning from demonstration has attracted intensive research interest as a promising way to address these problems, however, existing algorithms usually required a tremendous number of high-quality demonstrations or included a human expert in the learning loop, which were often difficult or unavailable.
To this end, an integrated algorithm has been proposed in this work, Policy Gradient from Demonstration and Curiosity (PGfDC), with the aim to facilitate exploration boosting and intrinsic reward learning from limited number of demonstrations in scenarios where the extrinsic reward signals were extremely sparse. The intuitions behind PGfDC were that: during interaction with the environment, when the extrinsic reward signals were sparse or even absent, an agent should imitate the demonstrated behaviors, and when got struggled in states where neither extrinsic reward nor demonstration data were available, an agent should attempt to explore novel states to minimize its uncertainty about the environment. After sufficient number of iterations, the agent could explore the environment on its own.
To facilitate PGfDC, the original extrinsic reward function was reformulated by two additional terms which were derived from demonstration and curiosity, respectively. The demonstration term was established by computing the Jensen-Shannon divergence [9] between the agent's current policy and that of the expert. The concept of occupancy measure was introduced to approximate the policy divergence, by measuring the difference between self-generated data and the expert demonstration. To estimate the curiosity term, a neural network has been implemented to embed the agent's observations and predict the consequences of its actions, uncertainties about the environment were measured to represent the curiosity reward. PGfDC was supposed to leverage expert demonstration and curiosity information to: (1) Reduce required number of demonstrations. (2) Improve exploration efficiency.
(3) Imitate the expert and meanwhile achieve high return. These properties were desired by and could benefit real-world applications, for instance, human robot interaction, autonomous driving, and game AI. Furthermore, PGfDC was compatible with most policy gradient algorithms, e.g., Proximal Policy Optimization (PPO) [17] and Trust Region Policy Optimization (TRPO) [16].In this work, PGfDC was evaluated on a range of grid world environments, where the original extrinsic reward signals were all extremely sparse.

Related work
Curiosity driven exploration. Various work focused on using curiosity to boost learning. Pathak et al., 2017 [14] designed an intrinsic curiosity module (ICM) by formulating curiosity reward as the uncertainty in an agent's ability to predict the consequence of its action. ICM improved exploration efficiency in scenarios where extrinsic reward signals were scarse or even absent. Burda et al., 2018 performed a large-scale study of purely curiosity-driven learning across 54 standard benchmark environments [5]. However, purely curiosity-driven learning might sometimes become infeasible or dangerous in real-world settings. For example, in autonomous driving and human robot interaction, unexpected movements might occur and lead to collisions.
Reinforcement learning from demonstration. Expert demonstrations have been introduced to guide the learning process. Hester et al., 2017 proposed the Deep Q-learning from Demonstrations (DQfD) and stored the demonstrations in experience replay buffer [21]. In Silver et al., 2016 [18], demonstration data was used to pre-train the policy network. Although reinforcement learning from demonstration had the potential to relieve exploration dilemma in sparse reward scenarios, existing algorithms tended to require a tremendous number of high-quality data but often failed to fully leverage the value of the demonstrations.
Inverse reinforcement learning. Inverse reinforcement learning (IRL) or inverse optimal control (IOC) have provided a set of algorithms to directly learn the reward functions from demonstrations, as in Ng et al., 2000 [12], Abbeel et al., 2004 [15], Ziebart et al., 2008 [24], and Finn et al., 2016 [6]. However, it was difficult to make an IRL algorithm effective since: (1) IRL asked for a number of high-quality expert demonstrations. (2) IRL was inherently underdefined as different reward functions might result in similar behaviors.
Reward learning from preference. A large amount of work have been conducted on reinforcement learning from human preferences or ratings. Christiano et al., 2017 explored learning objectives defined in terms of human preferences between pairs of trajectory segments, and demonstrated the effectiveness of the method on Atari games and simulated robot locomotion without access to the extrinsic reward signals [8]. In Ibarz et al., 2018 [1], expert demonstrations and trajectory preferences were combined, where a reward function was learned from the preferences and the demonstrations were used by a DQfD algorithm. However, preference learning might get struggled when encountered with tasks where qualified experts were not available. Moreover, the number and the quality of preferences required by an agent grew with the complexity of environments, making the learning process inefficient and sometimes even intractable.

Markov decision process
In this work, the problems considered were under the standard Markov Decision Process (MDP) setting. An MDP was formalized by the tuple: S, A, r, p 0 , T, γ , where S and A represented for the state space and action space, r = r(s, a, s ) was the reward function, p 0 was the probability distribution of the initial state, T = T (s |s, a) denoted the transition function of the environment, and γ ∈ (0, 1) was the discount factor. An agent interacted with the environment over time based on policy π(a|s), mapping state to action probability. At time step t, the agent received s t from the state space S, selected a t from the action space A according to π(a t |s t ), transitioned to the next state s t+1 based on T = T (s t+1 |s t , a t ), and received a scalar reward signal r t = r(s t , a t , s t+1 ). The discounted return was R t = ∞ k=0 γ k r t+k , and expectation of R t was usually evaluated to reflect performance of the policy π: where (s 0 , a 0 , s 1 , a 1 , s 2 , ...) was a trajectory generated from interaction with the environment. Correspondingly, the value function could be defined as V π (s) = E π [R t |s t = s], the action value function was Q π (s, a) = E π [R t |s t = s, a t = a], and the advantage function was A π (s, a) = Q π (s, a) − V π (s). The objective of RL algorithms was to discover policies maximizing the expectation of discounted return E π [R t ].

Policy gradient
Unlike value-based reinforcement learning, the policy gradient methods directly modelled and optimized the policy π θ (a|s) parameterized by θ. And the learning objective was defined: where J(π θ ) could be used to measure the performance of policy π θ (a|s), where d π θ (s) represented the stationary distribution of Markov chain for π θ (a|s). According to the policy gradient theorem: where θ could be optimized via gradient ascent. To solve ∇ θ J(π θ ), Q π θ (s, a) should be evaluated. Normally, Q π θ (s, a) could be computed with methods like Monte-Carlo estimation (REINFORCE), Temporal-Difference learning, or with an auxiliary critic model (actor-critic policy gradient). Furthermore, to reduce variance, the advantage function A π θ (s, a) was often used to substitute Q π θ (s, a),

Methodology
With the widespread use and advances of RL, the significance and difficulty of exploration and reward design have been highlighted. In real-world scenarios, the extrinsic reward signal was usually extremely sparse and hard to be reshaped, which affected the exploration efficiency. Introducing demonstrations or curiosity has proven to be effective in sparse reward settings. The demonstrations were often exploited in the following ways: (1) Store in the experience replay buffer.
(2) Pre-train the policy network. (3) Infer an intrinsic reward function. And the curiosity was deployed to encourage the agent to explore novel states or perform actions to reduce its uncertainty about the environment dynamics. In this work, to fully leverage demonstration data and curiosity, the above two ideas were combined to formulate a new policy gradient method which was boosted from both demonstration and curiosity (PGfDC). PGfDC was supposed to: (1) Require limited number of demonstrations. (2) Figure 1: Workflow of the PGfDC algorithm.
Guarantee superior exploration efficiency. (3) Imitate the expert and meanwhile achieve high return, which was desired in areas like human robot interaction, autonomous driving, and game AI.
The overall workflow of the proposed PGfDC algorithm was shown in Figure. 1. There were two sub-modules, namely the normal RL module and the intrinsic reward learner. For the reinforcement learning module, the agent interacted with the environment and received reward signals estimated from the current reward function r k : where r e was the original extrinsic reward function of the environment, r k d represented the intrinsic reward function learned from demonstrations at the k th iteration, r k c was the intrinsic reward function learned from curiosity at the k th iteration, and λ d and λ c were the corresponding weighting coefficients. The collected interaction data was stored as s, a, r, s and sent to the intrinsic reward learner. Within the intrinsic reward learner, the discriminator network was updated with pre-stored expert demonstrations and the interaction data, and the curiosity network was simultaneously optimized with gradients computed from the collected interaction data. Then the reward function was updated to r k+1 . The intrinsic reward learner could work synchronously or asynchronously [10] with the standard reinforcement learning module. Details of PGfDC were summarized in the appendix.

Reward learning from demonstration
Reinforcement learning from demonstration has proved to be an efficient and intuitive way of transferring expert's knowledge and preference to the agent. The agent could either infer a reward function from the demonstrations as in inverse reinforcement learning, or boost its exploration through a pre-trained policy. However, existing methods usually asked for a tremendous number of high-quality demonstration data while failed to fully exploit the value of the data. To address these issues, the demonstrations were used to formulate an additional penalty term to the original learning objective in this work, measuring the Jensen-Shannon divergence between the current policy π θ (a|s) and the demonstrations. Specifically, suppose the provided demonstrations were expressed as , and D E was generated from an implicit expert policy π E . Then the reformulated learning objective was obtained: where λ d ∈ (0, 1) was the weighting coefficient. It was impossible to directly estimate D JS (π θ , π E ) as π E was unknown, thus the concept of occupancy measure was introduced to approximate D JS (π θ , π E ). Definition 1. (Occupancy measure) Let ρ π θ (s) : S → R denote the unnormalized distribution of state visitation by following policy π θ in the environment, ρ π θ (s) = ∞ t=0 γ t P (s t = s|π θ ), then the unnormalized distribution of state-action pairs ρ π θ (s, a) = ρ π θ (s)π θ (a|s) was termed occupancy measure of policy π θ .
According to Theorem 2 of (Syed et al., 2008 [22]), π θ was the only policy whose occupancy measure was ρ π θ , given that ρ π θ was the occupancy measure for π θ (a|s) = ρπ θ (s,a) a ρπ θ (s,a ) . Therefore, the Jensen-Shannon divergence between π θ and π E could be substituted by: Kang et al., 2018 [4] derived a lower bound for D JS (ρ π θ , ρ π E ), which could be reformulated as: where D w (s, a) : S × A → (0, 1), and w was the parameters. Actually, the right side of Equation (5) could be viewed as the learning objective of discriminator in Generative Adversarial Network (GAN), with π θ working as the generator. In order to train D w (s, a), state-action pairs from ρ π E were labeled as true, while the state-action pairs generated by ρ π θ were labeled as false. Substitute Equation (5) into Equation (3), the following learning objective was obtained: which was equivalent to: Furthermore, Equation (7) could be re-organized as: In Equation (8), the original reward function was reshaped by λ d (logD w (s, a) − 1), as a constant −1 could be removed. Thus, provided with input tuple s, a, s , the demonstration reward was:

Reward learning from curiosity
Following [14], curiosity in PGfDC consisted of three sub-modules: feature embedding G e , inverse model G i , and the forward model G f . In G e , the input state s ∈ S was encoded as a feature vector F (s). Then the feature vectors of two consequent states, F (s t ) and F (s t+1 ) were concatenated and fed into G i to generate prediction for the action a t taken by the agent to move from s t to s t+1 . G e and G i could be combined to formulate a joint model: where θ ei was the network parameters and was optimized through minimizing L ei ( a t , a t ). As discrete actions were used in this work, L ei could be cross-entropy. For the forward model G f , feature vector F (s t ) and the corresponding action a t were taken as the input to predict feature vector F (s t+1 ) of the state at next time step: where the network parameters θ f were optimized by minimizing the mean squared loss function L f ( F (s t+1 ), F (s t+1 )) = 1 2 F (s t+1 ) − F (s t+1 ) 2 2 . In this work, θ ei and θ f were jointly updated and the loss functions L ei and L f were combined and formulated as: where β was the controlling weighting factor and β ∈ (0, 1). The training data was collected while the agent was interacting with the environment and was stored in the tuple s t , a t , s t+1 . L f was used to calculate the curiosity reward, and a transformation function e z −1 e z +1 was applied to L f to scale it to the range of [0, 1]. Therefore, given the input tuple s, a, s , the curiosity reward was: Figure. 2 illustrated the workflow of the curiosity reward module.

Experimental Evaluation
In this section, performance of the proposed PGfDC algorithm was empirically evaluated on the following aspects: (1) Given limited number of demonstrations, could PGfDC guarantee superior exploration efficiency. (2) Considering the extremely sparse extrinsic reward of environments, could PGfDC guarantee high return at convergence. (3) Given demonstrations from the expert, could PGfDC imitate the expert's behavioral preference and meanwhile achieve high empirical return.

Environment settings
In order to comprehensively investigate performance of the proposed algorithm, three variants of the grid world environment have been designed and implemented [7]: (1) An empty grid world with the size of 14 × 14, where the agent was required to reach for the terminal state (14 × 14 − GridW orld, Figure. Figure. 3(c)). For all of the three environments, a sparse extrinsic reward where n t was the number of time steps taken by the agent, and N max denoted the maximum number of time steps. For each environment, only one single demonstrated trajectory was provided to the agent: τ = {s t , a t } T t=0 , and the PGfDC algorithm was compared with four baselines: (1) A human expert. (2) Training the policy with Advantage Actor Critic (A2C) [2]. (3) Training the policy with Proximal Policy Optimization (PPO). (4) A random policy. In the following subsections, implementation details of PGfDC were briefly overviewed, including the policy network, the discriminator, and the curiosity.

Policy network architecture
A2C, PPO, and PGfDC shared the same policy network architecture, where the input state s t was passed through three consecutive convolution layers, with filter numbers of 16, 32, and 64, respectively, and the kernel size was 2 × 2. A rectified linear unit (RELU) was used after each convolution layer, and a max pooling operation with the size of 2 × 2 was deployed after the first convolution layer. The output of the last convolution layer was flattened to be fed into two separate fully connected layers to predict the action probability distribution and the value function, where each fully connected layer had the size of 64 and a Tanh activation function after it. To get the action probability distribution, another fully connected layer with size equivalent as that of the action space was implemented, followed by a softmax operation. On the other hand, an output layer with 1 hidden unit was used to predict the value function.

Discriminator network architecture
The input action a t was passed through a 2-layer MLP with RELU activations and 16 and 8 hidden units correspondingly to obtain the action feature vector. The input state s t was passed through a sequence of two convolution layers with filter numbers of 16 and 32, a RELU and a max pooling operation with size of 2 × 2 were deployed after each convolution layer. The output of the last convolution layer was flattened and fed into a 2-layer MLP with RELU activations and 16 and 4 hidden units correspondingly to obtain the state feature vector. The action feature vector and the state feature vector were concatenated and passed through a fully connected layer with size of 4, followed by a RELU. To predict the discriminator reward r d , an output layer with 1 hidden unit and a sigmoid activation function was used. Learning rate of the discriminator was set to be 10 −3 for all of the three environments.

Curiosity network architecture
Adapted from [14], architecture of the curiosity network was illustrated in Figure. 2. The curiosity module was composed of three components: the feature embedding G e , the inverse model G i , and the forward model G f . The feature embedding mapped the input states s t and s t+1 into feature vectors F (s t ) and F (s t+1 ) with a sequence of four convolution layers with the same filter number of 16 and kernel size of 3 × 3, an ELU activation function was used after each convolution layer. The output of the last convolution layer was flattened to generate a 32-dimension feature vector. For the inverse model, F (s t ) and F (s t+1 ) were concatenated and passed through a fully connected layer with RELU activation and 64 hidden units, followed by an output layer activated by the sigmoid function to predict the action. In the forward model, the embedded feature vector F (s t ) and action a t were concatenated and fed into a fully connected layer with 128 hidden units and the RELU, followed by an output layer with 32 hidden units to predict the feature vector of s t+1 , F (s t+1 ). A transformation function e z −1 e z +1 was applied to the output of the curiosity network z to scale it to the range of (0, 1) and obtain the curiosity reward r c , thus r c had the same scale as that of r d . For all the environments, learning rate of the curiosity was set to be 10 −3 , and β was 10 −2 . Figure. 3 showed the three experimental environments, and the corresponding learning curves were depicted in Figure. 4. Only one single demonstrated trajectory was provided to the agent in each environment. Table. 1 summarized the hyperparameters adopted by PGfDC. Figure. Figure. 4(c). In this task, the proposed PGfDC started to converge at 300 th iteration approximately, however, neither PPO nor A2C converged within 460 iterations in this task.

Experimental results
Evaluation of the learned policies was performed using 10 different random seeds, and for each seed, the average return was computed with 10 independent rollout episodes. The evaluation results were shown in Figure.  The abovementioned experimental evaluations have validated that: (1) Given one single demonstrated trajectory, the proposed PGfDC algorithm succeeded to converge at much higher exploration efficiency compared with A2C and PPO in all three environments. (2) Considering the extremely sparse extrinsic reward of the tasks, PGfDC successfully achieved higher average return against A2C and PPO in all of the tasks. In order to investigate the third aspect of the proposed algorithm, which was: provided with demonstrations from the expert, could PGfDC imitate the expert's behavior while achieve high return at the same time, the 14 × 14 − GridW orld environment was used as the testbed.
Two independent and distinct demonstrated trajectories were provided to the agent separately to facilitate two independent experimental runs, and thus two discriminators have been learned with the demonstrations to determine whether the input sample came from the expert or not (based on Equation (5)). The experimental results were shown in Figure. 5(b), and the average discriminator score, E π θ [D w (s, a)], was calculated to reflect the similarity between expert and the policy. Again, the evaluations were performed using 10 different random seeds, and for each seed, the average discriminator score was computed with 10 independent rollout episodes. Provided with demonstration-1, PGfDC achieved 0.859 on average. As comparison, the discriminator scored the expert for 0.910, while A2C and PPO only received 0.121 and 0.137, respectively, given that the random policy achieved an average discriminator score of 0.135. With demonstration-2, the proposed PGfDC algorithm achieved an average score of 0.550, the expert's average score was 0.586, while A2C, PPO and the random policy received 0.162, 0.156, and 0.194, respectively. Therefore, the experimental results have validated that the proposed algorithm had the potential of imitating expert and meanwhile achieving considerably high return.

Conclusion
Alongside the development of reinforcement learning algorithms, reward shaping and exploration remained challenging for existing methods. An agent might struggle to discover useful information, especially when interacting with environment where the extrinsic feedback was sparse. An integrated algorithm has been developed in this work, the PGfDC, with the purpose to boost exploration and facilitate intrinsic reward learning in sparse reward scenarios from only limited number of demonstrations. In PGfDC, the original reward function was reformulated by two additional terms, r d and r c , where r d was the intrinsic reward learned from demonstrations with a discriminator network, and r c represented the intrinsic reward signal derived from curiosity. To comprehensively evaluate the performance of PGfDC, three grid world-like environments have been designed, where the original extrinsic reward was extremely sparse. For each environment, only one single demonstrated trajectory was provided. The experimental results validated that: (1) Provided with limited number of demonstrations, PGfDC could guarantee superior exploration efficiency. (2) PGfDC could achieve high return in sparse reward scenarios. (3) PGfDC could imitate the expert's behavioral preference and meanwhile achieve high empirical return. Details of the implementation were given. And in theory, PGfDC was compatible with most policy gradient algorithms. Future work would be focused on extending PGfDC to real-world applications, e.g., computer games and human robot interaction.