{'fraction_non_alphanumeric': 0.038365137185442634, 'fraction_numerical': 0.01577381946997293, 'mean_word_length': 4.662062440870388, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowledge-Grounded Reinforcement Learning

Receiving knowledge, abiding by laws, and being aware of regulations are common behaviors in human society. Bearing in mind that reinforcement learning (RL) algorithms benefit from mimicking humanity, in this work, we propose that an RL agent can act on external guidance in both its learning process and model deployment, making the agent more socially acceptable. We introduce the concept, Knowledge-Grounded RL (KGRL), with a formal definition that an agent learns to follow external guidelines and develop its own policy. Moving towards the goal of KGRL, we propose a novel actor model with an embedding-based attention mechanism that can attend to either a learnable internal policy or external knowledge. The proposed method is orthogonal to training algorithms, and the external knowledge can be flexibly recomposed, rearranged, and reused in both training and inference stages. Through experiments on tasks with discrete and continuous action space, our KGRL agent is shown to be more sample efficient and generalizable, and it has flexibly rearrangeable knowledge embeddings and interpretable behaviors.


INTRODUCTION
Incorporating external guidance into learning is a commonly seen behavior among humans. We can speed up our learning process by referring to useful suggestions. At the same time, we beware of external regulations for safe and ethical reasons. On top of following external guidance, we humans can learn our own strategies to complete a task. We can also arbitrarily recompose, rearrange, and reuse those strategies and external guidelines to solve a new task and adapt to environmental changes. Imitating human behaviors has been shown to benefit reinforcement learning (RL) (Billard et al., 2016;Sutton and Barto, 2018;. However, how an RL agent can achieve the above capabilities remains challenging. Different approaches have been proposed to develop some of these abilities. One branch of previous work in RL has studied how an agent can learn from demonstrations provided externally as examples of completing a task (Ross et al., 2011;Rajeswaran et al., 2017;Duan et al., 2017;Nair et al., 2018;Goecks et al., 2019;Ding et al., 2019). Another branch of previous research has investigated how an agent can learn reusable policies. This branch of research includes (1) transferring knowledge among tasks with similar difficulty (Parisotto et al., 2015;Yin and Pan, 2017;Gupta et al., 2018a;Liu et al., 2019;Tao et al., 2021) and (2) learning multiple reusable skills to solve a complex task in a divideand-conquer manner (Bacon et al., 2017;Frans et al., 2017;Nachum et al., 2018a;Eysenbach et al., 2018;Kim et al., 2021;Tseng et al., 2021). These methods allow an agent to learn a new task with fewer training samples. However, demonstrations are too task-specific to be reused in a new task, and incorporating external guidelines from different sources into existing knowledge-reuse frameworks is not straightforward. Moreover, current learning-from-demonstration and knowledge-reuse approaches lack the flexibility to rearrange and recompose different demonstrations or knowledge, so they cannot dynamically adapt to environmental changes.
In this work, we introduce Knowledge Grounded Reinforcement Learning (KGRL), a novel problem with the following goal: An RL agent can learn its own policy (knowledge) while referring to external knowledge. Meanwhile, all knowledge can be arbitrarily recomposed, rearranged, and reused anytime in the learning and inference stages. A KGRL problem simultaneously considers the following three questions: (1) How can an agent follow a set of external knowledge from different sources? (2) How can an agent efficiently learn new knowledge by referring to external ones? (3) What is a proper representation of knowledge such that an agent can dynamically recompose, rearrange, and reuse knowledge in both the learning process and model deployment?
To address the challenges in KGRL, we propose a simple yet effective actor model with an embeddingbased attention mechanism. This model is orthogonal to training algorithms and easy to implement. In this model, each external-knowledge policy is paired with one learnable embedding. An internalknowledge policy, which the agent learns by itself, is also paired with one learnable embedding. Then another learnable query embedding performs an attention mechanism by attending to each knowledge embedding and then decides which knowledge to follow. With this attention mechanism, the agent can learn new skills by following a set of external guidance. In addition, the knowledge and query embeddings design disentangles each knowledge from the selection mechanism, so a set of knowledge can be dynamically recomposed or rearranged. Finally, all knowledge policies are encoded into a joint embedding space, so they can be reused once their embeddings are appropriately learned.
We evaluate our proposed KGRL method for grid navigation and robotic manipulation tasks. The results demonstrate that our method answers all the questions considered in KGRL. In analyses, we show that the proposed approach achieves sample efficiency, generalizability, compositionality, and incrementality, which are the four essential components of efficient learning (Kaelbling, 2020). At the same time, our method also enables behavior interpretability because of the embedding-based attention mechanism.
Our contributions are: • We introduce KGRL, a novel RL problem addressing how an agent can repeatedly refer to an arbitrary set of external knowledge for learning new skills. • We propose a novel actor model that achieves the goals of KGRL and is orthogonal to training algorithms. • We demonstrate in the experiments that our KGRL method satisfies the four properties of efficient learning (Kaelbling, 2020) and learns interpretable strategies.

RELATED WORK
Several lines of research in RL have studied how an agent can incorporate external demonstrations/regulations into learning or learn reusable policies. Here we summarize the differences between their research goals and the ones in the KGRL formulation.
Learning from demonstrations (LfD) and imitation learning (IL). Providing expert demonstrations to an RL agent is a popular way to incorporate external guidance into learning. Demonstrations are (sub-)optimal examples of completing a task and represented as state-action pairs. Previous work has leveraged demonstrations by introducing them into the policy-update steps of RL Rajeswaran et al., 2017;Nair et al., 2018;Pfeiffer et al., 2018;Goecks et al., 2019) or performing IL (Abbeel and Ng, 2004;Ross et al., 2011;Ho and Ermon, 2016;Duan et al., 2017;Peng et al., 2018a;b;Ding et al., 2019). LfD and IL require sufficient number of optimal demonstrations, which are difficult to collect, to achieve sample-efficient learning. Moreover, those demonstrations are task-dependent and can hardly be reused in other tasks. Compared to LfD and IL, KGRL can achieve efficient learning by referring to highly imperfect knowledge. Also, KGRL can incorporate not only task-related knowledge but also knowledge of commonsense, principles, or regulations to move toward safe and ethical learning. Finally, all knowledge, including the one learned by the agent itself, can be reused in new tasks.
Safe RL. Other than demonstrations, regulations are also essential guidance that can help an agent achieve effective learning. Safe RL, which incorporates safety regulations (constraints) into learning and inference stages, has gained more attention over the past decade. Current research in safe RL mainly includes safety in learning through (1) jointly optimizing the expected return and safe-related cost (Ammar et al., 2015;Achiam et al., 2017;Chow et al., 2018;Stooke et al., 2020;Ding et al., 2021) and (2) restricting exploration in a safe space (Garcia and Fernández, 2012;Berkenkamp et al., 2017;Alshiekh et al., 2018;Cheng et al., 2019;Thananjeyan et al., 2021;Liu et al., 2022). The constraints in safe RL can be softly or strictly imposed, where the former aims to decrease the number of safety violations, and the latter prohibits any of them in both learning and inference stages. Compared to KGRL, safe RL usually does not consider reusing the safety constraints in a new task or dynamically adjusting regulations according to the environmental changes.
Knowledge transfer and meta-RL. How an agent can transfer previously learned knowledge to a new task is a well-recognized challenge. Several studies have focused on transferring the knowledge learned in a single task to a multi-task policy (Parisotto et al., 2015;Rusu et al., 2015;Yin and Pan, 2017;Xu et al., 2020;Tao et al., 2021). On the other hand, a line of research in meta-RL investigates how to quickly adapt the knowledge learned in multiple tasks to a new one (Finn et al., 2017;Gupta et al., 2018a;b;Nagabandi et al., 2018;Liu et al., 2019). These two research directions do not enable incorporating external guidance from different sources into learning, so their transferred knowledge is limited to the one learned among similar environments (Glatt et al., 2016). In addition, they require extensive amount of training data before a set of knowledge can be reused in a new task. Compared to previous work in knowledge transfer and meta-RL, KGRL allows an agent to follow and reuse an arbitrary set of knowledge obtained from different sources in a new task, and these sources can be very different from the new task. Moreover, our KGRL approach does not require a separate process, such as policy distillation (Rusu et al., 2015), or numerous training samples collected from multiple environments to learn transferable knowledge.
Hierarchical RL (HRL). Other than transferring knowledge among tasks with similar difficulty, another popular line of research, HRL, aims to decompose a complex task into a hierarchy of subtasks and learn a reusable skill for each subtask. Previous research in HRL has focused on learning a hierarchy of a complex task (Dayan and Hinton, 1992;Sutton et al., 1999;Kulkarni et al., 2016;Levy et al., 2017;Nachum et al., 2018b;Jiang et al., 2019), discovering options/subtasks/subgoals (Stolle and Precup, 2002;Bacon et al., 2017;Vezhnevets et al., 2017;Nachum et al., 2018a;Riemer et al., 2018;Eysenbach et al., 2018;Khetarpal and Precup, 2019;Zhang and Whiteson, 2019;Kim et al., 2021), and transferring learned skills among different tasks (Konidaris and Barto, 2006;Frans et al., 2017;Tessler et al., 2017;Peng et al., 2019;Qureshi et al., 2019;Tseng et al., 2021). Compared to the skills in HRL, knowledge in KGRL can be partially contained in, be more complex than, or even has little to do with the task. That is, it is not limited to a skill that solves a subtask. This property enables flexibility of providing knowledge and improves exploration in an unseen environment. In addition, a skill in HRL is usually assigned with a fixed index or a task-dependant subgoal, making an agent difficult to dynamically recompose or rearrange skills in learning or inference stages. On the other hand, our KGRL method disentangles knowledge representation from the high-level selection mechanism of a task, making an agent able to dynamically adapt to environmental changes anytime.
Program-guided RL. Similar to HRL, program-guided RL also decomposes a complex task into subtasks and learns subpolicies for them. However, instead of learning a hierarchy of the task, program-guided RL guides an agent to complete a task with a program, which explicitly specifies the flow of executing the subtasks given the environmental conditions (Sun et al., 2019;Brooks et al., 2021;Trivedi et al., 2021;Yang et al., 2021;Zhao et al., 2021). This guidance requires a structured and unambiguous hierarchy of a task and is more task-dependent compared to knowledge in KGRL. Moreover, to reuse a previously learned subpolicy, program-guided RL needs to first generate a new program with a new control flow, hence lacking the flexibility to dynamically recompose and rearrange subpolicies.

KGRL PROBLEM FORMULATION
In this section, we formally define the problem formulation of Knowledge-Grounded Reinforcement Learning (KGRL). We consider the family of discrete-time finite-horizon Markov Decision Processes (MDPs) with knowledge guidance for an environment. Such an MDP is formulated as a tuple (S, A, T , G, R, ρ, γ), where S is the state space, A is the action space, T : S × A × S → R is the transition probability distribution, G is the set of high-level knowledge, R is the reward function, ρ is the initial state distribution, and γ is the discount factor. Each knowledge in the high-level knowledge set, g ∈ G, can be any commonsense, suggestion, guideline, principle, specification, rule, restriction, or previously learned knowledge described in formal or informal language. Note that each g ∈ G will be transformed to a low-level knowledge mapping π g : S → A, which describes its low-level execution strategy.
The goal of a KGRL agent is to maximize the expected rewards and develop the following attributes: (1) The agent learns its own knowledge by referring to external one. (2) The agent can recompose/rearrange/reuse knowledge in training and inference stages as well as new environments. (3) The agent's behaviors are interpretable in terms of which knowledge it refers to at each time step.

PROPOSED APPROACH
We propose a novel actor model for KGRL with the concept inspired by the self-attention mechanism (Vaswani et al., 2017). The actor model includes an inner actor and a set of external knowledge mappings, {π g1 , π g2 , . . . , π gn }. The actor architecture is illustrated in Figure 1. The idea behind this design is that the actor will decide to follow an external guidance or its own policy given a state s t ∈ S at the t-th time step. This dynamic decision making should eventually lead the agent to receive higher accumulated rewards based on the reward function R. Moreover, the external knowledge set can be recomposed/rearranged/reused anytime during training and testing phases. In the following sections, we introduce the detailed design of each component in the proposed actor model.

ACTOR MODEL ARCHITECTURE
External Knowledge. An important property of a KGRL policy is the flexibility of knowledge recomposition/rearrangement/reusability during training and testing phases. To obtain this flexibility, we propose a uniform representation for each knowledge. We learn a key (embedding) for each external knowledge mapping, which is a d k -dimensional vector denoted as k gj ∈ R d k , for j = 1, 2, . . . , n. Note that the knowledge key k gj is for the entire knowledge mapping π gj instead of each state-action pair suggested by this mapping. Since each knowledge is encoded as an independent key, which lies in a joint embedding space, one can change the number of knowledge by adding/removing keys to/from the actor model any time during training and testing. Meanwhile, one can also combine knowledge keys from different tasks in a new learning/evaluation process.

Inner
Actor. An inner actor contains two function approximators φ in and φ k . Given a state s t , the function approximator φ in predicts an inner policy, π in = φ in (s t ) : S → A, and φ k generates an inner key, k in = φ k (s t ) ∈ R d k . The inner policy serves the same purpose as a policy in regular RL, which represents the strategy that an agent learns by itself through interactions with the environment. Similar to an external knowledge mapping π gi , the inner policy π in can be viewed as the internal knowledge mapping with a paired learnable inner key k in . Note that unlike a knowledge key, the inner key varies according to a given state, because the inner strategy will change over time.
Query. To decide which inner/external knowledge to refer to, we add another function approximator φ u that generates a query vector, denoted as u = φ u (s t ) ∈ R d k , given a state s t . The query together Algorithm 1: Knowledge-Guided Reinforcement Learning (KGRL) with Our Actor Model Input: environment E with MDP (S, A, T , G, R, ρ, γ) Input: external knowledge mappings π g1 , π g2 , ..., π gn initialize k gj for j = 1, 2, ..., n initialize φ in , φ k , φ u for each time step do Generate k g1 , . . . , k gn , k in , π in , u Compute w in , w g1 , . . . , w gn according to equation (1) Sample an action a t ∼ π(·|s t ) according to equations (2) Observe a new state and reward s t+1 , r t from E Update φ in , φ k , φ u , k g1 , . . . , k gn by any RL algorithm return φ in , φ k , φ u , k g1 , . . . , k gn with the keys will be used to calculate the weights for each knowledge mapping. Note that the proposed model predicts the query and keys separately instead of directly predicting the weight for each guidance. In this way, we disentangle the decision component (the query) and the knowledge representations (the keys). The following section describes how we compute the weights.

ACTION PREDICTION
Attentions. We compute the attention of the query u over the inner keys and external knowledge keys by their dot products. The results become the weights for the inner and external guidance: If w in is larger, then the agent is more likely to adopt the inner policy. On the other hand, if w gj is larger, then the agent is more likely to follow the guidance from the j-th external knowledge.
Final Policy. A straightforward way to describe the final policy given the weights w in and w gj and their corresponding knowledge mappings π in and π gj , for j = 1, 2, . . . , n, is π(·|s t ) = w in π in (·|s t ) + n j=1 w gj π gj (·|s t ), which is a weighted sum of each strategy. For a stochastic policy in a continuous action space, equation (3) becomes the probabilistic density function of a Gaussian Mixture Model (GMM). To sample an action from a GMM that makes the entire actor model differentiable, we first sample an element from the set {in, g 1 , g 2 , . . . , g n } according to the weights {w in , w g1 , w g2 , . . . , w gn } using Gumbel softmax (Jang et al., 2016): Then given a state s t , an action is sampled from the knowledge mapping π e (·|s) using the reparameterization trick, where φ e can be one of {φ in , φ g1 , . . . , φ gn } according to the sampled e.

LEARNING ALGORITHMS
The proposed actor model can be easily learned with any existing RL algorithm, and the implementation is simple. Algorithm 1 describes our proposed KGRL algorithm.

EXPERIMENTS
We validate the proposed KGRL actor model on two sets of widely used environments with different properties: MiniGrid (Chevalier-Boisvert et al., 2018) and OpenAI-Robotics (Plappert et al., 2018). Through experiments on these environments, we answer the following four questions: (1) Can our KGRL agent learn effective new knowledge by referring to an arbitrary set of external guidance? (2) Can the knowledge policies and their learned embeddings be recomposed, rearranged, and reused to solve a new task? (3) Is our KGRL method sample efficient and generalizable? (4) Does our KGRL agent learn interpretable behaviors? The implementation details can be found in Appendix.

MINIGRID
Environments. We evaluate the proposed method in the following MiniGrid environments with discrete action space and sparse rewards: Empty-5x5, Empty-Random-5x5, Empty-16x16, Unlock, DoorKey-5x5, and DoorKey-8x8. Among them, DoorKey are the environments that an agent needs to pick up a key, open a door, and then reach a goal, which can use the skills learned in Unlock (pick up a key and open a door) and Empty (reach a goal). We choose these environments to study if our method can reuse previously learned knowledge. The observation space of these environments is the directed first person view with a range of 5x5 grid, and the action space has seven discrete actions.
Setup. We compare Proximal Policy Optimization algorithm   Results. Figure 2 shows the training results of PPO and PPO-KGRL with different knowledge combinations. Compared to PPO, PPO-KGRL achieves significantly better sample efficiency in five of the six environments. Also in general, the improvement is larger when the task is more complex (Empty < Unlock < DoorKey). Note that PPO does not show any success in DoorKey-8x8 within three million steps. However, we found that in Empty-16x16, our KGRL agent learns more quickly at the beginning but slows down its progress over time. The reason might be that Empty-16x16 requires only exploration instead of other skills, so the agent cannot benefit too much by following external guidance. Moreover, in a small environment such as DoorKey-5x5, the advantages of providing external knowledge is difficult to demonstrate due to extra learnable parameters in our actor model.
We also study the effects of different knowledge combinations for PPO-KGRL. In Unlock, we found that KG2 is more helpful than KG1; in DoorKey-8x8, KG3 is more helpful than KG1+KG2. We conjecture that the knowledge which directly helps receive the rewards guides an agent more effectively in these environments. Also, as shown in Figure 2(a-d), adding redundant knowledge will not affect the performance. This observation demonstrates the benefit of disentangling the knowledge embeddings and query, thus increasing the flexibility of providing external guidance.
1 2 3 4 Steps Step 6: pick up the key Step 15: open the door Step 22: search for goal Step 29: reach the goal Policy Interpretation. Figure 3 shows that our KGRL method learns interpretable agent behaviors. The weight changes indicate that in a DoorKey-8x8 environment, the agent chooses to sequentially "pick up the key", "open the door", "search for the goal", and "reach the goal". Note that the attention to the inner policy does not disappear throughout the entire episode. This shows that the inner policy helps an agent explore the environment.

Analysis of Generalizability.
To test the generalizability of PPO and PPO-KGRL, we train an agent in an empty environment with a fixed goal (Empty) and evaluate the agent in an empty environment with a randomized goal (Empty-Random). This experimental design verifies whether the agent memorizes the goal position or really understands how to find the goal. As shown in the left column of Table 1, the returns of PPO-KGRL achieve an average of 0.94 and a minimum of 0.77. This is much closer to the upper bound, which is the average return of a PPO-KGRL agent trained on the Empty-Random environment.   Table 1: Zero-shot evaluations on three environments (Empty, Empty-Random, Unlock) of RL agents trained on an initial environment (Empty and DoorKey) using PPO or PPO-KGRL algorithms. The average and minimal returns are shown for each evaluation. Our KGRL method is more generalizable and effective in transferring previously learned knowledge. Analysis of Reusing, Recomposing, and Rearranging Knowledge. For our KGRL method, we verify if the external and internal knowledge policies and their embeddings can be reused by the following two sets of experiments: (1) zeroshot transferring the knowledge policies and embeddings to a simpler task and (2) initializing the external guidance by those knowledge policies and embeddings when learning a more complex task. The second set of experiments also verifies if the knowledge policies and embeddings can be recomposed and rearranged.
In the first set of experiments, we zero-shot evaluate an agent trained in DoorKey environments on Empty, Empty-Random, and Unlock environments. The four columns on the right side of Table 1 show the average and minimal returns. We found that an agent trained with PPO-KGRL significantly outperforms the one trained with PPO in all environments. These results demonstrate that our KGRL policies have better reusability.    The two KGRL agents that reuse previously learned knowledge perform better than PPO and PPO-KGRL(all). Also surprisingly, we found that the number of samples required by PPO-KGRL that reuses the Unlock knowledge plus the one required by learning the Unlock knowledge is even less than the one required by PPO-KGRL(all), which learns with KG1+KG2+KG3. These results indicate that with the abilities to flexibly reuse, recompose, and rearrange knowledge, our KGRL method can further enhance the training efficiency.

ROBOTICS
Environments. We also verify the proposed KGRL approach in the following OpenAI robotic environments with a continuous action space and sparse rewards: Reach and Pick-And-Place. Both environments contain a task of reaching a randomly sampled goal. Similar to the MiniGrid environments, these two tasks can demonstrate the ability of our KGRL method reusing, recomposing, and rearranging previously learned knowledge when solving a new task. An observation of these environments includes the position and velocity of the end-effector and object (if it exists) and the position of the goal. An action of these environments contains the position variation of the end-effector and the distance between the two fingers.
Setup. We compare the soft actor-critic (Haarnoja et al., 2018) algorithm with hindsight experience replay  under RL (SAC) and KGRL (SAC-KGRL) formulations. In SAC-KGRL, we add the learnable inner key, knowledge keys, and query on top of the actor model of SAC. We define the following two external knowledge policies for SAC-KGRL, where p ee ∈ R 3 is the position of the end-effector and p obj ∈ R 3 is the position of the object. (KG1) move straightly to the goal: if p ee − p obj 2 < ε, move straightly to the goal with the gripper closed; otherwise, stay unmoved. Note that there is no object in Reach, so ε is set to infinity in this environment, and KG1 always guides the agent towards the goal. (KG2, for Pick-And-Place only) move straightly to the object: if p ee − p obj 2 ≥ ε, move straightly to the object with the gripper opened; otherwise, stay unmoved. ε is set to 0.03 in Pick-And-Place.
Results. Figure 5 shows the training results of Reach and Pick-And-Place. In both environments, SAC-KGRL requires less samples to succeed and shows more stability. We also study the effects of different knowledge combinations in Pick-And-Place. The learning curves indicate that using both KG1 and KG2 leads to slightly better sample efficiency than using only one of them, but there is no significant difference among the three combinations.
Analysis of Generalizability. We analyze if our KGRL policies are more generalizable by training in both environments with a smaller goal range and testing with larger goal ranges. The training goal range is 0.1 times smaller than the original one, and the testing goal ranges are the original one multiplied by 0.1 to 1. Figure 6 shows the generalizability results. The success rate of SAC and SAC-KGRL agents both decreases as the testing goal range increases, but the SAC-KGRL agent in Reach and the SAC-KGRL(all) agent in Pick-And-Place are less affected by the changes, especially when the goal ranges are larger. These results suggest that an KGRL agent has more potential to generalize to unseen environments.   Table 2 shows the testing results of both tasks after removing part of the knowledge set. The learned inner actors in both environments achieve high success rate without the external guidance, indicating that a KGRL agent learns to not only follow the guidance but also mimic its strategy. This imitation process allows an agent to develop an inner policy that outperforms external strategies, and itself can then become new knowledge useful for solving other tasks.  Table 3: Zero-shot transfer results of Pick-And-Place policies to Reach. Our proposed method is more effective and efficient in reusing learned knowledge.
Analysis of Knowledge Reusability. To demonstrate that our KGRL method enables better knowledge reusability, we apply zero-shot transfer from the learned Pick-And-Place policies to Reach. Note that the observation spaces of Reach and Pick-And-Place are different since there is no object in Reach. Hence, we apply zeros as the object's relative position and velocity in Reach to unify their observation spaces. Table 3 shows the success rate of different Pick-And-Place policies applied to Reach. The results indicate that SAC-KGRL is more efficient in reusing knowledge since the transfers are mostly successful after only 200k training steps.

CONCLUSION AND DISCUSSION
In this work, we introduce KGRL, a novel RL problem discussing how an agent can follow an arbitrary set of external guidance and learn its own policy. Also, the agent should be able to repeatedly reuse previously learned knowledge and dynamically recompose and rearrange knowledge to adapt to environmental changes. To address these challenges, we propose a simple yet effective actor model that performs an embedding-based attention mechanism. The experimental results show that the proposed model achieves the goal of KGRL. Our KGRL algorithm also satisfies the four properties of efficient learning (Kaelbling, 2020) and learns interpretable agent behaviors.
This work serves as an initial effort for the ultimate goal of KGRL, which is to automatically collect a large group of knowledge from different sources, such as books, the Internet, or previous training processes. This group of knowledge can be shared among multiple environments/tasks and continuously expanded, and a wide variety of artificial agents can use it as their guidance. Moving forward, the following research directions in KGRL are worth exploring: (1) incorporating restrictions that should be strictly followed during the learning and inference stages, (2) composing knowledge with different state and action spaces, and (3) dealing with more complex relationships among different knowledge, such as conditional dependence and conflicts. We hope this work can motivate future research in KGRL.