{'fraction_non_alphanumeric': 0.035779260157671314, 'fraction_numerical': 0.009096422073984234, 'mean_word_length': 4.40672131147541, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners

Successful teaching requires an assumption of how the learner learns - how the learner uses experiences from the world to update their internal states. We investigate what expectations people have about a learner when they teach them in an online manner using rewards and punishment. We focus on a common reinforcement learning method, Q-learning, and examine what assumptions people have using a behavioral experiment. To do so, we first establish a normative standard, by formulating the problem as a machine teaching optimization problem. To solve the machine teaching optimization problem, we use a deep learning approximation method which simulates learners in the environment and learns to predict how feedback affects the learner's internal states. What do people assume about a learner's learning and discount rates when they teach them an idealized exploration-exploitation task? In a behavioral experiment, we find that people can teach the task to Q-learners in a relatively efficient and effective manner when the learner uses a small value for its discounting rate and a large value for its learning rate. However, they still are suboptimal. We also find that providing people with real-time updates of how possible feedback would affect the Q-learner's internal states weakly helps them teach. Our results reveal how people teach using evaluative feedback and provide guidance for how engineers should design machine agents in a manner that is intuitive for people.


Introduction
People regularly teach other agents (e.g., children, pets, machines) in their environment using evaluative feedback (rewards and punishment). For example, Andrew is teaching his six-year-old daughter Jane to forage for wild berries. To do so, he first brings her to a bush with edible berries. What does he do next? If his goal is purely for her to eat some wild berries, then he has achieved his goal. However, to teach her to forage in a robust manner, he must provide her rewards to incentivize her to leave that bush and seek new ones. How do people teach agents using rewards and punishments, and how does their teaching depend on their knowledge of the internal dynamics of how the learner updates their beliefs?
Although not always presented from this perspective, foraging is an example of the explorationexploitation problem within reinforcement learning: How should an agent balance exploiting rewards based on their current knowledge while still exploring for berries is an example of an exploration-exploitation problem, which has been studied extensively for humans, animals, and idealized agents. Many natural agents (adults, children, animals, and other living creatures; [3,6,26,29]) stop exploiting rewards at a state to explore, providing an implicit or explicit punishment to the agent, in a manner that is optimal within their ecological niche. However, often the mechanisms used to learn and change their behavior are tuned to assumptions of their environment, and do not adjust well to those in other environments. Although foraging itself (and learning while foraging) has been extensively studied within reinforcement learning and other mathematical frameworks, to the best of our knowledge, teaching others to forage is an open question. In this paper, we explore this question for teaching Q-learning agents a full policy in a task that requires teaching sub-optimal actions so that the learner learns what they should do in states they otherwise would not encounter (because they start close to their ultimate goal).
Teaching the correct actions to take in a domain while exploring the domain is a social task involving the interaction of the teacher, a learner, and the environment. Researchers have examined how people teach others, formalized this process, and created automated methods to teach. One unifying computational framework across these areas is the Bayesian pedagogy framework, where the learner and teacher are Bayesian agents that assume both know the teacher is providing information to help the learner [27]. However, this framework is mechanism-agnostic and assumes humans are doing ideal Bayesian updates, which can be a problematic assumption. Instead, we take the perspective of machine teaching, where we examine human teaching from the perspective of optimized teaching for a particular learning mechanism -Q-learning algorithms.
Q-learning is a family of model-free reinforcement learning algorithms that is known to learn the optimal policy as the agent interacts with the MDP [30] -undisturbed by a teacher. Instead, we allow a human teacher to intervene in the agent-MDP loop, specifically by changing the reward signal, with the hope to teach the optimal policy faster. Concretely, we design multiple Q-learning agents (students) that differ in a number of critical parameters. If human teachers can teach some agents better than others, it means that those agents' parameters are closer to human teachers' assumption about how students learn.

Prior Work
From children to adults, when asked to teach, people provide different information than if they are simply asked to convey some information to another learner. For social situations that involve goals and rewards, assuming people are optimal agents within a Markov Decision Process captures human behavior well. For example, when asked to show how to do a task, people will take actions that are strictly unnecessary for completing the task, but convey information to a learner. However, if they are asked to do a task, they only do the necessary actions for completing the task [11].
Recent work in cognitive science and human-machine interaction has explored human teaching strategies and to what extent they are optimal. For example, work in Bayesian pedagogy [27] has shown that when teaching a range of simple concepts by example, people are capable of teaching other people near optimally. Similar research on linguistic pragmatics [7,5] demonstrates that people's use of language reflects an intention to be optimally informative. Moreover, these findings on human teaching have been shown to generalize to more complex settings, such as sequential decision making [11]. However, more complex settings also make it more likely that human teaching and learning processes are misaligned [12,10], which motivates research into learning algorithms tailored to human teaching strategies [16,21].
Although the Bayesian pedagogy and social cognition literature provide useful perspectives on how people ought to teach others who know they are being taught, it does so for an idealized Bayesian agent assumed to be maximizing environmentally provided rewards. Albeit a useful assumption for examining human teaching, people do not teach or learn as an idealized Bayesian agent. In the 1990s, researchers learned how difficult it can be to train a reinforcement learner to complete simple tasks that contained necessary conditions which need to be completed by the learner completes the last steps of their task which are closer in state space to their current location [e.g., 23]. Based on this intuition, recent work demonstrated that people fail to teach simple model-free and model-based reinforcement learners how to get from a start state to an end state while staying on a trail in a 3 × 3 Grid World [10]. The learners often pick up on "positive net reward cycles" which enable them to get arbitrarily large reward while not completing the task.
Computational Teaching: Since computational teaching was first proposed in [28], the teaching has been studied in various learning settings, see [35] for a recent survey. Of particular interest to us are works on teaching online learners such as Online Gradient Descent (OGD) [18,17], active learners [8,24], and other sequential learners [13,22,34,14,19,32]. The optimal control formulation is required when the learner becomes sequential. Several recent works studied teaching on Inverse Reinforcement Learning (IRL) [31,15,1,9,2], where learner learns from teacher demonstration. Finally, computational teaching in reinforcement learning have been studied recently [33,25,20], where teacher teach via rewards and/or state transitions. Our work instead focuses on using computational teaching theory to understand how humans teach.

Preparing a Family of Reinforcement Learners for Teaching
In the teaching problem of reinforcement learning, we study the interaction between three entities: the RL agent (student), the teacher, and the underlying environment. In this work, we assume that the environment is a rewardless Markov Decision Process (MDP) parametrized by M = (S, A, P, µ 0 ) where S is the state space, A is the action space, P : S × A × S → R is the transition probability, and µ 0 : S → R is the initial state distribution.

A Family of Q-Learners
In our human experiments, we focus on a family of Q learners, denoted by L, that is widely studied as potential theory of mind for humans, among which we describe two variants: (1) standard Q-learning and (2) Action Signaling (AS) [10]. The two learners mainly differ in their internal knowledge representation and how they perform learning updates.
A Q-learning agent stores an estimate of the Q table, Q : S × A → R, which approximates the future cumulative rewards that the agent can receive after perform an action a ∈ A in a state s ∈ S. The learning update rule of Q-learning is defined by 2 parameters, the learning rate α, and the discounting factor γ. α determines how aggressive the learner updates the current belief given the new experience, and γ indicates how much the learner value future rewards compared to immediate rewards. Specifically, given a new piece of experience e t = (s t , a t , s t+1 , r t ), Q-learning only updates the (s t , a t ) entry of its Q table as An Action Signaling (AS1) agent stores a multinomial distribution over actions for each state, representing its current belief distribution of the optimal action [10]. In this paper, we represent the multinomial distribution also with a table, Q : S × A → R, where a∈A Q(s, a) = 1. The learning update of an AS agent is defined by a single parameter, the learning rate κ, which serves the same role as α in Q-learning. Given a new piece of experience e t = (s t , a t , s t+1 , r t ), an AS agent will update the belief distribution over the current state s t . If r t > 0, the probability w.r.t. a t will increase, whereas if r t < 0, the probability w.r.t. a t will decrease. In contrast to Q-learning, an AS agent does not make use of the next state s t+1 , and does not aim at optimizing long term reward. Specifically, (2) We further observe that even without the normalization step, arg max a Q t (s, a) will not change. Therefore, the learning update rule 2 can be equivalently changed to Q t+1 (s t , a) = Q t (s t , a)e 1 [a=a t ] κr t . We again observe that if instead of Q t (s, a), we store Q t (s, a) = log Q t (s, a), then the update rule becomes Q t+1 (s, a) = Q t (s, a) + κr t . Finally, we can drop the κ factor, since it's a constant scaling on all (s, a)-pairs, and we get Q t+1 (s, a) = Q t (s, a) + r t . Therefore, we have established that AS is equivalent to a variant of Q-learning, which simply stores the sum of rewards received on each (s, a)-pair. This motivates us to define another variant of the AS agent that instead of taking the sum of rewards, takes the average of rewards on each (s, a)-pair.
An Action Signaling (AS2) agent stores the average of rewards on each (s, a)-pair, whose update rule can be equivalently written as where n t (s, a) denotes the number of times the current (s, a)-pair has been visited. Similar to AS1, AS2 do not take into account of future rewards, and is in fact equivalent to a Q-learning agent with γ = 0, and time-varying learning rate α t = 1/n t (s, a). For those who are familiar with the classic RL/multi-armed bandit literature, the AS2 agent is equivalent to a multi-armed bandit algorithm that treats the MDP as S parallel A-arm bandits.
We further assume that all learners will behave according to the ε-greedy policy w.r.t. the current Q table, i.e.
We distinguish two teaching settings: In a white-box teaching setting, we assume an omniscient teacher, who has knowledge of the underlying MDP M and the learning agent's parameters. It also observes the current interaction s t , a t , s t+1 as well as the internal state of the agent Q t , before providing the reward signal r t ; In a black-box teaching setting, we assume that the teacher can still observe (s t , a t , s t+1 , Q t ) but does not know the precise learner update rule, and therefore cannot accurately predict the consequence of the current reward signal.
The teacher's goal is to drive the learner to learn a target policy π † . For example, the teacher may want the learner to always perform a specific action a † ∈ A at state s † ∈ S. This example goal can be expressed as a target set of Q tables that satisfy Q † : The teaching succeeds if the learner's Q t falls into Q at some time step t, in which case the teaching process terminates. We are interested in calculating a teaching strategy that achieves the teaching goal with the fewest time steps.

An Optimal Teacher can Teach the Family of Q-Learners Equally Fast
One of our main observations is that the optimal teaching problem in the white-box teaching setting forms a higher level teaching MDP N = (Ξ, ∆, ρ, τ ): • The teacher observes the teacher state ξ t ∈ Ξ, which jointly characterizes the environment and the learner at time t: ξ t := (s t , a t , s t , Q t ). • The teacher's action space consists of all possible rewards r t ∈ R = ∆.
The optimal teaching strategy is one where the teacher minimize its cumulative attack cost, T t=0 ρ t , s.t. Q T ∈ Q. Due to the randomness of the MDP as well as the learner's behavior policy, this quantity is a random variable, and thus we instead minimize its expected value. Formally, the teacher seeks a time-invariant teaching policy φ * : φ * = arg min φ: We will also denote the shortest expected time step to achieve the teaching goal as the teaching dimension for the corresponding teaching problem instance, i.e.
The optimal control formulation allows us to computationally solve for the optimal teaching strategy on any specific teaching instance, defined by the MDP, learner type and target policy to be taught, using any optimal control solver. In this paper, we use Twin Delayed DDPG (TD3) [4], a state-ofthe-art Deep Reinforcement Learning (DRL) algorithm that solves continuous control problems.
Our second theoretical insight is that the any learner in the Q-learning family L has the same teaching dimension. Specifically, Theorem 1. For any two teaching instances defined by (M, L, Q 0 , π † ) and (M, L , Q 0 , π † ), where L and L are two Q-learning agents, if Q 0 and Q 0 satisfies that In other words, under the white-box setting, an optimal teacher can teach a learner in the family L equally fast.

Human Teaching on the Family of Q-Learners
But our human teachers do not really know what algorithm or parameters the students use. This allows us to probe the human assumption on the student: we expect that when an student algorithm is closer to human preconception of a student, the human teacher will be better at teaching that student.
How does human teaching compare to machine teaching in a task where a person needs to teach a reinforcement learner a full policy when the learner starts one action away from its environmental goal? Figure 1 presents an idealized exploration-exploitation scenario. In the scenario, the dog's initial state is one tile left of its goal (the door which symbolizes its home). The teaching goal is to teach the dog to go home from every tile. Whenever the dog reaches its home, the next step it starts on the initial state. To teach the dog to get home from every square, the teacher must incentivize the dog to explore, which results in the dog moving away from its ultimate goal: It must punish moving to the goal and/or reward going away from the goal. Once the dog has gotten all the way to the leftmost tile, the teacher can begin to "undo" their prior teaching and teach the dog to go right. Depending on how much reward/punishment the dog is given to explore, it may take many steps (or even be impossible given feedback with finite limits) to teach the dog to move right at every tile.
To analyze the difficulty of this task, we used the machine teaching method described in Section 3 to provide the optimal teaching policy. The learner is a Q-learner that uses an ε-greedy policy to select its action and it starts indifferent Q 0 (s, a) = 0 for every feasible state-action pair (s, a). For concreteness, we assume the learner is parametrized as follows: α = 0.1, γ = 0.9, and ε = 0.1. Given this configuration, the optimal teaching policy is precisely the one discussed above (get the dog to move all the way left with minimal reward/punishment and then teach it to move right at every state). The number of steps to teach is a random variable whose outcomes depend on what actions are selected when the dog is indifferent and when suboptimal actions are taken. Thus, we approximated the expected number of steps via Monte Carlo (simulating teaching dogs using the optimal teaching policy and recording the number of steps it took for the full target policy to be taught). On average, the optimal teaching policy takes 11 steps to teach the dog the full target policy. It is 11 regardless of the value of α and γ (assuming they are within the values that allow Qlearning to converge). Changing ε affects the optimal teaching policy and expected number of steps quantitatively, but the optimal teaching policy follows the same qualitative procedure as before. So, we set ε = 0.1 for the remainder of the paper.
Although our machine teaching results suggest all Q-learners should be trainable with the same expected number of steps, can people train all learners? Are there some learners that are easier for people to train? As discussed earlier, recent work has found that traditional parametrization for Q-learners are extremely challenging for people to train on even simple tasks and the actionsignaling model (which we proved is a special-case of Q-learning) is much easier for people to train [10]. However, they did not test a large set of parametrization for the discount rate (γ) and some of their analyses suggest that a Q-learner might be trainable when the discount rate is small. This is consistent with other work on hyperdiscounting by Taber and colleagues [16] that reinforcement learners with small discount rates are easier for people to teach. In this study, we investigate the role of the discounting parameter for a simple teaching task and examine how giving people the internal dynamics of the learning update affects their ability to teach efficiently.

Experimental Design
Participants. We recruited 791 participants through Amazon MTurk. The number of participants was chosen a priori based on the authors' intuition from similar previously conducted studies. We excluded 42 participants (11 for failing to complete the task, 19 due to experiment error, and 12 who selected "do nothing" for at least 36 steps). In addition, the dogs that were trained successfully with steps less than the optimal length (the number of steps the optimal teaching policy needs) were excluded from the analysis. For example, if the subject gave punishment on every step and the dog luckily went to the left at every tile all the way from the rightmost tile to the leftmost tile, the dog could be trained in just 4 steps. These data were excluded because they did not really reflect a success in training, but more likely a misunderstanding of the goal and luck. This composes 9.03 % of the total dogs (203 out of 2247 dogs).
Interface/Stimuli. The dog training task took place in a 4 × 1 MDP with an absorbing state on the right. Figure 1 shows the visual interface that the participants interacted with. Four states were represented by four tiles that the dog could walk on, and the absorbing state was represented by a door. At each step, the dog could only go right or left from a tile to the nearby tile or the door. If the dog went left at the leftmost tile, the dog would stay at the same tile. If the dog went right at the rightmost tile, the dog would go to the door and then placed back to the rightmost tile. The training for a dog ended if the dog had successfully learned the target policy, or the dog had already taken 40 steps but still had not learned the target policy. Once the training for the dog ended, a new dog with a different color would be shown and the participant was asked to train the new dog. The internal states were displayed as two rows provided by a "brain scanner", which corresponded to the Q table and current optimal policy. At the beginning of each dog, the dog was placed at the rightmost tile with an initial Q table where the Q value of each state-action pair is zero. Participants responded via a continuous slider (feedback of -1 to 1), or could select a button to "do nothing" (feedback of zero).
Procedure. Participants were asked to teach three dogs to go home from any tile. They were given an extensive quiz about the instructions and could not continue until every question was answered correctly (see Supplementary Materials). There were two between-subjects conditions: learner type × learning dynamics. Each participant was assigned to either one of the four traditional Qlearners (γ ∈ {0.0, 0.1, 0.45, 0.9}, α = 0.9) and two action-signaling models. Learning dynamics were either shown (the whitebox setting) or not shown (the blackbox setting) to the participant. if they were shown, then the internal states displayed to the participant were synchronized to the current position of the slider. Otherwise the internal states only updated after the participant provided feedback. On each trial, the dog would move from one state to another. It followed an ε-greedy policy (ε = 0.1), where the policy was the optimal one for the current Q-matrix. When a random action was selected for the dog, a squirrel would appear in the direction of that action (participants were told that when a squirrel appeared, the dog would move towards it no matter what its internal states were). After the dog moved, the participant would respond by dragging the feedback slider or hitting the "do nothing" button. Training concluded when the participant successfully taught the target policy or 40 steps had completed. Participants trained three dogs and then was given a short survey to ensure that they treated the slider symmetrically and gather standard demographic data. Figure 2 shows the success rate of participants at training the full policy and the average number of steps taken when a dog was successfully trained. To analyze the success rate, we fit a mixed-effects logistic regression model where learner type, internal dynamics, and their interaction were fixed effects, and participant was a random effect. Learner type significantly influenced the success rate for a participant (χ 2 (5) = 20.7, p < 0.001), but the internal dynamics did not (χ 2 (1) = 0.66, p = 0.42). To analyze the average number of steps for successfully trained dogs, we fit a mixed-effects  When the dog's policy at a tile dictated going towards the door, the background of the cell turned green, indicating the policy at that state matched the target policy. After the dog took its action, the participant was asked to provide feedback to the dog. The participant could used the slider to select the feedback value or clicked the "do nothing" button to give a zero feedback. After the participant chose the feedback, the dog's Q table and target would get updated accordingly, and the dog would perform its next action. In the bottom-right corner, there as a step counter that recorded the steps that the dog has taken so far.  How well did people teach the dogs? Clearly they were suboptimal -at best, the average success rate was 51% and when successful took about 21 steps (optimal has a success rate of 100% and takes an average of 11 steps). To analyze whether people adapted their teaching over time, we conducted the following permutation test. For each participant, we created 1000 simulated Q-learners. For each simulated Q-learner and state-action pair, we sampled a random permutation of the feedback provided by the participant for that state-action pair. This provides a Monte Carlo test of whether human teaching adapted with learning. Participants clearly adapted their behavior: none of the simulated Q-learners for any participant ever learned the target policy when the feedback for each action-state pair randomly permuted. Thus, human teaching is quite successful in this task (albeit still suboptimal).

Results and Discussion
In sum, we observed that human teachers are suboptimal: they did not always succeed in teaching the policy to an agent; and when they did, they required more steps compared to the 11 steps of the optimal teaching algorithm in Section 3. This is not surprising: we never expected humans to be optimal teachers. Our important discovery is that human teachers' "favorite student" is Q0, which differs from other Q-learning agents in two key parameters: the smallest possible discounting factor γ = 0, and a large Q-update step size α = 0.9. What does this mean? The favorite student is one which is predictable by the teacher (no complications from discounted future rewards because γ = 0) and amenable to myopic control from the teacher (large α emphasizes immediate teacher reward). We thus suggest that human teachers assume such predictable, myopically controllable students when they teach. On the other hand, whether the teacher can see the "guts" of the agent (whitebox / blackbox setting with effect of reward slider on / off, respectively) turns out to not be very important for the teacher.

Conclusions
In this paper, we conducted the first investigation of how people teach learners to solve the exploration-exploitation tradeoff, which is a common problem in everyday life. We did so by first formulating it as a machine teaching problem where the learner is assumed to be a Q-learner. We then ran a behavioral experiment of an idealized scenario to see how well people train Q-learners to solve the exploration-exploitation tradeoff when the teaching goal is an entire policy. We found that people are suboptimal, but quite successful, and that teaching was the easiest when the Q-learner had a small discount rate γ and large update step size α. We also showed that revealing how possible feedback would change the learner's beliefs weakly helped people teach in a more efficient manner.
Given that our study is the first to explore how people teach learners to solve an explorationexploitation tradeoff, there are potential concerns with the generalizability of the study. For example, it is unclear whether our results would generalize to other environments. It is also unclear how robust the results are to other types of reinforcement learners. For example, it seems plausible that a model-based reinforcement learning method would be easier to train on our idealized task. However, prior work suggests that model-based learners would struggle in more complex environments where positive net cycles are easily taught.

Broader Impacts
Over the last few decades, human-machine interactions have become a regular part of everyday life. Although these interactions are often successful, they also often fail, usually due to the machine agent not properly understanding the person's feedback. When they do succeed, it is often because people have learned to interact with the machines, rather than machines learning or being created to interact with people in a manner that is natural to people. Not everyone can learn to interact with machines, especially those who have not kept up with current technology. This provides a large gap in the tools available to different people, with those who are disadvantaged often struggling to learn how to interact properly with the machines due to lack of experience.
One main method people interact with one another and machines is to teach by providing evaluative feedback. Despite much progress, scientists neither fully understand the mechanisms underlying this behavior nor have a formal description that can guide or be "plugged into" machine agents. This paper provides the first mechanism-level analysis of how a learner should be taught, as well as an analysis of how people teach via evaluative feedback in an idealized exploration-exploitation task. It provides guidance for how engineers should parametrize machine agents who use Q-learning while interacting with people. This is a step towards the larger goal of natural human-machine interaction that is equally accessible to all. Proof. We denote Q t ≡ Q t if Q 0 t(s, a) ≥ Q t (s, a ) if and only if Q t (s, a) ≥ Q t (s, a ) for all s ∈ S, a, a ∈ A. The proof is based on a key technical lemma: Lemma 3. For any teaching policy φ for teaching instance (M, L, Q 0 , π † ), there exists a matching teaching policy φ for teaching instance (M, L , Q 0 , π † ), such that for any time t, if Q t ≡ Q t and ξ t = ξ t , then Q t+1 ≡ Q t+1 .
Lemma 3 implies that with the same random seed, the relationship Q t+1 ≡ Q t+1 will remain invariant through teaching when φ is used on (M, L, Q 0 , π † ) and φ used on (M, L , Q 0 , π † ). Thus, Q t ∈ Q † if and only if Q t ∈ Q † . Therefore, if there exists an optimal φ that achieves the teaching dimension for (M, L, Q 0 , π † ), then the matching φ also achieves the same teaching dimension for (M, L, Q 0 , π † ). Thus, the T D for both teaching instances much match. What remains is to prove Lemma 3.
Proof of Lemma 3. Given a particular Q t and an experience tuple ξ t = (s t , a t , s t ), the Q-learning update rule will only modify the value of Q t (s t , a t ) based on the teacher provided reward r t = φ(ξ t ). Define the rank of (s t , a t ) in Q t as rank Qt (s t , a t ) = |{a|Q t (s t , a) > Q t (s t , a t )}|. Under the Q-learning update rule it is obvious that there exist r t such that the rank of (s t , a t ) in Q t+1 can be any of 0, 1, ..., A − 1.
Assume now that the rank of (s t , a t ) in Q t+1 becomes k after updating with r t = φ(ξ t ). Define φ such that r t = φ (ξ t ) will also update the rank of (s t , a t ) in Q t+1 to be k. Then, since Q t ≡ Q t and for both Q t+1 and Q t+1 the rank of (s t , a t ) becomes k, we have Q t+1 ≡ Q t+1 . This concludes the proof.

B Experiment Setting and Hyperparameters for TD3
Throughout the experiments, we use the following set of hyperparameters for TD3, described in Table 2. The hyperparameters are selected via grid search on the Dog MDP. Each experiment is run for 5000 episodes, where each episode is of 200 iteration long. The learned policy is evaluated for every 50 episodes, and the policy with the best evaluation performance is used for the computation of the teaching dimension. In the computation of the teaching dimension, we run the best-found TD3 policy for 1000 episodes, and take the average number of steps to teach the target policy. This gives 11.0.

C.1 Data Collection Process
Participants accessed the study on Amazon MTurk. Each participant was first given an instruction about the task (Fig. 3) and was required to complete an extensive quiz about the instructions (Fig. 4) and could not continue until every question was answered correctly. Participants trained three dogs and then was given a short survey to ensure that they treated the slider symmetrically and gather standard demographic data. Each participant was reimbursed for $2.00 for taking the task.

C.2 Participants and Data Exclusion
We recruited 791 participants through Amazon MTurk. The number of participants was chosen a priori based on the authors' intuition from similar previously conducted studies. We excluded 42 participants (11 for failing to complete the task, 19 due to experiment error, and 12 who selected "do nothing" for at least 37 steps out of 40 steps for training a single dog). In addition, the dogs that were trained successfully with steps less than the optimal length (the number of steps the optimal teaching policy needs) were excluded from the analysis. For example, if the subject gave punishment on every step and the dog luckily went to the left at every tile all the way from the rightmost tile to the leftmost tile, the dog could be trained in just 4 steps. These data were excluded because they did not really reflect a success in training, but more likely a misunderstanding of the goal and luck. This composes 9.03 % of the total dogs (203 out of 2247 dogs). In sum, there are 749 subjects (2044 dogs) in the analysis.