{'fraction_non_alphanumeric': 0.03429902254843149, 'fraction_numerical': 0.008857052118987968, 'mean_word_length': 4.666866626674665, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bayesian Experience Reuse for Learning from Multiple Demonstrators

Learning from demonstrations (LfD) improves the exploration efficiency of a learning agent by incorporating demonstrations from experts. However, demonstration data can often come from multiple experts with conflicting goals, making it difficult to incorporate safely and effectively in online settings. We address this problem in the static and dynamic optimization settings by modelling the uncertainty in source and target task functions using normal-inverse-gamma priors, whose corresponding posteriors are, respectively, learned from demonstrations and target data using Bayesian neural networks with shared features. We use this learned belief to derive a quadratic programming problem whose solution yields a probability distribution over the expert models. Finally, we propose Bayesian Experience Reuse (BERS) to sample demonstrations in accordance with this distribution and reuse them directly in new tasks. We demonstrate the effectiveness of this approach for static optimization of smooth functions, and transfer learning in a high-dimensional supply chain problem with cost uncertainty.


Introduction
Learning from demonstrations (LfD) is a powerful approach to incorporate advice from experts in the form of demonstrations in order to speed up learning on new tasks. However, existing work in LfD assumes that demonstrations are generated from a single near-optimal agent with a single goal [3,14,38]. In practice, however, data may be available from multiple sub-optimal agents with conflicting goals. For example, when learning to operate a vehicle autonomously using human driver demonstrations [9], different drivers can have different goals and needs (e.g. destinations, priorities for safety or time) and experience levels. Relying on demonstrators whose goals are misaligned with the new target task can lead to unintended behaviors, which can be dangerous in performance-critical applications, and can be minimized by learning to trust the most relevant demonstrators.
In this paper, we focus on using LfD with multiple conflicting demonstrators for solving static and dynamic optimization problems. In order to measure the similarity between source and target task goals, we adopt a Bayesian framework. We first parameterize source and target task reward functions as linear functions in a common feature space, whose uncertainty is modeled using Normal-Inverse-Gamma priors that are updated to posterior distributions using source and target demonstrations.
Taking a fully Bayesian treatment allows us to avoid premature convergence and is more robust to non-stationary data [8]. To jointly learn the posteriors and feature space, we build a Bayesian neural network with a shared encoder and multiple heads, one for each source and target task. We then formulate a quadratic programming problem (QP) whose solution yields a probability distribution over the source demonstrators. This allows us to trade-off the mean and variance of the uncertainty in source and target task goals in a principled manner.
In order to transfer demonstrations to new tasks in LfD, one approach is to pre-train the learner directly from the source data [15], or learn and reuse auxiliary representations from the source data such as policies [16,45]. However, the former can be ineffective when demonstrations assume conflicting goals, while meaningful policies may be difficult to solicit from the latter when demonstrators are limited, sub-optimal or exploratory in nature [32,38,47]. On the other hand, experience collected from a failed or inexperienced demonstrator can be just as valuable as an experienced one [21]. We present an algorithm called Bayesian Experience Reuse (BERS), for directly reusing multiple demonstrations in a way that is consistent with the learned weighting over the source and target task goals (Algorithm 1). We demonstrate the efficacy of our approach for optimizing static functions, and continuous control of a supply chain using reinforcement learning with high-dimensional actions.

Related Work
Most papers in LfD incorporate demonstrations from a single optimal or near-optimal expert with a single goal [3,14,38]. Some papers in the area of LfD for reinforcement learning (RLfD) relax this assumption to a single sub-optimal demonstrator, using pre-training [15,19,22], reward shaping [13,42], ranking demonstrations [51], Bayesian inference [36], or other approaches [45] for transfer. However, the aforementioned papers cannot learn from multiple demonstrators with conflicting goals. On the other hand, papers on this topic typically assume multiple near-optimal demonstrators, so that recovering policies [16,28,29], value functions [20,27] or successor features [6,7] is possible. In this paper, we show how sub-optimal demonstrators with conflicting goals can be ranked according to their alignment with the target task goal in a safe Bayesian manner, and reused directly by the target agent without learning auxiliary representations such as policies or value functions. Our approach is related to encoder sharing [18,34,48,49] and uncertainty quantification of rewards or policies [5,12,25,46], but our problem setting and methodology are different.

Reinforcement Learning
Complex decision-making can be summarized in a Markov decision process (MDP), formally defined as a five-tuple S, A, P, R, γ , where: S is a set of states, A(s) is a set of possible actions in state s, P (·|s, a) gives the next-state distribution upon taking action a in state s, R(s, a, s ) is the reward received upon transition to state s after taking action a in state s, and γ ∈ [0, 1] is a discount factor that gives smaller priority to rewards realized further into the future. The objective of an agent is to find a policy µ : S → A that maximizes the long-run expected return where a t ∼ µ(s t ) and s t+1 ∼ P (·|s t , a t ). When P and R are known, an optimal policy can be found exactly using dynamic programming [35,37]. In the reinforcement learning setting, neither P nor R are known to the decision-maker. Instead, the agent interacts with the environment using a randomized exploration policy µ e , collecting reward and observing the state transitions [43]. In order to learn optimal policies in this setting, temporal difference methods first learn Q(s, a) and then recover an optimal policy [31,52], while policy gradient methods parameterize and recover an optimal policy µ * directly [44]. Actor-critic methods learn and use both a critic Q(s, a) and actor policy µ(s) simultaneously [26].

Common Feature Representations
In this paper, we consider the regression problem associated with learning a multivariate function y : X → R on a domain X . For instance, in the RL setting, we estimate reward functions where X = S × A × S and y(x) = R(s, a, s ). Given a feature map φ : X → R d , a function y can be expressed as a linear combination y(x) = φ(x) w, ∀x ∈ X , where w ∈ R d is a fixed vector.
We are interested in the transfer learning problem between similar tasks in a domain M φ on a common X , defined formally as (1) In the RL setting, M φ could include all MDPs with shared dynamics and different rewards R i . In (1), we have explicitly assumed that the (unknown) state features φ are shared among tasks. This is not a restrictive assumption in practice, as given a set of tasks T 1 , T 2 . . . T n ∈ M φ , we may trivially define φ k (x) = y T k (x), ∀x ∈ X for each k = 1, 2 . . . n. The main challenge in this paper is to simultaneously learn a suitable common latent feature embedding φ and posterior distributions for y(x), and leverage this uncertainty for measuring task similarity in M φ .

Bayesian Learning from Multiple Demonstrators
The agent is presented with sets of demonstrations D 1 , D 2 , . . . D N from source tasks T 1 , T 2 , . . . T N ∈ M φ , consisting of labeled pairs (x t , y t ), and would like to solve a new target task T target ∈ M φ . In order to make optimal use of the source tasks during training, the agent should learn to favor demonstrators whose underlying reward representation is closest to the target reward.

Bayesian Linear Regression with Common Feature Representations
In order to tractably learn these reward representations, we learn a shared feature spaceφ ≈ φ, together with Bayesian regressors P(w i |D i ) for the source task and a Bayesian regressor P(w target |D) for the target task such that y i (x) = φ(x) w i and y target (x) = φ(x) w target approximately hold for all i and all realizations x ∈ X , y ∈ R. As we will soon show, the sharing of the features φ is crucial to allow meaningful comparisons between source weights w i and target weights w.
In order to tractably learn the features φ as well as the corresponding posterior distributions, we parameterize φ(x) ≈ φ θ (x) using a deep neural network (encoder) with weight parameters θ, and model w 1 , . . . w N , w target using the normal-inverse-gamma (NIG) conjugate prior: The joint posterior P(w i , σ 2 i |D i ) can be shown to factor as where and where Φ i is the set of state features and y i is the set of observations of y i in D i [8]. We have also assumed that, conditional on data D i , weights w i and variances σ 2 i are mutually independent between tasks, a very mild assumption in practice. Adapting the methodology of Snoek et al. [40], we update θ by gradient ascent on the marginal log-likelihood function for each head i, defined here as where the key quantities are provided by (4) and depend implicitly on θ through Φ i .
Using this model, parameter sharing allows the learned features φ to be refined and transferred seamlessly from source to target tasks, or as new source and target tasks are incorporated, and provides an additional source of knowledge for transfer. However, our main contribution is to use the posterior distributions over task goals, w i and w target , to transfer experiences between tasks.

Source Task Selection via Quadratic Programming
In order to derive a Bayesian decision rule for source task selection, we first observe that source tasks that are most similar to the target task, and hence those that lead to better transfer, should have w i closest to the true target w target . In our setting, we instead have uncertain estimates for w i and w target modelled as multivariate normal random variables. We therefore look for a weighting N i=1 a i w i that is closest in expectation to w target with respect to the uncertainty in these estimates. More specifically, suppose that we have computed the posterior distributions of w i for each i = 1, 2 . . . N and w target from past data. We would like to weight the w i in such a way that the weighted sum i a i w i is closest to w target in expectation, or in other words, solve where D is the union of all source and target data and P is a convex polyhedron. Specifically, we would like to be able to sample source tasks according to a, so we take P = {a : 1 a = 1, a ≥ 0}, the set of probability distributions on {1, 2 . . . N }. In other applications, such as static regression problems [33,50,53], we may choose P = R N or put additional constraints on expert selection.
To derive a closed form for (6), first note that, given the true values of the variances Then, taking expectation of (7) with respect to the variance terms v, where in the last step we used (2) and the expectation of InvGamma(α i , β i ), and ignored all terms independent of any decision variables a. Hence, we have shown that optimizing the expected weighted mean-squared error (6) is equivalent to optimizing the weighted mean-squared error of the posterior means plus a penalty term equal to the product of the noise and posterior variances.
Rewriting (8) in this new notation, we can obtain the following quadratic programming formulation of our problem: Here, M M + S is positive definite, since it is the sum of the positive semi-definite M M and the positive definite S. Hence, the above QP problem can be formulated and solved exactly using an off-the-shelf solver in polynomial time in N [11], which does not grow with the feature dimension d nor the number of demonstrations |D i | and can be applied in online settings. Hence, the QP (9) remains tractable when the domain complexity is high or the number of demonstrations is large. This is not necessarily the case if omitting the second order terms S, since the QP may become rank-deficient and thus lack a unique solution. We can therefore interpret S as a data-dependent Bayesian regularizer for the QP solution.
In the case where N is considerably large, the solution to (9) can instead be approximated using neural networks [1,10]. Alternatively, since the posterior updates are relatively smooth (as we demonstrate experimentally), the solution of (9) should not change significantly between consecutive iterations, in which case warm starts may be particularly effective [17].

Bayesian Experience Reuse (BERS) for LfD
When transferring demonstrations from a single source, a simple yet effective approach is to pre-train or seed the target learning agent with the demonstrations prior to target task training [15]. When data originates from multiple demonstrators with differing goals, pre-training is no longer possible. Instead, we propose to train the target agent on the source data while concurrently learning the target task. In this manner, the source demonstrations provide an effective exploration bonus by allowing the agent to observe demonstrations from other tasks with similar goals.
More specifically, in each iteration of the target learning phase, we sample a source task T i ∈ M φ according to the probability distribution a obtained from the quadratic programming problem (9), and train the target agent on experiences drawn from the corresponding source data D i . In order for the target agent to improve beyond the source task solution and generalize to the new task, the agent must eventually learn from target demonstrations rather than the source data. In order to do this, we apply the framework of Probabilistic Policy Reuse [16,29] in our setting. Specifically in each training episode m, the target agent is trained on source data with probability p m , and otherwise uses target data with probability 1 − p m , where p m is annealed to zero over time. The choice of p m is typically problem dependent, but may also be annealed based on the QP solution.
The full implementation of our algorithm, Bayesian Experience Reuse (BERS), is illustrated in Figure 1 in a transfer learning setting. The left figure provides a visualization of the process used to weight source tasks. Here, Bayesian heads with parameters {µ i , Λ i , α i , β i } N i=1 and µ, Λ, α, β are maintained for source and target tasks respectively, while sharing the encoder parameters θ, and then used to construct the quadratic programming problem (9). The outputsŷ i andŷ target can also be used for making predictions (e.g. in regression tasks); in this paper we focus on using the posterior distributions over task goals, and the corresponding QP solution, to transfer source demonstrations in a Bayesian framework.
In the pseudo-code on the right, we have defined O base to be a base learning algorithm for solving the target task, which is assumed to be a static or dynamic (RL) optimization algorithm in this work. We pre-train the first N heads on source data to learn features and posteriors for source task goals. At the end of each episode, we train all source and target heads on source and target demonstrations to refine the features and posteriors for all tasks. With simple modifications, this algorithm can also be applied in multi-task settings, by maintaining one QP solution per task. The bottleneck of this procedure is the solution of the QP, which can be addressed using various approximation methods outlined above.

Empirical Evaluation
In order to demonstrate the effectiveness of BERS, we apply it to a static optimization problem and continuous control of a supply chain.

Static Function Optimization
We first consider the problem of minimizing a smooth function [24]. Transfer learning can be useful in this setting because the known solution of one smooth function can be used as a starting point or seed in the search for the minimum of another similar function. More specifically, we use the 10-D Rosenbrock, Ackley and Sphere functions as source tasks, whose global optimum solutions have been shifted to various locations. As target task, we use each source task as the ground truth as well as the Rastrigin function. The Sphere and Rastrigin functions are locally similar around

Algorithm 1 Bayesian Experience Reuse (BERS)
Require: (4), (5) for episode m = 1, 2, . . . do for step t = 1, 2, . . . T of episode m do explore T target using O base and collect d = (x, y) (4), (5) solve QP (9) (using, e.g. [2]) and set a to the solution end for return O base the optimum, so a successful transfer experiment should learn to transfer knowledge between them. We consider optimizing the Rastrigin function in both transfer and multi-task settings. As the base learning agent O base in Algorithm 1, we use Differential Evolution (DE) [41], a commonly-used evolutionary algorithm for global optimization of black-box functions 2 .
We compare BERS (Ours) to the UCB algorithm [4] with asymptotically optimal convergence (UCB), where the reward is improvement in best solution value between consecutive iterations. We also compare to the equal weighting of demonstrators (Equal), individual demonstrators (S1, S2. . . ), and standard DE (None). Figure 2 illustrates the function value of the best solution and the QP solution in each iteration. Figure 3 tracks the movement of the posterior means of w i and w target during target training for a simplified model with d = 2 for visualization.

Continuous Control of a Supply Chain Network
The second problem is continuous control of a supply chain network as illustrated in Figure 4a. Here, the nodes include a central factory and K = 6 warehouses (denoted A, B . . . F), and edges represent A prudent agent must learn to take advantage of the "cheapest" routes through the network while simultaneously learning optimal manufacturing and order quantities.
The state includes the current stock in the factory and warehouses. Actions are modelled as follows: (1) one continuous action for production as a proportion of the maximum; (2) one set of K + 1 actions for proportions of factory stock to ship to each warehouse (including to keep at the factory); and (3) one set of K actions per warehouse, for proportions of warehouse stocks to ship to all other warehouses (including itself). This leads to a 2 + K + K 2 = 44-dimensional action space. In order to tractably solve this problem, we use the actor-critic algorithm DDPG [30] as the base learning agent O base . The actor network is shown in Figure 4b, using sparse encoding for states and sigmoid/ softmax output activation to satisfy stock constraints. The critic network has a similar structure.
We evaluate BERS (Ours) against prioritized experience replay [23,39] with pre-loaded demonstrations from all source tasks (PER), and HAT [45] combined with a state-of-the-art policy reuse algorithm [29] (PPR). In the latter case, source policies are trained using the same architecture as the actor network in Figure 4b for 50 epochs using a cross-entropy loss. Figure 5 illustrates the total test profit obtained, as well as the weights assigned by the QP problems to the source tasks. Figure 6 how the posterior weights of the target task approach the weights of the ground truth.

Discussion
On the optimization task, BERS achieves a small performance gap relative to the single best expert, since it can quickly identity the most suitable source task ( Figure 2). While UCB is a strong baseline, BERS finds the solution in less iterations and with less variability. As expected, the QP solution favours the Sphere function as the source task when solving the Rastrigin function, because they are structurally similar functions, which leads to a quicker identification of the global minimum.
On the supply chain task, BERS also achieves results that are similar to the single best expert, and does better than PPR (while the asymptotic performance on Scenario 1 is similar, BERS achieves better jump-start performance). Both BERS and PPR are able to quickly surpass the performance of the exploration policy that generated the source data (shown as a horizontal line in Figure 5), whereas PER was not able to obtain satisfactory performance. We believe the latter is true because PER prioritizes experiences by TD error, which is not suitable when the rewards are sampled from different tasks, while BERS learns a common feature embedding to allow consistent comparison between tasks (Figure 3). We also believe that if the source data was considerably less optimal, then the performance gap between PPR and BERS would be greater. On the target scenario, it is interesting to see that the weights assigned to Scenarios 1 and 2 are roughly equal, which makes sense as the target task shares similarities with both scenarios (Figure 4a). By mixing two source tasks, BERS was able to perform substantially better on the target task than the two source tasks (S2 and S3) in isolation. Also by adopting a fully Bayesian treatment, we obtain smooth convergence of the task weights on all experiments.

Conclusion
We studied the problem of LfD with multiple sub-optimal demonstrators with different goals. To solve this problem, we proposed a multi-headed Bayesian neural network with a shared encoder to efficiently learn consistent representations of the source and target reward functions from the demonstrations. Reward functions were parameterized as linear models, whose uncertainty was modeled using Normal-Inverse-Gamma priors and updated using Bayes' rule. A quadratic programming problem was formulated to rank the demonstrators while trading off the mean and variance of the uncertainty in the learned reward representations, and Bayesian Experience Reuse (BERS) was proposed to incorporate demonstrations directly when learning new tasks. Empirical results for static function optimization and RL suggest that our approach can successfully transfer experience from conflicting demonstrators in diverse tasks.

Broader Impact
This work presents a general framework for reusing demonstrations data available in general static and dynamic optimization problems. It could lead to positive societal consequences in terms of safety and reliability of automated systems, such as medical diagnosis systems that often require pre-training from human experts, or driver-less cars that learn from human mistakes to avoid similar situations in the real world, potentially saving lives in the process. On the other hand, because our approach requires the learning of the (latent) intentions of users and their goals, there could potentially be privacy concerns depending on how the trained agent decisions are implemented or published. This is particularly true shortly after commencement of training, when the learning agent can resemble one or more demonstrators in their decision making. Our framework is also based on deep learning methodologies, and so it naturally inherits some of the risks prevalent in that work (although our work also tries to address some of these issues by using a Bayesian approach). For example, data that consists of only exceptional situations (such as pre-training of a search and rescue robot), may consist of many outliers. Such data sets must sometimes undergo pre-processing to remove the effects of outliers or other features that could skew the algorithm and lead to the wrong conclusions. This could be catastrophic in mission critical operations where every decision counts. However, by using prudent machine learning practices, we believe the benefits of this work outweigh the risks. equally scaled (for Rosenbrock we subsequently also divide by 10), to demonstrate whether we can actually "learn" each function from the data rather than distinguish it according to scale alone.
Solver Settings: To optimize all functions, we use the Differential Evolution (DE) algorithm [41]. The pseudo-code of this algorithm is outlined in Algorithm 2.
Here, CR is the crossover probability and denotes the probability of replacing a coordinate in an existing solution with the candidates (crossover), F is the differential weight and controls the strength of the crossover, and N P is the size of the population (larger implies a more global search). We use standard values CR = 0.7, F = 0.5 and N P = 32 for reporting experimental results, unless indicated otherwise. The search bounds are also set to [−4, +4] for all coordinates; all points that are generated during the initialization of the population and the crossover are clipped to this range.
Experimental Details: We consider both transfer and multi-task learning settings. For the transfer experiment, we use the Rosenbrock, Ackley and Sphere functions as the source tasks, and the Rastrigin function as the target task. We also consider each of the source tasks as the ground truth to see whether we can identify the correct source task in an ideal controlled setting. In the multi-task setting, we did not observe any advantages of our algorithm nor the baselines when solving the Rosenbrock, Ackley or Sphere functions, so we focus on the quality of solution obtained for the Rastrigin function only. Transfer learning: We first generate demonstrations (x i , f (x i )) from each source function f using DE (Algorithm 2) and configuration above until a fitness of 0.15 is achieved. Respectively, these have sizes 17693, 6452 and 5853 for each source task. We further transform outputs for training and prediction with the neural-linear model using a log-transform y → log(1 + y) to limit the effects of outliers and stabilize the training of the model. When solving the target

Details for Prioritized Experience Replay
Experimental Details: We evaluated the performance of Prioritized Experience Replay [39] (PER) on the Supply Chain benchmark. In summary, PER is a replay buffer that ranks experiences using the Bellman error, defined for the DDPG algorithm for a transition (s, a, r, s ) as δ = r + γQ (s , µ (s )) − Q(s, a), where Q is the critic network, and Q and µ are the target critic and target actor networks, respectively.
In order to use PER to transfer experiences, we load all source demonstrations into the replay buffer prior to target task training with initial Bellman error δ = 1. The source demonstrations are first combined into a single data set and shuffled. The capacity of the buffer is then fixed to the total number of source demonstrations from all tasks (around 28,000 examples). During target training, observed transitions are immediately added to the replay buffer and override the oldest experience. In this way, the agent is able to maximize the use of the source data at the beginning of training but eventually shift emphasis to target task data.
Detailed Analysis: Figure 7 demonstrates the composition of each batch (of size 32) according to source (Scenario 1, Scenario 2, Scenario 3, Target Task) for each ground truth. As illustrated in all four plots, in early stages of training, batches are predominantly composed of source data, while in later stages, they consist entirely of target data. Figure 7 shows that PER is unable to favor demonstrations from the source task that correspond to the ground truth, in all four problem settings. One possible explanation of this is that an implicit assumption of PER is violated, namely that experiences are drawn from the same distribution of rewards, whereas in our experiment, experiences can come from tasks with contradictory goals. In this case, experiences corresponding to the ground truth do not necessarily have larger Bellman error (in fact, the opposite may be true).  Figure 7: Shows the composition of each batch over time sampled from the prioritized replay (PER) according to whether the sample came from the source or target task data, for each ground truth. PER is unable to rank experiences correctly to match the context.