{'fraction_non_alphanumeric': 0.04504161294103857, 'fraction_numerical': 0.014007736490446607, 'mean_word_length': 4.607131120604667, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adversarial Option-Aware Hierarchical Imitation Learning

It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to compounding errors or suboptimal solutions. In this paper, we propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent. We theoretically prove the convergence of the proposed algorithm. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.


Introduction
Hierarchical Imitation Learning (HIL) has exhibited promises on acquiring long-term skills directly from demonstrations (Le et al., 2018;Yu et al., 2018;Byrne & Russon, 1998;Sharma et al., 2019;Hamidi et al., 2015). It contends the nature of sub-task decomposition in long-horizontal tasks, enjoying richer expressivity on characterizing complex decision-making than canonical Imitation Learning (IL). In general, HIL designs micro-policies for accomplishing the specific control for each sub-task, and employs a macro-policy for scheduling the switching of micro-policies. Such a two-level decision-making process is usually formu- lated by an Option model (Sutton et al., 1999) or goal-based framework (Le et al., 2018). Although some works (Fei et al., 2020) have assumed the help of sub-task segmentation annotations, this paper mainly focuses on learning from unsegmented demonstrations to allow more practicability.
Plenty of HIL methods have been proposed, including the Behavior-Cloning(BC) based and the Inverse Reinforcement Learning(IRL) based approaches. For the first avenue, Zhang & Paschalidis (2020); Le et al. (2018) customize BC to hierarchical modeling (Daniel et al., 2016), which, unfortunately, is prone to compounding errors (Ross et al., 2011) in case that the demonstrations are limited in practice. On the contrary, IRL is potential to avoid compounding errors by agent self-explorations. However, addressing IRL upon the Option model, by no means, is non-trivial considering the temporal coupling of the high-level and low-level policies. The works by Sharma et al. (2018); Lee & Seo (2020) relax this challenge by training the two-level policies separately. Nevertheless, this two-stage method potentially leads to sub-optimal solutions in the absence of training collaboration between the two stages.
To overcome the aforementioned issues, this work proposes a novel Hierarchical IRL framework-Option-GAIL, which is theoretically elegant, free of compounding errors, and end-to-end trainable. Basically, it is built upon GAIL (Ho & Ermon, 2016) with two nutritive enhancements: 1) replacing Markov Decision Process (MDP) with the Option model; 2) augmenting the occupancy measurement matching with options. Note that GAIL is a popular IL method that casts policy learning upon MDP into an occupancy measurement matching problem. Therefore, it is natural to replace the MDP with the Option model for hierarchical modeling. Besides, GAIL mimicking a policy from an expert is equivalent to matching its occupancy measurement. This equivalence holds when the policy is one-to-one corresponding to its induced occupancy measurement, which, yet, is not valid in our hierarchical context. As we will depict in the paper, the policy of HIL is related to options, and its one-to-one correspondence only exists with regard to option-extended occupancy measurement other than traditional occupancy measurement without options. Hence, it is indispensable to involve options into the matching goal in our second enhancement. Notably, the option switching is inherently guaranteed in our model, without extra regulators such as mutual information used in (Sharma et al., 2018) to encourage the correlation between action-state pairs and options.
It is non-straightforward to train our Option-GAIL, specifically when the expert options are unavailable. We thereby propose an EM-like learning algorithm to remedy the training difficulty. Specifically, in the E-step, we apply a Viterbi method (Viterbi, 1967) to infer the expert options conditional on the agent policy and expert actions/states. For the M-step, with the expert options provided, we optimize both the high-and low-level policies to minimize the discrepancy between the option-occupancy measurements of expert and agent. To be more specific, the M-step is implemented by a min-max game: maximizing a discriminator to estimate the discrepancy, and updating the policy to minimize the discriminative cost via a hierarchical RL method (Zhang & Whiteson, 2019). Notably, the convergence of the proposed EM algorithm is theoretically guaranteed if certain mild conditions hold.
In summary, our main contributions include: • We propose Option-GAIL, a theoretically-grounded framework that integrates hierarchical modeling with option-occupancy measurement matching. It is proved that Option-GAIL ensures the discrepancy minimization between the policies of demonstrator and imitator.
• We propose an EM-like algorithm to train the parameters of Option-GAIL end-to-end. This method alternates option inference and policy updating and is proved to converge eventually.
• We evaluate our proposed method on several robotic locomotion and manipulation tasks against state-of-theart HIL/IL baselines. The results demonstrate that our approach attains both dramatically faster convergence and better final performance over the counterparts. A complete set of ablation studies also verify the validity of each component we proposed.

Related Works
There have already been plenty of works researching on HIL, which, in general, can be categorized into two classes: Hierarchical Behavior Cloning (H-BC) and Hierarchical Inverse Reinforcement Learning (H-IRL).
Hierarchical BC. In these avenues, HIL is an extension of Behavior Cloning (BC) (Pomerleau, 1988;Atkeson & Schaal, 1997). As a result, the missing sub-task information needs to be provided or inferred to ensure the learnability of hierarchical policies. For example, Le et al. (2018) require predefined sub-goals when learning the goal-based policies; Kipf et al. (2019) try to alleviate the requirement of sub-task information by formulating the policy learning as an unsupervised trajectory soft-segmentation problem; Daniel et al. (2016) and Zhang & Paschalidis (2020) employ Baum-Welch algorithm (Baum et al., 1972) upon the option framework to infer the latent option from demonstrations, and directly optimize both the high-and low-level policies. Despite its easy implementation, behavior cloning is vulnerable to compounding errors (Ross et al., 2011) in case of the limited demonstrations, while our method enjoys better sample efficiency thanks to the IRL formulation.
Hierarchical IRL. Contrasted to the BC-based approaches, Hierarchical IRL avoids compounding errors by taking advantage of the agent's self-exploration and reward reconstruction. Following the Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016), the works by (Sharma et al., 2018;Lee & Seo, 2020) realize HIL by introducing a regularizer into the original IRL problem and maximizing the directed information between generated trajectories and options. However, the high-and low-level policies are trained separately in these approaches: the highlevel policy is learned with behavior cloning and remains fixed during the GAIL-based low-level policy learning. As we reported in the experiments, such a two-staged paradigm could lead to potentially poor convergence compared to our end-to-end training fashion. Henderson et al. (2018) propose an end-to-end HIL method without pre-training. However, it adopts a Mixture-of-Expert (MoE) strategy rather than the canonical option framework. Therefore, an option is exclusively determined by the corresponding state, ignoring its relation to the option of the previous step ( Figure 1). On the contrary, our method conducts option inference and policy optimization that are strictly amenable to the option dynamics (Sutton et al., 1999), thus delineating the hierarchy of sub-tasks in a more holistic manner.

Preliminaries
This section briefly introduces preliminary knowledge and notation used in our work.

Markov Decision Process: A Markov Decision Process
(MDP) is a 6-element-tuple S, A, P a s,s , R a s , µ 0 , γ , where (S, A) denote the continuous/discrete state-action space; P a s,s = P (s t+1 = s |s t = s, a t = a) is the transition probability of next state s ∈ S given current state s ∈ S and action a ∈ A, determined by the environment; R a s = E[r t |s t = s, a t = a] returns the expected reward from the task when taking action a on state s; µ 0 (s) denotes the initial state distribution and γ ∈ [0, 1) is a discounting factor. The effectiveness of a policy π(a|s) is evaluated by its expected infinite-horizon reward: η(π) = E s0∼µ0,at∼π(st),st+1∼P a t s t ,s t+1 ∞ t=0 γ t r t .
The Option Framework: Options O = {1, · · · , K} are introduced for modeling the policy-switching pro-cedure on long-term tasks (Sutton et al., 1999). An option model is defined as a tuple O = S, A, {I o , π o , β o } o∈O , π O (o|s), P a s,s , R a s , µ 0 , γ , where, S, A, P a s,s , R a s , µ 0 , γ are defined as the same as MDP; I o ⊆ S denotes an initial set of states, from which an option can be activated; β o (s) = P (b = 1|s) is a terminate function which decides whether current option should be terminated or not on a given state s; π o (a|s) is the intra-option policy that determines an action on a given state within an option o; a new option is activated in the call-and-return style by an inter-option policy π O (o|s) once the last option terminates.
Generative Adversarial Imitation Learning (GAIL): For simplicity, we denote (x 0 , · · · , x n ) as x 0:n for short throughout this paper. Given expert demonstrations on a specified task as D demo = τ E = (s 0:T , a 0:T ) , imitation learning aims at finding a policy π that can best reproduce the expert's behavior, without the access of the real reward. Ho & Ermon (2016); Ghasemipour et al. (2020) cast the original maximum-entropy inverse reinforcement learning problem into an occupancy measurement (Syed et al., 2008) matching problem: where, D f computes f -Divergence, ρ π (s, a) and ρ π E (s, a) are the occupancy measurements of agent and expert respectively. By introducing a generative adversarial structure (Goodfellow et al., 2014), GAIL minimizes the discrepancy via alternatively optimizing the policy and the estimated discrepancy. To be specific, a discriminator D θ (s, a) : S × A → (0, 1) parameterized by θ in GAIL is maximized and then the policy is updated to minimize the overall discrepancy along each trajectory of exploration. Such optimization process can be cast into: max θ min π E π log D θ (s, a) +

Our Framework: Option-GAIL
In this section, we provide the details of the proposed Option-GAIL. Our goal is towards imitation learning from demonstrations of long-horizon tasks consisting of small sub-tasks. We first introduce necessary assumptions to facilitate our analyses, and follow it up by characterizing the motivations and formulation of Option-GAIL. We then provide the EM-like algorithm towards the solution and conclude this section with theoretical analyses on the convergence of our algorithm.

Assumptions and Definitions
We have introduced the full option framework for modeling switching procedure on hierarchical tasks in Section 3. How- ever, it is inconvenient to directly learn the policy of this framework due to the difficulty of determining the initial set I o and break condition β o . Here, we introduce the following assumption given which the original option framework will be equivalently converted into the one-step option that is easier to be dealt with.
Assumption 1 (Valid Option Switching) We assume that, 1) each state is switchable: Assumption 1 asserts that each state is switchable for each option, and once the switching happens, it switches to a different option with probability 1. Such assumption usually holds in practice without the sacrifice of model expressiveness. Now, we define the following one-step model.
. = µ 0 (s)1 o=# , where 1 x=y is the indicator function, and it is equal to 1 iff x = y, otherwise 0. The high-and low-level policies are defined as: It can be derived that O one-step = O under Assumption 1 1 . This equivalence is beneficial as we can characterize the switching behavior by only looking at π H and π L without the need to justify the exact beginning/breaking condition of each option. We denoteπ . = (π H , π L ) andΠ = {π} as the set of stationary policies. We will employ the onestep option in the remainder of our paper. Note that in a current paper (Li et al., 2021) the rigorous notions and formulations have been provided for further discussing the relationship between the full option model and the one-step option model.
We also provide the definition of the option-occupancy mea-surement by borrowing the notion of the occupancy measurement ρ(s, a) from MDP (Syed et al., 2008).
The measurement ρπ(s, a, o, o ) can be explained as the distribution of the state-action-option tuple that generated by the policy π H and π L on a givenμ 0 and P a s,s . According to the Bellman Flow constraint, one can easily obtain that ρπ(s, a, o, o ) belongs to a feasible set of affine constraint

Problem Formulation
Now we focus on the imitation task on long-term demonstrations. Note that GAIL is no longer suitable for this scenario since it is hard to capture the hierarchy of sub-tasks by MDP. Upon GAIL, the natural idea is to model the long-term tasks via the one-step Option instead of MDP, and the policy is learned by minimizing the discrepancy of the occupancy measurement between expert and agent, namely, While this idea is straightforward and never explored before, we claim that it will cause solution ambiguity-we cannot make sure thatπ =π E , even we can get the optimal solution ρπ(s, a) = ρπ E (s, a) in Equation 3.
Intuitively, for the hierarchical tasks, the action we make depends not only on the current state we observe but also on the option we have selected. By Definition 1, the hierarchical policy is relevant to the information of current state, current action, last-time option and current option, which motivates us to leverage the option-occupancy measurement instead of conventional occupancy measurement to depict the discrepancy between expert and agent. Actually, we have a one-to-one correspondence betweenΠ and D.
Theorem 1 (Bijection) For each ρ ∈ D, it is the optionoccupancy measurement of the following policy: andπ = (π H , π L ) is the only policy whose optionoccupancy measure is ρ.
With Theorem 1, optimizing the option policy is equivalent to optimizing its induced option-occupancy measurement, since ρπ(s, a, o, o ) = ρπ E (s, a, o, o ) ⇔π =π E . Our hierarchical imitation learning problem becomes: .
Note that the optimization problem defined in Equation 5 implies that in Equation 3, but not vice versa: , addressing the problem defined in Equation 5 is addressing an upper bound of that defined in Equation 3. Indeed, we will show in § 4.6 that decreasing the goal in Equation 5 will definitely decrease that of Equation 3 given certain conditions. Figure 2 depicts the relationship between Equation 5 and Equation 3.
Yet, it is nontrivial to tackle Equation 5 particularly owing to the unobserved options in expert demonstrations. Here, we propose an EM-like algorithm to address it, which will be detailed in § 4.3 and § 4.4.

Option-Occupancy Measurement Matching
In this section, we assume the expert options are given and train the hierarchical policyπ to minimize the goal defined in Equation 5. We denote the option-extended expert demonstrations asD demo = {τ = (s 0:T , a 0:T , o −1:T )}.
Inspired from GAIL (Ho & Ermon, 2016), rather than calculating the exact value of the option-occupancy measurement, we propose to estimate the discrepancy by virtue of adversarial learning. We define a parametric discriminator as If specifying the f-divergence as Jensen-Shannon divergence, Equation 5 can be converted to a min-max game: The inner loop is to train D θ (s, a, o, o ) with the expert demonstrationD demo and samplesDπ generated by selfexploration. Regarding the outer loop, a hierarchical rein-forcement learning (HRL) method should be used to minimize the discrepancy: where c(s, a, o, o ) = log D θ (s, a, o, o ) and the causal entropy H(π) = Eπ [− log π H − log π L ] as a policy regularizer with λ ∈ [0, 1]. Notice that the cost function is related to options, which is slightly different from many HRL problems with option agnostic cost/reward (Bacon et al., 2017;Zhang & Whiteson, 2019). To deal with this case, we reformulate the option-reward and optimize Equation 7 using similar idea as Zhang & Whiteson (2019).
DAC characterizes the option model as two-level MDPs.
Here we borrow its theoretical result by re-interpreting the reward used by each MDP: For the high-level MDP, we de- p(o|s) . Other terms including the initial state distributions µ H 0 and µ L 0 , the state-transition dynamics P H and P L are defined similar to Zhang & Whiteson (2019). The HRL task in Equation 7 can be separated into two non-hierarchical ones with augmented MDPs: , whose action decisions depend on π H and π L , separately. Such two nonhierarchical problems can be solved alternatively by utilizing typical reinforcement learning methods like PPO (Schulman et al., 2017) with immediate imitation reward r t = −c(s t , a t , o t , o t−1 ) when in practice.
By alternating the inner loop and the outer loop, we finally deriveπ that addresses Equation 5.

Expert Option Inference
So far, we have supposed that the expert options are provided, which, however, is usually not true in practice. It thus demands a kind of method to infer the options from observed data (states and actions). Basically, given a policy, the options are supposed to be the ones that maximize the likelihood of the observed state-actions, according to the principle of Maximum-Likelihood-Estimation (MLE).
We approximate the expert policy withπ learned by the method above. With states and actions observed, the option model will degenerate to a Hidden-Markov-Model (HMM), therefore, the Viterbi method (Viterbi, 1967) is applicable for expert option inference. We call this method as Option-Viterbi for its specification to the option model.
Here a maximum foreword messageα t (o t ) is introduced for indicating the maximum probability value of the partial trajectory till time t given o t , and it can be calculated recursively as Equation 9: α 0 (o 0 ) = log π L (a 0 |s 0 , o 0 ) + log π H (o 0 |s 0 , #). (10) Clearly, Option-Viterbi has a computation complexity of O(T × K 2 ) for a T -step trajectory with K options. By back-tracing o t−1 that induces the maximization onα(o t ) at each time step, the option-extended expert trajectories D demo can finally be acquired.
Although our above option inference process implies that the expert demonstrations are assumed to be generated through a hierarchical policy as the same as the agent, our method is still applicable to the non-hierarchical expert, given the fact that a non-hierarchical expert can be imitated by a hierarchical agent (with fewer options activated).

Summary of Option-GAIL
We briefly give an overview of our proposed Option-GAIL in Algorithm 1. With expert-provided demonstrations D demo = {τ E = (s 0:T , a 0:T )} and initial policyπ, our method alternates the policy optimization and option inference for sufficient iterations.

Convergence Analysis
Algorithm 1 has depicted how our method is computed, but it is still unknown if it will converge in the end. To enable the analysis, we first define a surrogate distribution of the options as q(o −1:T ). We denote Q n as the expected objective in Equation 5 at iteration n by Algorithm 1, namely, Q n = s, a, o, o ) . We immediately have the following convergence theorem for Algorithm 1.
Theorem 2 (Convergence) Algorithm 1 only decreases the objective: Q n+1 ≤ Q n , if these two conditions hold: (1) q(o −1:T ) is our option sampling strategy in E-step and is equal to the posterior of the options given current policỹ π n , i.e. q(o −1:T ) = Pπn (o −1:T |s 0:T , a 0:T ); (2) M-step definitely decreases the objective in Equation 5 at each iteration.
The proof is devised by generalizing traditional EM algorithm to the f-divergence minimization problem. We provide the details in Appendix. Since Q n ≥ 0, Theorem 2 confirms that Algorithm 1 will converge eventually. The first condition in Theorem 2 guarantees that the objective of Equation 3 is equal to Q n after E-step, thus Algorithm 1 also helps minimize Equation 3. Besides, if the global minimum is achieved, In our implementation, as mentioned in § 4.4, we adopt the maximized sampling process instead of the posterior sampling that is required by Theorem 2, as, in this way, we find that it still maintains the convergence in our experiments while reducing the computation complexity with only sampling one option series per trajectory.

Experiments
We evaluate our model on four widely used robot learning benchmarks with locomotion and manipulation tasks. We first compare our model against other counterparts that do not employ hierarchical structure or self-explorations. Then we provide an ablated study to understand the impact of each component in our model.

Environments
Four environments are used for evaluations: Hopper-v2 and Walker2d-v2: The Hopper-v2 and the Walker2d-v2 are two standardized continuous-time locomotion environments implemented in the OpenAI Gym (Brockman et al., 2016) with the MuJoCo (Todorov et al., 2012) physics simulator. On these two tasks, the robot should move toward the desired direction by moving its leg periodically. We use expert demonstration containing 1,000 time-steps for learning on Hopper-v2 environment and 5,000 time-steps for learning on the Walker2d-v2 environment. Both of the expert demonstrations are generated by a policy trained by PPO (Schulman et al., 2017).
AntPush-v0: The AntPush-v0 is a navigation/locomotion task proposed in Nachum et al. (2018), where an Ant robot is required to navigate into a room that is blocked by a movable obstacle, as shown in Figure 10. Specifically, the robot should first navigate to and push away the obstacle and then go into the blocked room. For better comparing the learning performance, we slightly extend the original binary reward as r t = −∆ pos t − tar 2 2 + 100 × 1 pos t =tar where pos is the position of the robot and tar is the location of the blocked room. We use expert demonstrations containing 50,000 time-steps for learning in this environment, where the policy is trained with DAC (Zhang & Whiteson, 2019) regarding our modified reward.
CloseMicrowave2: The Closemicrowave2 is a more challenging robot operation environment in RLBench (James et al., 2020). A 6-DoF robot arm with a gripper is required to reach and close the door of a microwave by controlling the increments on joint positions of the robot arm, as shown in Figure 10. The reward is defined as r t = −∆θ t + 100 × 1 θt=0 , where θ denotes the rotation angle of the door, We use expert demonstrations containing 1,000 time-steps for learning in this environment generated by a handcrafted feedback controller.

Comparative Evaluations
To illustrate the effectiveness of the proposed method, we contrast several popular imitation learning baselines, including: 1) supervised Behavior Cloning (BC) (Pomerleau, 1988) that do not contain hierarchical structure or selfexploration; 2) GAIL (Ho & Ermon, 2016) that uses selfexploration without hierarchical structure; 3) Hierarchical Behavioral Cloning (H-BC) (Zhang & Paschalidis, 2020), which builds upon the Option structure, but optimizing both the high-and low-level policies by directly maximizing the probability of the observed expert trajectories without any self-exploration. This baseline can also be regarded as optimizing Equation 9 with forward-backward messages; 4) a variant of GAIL, denoted as GAIL-HRL that updates hierarchical policies according to Equation 3, where the immediate imitation reward r t = − log D θ (s t , a t ) is used instead. GAIL-HRL can be regarded as a simplified version of our Option-GAIL without using options in occupancy measurement matching, and it is designed for justifying the necessity of involving options during the whole occupancy measurement matching procedure.
We employ the same demonstration trajectories, network structures, and option numbers on each environment for fair comparisons. Specifically, we allow 4 available op-    and (128,128) on Closemicrowave2; the discriminator is realized by an MLP with hidden size (256, 256) on all environments. For methods that do not use self-exploration, we train the policy for 100 epochs and then evaluate it by average reward-sums; for methods that rely on self-exploration, we update the policy and record the average reward-sum every 4096 environmental steps, and record maximum average reward-sums over the whole training period. The evaluation results are displayed in Figure 10.
Obviously, our method gains faster convergence speed and better eventual performance than all baselines in general, as illustrated in Figure 10. For example, on CloseMicrowave2, which is clearly a long-horizon task, our Option-GAIL converges to a remarkably larger reward than all others. Besides, by introducing the option-based hierarchical structure, Option-GAIL, H-BC and GAIL-HRL are superior to the counterparts that use single-level policy, namely, BC and GAIL. When demonstrations are limited, for instance, on Hopper-v2, Walker2d-v2, and Closemicrowave2, GAIL-like

Demo Sample
Ours GAIL-HRL t t Figure 4. Visualization of the options activated at each step, learned respectively by our proposed method (blue) and GAIL-HRL (gray) on Walker2d-v2. 'Demo' denotes the options inferred from the expert, and 'Sample' denotes the options used by agent when doing self-explorations. The effectiveness of our proposed method on regularizing the option switching is obvious by comparing the consistent switching tendencies between Demo and Sample. methods including Option-GAIL and GAIL-HRL outperform their BC-like counterpart H-BC, probably thanks to the reduction of compounding errors by self-explorations. On AntPush-v0, the advantage of our method over H-BC is reduced. We conjecture this is because H-BC is data-hungry, and it can become better when sufficient demonstrations are fed. Comparing with GAIL, with the help of hierarchical modeling, our method can successfully imitate the behavior of the expert with fewer self-explorations.
To examine if our proposed method actually regularizes the options switching between expert and agent, Figure 9 illustrates the options including that are inferred from expert and that generated by agent on Walker2d-v2 with more examples deferred to Appendix. It is observed in our method that the expert's options are consistent with the agent's, while for GAIL-HRL, a dramatic conflict exists between expert and agent. This result confirms the benefit of our method since we explicitly minimize the discrepancy of the option-extended occupancy measurement between expert and agent in Equation 5.

Ablations and Analysis
In this section, we perform an in-depth analysis on each component of our model.
The necessity of using option-extended occupancy: We geometrically assess the difference between our Option-GAIL and GAIL-HRL on Walker2d-v2. To do so, we visualize the trajectories of the expert (squares) and the agent (dots) as well as their options (color) at each time step. Different colors indicate the different options used by the low-policy, and the expert's options are inferred with the agent's policy. The visualizations in Figure 5 read that the agent's trajectory by Option-GAIL almost overlaps with  Figure 6. Ablations on individual components of our proposed method. The curve on the left side compares the maximum average reward-sum within a given range of exploration steps. The options used by agent and inferred from expert are presented one the right side.
the demonstration, whereas the one by GAIL-HRL sometimes tracks a wrong direction with respect to the expert. Moreover, in Option-GAIL, the options of the expert and agent always switch collaboratively with each other, while GAIL-HRL occasionally derives inconsistent options (highlighted by an arrow) between the expert and agent. This phenomenon once again verifies the necessity of performing the option-extended occupancy measurement matching in Equation 5. If otherwise conducting the occupancy measurement matching in Equation 3, it will give rise to ambiguities on the learned policies and eventually affect the learning performance.
Ablations on individual components: To verify the rationality of the model design, we test different variants of our method in Figure 6.
1) Random o. In this case, we generate the expert's options by random without option inference. As observed, the random options mislead policy learning and thus yield worse performance. We further display the option switching in the right sub-figure in Figure 6. The high-level policy tends to be conservative on switching since the sampling options always keep unchanged, making our model less expressive to fit the expert's hierarchical policy.
2) No o t−1 . we simply omit the o t−1 in Equation 6 and implement the discriminator with only (s t , a t , o t ). Clearly, without the option information from the last step, the highlevel policy is not fully regularized and cannot capture the option dynamics, thereby delivering a performance drop compared to the full-option version.
3) Pretrain π H . We pre-train the high-level policy using H-BC for 50 epochs and then fix it during the subsequent learning procedure, which can be regarded as a version of the two-stage method by Sharma et al. (2018). Such ablation explains it is indeed demanded to simultaneously learn the high-and low-level policies end-to-end by examining the results in Figure 6. In the optimization perspective, the two-level policies are coupled with each other and will be improved better under an alternative training fashion.
In summary, all the results here support the optimal choice of our design. Ablations on the number of available option classes: Throughout our above experiments, we set the number of option classes |o| = 4 and find it works generally. In fact, changing |o| does not essentially change our performance if |o| is sufficiently large, as our algorithm will automatically inactivate redundant options (similar sub-task switching will be clustered into the same option class). For illustrating such robustness, we evaluate our method with |o| ∈ {2 · · · 6} on the Hopper-v2 environment. As observed from Figure7, when |o| ≥ 3, all variants share similar convergence behavior and option transition: the number of activated options is observed as 3 for all cases. When |o| ≥ 2, the convergence becomes worse, probably due to its less expressivity.

Discussion and Conclusion
In this paper, we have presented Option-GAIL, a theoretically-grounded framework that integrates hierarchical modeling with option occupancy measurement matching. We then provide an EM-like algorithm for training the policies of Option-GAIL with a provable guaranteed training convergence. Comparing to existing HIL methods, Option-GAIL can regularize both the high-and low-level policies to better fit the hierarchical behavior of the expert. Experiments on robotic locomotion and manipulation benchmarks demonstrate the effectiveness of the proposed Option-GAIL over other counterparts; Option-GAIL works well particularly for the tasks consisting of evidently separable sub-tasks. Our method is generally powerful and can be enhanced by absorbing external knowledge. For example, it could be a future work direction to consider a long-horizon plan as the prior knowledge when inferring options.

A.1. Proofs for Propositions and Theorems
Here we provide the proofs for the propositions and theorems used in our Option-GAIL.
A.1.1. PROOF FOR O ONE-STEP = O According to Assumption 1.1, an option can be activated at any state, thus the intra-option policy π o (a|s), break policy β o (s) and inter-option policy π O (o|s) are all well-defined on any s ∈ S, ∀o ∈ O. This suggests that π L (a|s, o) ≡ π o (a|s) holds over all options on any state. For β o (s), with Assumption 1.2, we have β o (s) = 1 − π H (o |s, o ) and for π O (o|s) o|s,o ) . Also, with o −1 ≡ #, it can be directly found that µ 0 (s, o) =μ 0 (s, o = #) ≡ µ 0 (s). Since S, A, R a s , P a s,s , γ are all defined the same between O one-step and O, we can get thatO one-step = O holds under Assumption 1, and there exists an one-to-one mapping between π H (o|s, o ), π L (a|s, o) and π o (a|s), β o (s), π O (o|s) .
Combining with Theorem 1, this equivalency also suggests: A.1.2. PROOF FOR THEOREM 1 The proof of Theorem 1 can be derived similar as that from Syed et al. (2008) by defining an augmented MDP with options: where we denote o t used inã t as o A t for better distinguish from the option chosen ins t+1 , despite they should actually be the same.
Now we build the relation between the option-occupancy measurement ρπ(s,ã) and the policyπ(ã|s).
With Equation 28 ρ should be a solution of Equation 13-14, and with Lemma 1-2, the solution is unique and equals to the occupancy measurement ofπ. With Lemma 3, ρ is also the unique occupancy measurement of (π H , π L ).