{'fraction_non_alphanumeric': 0.03564172235685785, 'fraction_numerical': 0.011333871875841473, 'mean_word_length': 4.6841519410045915, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploiting Variable Impedance for Energy Efficient Sequential Movements

Compliant robotics have seen successful applications in energy efficient locomotion and cyclic manipulation. However, fully exploitation of variable physical impedance for energy efficient sequential movements has not been extensively addressed. This work employs a hierarchical approach to encapsulate low-level optimal control for sub-movement generation into an outer loop of iterative policy improvement, thereby benefits of both optimal control and reinforcement learning are leveraged. The framework enables optimizing efficiency trade-off for minimal energy expenses in a model-free manner, by taking account of cost function weighting, variable impedance exploitation, and transition timing, which are associated with the skill of compliance. The effectiveness of the proposed method is evaluated using two consecutive reaching tasks on a variable impedance actuator. The results demonstrate significant energy saving by improving the skill of compliance, with a 30% electrical consumption reduction measured on hardware.


I. INTRODUCTION
Intrinsically compliant robots typically have elastic components for stiffness modulation and such elements are capable of storing elastic energy. The field of robotic locomotion has seen a series of successful developments of energy efficient robots with elastic joints or springy legs that can exploit this energy storage. It is of great interest to apply the same principle to robotic manipulators such that soft robots can behave in a human-like energy efficient way for a wide variety of tasks.
Biological springs, like tendons and various elastic elements in muscles, are embedded in human and animals and make them highly efficient runners and jumpers [1]. Utilizing elastic energy storage and recoil, which is associated with optimizing muscular stiffness and transition timing, is a crucial skill that can be practised and improved for many other athletic activities, not limited to locomotion [2].
Physical compliance incorporating elastic components is prominent for energy efficient lower limb locomotion [3], [4], [5]. Also, they have been demonstrated to reproduce the skill of energy buffering in explosive movements such as throwing [6], [7]. Storing and discharging elastic energy, which was called as "skill of compliance" by Okada [8], can amplify the output power, exceeding the power limit of the drive motor. Other recent studies attempt to improve energy efficiency for cyclic manipulation tasks, e.g., repetitive pick-and-place [9] and dribbling a basketball [10]. However, many tasks in unstructured environments are not periodic and variable physical impedance is hard to be fully exploited. For instance, the objects to be picked and placed may locate at random positions. The task given to a robot may consist of a sequence of different types of actions, such like "reach a cup, grasp it, and pour the water". These non-periodic but sequential tasks more commonly involve upper limbs and are complicated by the greater diversity. The problem of taskoriented sequential movement generation -in the context of compliant robotics -faces the difficulty imposed by inherent actuation redundancy. The control redundancy of the actuators, which is somehow equivalent to the muscle redundancy of musculoskeletal arms, makes it non-trivial to optimize the movements in the "muscle space".
Energetic economy is of great importance to reproduce human-like skilled movements. To address this, we aim at optimization of energy efficient sequential movements considering the following aspects for robots driven by variable impedance actuators (VIAs): 1) Cost function weighting. When the form of the cost function is determined, the weighting parameter can be adjusted to tune the energy efficiency. For simple quadratic control effort, the weight for each sub-movement need not be the same and can be optimized according to realistic energetics (by estimation or measurement). 2) Variable impedance exploitation. A movement can adjust physical impedance at transition phases to improve arXiv:2002.12075v1 [cs.RO] 27 Feb 2020 subsequent movements. 3) Relative timing. Temporal characteristics effects the energy efficiency. For a given time horizon, the relative timing is of importance for skilled efficient movements.
These three issues have been addressed separately in the literature. For example, inverse optimal control or inverse reinforcement learning is capable to learn the cost function from human demonstration [11], [12], [13]. [14] exploited variable stiffness actuation for multiphase movements by optimal control, where a brachiation task is used for demonstration. [15] extended optimal control (OC) to include optimization of movement durations. An analogue via approximate inference was provided in [16]. Other works focus on optimizing the sub-goals or attractors of movements encoded by dynamical systems [17], [18].
However, rarely have existing approaches addressed the above targets in the sequential context within one framework. Also, many optimization-based methods rely on combining cost functions of subtasks into a composite one, which intensifies the cost function shaping issue when competing terms join together. Therefore, this paper proposes a hierarchical approach that is capable of optimizing the three aspects identified above and mitigate the cost function shaping issue. More specifically, we employ a bi-level structure to encapsulate low-level OC for sub-movement generation into an outer loop of iterative policy improvement, thereby benefits of both OC and RL are leveraged. The high-level planning formulated as reinforcement learning problem enables optimizing the trade-off balance concerning (low-level) (1) cost function weighting, (2) variable impedance exploitation and (3) transition timing for minimal realistic energetics. The associated policy parameters can be optimized in a derivative-free fashion by evolutionary strategy (ES), which is a black-box optimization (BBO) method for policy improvement suggested by [19]. It can be viewed as a simplification of PI 2 [20] and evolution strategy CMA-ES [21]. OC at low-level naturally resolves the actuation redundancy and exploit variable impedance of VIAs [22], [7], for which there exists efficient solvers e.g., Iterative Linear Quadratic Regulator [23], [24].
The rest of this paper is organized as follows. In §II we discuss relevant literature and concepts. §III first introduces a simple OC example of point-to-point reaching on the single joint VIA. By investigating the efficient frontiers of the OC problem we show how the hyper-parameters is identified and how the reinforcement learning problem is formulated. The proposed method is introduced in §IV. Its effectiveness is evaluated by consecutive reaching tasks on a realistic VIA robot. Simulations demonstrate significant energy efficiency improvement and a reduction of electrical consumption about 30% is recorded on the hardware. Conclusions and future works are covered in §VI.
To avoid confusion, the sequential movements/tasks considered in this paper are sequences in predefined order. The problem of planning the order of executing a set of actions [25] is not within the scope of this work.

A. Sequential Movements
Sequential movements are common found in human daily life, from jaw movement for speech, finger movement for playing musical instruments, to many athletic whole body actions. How these skilful human movements can be learnt, executed and improved? Central to that is whether a hierarchical structure of representation, learning and control of movement sequences exists in human brain. The hypothesis of hierarchical organization of movement planning was proposed a long time ago in mid twentieth century by behaviourist Karl Lashley ([26]). Recent experimental studies have provided evidence of hierarchical representation of movement sequences in the brain. For instance, [27] found that individual finger presses are represented in the primary motor cortex, whereas activities about the sequential context happen mainly in the premotor and parietal cortices.
Many researchers in robot learning control treat the motion generation of complex skills in a hierarchical manner. This may because the skill learning are often investigated at the task level and a complex manipulation task can be naturally decomposed into a sequence of simple subtasks. A complex skill can be learnt from human demonstrations by motion segmentation into movement primitives ( [28]). Then a skilful movement can be composed by a "reportire" ( [29]) of such sequenced sub-movements. By doing so it is expected to realize more general motion intelligence and make robots master interactive tasks and tool use, which is a hallmark of human behaviour ( [30]).
Nevertheless, human can acquire a new skilled movement by sequencing simpler motion primitives and improve it via practice. Although each individual movement is possible to be fine-tuned during training, the increased performance through practice can be clearly attributed to improvements in planning processes, as shown by [31]. Consider a squat that can be composed of crouching and rising-up. By intuition, the contextual variables at the sequence planning level can possibly be transition timing, muscular stiffness, torque distribution, etc. [32] used OC to find optimal transition timing that can reproduce experimentally measured human squat movements. The role of stiffness was investigated by [33] also through biomechanical modelling and OC, which signifies the importance of exploiting elastic energy storage.

B. Optimisation of Sequential Movements
Improvement of a sequential movement necessitates existence of redundancy in either representational level or control level. In the above squat example, the transition timing is not predefined by the task or sub-movements, and thus can be tuned. While for playing a piece of music, the tempo and rhythm are determined, then the transition timing is specified by the task objective and not able to be exploited.
It is easy to notice that sequentially combining the submovements, which are optimized with respect to their subgoals, does not necessarily result in the optimal movement for the whole task. Loot at the example illustrated in Figure 3, III: π 1 := arg min J 1 π 2 := arg min J 2  the minimal-jerk trajectory 1 of a via-point task from point A to C via B (at a specific time) results a curved path in the X-Y plane. While the minimal-jerk model of a single point-topoint movement always shows a straight path. Consequently, if we sequence AB and BC (both individually are minimaljerk) directly, the resulting trajectory is not optimal w.r.t. the whole movement. The difference is simply due to that the velocity at the via point is not constrained to be zero. Though it is obvious, this common phenomenon in kinematic domain shows an example of exploiting the redundancy of velocity profile when concatenating discrete movements.
Based on the above reasoning, it follows that when a sequence is generated by chaining movement primitives, it may be suboptimal without appropriately planning each individual considering the whole trajectory or its subsequent ones. In general, one can structure the problem as a composite optimization or tackle it hierarchically. Depending on whether either way is adopted, or both, there are three possible approaches, as depicted in Figure 2. Throughout this paper, π denotes the control policy, ξ represents vector of hyper-parameters or policy parameter, J is used for cost functional.
To narrow down our discussion, we mainly consider the applications for (i) optimisation of cost function weighting, (ii) exploitation of variable impedance, and (iii) optimising temporal parameters such as the time horizon and relative timing.
1) Composite Optimisation: Composite optimization here means optimizing w.r.t. a composite cost function that consists of the objectives of subtasks. For instance, an optimisationbased approach usually consider the via-point problem by defining the cost function as v , x * f are the via-point and final targets respectively, H v , H f are diagonal matrices to penalize the deviation, and t v , t f represent the fixed via-point time and final time. The optimal control u(x, t) = π(x, t) with corresponding policy π is the one that minimizes the cost functional. The above minimal jerk via-point problem is one example that has analytical solution ( [34]). The shortcoming of this is that if t v is allowed to be adjusted, the optimization of (1) with a guess about t v may leads to suboptimal solutions.
Let us first consider the possibility to simultaneously optimize the control and some hyper-parameter like t v . This is categorized as Type I in Figure 2. For many non-linear real problems arising in robotics, a classical method is to convert the OC problem into a non-linear programming problem. Considering the computational efficiency, a more efficient paradigm for learning control is to transform the representation of the policy into lower-dimensional space, and then optimize the policy parameters and hyper-parameters simultaneously. For example, [18] implemented the (model-free) reinforcement learning algorithm PI 2 for sequential tasks (termed as PI 2 SEQ), with the help of dynamic motion primitives (DMP) for trajectory encoding using dynamical systems. The shape parameter of trajectories and the attractors of dynamical systems are optimized together, so that the trajectory and its final state is optimized for all subsequent actions. The limitation of composite cost function is that it faces the cost function shaping issue. When competing terms from different subtasks come together, optimality of sub-movements may be no longer achievable.
2) Hierarchical Optimisation: The second possible approach is to construct the optimization problem hierarchically. As shown by Type II in Figure 2, it is hierarchical in the sense that an inner loop and an outer loop optimize the control policy and hyper-parameters separately. Various previous studies addressing the multiphase optimal control can be found in this type. To name a few, temporal optimization with Iterative Linear Quadratic Regulator (ILQR-T) and approximate inference (AICO-T) was proposed by [15] and [16] respectively. [15] used finite difference to compute the gradient of total cost w.r.t. change of time durations. The evaluation of the gradient is based on running the time-scaled augmented control and hence is very efficient. This is done by leveraging a technique that maps the real time to a canonical time. It was demonstrated by [16] with similar technique on a via-point task, where the algorithm finds an optimal relative timing. In case J total is non-differentiable w.r.t. ξ, one can utilize derivative-free methods ( [35]) such as trust region technique ( [36]) and evolutionary strategy ( [21]) in the outer loop.
This hierarchical structure coincides with the so-called "bilevel" problem in inverse optimal control ( [11]). In inverse optimal control the outer loop optimizes the cost function shaping to match data demonstrated from human. What we interest in is the fact that the objective J total in the outer loop need not be the same as the composite cost. Suppose that, for the speed and robustness of optimization, the subtasks may be described with simple quadratic terms such as traditional "control effort", or even have different energetic functions individually, but on the high level, the parameter can be updated according to more realistic cost estimator or physical measurement. This potential can be realized within the bi-level architecture.
Note that, since Type II also employ a composite cost function in the inner loop, it shares the same shortcoming with Type I that composite optimization may fail to achieve optimality for all subtasks and thus need redesign. To overcome the drawback, we propose to optimize the sub-movements according to their own cost function as well as integrate the hierarchical (bi-level) architecture, which leads to Type III ( Figure 2). The comparison against previous two types is summarized in the table (Figure 2 (b)).

III. PROBLEM DEFINITION
In this section we first present a OC model of a single joint driven by a VIA, followed by an investigation of energy efficiency based on the concept of efficient frontiers. The intuition gained thereby helps with justifying the problem formulation. Finally, a reinforcement learning problem is formulated that enables optimizing high-level parameters using policy improvement methods.

A. A Simple Reaching Movement Model
Consider a point-to-point fast reaching task using one-link robot driven by a VIA. The robot used in this paper is MAC-CEPA [37] with variable damping [38] (VD). As illustrated in Figure 4, the equilibrium position (EP) is controlled with SERVO1 and stiffness is regulated by spring pretension via SERVO2. The mechanism was implemented in our previous work [39] with energy regenerative damping [40], where the damping is modulated by controlling a dedicated switching circuit that adjusts back-electromotive force on a DC motor τ q Joint Figure 4: Diagram of MACCEPA-VD [37], [38].
attached to the joint. The system model provided in Appendix A.
A fast reaching task is represented by a cost functional An optimal control problem can be formulated as to seek an optimal control u(t) ∈ U ∈ R 3 constrained by its admissible set U = {u ∈ R 3 | u min u u max }, that minimizes the cost functional (2) and subject to the state-space model of the robot dynamics. In the cost function, w e serves as a weighting parameter to enable adjustment of the performance-cost tradeoff.
In addition to trade-off balance via cost function weighting, the stiffness at transition could have a significant influence on the energy efficiency of the subsequent movement. This is explained as follows.

B. Efficient Frontiers of Optimal Control
Efficient fronter (EF) is a common tool to examine the tradeoff of two competing objectives in an optimization problem. The above OC problem has an efficient frontier by varying the weighting parameter w e . Then the distribution of optimal solutions can be visualized in performance-cost plane.
In addition, to investigate how pre-stored elastic energy affect the energy efficiency, we generate the optimal solutions by Iterative Linear Quadratic Regulator for different values of w e , with a certain minimal spring pretension to produce an EF. Multiple EFs are generated by changing the condition of minimal spring pretension. This is done by setting the initial stiffness motor angle θ 2 (0) and the lower bound u (2) min of u 2 to a preset value p s , i.e., let θ 2 (0) = u min The results are shown in Figure 5. The vertical axis represents the reaching accuracy performance, which is the terminal cost (3)  input mechanical work 2 E in and electric work E elec , both estimated by simulation. 3 where [·] + = max(0, ·). For in-depth analysis of modelling motor energy consumption, we refer the readers to [41]. Calculation of the mechanical and electrical power is given in Appendix A. Looking at Figure 5, when increasing the weight of control effort w e (as shown by the direction of green arrow), both mechanical and electrical consumption are decreased, with some loss of reaching performance. It demonstrates that even with a simple quadratic control cost, it is still possible to tune the trade-off between performance and realistic energy measures. Moreover, it can be seen in that, by increasing the minimal spring pre-tension p s , the efficient frontiers move towards the bottom-left, which signals an overall improvement of energy efficiency.
Overall, the above investigation based on the tool of efficient frontier suggests that control cost weight w e and minimal stiffness p s can be taken as hyper-parameters that tunes the performance-cost trade-off according to realistic energy measures.

C. Reinforcement Learning Formulation
Based on the previous rapid reaching OC model, now consider consecutive reaching task for example, that requires the arm to reach a sequence of targets {q * i } Ns i=1 from initial state x 0 , where N s is the number of subtasks. The sequential movement S := {M i } Ns i=1 consists of N s sub-movements generated by solving optimal control problem (OCP). The subproblems is denoted as {OCP i } Ns i=1 . We define ξ ∈ {ξ ∈ R dp | ξ min ξ ξ max } to be the stacked vector of weighting parameter w e = {w Note that, depending on the type of task at hand, p s may have different meaning. For example, as in §III-B it is used to set the minimal stiffness motor command. By doing so it constrains the minimal elastic energy to be stored and sets a target for the motor.
Our problem is to find an energy optimal trajectory S and ξ that minimizes energy cost while achieving all sub-goals. Mathematically, it is formulated as to minimize the episodic cost: The cost objective (7) is formulated as an episodic cost. J e is the energy consumption, and J p is the cost associated with task achievement. The amount of J p exceeding an upper boundJ p is penalized by a large constant C. The energy consumption can be estimated by a cost functional or measured on hardware. J p is evaluated by solving {OCP i } with initial ξ (0) .

IV. POLICY IMPROVEMENT FOR SEQUENTIAL MOVEMENTS
The policy improvement optimizes J in an iterative process. Figure 6 outlines the paradigm of general policy improvement procedures: exploration, evaluation, and policy update. Different from the vanilla reinforcement learning from exploration and evaluation we have an inner loop to solve {OCP i } sequentially.

... Exploration
Evaluation Update Initial trajectory p . The tolerance factor is introduced for user to trade-off the energy efficiency flexibly. A positive value allows the exploration for some worse samples to avoid being too "greedy".

A. Exploration and evaluation
The exploration phase generates K unconstrained perturbations in policy parameter space for K roll-outs. The perturb-ations˜ k ∼ N (0, γ n−1 Σ ), (k = 1, ..., K is assumed to obey normal distribution, where Σ is the covariance matrix and γ ∈ (0, 1) is the decay factor. Then the box constraintb and b is applied to yields When running each k-th roll-out, ξ (n)[k] is used to specify sub-problems with w e for cost functional J i , p s for stiffness motor constraint, and t d for time horizon. Without loss of generality, we assume the time horizon [t 0 , t f ] of i-th subproblem is from t 0 = 0 to t f = t for k = 1 to K do k-th rollout 6: Sample˜ k ∼ N (0, γ n−1 Σ ) Unconstrained perturbations 7: Specify hyper-parameters and constraints of {OCP i } according to ξ (n) [k] 10: 12: solve {OCP i }, u i = arg min J i 13: Update ξ (n+1) using (11) and (12) 20: Keep µ best samples for sample reuse 21: until ξ converges or maximum number of iterations reached dynamics and control u = {u i }. After running K roll-outs and collecting relevant costs, the total costs J [k] are calculated by (7).

B. High-level policy update
The policy update step (10)-(12) utilizes the rewardweighted averaging rule as introduced by [19].
First the cost J [k] is normalized according to their maximum and minimum by (10). The normalized costJ [k] is used to calculate probability P k for k-th roll-out according to (11), where c > 0 is a constant. 6 Finally, the update is computed by the weighted averaging rule (12).
The above weighted averaging technique is simplified from PI 2 ( [19]) and resembles the evolutionary strategy (µ, λ)-ES. It is appealing because it can solve non-linear non-convex black box optimization problems with reasonable efficiency. Unlike CMA-ES ( [21]), it does not have the covariance matrix adaption step. Instead, we manually specify a decay factor γ to gradually decrease the variance of perturbations.
For better robustness of convergence, another technique employed is sample reuse. After every update, we keep µ best samples among K roll-outs (at current iteration) for next update. Therefore, after the first iteration, we have K+µ samples. The exploration, evaluation and policy update procedures are repeated until ξ converges or reaches maximum steps. The whole algorithm is summarized in Algorithm 1 and termed as OC-ES, which stands for Optimal Control (at low-level) with Evolutionary Strategy (at high-level).

V. APPLICATIONS AND EVALUATIONS
Task 1 -consecutive fast reaching: To evaluate our proposed method, a consecutive fast reaching task is designed to test on the MACCEPA-VD robot. The task requires the joint actuated by MACCEPA-VD to reach a sequence of three targets {q * i } 3 i=1 := {0.7, −0.35, 0.3} (radians) rapidly within a fixed time horizon T i = 1, for i = 1, 2, 3, from initial state x 0 = (0, 0, 0, π/24, 0, 0) T . The cost functional J i for each subtask is defined by (2) -(4). Single fast reaching problem was used by [38] to investigate the role of variable damping for VIAs when appropriate amount of damping is needed to suppress oscillation of movements. In our previous work in [39], a consecutive (stochastic) fast reaching experiment was conducted to examine how combination of variable stiffness and regenerative damping influence the overall energy efficiency. There, the minimal spring pretension between two movements was manually set by user experience, whereby the variable stiffness was not fully exploited in the sequential context.
For comparison, a benchmark is generated by using Iterative Linear Quadratic Regulator to solve the sub-problems sequentially. The spring preset p (i) s = π/24 rad is lower bound of stiffness motor position command and weighting parameter w (i) e = 1 for i = 1, 2, 3. The resulting (approximately) optimal trajectory S is denoted by ILQR-0, and used as initial trajectory later for our proposed method.

A. Task 1: policy improvement with parametrised trajectory
The competing terms in the composite cost function may hinder the fulfilment of all sub-goals. To investigate this issue we directly optimize the trajectory and stiffness profile simultaneously w.r.t. the composite cost function of Task 1 The trajectories are parametrized by DMPs as introduced in Appendix B. Each sub-movement consists of 3 DMPs representing the trajectories of EP motor, stiffness motor and damping command. All DMP are initialized with shaping parameter w = 0, which is a 10 dimensional vector. The goals g 1 , g 2 , g 3 , for EP, stiffness motor, and damping respectively, are initialized as g 1 = q * , g 2 = 24/π e (3) , g 3 = 0.5 e (3) , 7 where e (d) represents a d-dimensional unit vector. The shaping parameter w is unconstrained. The box constraints are [−π/3, π/3], [π/24, π/2], [0, 1] for elements of g 1 , g 2 and g 3 respectively. The overall policy parameter ξ is a 99-dimensional stacked vector of g and w of all three sub-movements. Relevant hyper-parameters of the algorithm are γ = 0.95, µ = 15, K = 45 and Σ = diag(10 e (90) , 0.5 e (9) ) ∈ R 99×99 , Both goals and shaping parameters w of DMPs are optimized simultaneously by Algorithm 1 except that it doesn't have an inner loop. The policy update rule used is the same as the weighted averaging method (10)- (12). 8 The learning results of 10 sessions are illustrated in Figure 7. The policy update gradually converges and final energy cost evaluated by input mechanical work E in is successfully reduced from 0.2674 J of ILQR-0 to 0.1843 ± 0.0204 J. The final trajectory of one learning session shown in Figure 8 demonstrates that an optimized p s regulates the pretension at the transition phases. The effectiveness of policy improvement with parametrized trajectories for exploiting variable impedance of VIAs is verified despite some drawbacks. First it can be seen that the learning takes thousands of (trajectory) samples due to high dimension of ξ, which makes it less likely to be executed on the physical robot in an online fashion. Secondly, the joint trajectory in Figure 8 slightly but visibly deviated from first two goals, because the integral term in (15) competes with the terminal cost of its previous movement. To circumvent this issue the composite cost function needs redesign to adjust the cost terms, weights, or impose extra constraints.

B. Task 1: sequential reaching with OC-ES
Now we take both weighting and stiffness parameters w e , p s into account and employ the OC-ES framework. The policy parameter ξ for Task 1 consists of weights of control effort term and stiffness motor preset of each sub-problem. ξ is initialized as {w e = 1, p s = π/24 rad} 3 i=1 . We run K = 4 roll-outs for each policy update up to 100 iterations. The exploration noise Σ = 0.5 I ∈ R 6×6 and decay factor γ is set to 0.95. Sample reuse parameter is chosen to µ = 3. The initial trajectory is evaluated to record its energy cost E  Quadratic Regulator. After each policy update, the movement without perturbation is evaluated to record the learning performance. To verify the improvement of energy saving, both initial and final trajectories are executed on the hardware to record the energy consumption. The results are summarized in Figure 9 where ILQR-ES denotes the final trajectory.
1) Significant energy reduction: The learning curve in Figure 9 (a) shows a fast convergence after 50 iterations and very small variations as also shown in Figure 9 (b) (c). It demonstrates that the OC-ES method successfully reduced the energy cost of the whole task with worse performance cost within the tolerance. The mechanical energy cost in simulation is decreased about 44% from that of the initial ILQR-0 trajectory. The electrical consumption recorded on the servomotors verifies the result with a 29.6% reduction.
2) Optimally tuned cost function weighting: Looking at Figure 9 (d), all w e of sub-problems tend to increase from the initial settings. Despite relative large variations of the optimization result, it can obviously be seen that the second sub-movement takes the highest weight for control cost, which indicates the energy efficiency of it is most critical. In Figure 10 we can see that the second sub-movement has the largest travel distance among the three, and consumes the most energy in the initial trajectory (as shown by the cumulated energy cost in the right column in Figure 10). Hence, the result can be explained as the optimization adjusts the weight to balance the performance-cost trade-off more towards reducing energy cost.
3) Exploiting variable stiffness: It can be seen in Figure 10 that energy reduction occurs significantly during the second and third movements, compared with the initial trajectory. The stiffness motor maintains higher pretension at transition phases ( Figure 10 (e)) due to the constraint imposed by optimized p s , by which the acceleration of the subsequent movement consumes less energy in the EP motor. Also, the adjustment of stiffness motor causes a lot of electrical consumption (Figure 10 (h)), suggesting that the control effort may be lead to suboptimal solutions regarding real energy consumption. However, this highly depends on the variable stiffness mechanism and hardware design. For example, by implementing the variable stiffness actuators designed for minimizing energy cost for stiffness modulation ( [42], [43]), the energy cost of the stiffness motor of initial trajectory can be reduced so that the most saving occurs on the EP motor. However, it would raise another problem that if a variable stiffness actuator does not require energy input to adjust stiffness, then it may not be able to pre-store energy at equilibrium position (EP). As a result there may be no energy buffering effect for some movements starting from a static equilibrium phase. No.  Overall, the experiment demonstrates the effectiveness of applying OC-ES framework to improve energy efficiency by exploiting variable stiffness and cost function tuning. The learning takes only 4 explorations per iteration by leveraging model-based OC at the low-level, making it more feasible to run on the real robot.

C. Temporal and Stiffness Optimisation for Tracking Control
The second application is to show that the proposed framework can be applied to temporal optimization and work with low-level tracking controller.
Task 2 -Consecutive trajectory tracking: This task requires the arm to smoothly reach a sequence of targets with minimaljerk joint trajectory. In addition to exploit variable stiffness, the relative timing is allowed to be optimized. We set the targets as {q * i } = [π/5, −0.2, 1, 0.3] (rad). The arm starts at q(0) = 0 rad. The total time for the movement is 2.4 s. ξ is defined as . Since the total time is kept the same, the last time duration is excluded from the policy parameter. The box constraint on ξ is The minimal-jerk join trajectory can be computed analytically by formula introduced in [34], given the time duration t d and where it begins and ends. Then it becomes a join space tracking problem. The joint tracking with extended inverse dynamics controller (TIDC) derived in Appendix C to track the joint trajectory and resolve the actuation redundancy automatically. The controller serves as a feedback control law and reduces the inner loop OCP to a forward pass of dynamics. The stiffness parameter p s is used to impose a constraint on the target position of stiffness motor in each sub-movement, by adding a null-space controller for i-th trajectory tracking. This null-space controller encourages the EP motor moves towards the joint target and stiffness motor to p  Figure 11. It can be seen that the learning curve initially has a large variation but quickly converges after 40 iterations. Compared to the initial trajectory, by exploiting stiffness and temporal optimization the input mechanical energy E in reduces about 42%. Looking at the results in Table I, the optimized stiffness targets range from 1.26 rad for the first sub-movement to nearly 0 rad for the last one. It results in the pretension increasing during the first two sub-movements then decreasing towards the end (as shown in Figure 12). Moreover, the duration of third sub-movement is optimized to 692.6 ms, which is 92.6 ms more than the initial setting. While other three sub-movements have shorter durations. The result is coherent with the order in terms of movement distance.

D. Discussion
The experiments presented in this section demonstrated noticeable energy saving realized in consecutive reaching tasks. The consecutive fast reaching task was used in [39] where a default spring pretension was chosen manually for all movements.
The proposed method in this paper has been demonstrated to help the robot automatically regulate its stiffness with awareness of subsequent movements. The result of ILQR-ES for Task 1 regulates the stiffness motor to maintain at a small range around 0.5 rad, suggesting that a fixed value can be tuned for energy efficiency in practice if the movement distances are not distributed diversely. In general, it suggests that for VIAs that rely on spring pretension to modulate stiffness, the more efficient way to use them in consecutive point-to-point reaching is not reset its stiffness to minimum by default. According to this, the use of control effort is sceptical with an emphasis on minimal energy. However, due to the fact that accurate estimation of E in , E elec needs a much smaller time step for discretization of the continuous system dynamics, the quadratic control effort is preferred for less computation cost. It also enhances smoothness of the trajectory, although at a cost of energetic optimality. Nevertheless, this loss is alleviated -by applying the proposed framework -with an upper layer optimizer that adjusts the trade-off balance.

VI. CONCLUSIONS
This paper proposed a versatile framework that integrates Optimal Control and Evolutionary Strategy (OC-ES) in a bi-level structure to address the optimization of movement sequence specifically for VIAs. At the low-level OC is leveraged to resolve the actuation redundancy and exploit variable impedance naturally. The high-level planning in sequential context is formulated as a reinforcement learning problem, in the form of iterative policy improvement, and solved as a black box optimization using method inspired by evolutionary strategy.
The proposed framework was applied for two consecutive reaching tasks on a MACCEPA-VD actuator, one requires reaching as quickly as possible, the other tracks a smooth trajectory in joint space. In both cases natural dynamics is hard to be exploited for energy buffering as in periodic movement. By investigating the performance-cost trade-off via efficient frontiers, it can be seen how cost function weighting and minimal stiffness preset influence the energy efficiency. These two aspects can be addressed in the sequential movement context via the proposed framework, by which variable impedance can be fully exploited and the low-level trade-off is optimally balanced. In addition, a tracking controller that resolves the actuation redundancy was implemented to show the temporal and stiffness optimization at high-level can improve the energy efficiency of low-level sequential tracking control. All the experiments presented in §V demonstrated significant improvement of energy efficiency in both simulations and on hardware.
However, this work has been limited to reaching movements. More task types need to be considered in the future work to demonstrate more complex behaviours. Furthermore, it would be interesting to extend the application on compliant robots with multiple DOFs and consider problems involving contacts and interactions.

A. Model
The forward dynamics of MACCEPAVD can be written as: where q,q,q are the joint angle, velocity and acceleration, respectively, b is the viscous friction coefficient for the joint, m is the link inertia, τ s is the torque generated by the spring force, and τ ext is the joint torque due to external loading (the following reports results for the case of no external loading, i.e., τ ext = 0). θ 1 , θ 2 ,θ 1 ,θ 2 ,θ 1 ,θ 2 are the motor angles, velocities and accelerations. The motor angles θ 1 , θ 2 and damping d are controlled by control input u = (u 1 , u 2 , u 3 ) T . The servomotor dynamics (19), (20) are assumed to behave as a critically damped system, with β constraining the maximum acceleration of the 2nd order dynamical system.
On the hardware, SERVO1 and SERVO2 are two Robotis Dynamixel XM430-210-R servomotors with internal position and current sensors. The sensing data is transmitted from servos to a PC hosting connected with a dedicated U2D2 USB converter. The communication between servos and PC is based on ROS messages.

B. Dynamic Movement Primitives
A widely-used formalization is Dynamic Movement Primitive (DMP) proposed by [44] and [45], [46], based on the idea of dynamical system based modelling. Below is a formalization of DMPs for representing actuator variables θ = (θ 1 , θ 2 , θ 3 ) T represents the EP, stiffness motor and damping profile.
where τ > 0 represents the duration and g is the goal position of θ, the dynamics of θ is regulated by a dynamical system behaves like a mass-spring-damper model, with gains determined by α z , β z . f θ (s) is a forcing term manipulating the shape of trajectory. It is a function in the phase variable s, whose dynamics makes it asymptotically converge to 0, in a rate controlled by α s . As a result, The efficacy of forcing term gradually decays to zero. This behaviour is purposely designed by the [45], [46] to enhance convergence of θ to goal g. In addition to s, A is added to the forcing term to scale it according the movement distance, where the m-th element a m = g m −θ m corresponds to m-th forcing element f θ m . From (32) and (33) we can see that the forcing term is defined as the weighted sum of a set of N basis functions, of which each is an exponential function defined by centre point c i and width factor σ i .

C. Tracking Joint Trajectory with Inverse Dynamics Controller
The controller here is general for multiple DOF. q is the joint configuration vector Suppose that the robot is asked to track a desired trajectory {q des ,q des ,q des } and satisfies ( ... q− ... q des )+K 3 (q−q des )+K 2 (q−q des )+K 1 (q−q des ) = 0 Then taking derivatives of M(q)q + C(q,q)q + G(q) = τ a (q, θ) where J θ , J q , Jq are used to represent the Jacobians of τ a w.r.t. θ, q,q. For the following derivation we assume that θ is controlled in velocity domain by v.
Combining (34) and (36), after rearranging them, we get Denote the right-hand side as b, given a cost metric matrix N , the control law can be given as which is a closed form controller with joint feedback.