{'fraction_non_alphanumeric': 0.028138361881352894, 'fraction_numerical': 0.006759351716721714, 'mean_word_length': 5.004304381245196, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Benchmarking Deformable Object Manipulation with Differentiable Physics

Deformable Object Manipulation (DOM) is of significant importance to both daily and industrial applications. Recent successes in differentiable physics simulators allow learning algorithms to train a policy with analytic gradients through environment dynamics, which significantly facilitates the development of DOM algorithms. However, existing DOM benchmarks are either single-object-based or non-differentiable. This leaves the questions of 1) how a task-specific algorithm performs on other tasks and 2) how a differentiable-physics-based algorithm compares with the non-differentiable ones in general. In this work, we present DaXBench, a differentiable DOM benchmark with a wide object and task coverage. DaXBench includes 9 challenging high-fidelity simulated tasks, covering rope, cloth, and liquid manipulation with various difficulty levels. To better understand the performance of general algorithms on different DOM tasks, we conduct comprehensive experiments over representative DOM methods, ranging from planning to imitation learning and reinforcement learning. In addition, we provide careful empirical studies of existing decision-making algorithms based on differentiable physics, and discuss their limitations, as well as potential future directions.


INTRODUCTION
Deformable object manipulation (DOM) is critical to various applications, ranging from household scenarios (Maitin-Shepard et al., 2010;Miller et al., 2011; to industrial applications (Miller et al., 2012;Zhu et al., 2022). Built upon deformable object simulators, DOM benchmarks have provided a platform for algorithm development and prototyping . However, given the extremely high-dimensional state space and action space on a DOM task, learning a generalizable and effective policy remains challenging.
In contrast to standard non-differentiable simulators, differentiable physics simulation constructs physics rules as differentiable computational graphs, which allows direct optimization of a control policy with analytical environment gradients (Freeman et al., 2021;Hu et al., 2020). Because of their particle-based or mesh-based representations, deformable objects can be naturally connected with differentiable simulations. By directly propagating the gradients from optimization objectives, e.g., cumulative rewards, to a learning policy through environment dynamics, we can use the groundtruth simulation gradients to reason about the dynamics and improve sample efficiency during policy learning (Chen et al., 2022;Hu et al., 2020;Huang et al., 2021;Heiden et al., 2021), which is favorable in the context of DOM. However, most of the existing differentiable DOM benchmarks focus on a single task, and provide no evidence to the generalization ability of the resulting task-specific algorithms.
In this work, we present DaXBench, a differentiable and comprehensive high-performance benchmark for DOM. DaXBench relies on a Deformable object simulator, DaX, which connects the recent advances of deformable object simulation algorithms Chen et al., 2022) to the high-performance computational framework, JAX (Bradbury et al., 2018). In particular, powered by JAX, DaX enjoys the auto-differentiation and parallelization to multiple devices. From the task setup perspective, in contrast to single-task-based benchmarks, DaXBench covers a wide range of object types, including rope, cloth, liquid, and elasto-plastic object manipulations. Considering object types, a wide range of tasks with varying difficulty levels and well-defined reward functions are presented, such as liquid pouring, cloth folding, and rope wiping. Specifically, we design tasks with high-level macro actions and low-level control-based action spaces to fully examine the performance of different algorithms. All task environments are wrapped with OpenAI Gym APIs (Brockman et al., 2016) and Python interfaces, such that DaXBench can be seamlessly connected with a DOM algorithm for fast and easy development.
In our experiments, by comparing the algorithms with and without analytic gradients side-by-side on each task, we provide deeper understanding of the benefits and challenges to analytic policy optimization in DOM. We discuss the limitations of existing methods and potential future directions with DaXBench. In addition, we demonstrate that the dynamics model in DaXBenchis sufficiently realistic that allows direct sim-to-real transfer to a real robot on rope manipulation tasks.

DEFORMABLE OBJECT SIMULATORS
The progress in Deformable Object Manipulation (DOM) partially stems from the recent emergence of DOM simulators in the past years. These benchmarks and simulators have significantly advanced the state-of-the-art in modeling the complex dynamics of DOM and provide the firmament of the algorithmic innovations for the DOM methods. SoftGym  is the first DOM benchmark that models all liquid, fabric, and elastoplastic objects and introduces a wide range of standardized DOM tasks. However, SoftGym's simulator is non-differentiable and thus does not support methods based on differentiable physics. On the other hand, several differentiable DOM simulators emerge, such as ChainQueen (Hu et al., 2018b), Diff-PDE (Holl et al., 2020), PlasticineLab (Hu et al., 2020), DiSECt (Heiden et al., 2021), and DiffSim (Qiao et al., 2020). Each of these simulators specializes in modeling a single type of deformable object and supports a narrow range of object-specific tasks. For example, PlasticineLab (Hu et al., 2020) and DiSECt (Heiden et al., 2021) model only the elastoplastic objects, including tasks such as sculpting, rolling, and cutting the deformable objects. Since the dynamics of each deformable object are vastly different, the physic engine specialized in modeling one type of deformable objects cannot be easily extended to another. DaXBench bridges this gap by proposing a set of differentiable benchmarks that cover a wide range of deformable objects and the relevant tasks. We provide a fairground for comparing and developing all types of DOM methods, especially the differentiable ones.

DEFORMABLE OBJECT MANIPULATION ALGORITHMS
Deformable Object Manipulation (DOM) is more challenging than its rigid body counterpart due to the infinite dimensionality of the state space and the corresponding complex dynamics. These invalidate the direct application of the existing methods on rigid object manipulation. DOM methods need to explicitly handle the enormous DoFs in the state space and the complex dynamics. Despite the challenges, many interesting ideas have been proposed for DOM. Depending on the algorithmic paradigms, we categorize the DOM methods into three groups: planning, Imitation Learning (IL), and Reinforcement Learning (RL). Our discussion below focuses on the general DOM methods for a standardized set of tasks. We refer to (Sanchez et al., 2018;Khalil & Payeur, 2010) for a more detailed survey on prior methods for robot manipulation of deformable objects. Reinforcement Learning algorithms using differentiable physics such as SHAC  and APG (Freeman et al., 2021) are also explored. These methods use the ground-truth gradients on the dynamics to optimize the local actions directly, thereby bypassing the massive sampling for the policy gradient estimation.
Imitation Learning is another important learning paradigm for DOM. To overcome the "curse of dimensionality", the existing works such as Ganapathi et al., 2020) overcome the enormous state/action space by learning the low dimensional latent space and simplifying the primitive actions to only pick-and-place. ILD (Chen et al., 2022) is another line of IL method that utilizes the differentiable dynamics to match the entire trajectory all at once; therefore, it reduces the state coverage requirement by the prior works.
Motion planning for DOM has been explored in (Lippi et al., 2020;Nair & Finn, 2019;Yan et al., 2020). These methods overcome the enormous DoFs of the original state space by learning a lowdimensional latent state representation and a corresponding dynamic model for planning. The success of these planning methods critically depends on the dimensionality and quality of the learned latent space and the dynamic model, which in itself is challenging.
We believe that developing a holistic benchmark with its differentiable simulator can provide a fairground to compare with the existing methods. This allows the researchers to better understand the progress and limitations of current DOM methods and offers further insights to advance the progress of DOM methods from all paradigms.

DAXBENCH: DEFORMABLE-AND-JAX BENCHMARK
In this work, we introduce DaXBench, a holistic differentiable DOM benchmark that covers a wide range of deformable objects and manipulation tasks. To benchmark the DOM methods from all paradigms, especially the differentiable ones, we implemented our own differentiable simulator, Deformable-and-JAX (DaX), which is highly efficient and parallelizable. In this section, we will first briefly introduce our simulator DaX. Then we shall focus on introducing the benchmark tasks we implemented for DaXBench.
3.1 OUR SIMULATOR: DAX All materials including the soup are simulated by DaX using particles. Moreover, the entire simulation is fully differentiable and highly parallelizable.
We introduce DaX, a differentiable simulator that specializes in modeling various types of deformable object manipulation with high parallelization. Comparison with other existing simulators has been shown in Table 1. DaX is implemented using JAX (Bradbury et al., 2018), which supports highly paralleled computation with great automatic differentiation directly optimized on the GPU kernel level. In addition, DaX can connect seamlessly with the plethora of learning algorithms implemented using JAX, thus enabling the entire computation graph to be highly parallelized end-to-end.  (Sulsky et al., 1994;Hu et al., 2018a) to model their dynamics on the particle level. On the other hand, for the cloth that only undergoes limited deformation, DaX models its state using mesh and uses the less expensive mass-spring system to model its dynamics. With these modeling choices, DaX can strike a balance between the modeling capacity and its complexity. We refer the readers to the appendix for more details.

State Representations and Dynamic Models
Designed for Customization DaX allows full customization of new tasks based on the existing framework. Objects with arbitrary shapes and different rigidness can be added to the simulation. Primitive actions can also be customized.
Saving Memory DaX uses two tricks to optimize the efficiency and memory consumption of the end-to-end gradient computation, namely lazy dynamic update and re-evaluation of the gradients.
To optimize MPM's time and memory consumption for a single timestep, DaX lazily updates the dynamic of a small region affected by the manipulation at that step. In addition, DaX only stores the values of a few states at the forward pass; during the backward propagation, DaX re-computes segments of the forward values starting from every stored state in the reverse order sequentially for the overall gradient. We refer the reader to the appendix for details.

OUR BENCHMARK TASKS
DaXBench consists of a wide variety of Deformable Object Manipulation tasks, including manipulating liquid, rope, cloth, and elastoplastic objects, as illustrated in Figure 2. In addition to the tasks presented in the previous work, we propose a few new challenging long horizon tasks that require finer-grained control; these new tasks can better benchmark the scalability of the DOM methods. The horizon of the task is mainly determined by how many steps it takes human experts to finish the task. DaXBench, therefore, offers a common ground to compare the existing methods and future work. In this section, we detail the tasks supported by DaXBench.

Long Horizon Tasks
The policy complexity grows exponentially with the planning horizon, known as "curse of dimensionality"; this is exacerbated by the large state space of deformable objects. The existing DOM benchmark tasks have relatively short horizons. For example, the cloth folding and unfolding tasks in SoftGym consist of 1-3 steps. DaXBench provides tasks with longer horizons.
Tasks without Macro-Actions A macro-action can be seen as executing raw actions for multiple steps. For example, we can define a pick-and-place or push macro-action as a 6-dimensional real vector (x, y, z, x , y , z ) representing the start (x, y, z) and end (x , y , z ) positions of the gripper, and the raw actions are Cartesian velocities of the gripper. Macro-actions reduce the dimension of the action space and the effective horizon, and hence increase the scalability requirement on the DOM methods. However, macro-actions might be too coarse for some tasks. Taking whipping a rope to hit a target location as an example, the momentum needs to be adjusted at a relatively high frequency throughout the execution. DaXBench includes tasks that have no suitable macro-actions.

LIQUID MANIPULATION
Pour-Water Pour a bowl of water to the target bowl as quick as possible. The agent directly controls the velocity and rotation (in Cartesian space) of the gripper that holds the bowl. In particular, the action space of the agent is a 4-dimensional vector represents the linear velocity of the gripper, and w denotes the angular velocity around the wrist.
Pour-Soup Pouring a bowl of soup with various solid ingredients to the target bowl as fast as possible. The actions are the same as those of Pour-Water. This task is more challenging than Pour-Water due to the additional interactions between the liquid and the solid ingredients. In particular, the agent needs to promptly react when a solid ingredient is transferred, so as to 1) adjust for the sudden decrease in load on the gripper and 2) control the momentum of the ingredient when hitting the soup surface to minimize spilling.
Neither tasks have suitable high-level macro-actions and the task horizons are 100 steps.

ROPE MANIPULATION
Push-Rope Push the rope to the pre-specified configuration as quickly as possible. We randomly initialize the configuration of the rope. The agent uses a single gripper to execute the push macroactions. The horizon for this task is 6 steps.
Whip-Rope Whip the rope into a target configuration. The agent holds one end of the rope and controls the gripper velocity to whip the rope. In particular, the action space of this task is a 3dimensional vector a = (v x , v y , v z ) that denotes the velocity of the gripper in the Cartesian space. This task cannot be completed by the abstract macro-actions since the momentum and the deformation of the rope has to be adjusted at a relatively high frequency. The task horizon is 70 steps.

CLOTH MANIPULATION
Fold-Cloth Fold a piece of flattened cloth and move it to a target location. This task has two variants: the easy version requires the agent to execute 1 fold and the difficult version 3 folds. For both variants, the task begins with the cloth lying flat on the table at an arbitrary position. The agent executes pick-and-place macro-actions. The task horizon is 3 and 4 for the easy and difficult versions respectively.
Fold-T-shirt Fold a piece of T-shirt to a target location. The task begins with the T-shirt lying flat on the table at an arbitrary position. The agent executes pick-and-place macro-actions. The task horizon is 4.
Unfold-Cloth Flatten a piece of folded cloth to a target location as fast as possible. This task also has two variants that differ in the initial configurations of the folded cloth: the cloth in the easy version has been folded once and that in the difficult version has been folded three times. This task has the same action space as the Fold-Cloth task. The task horizon is 10 steps for both variants.
All of these tasks share similar APIs to the OpenAI Gym's standard environments. We provide a general template to define tasks in our simulator. This not only better organizes the existing task environments, but also allows the users to customize the existing task environments to better suit their research requirements. The variables and constants defined in the template are easily interpretable quantities that correspond to real physical semantics. This allows the users to better understand the environment, which can also help with the development and debugging process.

EXPERIMENTS
In this section, we aim to answer the following two questions: 1) Are the existing representative methods capable of solving the tasks in DaXBench? 2) Within each paradigm, will the differentiable simulator help in completing the tasks?
To investigate these two questions, we benchmark eight representative methods of deformable object manipulation, covering planning, imitation learning, and reinforcement learning. Each paradigm has different assumptions and requirements for input knowledge. We summarize the baseline methods and the relevant information in Table 2. In this section, we first introduce the general experiment setups for all methods. Then, we discuss the details of the evaluated methods and their performance analysis for each paradigm. Next, we summarize the insights and discuss the directions for future DOM methods. Finally, we demonstrate that our simulator DaX only incurs a small sim-2-real gap by conducting a real robot experiment.    Table 4 for the numerical results in the appendix.

EXPERIMENTAL SETUP
We report the performance evaluated under our ground-truth reward for all tasks for each method. We report the mean and variance of the performance for each task over 5 different seeds.

Reward function
For each task, the goal is specified by the desired final positions of the set of particles, g, representing the deformable object. In this setup, the reward can be intuitively defined by how well the current object particles match with those of the goal. Hence, we define the groundtruth reward as r gt (s, a) = exp(−λD(s , g)), where s is the next state resulted by a at the current state s, and D(s, g) is a non-negative distance measure between the positions of object's particles s and those of the goal g.
However, using the ground-truth reward along is inefficient for learning. The robot's end-effector can only change the current object pose by contacting it. Therefore, to most efficiently manipulate the object to the desired configuration, we add an auxiliary reward to encourage the robot to contact the object. We define the auxiliary reward as r aux (s, a) = exp(−L 2 (s, a)), where L 2 (s, a) measures the L2 distance between the end-effector and the object.
During training, we use sum of the ground-truth and auxiliary reward functions r gt +r aux , and during evaluation we only use the ground-truth reward r gt .

REINFORCEMENT LEARNING
Methods. We benchmark PPO (Schulman et al., 2017), SHAC , and APG (Freeman et al., 2021) as the representative RL methods. PPO is a widely used model-free nondifferentiable RL method with competitive performance, while SHAC and APG are the two latest RL methods that utilize the differentiable simulator. The computation graph of SHAC is illustrated in Figure 4a. By comparing the performance of the DaXBench tasks across these three methods, we aim to further study the benefits and limitations of applying the differentiable physics to RL.
Overall Performance. The learning curves of RL methods are shown in Figure 3. The performance difference between PPO and the differentiable RL methods shows opposite trends in the high-level macro-action-based and local-level control tasks. To our surprise, for most of the short-horizon  Table 3: Task performance for the planning and imitation learning methods. We report the mean and standard error for the policy/control sequences evaluated under 20 seeds.  macro-action-based tasks, PPO performs consistently better than the differentiable RL methods. This seems to contradict the experiment results of SHAC , which shows stronger performance than PPO in its own simulator on a wide range of context-rich MuJoCo tasks. However, for long-horizon tasks with low-level controls, the differentiable RL methods, especially APG, outperform PPO by a large margin.
Main Challenge for the differentiable-physics-based RL methods: Exploration. We argue that, on high-level macro-action-based tasks, the main limiting factor of the performance of differentiable-physics-based RL is the lack of exploration during training. The need to balance this trade-off is exaggerated by the enormous DoFs and the complex dynamics unique to the DOM tasks. Especially for the high-level macro-action-based tasks, their non-smooth/discontinuous/nonconvex optimization landscape necessitates using a good exploration strategy. However, the existing differentiable-physics-based RL methods rely exclusively on the simulation gradient to optimize the policy. They are not entropy regularized; hence, they may quickly collapse to the local optimal. Without explicit exploration, the differentiable-physics-based RL methods fail to reach the near-optimal state using the simulation gradients alone. Taking Fold-Cloth-3 as an example, if the randomly initialized policy never touches the cloth, the local simulation gradients cannot guide the agent to touch the cloth since the rewards and, consequently, the gradients for all not-in-contact actions are zero.
Differentiable-physics-based RL are sensitive to the Optimization Landscape. Differentiablephysics-based RL can largely improve the performance of some low-level control tasks if the optimization landscape is smooth and convex. Taking Whip-Rope as an example, the optimal low-level control sequence is a smooth motion trajectory that lies on a relatively smooth and well-behaved plane for the gradient-based methods. This explains why APG outperforms the non-differentiablephysics-based PPO by a large margin.

IMITATION LEARNING
Methods. We benchmark Transporter  and ILD (Chen et al., 2022) as the two representative Imitation Learning DOM methods. Transporter is a popular DOM method that is wellknown for its simple implementation and competitive performance. ILD is the latest differentiablephysics-based IL method; it utilizes the simulation gradients to reduce the required number of expert demonstrations and to improve the performance. Note that since Transporter abstracts its action space to pick-and-place actions to simplify the learning, it cannot be extended to the low-level control tasks. Our comparison for the IL methods will first focus on the high-level macro-actionbased tasks, then we will analyze the performance of ILD with the expert.
Overall performance For the high-level macro-action-based tasks, ILD outperforms Transporter by a large margin for all tasks. For example, ILD triples the performance Transporter in Fold-Cloth-1 and doubles it in Fold-Cloth-3 and Unfold-Cloth-1. We highlight that this significant performance gain is achieved by using a much smaller number of expert demonstrations. We attribute this improvement to the additional information provided by the simulation gradients, where now ILD can reason over the ground-truth dynamics to bring the trajectory closer to the expert, at least locally. As illustrated in Figure 4, the gradients used to optimize each action comes from the globally optimal expert demonstrations 1) at the current step and 2) in all the steps after. In contrast, the gradients used in SHAC only count the reward at the current step. The differentiable RL faces the challenge of the exploding/vanishing gradients and lack of exploration. Differentiable IL alleviates these problems by having step-wise, globally optimal guidance from expert demonstrations.
Challenge for ILD: Unbalanced State Representations. ILD does not perform well on tasks like Pour-Water and Pour-Soup with unbalanced state representations, for example, 1000 × 3 features of particles and 6 gripper states. The learned signals from the particles overwhelm the signals from the gripper states. Therefore, ILD cannot learn useful strategies with unbalanced learning signals. Softgym  also concurs with the results that policies with reduced state representation perform better than those with full state. A more representative representation of the states needs to be considered and we leave it for future study.
In conclusion, differentiable methods can optimize the policies efficiently if the optimization landscape is well-behaved. For the tasks that require tremendous explorations or with sparse rewards, gradient-based methods suffer severely from the local minima and gradient instability issues.

PLANNING
Methods. We benchmark the classic CEM-MPC (Richards, 2005) as a representative random shooting method that uses a gradient-free method to optimize the action sequences for the highest final reward. To study the effect of using differentiable physics, we implement two versions of differentiable MPC. The basic computation graph for both is illustrated in Figure 4c. In particular, both differentiable MPC baselines maximize the immediate reward of each action using differentiable physics. The difference lies in the initial action sequences: diff-MPC is initialized with an random action sequence, while diff-CEM-MPC with an action sequence optimized by CEM-MPC.
Overall Performance. The diff-CEM-MPC performs consistently better than or on par with CEM-MPC, while Diff-MPC fails to plan any reasonable control sequences for most of tasks.
Simulation Gradients as Additional Optimization Signals. The vastly different performance between the two versions of the differentiable baselines brings us insights on how the differentiable physics helps in planning. Simulation gradients provide the additional signals in the direction to optimize the actions, thereby reducing the sample complexity. However, similar to any gradient-based optimization methods, differentiable planning methods are sensitive to the non-convex/non-smooth optimization landscape and face a major limitation of being stuck at the local optimal solution. DOM's enormous DoFs of DOM and complex dynamics exacerbate these challenges for the differentiable DOM planning methods.
Differentiable Planning needs Good Initialization. We attribute the poor performance of Diff-MPC to its random control sequence initialization. In particular, this control sequence may significantly deviate from the optimal one over the non-smooth/non-convex landscape, such that the simulation gradients alone cannot bring the control sequence back to the optimal. In contrast, Diff-CEM-MPC is initialized with the optimized control sequence by CEM-MPC, which relies on iterative sampling followed by elite selection to escape the local optimality. Hence, the control sequence fed into the differentiable step is already sufficiently close to the globally optimal solution. The simulation gradients help diff-CEM-MPC to optimize the action sequences further locally, which explains the performance gain.
To summarize, the differentiable simulator can further optimize the near-optimal control sequences locally with much low sample complexity. Yet, its performance critically relies on a good initialization and a locally smooth optimization landscape.
(a) (b) Figure 5: We deploy CEM-MPC in a Push-Rope task to straighten the rope in 6 steps on a real robot. (a) The system structure. CEM-MPC uses our DaX simulator as the predictive model. Given current state estimated from a point cloud image, CEM-MPC plans the best action to be executed by the robot. (b) The states after each of the 6 pushes. We compare the dynamics in reality and in simulation. Top: state trajectory in a real robot experiment. Bottom: the next state predicted by DaX given the estimated current state and the corresponding planned action.

INSIGHTS AND FUTURE WORKS
The experiment results from the eight algorithms reveal a few shared insights. We believe these insights can inspire the researchers to improve and develop DOM methods from each community. Firstly, differentiable-physics-based methods fundamentally use the simulation gradients as additional signals to optimize the policy/control sequence. Therefore, these methods inherit the generic benefits of additional ground-truth gradients, such as faster convergence and more stable optimization under mild conditions. At the same time, the problems borne with the gradient-based methods are exacerbated in differentiable DOM methods. For example, the performance of the differentiable methods critically relies on good initialization and a well-behaved optimization landscape. Without these assumptions, additional effort is required to sufficiently explore the enormous state space and assist the simulation gradient-based optimization in escaping its local minima. We believe more research into the exploration strategy for differentiable DOM methods is necessary to further advance the progress in deformable object manipulation.

SIM-2-REAL GAP
The physics engine of DaX is built upon state-of-the-art simulation method. This ensures that the dynamics of our simulated tasks are of high fidelity and with a small sim2real gap. To testify to the correctness of our simulated dynamics to the real-world physics, we carry out a real robot experiment on a Kinova Gen 3 robot. We deploy CEM-MPC with DaX as the predictor model to a Push-Rope task, as shown in Figure 5. Our study finds that the resultant state trajectories are similar in simulation and in reality. This testifies that the tasks implemented based on our engine are of high fidelity, and the performance of the DOM methods in our simulator provides strong hints about their performance on the real robot. We refer the reader to the appendix for details.

CONCLUSION
We introduced DaXBench, a differentiable benchmark for various deformable object manipulation tasks. To our knowledge, DaXBench is the first benchmark that simulates liquid, rope, fabric, and elastoplastic materials while being differentiable and highly parallelizable. Our task coverage goes beyond the common DOM tasks; we innovate several novel tasks, such as WipeRope and PourSoup, to study the performance of the DOM methods on long-horizon tasks with low-level controls. Our rich task coverage provides a fairground to compare the existing DOM methods from all paradigms; further, our benchmark provides insights into how the differentiable-physics-based methods help in DOM and the caveats of using this new tool. We believe DaXBenchcan help assist with the algorithmic development of DOM methods, especially the differentiable-physics-based ones of all paradigms.

ETHICS STATEMENT
DaXBench provides a benchmark for deformable object manipulation (DOM). We believe that DaXBench and its simulator DaX have no potential negative societal impacts. In addition, our work is on benchmarking the DOM methods in simulator, so we do not collect any data or conduct experiments with human subjects.

REPRODUCIBILITY STATEMENT
In this paper, all experiments are averaged over 5 random seeds. We have included our source code as an easy-to-install package in the supplementary material.

A IMPLEMENTATION DETAILS
A.1 OBJECTS DaXBench models a variety of deformable objects of different dimensionalities, ranging from 0dimensional liquid, 1-dimensional rope, to 2-dimensional cloth. The underlying deformation and the dynamics vary distinctly for each dimension. In general, the lower the dimension is, the higher the degrees of freedom (DoFs) of the system, thus the more complex modeling that system becomes. We use two different representations to model the deformable objects to balance the modeling capacity with its complexity: water and rope are modeled using Material Point Method (MPM), while the cloth is modelled using a mass-spring system.
Modelling Water and Rope using MPM Material Point Method (MPM) demonstrates strong modeling capacity and flexibility. MPM models the deformable objects as a set of particles; each particle is represented by the position and some properties such as mass. The dynamics and the deformation of the object are computed by simulating the interactions among all individual particles, conforming to some pre-defined physics rules. This Particle-level representation and the dynamics simulation allow MPM to model the object under severe deformation. We therefore choose to use MPM to model water and rope that can undergo severe deformation, for they can exploit the model capacity of MPM to its full potential.
Modelling Cloth using Mass-Spring System Cloth only undergoes limited deformation as it has to conform to the canonical shape of the initial cloth through the manipulation. This limited deformable does not require the strong modeling capacity of MPM, which is computationally expensive. Therefore, rather than modeling the deformation on the Particle-level, we choose to model the cloth using the mass-spring system, which is a simpler model that uses the physical constraints as a prior.
A.2 CODE EXAMPLE (a) Detect the active region.
(b) Update the deformation.
(c) Shift back the update active region. Figure 6: We demonstrate how to use the active region to reduce the computation complexity for the simulation. Taking rope for example, the original environment, denoted as the black background, is of size 128 × 128 × 128, while the active region, where the region that will be active changed due to the current deformation highlight in red, is of size 32 × 6 × 32, as illustrated in (a). To avoid update the all particles in the environment, in ( b), we shift the active region to the top corner, the origin of the environment and update the dynamic and observation. Finally, in ( c), we shift the updated active region back to align its original coordinates. and the enormous state space. The complex governing equations invalidate most of the analytical approaches since solving the complex governing equations are intractable. On the other hand, the success of the sampling-based methods hinge on the reasonable coverage of the state space. The lack of support for highly paralleled sampling to better cover the state space is a major impediment for the works on Deformable Object Manipulation. DaXBench fills in this gap by implementing its simulator using JAX, which support highly paralleled sampling with great automatic differentiation. Using our implementation, we can finish a iteration (both forward and backward pass) for 128 rollouts with 80 timesteps on a server with 4 2080-Ti GPUs in 3 seconds.

A.4 SAVING MEMORY
Lazy Dynamic Update DaXBench optimizes the time and memory consumption of MPM by only updating the dynamic of a small region affected by the manipulation at each step. MPM models and updates the state and dynamics on the Particle-level; its time and memory consumption is proportional to the size of the environment, which can be arbitrarily large. Our insight is that, though the deformable object can potentially interact with the entire environment, only a small region will be affected at each time. This inspires DaXBench to optimize the time and memory consumption of running MPM by exclusively consider the active region during the interactions. Using this trick, DaXBench effectively reduces the computation time by two orders of magnitude on average. Taking the rope for example, the original environment is of size 128 × 128 × 128. However, a grid of size 32 × 6 × 32 is sufficient to cover the region of interaction between the environment and the rope. Therefore, we first detect the active region of size 32×6×32, then we perform linear transformation to shift the active region to the origin, perform the update due to deformation and interaction, finally we shift the updated active region to align with its original coordinates.
Lazy evaluation of Gradient DaXBench re-evaluate the sub-step gradients online during the backpropagation to reduce the memory consumption for long horizon problems. To make the simulator end-to-end differentiable, every value in the neural network that is computed during the forward pass has to be saved, as they are needed to compute the gradient during the backward pass. The computation complexity scales up quickly w.r.t. the length of the task horizon. This is a significant problem for our simulator for high DoFs deformable object manipulation, as the memory required for each timestep's update is enormous compared to its rigid body counterpart. To overcome the large memory consumption issue, DaXBench trade-offs the computation complexity slightly for large memory consumption improvement: DaXBench only stores the values of a few states at the forward pass; during the backward propagation, DaXBench re-compute segments of the forward values starting from every stored state in the reverse order sequentially for the overall gradient. With this trick, DaXBench now doubles the computation complexity, but saves arbitrary amount of memory, depending on the intervals of the states saved.  Table 4: Reinforcement Learning Methods Task Results. We report the mean and standard error over 20 random seeds for each entry.

A.5 DISCONTINUOUS GRADIENT
As an end-to-end differentiable simulator, the effectiveness of DaXBench rests on the accuracy of the gradient computed. We found that for deformable object manipultation, the action not in contact with the object will cause the gradient to be discontinuous, which causes the estimate of the gradient after this action to suffer from high variance and poor accuracy in the backward propagation. To mitigate this problem, for every action input by the agent or the human user, DaXBench will perform local adjustment such that the action will always make the gripper in contact with the deformable objects. This ensures that our gradient to be end-to-end continuous therefore improves the accuracy of the estimated gradient.

B SIM-2-REAL GAP
To verify that the dynamics of our simulated tasks has high-fidelity and small Sim2Real gap, we carry out a real robot experiment. We deploy CEM-MPC to the PushRope task on a Kinova Gen3 robot. CEM-MPC uses our DaX simulator as the predictive model. Given a state, CEM-MPC plans the next best push action (x, y, z, x , y , z ). The state is estimated from a point cloud image, and the Cartesian space push action is transformed to the joint space trajectory via an inverse kinematics module. Note we are not verifying the sim2real gap caused by the inaccuracy of state estimation or kinematics; we are interested in whether the dynamics of DaX can correctly guide the CEM-MPC to succeed in the task. The experiment is included in our supplementary video.

C RL ALGORITHMS NUMERICAL EXPERIMENT RESULTS
We report the numerical experiment performance for the RL algorithms in Table 4. This is to supplement the learning curve in the main text. This numerical results report the final performance of the learning curves in Figure 3.