#
Code Example: Batching a (Streaming) Dataset

This section shows how to access datasets hosted on Hugging Face using the LeRobotDataset class. We’ll start small and build up, so you can copy–paste the pattern that fits your use case.

First, any dataset on the Hub that follows the three pillars (tabular, visual, relational metadata) can be accessed with a single instruction.

Why windows? Most reinforcement learning (RL) and behavioral cloning (BC) algorithms operate on stacks of observations and actions. For brevity, we refer to joint states and camera frames collectively as a single “frame.” For example, RL often uses a history of observations $o{t-H_o:t}$ to mitigate partial observability, and BC typically regresses chunks of multiple actions ($a{t:t+H_a}$) rather than single controls.

To support these training patterns, LeRobotDataset provides native temporal windowing. You define the time offsets (before/after) around any frame using `delta_timestamps`. Unavailable frames are padded and a mask is returned to filter padded elements. This all happens inside LeRobotDataset and is transparent to higher‑level wrappers like `torch.utils.data.DataLoader`.

> Temporal Windows Explained:

> - Observation history: `[-0.2, -0.1, 0.0]` gives you 200ms, 100ms, and current observations
> - Action sequences: `[0.0, 0.1, 0.2]` provides current and next 2 actions (100ms apart)
> - Automatic padding: Missing frames at episode boundaries are handled automatically
> - Mask included: Know which frames are real vs. padded for proper training

> This is crucial for robot learning where decisions depend on recent history!

Conveniently, by using LeRobotDataset with a PyTorch `DataLoader` one can automatically collate the individual sample dictionaries from the dataset into a single dictionary of batched tensors for downstream training or inference. LeRobotDataset also natively supports streaming mode for datasets. Users can stream data of a large dataset hosted on the Hugging Face Hub, with a one-line change in their implementation. Streaming datasets supports high-performance batch processing (ca. 80-100 it/s, varying on connectivity) and high levels of frames randomization, key features for practical BC algorithms which otherwise may be slow or operating on highly non-i.i.d. data. This feature is designed to improve on accessibility so that large datasets can be processed by users without requiring large amounts of memory and storage.

Here are different ways to set up temporal windows depending on your use case. Skim the options and pick one to start—switching later is just a change to the dictionary.

basic-bc
history-bc
action-chunking

Basic Behavioral Cloning (learn current action from current observation):

    # Simple: current observation → current action
    delta_timestamps = {
        "observation.images.wrist_camera": [0.0],  # Just current frame
        "action": [0.0]  # Just current action
    }

    dataset = LeRobotDataset(
        "lerobot/svla_so101_pickplace",
        delta_timestamps=delta_timestamps
    )

###
Streaming Large Datasets

> When to use streaming:

> - Dataset > available storage - Stream datasets that don’t fit on your disk
> - Experimentation - Quickly try different datasets without downloading
> - Cloud training - Reduce startup time by streaming from Hugging Face Hub
> - Network available - Requires stable internet connection during training

> Performance: Streaming achieves 80-100 it/s with good connectivity!

download
streaming

Download Dataset (faster training, requires storage):

    from lerobot.datasets.lerobot_dataset import LeRobotDataset

    # Downloads dataset to local cache
    dataset = LeRobotDataset("lerobot/svla_so101_pickplace")

    # Fastest access after download
    sample = dataset[100]

##
Training Integration

###
PyTorch DataLoader

    import torch
    from torch.utils.data import DataLoader

    # Create DataLoader for training
    dataloader = DataLoader(
        dataset,
        batch_size=16,
        shuffle=True,
        num_workers=4
    )

    # Training loop
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    for batch in dataloader:
        # Move to device
        observations = batch["observation.state"].to(device)
        actions = batch["action"].to(device)
        images = batch["observation.images.wrist_camera"].to(device)

        # Your model training here
        # loss = model(observations, images, actions)
        # loss.backward()
        # optimizer.step()

##
Why This Matters

This simple API hides significant complexity:

- ✅ Multi-modal synchronization - Images and sensors perfectly aligned
- ✅ Efficient storage - Compressed videos, memory-mapped arrays
- ✅ Temporal handling - Easy access to observation/action sequences
- ✅ Scalability - Same code works for small and massive datasets

Compare this to traditional robotics data handling, which often requires:

- Custom parsers for each data format
- Manual synchronization across modalities
- Complex buffering for temporal windows
- Platform-specific loading code

LeRobotDataset standardizes and simplifies all of this!

##
Section Quiz

Test your understanding of LeRobot and its role in robot learning:

###
1. What makes LeRobot different from traditional robotics libraries?

It only works with simulation environments. It provides end-to-end integration across the entire robotics stack with state-of-the-art learning algorithms. It requires expensive industrial robots to function. It only supports classical control methods.

###
2. Which of the following is NOT a key component of LeRobot’s approach?

Unified low-level robot configuration handling Native robotics dataset format (LeRobotDataset) Requiring expert knowledge for each new robot platform State-of-the-art learning algorithms with PyTorch implementations

###
3. What is the main advantage of LeRobot’s optimized inference stack?

It makes training faster on GPUs. It reduces the memory requirements for storing datasets. It decouples action planning from action execution for better real-time performance. It automatically generates training data from robot interactions.

###
4. Which types of robotic platforms does LeRobot support?

Only manipulation robots like robotic arms. Manipulation, locomotion, and whole-body control platforms. Only robots that cost more than $10,000. Only robots manufactured by specific companies.

###
5. What does “end-to-end integration with the robotics stack” mean in the context of LeRobot?

It only handles high-level planning, not low-level control. It covers everything from low-level hardware control to high-level learning algorithms. It requires separate tools for data handling and model training. It only works with specific operating systems.

###
6. What is the primary purpose of the delta_timestamps parameter in LeRobotDataset?

It sets the frame rate for video recording. It defines temporal windows to access observation histories and action sequences. It synchronizes data across different robots. It compresses video data for storage efficiency.

###
7. Which of the following best describes the three main components of LeRobotDataset?

Images, Actions, and Rewards Tabular Data, Visual Data, and Metadata Training, Validation, and Test sets Simulation, Real Robot, and Hybrid data

###
8. What happens when you use StreamingLeRobotDataset instead of LeRobotDataset ?

The data is automatically augmented for better training. The dataset is downloaded faster to your local machine. Data is streamed from the Hugging Face Hub without downloading, saving storage space. The dataset is automatically split into train/validation sets.

###
9. In the context of robot learning, what does “temporal windowing” refer to?

The time it takes to train a robot learning model. Accessing multiple time steps of observations and actions around a given frame. The frequency at which robot sensors collect data. The duration of each robot episode or trajectory.

###
10. What is the main advantage of LeRobotDataset’s approach to storing video data?

Videos are stored in the highest possible quality. Each frame is stored as a separate file for easy access. Multiple episodes are concatenated into larger MP4 files to reduce file system stress. Videos are automatically compressed using AI algorithms.

###
11. Which statement about LeRobotDataset’s compatibility is correct?

It only works with specific robot brands like SO-100. It requires custom code for each new robot platform. It integrates seamlessly with PyTorch DataLoader and Hugging Face ecosystems. It only supports simulation data, not real robot data.
Update on GitHub