From the Hugging Face Hub to Robot Hardware with Strands Agents and LeRobot

From the Hugging Face Hub to Robot Hardware with Strands Agents and LeRobot

The intersection of large language models (LLMs) and robotics is rapidly expanding, enabling developers to move from simulation to real-world hardware with unprecedented speed. Two open-source projects—Strands Agents and LeRobot—are at the forefront of this shift, allowing you to download robot control policies from the Hugging Face Hub and deploy them on physical robots. This article provides a practical, step-by-step guide to bridging the gap between cloud-based AI models and tangible robotic hardware.

Why This Matters

Traditionally, robotics development required custom firmware, proprietary software, and extensive manual tuning. Today, thanks to standardized model repositories and agent frameworks, you can:

• **Download pre-trained policies** (e.g., for grasping, navigation, or manipulation) from the Hugging Face Hub.
• **Run them with Strands Agents**, a modular framework for orchestrating AI agents.
• **Execute them on real robot hardware** using LeRobot, a library for low-level robot control and simulation.

This pipeline reduces development time from months to days and democratizes access to advanced robotics AI.

Requirements

Before you begin, ensure your system meets the following requirements:

• **Operating System**: Ubuntu 22.04 or later (macOS and Windows with WSL2 are partially supported).
• **Python**: Version 3.10 or 3.11.
• **Hardware**: A robot with a supported interface (e.g., Franka Emika, UR5, or a custom arm with ROS 2). For testing without hardware, use LeRobot’s built-in simulation.
• **GPU**: Recommended for running large LLMs (e.g., NVIDIA GPU with CUDA 12.1+).
• **Internet**: To download models from Hugging Face Hub.

Software Dependencies

• **ROS 2 Humble or Iron** (for hardware communication).
• **Git** and **Git LFS** (to download large model files).
• **Conda** or **venv** (for Python environment management).

Step-by-Step Installation

We’ll set up a complete environment from scratch. Open a terminal and execute each command block sequentially.

1. Create a Python Virtual Environment

Isolate dependencies to avoid conflicts.

python3 -m venv strands_robot_env
source strands_robot_env/bin/activate

2. Install LeRobot

LeRobot provides the low-level interface for robot control. Install the latest version from PyPI.

pip install lerobot

If you plan to use simulation (recommended for testing), install with the simulation extras:

pip install "lerobot[sim]"

3. Install Strands Agents

Strands Agents is the high-level orchestration layer. Clone the repository and install in editable mode.

git clone https://github.com/huggingface/strands-agents.git
cd strands-agents
pip install -e .

4. Install ROS 2 (for Hardware Communication)

If you intend to run on a physical robot, install ROS 2 Humble. Follow the official ROS 2 installation guide for your OS. For Ubuntu 22.04:

sudo apt update
sudo apt install ros-humble-desktop
source /opt/ros/humble/setup.bash

Add the source command to your `~/.bashrc` for persistence:

echo "source /opt/ros/humble/setup.bash" >> ~/.bashrc

5. Authenticate with Hugging Face Hub

To download models, log in to the Hugging Face Hub. You’ll need a user access token (create one at [hf.co/settings/tokens](https://huggingface.co/settings/tokens)).

huggingface-cli login

Paste your token when prompted.

6. Verify Installation

Test that all components are accessible.

python -c "import lerobot; print('LeRobot version:', lerobot.__version__)"
python -c "import strands_agents; print('Strands Agents imported successfully')"

Downloading a Pre-trained Policy from the Hugging Face Hub

The Hugging Face Hub hosts hundreds of robot policies. For this example, we’ll use a policy for pick-and-place tasks, trained on the `lerobot/pusht` dataset.

1. List Available Policies

Use the LeRobot CLI to browse models.

lerobot list --datasets

This command shows datasets and associated policies. Look for entries like `lerobot/pusht` or `lerobot/aloha`.

2. Download a Specific Policy

Download the policy weights and configuration.

lerobot download --dataset lerobot/pusht --policy diffuser

This downloads the `diffuser` policy for the `pusht` dataset into `~/.cache/lerobot/`.

3. Inspect the Model

Check the policy’s structure and input/output specifications.

from lerobot.common.policies.diffusion.model import DiffusionPolicy
policy = DiffusionPolicy.from_pretrained("lerobot/pusht")
print(policy.config)

You’ll see configuration parameters such as observation dimensions, action dimensions, and model architecture.

Running the Policy with Strands Agents

Strands Agents wraps the policy into an agent that can interact with the environment or hardware.

1. Create a Simple Agent Script

Write a Python file `run_pusht_agent.py`:

from strands_agents import Agent
from lerobot.common.policies.diffusion.model import DiffusionPolicy
import numpy as np

# Load the policy
policy = DiffusionPolicy.from_pretrained("lerobot/pusht")

# Define a custom agent that uses the policy
class PushtAgent(Agent):
    def __init__(self, policy):
        super().__init__()
        self.policy = policy

    def step(self, observation):
        # Convert observation to tensor and run policy
        action = self.policy.select_action(observation)
        return action

# Instantiate the agent
agent = PushtAgent(policy)

# Test with a dummy observation (e.g., image and joint angles)
dummy_obs = {
    "image": np.random.rand(3, 96, 96).astype(np.float32),
    "joint_positions": np.random.rand(7).astype(np.float32)
}
action = agent.step(dummy_obs)
print("Predicted action:", action)

Run the script:

python run_pusht_agent.py

This demonstrates that the agent can process observations and output actions.

2. Connect to a Simulated Environment

LeRobot includes a Gym-style environment for the `pusht` task. Use it to test the agent in simulation.

from lerobot.envs import create_env
from strands_agents import Agent
from lerobot.common.policies.diffusion.model import DiffusionPolicy

# Create environment
env = create_env("lerobot/pusht", simulation=True)

# Load policy
policy = DiffusionPolicy.from_pretrained("lerobot/pusht")

# Wrap policy as agent
class PolicyAgent(Agent):
    def step(self, obs):
        return policy.select_action(obs)

agent = PolicyAgent(policy)

# Run one episode
obs, info = env.reset()
for _ in range(200):
    action = agent.step(obs)
    obs, reward, done, truncated, info = env.step(action)
    if done:
        break
env.close()
print("Episode finished.")

Deploying on Real Robot Hardware

Moving from simulation to a physical robot requires hardware-specific configuration. We’ll use a Franka Emika Panda arm as an example.

1. Configure ROS 2 Interface

Ensure your robot’s ROS 2 drivers are running. For Franka, start the `franka_ros2` driver.

ros2 launch franka_bringup franka.launch.py robot_ip:=192.168.1.100

2. Create a Hardware-Aware Agent

Modify the agent to communicate with the robot via ROS 2 topics. Use LeRobot’s `Robot` class.

from lerobot.robot import Robot
from strands_agents import Agent
from lerobot.common.policies.diffusion.model import DiffusionPolicy

# Initialize robot interface
robot = Robot("franka", control_mode="position")

# Load policy
policy = DiffusionPolicy.from_pretrained("lerobot/pusht")

class HardwareAgent(Agent):
    def step(self, obs):
        return policy.select_action(obs)

agent = HardwareAgent(policy)

# Main control loop
while True:
    # Get current observation from robot sensors
    obs = robot.get_observation()
    # Compute action
    action = agent.step(obs)
    # Send action to robot
    robot.send_action(action)

3. Safety Precautions

Before running on hardware:

• **Set velocity limits** in the robot configuration.
• **Enable emergency stop**.
• **Start with a low gain** mode.
• **Use a virtual joint limit** to prevent collisions.

LeRobot provides a `safety` module; add it to your script:

from lerobot.utils.safety import SafetyController
safety = SafetyController(velocity_limit=0.5)
action = safety.limit(action)

Usage Examples

Example 1: Pick-and-Place with Simulation

Download a pick-and-place policy and run it in a simulated environment.

lerobot download --dataset lerobot/aloha --policy act
python -c "
from lerobot.envs import create_env
from lerobot.common.policies.act.model import ACTPolicy
env = create_env('lerobot/aloha', simulation=True)
policy = ACTPolicy.from_pretrained('lerobot/aloha')
obs = env.reset()
for _ in range(500):
    action = policy.select_action(obs)
    obs, reward, done, _, _ = env.step(action)
    if done: break
print('Pick-and-place completed.')
"

Example 2: Custom Agent with Language Instructions

Strands Agents supports LLM-based reasoning. Combine a policy with a language model for high-level task planning.

from strands_agents import Agent
from lerobot.common.policies.diffusion.model import DiffusionPolicy
from transformers import pipeline

class LanguageGuidedAgent(Agent):
    def __init__(self, policy, llm):
        self.policy = policy
        self.llm = llm

    def step(self, obs, instruction):
        # Use LLM to refine action based on instruction
        prompt = f"Given observation {obs}, instruction: {instruction}"
        response = self.llm(prompt)
        # Map LLM output to policy parameters
        return self.policy.select_action(obs)

policy = DiffusionPolicy.from_pretrained("lerobot/pusht")
llm = pipeline("text-generation", model="gpt2")
agent = LanguageGuidedAgent(policy, llm)
action = agent.step(dummy_obs, "Move the block to the left")

Troubleshooting Common Issues

• **Model download fails**: Ensure Git LFS is installed (`git lfs install`).
• **ROS 2 connection errors**: Check that `ROS_DOMAIN_ID` matches across all nodes.
• **CUDA out of memory**: Reduce batch size or use CPU inference (`device="cpu"` in policy loading).
• **Simulation vs. hardware mismatch**: Normalize observation data consistently.

Conclusion

The combination of the Hugging Face Hub, Strands Agents, and LeRobot creates a powerful, open-source pipeline for deploying AI models on real robots. You can now download state-of-the-art policies, orchestrate them with flexible agent logic, and run them on hardware—all with a few lines of Python.

This approach lowers the barrier to entry for robotics research and development, enabling faster experimentation and iteration. As the ecosystem grows, expect even tighter integration between LLMs, simulation, and physical robots, making it easier than ever to go from a model on the Hub to a robot in your lab.

**Next steps**: Explore other datasets on the Hugging Face Hub (e.g., `lerobot/trifinger`, `lerobot/libero`), experiment with custom agent behaviors, and contribute your own trained policies back to the community.