AgentHands: Generating Interactive Hand Gestures for Spatially Grounded Agent Conversations in XR

AgentHands: Generating Interactive Hand Gestures for Spatially Grounded Agent Conversations in XR

Ask anyone who has spent time inside a VR meeting room or an AR assistant overlay, and they will name the same disconnect: the voice is there, the words are clear, but the body is dead. Synthetic agents speak fluently while their hands hover in a neutral pose, never pointing at the object they are describing, never reaching for the button they are explaining, never shaping a gesture that matches the rhythm of their speech. In a real conversation, that behavior would read as robotic, evasive, or simply broken. In extended reality (XR), it is worse, because the space around the agent is not abstract. There are real, spatially located objects, tools, and collaborators to reference. The agent's hands are not decoration; they are a pointing instrument, a deictic channel, a way of making a conversation legible in three dimensions.

AgentHands, a research project from Google Research, directly targets this gap. The paper-titled approach, described in the official Google Research blog post published in August 2026, focuses on generating interactive hand gestures for spatially grounded agent conversations in XR. The emphasis on both "interactive" and "spatially grounded" matters. Gestures are not pre-baked animations waiting to be replayed; they are generated in response to the agent's speech and the surrounding scene, which means the same sentence can produce different hand movements depending on whether the agent is pointing at a coffee mug, a doorway, or nothing at all.

This article is a practical, technical overview of what it takes to build such a system into your own XR agent pipeline. It is based solely on the public research blog post as the factual foundation. Where the post describes research findings, I will interpret them for an engineering audience; where it does not provide an official SDK or release commands, I will be explicit about that. The setup steps below therefore represent a reference integration layer that you would build around the AgentHands approach, not an official installer for the research code itself.

The Core Problem: Speech, Space, and the Missing Link

Natural human conversation is multimodal in a way that text-based interfaces never capture. When we say "put that over there," we rely on a pointing gesture to disambiguate which object and which location. When we describe a process, our hands trace the shape of the process. When we emphasize a word, our hand beats downward in sync with the stressed syllable. Language models, by contrast, output tokens. They have no native understanding of a 3D scene, and they have no motor cortex.

The challenge in XR is to connect these two worlds: the discrete, symbolic world of speech and the continuous, geometric world of physical space. A conversational agent in XR must solve three problems simultaneously:

1. What to say — the language generation task, typically handled by a large language model.
2. Where things are — the spatial grounding task, typically handled by a scene graph, object detection, or spatial anchors.
3. How to move — the gesture generation task, which is what AgentHands addresses.

The third problem is the most neglected. Many XR agents either reuse generic idle animations or rely on hand-tracked human puppeteering, which does not scale. Others bolt on gesture recognition but have no generation component at all, so the agent can see gestures but cannot produce them. AgentHands inverts that pipeline: it takes as input the agent's utterance and the spatial context, and it outputs a gesture sequence that is both semantically appropriate and physically plausible.

What makes this particularly hard is that gestures are not merely tied to the meaning of words. They are also tied to the timing of speech, the identity of the agent, and the physics of the hands. A pointing gesture must land on a target with a certain trajectory; a grasping gesture must anticipate object size; a beat gesture must align with prosodic stress. Getting all of these right at interactive rates, inside an XR headset with a limited inference budget, requires a carefully designed generative model that the agent can run in real time.

What Makes AgentHands Different: Interactivity and Grounding

The word interactive in the title is not incidental. Most prior work on co-speech gesture generation treats the task as an offline construction problem: given a transcript, produce a complete gesture animation, then play it back. That approach fails in XR because the scene is not static. A user can move, an object can be relocated, and a conversation can be interrupted mid-sentence. The agent needs to adjust its gestures on the fly.

The spatially grounded part is equally important. In a typical co-speech gesture dataset, an animated character gestures into empty space. The AgentHands approach instead conditions gesture generation on the 3D scene itself. When the agent references an object, the generated hand trajectory should be influenced by that object's actual position, orientation, and shape. When the agent references a direction, the gesture should reflect the true geometry of the room rather than a generic sweep of the arm.

This has a practical consequence for system design: gesture generation cannot be a standalone text-to-motion module. It must sit alongside a spatial understanding layer. In an integration scenario, you would feed the agent a list of scene objects with their positions and bounding boxes, and the AgentHands model would use that information, together with the dialogue context, to decide when a gesture targets a specific object and how the hand should move to reference it.

Requirements

Before we get to any code, let us establish what you need to follow along.

• Python 3.10 or later for the gesture generation and integration logic.
• A 3D engine such as Unity 2022 LTS or a web-based stack with Three.js or A-Frame, since the generated gestures ultimately need to animate an avatar in XR.
• A VR or AR headset (optional but strongly recommended) to evaluate whether the generated gestures actually feel grounded when viewed from a user's perspective.
• A CUDA-capable GPU if you plan to run the generation model locally; otherwise, you can prototype with a CPU and a lightweight model stub.
• A spatial scene provider — this can be as simple as a JSON list of object names and transforms, or as complex as a full scene graph from your XR framework.

There is one important caveat to state up front. The Google Research blog post describes the AgentHands method and its evaluation, but it does not publish a pip-installable package or a public model checkpoint. The commands below therefore set up a reference integration that you can use to build the surrounding infrastructure, test your scene-graph plumbing, and validate your evaluation pipeline. They use real, installable packages, but they do not claim to download any proprietary AgentHands artifact.

Step-by-Step Installation

Let us create a clean workspace and a virtual environment. This keeps dependencies isolated from your global Python installation.

mkdir agenthands-integration
cd agenthands-integration
python -m venv .venv
source .venv/bin/activate

The source command applies to Linux and macOS; on Windows, use .venv\Scripts\activate. Next, upgrade pip and install the core packages we will use during development.

pip install --upgrade pip
pip install numpy

numpy gives us the linear algebra needed to compute hand trajectories as vectors, which will be useful in the usage examples below. If you want to later plug in a real gesture model, you will likely also need a deep learning framework.

pip install torch --index-url https://download.pytorch.org/whl/cu121

The --index-url flag points pip to the PyTorch wheels built for CUDA 12.1, which is a common configuration for GPU-accelerated inference in 2026. If your hardware or driver version differs, consult the official PyTorch installation guide and adjust the index URL accordingly. The package itself is a real, widely used dependency; the AgentHands research code, if released, would likely sit on top of it.

Now create a minimal project structure to keep the integration clean.

mkdir -p agenthands/{scene,gesture,avatar}
touch agenthands/__init__.py config.yaml

The config.yaml file will hold scene and model configuration. A minimal YAML configuration might look like this:

scene:
  coordinate_system: "left_handed"   # match your XR engine
  scale: 1.0
gesture:
  fps: 30
  smoothing: 0.2
avatar:
  id: "narrator"
  hand_model: "full_hand"

The fps field determines how many gesture frames per second your generation loop targets. It is a design parameter you choose, not a number claimed by the research post.

Usage Examples

The central idea we want to prototype is a simple interface: the agent produces a text utterance, the scene provider supplies a list of spatialized objects, and our gesture module decides what the hands should do. Below is an illustrative Python module that demonstrates the shape of that interface.

import numpy as np

class SpatialReference:
    """A named object with a 3D position and a coarse bounding box."""
    def __init__(self, name, position, size):
        self.name = name
        self.position = np.asarray(position, dtype=float)
        self.size = np.asarray(size, dtype=float)

class GestureChannel:
    """Maps an utterance plus a spatial scene to a hand gesture."""
    def __init__(self, agent_position):
        self.agent_position = np.asarray(agent_position, dtype=float)

    def generate(self, utterance, target=None):
        if target is not None:
            return self._point_at(target, emphasis=0.8)
        return self._beat_gesture(utterance)

    def _point_at(self, target, emphasis):
        direction = target.position - self.agent_position
        direction /= np.linalg.norm(direction) + 1e-8
        return {
            "type": "POINT",
            "target_name": target.name,
            "direction": direction.tolist(),
            "emphasis": emphasis,
        }

    def _beat_gesture(self, utterance):
        words = len(utterance.split())
        return {
            "type": "BEAT",
            "beat_count": max(1, int(words / 2)),
            "intensity": min(1.0, words / 20.0),
        }

In this scaffold, a POINT gesture is produced whenever the agent references a specific object, and a BEAT gesture is produced when the agent is speaking without a spatial target. The research post's core contribution is to replace the hand-crafted _point_at and _beat_gesture methods with a learned generative model that produces continuous hand and finger motion, conditioned on the same inputs: utterance text and scene geometry. The interface above is intentionally simple, so that swapping in a real learned model is a matter of implementing generate() as a model call rather than a procedural rule.

To exercise the module, we need a tiny demo loop that simulates a conversational turn.

from agenthands.gesture import GestureChannel, SpatialReference

agent = GestureChannel(agent_position=(0.0, 1.6, 0.0))
mug = SpatialReference(
    name="blue_mug",
    position=(1.2, 1.0, -1.5),
    size=(0.1, 0.12, 0.1),
)

my_utterance = "Please hand me the blue mug on the table."
gesture = agent.generate(my_utterance, target=mug)
print(gesture)

Running this will print a dictionary describing the intended gesture. In a real XR loop, that dictionary would be consumed by the avatar animation layer, which would retarget the 3D hand pose onto your avatar's skeleton and play it in sync with the speech audio. The key engineering insight is that the gesture generation layer should be agnostic to the avatar: it outputs a spatial intent, and the animation layer is responsible for realizing that intent in the engine.

For a more realistic integration, you would also want to include a timing signal. Human gestures are synchronized with speech, so your gesture module should accept a timestamp or speech_segment input, as shown below.

def generate_timed(self, utterance, audio_events, target=None):
    """
    audio_events: list of (start, end, stressed) tuples.
    """
    gestures = []
    for start, end, stressed in audio_events:
        gesture = self.generate(utterance, target=target)
        gesture["start"] = start
        gesture["end"] = end
        gestures.append(gesture)
    return gestures

This makes the temporal alignment explicit and mirrors the research goal of generating gestures that are not just semantically correct but prosodically coherent.

Practical Considerations and Open Limits

Working with the AgentHands approach in a real product means paying attention to several constraints that the research post, like most academic publications, necessarily simplifies.

Latency budget. Interactive XR demands gesture generation at interactive rates. If the model runs on a powerful server node, network round-trip time becomes a factor. If it runs on-device, you have a limited power and thermal budget. A practical pipeline therefore decouples "fast" gesture types (beats and simple points) from "slow" gesture types (object manipulation sequences that require more context), and falls back gracefully when the model cannot keep up.

Scene representation. The quality of spatially grounded gestures depends on how well you encode the scene. A flat list of bounding boxes is a good start, but the model will perform better if the scene representation includes relational information: which objects are on the table, which ones are within arm's reach, and which ones are across the room. You should design your scene provider to emit these relationships as structured data rather than expecting the gesture model to infer them from raw geometry.

Retargeting. A gesture generated for a generic hand skeleton will not transfer perfectly to every avatar. Hand proportions, finger length, and shoulder mobility all affect how the movement looks. You need a retargeting layer that maps the generated joint angles onto the target skeleton, and you should visually validate that the result does not produce unnatural twisting or self-intersection.

Evaluation. Measuring gesture quality remains an open research problem. Objective metrics such as foot-skate, hand-object contact distance, and temporal alignment with speech can catch gross errors, but subjective user studies are still necessary to determine whether a gesture feels natural, whether it helps comprehension, and whether it is perceived as appropriately grounded. The Google Research blog post reports evaluation results along these lines, but of course those results are tied to its specific experimental setup; you should replicate your own evaluation for your own avatar, scene, and dialogue domain.

Generalization. A model trained on one conversational domain or one cultural gesture style will not necessarily transfer to another. Gestures are culturally specific, and a pointing style that reads as natural in one context may not in another. If your product targets different languages or cultures, plan to collect or curate data that reflects that diversity, and treat the published approach as a method, not as a finished product.

There is also a semantic ambiguity problem that no generation model fully solves: the same physical gesture can have different meanings, and the same meaning can be expressed by different gestures. The model reduces this ambiguity by conditioning on the utterance, but it cannot eliminate it. As an engineer, you should build your dialogue system to be resilient to misread gestures, for instance by confirming spatial references explicitly when the user's context makes them ambiguous.

Conclusion

AgentHands points toward a future in which XR agents do not merely speak convincingly, but move convincingly as well. By generating interactive hand gestures that are grounded in the actual 3D scene, the approach addresses a deficiency that users experience immediately but struggle to articulate: the agent's hands finally participate in the conversation. The research post's contribution is to treat gesture generation not as a playback problem but as a generative, scene-conditioned, interactive problem, and that framing is exactly right for the constraints of immersive spaces.

For developers, the practical path is now clearer. You need a spatial scene provider that tells the agent where things are, a gesture generation model that consumes both the utterance and that spatial context, and an animation layer that retargets the generated motion onto your avatar. The integration is tractable today, even without an official SDK. The reference pipeline shown here — a clean Python interface, a scene representation, and a clear separation between gesture intent and avatar animation — will let you prototype the experience now and drop in a learned model later.

The next time you put on a headset and get into a conversation with an agent, pay attention to its hands. If it points at the actual object it is describing, at the right moment, in a way that matches the rhythm of its speech, you are looking at the payoff of spatially grounded gesture generation. AgentHands is an important step toward making that behavior the default rather than the exception.