Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos
NVIDIA Cosmos Data Factory accelerates physical AI development by enabling dataset curation and synthetic generation through two prompts. This open platform offers five blueprints and six skills to generate driving scenarios, augment images, and create rare events, reportedly up to 60x faster than traditional methods.
Tags
Quick summary
NVIDIA Cosmos Data Factory accelerates physical AI development by enabling dataset curation and synthetic generation through two prompts. This open platform offers five blueprints and six skills to generate driving scenarios, augment images, and create rare events, reportedly up to 60x faster than traditional methods.
Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos
World model accuracy fundamentally relies on high-quality, diverse training datasets. However, capturing rare defects, nighttime incidents, and other challenging scenarios remains costly and time-consuming. Traditional workflows often involve disconnected tools and manual processes for curation, labeling, generation, and evaluation, adding significant complexity to turning raw data into training-ready datasets.
NVIDIA is addressing these challenges with NVIDIA Cosmos, an open platform designed for world model development. This platform integrates frontier models, datasets, frameworks, and skills to streamline the data pipeline. A key component of this ecosystem is the NVIDIA Cosmos Data Factory (CDF), which provides five cloud-native blueprints and six composable skills to accelerate data curation, processing, and synthetic data generation (SDG).
According to NVIDIA, developers can now curate, augment, generate, and label physical AI data, orchestrated by their coding agent, using just two prompts. This approach aims to reduce the friction between data needs and model training, enabling "prompt-to-data" workflows without the need for constant human steering.
The Two-Prompt Workflow
The core value proposition of the NVIDIA Cosmos Data Factory lies in its ability to orchestrate complex workflows through a simplified interface. The process is divided into two primary steps, each driven by a specific prompt.
The first step involves setting up and configuring Cosmos blueprints and skills. Users enter a prompt into their coding agent to initiate the environment. This action triggers preflight checks that verify the environment without making changes, flagging any missing components. Once these checks pass, the setup phase prepares the environment to run the specific blueprints required for the user's training data use case.
The second step involves running the blueprint for the specific training data use case. By entering a second prompt, the user directs the system to execute the chosen workflow. This orchestration is managed by the CDF Orchestration core skill, which coordinates workflow steps, deploys required services, and manages execution on the cluster. This separation of setup and execution allows developers to focus on their data goals while the underlying infrastructure handles the complexity of deployment and scaling.
Five Reference Blueprints for Physical AI
NVIDIA Cosmos Data Factory includes five reference blueprints designed to address common challenges in physical AI training. These blueprints leverage core agent skills to handle tasks ranging from autonomous vehicle simulation to visual inspection.
Use Case 1: Generate Driving Scenarios for AV Training
The Autonomous Vehicle Dataset Generation blueprint is designed for teams needing to curate existing footage or generate novel datasets for downstream model training and validation. This blueprint helps search large databases of driving scenarios and generate varied datasets.
The workflow involves NVIDIA Cosmos Curator extracting metadata for video dataset inputs, while NVIDIA Cosmos Dataset Search finds relevant videos through embeddings. Retrieved videos are passed to NVIDIA Cosmos3-Super-Image2Video for style transfer, transforming scenes across weather, lighting, and environmental conditions. Finally, Cosmos Evaluator checks the generated videos for physical plausibility. This ensures that the augmented data maintains the structural integrity required for safe autonomous vehicle training.
Use Case 2: Augment Image Attributes for Search and Re-Identification
For tasks requiring person search and re-identification, the Image Attribute Augmentation reference blueprint generates variations of person-crops with controlled attribute changes. These changes include clothing type, accessories, and color.
An LLM within the system turns selected attributes into structured modification prompts, which are used to generate augmented variations of the source image. Cosmos Evaluator confirms that the requested clothing attributes appear in the output. The blueprint then auto-labels the results with clothing attributes and search queries, creating a dataset ready for training search algorithms.
Use Case 3: Augment Videos Across Weather, Lighting, and Environments
The Video Data Augmentation reference blueprint transforms existing footage across various weather, lighting, and environmental conditions while preserving the original scene structure. A vision language model (VLM) describes the source video and combines that description with requested conditions into a structured prompt. Cosmos3-Nano then generates the augmented variant.
To ensure quality, Cosmos Evaluator checks each generated clip for motion artifacts not present in the source, confirms that the requested conditions actually appear, and scores whether the motion is physically plausible. The blueprint subsequently auto-labels both the source and generated videos with detection, tracking, and captions, providing a comprehensive training-ready dataset.
Use Case 4: Generate Rare Event Videos for Model Training
Capturing rare events like fires, falls, or shoplifting can be dangerous or expensive. The Event Video Generation reference blueprint addresses this by using a source image to create videos of rare events.
A VLM combines its description of the scene with the requested event into a structured prompt. Cosmos3-Super-Image2Video generates the clip while preserving the source viewpoint and layout. Cosmos Evaluator checks whether the event appears and whether the motion looks physically plausible, retrying failed clips automatically. Passing clips are auto-labeled with detections, tracks, captions, and semantic annotations for downstream training.
Use Case 5: Generate Defect Images for Visual Inspection
The Defect Image Generation reference blueprint creates defective examples from clean inspection images or 3D USD assets of parts such as circuit boards, glass panels, or metal surfaces. For 3D assets, NVIDIA Isaac Sim renders inspection images and masks, while Qwen-Image-Edit creates realistic clean reference images, known as golden images.
Cosmos 3 Super uses these images, defect masks, and requested defect types to generate defects while preserving the surrounding part. The system ships fine-tuned for printed circuit boards (PCBs), glass, and metal, with further fine-tuning skills available for other use cases. The output includes golden images, synthetic defect images, and manifests tracing each result to its source and generation settings.
Core Agent Skills and Tools
The five reference blueprints are built from six core agent skills that can be used independently or combined to build custom workflows around specific data and goals.
- Auto Labeling: Turns raw video and images into annotated datasets with detection, tracking, captions, and scene understanding.
- AnomalyGen: Creates realistic defect images from a handful of examples, complete with masks, specifically for visual inspection tasks.
- Augmentation: Generates visual variations of video and images using Cosmos models for conditions that are difficult to capture in the real world.
- Curation and Retrieval: Finds clips and images in large collections using Cosmos Curator and Cosmos Dataset Search. Cosmos Curator now supports multicamera video, IMU, and GPS inputs, plus initial lidar support in its sensor library.
- Simulation: Renders photorealistic, fully labeled images from 3D scenes using Isaac Sim and Omniverse Replicator.
- Orchestration: Runs reference blueprints on a cluster, deploying the services each one needs and scaling them as required.
For workflows focused on curation and retrieval, developers can use Cosmos Curator to process and auto-label raw video and multimodal data, and Cosmos Dataset Search to find targeted training scenarios across large datasets. Reusable Python processing components are also available to support these tasks.
Performance and Projected Outcomes
NVIDIA has measured the efficiency gains provided by this architecture. In tests conducted on a cloud server equipped with eight NVIDIA RTX PRO 6000 Blackwell Series GPUs, the organization reported significant improvements over traditional methods.
Specifically, NVIDIA measured daily throughput and reported up to 60x faster dataset assembly. Furthermore, the generation of autonomous vehicle scenarios was reported to be 45x cheaper compared to traditional approaches. These metrics highlight the potential for organizations to scale their data operations without proportionally increasing costs or time-to-market.
Conclusion
Building training-ready physical AI datasets is no longer solely dependent on manual capture and labor-intensive labeling. By leveraging NVIDIA Cosmos and the Cosmos Data Factory, developers can orchestrate complex data pipelines using just two prompts. Whether generating driving scenarios for autonomous vehicles, augmenting images for re-identification, or creating synthetic defect images for visual inspection, the platform offers a flexible and scalable solution.
With core skills like Auto Labeling, AnomalyGen, and Simulation working in concert, organizations can accelerate their world model development. As NVIDIA demonstrates, the combination of frontier models and composable skills allows for the creation of high-quality, diverse datasets that are essential for training robust physical AI systems.
Source:
- Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos — NVIDIA AI Blog — https://developer.nvidia.com/blog/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos



