Robotics labs use Runway's video models to generate synthetic data for training policies

Overview Debates Related claims Topics

Veo 3.1, Runway Gen-4.5, and Kling 3.0 generate audio synchronized with generated video footage

Kling 4.0 and Gemini Omni 1.1 generate AI video clips with enhanced controllability and duration

AI video generation models produce individual clips lasting seconds rather than full minutes

Dyna Robotics uses frontier AI models to generate synthetic environments for training its robots

Synthetic and simulation-based data are considered a next frontier for scaling robotics training

Dyna Robotics uses its own world action model, which can generate video, to create its own training data in a self-improving loop

In the DreamRSI framework, simulators are used to efficiently expand training data and update the original exploration policy

The "DreamRSI" paper proposes a method for RSI where an "explorer" policy generates data to create simulators

TypeSafe AI generated all of its own synthetic training data for the Jev model

Synthetic Personally Identifiable Information (PII) is used in training to teach models how to identify entities like names and addresses

Generative models are used to create synthetic training data, such as a voice pronouncing a name with simulated background noise like a baby crying

The Alpaca model was created by a Stanford team for $600 by fine-tuning Llama with synthetic data

Runway's Gen-1, released in January 2023, was a video-to-video model

World models allow for robotics simulation by providing just a single starting image of an environment

Runway released a 'turbo' version of Gen-3, which was a step-distilled model for faster inference

Runway announced its Gen-2 model two months after announcing its Gen-1 model

Runway developed its Gen-3 model a few months after OpenAI released its Sora model in February 2024

The development of camera controls for Gen-2 inspired Runway's research into world models

Step distillation allows large models to generate video in fewer steps, enabling real-time performance

Gen-1 functioned as a depth-conditioned model, converting video to a depth map before generating new pixels