Octo-small Open-source Robot Control Model - Freely Assist in Predicting Future Actions of Multi-source Robot Datasets

Octo Small

Developed by rail-berkeley

Octo Small is a robot control model trained based on diffusion policy, capable of predicting 7-dimensional actions for the next 4 steps, suitable for multi-source robot datasets.

Multimodal Fusion

Transformers

Open Source License:MIT #Multimodal Robot Control #Diffusion Policy Prediction #Lightweight Transformer

Downloads 335

Release Time : 12/13/2023

Model Overview

This model adopts a Transformer architecture, preprocesses image data through a lightweight convolutional encoder, and combines a T5 language encoder to process text instructions for robot action prediction and control.

Model Features

Multi-source Data Training

Integrates training data from 25 different robot datasets, covering a wide range of robot operation scenarios

Lightweight Architecture

Adopts a compact Transformer architecture with 27 million parameters, suitable for real-time robot control

Multimodal Input

Supports input from main camera and wrist camera images, combined with language instructions for action prediction

Diffusion Policy

Trained using diffusion policy, capable of predicting 7-dimensional actions for the next 4 steps

Model Capabilities

Robot Action Prediction

Multimodal Data Processing

Real-time Control

Language Instruction Understanding

Use Cases

Industrial Robots

Assembly Line Operations

Completes product assembly tasks based on visual input and language instructions

Service Robots

Object Grasping

Identifies and grasps specified items based on user instructions

🚀 Octo Small

Octo Small is a robotics model that predicts 7 - dimensional actions 4 steps into the future using a diffusion policy. It offers a practical solution for robotic tasks by leveraging a Transformer architecture.

🚀 Quick Start

See https://github.com/octo-models/octo for instructions for using this model.

✨ Features

Transformer Architecture: Octo Small is a Transformer with 27M parameters, equivalent to a ViT - S.
Tokenization: Images are pre - processed with a lightweight convolutional encoder and grouped into 16x16 patches. Language is tokenized using the T5 tokenizer and encoded by the T5 - Base language encoder.
Flexible Inference: At inference, you can pass in any subset of observation and task keys, with a history window up to 2 timesteps.

🔧 Technical Details

Octo Small is trained with a window size of 2. Observations and tasks conform to the following spec:

Observations

{
    image_primary: ('batch', 'history_window', 256, 256, 3),
    image_wrist: ('batch', 'history_window', 128, 128, 3),
}

Tasks

{
    image_primary: ('batch', 256, 256, 3),
    image_wrist: ('batch', 128, 128, 3),
    language_instruction: {
        attention_mask: ('batch', 16),
        input_ids: ('batch', 16),
    },
}

📚 Documentation

This model was trained on a mix of datasets from the Open X - Embodiment dataset.

Property	Details
Model Type	Transformer with 27M parameters (equivalent to a ViT - S)
Training Data	A mix of datasets from the Open X - Embodiment dataset

Dataset Proportions

Dataset	Proportion of batch
Fractal (Brohan et al, 2022)	17.0%
Kuka (Kalashnikov et al, 2018)	17.0%
Bridge (Walke et al, 2023)	17.0%
BC - Z (Jang et al, 2022)	9.1%
Stanford Hydra Dataset (Belkhale et al, 2023)	6.0%
Language Table~ (Lynch et al, 2023)	5.9%
Taco Play (Rosete - Beas et al, 2022, Mees et al., 2023)	3.6%
Furniture Bench Dataset (Heo et al, 2023)	3.3%
UTAustin Mutex (Shah et al, 2023)	3.0%
Austin Sailor Dataset (Nasiriany et al, 2022)	2.9%
Roboturk (Mandlekar et al, 2018)	2.8%
Toto (Zhou et al, 2023)	2.4%
Austin Sirius Dataset (Liu et al, 2023)	2.3%
Berkeley Autolab UR5 (Chen et al)	1.5%
IAMLab CMU Pickup Insert (Saxena et al, 2023)	1.2%
Viola (Zhu et al, 2023)	1.2%
Berkeley Fanuc Manipulation (Zhu et al, 2023)	1.0%
NYU Franka Play Dataset (Cui et al, 2022)	0.9%
UCSD Kitchen Dataset (Ge Yan and Wang, 2023)	<0.1%
Jaco Play (Dass et al, 2023)	0.6%
Berkeley Cable Routing (Luo et al, 2023)	0.3%
Austin Buds Dataset (Zhu et al, 2022)	0.3%
CMU Stretch (Mendonca et al, 2023)	0.2%
NYU Door Opening (Pari et al, 2021)	0.1%
DLR EDAN Shared Control (Quere et al, 2020)	0.1%

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご