Spatialvla 4b 224 Sft Bridge

🚀 SpatialVLA Fine-Tuned on fractal & bridge

This model is generated by fine-tuning the SpatialVLA model on the bridge dataset for the Simpler-env benchmark, offering high - performance vision - language - action capabilities.

🚀 Quick Start

SpatialVLA relies solely on HuggingFace Transformers 🤗, making deployment extremely easy. If your environment supports transformers >= 4.47.0, you can directly use the following code to load the model and perform inference. (requires 8.5GB of GPU memory).

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_name_or_path="IPEC-COMMUNITY/spatialvla-4b-224-pt"
processor = AutoProcessor.from_pretrained(model_name_or_path, trust_remote_code=True)

model = AutoModel.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda()

image = Image.open("example.png").convert("RGB")
prompt = "What action should the robot take to pick the cup?"
inputs = processor(images=[image], text=prompt, return_tensors="pt")
generation_outputs = model.predict_action(inputs)

actions = processor.decode_actions(generation_outputs, unnorm_key="bridge_orig/1.0.0")
print(actions)

✨ Features

• Multidisciplinary Team Development: Developed by the SpatialVLA team composed of researchers from Shanghai AI Laboratory, ShanghaiTech and TeleAI.
• Advanced Model Type: Vision - language - action model, converting language and image inputs into robot actions.
• Rich Data Support: Pretrained on large - scale datasets such as [Open X - Embodiment](https://robotics - transformer - x.github.io/) and RH20T.
• Easy Deployment: Relying on HuggingFace Transformers, it can be quickly deployed in environments that meet the requirements.

📦 Installation

If you want to use the model for fine - tuning or pre - training, you need to follow these steps:

Clone the Repository

git clone https://github.com/SpatialVLA/SpatialVLA.git

Create a Python Environment

conda create -n spatialvla python=3.10
conda activate spatialvla

Install Required Packages

pip install -r requirements.txt

Note that we use a customised dlimp to support seed setting for reproducibility. If you catch any problems, please manually install the dlimp form the dlimp_custom.

💻 Usage Examples

Basic Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_name_or_path="IPEC-COMMUNITY/spatialvla-4b-224-pt"
processor = AutoProcessor.from_pretrained(model_name_or_path, trust_remote_code=True)

model = AutoModel.from_pretrained(model_name_or_path, trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda()

image = Image.open("example.png").convert("RGB")
prompt = "What action should the robot take to pick the cup?"
inputs = processor(images=[image], text=prompt, return_tensors="pt")
generation_outputs = model.predict_action(inputs)

actions = processor.decode_actions(generation_outputs, unnorm_key="bridge_orig/1.0.0")
print(actions)

Advanced Usage

Train from Scratch

SpatialVLA is pre - trained with 1.1 Million real - robot demonstrations from the OXE and RH20T dataset on a cluster of 64 A100 GPUs for about 10 days, using a batch size of 2048. You can pre - train the model from scratch using the following command.

# torchrun
bash scripts/spatialvla_4b_pretrain/torchrun_pretrain.sh

# or in a slurm cluster
bash scripts/spatialvla_4b_pretrain/slurm_pretrain.sh

Fine - tuning

Most of our fine - tuning experiments are conducted using LoRA on 4 or 8 A100 GPUs. You can use the following scripts for full - parameter or LoRA fine - tuning. For real - world experiments with small datasets, we prefer using LoRA for fine - tuning.

# full fine - tuning
bash scripts/spatialvla_4b_finetune/finetune_full.sh

# LoRA fine - tuning
bash scripts/spatialvla_4b_finetune/finetune_lora.sh

📚 Documentation

Model Details

Evaluation

SimplerEnv evaluation on Google Robot tasks

SimplerEnv evaluation on WidowX Robot tasks

Zero - shot Robot Control Evaluation on WidowX Robot


Spatial Understanding Capability Evaluation


📄 License

This model is licensed under the MIT license.

🔧 Technical Details

The model is fine - tuned from [paligemma2 - 3b - pt - 224](https://huggingface.co/google/paligemma2 - 3b - pt - 224) on the bridge dataset for the Simpler - env benchmark. It is a vision - language - action model that can convert language and image inputs into robot actions. The model is pre - trained on large - scale datasets such as [Open X - Embodiment](https://robotics - transformer - x.github.io/) and RH20T, and can be easily deployed using HuggingFace Transformers.

📄 Citation

BibTeX:

@misc{qu2025spatialvlaexploringspatialrepresentations,
      title={SpatialVLA: Exploring Spatial Representations for Visual - Language - Action Model}, 
      author={Delin Qu and Haoming Song and Qizhi Chen and Yuanqi Yao and Xinyi Ye and Yan Ding and Zhigang Wang and JiaYuan Gu and Bin Zhao and Dong Wang and Xuelong Li},
      year={2025},
      eprint={2501.15830},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2501.15830}, 
}