---

library_name: transformers pipeline_tag: image-text-to-text license: mit

Model Card for Magma-8B

Agents

UI Navigation

Robot Manipulation

Gaming

Task: Model controls the robot to collect green blocks.

Model Details

Model Description

Magma is a multimodal agentic AI model that can generate text based on the input text and image. The model is designed for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, in particular the multimodal agentic AI. The main innovation of this model lies on the introduction of two technical innovations: Set-of-Mark and Trace-of-Mark, and the leverage of a large amount of unlabeled video data to learn the spatial-temporal grounding and planning. Please refer to our paper for more technical details.

Highlights

• Digital and Physical Worlds: Magma is the first-ever foundation model for multimodal AI agents, designed to handle complex interactions across both virtual and real environments!
• Versatile Capabilities: Magma as a single model not only possesses generic image and videos understanding ability, but also generate goal-driven visual plans and actions, making it versatile for different agentic tasks!
• State-of-the-art Performance: Magma achieves state-of-the-art performance on various multimodal tasks, including UI navigation, robotics manipulation, as well as generic image and video understanding, in particular the spatial understanding and reasoning!
• Scalable Pretraining Strategy: Magma is designed to be learned scalably from unlabeled videos in the wild in addition to the existing agentic data, making it strong generalization ability and suitable for real-world applications!

License

The model is developed by Microsoft and is funded by Microsoft Research. The model is shared by Microsoft Research and is licensed under the MIT License.

How to Get Started with the Model

To get started with the model, you first need to make sure that transformers and torch are installed, as well as installing the following dependencies:

pip install torchvision Pillow open_clip_torch

⚠️ Please note that you need to install our customized transformers lib:

pip install git+https://github.com/jwyang/transformers.git@dev/jwyang-v4.48.2

See here for the reason why you need this.

Then you can run the following code:

import torch
from PIL import Image
from io import BytesIO
import requests

from transformers import AutoModelForCausalLM, AutoProcessor

# Load the model and processor
dtype = torch.bfloat16
model = AutoModelForCausalLM.from_pretrained("microsoft/Magma-8B", trust_remote_code=True, torch_dtype=dtype)
processor = AutoProcessor.from_pretrained("microsoft/Magma-8B", trust_remote_code=True)
model.to("cuda")

# Inference
url = "https://assets-c4akfrf5b4d3f4b7.z01.azurefd.net/assets/2024/04/BMDataViz_661fb89f3845e.png"
image = Image.open(BytesIO(requests.get(url, stream=True).content))
image = image.convert("RGB")

convs = [
    {"role": "system", "content": "You are agent that can see, talk and act."},
    {"role": "user", "content": "<image_start><image><image_end>\nWhat is in this image?"},
]
prompt = processor.tokenizer.apply_chat_template(convs, tokenize=False, add_generation_prompt=True)
inputs = processor(images=[image], texts=prompt, return_tensors="pt")
inputs['pixel_values'] = inputs['pixel_values'].unsqueeze(0)
inputs['image_sizes'] = inputs['image_sizes'].unsqueeze(0)
inputs = inputs.to("cuda").to(dtype)

generation_args = { 
    "max_new_tokens": 128, 
    "temperature": 0.0, 
    "do_sample": False, 
    "use_cache": True,
    "num_beams": 1,
}

with torch.inference_mode():
    generate_ids = model.generate(**inputs, **generation_args)

generate_ids = generate_ids[:, inputs["input_ids"].shape[-1] :]
response = processor.decode(generate_ids[0], skip_special_tokens=True).strip()
print(response)

Training Details

Training Data

Our training data consists of:

• Generic Image SFT Data: LLaVA-Next, InfoGrpahicVQA, ChartQA_Augmented, FigureQA, TQA, ScienceQA.

• Generic Video SFT Data: ShareGPT4Video and LLaVA-Video.

• Instructional Video Data: Ego4d, Somethingv2, Epic-Kitchen and other related instructional videos.

• Robotics Manipulation Data: Open-X-Embodiment.

• UI Grounding Data: SeeClick.

• UI Navigation Data: Mind2web and AITW.

The data collection process involved sourcing information from publicly available documents, with a meticulous approach to filtering out undesirable documents and images. To safeguard privacy, we carefully filtered various image and text data sources to remove or scrub any potentially personal data from the training data.

More details can be found in our paper.

Microsoft Privacy Notice

Training Procedure

Preprocessing

In addition to the text-related preprocessing, we mainly undertake the following image and video preprocessing steps:

• UI Grounding and Navigation Data: For each UI screenshot, we extract the bounding boxes for the UI elements, and apply Set-of-Mark Prompting to overlay numeric marks on the raw image. The model is trained to generate the UI grounding text based on the image and the Set-of-Mark prompts.

• Instruction Video Data: For each video clip, we apply Co-Tracker to extract the grid traces and then apply filtering algorithm to remove the noisy or static points. For videos that bear camera motion, we further apply homography transformation to stabilize the video clips. In the end, we assign a numeric mark for each trace which gives us a set of trace-of-mark. The model is trained to generate the trace-of-mark given the video clips and instructional text.

• Robotics Manipulation Data: For robotics data in Open-X Embodiment, we extract the 7 DoF robot gripper state and also extract the trace-of-mark from the video clips. Similar filtering and stabilization steps are applied to the video clips. The model is trained to generate the robot manipulation action as well as the trace-of-mark given the video clips and instructional text.

After all these preprocessing, we combine them with existing text annotations to form our final multimodal training data. We refer to our paper for more technical details.

Training Hyperparameters

We used bf16 mixed precision for training on H100s and MI300s. We used the following hyperparameters for training:

• Batch size: 1024
• Learning rate: 1e-5
• Max sequence length: 4096
• Resolution: maximally 1024x1024 for image, 512x512 for video frame.
• Pretraining Epochs: 3

Evaluation

We evaluate the model in zero-shot manner on a wide range of tasks, mostly agent-related tasks.

Testing Data, Factors & Metrics

Zero-shot Testing Data

We evaluate the model's zero-shot performance on the following datasets:

• UI Grounding: ScreenSpot and VisualWebArena.

• Robotics Manipulation: SimplerEnv and WidowX real robot.

• Spatial Understanding and Reasoning: VSR, BLINK and SpatialEval.

Finetuned Testing Data

We evaluate the model's performance after finetuning on the following datasets:

• UI Navigation: Mind2Web and AITW.

• Robotics Manipulation: SimplerEnv and WidowX real robot.

• Multimodal Image Understanding and Reasoning: VQAv2, GQA, MME, POPE, TextVQA, ChartQA, DocVQA.

• Multimodal Video Understanding and Reasoning: Next-QA, VideoMME, MVBench.

Metrics

We follow the individual dataset's evaluation metrics for the evaluation. Please refer to the original dataset for more details.

Results on Agentic Intelligence

Zero-shot evaluation on agentic intelligence. We report the results for pretrained Magma without any domain-specific finetuning. Magma is the only model that can conduct the full task spectrum.

Model	VQAv2	TextVQA	POPE	SS-Mobile	SS-Desktop	SS-Web	VWB-Ele-G	VWB-Act-G	SE-Google Robot	SE-Bridge
GPT-4V	77.2	78.0	n/a	23.6	16.0	9.0	67.5	75.7	-	-
GPT-4V-OmniParser	n/a	n/a	n/a	71.1	45.6	58.5	-	-	-	-
LLava-1.5	78.5	58.2	85.9	-	-	-	12.1	13.6	-	-
LLava-Next	81.3	64.9	86.5	-	-	-	15.0	8.7	-	-
Qwen-VL	78.8	63.8	n/a	6.2	6.3	3.0	14.0	0.7	-	-
Qwen-VL-Chat	78.2	61.5	n/a	-	-	-	-	-	-	-
Fuyu	74.2	n/a	n/a	21.2	20.8	19.2	19.4	15.5	-	-
SeeClick	-	-	-	65.0	51.1	44.1	9.9	1.9	-	-
Octo	-	-	-	-	-	-	-	-	-	-
RT-1-X	-	-	-	-	-	-	-	-	6.0	15.9
OpenVLA	-	-	-	-	-	-	-	-	34.2	1.1
Magma-8B	80.0	66.5	87.4	59.5	64.1	60.6	96.3	71.8	52.3	35.4

Notes: SS - ScreenSpot, VWB - VisualWebArena, SE - SimplerEnv

Technical Specifications

Model Architecture and Objective

• Language Model: We use Meta LLama-3 as the backbone LLM.
• Vision Encoder: We use CLIP-ConvneXt-XXLarge trained by LAION team as the vision encoder to tokenize the images and videos.

The whole pipeline follows the common practice in the multimodal LLMs, where the vision encoder is used to tokenize the images and videos, and then the visual tokens are fed into the LLM along with the textual tokens to generate the text outputs.

Compute Infrastructure

We used Azure ML for our model training.

Hardware

Our model is trained on two GPUs:

• Nvidia H100
• AMD MI300

Software

Our model is built based on:

• Pytorch
• Transformers
• TorchVision
• DeepSpeed
• FlashAttention

Intended Uses

This model is intended for broad research use in English. It is designed only for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, particularly in multimodal agentic AI. It is intended to be used by domain experts who are independently capable of evaluating the quality of outputs before acting on them.

Direct Use

The model takes images and text as inputs, and produces the textual outputs for the following uses:

• Image/Video-Conditioned Text Generation: The model can generate text (e.g., descriptions, answers) based on the input text and image.

• Visual Planning Capabilities: The model can also produce the visual trace as the future planning to accomplish a task (e.g., move object from one place to another).

• Agentic Capabilities: The model can also generate UI grounding (e.g., click ``search'' button) and robotics manipulations (e.g., 7 DoF for the robot gripper).

Downstream Use

The model can be further finetuned for different downstream tasks, such as:

• Image Captioning and QA: We can further finetune this model for image captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better spatial understanding and reasoning on these tasks.

• Video Captioning and QA: We can further finetune this model for video captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better temporal understanding and reasoning on these tasks.

• UI Navigation: We can finetune this model for specific UI navigation tasks, such as web navigation or mobile navigation. The model can achieve superior performance on these tasks.

• Robotics Manipulation: Our model can be further finetuned for robotics tasks given its general agentic capabilities as a vision-language-action model. After finetuning, our model significantly outperforms the state-of-the-art models such as OpenVLA on robotics manipulation tasks.

Bias, Risks, and Limitations

Please note that this model is not specifically designed or evaluated for all downstream purposes.

The model is not intended to be deployed in production settings. It should not be used in high-risk scenarios, such as military and defense, financial services, and critical infrastructure systems.

Developers should consider common limitations of multimodal models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case.

Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. Like other multimodal models, Magma can potentially behave in ways that are unfair, unreliable, or offensive.

The models' outputs do not reflect the opinions of Microsoft.

Some of the limiting behaviors to be aware of include:

• Quality of Service: The model is trained primarily on English text. Languages other than English will experience worse performance. English language varieties with less representation in the training data might experience worse performance than standard American English. Magma is not intended to support multilingual use.

• Representation of Harms & Perpetuation of Stereotypes: These models can over- or under-represent groups of people, erase representation of some groups, or reinforce demeaning or negative stereotypes. Despite safety post-training, these limitations may still be present due to differing levels of representation of different groups or prevalence of examples of negative stereotypes in training data that reflect real-world patterns and societal biases.

• Inappropriate or Offensive Content: These models may produce other types of inappropriate or offensive content, which may make it inappropriate to deploy for sensitive contexts without additional mitigations that are specific to the use case.

• Information Reliability: Multimodal models can generate nonsensical content or fabricate content that might sound reasonable but is inaccurate or outdated.

Developers should apply responsible AI best practices and are responsible for ensuring that a specific use case complies with relevant laws and regulations (e.g. privacy, trade, etc.). Using safety services like Azure AI Content Safety that have advanced guardrails is highly recommended.

Recommendations

Magma was developed for research purposes only. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

The recommended usage for the finetuned models is within the research settings they were trained on — namely,

• an android simulator running on a computer for UI manipulation.
• an enclosure equipped with a robotic arm and everyday objects for Robotic manipulation

For UI navigation task, researchers should make sure a human is in the loop and in control for every action the agentic system generates. Since the model cannot act by itself, the sub-module a researcher uses to actually perform the UI navigation action should ensure no unintended consequences can occur as a result of performing the UI action proposed by the model.

For the robotic manipulation task, some mitigation strategies to use for human safety when operating robotic arms include:

• Safety Zones and Barriers: Establish physical barriers or safety zones around robotic workspaces to prevent unauthorized access.
• Emergency Stop Systems: Equip robotic arms with easily accessible emergency stop buttons. Implement a fail-safe mechanism that triggers an immediate stop of operations in case of an emergency
• Safety Standards and Compliance: Adhere to established safety standards (e.g., ISO 10218, ISO/TS 15066) for industrial robots and collaborative robots.
• User Training and Awareness: Provide comprehensive training for all personnel working around robotic arms to understand their functions, safety features, and emergency procedures. Promote awareness of the potential risks associated with robotic manipulation.

Citation

@misc{yang2025magmafoundationmodelmultimodal,
      title={Magma: A Foundation Model for Multimodal AI Agents}, 
      author={Jianwei Yang and Reuben Tan and Qianhui Wu and Ruijie Zheng and Baolin Peng and Yongyuan Liang and Yu Gu and Mu Cai and Seonghyeon Ye and Joel Jang and Yuquan Deng and Lars Liden and Jianfeng Gao},
      year={2025},
      eprint={2502.13130},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.13130}, 
}