T5gemma S S Ul2

🚀 T5Gemma model card

T5Gemma is a lightweight yet powerful encoder - decoder research model from Google, suitable for various generative tasks and capable of being deployed in resource - limited environments.

🚀 Quick Start

This repository corresponds to T5Gemma (pretrained) with S encoder and S decoder (adapted using UL2). To quickly start using the model, follow these steps:

First, install the Transformers library:

pip install -U transformers

Basic Usage

import torch
from transformers import pipeline

pipe = pipeline(
    "text2text-generation",
    model="google/t5gemma-s-s-ul2",
    device="cuda",  # replace with "mps" to run on a Mac device
)

text = "Once upon a time,"
outputs = pipe(text, max_new_tokens=32)
response = outputs[0]["generated_text"]
print(response)

Advanced Usage

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

tokenizer = AutoTokenizer.from_pretrained("google/t5gemma-s-s-ul2")
model = AutoModelForSeq2SeqLM.from_pretrained(
    "google/t5gemma-s-s-ul2",
    device_map="auto",
)

input_text = "Write me a poem about Machine Learning. Answer:"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=32)
print(tokenizer.decode(outputs[0]))

✨ Features

• Flexible Architecture: Allows pairing encoders and decoders of different sizes, such as a 9B encoder with a 2B decoder.
• Multiple Series: Released in Gemma 2 Series and T5 - compatible Series, covering a wide range of model sizes and architectures.
• Suitable for Various Tasks: Well - suited for generative tasks like question answering, summarization, and reasoning, and its encoders can be used for discriminative tasks.
• Resource - Friendly: Relatively small size enables deployment in resource - limited environments like laptops, desktops, or personal cloud infrastructure.

📦 Installation

To install the necessary library for using T5Gemma, run the following command:

pip install -U transformers

💻 Usage Examples

Inputs and outputs

• Input: A text string, such as a question, a prompt, or a document to be summarized.
• Output: Generated English - language text in response to the input, such as an answer to a question or a summary of a document.

Citation

@article{t5gemma_2025,
  title={Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation},
  author={Zhang, Biao and Moiseev, Fedor and Ainslie, Joshua and Suganthan, Paul and Ma, Min and Bhupatiraju, Surya and Lebron, Fede and Firat, Orhan and Joulin, Armand and Dong, Zhe},
  year={2025}
}

📚 Documentation

Model Information

T5Gemma is a family of lightweight yet powerful encoder - decoder research models from Google. It is created by adapting pretrained decoder - only models into an encoder - decoder, inheriting the capabilities of decoder - only models while offering a better quality - efficiency trade - off.

It has two series:

• Gemma 2 Series: Models directly adapted from the official Gemma 2 2B and 9B checkpoints, including 2B - 2B, 9B - 9B, and 9B - 2B variants.
• T5 - compatible Series: Models pretrained from scratch using the Gemma 2 recipe, with architectures and parameter counts aligning with traditional T5 models (Small, Base, Large, XL), and an ML (Medium - Large, ~2B) model.

Model Data

Training Dataset

The models were trained on a diverse text dataset:

• Web Documents: Provide a wide range of linguistic styles, topics, and vocabulary, mainly in English.
• Code: Helps the model learn programming language syntax and patterns.
• Mathematics: Enables the model to learn logical reasoning and address mathematical queries.

Data Preprocessing

• CSAM Filtering: Rigorous filtering to exclude child sexual abuse material.
• Sensitive Data Filtering: Automated techniques to filter out personal and sensitive data.
• Additional Methods: Filtering based on content quality and safety according to Google's policies.

Implementation Information

Hardware

T5Gemma was trained using Tensor Processing Unit (TPU) hardware (TPUv4p, TPUv5p, and TPUv5e), which offers advantages in performance, memory, scalability, and cost - effectiveness.

Software

Training was done using JAX and ML Pathways, as described in the paper about the Gemini family of models.

Evaluation

Benchmark Results

The models were evaluated against a large collection of datasets and metrics for text generation. The following table shows the results:

Ethics and Safety

Evaluation Approach

The evaluation methods include structured evaluations and internal red - teaming testing of relevant content policies. Categories for evaluation include child safety, content safety, and representational harms.

Evaluation Results

Significant improvements were seen in child safety, content safety, and representational harms compared to previous Gemma models. All testing was conducted without safety filters, and the models produced minimal policy violations. However, the evaluations only included English - language prompts.

Usage and Limitations

Intended Usage

The models can be used for content creation, communication, research, and education, such as text generation and summarization, and NLP research.

Limitations

• Training Data: Quality and diversity of training data can affect the model's capabilities.
• Context and Task Complexity: Struggle with open - ended or highly complex tasks.
• Language Ambiguity and Nuance: Difficulty in understanding subtle nuances in natural language.
• Factual Accuracy: May generate incorrect or outdated factual statements.
• Common Sense: Lack the ability to apply common sense reasoning in some situations.

Ethical Considerations and Risks

• Bias and Fairness: Trained models may reflect socio - cultural biases, but pre - processing and evaluations were conducted.
• Misinformation and Misuse: Can be misused to generate false or harmful text, and guidelines for responsible use are provided.
• Transparency and Accountability: This model card provides details about the model, and the open model promotes innovation and accountability.

Benefits

At release, this family of models offers high - performance open encoder - decoder large language model implementations for Responsible AI development, outperforming similarly sized models in benchmark evaluations.

🔧 Technical Details

• Model Type: Encoder - decoder research model
• Training Data: A diverse dataset including web documents, code, and mathematics, with specific pre - processing steps.
• Hardware: Trained on Tensor Processing Units (TPUv4p, TPUv5p, and TPUv5e).
• Software: Trained using JAX and ML Pathways.

📄 License

The license for this model is Gemma. To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license. To do this, please ensure you’re logged in to Hugging Face and click below. Requests are processed immediately. Acknowledge license

⚠️ Important Note

This repository corresponds to T5Gemma (pretrained) with S encoder and S decoder (adapted using UL2).

💡 Usage Tip

When using the model, make sure to follow the responsible use guidelines provided in the Responsible Generative AI Toolkit.