Sarvam-1 Open-Source Language Model - Optimized Specifically for Indian Languages, Performs Exceptionally Well in 10 Indian Languages

Sarvam 1

Developed by sarvamai

Sarvam-1 is a 2 billion parameter language model optimized for Indian languages, excelling in 10 Indian languages.

Supports Multiple Languages#Indian Language Optimization #Efficient Multilingual Generation #Low-Resource Efficient Inference

Downloads 3,386

Release Time : 10/23/2024

Model Overview

Sarvam-1 is a 2 billion parameter language model specifically optimized for Indian languages, supporting 10 Indian languages and English, showcasing top-tier performance in Indian language tasks.

Model Features

Multilingual Optimization

Designed for 10 Indian languages from the ground up, supporting major Indian languages and English

Efficient Token Processing

Achieves a fertility rate of 1.4-2.1 for all supported languages, 2-4 times more efficient than existing multilingual models

High-Quality Training Data

Trained on a curated corpus of approximately 4 trillion tokens, including 2 trillion high-quality Indian language tokens

Fast Inference

Delivers equivalent or better performance in Indian language tasks while being 4-6 times faster in inference compared to larger models

Model Capabilities

Text Generation

Multilingual Processing

Text Completion

Use Cases

Language Processing

Cross-Language Translation

Supports translation tasks between Indian languages

Flores English-Indian Translation: 39.83 chrF++

Cross-Language Summarization

Generates text summaries in different Indian languages

20.48 chrF++

Question Answering System

Builds multilingual question-answering systems

25.27 F1

🚀 Sarvam-1

Sarvam-1 is a 2-billion parameter language model specifically optimized for Indian languages. It offers top-notch performance in 10 Indic languages (bn, gu, hi, kn, ml, mr, or, pa, ta, te) when compared to popular models like Gemma-2-2B and Llama-3.2-3B. It can also compete with much larger models such as Llama-3.1-8B in these languages. More details can be found in our release blog.

🚀 Quick Start

The model was trained with NVIDIA NeMo™ Framework on the Yotta Shakti Cloud using HGX H100 systems.

⚠️ Important Note

This is a text-completion model. It is meant to be finetuned on downstream tasks, and cannot be used directly as a chat or an instruction-following model.

✨ Features

Optimized for 10 Indian Languages: Built from the ground up to support major Indian languages alongside English
Superior Token Efficiency: Achieves fertility rates of 1.4-2.1 across all supported languages, 2-4x more efficient than existing multilingual models
High-Quality Training Data: Trained on a curated corpus of ~4 trillion tokens with 2 trillion high-quality Indic tokens
Efficient Inference: 4-6x faster inference compared to larger models while matching or exceeding their performance on Indic language tasks

🔧 Technical Details

Model Architecture

Property	Details
Hidden size	2048
Intermediate size	11,008
Number of attention heads	16
Number of hidden layers	28
Number of key-value heads	8
Maximum position embeddings	8,192
Activation function	SwiGLU
Positional embeddings	Rotary (RoPE) with theta=10,000
Training	Grouped-query attention and bfloat16 mixed-precision

Training Details

Property	Details
Training Infrastructure	Yotta's Shakti cluster
Hardware	1,024 GPUs
Training Duration	5 days
Framework	NVIDIA NeMo

📈 Performance

Translated Academic Benchmarks (Zero-shot)

Benchmark	Score
MMLU	44.44
ARC-Challenge	58.50
TriviaQA	90.62
BoolQ	80.68

IndicGenBench (One-shot)

Benchmark	Score
Flores English-to-Indic translation	39.83 chrF++
CrossSum	20.48 chrF++
XORQA	25.27 F1
XQUAD	41.58 F1

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("sarvamai/sarvam-1")
tokenizer = AutoTokenizer.from_pretrained("sarvamai/sarvam-1")

# Example usage
text = "कर्नाटक की राजधानी है:"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=5)
result = tokenizer.decode(outputs[0])

📄 License

Sarvam non-commercial license: See the LICENSE file

🙏 Acknowledgements

NVIDIA: for support with the NeMo codebase
Yotta: for access to the Shakti GPU cluster
AI4Bharat: for their academic partnership and expertise in Indian language technologies

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご