Hebrew Gemma 11B

🚀 Hebrew-Gemma-11B

Hebrew-Gemma-11B is an open - source Large Language Model (LLM). It's a Hebrew/English pretrained generative text model with 11 billion parameters, based on the Gemma - 7B architecture from Google. It extends Gemma - 7B, scales up, and is trained on an additional 3B tokens of both English and Hebrew text data. The resulting Gemma - 11B is a powerful general - purpose language model for various natural language processing tasks, with a focus on Hebrew language understanding and generation.

✨ Features

Base Models:

• 07.03.2024: Hebrew-Gemma-11B
• 16.03.2024: Hebrew-Gemma-11B-V2

Instruct Models:

• 07.03.2024: Hebrew-Gemma-11B-Instruct

📦 Installation

First make sure to pip install -U transformers.

💻 Usage Examples

Basic Usage

Running on CPU

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("yam-peleg/Hebrew-Gemma-11B")
model = AutoModelForCausalLM.from_pretrained("yam-peleg/Hebrew-Gemma-11B")

input_text = "שלום! מה שלומך היום?"
input_ids = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))

Running on GPU

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("yam-peleg/Hebrew-Gemma-11B")
model = AutoModelForCausalLM.from_pretrained("yam-peleg/Hebrew-Gemma-11B", device_map="auto")

input_text = "שלום! מה שלומך היום?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))

Advanced Usage

Running with 4 - Bit precision

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

tokenizer = AutoTokenizer.from_pretrained("yam-peleg/Hebrew-Gemma-11B")
model = AutoModelForCausalLM.from_pretrained("yam-peleg/Hebrew-Gemma-11B", quantization_config = BitsAndBytesConfig(load_in_4bit=True))

input_text = "שלום! מה שלומך היום?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))

📚 Documentation

Terms of Use

As an extension of Gemma - 7B, this model is subject to the original license and terms of use by Google.

Gemma-7B original Terms of Use: Terms

Notice

Hebrew-Gemma-11B is a pretrained base model and therefore does not have any moderation mechanisms.

📄 License

👥 Authors

• Trained by Yam Peleg.
• In collaboration with Jonathan Rouach and Arjeo, inc.