Granite 7b Base

🚀 Granite-7b-base

Granite-7b-base is a pre - trained LLM from IBM's Granite model series, offering an open - source option for community and commercial use.

✨ Features

• Model Architecture: It replicates Meta’s Llama2 - 7B base variant with MHA and was trained with a 1M batch size on 2T tokens.
• Context Length: Supports up to 4k tokens.
• Tokenizer: Utilizes the Llama2 tokenizer.
• Language Support: Primarily supports English.

📦 Installation

No installation steps are provided in the original document.

📚 Documentation

Model Information

Pre - Training Data

The model was trained on 2T tokens, with sampling proportions designed to match the sampling distributions released in the Llama1 paper as closely as possible.

Evaluation Results

LM - eval Harness Scores

Bias, Risks, and Limitations

⚠️ Important Note

Granite-7b-base is a base model and has not undergone any safety alignment, so it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.

📄 License

This project is licensed under the Apache - 2.0 license.

For more information about training this model, please check out the blog: https://pytorch.org/blog/maximizing-training/