Marin 8b Instruct

🚀 Marin 8B

Marin 8B is an SFT model developed by the Marin team at Stanford CRFM. It is a Transformer-style autoregressive language model trained on various datasets, aiming to provide high - quality text generation capabilities.

🚀 Quick Start

Marin 8B uses the Llama architecture and can work out - of - the - box with the Hugging Face Transformers library and other libraries supporting the Llama architecture. You can use the following code to perform inference:

from transformers import AutoModelForCausalLM, AutoTokenizer
marin = AutoModelForCausalLM.from_pretrained("marin-community/marin-8b-base")
tokenizer = AutoTokenizer.from_pretrained("marin-community/marin-8b-base")
message = ["The Marin wind is"]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
response = marin.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])

To load a specific checkpoint, add the revision argument:

marin = AutoModelForCausalLM.from_pretrained("marin-community/marin-8b-base", revision="deeper-starling")

✨ Features

• Diverse Datasets: Trained on a wide range of datasets, including [Nemotron - CC](https://data.commoncrawl.org/contrib/Nemotron/Nemotron - CC/index.html), Starcoder Data, etc., to enhance its knowledge and generalization ability.
• Multiple Checkpoints: A number of training checkpoints are released, allowing users to choose according to their needs.
• Good Performance: In benchmark tests, it shows good performance compared to other models such as Llama 3.1 Base and OLMo 2 Base.

📦 Installation

Marin 8B uses a variant of the Llama 3 tokenizer: [stanford - crfm/marin - tokenizer](https://huggingface.co/stanford - crfm/marin - tokenizer/). It can be used with the Hugging Face Transformers library.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
marin = AutoModelForCausalLM.from_pretrained("marin-community/marin-8b-base")
tokenizer = AutoTokenizer.from_pretrained("marin-community/marin-8b-base")
message = ["The Marin wind is"]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
response = marin.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])

Advanced Usage

marin = AutoModelForCausalLM.from_pretrained("marin-community/marin-8b-base", revision="deeper-starling")

📚 Documentation

Model Description

Model Sources

• Project Page: https://marin.community
• Repositories:
  • Core repo (data and experiment management): https://github.com/marin - community/marin
  • Training code: https://github.com/stanford - crfm/levanter
• Retrospective: https://marin.readthedocs.io/en/latest/reports/marin - 8b - retro.html
• W&B Logs: [Marin 8B](https://wandb.ai/stanford - mercury/marin/reports/Tootsie - 8B---VmlldzoxMTY3MzU3OA)

🔧 Technical Details

Architecture Details

• Architecture: Llama 3 8B
• Hidden size: 4096
• Feedforward size: 14336
• Number of layers: 32
• Number of attention heads: 32
• Number of KV heads: 8

Tokenizer Details

Marin 8B uses a variant of the Llama 3 tokenizer: [stanford - crfm/marin - tokenizer](https://huggingface.co/stanford - crfm/marin - tokenizer/). It has the same vocabulary but bundles a chat template into the base tokenizer for convenience.

Training Phases

Pre - training Phases

• Kestrel (DCLM WSD - S Phase): DCLM+StarCoder+Proofpile2 using WSD - S (0->2.7T tokens)
• Ocelot (DCLM WSD Phase): Increased batch size, using WSD. (2.7T->3.78T tokens)
• Jellyfish (First Cooldown): Higher quality data (~Dolmino+Fine Math). (3.78T->4.78T tokens)
• Phoenix (Reheated): Rapid rewarming + Nemotron - CC (plus Starcoder). (4.78T->11.1T tokens)
• Starling (Second Cooldown): Another cooldown. A similar process to the first cooldown was followed, but a few new datasets were added. (11.1T->12.75T tokens)
• Deeper Starling: Somewhat more pretraining. (12.75T->13.7T tokens)

All released pre - training checkpoints except Kestrel use an exponential moving average of the model weights.

SFT Phase

SFT was comparably simple, consisting of only one phase for 5.3B tokens.

📄 License

The code and model are released under Apache 2.0.

⚠️ Important Note

Like any base language model or fine - tuned model without safety filtering, these models can easily be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias. Marin 8B has not undergone any safety tuning or evaluation. This model is not intended for fully autonomous use.

💡 Usage Tip

Users should use this model with caution, consider the risks when applying this technology, and verify the responses as many statements from Marin or any LLM are often inaccurate.