Mamba 7b Rw

🚀 Mamba-7B

Mamba-7B is a 7B parameter model based on the Mamba architecture, trained on the RefinedWeb dataset. It offers strong performance on various natural language benchmarks and serves as a baseline for the paper "Linearizing Large Language Models".

🚀 Quick Start

This model was trained using OpenLM. The weights have been converted to be compatible with HuggingFace.

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("tri-ml/mamba-7b-rw")
model = AutoModelForCausalLM.from_pretrained("tri-ml/mamba-7b-rw")

inputs = tokenizer(["The Toyota Supra"], return_tensors="pt")
gen_kwargs = {"max_new_tokens": 50, "top_p": 0.8, "temperature": 0.8, "do_sample": True, "repetition_penalty": 1.1}
output = model.generate(inputs['input_ids'], **gen_kwargs)
output = tokenizer.decode(output[0].tolist(), skip_special_tokens=True)
print(output)
# The Toyota Supra is a sports car that has been in production since 1978. The car was discontinued in 2002, but it has recently been revived and will be available again in 2020. The Supra has always been known for its powerful engines and agile handling. 

✨ Features

• Innovative Architecture: Based on the Mamba architecture, a state - space model that doesn't use self - attention, unlike the standard transformer architecture.
• Strong Performance: Demonstrates good performance on various natural language benchmarks.
• Large - Scale Training: Trained on 1.2T tokens of the RefinedWeb dataset.

📦 Installation

This model can be used with HuggingFace's transformers library. You can install the necessary libraries using the following command:

pip install transformers

📚 Documentation

Model Details

Training Details

• Mamba-7B was trained using AWS SageMaker on 128 H100 80GB GPUs.
• Training began in March 2024 and lasted three weeks.

Performance Evaluation

Our evaluations were done using the Eleuther LM Eval Harness repo.

🔧 Technical Details

This model was trained as a baseline for our paper Linearizing Large Language Models. The Mamba architecture is a state - space model that shows strong performance on natural language tasks without using self - attention.

📄 License

This model is licensed under Apache License, Version 2.0.

How to Cite

If you use this model, please cite our paper on Linearizing Large Language Models.

@article{Mercat2024Linearizing,
  title={Linearizing Large Language Models},
  author={Jean Mercat and Igor Vasiljevic and Sedrick Keh and Kushal Arora and Achal Dave and Adrien Gaidon and Thomas Kollar},
  journal={arXiv preprint arXiv:2405.06640},
  year={2024}
}

Citations

Mamba

@article{mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}

OpenLM

@misc{open_lm,
  author = {Gururangan, Suchin and Wortsman, Mitchell and Gadre, Samir Yitzhak and Dave, Achal and Kilian, Maciej and Shi, Weijia and Mercat, Jean and Smyrnis, Georgios and Ilharco, Gabriel and Jordan, Matt and Heckel, Reinhard and Dimakis, Alex and Farhadi, Ali and Shankar, Vaishaal and Schmidt, Ludwig},
  title = {{open_lm}:  a minimal but performative language modeling (LM) repository},
  year = {2023},
  note = {GitHub repository},
  url = {https://github.com/mlfoundations/open_lm/}
}