Molm 700M 4B

🚀 MoLM

MoLM is a collection of MoE-based language models with parameter scales ranging from 4 billion to 8 billion. This repository is for the MoLM-700M-4B pretrained model, converted to the Hugging Face Transformers format. You can find links to other models in the index at the bottom.

🚀 Quick Start

To load the model, you need to install the ModuleFormer package. Then you can load the model with the following code:

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, AutoModelForSequenceClassification
from moduleformer import ModuleFormerForCausalLM, ModuleFormerConfig, ModuleFormerForSequenceClassification
AutoConfig.register("moduleformer", ModuleFormerConfig)
AutoModelForCausalLM.register(ModuleFormerConfig, ModuleFormerForCausalLM)
AutoModelForSequenceClassification.register(ModuleFormerConfig, ModuleFormerForSequenceClassification)

tokenizer = AutoTokenizer.from_pretrained('ibm/MoLM-700M-4B')
model = AutoModelForCausalLM.from_pretrained('ibm/MoLM-700M-4B')

✨ Features

• Scalable Architecture: MoLM comes in different parameter sizes (4B and 8B), with variants offering different computational costs (350M and 700M).
• Efficient Computation: Each input token only activates a fraction of the total parameters, making it computationally equivalent to smaller dense models.
• Trained on Large Datasets: All models are trained on 300 billion tokens from publicly available sources.

📚 Documentation

Model Details

• MoLM-350M-4B: It has 4 billion parameters, but each input token only activates 350M parameters, making it computationally equivalent to a 350M dense model.
• MoLM-700M-4B: It has 4 billion parameters and is computationally equivalent to a 700M dense model.
• MoLM-700M-8B: It has 8 billion parameters and is computationally equivalent to a 700M dense model.

All models are trained on 300 billion tokens from publicly available sources, with a learning rate of 3.0 x 10^-4 and a global batch-size of 3M tokens.

Model Developers

IBM

Variations

MoLM comes in two different parameter sizes — 4B and 8B. The 4B models have two variants with different computation costs — 350M and 700M.

Input

The models accept text as input.

Output

The models generate text as output.

Model Architecture

MoLM is an auto-regressive language model that uses the ModuleFormer architecture. It has 16 attention modules in each attention layer and 32 MLP modules in each MLP layer. During inference, in each layer, MoLM-350M-4B and MoLM-700M-8B activate 2 modules for each token, while MoLM-700M-4B activates 4 modules. MoLM-350M-4B and MoLM-700M-4B have 24 blocks and MoLM-700M-8B has 48 blocks.

Status

This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.

Research Paper

"ModuleFormer: Modularity Emerges from Mixture-of-Experts"

📦 Installation

To use the MoLM models, you need to install the ModuleFormer package.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, AutoModelForSequenceClassification
from moduleformer import ModuleFormerForCausalLM, ModuleFormerConfig, ModuleFormerForSequenceClassification
AutoConfig.register("moduleformer", ModuleFormerConfig)
AutoModelForCausalLM.register(ModuleFormerConfig, ModuleFormerForCausalLM)
AutoModelForSequenceClassification.register(ModuleFormerConfig, ModuleFormerForSequenceClassification)

tokenizer = AutoTokenizer.from_pretrained('ibm/MoLM-700M-4B')
model = AutoModelForCausalLM.from_pretrained('ibm/MoLM-700M-4B')

🔧 Technical Details

Training Data

MoLM models are pretrained on 300 billion tokens of data from publicly available sources.

Evaluation Results

In this section, we report the results for the MoLM models on standard academic benchmarks. For all the evaluations, we use LM evaluations Harness.

⚠️ Important Note

MoLM is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, MoLM’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of MoLM, developers should perform safety testing and tuning tailored to their specific applications of the model.

📄 License

This project is licensed under the Apache-2.0 license.

📖 Citation

Please cite the following paper if you use the data or code in this repo.

@article{shen2023moduleformer,
  title={ModuleFormer: Learning Modular Large Language Models From Uncurated Data},
  author={Shen, Yikang and Zhang, Zheyu and Cao, Tianyou and Tan, Shawn and Chen, Zhenfang and Gan, Chuang},
  journal={arXiv preprint arXiv:2306.04640},
  year={2023}
}

MoLM Model Index