Doge 320M

🚀 Doge 320M

Doge 320M is a language model that uses Dynamic Mask Attention for sequence transformation. It can leverage Multi - Layer Perceptron or Cross Domain Mixture of Experts for state transformation. This model is trained by the SmallDoge community.

🚀 Quick Start

Doge uses Dynamic Mask Attention as sequence transformation and can use Multi - Layer Perceptron or Cross Domain Mixture of Experts as state transformation. Dynamic Mask Attention allows the Transformer to use self - attention during training and state space during inference, and Cross Domain Mixture of Experts can directly inherit the weights of Multi - Layer Perceptron for further training. This model is trained by SmallDoge community. For detailed algorithm and model architecture, a paper is coming soon. All training details and code are available in the [small - doge](https://github.com/SmallDoges/small - doge) repository.

💻 Usage Examples

Basic Usage

>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("SmallDoge/Doge-320M")
>>> model = AutoModelForCausalLM.from_pretrained("SmallDoge/Doge-320M", trust_remote_code=True)
>>> inputs = tokenizer("Hey how are you doing?", return_tensors="pt")

>>> out = model.generate(**inputs, max_new_tokens=100)
>>> print(tokenizer.batch_decode(out))

📚 Documentation

Model Details

We build the Doge by doing Per - Training on [Smollm - Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm - corpus). If you want to continue pre - training this model, you can find the unconverged checkpoint [here](https://huggingface.co/SmallDoge/Doge - 320M - checkpoint). These models have not been fine - tuned for instruction, the instruction model is [here](https://huggingface.co/SmallDoge/Doge - 320M - Instruct).

Pre - Training:

Evaluation:

Procedure:

Environment:

• Image: nvcr.io/nvidia/pytorch:24.12 - py3
• Hardware: 1x NVIDIA RTX 4090
• Software: Transformers

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Citation

@misc{smalldoges,
  title={SmallDoges: A Family of Dynamic UltraFast Small Language Models}, 
  author={Jingze, Shi and Yifan, Wu and Bingheng, Wu and Yuyu, Luo},
  year={2025},
  month={March},
  url={https://github.com/SmallDoges/small-doge}
}