Doge 20M Chinese

🚀 Doge 20M CN

Doge 20M CN is a language model that uses Dynamic Mask Attention for sequence transformation. It offers flexibility in state transformation and is trained by the SmallDoge community.

🚀 Quick Start

Install the Required Libraries

First, make sure you have the transformers library installed. If not, you can install it using the following command:

pip install transformers

Use the Model

Here is a basic example of using the Doge-20M-Chinese model for text generation:

>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("wubingheng/Doge-20M-Chinese")
>>> model = AutoModelForCausalLM.from_pretrained("wubingheng/Doge-20M-Chinese", trust_remote_code=True)
>>> inputs = tokenizer("你好", return_tensors="pt")

>>> out = model.generate(**inputs, max_new_tokens=100)
>>> print(tokenizer.batch_decode(out))

✨ Features

• Dynamic Mask Attention: Doge uses Dynamic Mask Attention as sequence transformation, allowing the Transformer to use self - attention during training and state space during inference.
• Flexible State Transformation: It can use Multi - Layer Perceptron or Cross Domain Mixture of Experts as state transformation. The Cross Domain Mixture of Experts can directly inherit the weights of Multi - Layer Perceptron for further training.

📚 Documentation

Model Details

Environment:

• Image: nvcr.io/nvidia/pytorch:24.12-py3
• Hardware: 1x NVIDIA RTX 4090
• Software: Transformers

Model Information Table

📄 License

This project is licensed under the Apache-2.0 license. You can find the full license text in the small-doge repository.

📚 Citation

If you use this model in your research, please cite it using the following BibTeX entry:

@misc{smalldoges,
  title={SmallDoges: A Family of Dynamic UltraFast Small Language Models}, 
  author={Jingze, Shi and Yifan, Wu and Bingheng, Wu and Yuyu, Luo},
  year={2025},
  month={March},
  url={https://github.com/SmallDoges/small-doge}
}