Fanformer 1B

🚀 FANformer-1B

FANformer-1B is a 1.1-billion-parameter autoregressive language model. It enhances language modeling through effective periodicity mechanisms and has potential in various text-related tasks.

🚀 Quick Start

Inference Example

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)

input_text = "The concept of periodicity serves as a fundamental organizing principle across the natural world, human societies, and even abstract systems. From the rhythmic cycles of celestial bodies governing seasons and tides to the biological clocks regulating sleep and metabolism in living organisms, recurring patterns create stability amid chaos. In ecosystems, predator-prey population oscillations maintain balance, while the carbon cycle ensures Earth's climate resilience. Culturally, humanity has structured civilizations around agricultural cycles, religious calendars, and economic fluctuations—harvest festivals marking seasonal abundance, financial markets swaying between boom and bust. Even at the quantum level, wave functions reveal inherent periodicity that underpins material reality. This universal recurrence enables prediction, adaptation, and innovation: by recognizing cycles, we"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=512, do_sample=True, temperature=0.6, top_p=0.8)

print(tokenizer.decode(outputs[0]))

✨ Features

• Enhanced Periodicity Modeling: FANformer-1B is a decoder-only LLM with enhanced periodicity modeling, which can capture periodic patterns in training data.
• Versatile Applications: It can be used for general - purpose text generation and understanding, and can also be fine - tuned for tasks like summarization, question answering, and dialogue.

📚 Documentation

Model Description

FANformer-1B is a 1.1-billion-parameter autoregressive language model pre-trained from scratch to enhance language modeling through effective periodicity mechanisms. Its revised architecture (olmo/model.py) introduces the FAN Layer, a novel component designed to capture periodic patterns in training data, enabling superior learning efficiency and performance.

Training Details

Intended Uses

• Primary Use: General - purpose text generation and understanding.
• Downstream Use: Can be fine - tuned for tasks like summarization, question answering, and dialogue.
• Limitations: May inherit biases from training data. Performance on low - resource languages is not guaranteed.

Evaluation

📄 License

This model is released under the MIT License.

📖 Citation

@article{dong2025fanformer,
  title={FANformer: Improving Large Language Models Through Effective Periodicity Modeling},
  author={Dong, Yihong and Li, Ge and Jiang, Xue and Tao, Yongding and Zhang, Kechi and Zhu, Hao and Liu, Huanyu and Ding, Jiazheng and Li, Jia and Deng, Jinliang and Mei, Hong},
  journal={arXiv preprint arXiv:2502.21309},
  year={2025}
}

@article{dong2024fan,
  title={FAN: Fourier Analysis Networks},
  author={Yihong Dong and Ge Li and Yongding Tao and Xue Jiang and Kechi Zhang and Jia Li and Jing Su and Jun Zhang and Jingjing Xu},
  journal={arXiv preprint arXiv:2410.02675},
  year={2024}
}