Fanformer 1B

🚀 FANformer-1B模型

FANformer-1B是一个拥有11亿参数的自回归语言模型，它通过有效的周期性机制来增强语言建模能力。该模型可用于通用文本生成和理解，还能针对特定任务进行微调。

🚀 快速开始

你可以按照以下代码示例进行推理：

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)

input_text = "The concept of periodicity serves as a fundamental organizing principle across the natural world, human societies, and even abstract systems. From the rhythmic cycles of celestial bodies governing seasons and tides to the biological clocks regulating sleep and metabolism in living organisms, recurring patterns create stability amid chaos. In ecosystems, predator-prey population oscillations maintain balance, while the carbon cycle ensures Earth's climate resilience. Culturally, humanity has structured civilizations around agricultural cycles, religious calendars, and economic fluctuations—harvest festivals marking seasonal abundance, financial markets swaying between boom and bust. Even at the quantum level, wave functions reveal inherent periodicity that underpins material reality. This universal recurrence enables prediction, adaptation, and innovation: by recognizing cycles, we"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=512, do_sample=True, temperature=0.6, top_p=0.8)

print(tokenizer.decode(outputs[0]))

✨ 主要特性

• 有效周期性建模：通过有效的周期性机制增强语言建模能力。
• 新型架构：引入FAN层，能够捕捉训练数据中的周期性模式，提高学习效率和性能。
• 多功能用途：可用于通用文本生成和理解，还能针对特定任务进行微调。

📦 安装指南

文档未提供安装步骤，可参考transformers库的官方安装指南进行安装。

💻 使用示例

基础用法

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("dongyh/FANformer-1B", trust_remote_code=True)

input_text = "The concept of periodicity serves as a fundamental organizing principle across the natural world, human societies, and even abstract systems. From the rhythmic cycles of celestial bodies governing seasons and tides to the biological clocks regulating sleep and metabolism in living organisms, recurring patterns create stability amid chaos. In ecosystems, predator-prey population oscillations maintain balance, while the carbon cycle ensures Earth's climate resilience. Culturally, humanity has structured civilizations around agricultural cycles, religious calendars, and economic fluctuations—harvest festivals marking seasonal abundance, financial markets swaying between boom and bust. Even at the quantum level, wave functions reveal inherent periodicity that underpins material reality. This universal recurrence enables prediction, adaptation, and innovation: by recognizing cycles, we"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=512, do_sample=True, temperature=0.6, top_p=0.8)

print(tokenizer.decode(outputs[0]))

📚 详细文档

模型描述

训练详情

预期用途

• 主要用途：通用文本生成和理解。
• 下游用途：可针对总结、问答和对话等任务进行微调。
• 局限性：可能继承训练数据中的偏差，对低资源语言的性能无法保证。

评估

🔧 技术细节

FANformer-1B的修订架构（olmo/model.py）引入了FAN层，这是一个新型组件，旨在捕捉训练数据中的周期性模式，从而提高学习效率和性能。

📄 许可证

本项目采用MIT许可证。

📖 引用

@article{dong2025fanformer,
  title={FANformer: Improving Large Language Models Through Effective Periodicity Modeling},
  author={Dong, Yihong and Li, Ge and Jiang, Xue and Tao, Yongding and Zhang, Kechi and Zhu, Hao and Liu, Huanyu and Ding, Jiazheng and Li, Jia and Deng, Jinliang and Mei, Hong},
  journal={arXiv preprint arXiv:2502.21309},
  year={2025}
}

@article{dong2024fan,
  title={FAN: Fourier Analysis Networks},
  author={Yihong Dong and Ge Li and Yongding Tao and Xue Jiang and Kechi Zhang and Jia Li and Jing Su and Jun Zhang and Jingjing Xu},
  journal={arXiv preprint arXiv:2410.02675},
  year={2024}
}