Gpt2 Wechsel French

🚀 GPT2-Wechsel法语模型

GPT2-Wechsel法语模型是使用WECHSEL方法训练的，该方法可以有效地初始化子词嵌入，用于单语言模型的跨语言迁移。本项目提供了模型的代码和相关研究论文，旨在为跨语言自然语言处理任务提供更高效的解决方案。

🚀 快速开始

你可以通过以下链接查看模型代码和相关论文：

• 代码：https://github.com/CPJKU/wechsel
• 论文：https://aclanthology.org/2022.naacl-main.293/

✨ 主要特性

• 跨语言迁移：通过WECHSEL方法，实现单语言模型在多种语言间的有效迁移。
• 性能提升：在多个自然语言处理任务中，相较于传统模型有更好的表现。

📚 详细文档

RoBERTa模型性能对比

GPT2模型性能对比

📄 许可证

本项目采用MIT许可证。

📖 引用

如果您使用了本项目的模型或方法，请引用以下论文：

@inproceedings{minixhofer-etal-2022-wechsel,
    title = "{WECHSEL}: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models",
    author = "Minixhofer, Benjamin  and
      Paischer, Fabian  and
      Rekabsaz, Navid",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.293",
    pages = "3992--4006",
    abstract = "Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method {--} called WECHSEL {--} to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.",
}