Roberta Base Wechsel Chinese

🚀 roberta-base-wechsel-chinese

本項目的模型使用WECHSEL方法進行訓練，該方法可有效初始化子詞嵌入，用於單語言模型的跨語言遷移。

查看代碼請訪問：https://github.com/CPJKU/wechsel

查看論文請訪問：https://aclanthology.org/2022.naacl-main.293/

🚀 快速開始

本項目提供了使用WECHSEL方法訓練的多語言模型，包括法語、德語、中文和斯瓦希里語等。你可以通過上述鏈接查看代碼和論文獲取更多信息。

✨ 主要特性

本項目使用WECHSEL方法對單語言模型進行跨語言遷移，提高了模型在不同語言上的性能。通過與其他基準模型對比，展示了該方法在多種語言任務上的有效性。

📚 詳細文檔

RoBERTa模型性能

GPT2模型性能

更多詳細信息請參考我們的論文。

📄 許可證

本項目採用MIT許可證。

📖 引用信息

如果你使用了本項目的模型或方法，請引用以下論文：

@inproceedings{minixhofer-etal-2022-wechsel,
    title = "{WECHSEL}: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models",
    author = "Minixhofer, Benjamin  and
      Paischer, Fabian  and
      Rekabsaz, Navid",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.293",
    pages = "3992--4006",
    abstract = "Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method {--} called WECHSEL {--} to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.",
}