Banglat5

🚀 BanglaT5

BanglaT5 是一个序列到序列的Transformer模型，本仓库包含该模型的预训练检查点。此模型通过 "Span Corruption" 目标进行预训练。使用此检查点进行微调的模型在许多孟加拉语自然语言生成（NLG）任务中取得了最先进的成果。

对于不同下游任务（如 机器翻译、抽象文本摘要、问答 等）的微调，请参考官方GitHub 仓库 中的脚本。

⚠️ 重要提示

该模型使用特定的归一化管道进行预训练，可在 此处 获取。官方GitHub仓库中的所有微调脚本默认使用此归一化方法。如果需要将预训练模型应用于其他任务，请确保在分词前使用此管道对文本单元进行归一化处理，以获得最佳效果。下面给出一个基本示例。

🚀 快速开始

✨ 主要特性

• 基于序列到序列的Transformer架构，通过 "Span Corruption" 目标进行预训练。
• 在多个孟加拉语NLG任务中取得了最先进的成果。

📦 安装指南

使用此模型，你需要安装 transformers 和 normalizer 库：

pip install transformers
pip install git+https://github.com/csebuetnlp/normalizer

💻 使用示例

基础用法

以下是在 transformers 库中使用该模型的示例代码（测试版本为 4.11.0.dev0）：

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from normalizer import normalize # pip install git+https://github.com/csebuetnlp/normalizer

model = AutoModelForSeq2SeqLM.from_pretrained("csebuetnlp/banglat5")
tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglat5", use_fast=False)

input_sentence = ""
input_ids = tokenizer(normalize(input_sentence), return_tensors="pt").input_ids
generated_tokens = model.generate(input_ids)
decoded_tokens = tokenizer.batch_decode(generated_tokens)[0]

print(decoded_tokens)

📚 详细文档

Benchmarks

以下是监督微调的基准测试结果：

基准测试数据集如下：

• 机器翻译 (MT): Machine Translation
• 抽象文本摘要 (TS): Abstractive Text Summarization
• 问答 (QA): Question Answering
• 多轮对话生成 (MD): Multi Turn Dialogue Generation
• 新闻标题生成 (NHG): News Headline Generation
• 跨语言摘要 (XLS): Cross-lingual Summarization

📄 许可证

本项目采用 cc-by-nc-sa-4.0 许可证。

📚 引用

如果你使用此模型，请引用以下论文：

@article{bhattacharjee2022banglanlg,
  author    = {Abhik Bhattacharjee and Tahmid Hasan and Wasi Uddin Ahmad and Rifat Shahriyar},
  title     = {BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla},
  journal   = {CoRR},
  volume    = {abs/2205.11081},
  year      = {2022},
  url       = {https://arxiv.org/abs/2205.11081},
  eprinttype = {arXiv},
  eprint    = {2205.11081}
}

如果你使用了归一化模块，请引用以下论文：

@inproceedings{hasan-etal-2020-low,
    title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
    author = "Hasan, Tahmid  and
      Bhattacharjee, Abhik  and
      Samin, Kazi  and
      Hasan, Masum  and
      Basak, Madhusudan  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
    doi = "10.18653/v1/2020.emnlp-main.207",
    pages = "2612--2623",
    abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
}