Banglat5 Small

🚀 孟加拉语T5模型（BanglaT5）

本仓库包含了孟加拉语T5模型（小版本） 的预训练检查点。这是一个基于序列到序列的Transformer模型，通过"Span Corruption"目标进行预训练。使用此检查点进行微调的模型在许多孟加拉语自然语言生成（NLG）任务中取得了最先进的成果。

该模型可用于不同的下游任务，如机器翻译、抽象文本摘要、问答系统等。若要进行微调，请参考官方GitHub 仓库 中的脚本。

⚠️ 重要提示

该模型使用了特定的归一化管道进行预训练，可在 此处 获取。官方GitHub仓库中的所有微调脚本默认使用此归一化方法。如果需要将预训练模型应用于其他任务，请确保在分词前使用此管道对文本单元进行归一化，以获得最佳效果。下面给出一个基本示例。

🚀 快速开始

✨ 主要特性

• 基于序列到序列的Transformer架构，适用于多种孟加拉语NLG任务。
• 预训练模型在多个NLG任务中达到了最先进的性能。

📦 安装指南

在使用该模型之前，你需要安装必要的库。可以使用以下命令安装归一化库：

pip install git+https://github.com/csebuetnlp/normalizer

💻 使用示例

基础用法

以下是在transformers库中使用该模型的示例代码（测试版本为4.11.0.dev0）：

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from normalizer import normalize # pip install git+https://github.com/csebuetnlp/normalizer

model = AutoModelForSeq2SeqLM.from_pretrained("csebuetnlp/banglat5_small")
tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglat5_small", use_fast=False)

input_sentence = ""
input_ids = tokenizer(normalize(input_sentence), return_tensors="pt").input_ids
generated_tokens = model.generate(input_ids)
decoded_tokens = tokenizer.batch_decode(generated_tokens)[0]

print(decoded_tokens)

📚 详细文档

🔍 基准测试

以下是模型在监督微调下的基准测试结果：

基准测试数据集如下：

• 机器翻译（MT）：数据集链接
• 抽象文本摘要（TS）：数据集链接
• 问答系统（QA）：数据集链接
• 多轮对话生成（MD）：数据集链接
• 新闻标题生成（NHG）：数据集链接
• 跨语言摘要（XLS）：数据集链接

📄 许可证

本项目采用CC - BY - NC - SA 4.0许可证。

📚 引用说明

如果你使用了该模型，请引用以下论文：

@article{bhattacharjee2022banglanlg,
  author    = {Abhik Bhattacharjee and Tahmid Hasan and Wasi Uddin Ahmad and Rifat Shahriyar},
  title     = {BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla},
  journal   = {CoRR},
  volume    = {abs/2205.11081},
  year      = {2022},
  url       = {https://arxiv.org/abs/2205.11081},
  eprinttype = {arXiv},
  eprint    = {2205.11081}
}

如果你使用了归一化模块，请引用以下论文：

@inproceedings{hasan-etal-2020-low,
    title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
    author = "Hasan, Tahmid  and
      Bhattacharjee, Abhik  and
      Samin, Kazi  and
      Hasan, Masum  and
      Basak, Madhusudan  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
    doi = "10.18653/v1/2020.emnlp-main.207",
    pages = "2612--2623",
    abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
}