Hplt Bert Base Sk

🚀 HPLT斯洛伐克語Bert模型

HPLT斯洛伐克語Bert模型是由HPLT項目首次發佈的僅編碼器單語言模型之一。它是一種掩碼語言模型，具體採用了經典BERT模型的改進版本LTG - BERT。

🚀 快速開始

本模型是基於Transformer架構的單語言掩碼語言模型，專為斯洛伐克語設計，可用於多種自然語言處理任務。

✨ 主要特性

• 模型類型：基於LTG - BERT架構的掩碼語言模型。
• 訓練數據：使用特定於斯洛伐克語的HPLT數據進行訓練，來自HPLT 1.2數據發佈。
• 超參數設置：
  • 隱藏層大小：768
  • 注意力頭數：12
  • 層數：12
  • 詞彙表大小：32768
• 分詞器：每個模型使用在特定語言的HPLT數據上訓練的分詞器。

📦 安裝指南

此模型目前需要來自modeling_ltgbert.py的自定義包裝器，因此在加載模型時應使用trust_remote_code = True。

💻 使用示例

基礎用法

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("HPLT/hplt_bert_base_sk")
model = AutoModelForMaskedLM.from_pretrained("HPLT/hplt_bert_base_sk", trust_remote_code=True)

mask_id = tokenizer.convert_tokens_to_ids("[MASK]")
input_text = tokenizer("It's a beautiful[MASK].", return_tensors="pt")
output_p = model(**input_text)
output_text = torch.where(input_text.input_ids == mask_id, output_p.logits.argmax(-1), input_text.input_ids)

# 應輸出: '[CLS] It's a beautiful place.[SEP]'
print(tokenizer.decode(output_text[0].tolist()))

高級用法

目前實現了以下類：AutoModel、AutoModelMaskedLM、AutoModelForSequenceClassification、AutoModelForTokenClassification、AutoModelForQuestionAnswering和AutoModeltForMultipleChoice。

🔧 技術細節

所有HPLT僅編碼器模型使用相同的超參數，大致遵循BERT - base設置。每個模型使用在特定語言的HPLT數據上訓練的分詞器。有關訓練語料庫的大小、評估結果等更多信息，請參閱語言模型訓練報告。

中間檢查點

我們為每個模型在每3125個訓練步驟的間隔發佈10箇中間檢查點，存儲在單獨的分支中。命名約定為stepXXX，例如step18750。

可以使用transformers庫的revision參數加載特定的模型版本：

model = AutoModelForMaskedLM.from_pretrained("HPLT/hplt_bert_base_sk", revision="step21875", trust_remote_code=True)

可以使用以下代碼訪問模型的所有版本：

from huggingface_hub import list_repo_refs
out = list_repo_refs("HPLT/hplt_bert_base_sk")
print([b.name for b in out.branches])

📄 許可證

本項目採用Apache - 2.0許可證。

引用

@inproceedings{samuel-etal-2023-trained,
    title = "Trained on 100 million words and still in shape: {BERT} meets {B}ritish {N}ational {C}orpus",
    author = "Samuel, David  and
      Kutuzov, Andrey  and
      {\O}vrelid, Lilja  and
      Velldal, Erik",
    editor = "Vlachos, Andreas  and
      Augenstein, Isabelle",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2023",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-eacl.146",
    doi = "10.18653/v1/2023.findings-eacl.146",
    pages = "1954--1974"
}

@inproceedings{de-gibert-etal-2024-new-massive,
    title = "A New Massive Multilingual Dataset for High-Performance Language Technologies",
    author = {de Gibert, Ona  and
      Nail, Graeme  and
      Arefyev, Nikolay  and
      Ba{\~n}{\'o}n, Marta  and
      van der Linde, Jelmer  and
      Ji, Shaoxiong  and
      Zaragoza-Bernabeu, Jaume  and
      Aulamo, Mikko  and
      Ram{\'\i}rez-S{\'a}nchez, Gema  and
      Kutuzov, Andrey  and
      Pyysalo, Sampo  and
      Oepen, Stephan  and
      Tiedemann, J{\"o}rg},
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.100",
    pages = "1116--1128",
    abstract = "We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of {\mbox{$\approx$}} 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.",
}

相關鏈接

• 訓練代碼
• 所有75次運行的訓練統計信息