Bert Large Cantonese

🚀 bert-large-cantonese

This is a BERT model trained from scratch on Cantonese text, featuring a large architecture.

🚀 Quick Start

This model can be directly used with a pipeline for masked language modeling.

✨ Features

• Trained from scratch on Cantonese text.
• Adopts a large BERT architecture (24 - layer, 1024 - hidden, 16 - heads, 326M parameters).

💻 Usage Examples

Basic Usage

from transformers import pipeline

mask_filler = pipeline(
    "fill-mask",
    model="hon9kon9ize/bert-large-cantonese"
)

mask_filler("雞蛋六隻，糖呢就兩茶匙，仲有[MASK]橙皮添。")

; [{'score': 0.08160534501075745,
;   'token': 943,
;   'token_str': '個',
;   'sequence': '雞 蛋 六 隻 ， 糖 呢 就 兩 茶 匙 ， 仲 有 個 橙 皮 添 。'},
;  {'score': 0.06182105466723442,
;   'token': 1576,
;   'token_str': '啲',
;   'sequence': '雞 蛋 六 隻 ， 糖 呢 就 兩 茶 匙 ， 仲 有 啲 橙 皮 添 。'},
;  {'score': 0.04600336775183678,
;   'token': 1646,
;   'token_str': '嘅',
;   'sequence': '雞 蛋 六 隻 ， 糖 呢 就 兩 茶 匙 ， 仲 有 嘅 橙 皮 添 。'},
;  {'score': 0.03743772581219673,
;   'token': 3581,
;   'token_str': '橙',
;   'sequence': '雞 蛋 六 隻 ， 糖 呢 就 兩 茶 匙 ， 仲 有 橙 橙 皮 添 。'},
;  {'score': 0.031560592353343964,
;   'token': 5148,
;   'token_str': '紅',
;   'sequence': '雞 蛋 六 隻 ， 糖 呢 就 兩 茶 匙 ， 仲 有 紅 橙 皮 添 。'}]

🔧 Technical Details

The training process of this model consists of two stages:

First Training Stage

• Batch size: 512
• Learning rate: 1e - 4
• Learning rate scheduler: linear decay
• Epochs: 1
• Warmup ratio: 0.1
• Loss plot: WanDB

Second Training Stage

• Batch size: 512
• Learning rate: 5e - 5
• Learning rate scheduler: linear decay
• Epochs: 1
• Warmup ratio: 0.1
• Loss plot: WanDB

📄 License

This model is licensed under the CC - BY - 4.0 license.

Additional Information