bert-base-han-chinese-ws Open-source Model - Provides Free Classical Chinese Word Segmentation Function

Bert Base Han Chinese Ws

Developed by ckiplab

This model provides word segmentation functionality for classical Chinese, with training datasets covering four historical periods of Chinese language development.

Sequence Labeling

Transformers

ChineseOpen Source License:Gpl-3.0 #Classical Chinese Word Segmentation #Multi-period Corpus #BERT Architecture

Downloads 14

Release Time : 7/1/2022

Model Overview

A BERT-based Chinese word segmentation model specifically designed for classical Chinese texts, supporting word segmentation tasks from ancient to modern Chinese.

Model Features

Historical Chinese Support

Training data covers four developmental periods of Chinese: Ancient, Middle, Early Modern, and Modern

Academic-grade Corpus

Trained on authoritative annotated corpora from Academia Sinica's Linguistics Research Institute

BERT Architecture

Utilizes BERT-base architecture with excellent contextual understanding capabilities

Model Capabilities

Chinese Word Segmentation

Historical Chinese Processing

Sequence Labeling

Use Cases

Academic Research

Ancient Text Analysis

Automatic word segmentation for ancient Chinese texts

Accurately identifies word boundaries in classical Chinese

Language Evolution Studies

Comparing word segmentation characteristics across different historical periods

Assists linguists in studying the historical evolution of Chinese

Digital Humanities

Ancient Text Digitization

Provides preprocessing support for digitizing classical texts

Enhances searchability and analyzability of ancient texts

🚀 CKIP BERT Base Han Chinese WS

This model offers word segmentation for the ancient Chinese language. Its training dataset encompasses four eras of the Chinese language.

🚀 Quick Start

This model provides word segmentation capabilities for ancient Chinese. It has been trained on datasets covering four different eras of the Chinese language, enabling accurate processing of ancient Chinese texts.

✨ Features

Ancient Chinese Word Segmentation: Specifically designed to handle the unique linguistic characteristics of ancient Chinese.
Comprehensive Training Data: Trained on datasets from four distinct eras of the Chinese language, ensuring broad coverage and high accuracy.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import (
  AutoTokenizer,
  AutoModel,
)

tokenizer = AutoTokenizer.from_pretrained("ckiplab/bert-base-han-chinese-ws")
model = AutoModel.from_pretrained("ckiplab/bert-base-han-chinese-ws")

Advanced Usage

>>> from transformers import pipeline
>>> classifier = pipeline("token-classification", model="ckiplab/bert-base-han-chinese-ws")
>>> classifier("帝堯曰放勳")

# output
[{'entity': 'B',
  'score': 0.9999793,
  'index': 1,
  'word': '帝',
  'start': 0,
  'end': 1},
 {'entity': 'I',
  'score': 0.9915047,
  'index': 2,
  'word': '堯',
  'start': 1,
  'end': 2},
 {'entity': 'B',
  'score': 0.99992275,
  'index': 3,
  'word': '曰',
  'start': 2,
  'end': 3},
 {'entity': 'B',
  'score': 0.99905187,
  'index': 4,
  'word': '放',
  'start': 3,
  'end': 4},
 {'entity': 'I',
  'score': 0.96299917,
  'index': 5,
  'word': '勳',
  'start': 4,
  'end': 5}]

📚 Documentation

Homepage

ckiplab/han-transformers

Training Datasets

The copyright of the datasets belongs to the Institute of Linguistics, Academia Sinica.

Contributors

Chin-Tung Lin at CKIP

📄 License

This project is licensed under the GPL-3.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご