roberta-classical-chinese-base-sentence-segmentation Open-source Model - Automatically Identify Classical Chinese Sentence Boundaries

Roberta Classical Chinese Base Sentence Segmentation

Developed by KoichiYasuoka

This is a RoBERTa model pre-trained on Classical Chinese, specifically designed for sentence segmentation tasks, capable of automatically identifying sentence boundaries in Classical Chinese texts.

Sequence Labeling

Transformers

OtherOpen Source License:Apache-2.0 #Classical Chinese sentence segmentation #Ancient Chinese processing #RoBERTa fine-tuning

Downloads 34

Release Time : 3/2/2022

Model Overview

This model is used for sentence segmentation tasks in Classical Chinese texts, capable of automatically identifying sentence boundaries. Each segmented sentence starts with the token label 'B' and ends with 'E' (single-character sentences use the token label 'S').

Model Features

Specialized for Classical Chinese

Pre-trained and optimized specifically for Classical Chinese, accurately identifying sentence boundaries in Classical Chinese texts.

Based on RoBERTa Architecture

Utilizes the RoBERTa architecture, offering robust contextual understanding capabilities.

Token Classification

Employs a B/E/S tagging system to mark sentence boundaries, suitable for complex Classical Chinese structures.

Model Capabilities

Classical Chinese processing

Sentence segmentation

Text token classification

Use Cases

Ancient text digitization

Automatic segmentation of ancient texts

Automatically segments sentences in ancient literature for subsequent analysis and processing.

Accurately identifies sentence boundaries in Classical Chinese

Classical Chinese education

Preprocessing teaching materials

Automatically segments sentences in Classical Chinese textbooks for educational use.

Improves efficiency in preparing teaching materials

🚀 roberta-classical-chinese-base-sentence-segmentation

This is a RoBERTa model designed for sentence segmentation in Classical Chinese texts, offering high - precision segmentation capabilities.

🚀 Quick Start

This is a RoBERTa model pre-trained on Classical Chinese texts for sentence segmentation, derived from roberta-classical-chinese-base-char. Every segmented sentence begins with token-class "B" and ends with token-class "E" (except for single-character sentence with token-class "S").

✨ Features

Precise Segmentation: Specifically pre - trained on Classical Chinese texts to accurately segment sentences.
Clear Token Classification: Each sentence has clear token - class markers for easy identification.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer,AutoModelForTokenClassification
tokenizer=AutoTokenizer.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
model=AutoModelForTokenClassification.from_pretrained("KoichiYasuoka/roberta-classical-chinese-base-sentence-segmentation")
s="子曰學而時習之不亦説乎有朋自遠方來不亦樂乎人不知而不慍不亦君子乎"
p=[model.config.id2label[q] for q in torch.argmax(model(tokenizer.encode(s,return_tensors="pt"))["logits"],dim=2)[0].tolist()[1:-1]]
print("".join(c+"。" if q=="E" or q=="S" else c for c,q in zip(s,p)))

📚 Documentation

Reference

Koichi Yasuoka: Sentence Segmentation of Classical Chinese Texts Using Transformers and BERT/RoBERTa Models, IPSJ Symposium Series, Vol.2021, No.1 (December 2021), pp.104 - 109.

📄 License

This model is under the "apache - 2.0" license.

Property	Details
Model Type	RoBERTa model for Classical Chinese sentence segmentation
Base Model	KoichiYasuoka/roberta - classical - chinese - base - char
License	apache - 2.0
Pipeline Tag	token - classification
Tags	classical chinese, literary chinese, ancient chinese, sentence segmentation, token - classification
Language	lzh

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご