Mdeberta V3 Base Multilingual Pos Tagger

🚀 多语言词性标注模型

本项目提供了使用Hugging Face进行词性标注的脚本，可提取文本中的词性信息，还能自动检测和提取名词与停用词。同时，对多语言词性标注模型的评估框架和训练配置进行了概述。

🚀 快速开始

本项目提供了使用Hugging Face进行词性标注的脚本，可提取文本中的词性信息，还能自动检测和提取名词与停用词。

✨ 主要特性

• 多语言支持：能够处理多种语言的词性标注任务。
• 词性分类：可以准确识别不同词性的类别。
• 停用词提取：自动检测并提取文本中的名词和停用词。

📦 安装指南

本项目依赖于Hugging Face的transformers库，可使用以下命令进行安装：

pip install transformers

💻 使用示例

基础用法

以下代码展示了如何使用Hugging Face的pipeline进行词性标注：

from transformers import pipeline

# Load model and tokenizer
pos_pipeline = pipeline("token-classification", model="jordigonzm/mdeberta-v3-base-multilingual-pos-tagger")

# Input text
text = "On January 3rd, 2024, the $5.7M prototype—a breakthrough in AI-driven robotics—successfully passed all 37 rigorous performance tests!"

# Run POS tagging
words = text.split(" ")
tokens = pos_pipeline(words)

# Print tokens and their categories
for word, group_token in zip(words, tokens):
    print(f"{word:<15}", end=" ")
    for token in group_token:
        print(f"{token['word']:<8} → {token['entity']:<8}", end=" | ")
    print("\n" + "-" * 80)

高级用法

以下代码展示了如何进行词性标注并提取名词和停用词：

from transformers import pipeline

# Load the pre-trained POS tagging model
pos_pipeline = pipeline("ner", model="jordigonzm/mdeberta-v3-base-multilingual-pos-tagger")

# Input text
text = "Companies interested in providing the service must take care of signage and information boards."

# Run POS tagging
tokens = pos_pipeline(text)

# Print raw tokens and their POS tags
print("\nTokens POS tagging:")
for token in tokens:
    print(f"{token['word']:10} → {token['entity']}")

# Reconstruct words correctly
words, buffer, labels = [], [], []

for token in tokens:
    raw_word = token["word"]

    if raw_word.startswith("▁"):  # New word starts
        if buffer:
            words.append("".join(buffer))  # Add the completed word
            labels.append(buffer_label)
        buffer = [raw_word.replace("▁", "")]
        buffer_label = token["entity"]
    else:
        buffer.append(raw_word)  # Continue word construction

# Add last word in buffer
if buffer:
    words.append("".join(buffer))
    labels.append(buffer_label)

# Print final POS tagging results
print("\nPOS tagging results:")
for word, label in zip(words, labels):
    print(f"{word:<15} → {label}")

# Define valid POS tags for extraction
noun_tags = {"NOUN", "PROPN"}  # Nouns & Proper Nouns
stopword_tags = {"DET", "ADP", "PRON", "AUX", "CCONJ", "SCONJ", "PART"}  # Common stopword POS tags

# Extract nouns and stopwords separately
filtered_nouns = [word for word, tag in zip(words, labels) if tag in noun_tags]
stopwords = [word for word, tag in zip(words, labels) if tag in stopword_tags]

# Print extracted words
print("\nFiltered Nouns and Proper Nouns:", filtered_nouns)
print("\nStopwords detected:", stopwords)

📚 详细文档

多语言词性标注概述

本报告概述了多语言词性标注模型的评估框架和潜在的训练配置。该模型基于Transformer架构，并在有限的训练轮次后进行评估。

预期范围

训练配置