Mdeberta V3 Base Multilingual Pos Tagger

🚀 多語言詞性標註模型

本項目提供了使用Hugging Face進行詞性標註的腳本，可提取文本中的詞性信息，還能自動檢測和提取名詞與停用詞。同時，對多語言詞性標註模型的評估框架和訓練配置進行了概述。

🚀 快速開始

本項目提供了使用Hugging Face進行詞性標註的腳本，可提取文本中的詞性信息，還能自動檢測和提取名詞與停用詞。

✨ 主要特性

• 多語言支持：能夠處理多種語言的詞性標註任務。
• 詞性分類：可以準確識別不同詞性的類別。
• 停用詞提取：自動檢測並提取文本中的名詞和停用詞。

📦 安裝指南

本項目依賴於Hugging Face的transformers庫，可使用以下命令進行安裝：

pip install transformers

💻 使用示例

基礎用法

以下代碼展示瞭如何使用Hugging Face的pipeline進行詞性標註：

from transformers import pipeline

# Load model and tokenizer
pos_pipeline = pipeline("token-classification", model="jordigonzm/mdeberta-v3-base-multilingual-pos-tagger")

# Input text
text = "On January 3rd, 2024, the $5.7M prototype—a breakthrough in AI-driven robotics—successfully passed all 37 rigorous performance tests!"

# Run POS tagging
words = text.split(" ")
tokens = pos_pipeline(words)

# Print tokens and their categories
for word, group_token in zip(words, tokens):
    print(f"{word:<15}", end=" ")
    for token in group_token:
        print(f"{token['word']:<8} → {token['entity']:<8}", end=" | ")
    print("\n" + "-" * 80)

高級用法

以下代碼展示瞭如何進行詞性標註並提取名詞和停用詞：

from transformers import pipeline

# Load the pre-trained POS tagging model
pos_pipeline = pipeline("ner", model="jordigonzm/mdeberta-v3-base-multilingual-pos-tagger")

# Input text
text = "Companies interested in providing the service must take care of signage and information boards."

# Run POS tagging
tokens = pos_pipeline(text)

# Print raw tokens and their POS tags
print("\nTokens POS tagging:")
for token in tokens:
    print(f"{token['word']:10} → {token['entity']}")

# Reconstruct words correctly
words, buffer, labels = [], [], []

for token in tokens:
    raw_word = token["word"]

    if raw_word.startswith("▁"):  # New word starts
        if buffer:
            words.append("".join(buffer))  # Add the completed word
            labels.append(buffer_label)
        buffer = [raw_word.replace("▁", "")]
        buffer_label = token["entity"]
    else:
        buffer.append(raw_word)  # Continue word construction

# Add last word in buffer
if buffer:
    words.append("".join(buffer))
    labels.append(buffer_label)

# Print final POS tagging results
print("\nPOS tagging results:")
for word, label in zip(words, labels):
    print(f"{word:<15} → {label}")

# Define valid POS tags for extraction
noun_tags = {"NOUN", "PROPN"}  # Nouns & Proper Nouns
stopword_tags = {"DET", "ADP", "PRON", "AUX", "CCONJ", "SCONJ", "PART"}  # Common stopword POS tags

# Extract nouns and stopwords separately
filtered_nouns = [word for word, tag in zip(words, labels) if tag in noun_tags]
stopwords = [word for word, tag in zip(words, labels) if tag in stopword_tags]

# Print extracted words
print("\nFiltered Nouns and Proper Nouns:", filtered_nouns)
print("\nStopwords detected:", stopwords)

📚 詳細文檔

多語言詞性標註概述

本報告概述了多語言詞性標註模型的評估框架和潛在的訓練配置。該模型基於Transformer架構，並在有限的訓練輪次後進行評估。

預期範圍

訓練配置