Pii Entity Extractor

🚀 使用DeBERTa進行PII檢測的模型

本模型是一個專為命名實體識別（NER）任務微調的模型，基於microsoft/deberta 。它能夠精準檢測各類個人身份信息（PII）實體，如姓名、社保號碼、電話號碼、信用卡號、地址等，在保護個人隱私和數據安全方面具有重要價值。

🚀 快速開始

from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

model_name = "AI-Enthusiast11/pii-entity-extractor"  
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Post processing logic to combine the subword tokens
def merge_tokens(ner_results):
    entities = {}
    for entity in ner_results:
        entity_type = entity["entity_group"]
        entity_value = entity["word"].replace("##", "")  # Remove subword prefixes

        # Handle token merging
        if entity_type not in entities:
            entities[entity_type] = []
        if entities[entity_type] and not entity_value.startswith(" "):
            # If the previous token exists and this one isn't a new word, merge it
            entities[entity_type][-1] += entity_value
        else:
            entities[entity_type].append(entity_value)

    return entities

def redact_text_with_labels(text):
    ner_results = nlp(text)

    # Merge tokens for multi-token entities (if any)
    cleaned_entities = merge_tokens(ner_results)

    redacted_text = text
    for entity_type, values in cleaned_entities.items():
        for value in values:
            # Replace each identified entity with the label
            redacted_text = redacted_text.replace(value, f"[{entity_type}]")

    return redacted_text

#Loading the pipeline
nlp = pipeline("ner", model=model_name, tokenizer=tokenizer, aggregation_strategy="simple")

# Example input (choose one from your examples)
example = "Hi, I’m Mia Thompson. I recently noticed that my electricity bill hasn’t been updated despite making the payment last week. I used account number 4893172051 linked with routing number 192847561. My service was nearly suspended, and I’d appreciate it if you could verify the payment. You can reach me at 727-814-3902 if more information is needed."

# Run pipeline and process result
ner_results = nlp(example)
cleaned_entities = merge_tokens(ner_results)

# Print the NER results
print("\n==NER Results:==\n")
for entity_type, values in cleaned_entities.items():
    print(f"  {entity_type}: {', '.join(values)}")

# Redact the single example with labels
redacted_example = redact_text_with_labels(example)

# Print the redacted result
print(f"\n==Redacted Example:==\n{redacted_example}")

✨ 主要特性

• 精準檢測：能夠準確識別多種個人身份信息（PII）實體，包括姓名、社保號碼、電話號碼等。
• 微調優化：基於microsoft/deberta模型進行微調，在PII檢測任務上表現出色。
• 易於使用：提供了清晰的代碼示例，方便用戶快速上手。

📦 安裝指南

文檔未提及安裝步驟，故跳過該章節。

💻 使用示例

基礎用法

# 以下是使用模型進行PII檢測的基礎代碼示例
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

model_name = "AI-Enthusiast11/pii-entity-extractor"  
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# 加載pipeline
nlp = pipeline("ner", model=model_name, tokenizer=tokenizer, aggregation_strategy="simple")

# 示例輸入
example = "Hi, I’m Mia Thompson. I recently noticed that my electricity bill hasn’t been updated despite making the payment last week. I used account number 4893172051 linked with routing number 192847561. My service was nearly suspended, and I’d appreciate it if you could verify the payment. You can reach me at 727-814-3902 if more information is needed."

# 運行pipeline並處理結果
ner_results = nlp(example)

高級用法

# 以下代碼展示瞭如何對檢測結果進行後處理，合併子詞標記並對文本進行脫敏處理
# Post processing logic to combine the subword tokens
def merge_tokens(ner_results):
    entities = {}
    for entity in ner_results:
        entity_type = entity["entity_group"]
        entity_value = entity["word"].replace("##", "")  # Remove subword prefixes

        # Handle token merging
        if entity_type not in entities:
            entities[entity_type] = []
        if entities[entity_type] and not entity_value.startswith(" "):
            # If the previous token exists and this one isn't a new word, merge it
            entities[entity_type][-1] += entity_value
        else:
            entities[entity_type].append(entity_value)

    return entities

def redact_text_with_labels(text):
    ner_results = nlp(text)

    # Merge tokens for multi-token entities (if any)
    cleaned_entities = merge_tokens(ner_results)

    redacted_text = text
    for entity_type, values in cleaned_entities.items():
        for value in values:
            # Replace each identified entity with the label
            redacted_text = redacted_text.replace(value, f"[{entity_type}]")

    return redacted_text

# 示例輸入
example = "Hi, I’m Mia Thompson. I recently noticed that my electricity bill hasn’t been updated despite making the payment last week. I used account number 4893172051 linked with routing number 192847561. My service was nearly suspended, and I’d appreciate it if you could verify the payment. You can reach me at 727-814-3902 if more information is needed."

# 運行pipeline並處理結果
ner_results = nlp(example)
cleaned_entities = merge_tokens(ner_results)

# 打印NER結果
print("\n==NER Results:==\n")
for entity_type, values in cleaned_entities.items():
    print(f"  {entity_type}: {', '.join(values)}")

# 對示例文本進行脫敏處理
redacted_example = redact_text_with_labels(example)

# 打印脫敏後的結果
print(f"\n==Redacted Example:==\n{redacted_example}")

📚 詳細文檔

模型詳情

模型描述

這是一個基於Transformer架構的模型，通過在自定義數據集上進行微調，能夠有效檢測敏感信息，通常歸類為個人身份信息（PII）。該模型採用序列標註的方式，通過標記級別的分類來識別實體。

訓練詳情

訓練數據

模型在包含以下PII實體類型的自定義標註數據集上進行了微調：

• 姓名（NAME）
• 社保號碼（SSN）
• 電話號碼（PHONE-NO）
• 信用卡號（CREDIT-CARD-NO）
• 銀行賬號（BANK-ACCOUNT-NO）
• 銀行路由號（BANK-ROUTING-NO）
• 地址（ADDRESS）

訓練輪次日誌

SeqEval分類報告

總結

• 微平均：0.95
• 宏平均：0.95
• 加權平均：0.95

評估

測試數據

評估在同一標註數據集的預留部分上進行。

評估指標

• 精確率
• 召回率
• F1分數（通過seqeval計算）
• 實體級別的細分指標
• 標記級別的準確率

評估結果

• 大多數標籤的F1分數始終高於0.95，表明模型在PII檢測方面具有較強的魯棒性。

建議

⚠️ 重要提示

在高風險環境中，建議進行人工審核。

💡 使用建議

在部署模型之前，建議在自己的特定領域數據上進行評估。