Roberta Multilingual Medieval Ner

🚀 多语言中世纪命名实体识别模型（roberta-multilingual-medieval-ner）

这是一个基于多语言Roberta模型在中世纪宪章文本上微调的模型，旨在以扁平与嵌套的方式识别中世纪文本中的地点和人物。训练数据集包含8000篇标注文本，涵盖了11世纪至15世纪的中世纪拉丁语、古法语和古西班牙语。

🚀 快速开始

该模型的使用方式非常简单：

import torch
from transformers import pipeline

pipe = pipeline("token-classification", model="magistermilitum/roberta-multilingual-medieval-ner")

results = list(map(pipe, list_of_sentences))
results =[[[y["entity"],y["word"], y["start"], y["end"]] for y in x] for x in results]
print(results)

✨ 主要特性

• 多语言支持：支持拉丁语、法语、西班牙语等多种语言，适用于中世纪多语言文本的命名实体识别。
• 高精度识别：在中世纪文本的地点和人物识别任务中，精度达到98.01%，召回率达到97.08%。
• 灵活的输出格式：可以将模型推理结果转换为CONLL格式，方便后续处理。

📦 安装指南

文档未提及安装步骤，故跳过该章节。

💻 使用示例

基础用法

import torch
from transformers import pipeline

pipe = pipeline("token-classification", model="magistermilitum/roberta-multilingual-medieval-ner")

results = list(map(pipe, list_of_sentences))
results =[[[y["entity"],y["word"], y["start"], y["end"]] for y in x] for x in results]
print(results)

高级用法

以下代码展示了如何将模型推理结果转换为CONLL格式：

class TextProcessor:
    def __init__(self, filename):
        self.filename = filename
        self.sent_detector = nltk.data.load("tokenizers/punkt/english.pickle") #sentence tokenizer
        self.sentences = []
        self.new_sentences = []
        self.results = []
        self.new_sentences_token_info = []
        self.new_sentences_bio = []
        self.BIO_TAGS = []
        self.stripped_BIO_TAGS = []

    def read_file(self):
        #Reading a txt file with one document per line.
        with open(self.filename, 'r') as f:
            text = f.read()
        self.sentences = self.sent_detector.tokenize(text.strip())

    def process_sentences(self): #We split long sentences as encoder has a 256 max-lenght. Sentences with les of 40 words will be merged.
        for sentence in self.sentences:
            if len(sentence.split()) < 40 and self.new_sentences:
                self.new_sentences[-1] += " " + sentence
            else:
                self.new_sentences.append(sentence)

    def apply_model(self, pipe):
        self.results = list(map(pipe, self.new_sentences))
        self.results=[[[y["entity"],y["word"], y["start"], y["end"]] for y in x] for x in self.results]

    def tokenize_sentences(self):
        for n_s in self.new_sentences:
            tokens=n_s.split() # Basic tokenization
            token_info = []

            # Initialize a variable to keep track of character index
            char_index = 0
            # Iterate through the tokens and record start and end info
            for token in tokens:
                start = char_index
                end = char_index + len(token)  # Subtract 1 for the last character of the token
                token_info.append((token, start, end))

                char_index += len(token) + 1  # Add 1 for the whitespace
            self.new_sentences_token_info.append(token_info)

    def process_results(self): #merge subwords and BIO tags
        for result in self.results:
            merged_bio_result = []
            current_word = ""
            current_label = None
            current_start = None
            current_end = None
            for entity, subword, start, end in result:
                if subword.startswith("▁"):
                    subword = subword[1:]
                    merged_bio_result.append([current_word, current_label, current_start, current_end])
                    current_word = "" ; current_label = None ; current_start = None ; current_end = None
                if current_start is None:
                    current_word = subword ; current_label = entity ; current_start = start+1 ; current_end= end
                else:
                    current_word += subword ; current_end = end
            if current_word:
                merged_bio_result.append([current_word, current_label, current_start, current_end])
            self.new_sentences_bio.append(merged_bio_result[1:])

    def match_tokens_with_entities(self): #match BIO tags with tokens
        for i,ss in enumerate(self.new_sentences_token_info):
            for word in ss:
                for ent in self.new_sentences_bio[i]:
                    if word[1]==ent[2]:
                        if ent[1]=="L-PERS":
                            self.BIO_TAGS.append([word[0], "I-PERS", "B-LOC"])
                            break
                        else:
                            if "LOC" in ent[1]:
                                self.BIO_TAGS.append([word[0], "O", ent[1]])
                            else:
                                self.BIO_TAGS.append([word[0], ent[1], "O"])
                            break
                else:
                    self.BIO_TAGS.append([word[0], "O", "O"])

    def separate_dots_and_comma(self): #optional
        signs=[",", ";", ":", "."]
        for bio in self.BIO_TAGS:
            if any(bio[0][-1]==sign for sign in signs) and len(bio[0])>1:
                self.stripped_BIO_TAGS.append([bio[0][:-1], bio[1], bio[2]]); 
                self.stripped_BIO_TAGS.append([bio[0][-1], "O", "O"])
            else:
                self.stripped_BIO_TAGS.append(bio)

    def save_BIO(self):
        with open('output_BIO_a.txt', 'w', encoding='utf-8') as output_file:
            output_file.write("TOKEN\tPERS\tLOCS\n"+"\n".join(["\t".join(x) for x in self.stripped_BIO_TAGS]))

# Usage:
processor = TextProcessor('my_docs_file.txt')
processor.read_file()
processor.process_sentences()
processor.apply_model(pipe)
processor.tokenize_sentences()
processor.process_results()
processor.match_tokens_with_entities()
processor.separate_dots_and_comma()
processor.save_BIO()

直接使用示例

对于句子 "Ego Radulfus de Francorvilla miles, notum facio tam presentibus cum futuris quod, cum Guillelmo Bateste militi de Miliaco"，模型将以BIO格式进行标注：

('Ego', 'O', 'O')
('Radulfus', 'B-PERS')
('de', 'I-PERS', 'O')
('Francorvilla', 'I-PERS', 'B-LOC')
('miles', 'O')
(',', 'O', 'O')
('notum', 'O', 'O')
('facio', 'O', 'O')
('tam', 'O', 'O')
('presentibus', 'O', 'O')
('quam', 'O', 'O')
('futuris', 'O', 'O')
('quod', 'O', 'O')
(',', 'O', 'O')
('cum', 'O', 'O')
('Guillelmo', 'B-PERS', 'O')
('Bateste', 'I-PERS', 'O')
('militi', 'O', 'O')
('de', 'O', 'O')
('Miliaco', 'O', 'B-LOC')

📚 详细文档

模型信息

训练过程

该模型在XML-Roberta-Large的基础上进行了5个epoch的微调，学习率为5e-5，批次大小为16。

BibTeX引用

@inproceedings{aguilar2022multilingual,
  title={Multilingual Named Entity Recognition for Medieval Charters Using Stacked Embeddings and Bert-based Models.},
  author={Aguilar, Sergio Torres},
  booktitle={Proceedings of the second workshop on language technologies for historical and ancient languages},
  pages={119--128},
  year={2022}
}

🔧 技术细节

该模型在XML-Roberta-Large基础上进行微调，训练过程中使用了5e-5的学习率和16的批次大小，经过5个epoch的训练，在中世纪文本的命名实体识别任务中取得了较好的效果，精度达到98.01%，召回率达到97.08%。

📄 许可证

本模型采用CC BY-NC 4.0许可证。

模型卡片联系方式

[sergio.torres@uni.lu]