Cuckoo C4

🚀 布谷鸟模型（Cuckoo）🐦

布谷鸟（Cuckoo）是一个小型（3亿参数）的信息提取（IE）模型，它模仿大语言模型的下一个标记预测范式。该模型通过在给定的输入上下文中标记来预测下一个标记，而非从词汇表中检索。本仓库包含了论文 Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest 中的模型。

布谷鸟与以往的信息提取预训练模型有很大不同，因为它可以利用任何文本资源来提升自身，尤其是借助为大语言模型精心整理的数据！

目前，我们开源了在以下数据上预训练的布谷鸟模型检查点：

1. 从C4转换而来的1亿个下一个标记提取（NTE）实例。(布谷鸟 - C4 🐦)
2. 布谷鸟 - C4 + 从有监督微调数据集TuluV3转换而来的260万个下一个标记提取（NTE）实例。(布谷鸟 - C4 - 指令 🐦🛠️)
3. 布谷鸟 - C4 - 指令 + MultiNERD、MetaIE、NuNER、MRQA（不包括SQuAD、DROP）。(布谷鸟 - C4 - 彩虹 🌈🐦🛠️)
4. 布谷鸟 - C4 - 彩虹 + 多个命名实体识别（NER）数据集、WizardLM数据集、多项选择问答数据集、MMLU、SQuAD、DROP、MNLI、SNLI。(布谷鸟 - C4 - 超级彩虹 🦸🌈🐦🛠️)

✨ 主要特性

• 创新的预测范式：模仿大语言模型的下一个标记预测范式，通过在输入上下文中标记来预测下一个标记，而非传统的从词汇表中检索方式。
• 数据利用高效：能够利用任何文本资源进行自我提升，特别是可以借助为大语言模型整理的数据，实现数据的高效利用。
• 多场景适应性：在多种信息提取任务中表现出色，如实体识别、关系理解、问答等，具有广泛的应用场景。
• 模型规模小巧：仅有3亿参数，在保证性能的同时，具有较低的计算资源需求和更快的推理速度。

🚀 快速开始

性能展示 🚀

开启布谷鸟模型的探索之旅，体验它在各类信息提取任务中不可思议的适应效率！

快速体验布谷鸟模型的下一个标记提取 ⚡

我们建议使用最强的超级彩虹布谷鸟 🦸🌈🐦🛠️ 进行零样本提取。

1️⃣ 首先加载模型和分词器

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
import spacy

nlp = spacy.load("en_core_web_sm")

device = torch.device("cuda:0")
path = f"KomeijiForce/Cuckoo-C4-Super-Rainbow"
tokenizer = AutoTokenizer.from_pretrained(path)
tagger = AutoModelForTokenClassification.from_pretrained(path).to(device)

2️⃣ 定义下一个标记提取函数

def next_tokens_extraction(text):

    def find_sequences(lst):
        sequences = []
        i = 0
        while i < len(lst):
            if lst[i] == 0:
                start = i
                end = i
                i += 1
                while i < len(lst) and lst[i] == 1:
                    end = i
                    i += 1
                sequences.append((start, end+1))
            else:
                i += 1
        return sequences

    text = " ".join([token.text for token in nlp(text)])

    inputs = tokenizer(text, return_tensors="pt").to(device)
    tag_predictions = tagger(**inputs).logits[0].argmax(-1)

    predictions = [tokenizer.decode(inputs.input_ids[0, seq[0]:seq[1]]).strip() for seq in find_sequences(tag_predictions)]
    
    return predictions

3️⃣ 调用函数进行提取！

案例1：基本实体和关系理解

text = "Tom and Jack went to their trip in Paris."

for question in [
    "What are the people mentioned here?",
    "What is the city mentioned here?",
    "Who goes with Tom together?",
    "What do Tom and Jack go to Paris for?",
    "Which city does George live in?",
]:
    text = f"User:\n\n{text}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(text)
    print(question, predictions)

你将得到类似如下的结果：

What are the people mentioned here? ['Tom', 'Jack']
What is the city mentioned here? ['Paris']
Who goes with Tom together? ['Jack']
What do Tom and Jack go to Paris for? ['trip']
Which city does George live in? []

其中 [] 表示布谷鸟模型认为没有可提取的下一个标记。

案例2：更长的上下文

passage = f'''Ludwig van Beethoven (17 December 1770 – 26 March 1827) was a German composer and pianist. He is one of the most revered figures in the history of Western music; his works rank among the most performed of the classical music repertoire and span the transition from the Classical period to the Romantic era in classical music. His early period, during which he forged his craft, is typically considered to have lasted until 1802. From 1802 to around 1812, his middle period showed an individual development from the styles of Joseph Haydn and Wolfgang Amadeus Mozart, and is sometimes characterised as heroic. During this time, Beethoven began to grow increasingly deaf. In his late period, from 1812 to 1827, he extended his innovations in musical form and expression.'''

for question in [
    "What are the people mentioned here?",
    "What is the job of Beethoven?",
    "How famous is Beethoven?",
    "When did Beethoven's middle period showed an individual development?",
]:
    text = f"User:\n\n{passage}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(text)
    print(question, predictions)

你将得到类似如下的结果：

What are the people mentioned here? ['Ludwig van Beethoven', 'Joseph Haydn', 'Wolfgang Amadeus Mozart']
What is the job of Beethoven? ['composer and pianist']
How famous is Beethoven? ['one of the most revered figures in the history of Western music']
When did Beethoven's middle period showed an individual development? ['1802']

案例3：知识问答

for obj in ["grass", "sea", "fire", "night"]:
    text = f"User:\n\nChoices:\nred\nblue\ngreen.\n\nQuestion: What is the color of the {obj}?\n\nAssistant:\n\nAnswer:"
    predictions = next_tokens_extraction(text)
    print(obj, predictions)

你将得到类似如下的结果：

grass ['green']
sea ['blue']
fire ['red']
night []

这表明布谷鸟模型并非简单地提取可能的文本片段，而是具备理解上下文的知识。

📚 详细文档

文件信息

仓库包含以下文件信息：

special_tokens_map.json

{
  "bos_token": {
    "content": "<s>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "cls_token": {
    "content": "<s>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "eos_token": {
    "content": "</s>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "mask_token": {
    "content": "<mask>",
    "lstrip": true,
    "normalized": false,
    "rstrip": false,
    "single_word": false
  },
  "pad_token": {
    "content": "<pad>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "sep_token": {
    "content": "</s>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "unk_token": {
    "content": "<unk>",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  }
}

tokenizer_config.json

{
  "add_prefix_space": true,
  "added_tokens_decoder": {
    "0": {
      "content": "<s>",
      "lstrip": false,
      "normalized": true,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "1": {
      "content": "<pad>",
      "lstrip": false,
      "normalized": true,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "2": {
      "content": "</s>",
      "lstrip": false,
      "normalized": true,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "3": {
      "content": "<unk>",
      "lstrip": false,
      "normalized": true,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "50264": {
      "content": "<mask>",
      "lstrip": true,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    }
  },
  "bos_token": "<s>",
  "clean_up_tokenization_spaces": false,
  "cls_token": "<s>",
  "eos_token": "</s>",
  "errors": "replace",
  "mask_token": "<mask>",
  "max_length": 512,
  "model_max_length": 512,
  "pad_token": "<pad>",
  "sep_token": "</s>",
  "stride": 0,
  "tokenizer_class": "RobertaTokenizer",
  "trim_offsets": true,
  "truncation_side": "right",
  "truncation_strategy": "longest_first",
  "unk_token": "<unk>"
}

merges.txt

内容："文件内容超过50 KB，过长无法显示。"

vocab.json

内容："文件内容超过50 KB，过长无法显示。"

config.json

{
  "_name_or_path": "models/ptr-large-c4-stage9",
  "architectures": [
    "RobertaForTokenClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "classifier_dropout": null,
  "eos_token_id": 2,
  "finetuning_task": "ner",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "id2label": {
    "0": "B",
    "1": "I",
    "2": "O"
  },
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "label2id": {
    "B": 0,
    "I": 1,
    "O": 2
  },
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "torch_dtype": "float32",
  "transformers_version": "4.45.2",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}

tokenizer.json

内容："文件内容超过50 KB，过长无法显示。"

📄 许可证

本项目采用MIT许可证。