Cuckoo C4 Rainbow

🚀 Cuckoo 🐦

Cuckoo is a small (300M) information extraction (IE) model that imitates the next token prediction paradigm of large language models, offering high adaptation efficiency for various IE tasks.

🚀 Quick Start

Cuckoo is an innovative information extraction model. It can use any text resource to enhance itself, especially by leveraging data curated for LLMs. You can start using Cuckoo through the following steps:

Load the Model and Tokenizers

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
import spacy

nlp = spacy.load("en_core_web_sm")

device = torch.device("cuda:0")
path = f"KomeijiForce/Cuckoo-C4-Super-Rainbow"
tokenizer = AutoTokenizer.from_pretrained(path)
tagger = AutoModelForTokenClassification.from_pretrained(path).to(device)

Define the Next Tokens Extraction Function

def next_tokens_extraction(text):

    def find_sequences(lst):
        sequences = []
        i = 0
        while i < len(lst):
            if lst[i] == 0:
                start = i
                end = i
                i += 1
                while i < len(lst) and lst[i] == 1:
                    end = i
                    i += 1
                sequences.append((start, end+1))
            else:
                i += 1
        return sequences

    text = " ".join([token.text for token in nlp(text)])

    inputs = tokenizer(text, return_tensors="pt").to(device)
    tag_predictions = tagger(**inputs).logits[0].argmax(-1)

    predictions = [tokenizer.decode(inputs.input_ids[0, seq[0]:seq[1]]).strip() for seq in find_sequences(tag_predictions)]
    
    return predictions

Call the Function for Extraction

text = "Tom and Jack went to their trip in Paris."

for question in [
    "What is the person mentioned here?",
    "What is the city mentioned here?",
    "Who goes with Tom together?",
    "What do Tom and Jack go to Paris for?",
    "Where does George live in?",
]:
    prompt = f"User:\n\n{text}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(prompt)
    print(question, predictions)

✨ Features

• Versatile Pre - training: Cuckoo can be pre - trained on various datasets, including C4, TuluV3, and multiple NER and QA datasets.
• High Adaptation Efficiency: It shows excellent performance in few - shot adaptation to different IE tasks.
• Knowledge Understanding: Cuckoo can understand the context and extract relevant information even in knowledge quiz scenarios.

💻 Usage Examples

Basic Usage

text = "Tom and Jack went to their trip in Paris."

for question in [
    "What is the person mentioned here?",
    "What is the city mentioned here?",
    "Who goes with Tom together?",
    "What do Tom and Jack go to Paris for?",
    "Where does George live in?",
]:
    prompt = f"User:\n\n{text}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(prompt)
    print(question, predictions)

Advanced Usage

passage = f'''Ludwig van Beethoven (17 December 1770 – 26 March 1827) was a German composer and pianist. He is one of the most revered figures in the history of Western music; his works rank among the most performed of the classical music repertoire and span the transition from the Classical period to the Romantic era in classical music. His early period, during which he forged his craft, is typically considered to have lasted until 1802. From 1802 to around 1812, his middle period showed an individual development from the styles of Joseph Haydn and Wolfgang Amadeus Mozart, and is sometimes characterised as heroic. During this time, Beethoven began to grow increasingly deaf. In his late period, from 1812 to 1827, he extended his innovations in musical form and expression.'''

for question in [
    "What are the people mentioned here?",
    "What is the job of Beethoven?",
    "How famous is Beethoven?",
    "When did Beethoven's middle period showed an individual development?",
]:
    text = f"User:\n\n{passage}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(text)
    print(question, predictions)

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Few - shot Adaptation

Cuckoo is highly adaptable to few - shot tasks. For example, to adapt to CoNLL2003:

bash run_downstream.sh conll2003.5shot KomeijiForce/Cuckoo-C4-Rainbow

The fine - tuned model will be saved in models/cuckoo-conll2003.5shot. You can then benchmark it using:

python eval_conll2003.py

To adapt to SQuAD:

bash run_downstream.sh squad.32shot KomeijiForce/Cuckoo-C4-Rainbow

The fine - tuned model will be in models/cuckoo-squad.32shot, and you can benchmark it with:

python eval_squad.py

Build Your Own Downstream Dataset

Create a Jsonlines file, each line containing {"words": [...], "ner": [...]}, for example:

{"words": ["I", "am", "John", "Smith", ".", "Person", ":"], "ner": ["O", "O", "B", "I", "O", "O", "O"]}

Save your dataset as my_downstream.json and run:

bash run_downstream.sh my_downstream KomeijiForce/Cuckoo-C4-Rainbow

The adapted model will be in models/cuckoo-my_downstream.

Fly Your Own Cuckoo

Use the script in nte_data_collection.py to transform texts to NTE instances. For example, with C4:

python run_ner.py \
  --model_name_or_path roberta-large \
  --train_file cuckoo.c4.example.json \
  --output_dir models/cuckoo-c4-example \
  --per_device_train_batch_size 4\
  --gradient_accumulation_steps 16\
  --num_train_epochs 1\
  --save_steps 1000\
  --learning_rate 0.00001\
  --do_train \
  --overwrite_output_dir

🔧 Technical Details

Cuckoo imitates the next token prediction paradigm of large language models. Instead of retrieving from the vocabulary, it predicts the next tokens by tagging them in the given input context. It can use any text resource to enhance itself, especially by leveraging data curated for LLMs.

📄 License

This project is licensed under the Apache - 2.0 license.

🐾 Citation

@article{DBLP:journals/corr/abs-2502-11275,
  author       = {Letian Peng and
                  Zilong Wang and
                  Feng Yao and
                  Jingbo Shang},
  title        = {Cuckoo: An {IE} Free Rider Hatched by Massive Nutrition in {LLM}'s Nest},
  journal      = {CoRR},
  volume       = {abs/2502.11275},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2502.11275},
  doi          = {10.48550/arXiv.2502.11275},
  eprinttype   = {arXiv},
  eprint       = {2502.11275},
  timestamp    = {Mon, 17 Feb 2025 19:32:20 +0000},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2502-11275.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}