Cuckoo C4 Instruct

🚀 Cuckoo 🐦

Cuckoo is a family of extractive question answering models. It's a small (300M) information extraction (IE) model that imitates the next token prediction paradigm of large language models. Cuckoo can use various text resources to enhance itself, especially by leveraging data curated for LLMs. This README provides details about its pre - trained checkpoints, performance, usage examples, and more.

🚀 Quick Start

Quick Experience with Cuckoo in Next Tokens Extraction ⚡

We recommend using the strongest Super Rainbow Cuckoo 🦸🌈🐦🛠️ for zero - shot extraction. You can directly run the cases below in case_next_tokens_extraction.py.

1️⃣ First load the model and the tokenizers

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
import spacy

nlp = spacy.load("en_core_web_sm")

device = torch.device("cuda:0")
path = f"KomeijiForce/Cuckoo-C4-Super-Rainbow"
tokenizer = AutoTokenizer.from_pretrained(path)
tagger = AutoModelForTokenClassification.from_pretrained(path).to(device)

2️⃣ Define the next tokens extraction function

def next_tokens_extraction(text):

    def find_sequences(lst):
        sequences = []
        i = 0
        while i < len(lst):
            if lst[i] == 0:
                start = i
                end = i
                i += 1
                while i < len(lst) and lst[i] == 1:
                    end = i
                    i += 1
                sequences.append((start, end+1))
            else:
                i += 1
        return sequences

    text = " ".join([token.text for token in nlp(text)])

    inputs = tokenizer(text, return_tensors="pt").to(device)
    tag_predictions = tagger(**inputs).logits[0].argmax(-1)

    predictions = [tokenizer.decode(inputs.input_ids[0, seq[0]:seq[1]]).strip() for seq in find_sequences(tag_predictions)]
    
    return predictions

3️⃣ Call the function for extraction!

Case 1: Basic entity and relation understanding

text = "Tom and Jack went to their trip in Paris."

for question in [
    "What is the person mentioned here?",
    "What is the city mentioned here?",
    "Who goes with Tom together?",
    "What do Tom and Jack go to Paris for?",
    "Where does George live in?",
]:
    prompt = f"User:\n\n{text}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(prompt)
    print(question, predictions)

Case 2: Longer context

passage = f'''Ludwig van Beethoven (17 December 1770 – 26 March 1827) was a German composer and pianist. He is one of the most revered figures in the history of Western music; his works rank among the most performed of the classical music repertoire and span the transition from the Classical period to the Romantic era in classical music. His early period, during which he forged his craft, is typically considered to have lasted until 1802. From 1802 to around 1812, his middle period showed an individual development from the styles of Joseph Haydn and Wolfgang Amadeus Mozart, and is sometimes characterised as heroic. During this time, Beethoven began to grow increasingly deaf. In his late period, from 1812 to 1827, he extended his innovations in musical form and expression.'''

for question in [
    "What are the people mentioned here?",
    "What is the job of Beethoven?",
    "How famous is Beethoven?",
    "When did Beethoven's middle period showed an individual development?",
]:
    text = f"User:\n\n{passage}\n\nQuestion: {question}\n\nAssistant:"
    predictions = next_tokens_extraction(text)
    print(question, predictions)

Case 3: Knowledge quiz

for obj in ["grass", "sea", "fire", "night"]:
    text = f"User:\\n\\nChoices:\\nred\\nblue\\ngreen.\\n\\nQuestion: What is the color of the {obj}?\\n\\nAssistant:\\n\\nAnswer:"
    predictions = next_tokens_extraction(text)
    print(obj, predictions)

✨ Features

• Unique IE Paradigm: Cuckoo imitates the next token prediction paradigm of large language models, predicting the next tokens by tagging them in the given input context instead of retrieving from the vocabulary.
• Versatile Pre - training: Open - source checkpoints are available for different pre - training scenarios, including on C4, C4 + supervised fine - tuning datasets, and combinations with multiple NER and QA datasets.
• High Adaptability: It shows excellent performance in few - shot adaptation to various downstream tasks, such as named entity recognition and machine reading comprehension.

📦 Installation

The README does not provide specific installation steps. However, it is assumed that the necessary libraries like transformers, torch, and spacy need to be installed. You can use pip to install them:

pip install transformers torch spacy
python -m spacy download en_core_web_sm

💻 Usage Examples

Few - shot Adaptation

Taking CoNLL2003 as an example, run bash run_downstream.sh conll2003.5shot KomeijiForce/Cuckoo-C4-Rainbow, you will get a fine - tuned model in models/cuckoo-conll2003.5shot. Then you can benchmark the model with the script python eval_conll2003.py, which will show you an F1 performance of around 80.

For machine reading comprehension (SQuAD), run bash run_downstream.sh squad.32shot KomeijiForce/Cuckoo-C4-Rainbow, you will get a fine - tuned model in models/cuckoo-squad.32shot. Then you can benchmark the model with the script python eval_squad.py, which will show you an F1 performance of around 88.

Fly your own Cuckoo

We include the script to transform texts to NTE instances in the file nte_data_collection.py, which takes C4 as an example, the converted results can be checked in cuckoo.c4.example.json. Run the run_cuckoo.sh script to try an example pre - training:

python run_ner.py \
  --model_name_or_path roberta-large \
  --train_file cuckoo.c4.example.json \
  --output_dir models/cuckoo-c4-example \
  --per_device_train_batch_size 4\
  --gradient_accumulation_steps 16\
  --num_train_epochs 1\
  --save_steps 1000\
  --learning_rate 0.00001\
  --do_train \
  --overwrite_output_dir

📚 Documentation

Prompt Templates

Downstream Dataset Creation

For fine - tuning your own task, you need to create a Jsonlines file, each line contains {"words": [...], "ner": [...]}, For example:

{"words": ["I", "am", "John", "Smith", ".", "Person", ":"], "ner": ["O", "O", "B", "I", "O", "O", "O"]}

After building your own downstream dataset, save it into my_downstream.json, and then run the command bash run_downstream.sh my_downstream KomeijiForce/Cuckoo-C4-Rainbow. You will find an adapted Cuckoo in models/cuckoo-my_downstream.

🔧 Technical Details

The Cuckoo family of models are extractive question answering models as described in the paper Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest. It is a small (300M) information extraction (IE) model that imitates the next token prediction paradigm of large language models.

📄 License

The project is licensed under the MIT license.

🐾 Citation

@article{DBLP:journals/corr/abs-2502-11275,
  author       = {Letian Peng and
                  Zilong Wang and
                  Feng Yao and
                  Jingbo Shang},
  title        = {Cuckoo: An {IE} Free Rider Hatched by Massive Nutrition in {LLM}'s Nest},
  journal      = {CoRR},
  volume       = {abs/2502.11275},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2502.11275},
  doi          = {10.48550/arXiv.2502.11275},
  eprinttype   = {arXiv},
  eprint       = {2502.11275},
  timestamp    = {Mon, 17 Feb 2025 19:32:20 +0000},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2502-11275.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

Performance Demonstration 🚀