Curie 7B V1

🚀 Curie-7B-v1

This project demonstrates the potential of fine - tuning English Large Language Models (LLMs) for Polish text generation, achieving remarkable performance with high efficiency.

🚀 Quick Start

The Curie-7B-v1 model is now open - source. You can refer to the following sections for details about its training, performance, and related research.

✨ Features

• Efficient Fine - tuning: By employing Language Adaptive Pre - training (LAPT) on a relatively small dataset, the model achieves excellent performance in Polish text generation and downstream tasks.
• High - Quality Output: It generates Polish text with a low perplexity of 3.02 among decoder - based models and competes closely with the best Polish encoder - decoder models.
• Versatility: Can be used as classifiers, regressors, and AI assistants for various NLP tasks in Polish.

📚 Documentation

Introduction

This research shows the potential of fine - tuning English LLMs for Polish text generation. Using LAPT on a 3.11 GB (276 million Polish tokens) high - quality dataset and subsequent fine - tuning on the KLEJ challenges, the Curie-7B-v1 model performs remarkably. It generates Polish text with the lowest perplexity of 3.02 among decoder - based models and closely rivals the best Polish encoder - decoder models, with a minimal performance gap on 8 out of 9 tasks, using only about 2 - 3% of the typical dataset size.

Language Adaptive Pre - training Dataset

The LAPT phase used the SpeakLeash dataset, focusing on a high - quality extract of approximately 2 GB from the original 1TB of Polish texts.

Hardware and Software Stack

Experiments were carried out on a server with an [NVIDIA RTX A6000 ADA GPU](https://www.nvidia.com/content/dam/en - zz/Solutions/design - visualization/rtx - 6000/proviz - print - rtx6000 - datasheet - web - 2504660.pdf) (48GB VRAM), an AMD Epyc 7742 processor, running Ubuntu with Pytorch 2.0 and CUDA 12.2.

The Adaptive Pre - training

The model was trained with the AdamW optimizer. Training completed in one epoch (106 hours), after which overfitting started to occur.

Hyperparameters

• lora_rank: 32
• lora_dropout: 0.05
• lora_alpha: 16
• warmup_steps: 0.1
• learning_rate: 2.5 x 10^-5
• neftune_noise_alpha: 2
• batch_size: 128
• max_seq_len: 128

Fine - tuning for KLEJ Downstream Tasks

Curie-7B-v1 shows high efficiency and capability in handling Polish NLP tasks, being very close to the best baseline models on 8 out of 9 KLEJ tasks with significantly less data.

Performance Highlights

• NKJP - NER: 93.4
• CDSC - E: 92.2
• CDSC - R: 94.9
• CBD: 49.0 (Room for improvement)
• PolEmo2.0 - IN: 92.7
• PolEmo2.0 - OUT: 80.0
• DYK: 76.2
• PSC: 98.6
• AR: 86.8

Conclusions

The Curie-7B-v1 model matches foundational models on eight downstream tasks with much less data. Its versatility in Polish text generation and transformation into different types of models highlight the effectiveness of the method. This open - source Polish LLM provides a basis for efficient business solutions.

Research Paper

The work and details of this model are described in the research paper Efficient Language Adaptive Pre - training: Extending State - of - the - Art Large Language Models for Polish by Szymon Ruciński.

📄 License

The model is released under the Apache - 2.0 license.