Qra 1b

🚀 Qra - Polish Language LLMs

Qra is a series of large language models (LLMs) tailored for the Polish language. It's a collaborative effort between the National Information Processing Institute (OPI) and Gdańsk University of Technology (PG). These models were trained on the infrastructure of PG TASK Computing Center with 21 Nvidia A100 cards. The published Qra models were initialized with the weights of English Llama 2 checkpoints and then further trained on a meticulously cleaned, filtered, and deduplicated Polish text corpus, which amounts to approximately 90 billion tokens. The original corpus mainly consisted of web data, including CommonCrawl dumps and the MADLAD - 400 corpus.

🚀 Quick Start

This section provides a high - level overview of the Qra models. For more detailed information, please refer to the subsequent sections.

✨ Features

• Polish Adaptation: Specifically adapted to the Polish language, making it suitable for Polish - related NLP tasks.
• Modern Optimizations: Trained with many modern optimizations such as torch.compile, adamw_apex_fused optimizer, Flash Attention 2, etc.

📚 Documentation

Preprocessing Pipeline

The preprocessing pipeline included the following steps:

• Text Normalization: Normalized text and removed URLs.
• Length Filtering: Removed documents shorter than 500 characters.
• Heuristic Cleaning: Cleaned sentences in documents using a set of heuristic rules. Removed sentences consisting mostly of non - alphabetical characters and sentences in languages other than Polish and English.
• Quality Classification: Filtered documents using a quality classifier trained on a set of several thousand documents manually labeled as high or low quality. The classifier uses several statistics ("quality signals") like the percentage of Polish words, average word and sentence length, number of word and character duplications, and proportion of different character classes in the text.
• Perplexity Filtering: Filtered documents based on the perplexity value calculated by a lightweight KenLM language model.
• Topic Classification: Assigned the document to one of 18 topical domains using a trained classifier.
• Deduplication: Performed fuzzy deduplication using the MinHash algorithm within each topical domain.

The final distribution of documents by topic is shown in the chart below:

Model details

The models were trained for one epoch on sequences of 4096 tokens. During training, many modern optimizations were used:

• torch.compile
• adamw_apex_fused optimizer
• Flash Attention 2
• Mixed precision (--bf16 and --tf32 options)
• Gradient accumulation
• Fully Sharded Data Parallel (FSDP) with the SHARD_GRAD_OP mode
• Gradient checkpointing (only for the 13B model)

The following table summarizes the Qra - 1B model:

Evaluation

In this section, we compare the perplexity of Qra models on Polish texts with other Polish and English LLMs. Note that perplexity values between different text segmentations are not directly comparable. Therefore, we can draw conclusions based on comparisons only between models using the same tokenizer, such as Qra and the original Llama / TinyLlama.

PolEval - 2018

In 2018, the PolEval competition included a language modeling task, for which training and test sets totaling over 20 million Polish sentences were made available. We used the first 10k sentences from the test set to evaluate modern neural language models. To calculate the perplexity, we used a script from the HuggingFace Evaluate library.

Long documents (2024)

Currently, LLMs support contexts of thousands of tokens. Their practical applications usually also involve processing long documents. Therefore, evaluating perplexity on a sentence - based dataset such as PolEval - 2018 may not be meaningful. Additionally, the PolEval corpus has been publicly available on the internet for the past few years, which raises the possibility that for some models the training sets have been contaminated by this data. For this reason, we have prepared a new collection consisting of long papers published exclusively in 2024, which will allow us to more reliably test the perplexities of the models on new knowledge that was not available to them at the time of training. The corpus consists of 5,000 documents ranging from several hundred to about 20,000 tokens. Half of the set consists of press texts from Polish news portals from February 2024, the other half are scientific articles published since January 2024. Most of the documents exceed the context size of the evaluated models. To calculate perplexity for these documents, we divided them into chunks of size equal to the model's context length with a stride of 512 tokens, following this example.

📄 License

The Qra models are licensed under the Apache 2.0 license.

⚠️ Important Note

Qra are foundation language models trained with causal language modeling objective on a large corpus of texts. They are therefore not intended for conversational or instruction - following purposes, and should be further fine - tuned to be used for such tasks.