PULI LlumiX 32K

🚀 PULI LlumiX 32K base (6.74B billion parameter)

This is a model with 6.74 billion parameters. It offers extended context length and is trained on multiple datasets. For more details or to test the instruct model, visit the provided demo site.

🚀 Quick Start

For further details or testing our instruct model, see our demo site.

✨ Features

• Trained with OpenChatKit github.
• The LLaMA-2-7B-32K model was continuously pretrained on a Hungarian dataset.
• The model has been extended to a context length of 32K with position interpolation.
• Checkpoint: 100,000 steps.

📦 Installation

No installation steps provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import pipeline, LlamaForCausalLM, LlamaTokenizer

model = LlamaForCausalLM.from_pretrained("NYTK/PULI-LlumiX-32K")
tokenizer = LlamaTokenizer.from_pretrained("NYTK/PULI-LlumiX-32K")
prompt = "Elmesélek egy történetet a nyelvtechnológiáról."
generator = pipeline(task="text-generation", model=model, tokenizer=tokenizer)

print(generator(prompt, max_new_tokens=30)[0]["generated_text"])

Advanced Usage

No advanced usage code provided in the original document, so this part is skipped.

📚 Documentation

Dataset for continued pretraining

• Hungarian: 7.9 billion words, documents (763K) that exceed 5000 words in length.
• English: Long Context QA (2 billion words), BookSum (78 million words).

Limitations

• max_seq_length = 32,768
• float16
• vocab size: 32,000

🔧 Technical Details

No detailed technical implementation details provided in the original document, so this section is skipped.

📄 License

• License: llama2