Llammas

🚀 LLammas 🐑

LLammas是針對愛沙尼亞語對Llama - 2 - 7B進行兩階段指令微調後的模型。它解決了在愛沙尼亞語場景下缺乏高效語言模型的問題，為愛沙尼亞語的文本生成等任務提供了有力支持。

🚀 快速開始

模型使用

• 文本生成管道：

from transformers import pipeline
import torch

pipe = pipeline("text-generation", model="tartuNLP/Llammas", torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "user", "content": "Tere!"},
    {"role": "assistant", "content": "Tere! Kas saaksin teid kuidagi aidata?"},
    {"role": "user", "content": "Kuidas alustada kirja kirjutamist?"}
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.6, top_k=50, top_p=0.9)
print(outputs[0]["generated_text"][len(prompt):])

• 對話管道：（適用於transformers==4.36.2版本，新版本可能存在輸出問題）

from transformers import pipeline, Conversation
import torch

pipe = pipeline("conversational", model="tartuNLP/Llammas", torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "user", "content": "Tere!"},
    {"role": "assistant", "content": "Tere! Kas saaksin teid kuidagi aidata?"},
    {"role": "user", "content": "Kuidas alustada kirja kirjutamist?"}
]

conversation = Conversation(messages)
conversation = pipe(conversation)

• 對話格式：

<|user|>
Tere!
<|assistant|>
Tere! Kas saaksin teid kuidagi aidata?</s>
<|user|>
Kuidas alustada kirja kirjutamist?
<|assistant|>
Kirja kirjutamiseks alustage tervitusega, näiteks "Tere!" või "Tere hommikust!". Seejärel tutvustage ennast ja mainige, kellega kirjutate. Kirjeldage oma mõtteid või küsimusi, mida soovite arutada. Lõpetage kiri viisakalt, näiteks "Tänan teid tähelepanu eest!" või "Parimate soovidega!"</s>

✨ 主要特性

• 兩階段微調：
  1. 持續預訓練：使用包含50億個標記的CulturaX數據集進行持續預訓練，其中75%的文檔為愛沙尼亞語，25%為英語（詳見 Llammas-base）。
  2. 指令微調：使用Alpaca - cleaned、Alpaca - est、OASST1排名前1的英語對話、CoT和FLAN - V2（遵循open - instruct，各10000條）、WMT18英愛翻譯開發數據（作為文檔）以及通用MTee驗證英愛保留數據進行指令微調。
• 特定數據集：Alpaca - est 是一個使用 gpt - 3.5 - turbo - 0613 為愛沙尼亞語生成的指令數據集，遵循Alpaca格式。更多詳細信息可參考我們的 論文。

📚 詳細文檔

額外資源

• 論文：https://aclanthology.org/2024.findings-naacl.210/
• 代碼：github.com/TartuNLP/llammas
• 基礎模型：tartuNLP/Llammas-base
• 4位量化GGUF模型：AlbertUnn/LlammasGGUF
• Alpaca - est數據集：github.com/TartuNLP/alpaca-est

📄 許可證

引用信息

@inproceedings{kuulmets-etal-2024-teaching,
    title = "Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer",
    author = "Kuulmets, Hele-Andra  and
      Purason, Taido  and
      Luhtaru, Agnes  and
      Fishel, Mark",
    editor = "Duh, Kevin  and
      Gomez, Helena  and
      Bethard, Steven",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-naacl.210",
    doi = "10.18653/v1/2024.findings-naacl.210",
    pages = "3309--3325",
    abstract = "This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named Llammas, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.",
}

📦 信息表格