Snakmodel 7b Instruct

🚀 SnakModel

SnakModel is a 7B-parameter Danish language model, offering high - performance text generation capabilities based on Llama 2 architecture.

🚀 Quick Start

Here is a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "NLPnorth/snakmodel-7b-instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Hvor ligger IT Universitet?"
messages = [
    {"role": "system", "content": "Du er Snakmodel, skabt af IT-Universitetet i København. Du er en hjælpsom assistent."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=20
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

✨ Features

• Danish Language Specialization: Specifically designed for the Danish language, trained on a large - scale Danish corpus.
• Instruction - Tuned Variant: The SnakModel - 7B (instruct) version can follow instructions in the [INST] {instruction} [/INST] format.
• Multiple Versions: Available in base and instruction - tuned versions, with intermediate checkpoints.

📚 Documentation

Model Details

SnakModel is a 7B - parameter model specifically designed for the Danish language. This is the instruction - tuned variant: SnakModel - 7B (instruct). Our models build upon [Llama 2](https://huggingface.co/meta - llama/Llama - 2 - 7b - hf), which we continuously pre - train on a diverse collection of Danish corpora comprising 350M documents and 13.6B words, before tuning it on 3.7M Danish instruction - answer pairs.

Model Developers: NLPnorth research unit at the IT University of Copenhagen, Denmark.

Variations: SnakModel comes as an instruction - tuned, and a base version. In addition, each model includes intermediate checkpoints (under model revisions).

Input: Text only, with instructions following the [INST] {instruction} [/INST] template.

Output: Text only.

Model Architecture: SnakModel is an auto - regressive, transformer - based language model. The instruct version uses supervised fine - tuning (SFT) to enable instruction following in Danish.

Model Dates: SnakModel was trained between January 2024 and September 2024.

Intended Use & Limitations

Intended Use Cases: SnakModel is intended for use in Danish. The instruction - tuned variant is intended for assistant - like chat. The instruct variant follows the Llama 2 (chat) instruction template, in which instructions are encapsulated in special tokens, i.e., [INST] {instruction} [/INST].

Limitations: SnakModel variants are fine - tuned on Danish data. As such, the use in other languages falls out - of - scope. While we found SnakModel to be more proficient in Danish than other Llama 2 - based models, it still frequently generates factually incorrect output. Make sure to carefully evaluate and weigh these factors before deploying the model. In addition, make sure to adhere to the original [Llama 2 license agreement](https://huggingface.co/meta - llama/Llama - 2 - 7b - hf/blob/main/LICENSE.txt).

Hardware and Software

Training Factors: SnakModel is trained on private infrastructure with one node, containing four NVIDIA A100 - PCIe 40GB GPUs. The node has an AMD Epyc 7662 128 Core Processor and 1TB of RAM.

Carbon Footprint: Total training time accounted to 8,928 GPU hours, with an average carbon efficiency at 0.122kg CO2eq / kWh. This is equivalent to 272.3kg CO2eq emitted, based on the Machine Learning Impact calculator.

Training Data

Overview: SnakModel was continuously pre - train on a diverse collection of Danish corpora comprising 350M documents and 13.6B words. The instruct version is further tuned on 3.7M Danish instruction - answer pairs.

Data Freshness: The pre - training data has a cutoff of January 2024.

Evaluation Results

🔧 Technical Details

• Model Type: Auto - regressive, transformer - based language model.
• Training Data: Pre - trained on 350M Danish documents (13.6B words), and the instruct version is further tuned on 3.7M Danish instruction - answer pairs.
• Training Hardware: One node with four NVIDIA A100 - PCIe 40GB GPUs, AMD Epyc 7662 128 Core Processor, and 1TB of RAM.

📄 License

This model follows the original [Llama 2 license agreement](https://huggingface.co/meta - llama/Llama - 2 - 7b - hf/blob/main/LICENSE.txt).

📚 Citation

@inproceedings{zhang-etal-2025-snakmodel,
    title = "{SnakModel}: {Lessons} Learned from Training an Open {Danish} Large Language Model",
    author = {Zhang, Mike  and
      M{\"u}ller-Eberstein, Max  and
      Bassignana, Elisa  and
      Goot, Rob van der},
    editor = "Johansson, Richard  and
      Stymne, Sara",
    booktitle = "Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)",
    month = mar,
    year = "2025",
    address = "Tallinn, Estonia",
    publisher = "University of Tartu Library",
    url = "https://aclanthology.org/2025.nodalida-1.80/",
    pages = "812--825",
    ISBN = "978-9908-53-109-0",
    abstract = "We present SnakModel, a Danish large language model (LLM) based on Llama2-7B, which we continuously pre-train on 13.6B Danish words, and further tune on 3.7M Danish instructions. As best practices for creating LLMs for smaller language communities have yet to be established, we examine the effects of early modeling and training decisions on downstream performance throughout the entire training pipeline, including (1) the creation of a strictly curated corpus of Danish text from diverse sources; (2) the language modeling and instruction-tuning training process itself, including the analysis of intermediate training dynamics, and ablations across different hyperparameters; (3) an evaluation on eight language and culturally-specific tasks. Across these experiments SnakModel achieves the highest overall performance, outperforming multiple contemporary Llama2-7B-based models. By making SnakModel, the majority of our pre-training corpus, and the associated code available under open licenses, we hope to foster further research and development in Danish Natural Language Processing, and establish training guidelines for languages with similar resource constraints."
}