Merak 7B V2 GGML

🚀 MERAK-7B-V2 GGML

These GGML format model files are designed for CPU + GPU inference, offering a powerful solution for various applications.

This README is adapted from TheBloke. These files are in GGML format for MERAK-7B-V2.

GGML files support CPU + GPU inference using llama.cpp and are compatible with numerous libraries and UIs, including:

• KoboldCpp: A robust GGML web UI with built - in full GPU acceleration, ideal for storytelling.
• LoLLMS Web UI: A great web UI that enables GPU acceleration via the c_transformers backend.
• LM Studio: A fully - featured local GUI. It supports full GPU acceleration on macOS and also works on Windows without GPU acceleration.
• [text - generation - webui](https://github.com/oobabooga/text - generation - webui): The most popular web UI, though it requires additional steps to enable GPU acceleration via the llama.cpp backend.
• ctransformers: A Python library with LangChain support and an OpenAI - compatible AI server.
• [llama - cpp - python](https://github.com/abetlen/llama - cpp - python): A Python library with an OpenAI - compatible API server.

✨ Features

• Multi - language support: Supports languages such as Indonesian (id) and English (en).
• Diverse quantization methods: Offers both original llama.cpp quant methods and new k - quant methods.
• Efficient performance: Leveraging QLoRA, it can run with 16 GB VRAM.

📦 Installation

Please ensure you have installed the CUDA driver, Python 3.10, and PyTorch 2 in your system. Then, install the following libraries in the terminal:

pip install bitsandbytes==0.39.1
pip install transformers==4.31.0
pip install peft==0.4.0
pip install accelerate==0.20.3
pip install einops==0.6.1 scipy sentencepiece datasets

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM, BitsAndBytesConfig, LlamaTokenizer
from peft import PeftModel, PeftConfig

model_id = "Ichsan2895/Merak-7B-v2"
config = AutoConfig.from_pretrained(model_id)

BNB_CONFIG = BitsAndBytesConfig(load_in_4bit=True,
                                bnb_4bit_compute_dtype=torch.bfloat16,
                                bnb_4bit_use_double_quant=True,
                                bnb_4bit_quant_type="nf4",
    )

model = AutoModelForCausalLM.from_pretrained(model_id,
                                             quantization_config=BNB_CONFIG,
                                             device_map="auto",
                                             trust_remote_code=True)

tokenizer = LlamaTokenizer.from_pretrained(model_id)

def generate_response(question: str) -> str:
  prompt = f"<|prompt|>{question}\n<|answer|>".strip()

  encoding = tokenizer(prompt, return_tensors='pt').to("cuda")
  with torch.inference_mode():
    outputs = model.generate(input_ids=encoding.input_ids,
                             attention_mask=encoding.attention_mask,
                             eos_token_id=tokenizer.pad_token_id,
                             do_sample=False,
                             num_beams=2,
                             temperature=0.3,
                             repetition_penalty=1.2,
                             max_length=200)

    response = tokenizer.decode(outputs[0], skip_special_tokes=True)

    assistant_start = "<|answer|>"
    response_start = response.find(assistant_start)
return response[response_start + len(assistant_start) :].strip()

prompt = "Siapa penulis naskah proklamasi kemerdekaan Indonesia?"
print(generate_response(prompt))

Advanced Usage

For better answers, avoid using BitsandBytes 4 - bit Quantization, though it requires more VRAM.

import torch
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM, BitsAndBytesConfig, LlamaTokenizer
from peft import PeftModel, PeftConfig

model_id = "Ichsan2895/Merak-7B-v2"
config = AutoConfig.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             device_map="auto",
                                             trust_remote_code=True)

tokenizer = LlamaTokenizer.from_pretrained(model_id)

def generate_response(question: str) -> str:
  prompt = f"<|prompt|>{question}\n<|answer|>".strip()

  encoding = tokenizer(prompt, return_tensors='pt').to("cuda")
  with torch.inference_mode():
    outputs = model.generate(input_ids=encoding.input_ids,
                             attention_mask=encoding.attention_mask,
                             eos_token_id=tokenizer.pad_token_id,
                             do_sample=False,
                             num_beams=2,
                             temperature=0.3,
                             repetition_penalty=1.2,
                             max_length=200)

    response = tokenizer.decode(outputs[0], skip_special_tokes=True)

    assistant_start = "<|answer|>"
    response_start = response.find(assistant_start)
return response[response_start + len(assistant_start) :].strip()

prompt = "Siapa penulis naskah proklamasi kemerdekaan Indonesia?"
print(generate_response(prompt))

📚 Documentation

Compatibility

Original llama.cpp quant methods: q4_0, q4_1, q5_0, q5_1, q8_0

These methods are guaranteed to be compatible with any UIs, tools, and libraries released since late May. However, they may be phased out soon as they are largely replaced by the new k - quant methods.

New k - quant methods: q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K

These new quantization methods are compatible with llama.cpp as of June 6th, commit 2d43387. They are also compatible with recent releases of text - generation - webui, KoboldCpp, llama - cpp - python, ctransformers, rustformers, and most other tools. For compatibility with other tools and libraries, please refer to their documentation.

Explanation of the new k - quant methods

Provided files

How to run in text - generation - webui

Further instructions can be found here: [text - generation - webui/docs/llama.cpp - models.md](https://github.com/oobabooga/text - generation - webui/blob/main/docs/llama.cpp - models.md).

Original model card: 6TH PROTOTYPE OF MERAK - 7B - V2!

Merak - 7B is a Large Language Model for the Indonesian language. This model is based on Meta Llama - 2 - 7B - Chat - HF and fine - tuned using some pre - cleaned Indonesian Wikipedia articles.

Leveraging QLoRA (QLora: Efficient Finetuning of Quantized LLMs), Merak - 7B can run with 16 GB VRAM.

Licensed under Creative Commons - By Attribution - Share Alike - Non Commercial (CC - BY - SA - NC 4.0), Merak - 7B empowers AI enthusiasts and researchers alike.

Big thanks to all my friends and communities that helped build our first model. Feel free to ask me about the model and share the news on your social media.

CHANGELOG

• v1: The first Merak - 7B model. We selected and cleaned about 200k ID Wikipedia articles.
• v2: A finetuned version of the first Merak - 7B model. We finetuned it again with the same ID Wikipedia articles, except we changed the prompt - style in the questions.

CITATION

@Paper{arXiv,
  author  = {Touvron, et al},
  title   = {Llama 2: Open Foundation and Fine-Tuned Chat Models},
  journal = {arXiv preprint arXiv:2307.09288},
  year    = {2023}
}

@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"
}

@inproceedings{wolf-etal-2020-transformers,
    title = "Transformers: State-of-the-Art Natural Language Processing",
    author = "Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = oct,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-demos.6",
    pages = "38--45"
}

@article{dettmers2023qlora,
  title   = {QLoRA: Efficient Finetuning of Quantized LLMs},
  author  = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal = {arXiv preprint arXiv:2305.14314},
  year    = {2023}
}

📄 License

The model is licensed under Creative Commons - By Attribution - Share Alike - Non Commercial (CC - BY - SA - NC 4.0).

📋 Information Table