Jais 7b Chat

🚀 Jais-7b-chat (Its a double quantized version)

This is a double quantized version of jais-13b-chat by Core42. Its purpose is to enable the model to run on machines with limited GPU resources. For high - quality tasks, it's recommended to use the non - quantized 13b model.

Model creator: Core42

Original model: jais-13b-chat

🚀 Quick Start

You can run this model as a text - generation pipeline task.

📦 Installation

1. Install necessary libraries

pip install -Uq huggingface_hub transformers bitsandbytes xformers accelerate

💻 Usage Examples

Basic Usage

1. Create the pipeline

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, TextStreamer, BitsAndBytesConfig

tokenizer = AutoTokenizer.from_pretrained("erfanvaredi/jais-7b-chat")
model = AutoModelForCausalLM.from_pretrained(
    "erfanvaredi/jais-7b-chat",
    trust_remote_code=True,
    device_map='auto',
)

# Create a pipeline
pipe = pipeline(model=model, tokenizer=tokenizer, task='text-generation')

2. Create prompt

chat = [
    {"role": "user", "content": 'Tell me a funny joke about Large Language Models.'},
]
prompt = pipe.tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

3. Create streamer (Optional)

streamer = TextStreamer(
    tokenizer,
    skip_prompt=True,
    stop_token=[tokenizer.eos_token]
)

4. Ask the model

pipe(
  prompt,
  streamer=streamer,
  max_new_tokens=256,
  temperature=0,
)

🔧 Technical Details

This model has been successfully tested on Google Colab Pro T4 instance.

📄 License

This model is licensed under the Apache - 2.0 license.