Jais 30b V1

🚀 Jais-30b-v1

Jais-30b-v1 is a pre-trained bilingual large language model with 30 billion parameters, supporting both Arabic and English. It offers high performance in various language tasks and is suitable for research and commercial applications.

🚀 Quick Start

Below is sample code to use the model. Note that the model requires a custom model class, so users must enable trust_remote_code=True while loading the model. Also, note that this code is tested on transformers==4.32.0.

# -*- coding: utf-8 -*-

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_path = "core42/jais-30b-v1"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", trust_remote_code=True)


def get_response(text,tokenizer=tokenizer,model=model):
    input_ids = tokenizer(text, return_tensors="pt").input_ids
    inputs = input_ids.to(device)
    input_len = inputs.shape[-1]
    generate_ids = model.generate(
        inputs,
        top_p=0.9,
        temperature=0.3,
        max_length=200,
        min_length=input_len + 4,
        repetition_penalty=1.2,
        do_sample=True,
    )
    response = tokenizer.batch_decode(
        generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
    )[0]
    return response


text= "عاصمة دولة الإمارات العربية المتحدة ه"
print(get_response(text))

text = "The capital of UAE is"
print(get_response(text))

✨ Features

• Bilingual Support: Optimized for both Arabic and English, enabling seamless communication in these two languages.
• Transformer-based Architecture: Built on a decoder-only (GPT-3) architecture with SwiGLU non-linearity, ensuring high performance.
• ALiBi Position Embeddings: Allows the model to handle long sequences effectively, improving context handling and precision.

📚 Documentation

Model Details

Intended Use

We release the Jais 30B model under a full open source license and welcome all feedback and collaboration opportunities.

Potential Downstream Uses

• Research: Suitable for researchers and developers in the field of natural language processing.
• Commercial Use: Can be used as a base model for further fine-tuning in specific scenarios, such as chat-assistants and customer service.

Target Audiences

• Academics: Those researching Arabic natural language processing.
• Businesses: Companies targeting Arabic-speaking audiences.
• Developers: Those integrating Arabic language capabilities into apps.

Out-of-Scope Use

While Jais-30b is a powerful bilingual model, it has limitations and potential for misuse.

• Malicious Use: Prohibited from generating harmful, misleading, or inappropriate content, including hate speech, misinformation, and illegal activities.
• Sensitive Information: Should not be used to handle or generate personal, confidential, or sensitive information.
• Generalization Across All Languages: Not assumed to have equal proficiency in other languages or dialects.
• High-Stakes Decisions: Should not be used for high-stakes decisions without human oversight.

Bias, Risks, and Limitations

The model is trained on publicly available data curated in part by Inception. Although efforts have been made to reduce bias, it may still exhibit some bias, as with all LLM models.

It is designed as an AI assistant for Arabic and English speakers and may not produce appropriate responses for other languages.

By using Jais, you acknowledge that it may generate incorrect, misleading, or offensive information. The information is not intended as advice, and we are not responsible for its content or consequences. We welcome feedback to improve the model.

Training Details

Training Data

For pre-training Jais-30b, we used a diverse bilingual corpus from the web and other sources, as well as publicly available English and code datasets. Arabic data was collected from multiple sources, including web pages, Wikipedia articles, news articles, Arabic books, and social network content. We augmented the Arabic data by translating high-quality English resources using an in-house machine translation system. Our data acquisition strategy is similar to that of Jais-13b.

Training Procedure

Training was performed on the Condor Galaxy 1 (CG-1) supercomputer platform.

Training Hyperparameters

Evaluation

We conducted a comprehensive evaluation of Jais and benchmarked it against other leading base language models in both English and Arabic. The evaluation criteria included knowledge, reasoning, and susceptibility to misinformation/bias.

Arabic Evaluation Results

English Evaluation Results

Citation

@misc{sengupta2023jais,
      title={Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models}, 
      author={Neha Sengupta and Sunil Kumar Sahu and Bokang Jia and Satheesh Katipomu and Haonan Li and Fajri Koto and Osama Mohammed Afzal and Samta Kamboj and Onkar Pandit and Rahul Pal and Lalit Pradhan and Zain Muhammad Mujahid and Massa Baali and Alham Fikri Aji and Zhengzhong Liu and Andy Hock and Andrew Feldman and Jonathan Lee and Andrew Jackson and Preslav Nakov and Timothy Baldwin and Eric Xing},
      year={2023},
      eprint={2308.16149},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Copyright Inception Institute of Artificial Intelligence Ltd.