Mistral 7B Indian Law

🚀 Fine Tuned Mistral-7B for Indian Law

A fine-tuned version of the Mistral 7B model, optimized for Indian law understanding and response generation using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA and LoRA techniques.

✨ Features

• Task: Legal Text Understanding and Generation
• Fine-Tuning Dataset: Custom Indian Law Corpus (jizzu/llama2_indian_law_v3)
• Fine-Tuning Method: PEFT with QLoRA and LoRA
• Perplexity Score: 37.32
• Repository: mistralai/Mistral-7B-v0.1

Out-of-Scope Use

• May struggle with highly ambiguous legal queries or non-Indian legal systems.
• Perplexity suggests potential for improvement with extended training or data.

📦 Installation

Use pip install transformers peft torch, use torch with cuda.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model_name = "ajay-drew/midtral-7b-indian-law"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-7B-v0.1")

# Load fine-tuned weights with PEFT
model = PeftModel.from_pretrained(base_model, model_name)

text = "What is the penalty for using forged document? " # Ask custom questions on Indian Law
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Advanced Usage

To check the perplexity of the model use the below code after you run pip install transformers datasets torch use torch with cuda support for reduced metrics check.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from datasets import load_dataset
import torch

dataset = load_dataset("kshitij230/Indian-Law", split="train")

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model_name = "ajay-drew/Mistral-7B-Indian-Law"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map="auto"
)


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


model.eval()
total_loss = 0
total_tokens = 0

test_texts = dataset['Instruction'][:500]

with torch.no_grad():
    for text in test_texts:
        inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512).to(device)
        outputs = model(**inputs, labels=inputs["input_ids"])
        loss = outputs.loss
        if loss is not None:  # Ensure loss is valid
            total_loss += loss.item() * inputs["input_ids"].size(1)
            total_tokens += inputs["input_ids"].size(1)

if total_tokens > 0:
    perplexity = torch.exp(torch.tensor(total_loss / total_tokens)).item()
    print(f"Perplexity: {perplexity}")
    print(f"Total tokens: {total_tokens}")
    print(f"Total loss: {total_loss}")
else:
    print("Error: No tokens processed. Check dataset or tokenization.")

🔧 Technical Details

Hardware Used

• Hardware Type: NVIDIA GeForce RTX 4050 Laptop GPU
• Hours used: 24:19:47

Model Architecture

• Base Model: Mistral 7B (a transformer-based language model with 7 billion parameters)
• Architecture:
  • Decoder-only transformer with multi-head self-attention layers.
  • 32 layers, 4096 hidden size, and 16 attention heads (inherited from Mistral 7B).
  • Modified with Low-Rank Adaptation (LoRA) layers for efficient fine-tuning.
• Fine-Tuning Approach:
  • PEFT: Parameter-Efficient Fine-Tuning to reduce memory footprint.
  • QLoRA: Quantized LoRA, using 4-bit quantization to adapt weights efficiently.
• Parameters Fine-Tuned: LoRA targets specific weight matrices leaving the base model largely frozen.

Software

• CUDA: PyTorch - 2.6.0+cu126

📚 Documentation

Model Card Contact

• Gmail: drewjay05@gmail.com
• GitHuB: github.com/ajay-drew
• Linkedin: linkedin.com/in/ajay-a-133b1326a/

Framework versions

• PEFT 0.14.0

📄 License

This model is released under the MIT license.