Mistrallite

🚀 MistralLite Model

MistralLite is a fine - tuned Mistral - 7B - v0.1 language model. It enhances the ability to process long contexts (up to 32K tokens). By adapting Rotary Embedding and using a sliding window during fine - tuning, MistralLite significantly outperforms in long - context retrieval and answering tasks while maintaining the simple structure of the original model. It is suitable for various applications like long - context line and topic retrieval, summarization, and question - answering. MistralLite can be deployed on a single AWS g5.2x instance with Sagemaker Huggingface Text Generation Inference (TGI) endpoint, making it ideal for high - performance applications in resource - constrained environments. It also supports other serving methods such as [vLLM](https://github.com/vllm - project/vllm) and can be used in Python with HuggingFace transformers and [FlashAttention - 2](https://github.com/Dao - AILab/flash - attention) libraries.

✨ Features

MistralLite is similar to Mistral - 7B - Instruct - v0.1. Here are their similarities and differences:

⚠️ Important Note

Use the following prompt template for MistralLite: <|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>

📚 Documentation

Motivation of Developing MistralLite

Since the release of Mistral - 7B - Instruct - v0.1, it has become increasingly popular due to its strong performance on a wide range of benchmarks. However, most benchmarks are evaluated on short context, and its performance on long - context tasks has not been well - investigated. We evaluated Mistral - 7B - Instruct - v0.1 against benchmarks designed for assessing LLMs' long - context handling capabilities. Although it was fairly competitive on long contexts less than 4096 tokens, there were limitations on longer contexts. To improve its long - context performance, we fine - tuned the Mistral 7B model and produced Mistrallite, which significantly boosts the long - context handling performance compared to Mistral - 7B - Instruct - v0.1. The detailed long context evaluation results are as follows:

1. [Topic Retrieval](https://lmsys.org/blog/2023 - 06 - 29 - longchat/) | Model Name | Input length | Input length | Input length | Input length | Input length | |----------|-------------:|-------------:|------------:|-----------:|-----------:| | | 2851 | 5568 | 8313 | 11044 | 13780 | | Mistral - 7B - Instruct - v0.1 | 100% | 50% | 2% | 0% | 0% | | MistralLite | 100% | 100% | 100% | 100% | 98% |

2. [Line Retrieval](https://lmsys.org/blog/2023 - 06 - 29 - longchat/#longeval - results) | Model Name | Input length | Input length | Input length | Input length | Input length | Input length | |----------|-------------:|-------------:|------------:|-----------:|-----------:|-----------:| | | 3818 | 5661 | 7505 | 9354 | 11188 | 12657 | | Mistral - 7B - Instruct - v0.1 | 98% | 62% | 42% | 42% | 32% | 30% | | MistralLite | 98% | 92% | 88% | 76% | 70% | 60% |

3. [Pass key Retrieval](https://github.com/epfml/landmark - attention/blob/main/llama/run_test.py#L101) | Model Name | Input length | Input length | Input length | Input length | |----------|-------------:|-------------:|------------:|-----------:| | | 3264 | 5396 | 8329 | 10197 | | Mistral - 7B - Instruct - v0.1 | 100% | 50% | 20% | 30% | | MistralLite | 100% | 100% | 100% | 100% |

4. [Question Answering with Long Input Texts](https://nyu - mll.github.io/quality/) | Model Name | Test set Accuracy | Hard subset Accuracy | |----------|-------------:|-------------:| | Mistral - 7B - Instruct - v0.1 | 44.3% | 39.7% | | MistralLite | 64.4% | 56.2% |

Example test on long context of 13400 tokens

• Context is from Amazon Aurora FAQs
• Question: please tell me how does pgvector help with Generative AI and give me some examples.
• Answer from MistralLite:

pgvector is an open - source extension for PostgreSQL supported by Amazon Aurora PostgreSQL - Compatible Edition.

You can use pgvector to store, search, index, and query billions of embeddings that are generated from machine learning (ML) and artificial intelligence (AI) models in your database, such as those from Amazon Bedrock (limited preview) or Amazon SageMaker. A vector embedding is a numerical representation that represents the semantic meaning of content such as text, images, and video.

With pgvector, you can query embeddings in your Aurora PostgreSQL database to perform efficient semantic similarity searches of these data types, represented as vectors, combined with other tabular data in Aurora. This enables the use of generative AI and other AI/ML systems for new types of applications such as personalized recommendations based on similar text descriptions or images, candidate match based on interview notes, customer service next best action recommendations based on successful transcripts or chat session dialogs, and more.

Model Details

MistralLite LM - Eval Results

Methodology

• Please see https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
• revision = 4ececff
• Note: we used --model hf - causal - experimental instead of --model hf - causal

Results

💻 Usage Examples

How to Use MistralLite from Python Code (HuggingFace transformers)

⚠️ Important Note

For an end - to - end example Jupyter notebook, please refer to [this link](https://github.com/awslabs/extending - the - context - length - of - open - source - llms/blob/main/MistralLite/huggingface - transformers/example_usage.ipynb).

Install the necessary packages

Requires: transformers 4.34.0 or later, [flash - attn](https://pypi.org/project/flash - attn/) 2.3.1.post1 or later, and accelerate 0.23.0 or later.

pip install transformers==4.34.0
pip install flash - attn==2.3.1.post1 --no - build - isolation
pip install accelerate==0.23.0

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import transformers
import torch

model_id = "amazon/MistralLite"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             torch_dtype=torch.bfloat16,
                                             use_flash_attention_2=True,
                                             device_map="auto",)
pipeline = transformers.pipeline(
    "text - generation",
    model=model,
    tokenizer=tokenizer,
)
prompt = "<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>"

sequences = pipeline(
    prompt,
    max_new_tokens=400,
    do_sample=False,
    return_full_text=False,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"{seq['generated_text']}")

⚠️ Important Note

Use the following prompt template for MistralLite:

<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>

How to Serve MistralLite on TGI

⚠️ Important Note

• For an end - to - end example Jupyter notebook using the native TGI container, please refer to [this link](https://github.com/awslabs/extending - the - context - length - of - open - source - llms/blob/main/MistralLite/tgi/example_usage.ipynb).
• If the input context length is greater than 12K tokens, it is recommended using a custom TGI container, please refer to [this link](https://github.com/awslabs/extending - the - context - length - of - open - source - llms/blob/main/MistralLite/tgi - custom/example_usage.ipynb).

Start TGI server

Use TGI version 1.1.0 or later. The official Docker container is: ghcr.io/huggingface/text - generation - inference:1.1.0

Example Docker parameters:

docker run -d --gpus all --shm - size 1g -p 443:80 -v $(pwd)/models:/data ghcr.io/huggingface/text - generation - inference:1.1.0 \
      --model - id amazon/MistralLite \
      --max - input - length 16000 \
      --max - total - tokens 16384 \
      --max - batch - prefill - tokens 16384 \
      --trust - remote - code

Advanced Usage

Example Python code for inference with TGI (requires text_generation 0.6.1 or later):

pip install text_generation==0.6.1

from text_generation import Client

SERVER_PORT = 443
SERVER_HOST = "localhost"
SERVER_URL = f"{SERVER_HOST}:{SERVER_PORT}"
tgi_client = Client(f"http://{SERVER_URL}", timeout=60)

def invoke_tgi(prompt, 
                      random_seed=1, 
                      max_new_tokens=400, 
                      print_stream=True,
                      assist_role=True):
    if (assist_role):
        prompt = f"<|prompter|>{prompt}</s><|assistant|>"
    output = ""
    for response in tgi_client.generate_stream(
        prompt,
        do_sample=False,
        max_new_tokens=max_new_tokens,
        return_full_text=False,
        #temperature=None,
        #truncate=None,
        #seed=random_seed,
        #typical_p=0.2,
    ):
        if hasattr(response, "token"):
            if not response.token.special:
                snippet = response.token.text
                output += snippet
                if (print_stream):
                    print(snippet, end='', flush=True)
    return output

prompt = "What are the main challenges to support a long context for LLM?"
result = invoke_tgi(prompt)

⚠️ Important Note

When using MistralLite for inference for the first time, it may require a brief 'warm - up' period that can take 10s of seconds. However, subsequent inferences should be faster and return results in a more timely manner. This warm - up period is normal and should not affect the overall performance of the system once the initialisation period has been completed.

How to Deploy MistralLite on Amazon SageMaker

⚠️ Important Note

• For an end - to - end example Jupyter notebook using the SageMaker built - in container, please refer to [this link](https://github.com/awslabs/extending - the - context - length - of - open - source - llms/blob/main/MistralLite/sagemaker - tgi/example_usage.ipynb).
• If the input context length is greater than 12K tokens, it is recommended using a custom docker container, please refer to [this link](https://github.com/awslabs/extending - the - context - length - of - open - source - llms/blob/main/MistralLite/sagemaker - tgi - custom/example_usage.ipynb).

Install the necessary packages

Requires...

📄 License

This project is licensed under the Apache 2.0 license.