Prosparse Llama 2 7b

🚀 ProSparse-LLaMA-2-7B

ProSparse-LLaMA-2-7B is a fine - tuned large language model that uses the "ProSparse" method to achieve high activation sparsity while maintaining comparable performance to the original model, and it shows significant inference acceleration effects.

✨ Features

• High Activation Sparsity: By applying the "ProSparse" method, it achieves high activation sparsity, such as 89.32% for ProSparse - 7B, which is higher than many existing models.
• Comparable Performance: Maintains comparable performance to the original Swish - activated LLaMA2 models on various benchmarks.
• Inference Acceleration: Demonstrates practical speed - up effects on both the PowerInfer framework and custom sparse GPU operators.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

The README does not provide code examples for using the model, so this section is skipped.

📚 Documentation

Introduction

The utilization of activation sparsity is a promising method for inference acceleration of large language models (LLMs). Most recent mainstream LLMs use non - sparse activation functions. In this work, the "ProSparse" method is introduced to push LLMs for higher activation sparsity while maintaining comparable performance. By applying it to different models, high - sparsity ReLU - activated models are obtained, and their performance is comparable to the original versions.

Training Dataset

The 7B model is trained on about 34.60 billion tokens within 16,500 steps. The training data consists of two categories:

• Language modeling datasets: StarCoder, Wikipedia, Pile, and other collected datasets.
• Instruction tuning datasets: UltraChat, P3 (multiple - choice QA), PAQ, Unnatural Instructions, Flan, Super - Natural Instructions, and other collected datasets.

ProSparse: Training Methodology

The training process of ProSparse consists of three steps:

1. Activation Function Substitution: Substitute the activation function of FFNs with ReLU and apply continual training.
2. Progressive Sparsity Regularization: Jointly optimize the model on the conventional next - token prediction loss and (L_1) regularization loss. The regularization factor (\lambda) increases progressively in multiple stages.
3. Activation Threshold Shifting: Replace ReLU with FATReLU, a ReLU variant with a positive threshold, to further boost sparsity.

The 7B model is trained on 8 A100 GPUs. The learning rate is controlled by a cosine scheduler with a peak LR of (3e - 5). The hyper - parameters for each stage are as follows:

Evaluation Results

The evaluation results on various benchmarks show the advantage of ProSparse. It is the only method achieving high sparsity and comparable performance to the original Swish - activated LLaMA2. The evaluation is based on the framework UltraEval.

Evaluation Issues with LM - Eval

The results can be replicated with UltraEval. Abnormal results with [LM - Eval](https://github.com/EleutherAI/lm - evaluation - harness) may be due to the absence of the cls token <s>. A quick fix is shown in the following code:

# https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py#L945
for _, context_enc, continuation_enc in chunk:
    # sanity check
    assert len(context_enc) > 0
    # Note: a trivial fix here
    if context_enc[0] != 1:
        context_enc = [1] + context_enc
    assert len(continuation_enc) > 0
    assert len(continuation_enc) <= self.max_length

Adapting vLLM to ProSparse LLaMA models

1. Replace the file vllm/model_executor/models/llama.py in original vLLM with this [file](https://github.com/Raincleared - Song/DejaVu_predictor/blob/main/llama.py).
2. Replace the contents of the original [config.json](https://huggingface.co/SparseLLM/prosparse - llama - 2 - 7b/blob/main/config.json) with this [file](https://github.com/Raincleared - Song/DejaVu_predictor/blob/main/config.json).
3. Set the environment variable ACT_INFO. To test the version without activation threshold shifting, export ACT_INFO = relu. To test the version with activation threshold shifting, export ACT_INFO = fatrelu_0.01.

Inference Acceleration Effects

• PowerInfer: Utilize PowerInfer, a state - of - the - art acceleration framework. Report activation recall, predicted sparsity, and the number of tokens generated per second. The GGUF files and activation predictors for ProSparse - 7B are available at [ProSparse - LLaMA - 2 - 7B - GGUF](https://huggingface.co/PowerInfer/prosparse - llama - 2 - 7b - gguf) and [ProSparse - LLaMA - 2 - 7B - Predictor](https://huggingface.co/PowerInfer/prosparse - llama - 2 - 7b - predictor).
• Sparse GPU Operators: Implement two sparse GPU [operators](https://github.com/Raincleared - Song/sparse_gpu_operator) for faster accurate inference. They are responsible for the speedup of two key steps in a gated FFN.

🔧 Technical Details

The README provides detailed technical information about the ProSparse method, including the training process (activation function substitution, progressive sparsity regularization, activation threshold shifting), hyper - parameters, and the implementation of sparse GPU operators.

📄 License

The model uses the llama2 license.