Llama Xlam 2 8b Fc R GGUF

🚀 Experimental layer-wise quantization of Salesforce/Llama-xLAM-2-8b-fc-r

This project focuses on the experimental layer-wise quantization of the Salesforce/Llama-xLAM-2-8b-fc-r model, aiming to optimize the inference performance of large language models in resource-constrained environments.

Model Information

🚀 Quick Start

The experimental versions of the model are generated using a custom quantization method. Here is a high - level overview of the process:

1. Convert the original model's tensors to GGUF F16*.
2. Estimate the Perplexity score for the F16 model (baseline) using the wikitext-2-raw-v1 dataset, and save the logits.
3. Generate an imatrix from selected calibration datasets.
4. Determine tensor and layer Importance Score contribution using the modified version of llama-imatrix.
5. Select an appropriate quant level for each tensor and quantize the model using llama-quantize.
6. Calculate Perplexity, KL Divergence, ARC (Easy+Challenge), HellaSwag, MMLU, Truthful QA and WinoGrande scores for each quantized model.
7. Keep versions with the best scores.
8. Repeat until all desired quants are created.

*BF16 would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16

✨ Features

• Layer-wise Quantization: Inspired by Dumitru's et al Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels, different quantization types are applied to different tensors/layers.
• Optimized Inference: Aims to improve the inference performance of LLMs in resource - constrained environments.
• Custom Tools: A custom version of llama-imatrix and llama-quantize is used to identify influential tensors and perform quantization.

📚 Documentation

Original Model Introduction

Large Action Models (LAMs) are advanced language models designed to enhance decision - making by translating user intentions into executable actions. As the brains of AI agents, LAMs autonomously plan and execute tasks to achieve specific goals, making them invaluable for automating workflows across diverse domains.

This model release is for research purposes only.

The new xLAM - 2 series, built on our most advanced data synthesis, processing, and training pipelines, marks a significant leap in multi - turn conversation and tool usage. Trained using our novel APIGen - MT framework, which generates high - quality training data through simulated agent - human interactions. Our models achieve state - of - the - art performance on BFCL and τ - bench benchmarks, outperforming frontier models like GPT - 4o and Claude 3.5. Notably, even our smaller models demonstrate superior capabilities in multi - turn scenarios while maintaining exceptional consistency across trials.

Experimental Version Production

An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource - constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. There are many approaches to accomplish this, including architecture simplification and knowledge distillation, but the focus has been primarily on quantization and pruning.

The method used to produce these experimental versions is covered in Squeezing Tensor Bits: the quest for smaller LLMs. At a high level, it involves using a custom version of llama-imatrix and llama-quantize to identify influential tensors, and quantize the most important layers to higher bit precision and the less important to lower bits.

As of version b5125 llama - quantize can now perform tensor - wide quantization (TWQ), whereby user - defined tensors are quantized at a specific level, or perform layer - wise quantization (LWQ) by selecting different quantization types per tensor/layer.

The modified version of llama - imatrix generates useful statistics to guide the tensor selection process. --show - statistics will display:

• Œ£(Bias): the sum of all activations over the tensor (i.e. the Importance Scores)
• Min & Max: minimum and maximum activation values
• Œº & œÉ: activations' mean and standard deviation
• % Active: proportion of elements whose average activation exceeds a very small threshold (1e - 6). Helpful to determine how alive/dormant the tensor is during inference
• N: number of activations in the tensor
• Entropy: entropy of the activation distribution, in bits (standard Shannon entropy measurement)
• E (norm): Normalized entropy.
• ZD Score: z - score distribution as described in 3.1 Layer Importance Scores in the Layer - Wise Quantization paper
• CosSim: cosine similarity between same type tensors with respect to the previous layer (i.e. blk.7.attn_k and blk.6.attn_k)

Please note that statistics are calculated for each individual tensor and should be used to compare between tensors of the same type only.

There's a pull request to merge these changes back into the core llama.cpp project. This may or may not ever happen so, until then, the modified version will be available on GitHub.

Testing and Comparison

For testing and comparison, normally models produced by Unsloth and Bartowski are used. But they don't provide GGUF versions of this model, so all tests and comparisons are done against naive quantizations obtained by simply running llama - quantize with no further optimization.

All experimental versions were generated using an appropriate imatrix created from calibration datasets available at eaddario/imatrix - calibration.

🔧 Technical Details

Model Sizes

Perplexity and KL Divergence scores

ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores

Scores generated using llama - perplexity with 750 tasks per test, and a context size of 768 tokens.

For the test data used in the generation of these scores, follow the appropriate links: HellaSwag, ARC, MMLU, Truthful QA and WinoGrande

📄 License

This project is licensed under the cc - by - nc - 4.0 license.