đ LUXIA-21.4B-Alignment
LUXIA-21.4B-Alignment is a large language model with 21.4 billion parameters, excelling in various natural language processing tasks.
đ Quick Start
LUXIA-21.4B-Alignment, a large language model (LLM) with 21.4 billion parameters, showcases outstanding performance in multiple natural language processing (NLP) tasks. It achieves unparalleled state-of-the-art performance among models with parameters under 35B and even outperforms the 72B model and the 34Bx2 MoE (Mixture of Experts) model. For detailed information, please refer to the evaluation results table.
The luxia-21.4b-alignment model is derived from the luxia-21.4b-instruct model through DPO training, and the luxia-21.4b-instruct model is an SFT trained version of the luxia-21.4b model. We plan to release both the pretrained model and the instruction-tuned model soon.
⨠Features
Instruction Fine - Tuning Strategy
luxia-21.4b
We created the base model by expanding the layers through a passthrough method based on the internlm2-20b-llama model. To recover the performance of the created model, we conducted continual pretraining.
luxia-21.4b-instruct model
We utilize state - of - the - art instruction fine - tuning methods including supervised fine - tuning (SFT).
We used a mixture of the following datasets:
- c-s-ale/alpaca-gpt4-data
- Open-Orca/SlimOrca
- in - house generated data utilizing Metamath
luxia-21.4b-alignment model
We utilize state - of - the - art instruction fine - tuning methods including direct preference optimization (DPO).
We used a mixture of the following datasets:
- jondurbin/truthy-dpo-v0.1
- abacusai/ARC_DPO_FewShot
- abacusai/HellaSwag_DPO_FewShot
đ Documentation
Data Contamination Test Results
We generate our contamination numbers using https://github.com/swj0419/detect-pretrain-code-contamination/tree/master, with internlm2-20b-llama as our reference model.
luxia-21.4b-alignment-v1.2 has the following results:
Model |
ARC |
MMLU |
TruthfulQA |
GSM8K |
luxia-21.4b-alignment-v1.2 |
0.00 |
0.07 |
0.13 |
0.34 |
Open LLM Leaderboard Evaluation Results
Model |
ARC |
HellaSwag |
MMLU |
TruthfulQA |
Winogrande |
GSM8K |
luxia-21.4b-alignment-v1.2 |
77.73 |
90.86 |
67.86 |
79.16 |
86.27 |
66.94 |
đģ Usage Examples
Basic Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("saltlux/luxia-21.4b-alignment-v1.2")
model = AutoModelForCausalLM.from_pretrained(
"saltlux/luxia-21.4b-alignment-v1.2",
device_map="auto",
torch_dtype=torch.bfloat16,
)
đ License
đŦ Contact Us
Any questions and suggestions are welcomed at the discussion tab.