🚀 una-xaberius-34b-v1-beta (UNA: Uniform Neural Alignment)
This is an experimental 34B model based on LLaMa-Yi-34B, developed by juanako.ai. It is trained using SFT, DPO, and UNA (Unified Neural Alignment) techniques on multiple datasets, aiming to provide high - quality text - generation capabilities.
🚀 Quick Start
The model is very good and works well on almost any prompt. The ChatML format and Alpaca System can get the best results. Here is an example:
<|im_start|>system
- You are a helpful assistant chatbot trained by MosaicML.
- You answer questions.
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>
<|im_start|>user
Explain QKV<|im_end|>
<|im_start|>assistant
### Assistant: I am StableVicuna, a large language model created by CarperAI. I am here to chat!
### Human: Explain QKV
### Assistant:
[Round <|round|>]
问:Explain QKV
答:
[Round <|round|>]
Question:Explain QKV
Answer:
Question:Explain QKV
Answer:
✨ Features
- High - performance on benchmarks: It scores 74.21, outperforming the former leader tigerbot - 70b - chat and landing on the #1 position of the HuggingFace LeaderBoard on 08 December 2023. It also scores 79.13 in MMLU, setting a new record not just for 34B but also for all OpenSource LLM's.
- Advanced training technique: Trained with UNA: Uniform Neural Alignment technique, which is a unique formula and method to "tame" models.
📚 Documentation
Model Details
The model is trained with the UNA: Uniform Neural Alignment technique (a related paper will be released soon).
- What is NOT UNA? It's not a merged layers model, nor SLERP, SLURP, or similar.
- What is UNA? It's a formula and a technique to "tame" models.
- Code and paper release: The code and paper will be released when there is time. Contributions can speed up the process.
Model Description
Framework versions
- Transformers 4.35.2 - UNA
- Pytorch 2.1.0
- Datasets 2.14.6
- Tokenizers 0.14.1
Citations
If you find Xaberius, Cybertron, Juanako, or any of our models useful, especially if you use it for your big brand or clone/merge/SLERP our models, please cite:
@misc{unaxaberius34b,
title={Xaberius 34B: Uniform Neural Alignment},
author={Xavier Murias},
year={2023},
publisher = {HuggingFace},
journal = {HuggingFace repository},
howpublished = {\url{https://huggingface.co/fblgit/una-xaberius-34b-v1beta}},
}
📄 License
The model is licensed under cc - by - nc - nd - 4.0.
🔧 Technical Details
Datasets
The model is trained on the following datasets:
- fblgit/tree - of - knowledge
- garage - bAInd/Open - Platypus
- allenai/ultrafeedback_binarized_cleaned
- Open - Orca/OpenOrca
Model Performance
Model |
Average |
ARC (25 - s) |
HellaSwag (10 - s) |
MMLU (5 - s) |
TruthfulQA (MC) (0 - s) |
Winogrande (5 - s) |
GSM8K (5 - s) |
[fblgit/una - cybertron - 7b - v1 - fp16](https://huggingface.co/fblgit/una - cybertron - 7b - v1 - fp16) |
69.49 |
68.43 |
85.85 |
63.34 |
63.28 |
80.90 |
55.12 |
[fblgit/una - cybertron - 7b - v2 - bf16](https://huggingface.co/fblgit/una - cybertron - 7b - v2 - bf16) |
69.67 |
68.26 |
85.?4 |
63.23 |
64.63 |
81.37 |
55.04 |
[fblgit/una - xaberius - 34b - v1beta](https://huggingface.co/fblgit/una - xaberius - 34b - v1beta) |
74.18 |
70.39 |
86.77 |
78.15 |
61.45 |
84.93 |
63.38 |
Evaluation Results
Metric |
Value |
Avg. |
74.18 |
AI2 Reasoning Challenge (25 - Shot) |
70.39 |
HellaSwag (10 - Shot) |
86.77 |
MMLU (5 - Shot) |
78.15 |
TruthfulQA (0 - shot) |
61.45 |
Winogrande (5 - shot) |
84.93 |
GSM8k (5 - shot) |
63.38 |
The detailed evaluation results can be found [here](https://huggingface.co/datasets/open - llm - leaderboard/details_fblgit__una - xaberius - 34b - v1beta).
Timeline
- 05 - Dec - 2023: v1 - beta released
- 08 - Dec - 2023: Evaluation been "RUNNING" for 2 days.. no results yet
- 09 - Dec - 2023: Evaluation been "FINISHED", confirming #1 spot outperforming the contaminated - disqualified tigerbot :)
[Results Here](https://huggingface.co/datasets/open - llm - leaderboard/details_fblgit__una - xaberius - 34b - v1beta/blob/main/results_2023 - 12 - 09T11 - 16 - 37.904970.json)