UNA-SimpleSmaug-34b-v1beta Open Source Model - Significantly Improves Mathematical and Reasoning Abilities, Exceptional Performance!

UNA SimpleSmaug 34b V1beta

Developed by fblgit

A supervised fine-tuned model based on Smaug-34B, focused on enhancing mathematical and reasoning capabilities, excelling among 34B-scale models.

Large Language Model

Transformers

Open Source License:Apache-2.0 #Mathematical Reasoning Optimization #High-Precision Text Generation #SFT Fine-Tuning Enhancement

Downloads 18

Release Time : 2/5/2024

Model Overview

This model is a variant of Smaug-34B-v0.1 fine-tuned via supervised learning (SFT) on simple mathematical datasets, improving mathematical and reasoning abilities while retaining the original model's performance. It performs exceptionally well on the Open LLM Leaderboard and is a leader among 34B-scale models.

Model Features

Enhanced Mathematical Capabilities

Supervised fine-tuning on simple mathematical datasets significantly improves mathematical and reasoning abilities.

Leader in 34B Scale

Outstanding performance among 34B-parameter models, scoring 77.41 on the Open LLM Leaderboard.

Attention Layer Optimization

UNA technology is applied only to attention layers without modifying MLP layers, preserving the original model's core capabilities.

Model Capabilities

Text Generation

Mathematical Reasoning

Logical Reasoning

Question Answering

Use Cases

Education

Math Problem Solving

Solves various mathematical problems, including arithmetic, algebra, etc.

72.48% accuracy on the GSM8k dataset

Evaluation Testing

Standard Test Evaluation

Used for standard test evaluations such as AI2 Reasoning Challenge and MMLU.

74.57 on AI2 Reasoning Challenge, 76.68 on MMLU

🚀 UNA-SimpleSmaug-34b-v1beta

UNA-SimpleSmaug-34b-v1beta is a 34B model that scored top on February 4, 2024. It outperforms its base model Smaug-34B-v0.1 with a score of 77.41. After SFT, the internal mechanism of Smaug returns to normal, allowing for further training.

📄 License

This project is licensed under the Apache-2.0 license.

🔖 Tags

UNA
simple-math
juanako

📦 Model Information

Base Model: abacusai/Smaug-34B-v0.1
Datasets:
- fblgit/simple-math
- jondurbin/bagel-v0.3

📊 Model Index

Model Name: UNA-SimpleSmaug-34b-v1beta
Results: | Task | Dataset | Metric | Value | Source | |------|---------|--------|-------|--------| | Text Generation | AI2 Reasoning Challenge (25-Shot) | acc_norm | 74.57 | Open LLM Leaderboard | | Text Generation | HellaSwag (10-Shot) | acc_norm | 86.74 | Open LLM Leaderboard | | Text Generation | MMLU (5-Shot) | acc | 76.68 | Open LLM Leaderboard | | Text Generation | TruthfulQA (0-shot) | mc2 | 70.17 | Open LLM Leaderboard | | Text Generation | Winogrande (5-shot) | acc | 83.82 | Open LLM Leaderboard | | Text Generation | GSM8k (5-shot) | acc | 72.48 | Open LLM Leaderboard | | Text Generation | IFEval (0-Shot) | inst_level_strict_acc and prompt_level_strict_acc | 45.56 | Open LLM Leaderboard | | Text Generation | BBH (3-Shot) | acc_norm | 32.78 | Open LLM Leaderboard | | Text Generation | MATH Lvl 5 (4-Shot) | exact_match | 0.15 | Open LLM Leaderboard | | Text Generation | GPQA (0-shot) | acc_norm | 8.95 | Open LLM Leaderboard | | Text Generation | MuSR (0-shot) | acc_norm | 11.96 | Open LLM Leaderboard | | Text Generation | MMLU-PRO (5-shot) | acc | 39.33 | Open LLM Leaderboard |

🌟 Model Features

Based on Smaug: This model is built upon the Smaug architecture.
SimpleMath Dataset: Trained with the SimpleMath dataset to enhance mathematical and reasoning capabilities.
Trained on Axolotl: Utilized the Axolotl framework for training.

📸 Model Image

UNA

⚙️ Experiment

The experiment aims to understand the impact of applying the SimpleMath dataset at the attention layer during the SFT session on the overall neural network.

Results: The model improves mathematical and reasoning capabilities without degrading previous training results.

🛠️ Tool

Model Similarities Tool Detector: https://github.com/fblgit/model-similarity. This tool numerically confirms the model's lineage.

📈 Evals

First Evaluation Table

Metric	Value
Avg.	77.41
AI2 Reasoning Challenge (25-Shot)	74.57
HellaSwag (10-Shot)	86.74
MMLU (5-Shot)	76.68
TruthfulQA (0-shot)	70.17
Winogrande (5-shot)	83.82
GSM8k (5-shot)	72.48

Second Evaluation Table

Task	Version	Metric	Value
arc_challenge	HF	acc_norm	0.7457337883959
gsm8k	HF	acc	0.7247915087187
mmlu	HF	acc	0.7649553475572
mmlu	HF	acc_norm	0.7681713551647
hellaswag	HF	acc_norm	0.8673571001792
truthfulqa	HF	mc2	0.7016557407771
winogrande	HF	acc	0.8382004735595

Observation: The model shows improvements in GSM, MMLU, ARC, and WINO.

Third Evaluation Table

Metric	Value
Avg.	23.12
IFEval (0-Shot)	45.56
BBH (3-Shot)	32.78
MATH Lvl 5 (4-Shot)	0.15
GPQA (0-shot)	8.95
MuSR (0-shot)	11.96
MMLU-PRO (5-shot)	39.33

📚 Citations

We would like to thank abacusai for creating Smaug-34B, the Bagel, and all the underlying technology of the base model.

If you use this model, please provide citations, even for merged models or other related works.

🔗 Links

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご