đ TinyMistral-248m: A Compact Language Model
This is a pre - trained language model derived from the Mistral 7B model, scaled down to approximately 248 million parameters. It has been trained on 7,488,000 examples. The model is designed not for direct use but for fine - tuning on downstream tasks. It is expected to have a context length of around 32,768 tokens. Due to issues with saving model weights, safe serialization has been removed.
đ Quick Start
This model is mainly for fine - tuning. You can refer to the following inference parameters:
{
"parameters": {
"do_sample": true,
"temperature": 0.5,
"top_p": 0.5,
"top_k": 50,
"max_new_tokens": 250,
"repetition_penalty": 1.176
}
}
⨠Features
- Compact Size: With approximately 248 million parameters, it is a scaled - down version of the Mistral 7B model.
- Small - scale Training: Trained on a relatively small number of examples (7,488,000).
- Fine - tuning Oriented: Intended for fine - tuning on downstream tasks rather than direct use.
- Long Context Length: Expected to have a context length of around 32,768 tokens.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
No code examples are provided in the original document.
đ Documentation
Evaluation Results
During evaluation on InstructMix, this model achieved an average perplexity score of 6.3. More epochs are planned for this model on different datasets.
You can find the Open LLM Leaderboard Evaluation Results (outdated). Detailed results are available here.
| Metric |
Value |
| Avg. |
24.18 |
| ARC (25 - shot) |
20.82 |
| HellaSwag (10 - shot) |
26.98 |
| MMLU (5 - shot) |
23.11 |
| TruthfulQA (0 - shot) |
46.89 |
| Winogrande (5 - shot) |
50.75 |
| GSM8K (5 - shot) |
0.0 |
| DROP (3 - shot) |
0.74 |
Purpose
The purpose of this model is to prove that trillion - scale datasets are not needed to pretrain a language model. As a result of needing small datasets, this model was pretrained on a single GPU (Titan V).
đ§ Technical Details
- Training Datasets:
- Skylion007/openwebtext
- JeanKaddour/minipile
- Model Parameters: Scaled down to approximately 248 million parameters from the Mistral 7B model.
- Training Examples: 7,488,000 examples.
- Context Length: Around 32,768 tokens.
- Serialization: Safe serialization has been removed due to issues saving model weights.
đ License
This model is licensed under the Apache 2.0 license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
