Arsh Llm Gpt

🚀 Arsh LLM

Arsh LLM is a new research - based model built on the GPT2 architecture, aiming to prove that large - scale models can be developed without extremely powerful hardware.

🚀 Quick Start

This model is based on the GPT2 architecture. Check out the main model. Arsh LLM is the newest project developed to assist in research. This version is pre - trained on the olmo mix dataset using a T4 GPU. (This process took about 4 - 5 days. Due to limitations, I split it into 8 parts and completed each part in one or two days!)

✨ Features

Arsh LLM is my latest research project. It requires more training, and this is just one of its initial versions. My goal is to demonstrate that large models don't necessarily need high - end hardware, especially with all the available tools to speed up the process. First, I designed an architecture using GPT NeoX and Llama documentation with the help of AI for optimal optimization (I'm not certain if it's fully optimized). Then, I created a model and calculated the initial weights using Phi - 4. After that, I trained it on a subset of the PILE dataset to stabilize the model. Next, I trained a model named ArshGPT on the olmo - mix - 1124 dataset. My aim was to more easily transfer data to my new large - scale model. This is a merged model fine - tuned using some small conversational open - source datasets. I believe its performance has improved.

📄 License

I've chosen the MIT license for this model, the best license to embrace freedom 😊!

📦 Information Table