Slam Scaled

🚀 Slamming: Training a Speech Language Model on One GPU in a Day

This project presents a method to train high - quality Speech Language Models (SLMs) on a single academic GPU in 24 hours, aiming to make SLM training and research more accessible.

🚀 Quick Start

We refer users to the official repository for full usage explanations - github.

✨ Features

• Efficient Training: The model can be trained on a single academic GPU in 24 hours.
• Good Scalability: The training recipe scales well with more compute, achieving results comparable to leading SLMs at a fraction of the compute cost.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

The README does not contain code examples, so this section is skipped.

📚 Documentation

Model Details

Model Description

This Speech Language Model, introduced in "Slamming: Training a Speech Language Model on One GPU in a Day", focuses on efficient training. It was fine - tuned from Qwen/Qwen2.5 - 0.5B over a vocabulary of 500 speech tokens extracted from the 11 - th layer of mhubert - 25hz.

The model was pre - trained using next - token prediction on a subset of LibriSpeech, Libri - Light and a synthetic dataset sTinyStories. It was subsequently fine - tuned with DPO on SpokenSwag.

Uses

This base SpeechLM can be used to generate continuations for speech segments, or as a base for further tuning. See the SlamKit codebase for more details on usage, and checkout the demo page for some generation examples.

⚠️ Important Note

This model was trained on curated speech datasets which contain mainly audio - books and stories, as such the outputs should not be treated as factual in any way.

Training Details

We highly encourage users to read the full paper for full training details. A brief overview is provided below.

Training Data

This model was trained on a subset of LibriSpeech train, Libri - Light and the synthetic dataset sTinyStories for the pre - training phase. It was also trained with DPO on the synthetic dataset SpokenSwag.

Training Procedure

This model was trained by next token prediction over several datasets, and then trained with DPO over SpokenSwag. Please refer to the paper or code for the full training recipes.

Preprocessing

Speech tokens are extracted from the audio using Hubert - 25hz, and quantised using the official kmeans released with the model in textlesslib. Units are de - duplicated. We encourage you to explore the official repository for full details - github.

Evaluation

The paper provides full results. We give here some results and also refer to the demo page to listen to some samples.

Compute Infrastructure

This model was trained as part of "Slamming: Training a Speech Language Model on One GPU in a Day", focusing on efficient training.

Hardware

This model was trained using only 2 Nvidia A100 GPU for 48 hours.

Software

The model was trained using the SlamKit codebase which builds upon 🤗transformers extending it to support easy and efficient training of Speech Language Models.

🔧 Technical Details

The model was presented in the paper Slamming: Training a Speech Language Model on One GPU in a Day. Through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components, it achieves efficient training of high - quality SLMs.

📄 License

This model is licensed under the MIT license.

Citation

BibTeX:

@misc{maimon2025slamming,
      title={Slamming: Training a Speech Language Model on One GPU in a Day}, 
      author={Gallil Maimon and Avishai Elmakies and Yossi Adi},
      year={2025},
      eprint={2502.15814},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2502.15814}, 
}