Slam

🚀 SLAM Model Card

This is a Speech Language Model designed to generate speech continuations over discrete Hubert tokens, offering efficient speech processing capabilities.

✨ Features

• Efficient Training: Focuses on efficient training, as introduced in "Slamming: Training a Speech Language Model on One GPU in a Day".
• Fine - Tuned Model: Fine - tuned from Qwen/Qwen2.5 - 0.5B over a vocabulary of 500 speech tokens.
• Versatile Usage: Can be used to generate speech continuations or as a base for further tuning.

📚 Documentation

📋 Model Details

Model Description

This Speech Language Model, presented in "Slamming: Training a Speech Language Model on One GPU in a Day", emphasizes efficient training. It was fine - tuned from Qwen/Qwen2.5 - 0.5B using a vocabulary of 500 speech tokens extracted from the 11 - th layer of mhubert - 25hz. For a more powerful version trained with slightly more compute (2*A100 for 2 days), refer to slam_scaled.

The model was trained by next - token prediction on a subset of LibriSpeech, Libri - Light, and the synthetic data sTinyStories, and then trained with DPO on SpokenSwag.

• Developed by: SLP - RL
• Model type: SpeechLM
• License: MIT
• Finetuned from model: Qwen/Qwen2.5 - 0.5B

Model Sources

• Repository: https://github.com/slp - rl/slamkit
• Paper: https://arxiv.org/abs/2502.15814
• Demo: https://pages.cs.huji.ac.il/adiyoss - lab/slamming/

🛠️ Uses

This is a base SpeechLM that can generate continuations for speech segments or serve as a base for further tuning. For more usage details, see the SlamKit [codebase](https://github.com/slp - rl/slamkit), and check out the [demo page](https://pages.cs.huji.ac.il/adiyoss - lab/slamming/) for some generation examples.

⚠️ Important Note

This model was trained on curated speech datasets mainly containing audio - books and stories. Thus, the outputs should not be considered factual in any way.

🚀 Quick Start

Refer to the official repository for full usage explanations - [github](https://github.com/slp - rl/slamkit).

🔧 Technical Details

Training Data

This model was pre - trained on a subset of LibriSpeech train, [Libri - Light](https://ai.meta.com/tools/libri - light/), and the synthetic dataset sTinyStories. It was also trained with DPO on the synthetic dataset SpokenSwag.

Training Procedure

The model was trained by next - token prediction on several datasets and then trained with DPO on SpokenSwag. For the full training recipes, please refer to the paper or [code](https://github.com/slp - rl/slamkit).

Preprocessing

Speech tokens are extracted from the audio using [Hubert - 25hz](https://huggingface.co/slprl/mhubert - base - 25hz) and quantized using the official kmeans released with the model in textlesslib. Units are de - duplicated. Explore the official repository for full details - [github](https://github.com/slp - rl/slamkit).

📊 Evaluation

The paper provides full results. Here are some results, and you can also refer to the [demo page](https://pages.cs.huji.ac.il/adiyoss - lab/slamming/) to listen to some samples.

Compute Infrastructure

This model was trained as part of "Slamming: Training a Speech Language Model on One GPU in a Day", focusing on efficient training.

Hardware

The model was trained using only a single Nvidia A5000 GPU, 16 CPU cores, and 24 GB of RAM for 24 hours.

Software

The model was trained using the [SlamKit](https://github.com/slp - rl/slamkit) codebase, which builds upon 🤗transformers and extends it to support easy and efficient training of Speech Language Models.

📄 License

This model is released under the MIT license.

📖 Citation

BibTeX:

@misc{maimon2025slamming,
      title={Slamming: Training a Speech Language Model on One GPU in a Day}, 
      author={Gallil Maimon and Avishai Elmakies and Yossi Adi},
      year={2025},
      eprint={2502.15814},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2502.15814}, 
}