Distilhubert Open-source Speech Learning Model - Lightweight Design Reduces Costs While Still Boasting Excellent Performance

Distilhubert

Developed by ntu-spml

DistilHuBERT is a lightweight speech representation learning model achieved through hierarchical distillation of the HuBERT model, significantly reducing model size and computational costs while maintaining performance.

Speech Recognition

Transformers

EnglishOpen Source License:Apache-2.0 #Voice Representation Distillation #Lightweight HuBERT #Self-supervised Learning

Downloads 2,962

Release Time : 3/2/2022

Model Overview

A lightweight version of speech representation learning model obtained via hierarchical distillation of the HuBERT model within a multi-task learning framework, suitable for various speech processing tasks.

Model Features

Efficient Distillation

Reduces the HuBERT model size by 75% and improves speed by 73% through hierarchical distillation technology.

Multi-task Learning

Adopts a multi-task learning framework to directly distill hidden representations from the HuBERT model.

Low Resource Requirements

Requires minimal training time and data volume, making it suitable for personal and edge devices.

Performance Retention

Maintains most of the performance across ten different tasks.

Model Capabilities

Speech Representation Extraction

Speech Recognition (requires fine-tuning)

Support for Speech Processing Tasks

Use Cases

Speech Processing

Speech Recognition System

Can be fine-tuned to build a speech recognition system.

Maintains performance close to the original HuBERT.

Edge Device Speech Processing

Suitable for deployment on resource-limited edge devices for speech processing.

Small model size with high computational efficiency.

🚀 DistilHuBERT

DistilHuBERT is a speech representation model developed by NTU Speech Processing & Machine Learning Lab. It offers an efficient solution for speech processing tasks by distilling hidden representations from a HuBERT model.

🚀 Quick Start

DistilHuBERT is a base model pretrained on 16kHz sampled speech audio. When using the model, ensure that your speech input is also sampled at 16Khz.

✨ Features

Efficient Distillation: DistilHuBERT reduces HuBERT's size by 75% and speeds up processing by 73% while retaining most of the performance in ten different tasks.
Less Training Requirements: It requires little training time and data, enabling the pre - training of personal and on - device SSL models for speech.

📚 Documentation

Paper

DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT

Authors

Heng - Jui Chang, Shu - wen Yang, Hung - yi Lee

Abstract

Self - supervised speech representation learning methods like wav2vec 2.0 and Hidden - unit BERT (HuBERT) leverage unlabeled speech data for pre - training and offer good representations for numerous speech processing tasks. Despite the success of these methods, they require large memory and high pre - training costs, making them inaccessible for researchers in academia and small companies. Therefore, this paper introduces DistilHuBERT, a novel multi - task learning framework to distill hidden representations from a HuBERT model directly. This method reduces HuBERT's size by 75% and 73% faster while retaining most performance in ten different tasks. Moreover, DistilHuBERT required little training time and data, opening the possibilities of pre - training personal and on - device SSL models for speech.

💻 Usage Examples

Basic Usage

See this blog for more information on how to fine - tune the model. Note that the class Wav2Vec2ForCTC has to be replaced by HubertForCTC.

📄 License

This project is licensed under the Apache 2.0 license.

⚠️ Important Note

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine - tuned on labeled text data. Check out this blog for more in - detail explanation of how to fine - tune the model.

Property	Details
Datasets	librispeech_asr
Tags	speech
License	apache - 2.0
Original Model	https://github.com/s3prl/s3prl/tree/master/s3prl/upstream/distiller

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご