Meralion AudioLLM Whisper SEA LION

🚀 MERaLiON

MERaLiON-AudioLLM is a Speech-Text Large Language Model specifically designed for Singapore’s multilingual and multicultural landscape. By integrating a localised Whisper-large-v2 speech encoder and SEA-LION V3 text decoder, and fine-tuning it on 260,000 hours of speech and audio data for 6 various tasks, MERaLiON-AudioLLM is capable of addressing the diverse linguistic nuances of Singapore's local accents and dialects.

MERaLiON stands for Multimodal Empathetic Reasoning and Learning in One Network.

• Developed by: I²R, A*STAR, in collaboration with AISG, Singapore
• Model type: Multimodal LLM
• Language(s): Primarily English (Global and Singapore), supporting input and output in other languages compatible with Whisper and SEA-LION models.
• Audio: Mono channel audio, 16000 hz, within 30 seconds.
• License: MERaLiON Public License
• Demo: MERaLiON-AudioLLM Web Demo

We support model inference using the Huggingface and vLLM frameworks, enabling lightning inference speed. For more technical details, please refer to our technical report.

🚀 Quick Start

MERaLiON-AudioLLM can be used for various speech-related tasks. You can start by referring to the provided demos and examples in the repository.

✨ Features

• Tailored for Singapore’s multilingual and multicultural landscape.
• Integrates a localised speech encoder and text decoder.
• Fine-tuned on a large amount of speech and audio data for multiple tasks.
• Supports multiple speech-related tasks such as ASR, ST, SQA, SDS, SI, and PARA.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Sentence-level ASR

• Input: librispeech_clean.wav
• Output:

USER Instruction: Please transcribe this speech.
MODEL Output: <Speaker1> When they were outside, Ung simply latched the door and started up the path.

Dialogue-level ASR

• Input: IMDA_conversation.wav
• Output:

USER Instruction: Please turn this speech into written format.
MODEL Output: <Speaker1> Okay, (um) in that case, (uh) how do I apply? <Speaker2> Alright, you can just (um) apply it online and then (um) we'll need some documents from you. (um) let's say the bank statement with your address and your name, and also the (um) Nsf card that you have to enjoy the promotion. <Speaker1> (mmhmm) (mmhmm) (mmhmm) [ah] I see the green one, right?

Spoken Dialogue Summarization

• Input: IMDA_conversation.wav
• Output:

USER Instruction: Please briefly summarize this conversation.
MODEL Output: Speaker1 and Speaker2 discussed the application process for a promotion. Speaker2 explained that the application can be done online and provided a list of required documents, including a bank statement with the applicant's address and name, and an NSF card. Speaker1 acknowledged the information and confirmed the details of the required documents.

📚 Documentation

Model Description

MERaLiON-AudioLLM is designed to take in an audio-text pair as input and generate a text output.

The architecture comprises three key components: an audio encoder that transforms speech or audio inputs into sequences of vector representations, a text decoder that interprets and responds to natural language instructions, and an adaptor module that compresses the encoder representations while aligning the encoder’s hidden dimension with the text decoder’s embedding size.

Specifically, we fine-tuned the MERaLiON-Whisper encoder from Whisper-large-v2 for the audio encoder and used SEA-LION V3, a localised LLM developed by our partner AI Singapore as the text decoder.

Capabilities

MERaLiON-AudioLLM is trained to mainly address 6 tasks, namely Automatic Speech Recognition (ASR), Speech Translation (ST), Spoken Question Answering (SQA), Spoken Dialogue Summarization (SDS), Speech Instruction (SI), and Paralinguistics (PARA).

We benchmark MERaLiON-AudioLLM with a series of test sets from the AudioBench benchmark against three well-known AudioLLMs: Qwen2-Audio 7B, WavLLM, SALMONN, and a cascaded model. As is shown in the following table, MERaLiON-AudioLLM performs better in the Singapore local context, as evidenced by evaluation results on Singapore's Multitask National Speech Corpus (MNSC) datasets.

⚠️ Important Note

MNSC is a multitask speech understanding dataset derived and further annotated from IMDA NSC Corpus. It focuses on the knowledge of Singapore's local accent, localised terms, and code-switching.

We assess ASR and ST tasks using Word Error Rate (WER) and BLEU scores, respectively. For other tasks, we employ the LLM-as-a-Judge framework, which uses a pre-trained large language model to evaluate task performance by generating and scoring responses based on relevance, coherence, and accuracy criteria. Refer to the AudioBench paper for more details.

🔧 Technical Details

The README does not provide specific technical details, so this section is skipped.

📄 License

This model is licensed under the MERaLiON Public License.