BioLingual Open-source Bioacoustic Model - Free Zero-shot Classification and Task Fine-tuning

Biolingual

Developed by davidrrobinson

A contrastive language-audio pretrained bioacoustic audio-text model supporting zero-shot classification and task fine-tuning

Audio Classification

Transformers

#Bioacoustic Classification #Zero-shot Learning #Audio-Text Contrastive Learning

Downloads 483

Release Time : 7/24/2023

Model Overview

BioLingual is a transferable bioacoustic model based on human language supervision, achieving bioacoustic audio-text alignment through contrastive language-audio pretraining

Model Features

Contrastive Language-Audio Pretraining

Achieves cross-modal alignment between audio and text through contrastive learning

Zero-shot Classification Capability

Classifies new audio categories without fine-tuning

Bioacoustics Specialization

Specifically optimized for bioacoustic data like animal sounds

Model Capabilities

Bioacoustic audio classification

Audio-text embedding extraction

Zero-shot learning

Cross-modal retrieval

Use Cases

Wildlife Monitoring

Animal Sound Recognition

Identifying specific animal species in field recordings

Example accurately recognized dog sounds

Ecological Research

Biodiversity Assessment

Evaluating biodiversity in specific areas through sound analysis

🚀 BioLingual

Transferable Models for bioacoustics with Human Language Supervision, an audio - text model for bioacoustics based on contrastive language - audio pretraining.

🚀 Quick Start

This model can be used for bioacoustic zero - shot audio classification or fine - tuning on bioacoustic tasks.

✨ Features

An audio - text model for bioacoustics based on contrastive language - audio pretraining.
Enables zero - shot audio classification in bioacoustics.
Can be fine - tuned on bioacoustic tasks.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

Perform zero - shot audio classification using `pipeline`

from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("ashraq/esc50")
audio = dataset["train"]["audio"][-1]["array"]

audio_classifier = pipeline(task="zero-shot-audio-classification", model="davidrrobinson/BioLingual")
output = audio_classifier(audio, candidate_labels=["Sound of a sperm whale", "Sound of a sea lion"])
print(output)
>>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]

Advanced Usage

Run the model on CPU

from datasets import load_dataset
from transformers import ClapModel, ClapProcessor

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]

model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
processor = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")

inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")
audio_embed = model.get_audio_features(**inputs)

Run the model on GPU

from datasets import load_dataset
from transformers import ClapModel, ClapProcessor

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]

model = ClapModel.from_pretrained("laion/clap-htsat-unfused").to(0)
processor = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")

inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)
audio_embed = model.get_audio_features(**inputs)

📚 Documentation

Datasets

davidrrobinson/AnimalSpeak

🔧 Technical Details

No technical details are provided in the original document, so this section is skipped.

📄 License

No license information is provided in the original document, so this section is skipped.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご