Finvoc2vec

🚀 FinVoc2Vec

FinVoc2Vec 是一款專為現實世界中的企業披露設計的語音語調分類器。它可以有效分析企業會議中的語音語調，為企業信息分析提供有力支持。

🚀 快速開始

FinVoc2Vec 是一個用於企業披露的語音語調分類器。它分兩個階段進行訓練：首先，使用 500,000 個未標記的電話會議語音句子樣本進行自監督預訓練，使基礎模型適應披露環境的聲學特徵；然後，進行有監督的微調，使用包含 5,000 個來自電話會議的語言中立句子的語音語料庫，並手動標記每個句子的語音語調（積極、消極或中立），讓模型學習人類標記的語音語調錶示。

💻 使用示例

基礎用法

使用示例數據集進行語音語調分類：

import torch
from datasets import load_dataset
from transformers import Wav2Vec2FeatureExtractor, AutoModel
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

# load model and feature extractor
model = AutoModel.from_pretrained("waiv/FinVoc2Vec", trust_remote_code=True).to(device)
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("waiv/FinVoc2Vec")

# load dataset
demo_dataset = load_dataset("waiv/FinVoc2Vec_demo")

arrays = [demo['audio']['array'] for demo in demo_dataset['test']]

# extract features
features = feature_extractor(
    arrays,
    sampling_rate=feature_extractor.sampling_rate,
    padding=True,
    truncation=False)

inputs = torch.tensor(np.array(features['input_values']), dtype=torch.float32).to(device)
attention_mask = torch.tensor(np.array(features['attention_mask']), dtype=torch.long).to(device)

prob_dict = {}
with torch.no_grad():
    model_output = model(inputs, attention_mask=attention_mask)
    logits = model_output['logits'].to(torch.float32).to('cpu')
    probs = torch.nn.functional.softmax(logits, dim=1).numpy()
    
    label_to_id = model.config.label2id
    for i, id in enumerate(demo_dataset['test']['id']):
        prob_dict[id] = {'prob_negative': probs[i, label_to_id['negative']],
                         'prob_neutral': probs[i, label_to_id['neutral']],
                         'prob_positive': probs[i, label_to_id['positive']]}

高級用法

使用音頻文件進行語音語調分類：

import torch
from torch.utils.data import DataLoader
from datasets import load_dataset
from dataclasses import dataclass
from typing import Dict, List, Optional, Union
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Processor, AutoModel
import torchaudio

device = "cuda" if torch.cuda.is_available() else "cpu"

@dataclass
class DataCollatorWithPadding:

    processor: Union[Wav2Vec2Processor, Wav2Vec2FeatureExtractor]
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None

    def __call__(self,
                 features: List[Dict[str, Union[List[int], torch.Tensor]]]
                 )-> Dict[str, torch.Tensor]:
        
        input_features = [{"input_values": feature["input_values"]} for feature in features]

        # trunc and pad max lengths, get attention mask
        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt")
        return batch


def preprocess_audio(batch: Dict,
                     feature_extractor: Wav2Vec2FeatureExtractor = None,
                     max_duration: Optional[float] = 20.0):
    
    target_sr = feature_extractor.sampling_rate # 16kHz
    audio_arrays = []

    for path in batch['path']:
        audio_array, sampling_rate = torchaudio.load(path)
        
        # split to mono if multiple channels exist
        if audio_array.shape[0] > 1:
            audio_array = torch.mean(audio_array, dim=0, keepdim=True)
        
        # resample audio
        resampler = torchaudio.transforms.Resample(sampling_rate, target_sr)
        audio_array = resampler(audio_array).squeeze().numpy()
        audio_arrays.append(audio_array)
    
    # set params for feature extractor
    max_length = int(target_sr*max_duration) if max_duration is not None else None

    # use feature extractor to normalize inputs and trunc data
    result = feature_extractor(
        audio_arrays,
        sampling_rate=target_sr,
        max_length=max_length,
        truncation=bool(max_length))
    return result

# load model
model = AutoModel.from_pretrained("waiv/FinVoc2Vec", trust_remote_code=True).to(device)

# load feature extractor
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("waiv/FinVoc2Vec")

# load dataset
# NOTE: Needed feature: 'path' -> path to the audio-data
test_dataset = load_dataset(r'path/to/dataset')

# preprocess audio data
test_dataset = test_dataset.map(
    preprocess_audio,
    batch_size=1000,
    batched=True,
    num_proc=4,
    fn_kwargs={'feature_extractor': feature_extractor,
                'max_duration': 20.0})

data_collator = DataCollatorWithPadding(feature_extractor)

data_loader = DataLoader(
    test_dataset, 
    batch_size=16,
    shuffle=False,
    collate_fn=data_collator,
    num_workers=4)

with torch.no_grad():
    for batch in data_loader:
        
        attention_mask, inputs = batch['attention_mask'], batch['input_values']
        
        inputs.to(device)
        attention_mask.to(device)
        
        model_output = model(inputs, attention_mask=attention_mask)

        logits = model_output['logits'].to(torch.float32).to('cpu')
        probs = torch.nn.functional.softmax(logits, dim=1).numpy()
        
        label_to_id = model.config.label2id
        dict_probs = {f'prob_negative': probs[:, label_to_id['negative']],
                      f'prob_neutral': probs[:, label_to_id['neutral']],
                      f'prob_positive': probs[:, label_to_id['positive']]}

📚 詳細文檔

為本地自動類註冊模型

若要為本地自動類註冊該模型，請使用以下代碼：

from transformers import AutoConfig, AutoModel

# download model and config
finvoc2vec_config = AutoConfig.from_pretrained("waiv/FinVoc2Vec", trust_remote_code=True)
finvoc2vec_model = AutoModel.from_pretrained("waiv/FinVoc2Vec", trust_remote_code=True)

# register model and config for automodel class
AutoConfig.register("finvoc2vec", FinVoc2VecConfig)
AutoModel.register(FinVoc2VecConfig, FinVoc2Vec)

更多資源

查看 🤗 Hugging Face Wav2Vec2 模型描述以獲取更多資源和配置信息。

📄 許可證

• 此模型基於 Wav2Vec2（Apache-2.0）派生而來。
• 此模型遵循知識共享署名非商業性使用 4.0 國際許可協議（CC-BY-NC-4.0）。

🔖 論文

• Listen Closely: Measuring Vocal Tone in Corporate Disclosures

📑 BibTeX

@article{ewertz2024,
  title={Listen Closely: Measuring Vocal Tone in Corporate Disclosures},
  author={Ewertz, Jonas and Knickrehm, Charlotte and Nienhaus, Martin and Reichmann, Doron},
  year={2024},
  note={Available at SSRN: \url{https://ssrn.com/abstract=4307178}}
}

模型信息