Bge M3 Financial Matryoshka

🚀 BGE-M3 Financial Matryoshka

BGE-M3 Financial Matryoshka 是一個基於 sentence-transformers 框架，從 BAAI/bge-m3 微調而來的模型。它能夠將句子和段落映射到一個 1024 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種自然語言處理任務。

🚀 快速開始

安裝依賴庫

首先，你需要安裝 sentence-transformers 庫：

pip install -U sentence-transformers

加載模型並進行推理

安裝完成後，你可以加載該模型並進行推理：

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載模型
model = SentenceTransformer("haophancs/bge-m3-financial-matryoshka")
# 進行推理
sentences = [
    'As of January 28, 2024 the net carrying value of our inventories was $1.3 billion, which included provisions for obsolete and damaged inventory of $139.7 million.',
    "What is the status of the company's inventory as of January 28, 2024, in terms of its valuation and provisions for obsolescence?",
    'What is the relationship between the ESG goals and the long - term growth strategy?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 多任務適用性：可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種自然語言處理任務。
• 高維向量表示：能夠將句子和段落映射到 1024 維的密集向量空間，更好地捕捉語義信息。
• 長序列處理：最大序列長度可達 8192 個標記，適合處理較長的文本。

📦 安裝指南

安裝 sentence-transformers 庫即可使用該模型：

pip install -U sentence-transformers

💻 使用示例

基礎用法

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載模型
model = SentenceTransformer("haophancs/bge-m3-financial-matryoshka")
# 進行推理
sentences = [
    'As of January 28, 2024 the net carrying value of our inventories was $1.3 billion, which included provisions for obsolete and damaged inventory of $139.7 million.',
    "What is the status of the company's inventory as of January 28, 2024, in terms of its valuation and provisions for obsolescence?",
    'What is the relationship between the ESG goals and the long - term growth strategy?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers Documentation
• 代碼倉庫：Sentence Transformers on GitHub
• Hugging Face：Sentence Transformers on Hugging Face

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

評估

信息檢索指標

以下是在不同數據集上的評估指標：

dim_1024 數據集

dim_768 數據集

dim_512 數據集

dim_384 數據集

訓練詳情

訓練數據集

• 未命名數據集
  • 數據規模：6300 個訓練樣本
  • 數據列：<code>positive</code> 和 <code>anchor</code>
  • 近似統計信息（基於前 1000 個樣本）：

    	positive	anchor
    類型	字符串	字符串
    詳情	最小：11 個標記 平均：51.97 個標記 最大：1146 個標記	最小：7 個標記 平均：21.63 個標記 最大：47 個標記

  • 樣本示例：

    positive	anchor
    From fiscal year 2022 to 2023, the cost of revenue as a percentage of total net revenue decreased by 3 percent.	What was the percentage change in cost of revenue as a percentage of total net revenue from fiscal year 2022 to 2023?
    •Operating income increased $321 million, or 2%, to $18.1 billion versus year ago due to the increase in net sales, partially offset by a modest decrease in operating margin.	What factors contributed to the increase in operating income for Procter & Gamble in 2023?
    market specific brands including 'Aurrera,' 'Lider,' and 'PhonePe.'	What specific brands does Walmart International market?

  • 損失函數：MatryoshkaLoss，參數如下：

    {
        "loss": "MultipleNegativesRankingLoss",
        "matryoshka_dims": [
            1024,
            768,
            512,
            384
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

訓練超參數

• 非默認超參數：
  • eval_strategy: epoch
  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 2
  • gradient_accumulation_steps: 2
  • learning_rate: 2e-05
  • num_train_epochs: 4
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • bf16: True
  • tf32: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

框架版本

• Python: 3.12.2
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.2.0+cu121
• Accelerate: 0.31.0
• Datasets: 2.19.1
• Tokenizers: 0.19.1

🔧 技術細節

訓練日誌

📄 許可證

本模型使用 apache-2.0 許可證。

📚 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}