Bge M3 Financial Matryoshka

🚀 BGE-M3 Financial Matryoshka

BGE-M3 Financial Matryoshka 是一个基于 sentence-transformers 框架，从 BAAI/bge-m3 微调而来的模型。它能够将句子和段落映射到一个 1024 维的密集向量空间，可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等多种自然语言处理任务。

🚀 快速开始

安装依赖库

首先，你需要安装 sentence-transformers 库：

pip install -U sentence-transformers

加载模型并进行推理

安装完成后，你可以加载该模型并进行推理：

from sentence_transformers import SentenceTransformer

# 从 🤗 Hub 下载模型
model = SentenceTransformer("haophancs/bge-m3-financial-matryoshka")
# 进行推理
sentences = [
    'As of January 28, 2024 the net carrying value of our inventories was $1.3 billion, which included provisions for obsolete and damaged inventory of $139.7 million.',
    "What is the status of the company's inventory as of January 28, 2024, in terms of its valuation and provisions for obsolescence?",
    'What is the relationship between the ESG goals and the long - term growth strategy?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 获取嵌入向量的相似度分数
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 多任务适用性：可用于语义文本相似度计算、语义搜索、释义挖掘、文本分类、聚类等多种自然语言处理任务。
• 高维向量表示：能够将句子和段落映射到 1024 维的密集向量空间，更好地捕捉语义信息。
• 长序列处理：最大序列长度可达 8192 个标记，适合处理较长的文本。

📦 安装指南

安装 sentence-transformers 库即可使用该模型：

pip install -U sentence-transformers

💻 使用示例

基础用法

from sentence_transformers import SentenceTransformer

# 从 🤗 Hub 下载模型
model = SentenceTransformer("haophancs/bge-m3-financial-matryoshka")
# 进行推理
sentences = [
    'As of January 28, 2024 the net carrying value of our inventories was $1.3 billion, which included provisions for obsolete and damaged inventory of $139.7 million.',
    "What is the status of the company's inventory as of January 28, 2024, in terms of its valuation and provisions for obsolescence?",
    'What is the relationship between the ESG goals and the long - term growth strategy?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 获取嵌入向量的相似度分数
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 详细文档

模型详情

模型描述

模型来源

• 文档：Sentence Transformers Documentation
• 代码仓库：Sentence Transformers on GitHub
• Hugging Face：Sentence Transformers on Hugging Face

完整模型架构

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

评估

信息检索指标

以下是在不同数据集上的评估指标：

dim_1024 数据集

dim_768 数据集

dim_512 数据集

dim_384 数据集

训练详情

训练数据集

• 未命名数据集
  • 数据规模：6300 个训练样本
  • 数据列：<code>positive</code> 和 <code>anchor</code>
  • 近似统计信息（基于前 1000 个样本）：

    	positive	anchor
    类型	字符串	字符串
    详情	最小：11 个标记 平均：51.97 个标记 最大：1146 个标记	最小：7 个标记 平均：21.63 个标记 最大：47 个标记

  • 样本示例：

    positive	anchor
    From fiscal year 2022 to 2023, the cost of revenue as a percentage of total net revenue decreased by 3 percent.	What was the percentage change in cost of revenue as a percentage of total net revenue from fiscal year 2022 to 2023?
    •Operating income increased $321 million, or 2%, to $18.1 billion versus year ago due to the increase in net sales, partially offset by a modest decrease in operating margin.	What factors contributed to the increase in operating income for Procter & Gamble in 2023?
    market specific brands including 'Aurrera,' 'Lider,' and 'PhonePe.'	What specific brands does Walmart International market?

  • 损失函数：MatryoshkaLoss，参数如下：

    {
        "loss": "MultipleNegativesRankingLoss",
        "matryoshka_dims": [
            1024,
            768,
            512,
            384
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

训练超参数

• 非默认超参数：
  • eval_strategy: epoch
  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 2
  • gradient_accumulation_steps: 2
  • learning_rate: 2e-05
  • num_train_epochs: 4
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • bf16: True
  • tf32: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

框架版本

• Python: 3.12.2
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.2.0+cu121
• Accelerate: 0.31.0
• Datasets: 2.19.1
• Tokenizers: 0.19.1

🔧 技术细节

训练日志

📄 许可证

本模型使用 apache-2.0 许可证。

📚 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}