Financial Rag Matryoshka

🚀 financial-rag-matryoshka

該模型是基於 Alibaba-NLP/gte-large-en-v1.5 針對金融用例進行微調的模型。它可以將句子和段落映射到一個 1024 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等任務。該模型在金融文檔檢索任務中表現出色，同時也能保持較高的通用性能。

🚀 快速開始

直接使用（Sentence Transformers）

首先，安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

然後，你可以加載該模型並進行推理：

from sentence_transformers import SentenceTransformer

# 從 Hugging Face Hub 下載
model = SentenceTransformer("rbhatia46/gte-large-en-v1.5-financial-rag-matryoshka")
# 運行推理
sentences = [
    'JP Morgan reported total deposits of $2.6 trillion in the year ending December 31, 2023.',
    "What were JP Morgan's total deposits in 2023?",
    'What is the primary source of revenue for the software company, Microsoft?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 多任務適用：可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種任務。
• 金融領域優化：針對金融用例進行了微調，在金融文檔檢索任務中表現出色。
• 高維向量映射：將句子和段落映射到 1024 維的密集向量空間。

📦 安裝指南

若要使用該模型，你需要安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

💻 使用示例

基礎用法

from sentence_transformers import SentenceTransformer

# 從 Hugging Face Hub 下載
model = SentenceTransformer("rbhatia46/gte-large-en-v1.5-financial-rag-matryoshka")
# 運行推理
sentences = [
    'JP Morgan reported total deposits of $2.6 trillion in the year ending December 31, 2023.',
    "What were JP Morgan's total deposits in 2023?",
    'What is the primary source of revenue for the software company, Microsoft?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers Documentation
• 倉庫：Sentence Transformers on GitHub
• Hugging Face：Sentence Transformers on Hugging Face

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

評估

信息檢索指標

以下是不同數據集上的評估指標：

訓練詳情

訓練數據集

未命名數據集

• 大小：4275 個訓練樣本
• 列：positive 和 anchor
• 近似統計信息（基於前 1000 個樣本）：

  	positive	anchor
  類型	字符串	字符串
  詳情	最小：15 個標記 平均：44.74 個標記 最大：114 個標記	最小：9 個標記 平均：18.12 個標記 最大：32 個標記

• 樣本：

  positive	anchor
  At the end of fiscal year 2023, Exxon Mobil reported a debt - to - equity ratio of 0.32, implying that the company used more equity than debt in its capital structure.	What was the debt - to - equity ratio for Exxon Mobil at the end of fiscal year 2023?
  Amazon Web Services (AWS) generated $12.7 billion in net sales in the fourth quarter of 2020, up 28% from the same period of the previous year. It accounted for about 10% of Amazon’s total net sales for the quarter.	How did Amazon's AWS segment perform in the fourth quarter of 2020?
  JPMorgan Chase generates revenues by providing a wide range of banking and financial services. These include investment banking (M&As, advisory), consumer and community banking (home mortgages, auto loans), commercial banking, and asset and wealth management.	What are the key revenue sources for JPMorgan Chase?

• 損失函數：MatryoshkaLoss，參數如下：

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

訓練超參數

非默認超參數

• eval_strategy：epoch
• per_device_train_batch_size：32
• per_device_eval_batch_size：16
• gradient_accumulation_steps：16
• learning_rate：2e - 05
• num_train_epochs：10
• lr_scheduler_type：cosine
• warmup_ratio：0.1
• bf16：True
• tf32：True
• load_best_model_at_end：True
• optim：adamw_torch_fused
• batch_sampler：no_duplicates

訓練日誌

注：加粗行表示保存的檢查點。

框架版本

• Python：3.10.6
• Sentence Transformers：3.0.1
• Transformers：4.41.2
• PyTorch：2.1.2 + cu121
• Accelerate：0.32.1
• Datasets：2.19.1
• Tokenizers：0.19.1

🔧 技術細節

該模型基於 Alibaba - NLP/gte - large - en - v1.5 進行微調，使用了 MatryoshkaLoss 損失函數，結合了 MultipleNegativesRankingLoss。在訓練過程中，通過不同維度的向量空間進行學習，以提高模型在金融領域的性能。同時，使用了多種超參數進行優化，如學習率調度、批量大小等，以確保模型的收斂和泛化能力。

📄 許可證

該模型使用 apache - 2.0 許可證。

引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard - Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al - Rfou and Brian Strope and Yun - hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}