Bge Base Financial Matryoshka Test 1

🚀 BGE base Financial Matryoshka

這是一個基於 sentence-transformers 框架，從 BAAI/bge-base-en-v1.5 微調而來的模型。它可以將句子和段落映射到一個 768 維的密集向量空間，可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等任務。

🚀 快速開始

直接使用（Sentence Transformers）

首先安裝 Sentence Transformers 庫：

pip install -U sentence-transformers

然後你可以加載這個模型並進行推理。

from sentence_transformers import SentenceTransformer

# 從 🤗 Hub 下載
model = SentenceTransformer("NickyNicky/bge-base-financial-matryoshka")
# 進行推理
sentences = [
    'Non-GAAP earnings from operations and non-GAAP operating profit margin consist of earnings from operations or earnings from operations as a percentage of net revenue excluding the items mentioned above and charges relating to the amortization of intangible assets, goodwill impairment, transformation costs and acquisition, disposition and other related charges. Hewlett Packard Enterprise excludes these items because they are non-cash expenses, are significantly impacted by the timing and magnitude of acquisitions, and are inconsistent in amount and frequency.',
    "What specific charges are excluded from Hewlett Packard Enterprise's non-GAAP operating profit margin and why?",
    'How many shares were outstanding at the beginning of 2023 and what was their aggregate intrinsic value?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# 獲取嵌入向量的相似度分數
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

✨ 主要特性

• 基於 sentence-transformers 框架，從 BAAI/bge-base-en-v1.5 微調而來。
• 能夠將句子和段落映射到 768 維的密集向量空間。
• 可用於語義文本相似度計算、語義搜索、釋義挖掘、文本分類、聚類等多種任務。

📚 詳細文檔

模型詳情

模型描述

模型來源

• 文檔：Sentence Transformers 文檔
• 倉庫：GitHub 上的 Sentence Transformers
• Hugging Face：Hugging Face 上的 Sentence Transformers

完整模型架構

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

評估

指標

信息檢索（dim_768 數據集）

使用 InformationRetrievalEvaluator 進行評估。

信息檢索（dim_512 數據集）

使用 InformationRetrievalEvaluator 進行評估。

信息檢索（dim_256 數據集）

使用 InformationRetrievalEvaluator 進行評估。

信息檢索（dim_128 數據集）

使用 InformationRetrievalEvaluator 進行評估。

信息檢索（dim_64 數據集）

使用 InformationRetrievalEvaluator 進行評估。

訓練詳情

訓練數據集

未命名數據集

• 大小：6300 個訓練樣本
• 列：positive 和 anchor
• 基於前 1000 個樣本的近似統計信息： | | 正樣本 | 錨樣本 | |------|------|------| | 類型 | 字符串 | 字符串 | | 詳情 |
  • 最小值：6 個標記
  • 平均值：46.8 個標記
  • 最大值：512 個標記
  |
  • 最小值：8 個標記
  • 平均值：20.89 個標記
  • 最大值：51 個標記
  |
• 樣本： | 正樣本 | 錨樣本 | |------|------| | Retail sales mix by product type for company-operated stores shows beverages at 74%, food at 22%, and other items at 4%. | What are the primary products sold in Starbucks company-operated stores? | | The pre-tax adjustment for transformation costs was $136 in 2021 and $111 in 2020. Transformation costs primarily include costs related to store and business closure costs and third party professional consulting fees associated with business transformation and cost saving initiatives. | What was the purpose of pre-tax adjustments for transformation costs by The Kroger Co.? | | HP's Consolidated Financial Statements are prepared in accordance with United States generally accepted accounting principles (GAAP). | What principles do HP's Consolidated Financial Statements adhere to? |
• 損失函數：MatryoshkaLoss，參數如下：

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

訓練超參數

非默認超參數

• eval_strategy：epoch
• per_device_train_batch_size：40
• per_device_eval_batch_size：16
• gradient_accumulation_steps：16
• learning_rate：2e-05
• num_train_epochs：10
• lr_scheduler_type：cosine
• warmup_ratio：0.1
• bf16：True
• tf32：True
• optim：adamw_torch_fused
• batch_sampler：no_duplicates

所有超參數

訓練日誌

框架版本

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.2.0+cu121
• Accelerate: 0.31.0
• Datasets: 2.19.1
• Tokenizers: 0.19.1

📄 許可證

本模型使用 apache-2.0 許可證。

📖 引用

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}