Snowflake Arctic Embed L V2.0 Ko

🚀 SentenceTransformer based on Snowflake/snowflake-arctic-embed-l-v2.0

This is a sentence-transformers model fine-tuned from Snowflake/snowflake-arctic-embed-l-v2.0 on clustered datasets. It maps sentences and paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity and semantic search.

The Snowflake/snowflake-arctic-embed-l-v2.0 model has been further trained with Korean data to enhance its performance in Korean retrieval tasks. It is a powerful model that achieves state-of-the-art (SOTA) performance across multiple retrieval benchmarks.

✨ Features

• Maps sentences and paragraphs to a 1024 - dimensional dense vector space.
• Suitable for semantic textual similarity and semantic search.
• Enhanced performance in Korean retrieval tasks after training with Korean data.
• Achieves SOTA performance across multiple retrieval benchmarks.

📦 Installation

First, install the Sentence Transformers library and xformers library:

pip install -U sentence-transformers
pip install xformers

💻 Usage Examples

Basic Usage

Using Sentence Transformers

from sentence_transformers import SentenceTransformer

# Load the model
# Please use bf16 when inferring with half precision
model_name = 'dragonkue/snowflake-arctic-embed-l-v2.0-ko'
model = SentenceTransformer(model_name)

# Define the queries and documents
queries = ['대한민국의 수도는 어디인가?', '한글을 만든 사람은 누구인가?']
documents = ['대한민국의 수도는 서울이다.', '한글은 세종대왕이 창제하였다.']

# Compute embeddings: use `prompt_name="query"` to encode queries!
query_embeddings = model.encode(queries, prompt_name="query") 
document_embeddings = model.encode(documents)

# Compute cosine similarity scores
scores = model.similarity(query_embeddings, document_embeddings)

# Output the results
for query, query_scores in zip(queries, scores):
    doc_score_pairs = list(zip(documents, query_scores))
    doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
    print("Query:", query)
    for document, score in doc_score_pairs:
        print(score, document)

Using Huggingface Transformers

import torch
from transformers import AutoModel, AutoTokenizer

# Load the model
# Please use bf16 when inferring with half precision
model_name = 'dragonkue/snowflake-arctic-embed-l-v2.0-ko'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, add_pooling_layer=False)
model.eval()

# Define the queries and documents
query_prefix = 'query: '
queries  = ['대한민국의 수도는 어디인가?', '한글을 만든 사람은 누구인가?']
queries_with_prefix = ["{}{}".format(query_prefix, i) for i in queries]
query_tokens = tokenizer(queries_with_prefix, padding=True, truncation=True, return_tensors='pt', max_length=8192)

documents = ['대한민국의 수도는 서울이다.', '한글은 세종대왕이 창제하였다.']
document_tokens = tokenizer(documents, padding=True, truncation=True, return_tensors='pt', max_length=8192)

# Compute token embeddings
with torch.no_grad():
    query_embeddings = model(**query_tokens)[0][:, 0]
    document_embeddings = model(**document_tokens)[0][:, 0]

# Normalize embeddings
query_embeddings = torch.nn.functional.normalize(query_embeddings, p=2, dim=1)
document_embeddings = torch.nn.functional.normalize(document_embeddings, p=2, dim=1)

scores = torch.mm(query_embeddings, document_embeddings.transpose(0, 1))

for query, query_scores in zip(queries, scores):
    doc_score_pairs = list(zip(documents, query_scores))
    doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
    # Output passages & scores
    print("Query:", query)
    for document, score in doc_score_pairs:
        print(score, document)

📚 Documentation

Model Details

Model Description

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Evaluation

• This evaluation references the KURE GitHub repository. (https://github.com/nlpai-lab/KURE)
• We conducted an evaluation on all Korean Retrieval Benchmarks registered in MTEB.

Korean Retrieval Benchmark

• Ko - StrategyQA: A Korean ODQA multi - hop retrieval dataset, translated from StrategyQA.
• AutoRAGRetrieval: A Korean document retrieval dataset constructed by parsing PDFs from five domains: finance, public, medical, legal, and commerce.
• MIRACLRetrieval: A Korean document retrieval dataset based on Wikipedia.
• PublicHealthQA: A retrieval dataset focused on medical and public health domains in Korean.
• BelebeleRetrieval: A Korean document retrieval dataset based on FLORES - 200.
• MrTidyRetrieval: A Wikipedia - based Korean document retrieval dataset.
• MultiLongDocRetrieval: A long - document retrieval dataset covering various domains in Korean.
• XPQARetrieval: A cross - domain Korean document retrieval dataset.

Metrics

• Standard metric: NDCG@10

Information Retrieval

• Achieves state - of - the - art (SOTA) performance across various benchmarks.
• For each benchmark, the highest score is highlighted in bold, and the second - highest score is italicized.

Capabilities Beyond Benchmarks

• Supports phrase - based queries in addition to full - sentence queries. Example: "What products does Samsung sell?" or "Samsung's products"
• Trained to handle diverse query formats, regardless of phrasing variations. Example: "Tell me about Samsung.", "I'm curious about Samsung.", "What is Samsung?"
• Optimized for Markdown table search, allowing retrieval of answers embedded within tables when present in documents.
• Efficient clustering without hard negatives:
  • Samples within the same batch are clustered together.
  • Uses efficient embedding formation for clustering by truncating embeddings from the Snowflake/snowflake - arctic - embed - l - v2.0 model to 256 dimensions.
  • The clustering approach is inspired by the findings in the following papers:
    • Embedding And Clustering Your Data Can Improve Contrastive Pretraining
    • CONTEXTUAL DOCUMENT EMBEDDINGS
• Strong performance across different domains:
  • The Arctic - Embed 2.0: Multilingual Retrieval Without Compromise paper states:
    "While models like mE5, mGTE, and BGE - M3 excel on MIRACL, their performance on CLEF is notably weaker compared to ours and closed - source offerings, suggesting the potential of overfitting to MIRACL or its Wikipedia - based domain."
  • Based on personal experience, Snowflake/snowflake - arctic - embed - l - v2.0 has consistently outperformed BGE - M3 in different domains, further validating this observation.

Bias, Risks and Limitations

To prevent excessive GPU usage costs, the model was trained with a maximum sequence length of 1300 tokens. As a result, its performance may degrade on benchmarks like MultiLongDocRetrieval (MLDR).

The previous model, BGE - m3 - ko, was trained with a token length of 1024, which imposed limitations on its MLDR benchmark performance.

In the case of snowflake - arctic - embed - l - v2.0 - ko, if the document length exceeds 1300 tokens or approximately 2500 characters, it is recommended to consider the following models instead.

Training Details

• Loss: CachedGISTEmbedLoss with specific parameters.

Training Hyperparameters

Non - Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 20000
• per_device_eval_batch_size: 4096
• learning_rate: 2e - 05
• num_train_epochs: 2
• lr_scheduler_type: warmup_stable_decay
• lr_scheduler_kwargs: {'num_decay_steps': 160}
• warmup_ratio: 0.05
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Framework Versions

📄 License

Arctic is licensed under the Apache 2.0. The released models can be used for commercial purposes free of charge.

Citation

BibTeX

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084"
}

@misc{KURE,
  publisher = {Youngjoon Jang, Junyoung Son, Taemin Lee},
  year = {2024},
  url = {https://github.com/nlpai-lab/KURE}
}

@article{yu2024arcticembed,
  title = "Arctic-Embed 2.0: Multilingual Retrieval Without Compromise",
  author = "Puxuan Yu, Luke Merrick, Gaurav Nuti, Daniel Campos",
  journal = "arXiv preprint arXiv:2412.04506",
  year = "2024",
  url = "https://arxiv.org/abs/2412.04506"
}

@article{merrick2024embedding,
  title = "Embedding And Clustering Your Data Can Improve Contrastive Pretraining",
  author = "Luke Merrick",
  journal = "arXiv preprint arXiv:2407.18887",
  year = "2024",
  url = "https://arxiv.org/abs/2407.18887"
}

@article{morris2024contextual,
  title = "Contextual Document Embeddings",
  author = "John X. Morris, Alexander M. Rush",
  journal = "arXiv preprint arXiv:2410.02525",
  year = "2024",
  url = "https://arxiv.org/abs/2410.02525"
}