Stella Pl Retrieval

🚀 Stella-PL-retrieval

This is a text encoder designed for Polish information retrieval tasks. It's built upon stella_en_1.5B_v5 and further fine - tuned for better performance in Polish.

✨ Features

• Multilingual Adaptation: In the first step, we adapted the model for Polish using the multilingual knowledge distillation method with a diverse corpus of 20 million Polish - English text pairs.
• Contrastive Fine - Tuning: In the second step, we fine - tuned the model with contrastive loss using a dataset of 1.4 million queries. Positive and negative passages for each query were selected with the help of [BAAI/bge - reranker - v2.5 - gemma2 - lightweight](https://huggingface.co/BAAI/bge - reranker - v2.5 - gemma2 - lightweight) reranker. The model was trained for three epochs with a batch size of 1024 queries.
• Vector Encoding: The encoder transforms texts into 1024 - dimensional vectors, optimized specifically for Polish information retrieval tasks. For more versatile tasks like semantic similarity or clustering, consider using the distilled version [sdadas/stella - pl](https://huggingface.co/sdadas/stella - pl).

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

The model utilizes the same prompts as the original stella_en_1.5B_v5.

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer(
    "sdadas/stella-pl-retrieval",
    trust_remote_code=True,
    device="cuda",
    model_kwargs={"attn_implementation": "flash_attention_2", "trust_remote_code": True}
)
model.bfloat16()

# Retrieval example
query_prefix = "Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: "
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
    "Trzeba zdrowo się odżywiać i uprawiać sport.",
    "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)
best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])

# Semantic similarity example
sim_prefix = "Instruct: Retrieve semantically similar text.\nQuery: "
sentences = [
    sim_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
    sim_prefix + "Warto jest prowadzić zdrowy tryb życia, uwzględniający aktywność fizyczną i dietę.",
    sim_prefix + "One should eat healthy and engage in sports.",
    sim_prefix + "Zakupy potwierdzasz PINem, który bezpiecznie ustalisz podczas aktywacji."
]
emb = model.encode(sentences, convert_to_tensor=True, show_progress_bar=False)
print(cos_sim(emb, emb))

📚 Documentation

• Prompt Usage:
  • For retrieval, queries should be prefixed with "Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: ".
  • For symmetric tasks such as semantic similarity, both texts should be prefixed with "Instruct: Retrieve semantically similar text.\nQuery: ".
• Model Loading: The model uses a custom implementation, so you should add trust_remote_code=True argument when loading it. It is also recommended to use Flash Attention 2, which can be enabled with attn_implementation argument.

🔧 Technical Details

The model is optimized for Polish information retrieval tasks. It first adapts to the Polish language through multilingual knowledge distillation and then fine - tunes with contrastive loss. The encoder transforms texts to 1024 - dimensional vectors.

📄 License

The license of this model is gemma.

📋 Model Information

📈 Evaluation Results

The model achieves NDCG@10 of 62.32 on the Polish Information Retrieval Benchmark. See PIRB Leaderboard for detailed results.

📖 Citation

@article{dadas2024pirb,
  title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods}, 
  author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata},
  year={2024},
  eprint={2402.13350},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}