Silver Retriever Base V1

🚀 Silver Retriever Base (v1)

The Silver Retriever model encodes Polish sentences or paragraphs into a 768 - dimensional dense vector space. It can be used for tasks like document retrieval or semantic search, offering an effective solution for information retrieval in the Polish language.

🚀 Quick Start

The Silver Retriever model is initialized from the HerBERT - base model and fine - tuned on the PolQA and MAUPQA datasets for 15,000 steps with a batch size of 1,024. For more details, please refer to the SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering.

✨ Features

• Encoding Capability: Encodes Polish sentences or paragraphs into a 768 - dimensional dense vector space.
• Task Suitability: Ideal for document retrieval and semantic search tasks.
• Fine - Tuned: Fine - tuned on specific Polish datasets for better performance.

📦 Installation

If you want to use this model with sentence - transformers, you can install it using the following command:

pip install -U sentence-transformers

💻 Usage Examples

Basic Usage

Using Sentence - Transformers

from sentence_transformers import SentenceTransformer
sentences = [
    "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", 
    "Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg."
]

model = SentenceTransformer('ipipan/silver-retriever-base-v1')
embeddings = model.encode(sentences)
print(embeddings)

Using HuggingFace Transformers

from transformers import AutoTokenizer, AutoModel
import torch


def cls_pooling(model_output, attention_mask):
    return model_output[0][:,0]


# Sentences we want sentence embeddings for
sentences = [
    "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", 
    "Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg."
]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1')
model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, cls pooling.
sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

📚 Documentation

Evaluation

Legend:

• Acc is the Accuracy at 10
• NDCG is the Normalized Discounted Cumulative Gain at 10

Usage

Preparing inputs

The model was trained on question - passage pairs and works best when the input is in the same format as that used during training:

• Add the phrase Pytanie: to the beginning of the question.
• The training passages consist of title and text concatenated with the special token </s>. Even if your passages don't have a title, it is still beneficial to prefix a passage with the </s> token.
• Although the dot product was used during training, the model usually works better with the cosine distance.

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)

Additional Information

Model Creators

The model was created by Piotr Rybak from the Institute of Computer Science, Polish Academy of Sciences.

This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00 - 00C002/19.

Licensing Information

CC BY - SA 4.0

Citation Information

@inproceedings{rybak-ogrodniczuk-2024-silver-retriever,
    title = "Silver Retriever: Advancing Neural Passage Retrieval for {P}olish Question Answering",
    author = "Rybak, Piotr  and
      Ogrodniczuk, Maciej",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1291",
    pages = "14826--14831",
    abstract = "Modern open-domain question answering systems often rely on accurate and efficient retrieval components to find passages containing the facts necessary to answer the question. Recently, neural retrievers have gained popularity over lexical alternatives due to their superior performance. However, most of the work concerns popular languages such as English or Chinese. For others, such as Polish, few models are available. In this work, we present Silver Retriever, a neural retriever for Polish trained on a diverse collection of manually or weakly labeled datasets. Silver Retriever achieves much better results than other Polish models and is competitive with larger multilingual models. Together with the model, we open-source five new passage retrieval datasets.",
}