Bloomz 560m Reranking

🚀 Bloomz-560m Reranking

This reranking model measures the semantic correspondence between a question and a context, supporting both French and English. It helps filter and reorder results in an ODQA context, despite high computational cost.

✨ Features

• Semantic Scoring: Measures the semantic match between a query and a context.
• Multilingual Support: Supports both French and English, performing well in cross - language contexts.
• Result Reordering: Reorders the results from a retriever more efficiently.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import pipeline

reranker = pipeline(
    task='text-classification',
    model='cmarkea/bloomz-560m-reranking',
    top_k=None
)

query: str
contexts: List[str]

similarities = reranker(
    [
        dict(
            text=context, # the model was trained with context in `text`
            text_pair=query # and query in `text_pair` argument.
        )
        for context in contexts
    ]
)

score_label_1 = [
    next(item['score'] for item in entry if item['label'] == 'LABEL_1') 
    for entry in similarities
]
contexts_reranked = sorted(
    zip(score_label_1, contexts),
    key=lambda x: x[0],
    reverse=True
)

score, contexts_cleaned = zip(
    *filter(
        lambda x: x[0] >= 0.8,
        contexts_reranked
    )
)

📚 Documentation

Dataset

The training dataset combines the mMARCO dataset, which consists of query/positive/hard negative triplets, and the "train" split data from SQuAD, also forming query/positive/hard negative triplets. To generate hard negative data for SQuAD, contexts from the same theme as the query but from a different set of queries are considered. The triplets are then flattened into pairs of query/context sentences, labeled 1 for query/positive and 0 for query/negative. The language (French or English) of each element in the pair is randomly and uniformly chosen.

Evaluation

To evaluate the reranker, the "validation" split of the SQuAD dataset is used. The first question from each paragraph and the corresponding paragraph as the context are selected. The evaluation corpus contains 1204 pairs of query/context to be ranked.

Monolingual Evaluation (French/French)

Cross - language Evaluation (French/English)

As shown, the cross - language context has little impact on the model's performance. A threshold of 0.8 can be applied to filter contexts in RAG - type applications.

Citation

@online{DeBloomzReranking,
  AUTHOR = {Cyrile Delestre},
  ORGANIZATION = {Cr{\'e}dit Mutuel Ark{\'e}a},
  URL = {https://huggingface.co/cmarkea/bloomz-3b-reranking},
  YEAR = {2024},
  KEYWORDS = {NLP ; Transformers ; LLM ; Bloomz},
}

📄 License

The model is licensed under the bigscience - bloom - rail - 1.0 license.