đ lang-uk/ukr-paraphrase-multilingual-mpnet-base
This is a fine-tuned sentence-transformers model for the Ukrainian language. It maps sentences and paragraphs into a 768-dimensional dense vector space and can be applied to tasks such as clustering or semantic search. The original model for fine-tuning is sentence-transformers/paraphrase-multilingual-mpnet-base-v2
. For more details, refer to our paper Contextual Embeddings for Ukrainian: A Large Language Model Approach to Word Sense Disambiguation.
đ Quick Start
⨠Features
- Multilingual Support: Supports multiple languages including ar, bg, ca, cs, da, de, el, en, es, et, fa, fi, fr, gl, gu, he, hi, hr, hu, hy, id, it, ja, ka, ko, ku, lt, lv, mk, mn, mr, ms, my, nb, nl, pl, pt, ro, ru, sk, sl, sq, sr, sv, th, tr, uk, ur, vi.
- Sentence Similarity: Ideal for tasks like sentence similarity, clustering, and semantic search.
- Fine-tuned for Ukrainian: Specifically fine-tuned for the Ukrainian language to provide better performance.
đĻ Installation
To use this model, you need to install the sentence-transformers
library. You can install it using the following command:
pip install -U sentence-transformers
đģ Usage Examples
Basic Usage
If you have sentence-transformers installed, using this model is straightforward:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('lang-uk/ukr-paraphrase-multilingual-mpnet-base')
embeddings = model.encode(sentences)
print(embeddings)
Advanced Usage
Without sentence-transformers, you can use the model as follows. First, pass your input through the transformer model, then apply the appropriate pooling operation on top of the contextualized word embeddings.
from transformers import AutoTokenizer, AutoModel
import torch
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output[0]
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
sentences = ['This is an example sentence', 'Each sentence is converted']
tokenizer = AutoTokenizer.from_pretrained('lang-uk/ukr-paraphrase-multilingual-mpnet-base')
model = AutoModel.from_pretrained('lang-uk/ukr-paraphrase-multilingual-mpnet-base')
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
model_output = model(**encoded_input)
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
print("Sentence embeddings:")
print(sentence_embeddings)
đ Documentation
- Supported Languages: The model supports a wide range of languages, including but not limited to Ukrainian, French (Canada), Portuguese (Brazil), Chinese (Simplified), and Chinese (Traditional).
- Model Details: The model is fine-tuned on the
sentence-transformers/paraphrase-multilingual-mpnet-base-v2
architecture.
đ License
This project is licensed under the Apache-2.0 license.
Citing & Authors
If you find this model helpful, please cite our publication Contextual Embeddings for {U}krainian: A Large Language Model Approach to Word Sense Disambiguation:
@inproceedings{laba-etal-2023-contextual,
title = "Contextual Embeddings for {U}krainian: A Large Language Model Approach to Word Sense Disambiguation",
author = "Laba, Yurii and
Mudryi, Volodymyr and
Chaplynskyi, Dmytro and
Romanyshyn, Mariana and
Dobosevych, Oles",
editor = "Romanyshyn, Mariana",
booktitle = "Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)",
month = may,
year = "2023",
address = "Dubrovnik, Croatia",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.unlp-1.2",
doi = "10.18653/v1/2023.unlp-1.2",
pages = "11--19",
abstract = "This research proposes a novel approach to the Word Sense Disambiguation (WSD) task in the Ukrainian language based on supervised fine-tuning of a pre-trained Large Language Model (LLM) on the dataset generated in an unsupervised way to obtain better contextual embeddings for words with multiple senses. The paper presents a method for generating a new dataset for WSD evaluation in the Ukrainian language based on the SUM dictionary. We developed a comprehensive framework that facilitates the generation of WSD evaluation datasets, enables the use of different prediction strategies, LLMs, and pooling strategies, and generates multiple performance reports. Our approach shows 77,9{\%} accuracy for lexical meaning prediction for homonyms.",
}
Copyright: Yurii Laba, Volodymyr Mudryi, Dmytro Chaplynskyi, Mariana Romanyshyn, Oles Dobosevych, Ukrainian Catholic University, lang-uk project, 2023
The original model used for fine-tuning was trained by sentence-transformers.