Lsg 4096 Sentence Similarity Spanish

🚀 LSG Variant of hiiamsid/sentence_similarity_spanish_es

An enhanced sentence similarity model leveraging the LSG attention mechanism for efficient long - sequence processing

🚀 Quick Start

This model is an enhanced version of hiiamsid/sentence_similarity_spanish_es, now transformed using Local Sparse Global (LSG) attention mechanism. The adaptation to LSG allows for efficient handling of longer sequences, making the model more versatile and robust in a wider range of natural language processing tasks. This LSG adaptation enables the model to efficiently process sequences up to 4096 tokens in length.

✨ Features

About the LSG architecture

LSG (Local Sparse Global) attention is a cutting-edge approach designed to mitigate the limitations of the traditional self-attention mechanism in Transformer models, particularly for processing long sequences. By incorporating local, sparse, and global attention, LSG attention significantly reduces computational complexity while maintaining, and often enhancing, model performance.

Model adaptation

This LSG variant has been adapted from the original model with the primary goal of extending its capabilities to efficiently handle longer text inputs. This enhancement enables the model to maintain high accuracy and efficiency, even with extended sequence lengths that were previously challenging for the base model.

Use cases

The LSG-enhanced model is particularly adept at tasks involving embeddings for longer documents.

💻 Usage Examples

Basic Usage

import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained('prudant/lsg_4096_sentence_similarity_spanish')
model = AutoModel.from_pretrained('prudant/lsg_4096_sentence_similarity_spanish', trust_remote_code=True)

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences
sentences = [
  'Esa es una persona feliz',
  "Ese es un perro feliz",
  "Esa es una persona muy feliz",
  "Hoy es un día soleado",
  "Esa es una persona alegre",
  ]

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

# Norm embeddings
normalized_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)

# Sentence similarity
cosine_similarities = F.cosine_similarity(normalized_embeddings[0].unsqueeze(0), normalized_embeddings[1:], dim=1)

print(cosine_similarities)

The output of sentence embeddings and cosine similarities are as follows:

Sentence embeddings:

tensor([[-0.1691, -0.2517, -1.3000,  ...,  0.1557,  0.3824,  0.2048],
        [ 0.1872, -0.7604, -0.4863,  ..., -0.4922, -0.1511, -0.8539],
        [-0.2467, -0.2373, -1.1708,  ...,  0.4637,  0.0616,  0.2841],
        [-0.2384,  0.1681, -0.3498,  ..., -0.2744, -0.1722, -1.2513],
        [ 0.2273, -0.2393, -1.6124,  ...,  0.6065,  0.2784, -0.3354]])

tensor([0.5132, 0.9346, 0.3471, 0.8543])

📄 License

This model is released under the apache - 2.0 license.

👏 Acknowledgments

This model has been adapted by Darío Muñoz Prudant, thanks to the Hugging Face community and contributors to the LSG attention mechanism for their resources and support.

📋 Model Information