Pearl Small

🚀 PEARL-small

PEARL-small is a lightweight string embedding model. It excels at semantic similarity computation for strings and is ideal for tasks like string matching, entity retrieval, entity clustering, and fuzzy join. Unlike typical sentence embedders, it incorporates phrase type information and morphological features, better capturing string variations. It's a variant of E5-small, finetuned on a context-free dataset to generate superior phrase and string representations.

🚀 Quick Start

PEARL-small offers an efficient solution for string - related tasks. It's based on the research paper Learning High - Quality and General - Purpose Phrase Representations and is accepted by EACL Findings 2024. The model is developed by Lihu Chen, [Gaël Varoquaux](https://gael - varoquaux.info/), and Fabian M. Suchanek.

✨ Features

• Lightweight: With a relatively small size, it can be deployed efficiently.
• Innovative Design: Incorporates phrase type information and morphological features for better string representation.
• Versatile: Suitable for multiple string - related tasks such as string matching, entity retrieval, etc.

📚 Documentation

Model Comparison

Cost Comparison

Cost comparison of FastText and PEARL. The estimated memory is calculated by the number of parameters (float16). The unit of inference speed is *ms/512 samples. The FastText model here is crawl - 300d - 2M - subword.bin.

💻 Usage Examples

Basic Usage

Sentence Transformers

from sentence_transformers import SentenceTransformer, util

query_texts = ["The New York Times"]
doc_texts = [ "NYTimes", "New York Post", "New York"]
input_texts = query_texts + doc_texts

model = SentenceTransformer("Lihuchen/pearl_small")
embeddings = model.encode(input_texts)
scores = util.cos_sim(embeddings[0], embeddings[1:]) * 100
print(scores.tolist())
# [[90.56318664550781, 79.65763854980469, 75.52056121826172]]

Transformers

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]

def encode_text(model, input_texts):
    # Tokenize the input texts
    batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

    outputs = model(**batch_dict)
    embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
    
    return embeddings


query_texts = ["The New York Times"]
doc_texts = [ "NYTimes", "New York Post", "New York"]
input_texts = query_texts + doc_texts

tokenizer = AutoTokenizer.from_pretrained('Lihuchen/pearl_small')
model = AutoModel.from_pretrained('Lihuchen/pearl_small')

# encode
embeddings = encode_text(model, input_texts)

# calculate similarity
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())

# expected outputs
# [[90.56318664550781, 79.65763854980469, 75.52054595947266]]

📄 License

This project is licensed under the Apache - 2.0 license.

📚 Training and Evaluation

Have a look at our code on Github

📚 Citation

If you find our work useful, please give us a citation:

@inproceedings{chen2024learning,
  title={Learning High - Quality and General - Purpose Phrase Representations},
  author={Chen, Lihu and Varoquaux, Gael and Suchanek, Fabian},
  booktitle={Findings of the Association for Computational Linguistics: EACL 2024},
  pages={983--994},
  year={2024}
}

Useful Links

🤗 PEARL - small 🤗 PEARL - base 📐 PEARL Benchmark 🏆 PEARL Leaderboard