Gte Qwen1.5 7B Instruct

🚀 gte-Qwen1.5-7B-instruct

gte-Qwen1.5-7B-instruct 是 gte 嵌入模型家族的最新成员。该模型基于 Qwen1.5-7B 大语言模型开发，充分借鉴了 Qwen1.5-7B 模型强大的自然语言处理能力。通过我们先进的嵌入训练技术进行优化，该模型具备了以下关键改进：

• 集成双向注意力机制，增强上下文理解能力。
• 仅在查询端进行指令微调，提高效率。
• 在涵盖不同领域和场景的大规模多语言文本语料库上进行全面训练。这种训练结合了弱监督和监督数据，确保模型适用于多种语言和众多下游任务。

我们还推出了 gte-base-en-v1.5 和 gte-large-en-v1.5，这两款英语嵌入模型在 MTEB 基准测试中，于相同模型规模类别下取得了最优成绩，并且支持最长达 8192 的上下文长度。

✨ 主要特性

• 强大的上下文理解：通过双向注意力机制，模型能够更深入地理解文本的上下文信息，从而提供更准确的嵌入表示。
• 高效的指令微调：仅在查询端进行指令微调，避免了不必要的计算开销，提高了模型的运行效率。
• 多语言支持：在大规模多语言语料库上进行训练，使得模型能够处理多种语言的文本，适用于全球范围内的应用场景。
• 广泛的下游任务适用性：经过全面的训练，模型能够在多种下游任务中表现出色，如文本分类、检索、聚类等。

📦 安装指南

运行该模型需要安装以下库：

transformers>=4.39.2
flash_attn>=2.5.6

💻 使用示例

Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Alibaba-NLP/gte-Qwen1.5-7B-instruct", trust_remote_code=True)
# 如果你想减少最大长度：
model.max_seq_length = 8192

queries = [
    "how much protein should a female eat",
    "summit define",
]
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "Definition of summit for English Language Learners. : 1  the highest point of a mountain : the top of a mountain. : 2  the highest level. : 3  a meeting or series of meetings between the leaders of two or more governments.",
]

query_embeddings = model.encode(queries, prompt_name="query")
document_embeddings = model.encode(documents)

scores = (query_embeddings @ document_embeddings.T) * 100
print(scores.tolist())
# [[70.00668334960938, 8.184843063354492], [14.62419319152832, 77.71407318115234]]

你可以查看 config_sentence_transformers.json 了解所有预定义的提示名称。此外，你也可以使用 model.encode(queries, prompt="Instruct: ...\nQuery: " 来使用自定义提示。

Transformers

import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def last_token_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'


# 每个查询都必须附带一个描述任务的单句指令
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, 'summit define')
]
# 检索文档无需添加指令
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "Definition of summit for English Language Learners. : 1  the highest point of a mountain : the top of a mountain. : 2  the highest level. : 3  a meeting or series of meetings between the leaders of two or more governments."
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen1.5-7B-instruct', trust_remote_code=True)
model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen1.5-7B-instruct', trust_remote_code=True)

max_length = 8192

# 对输入文本进行分词
batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# 对嵌入进行归一化处理
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[70.00666809082031, 8.184867858886719], [14.62420654296875, 77.71405792236328]]

📚 详细文档

模型信息

评估

MTEB & C-MTEB

你可以使用 scripts/eval_mteb.py 脚本来复现 gte-Qwen1.5-7B-instruct 模型在 MTEB（英语）/C-MTEB（中文）上的以下评估结果：

📄 许可证

本项目采用 Apache 2.0 许可证。

📖 引用

如果你觉得我们的论文或模型有帮助，请考虑引用以下文献：

@article{li2023towards,
  title={Towards general text embeddings with multi-stage contrastive learning},
  author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},
  journal={arXiv preprint arXiv:2308.03281},
  year={2023}
}