Japanese-e5-mistral-7b_slerp Open-Source Japanese Text Embedding Model - Accurately Handle Sentence Similarity Tasks

Japanese E5 Mistral 7b Slerp

Developed by oshizo

A Japanese text embedding model created by merging E5 Mistral 7B Instruct and Japanese StableLM Base Gamma 7B, focusing on sentence similarity tasks

Text Embedding

Transformers

JapaneseOpen Source License:MIT #Japanese sentence embedding #Hybrid model architecture #Text similarity calculation

Downloads 94

Release Time : 1/4/2024

Model Overview

This model combines the embedding generation capability of E5 Mistral with the Japanese language understanding of Japanese StableLM, suitable for semantic similarity calculation and embedding representation generation in Japanese text

Model Features

Bilingual model fusion

Combines the embedding generation capability of E5 Mistral with the Japanese language understanding advantage of Japanese StableLM

Layered parameter fusion

Uses more of the Japanese model at the input layer and more of the embedding model at the output layer, optimizing performance

Optimized merging method

Adopts the slerp merging method, the best configuration selected after comparing various methods

Model Capabilities

Japanese text embedding generation

Sentence similarity calculation

Semantic search

Use Cases

Information retrieval

Japanese document similarity matching

Finding semantically similar documents in a collection of Japanese documents

Question answering systems

Japanese FAQ matching

Semantically matching user questions with a FAQ database

🚀 Sentence Similarity Model

This model is designed for sentence similarity tasks, created by merging two powerful models to enhance performance.

🚀 Quick Start

This model was created by merging intfloat/e5-mistral-7b-instruct and stabilityai/japanese-stablelm-base-gamma-7b.
See intfloat/e5-mistral-7b-instruct page or evaluation notebook of oshizo/JapaneseEmbeddingEval for model usage.

💻 Usage Examples

Basic Usage

The steps to merge are as follows.

Step 1: Load and Save intfloat/e5-mistral-7b-instruct

Load intfloat/e5-mistral-7b-instruct as a "MistralForCausalLM" class and save_pretrained as is.

Because e5-mistral-7b-instruct is made with the "MistralModel" class, it could not be merged with "MistraForCausalLM" as is.
In my environment, I had to load into the CPU, not the GPU, or I would get an error.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "intfloat/e5-mistral-7b-instruct"
model = AutoModelForCausalLM.from_pretrained(model_id)#, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

model.save_pretrained("./e5-mistral-7b-instruct_with_lm_head")

Step 2: Merge using mergekit

Merge using mergekit with the following yaml configuration

merge_config.yaml

models:
  - model: stabilityai/japanese-stablelm-base-gamma-7b
  - model: ./e5-mistral-7b-instruct_with_lm_head
base_model: stabilityai/japanese-stablelm-base-gamma-7b
parameters:
  t:
    - value: [0.5, 0.9]

merge_method: slerp
dtype: float16

I tried the "linear", "slerp", and "task_arithmetic" merging methods, and this setting seemed to be the best.
The choice of "t" parameters was set to use more japanese-stablelm-base-gamma-7b for the layer closer to the input to take advantage of Japanese word understanding, and more e5-mistral-7b-instruct for the layer closer to the output to generate good embeddings.
As for the "ties" method, I could not find any parameters for density and weight that worked properly.

Step 3: Copy Pad Token Settings

Copy settings related to pad_token from the e5-mistral-7b-instruct repository.

config.json
tokenizer.json
tokenizer.model
tokenizer_config.json
special_tokens_map.json

📄 License

This project is licensed under the MIT License.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご