Lodestone-base-4096-v1 Open Source Model - Free Support for Long Text Embedding for Semantic Search and Clustering

Lodestone Base 4096 V1

Developed by Hum-Works

A sentence-transformers model developed by Hum, supporting 4096 tokens long text embedding, suitable for semantic search and clustering tasks

Text Embedding

PyTorch

EnglishOpen Source License:Apache-2.0 #Long Text Embedding #ALiBi Extrapolation #Semantic Search Optimization

Downloads 132

Release Time : 8/25/2023

Model Overview

An innovative long-text encoder based on Transformer architecture, integrating FlashAttention, ALiBi, and GLU technologies, capable of mapping sentences and paragraphs to a 768-dimensional vector space

Model Features

Ultra-long Context Support

Extended to 4096 tokens input length via ALiBi technology, suitable for processing long documents

Efficient Attention Mechanism

Integrated FlashAttention optimizes computational efficiency, supporting automatic invocation of Triton high-performance implementation

Lightweight Design

Can run on GPU/CPU, balancing performance and resource consumption

Multi-source Training Data

Fine-tuned on 1.5 billion sentence pairs from multiple domains (academic, Q&A, community discussions, etc.)

Model Capabilities

Text Vectorization

Semantic Similarity Calculation

Information Retrieval

Text Clustering

Use Cases

Knowledge Management

Academic Literature Retrieval

Embeddings trained on S2ORC data can be used for paper recommendation systems

Community Content Processing

Q&A Pair Matching

Identify similar questions on platforms like StackExchange

🚀 lodestone-base-4096-v1

This is a model for sentence similarity tasks. It has been trained on a wide range of datasets and has shown performance on various MTEB tasks.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

📚 Documentation

Model Information

Property	Details
Model Type	Sentence Transformer for Sentence Similarity
Training Data	s2orc, flax-sentence-embeddings/stackexchange_title_body_jsonl, flax-sentence-embeddings/stackexchange_titlebody_best_voted_answer_jsonl, flax-sentence-embeddings/stackexchange_title_best_voted_answer_jsonl, flax-sentence-embeddings/stackexchange_titlebody_best_and_down_voted_answer_jsonl, sentence-transformers/reddit-title-body, msmarco, gooaq, yahoo_answers_topics, code_search_net, search_qa, eli5, snli, multi_nli, wikihow, natural_questions, trivia_qa, embedding-data/sentence-compression, embedding-data/flickr30k-captions, embedding-data/altlex, embedding-data/simple-wiki, embedding-data/QQP, embedding-data/SPECTER, embedding-data/PAQ_pairs, embedding-data/WikiAnswers, sentence-transformers/embedding-training-data

Property

Details

Model Type

Sentence Transformer for Sentence Similarity

Training Data

s2orc, flax-sentence-embeddings/stackexchange_title_body_jsonl, flax-sentence-embeddings/stackexchange_titlebody_best_voted_answer_jsonl, flax-sentence-embeddings/stackexchange_title_best_voted_answer_jsonl, flax-sentence-embeddings/stackexchange_titlebody_best_and_down_voted_answer_jsonl, sentence-transformers/reddit-title-body, msmarco, gooaq, yahoo_answers_topics, code_search_net, search_qa, eli5, snli, multi_nli, wikihow, natural_questions, trivia_qa, embedding-data/sentence-compression, embedding-data/flickr30k-captions, embedding-data/altlex, embedding-data/simple-wiki, embedding-data/QQP, embedding-data/SPECTER, embedding-data/PAQ_pairs, embedding-data/WikiAnswers, sentence-transformers/embedding-training-data

Model Performance

The model lodestone-base-4096-v1 has been evaluated on multiple tasks and datasets. Here are some of the key results:

Classification Tasks

Task	Dataset	Accuracy	AP	F1
Classification	MTEB AmazonCounterfactualClassification (en)	69.7313432835821	31.618259511417733	63.30313825394228
Classification	MTEB AmazonPolarityClassification	86.89837499999999	82.39500885672128	86.87317947399657
Classification	MTEB AmazonReviewsClassification (en)	44.05	N/A	42.67624383248947
Classification	MTEB Banking77Classification	75.40584415584415	N/A	74.29514617572676

Retrieval Tasks

Task	Dataset	MAP@1	MAP@10	MAP@100	MAP@1000	MRR@1	MRR@10	MRR@100	MRR@1000	NDCG@1	NDCG@10	NDCG@100	NDCG@1000	Precision@1	Precision@10	Precision@100	Precision@1000	Recall@1	Recall@10	Recall@100	Recall@1000
Retrieval	MTEB ArguAna	26.173999999999996	40.976	42.067	42.075	26.814	41.252	42.337	42.345	26.173999999999996	49.819	54.403999999999996	54.59	26.173999999999996	7.838000000000001	0.9820000000000001	0.1	26.173999999999996	78.378	98.222	99.644
Retrieval	MTEB CQADupstackAndroidRetrieval	26.595000000000002	36.556	37.984	38.134	32.761	41.799	42.526	42.582	32.761	42.549	47.915	50.475	32.761	8.312	1.403	0.197	26.595000000000002	54.332	76.936	93.914
Retrieval	MTEB CQADupstackEnglishRetrieval	22.528000000000002	30.751	31.855	31.972	28.662	35.912	36.726	36.777	28.662	35.452	40.1	42.323	28.662	6.688	1.13	0.16	22.528000000000002	43.748	64.235	78.609
Retrieval	MTEB CQADupstackGamingRetrieval	33.117999999999995	44.339	45.367000000000004	45.437	38.37	47.786	48.522	48.567	38.37	50.019999999999996	54.36299999999999	55.897	38.37	8.288	1.139	0.132	33.117999999999995	63.451	82.767	93.786
Retrieval	MTEB CQADupstackGisRetrieval	16.028000000000002	23.186999999999998	24.236	24.337	17.514	24.84	25.838	25.924999999999997	17.514	27.391	32.684999999999995	35.367	17.514	4.463	0.745	0.101	16.028000000000002	38.81	63.295	83.762
Retrieval	MTEB CQADupstackMathematicaRetrieval	11.962	17.218	18.321	18.455	14.677000000000001	20.381	21.471999999999998	N/A	14.677000000000001	N/A	N/A	N/A	14.677000000000001	N/A	N/A	N/A	11.962	N/A	N/A	N/A

Clustering Tasks

Task	Dataset	V-Measure
Clustering	MTEB ArxivClusteringP2P	42.31689035788179
Clustering	MTEB ArxivClusteringS2S	31.280245136660984
Clustering	MTEB BiorxivClusteringP2P	37.41860080664014
Clustering	MTEB BiorxivClusteringS2S	29.319217023090705

Reranking Tasks

Task	Dataset	MAP	MRR
Reranking	MTEB AskUbuntuDupQuestions	58.79109720839415	71.79615705931495

STS Tasks

Task	Dataset	Cosine Similarity Pearson	Cosine Similarity Spearman	Euclidean Pearson	Euclidean Spearman	Manhattan Pearson	Manhattan Spearman
STS	MTEB BIOSSES	76.44918756608115	70.86607256286257	74.12154678100815	70.86607256286257	74.0078626964417	70.68353828321327

🔧 Technical Details

No specific technical implementation details are provided in the original document, so this section is skipped.

📄 License

The model is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご