Armenian Text Embeddings 1
Armenian text embedding model optimized based on multilingual-e5-base, supporting semantic search and cross-lingual understanding
Text Embedding
Transformers Supports Multiple Languages#Armenian semantic search#Multilingual text embedding#RAG-enhanced generation

Downloads 578
Release Time : 11/12/2024
Model Overview
This model specializes in generating high-quality dense vector representations for Armenian text, suitable for tasks like retrieval-augmented generation (RAG), semantic search, and document similarity calculation
Model Features
Armenian language optimization
Specifically optimized for Armenian, providing more accurate semantic representations
Cross-lingual capability
Based on multilingual-e5-base, retaining cross-lingual understanding capabilities between English and Armenian
Efficient training
Utilizes DeepSpeed Stage 2 optimization and mixed-precision training to improve training efficiency
Weight averaging
Combines the strengths of base model (0.6) and fine-tuned model (0.4) to enhance performance
Model Capabilities
Armenian semantic search
Document similarity calculation
Cross-lingual text understanding
Text classification
Information retrieval
Use Cases
Information retrieval
Armenian recipe search
Match user queries with the most relevant traditional Armenian recipes
Can accurately match queries with relevant recipe content
Health consultation
Nutrition information retrieval
Answer health-related questions such as protein intake
Can accurately match scientific advice with user queries
🚀 Armenian-Text-Embeddings-1
This model provides text embeddings for the Armenian language, enabling various NLP tasks such as semantic search, document similarity computation, and text classification.
🚀 Quick Start
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('Metric-AI/armenian-text-embeddings-1')
model = AutoModel.from_pretrained('Metric-AI/armenian-text-embeddings-1')
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = [
'query: Ինչպե՞ս պատրաստել տոլմա', # How to make tolma
'query: Քանի՞ գրամ սպիտակուց է հարկավոր օրական', # How many grams of protein needed daily
"""passage: Տոլմայի բաղադրատոմս՝
Բաղադրիչներ՝
- 500գ աղացած միս
- 1 բաժակ բրինձ
- Խաղողի տերևներ
- 2 գլուխ սոխ
- Համեմունքներ՝ աղ, սև պղպեղ, քարի
Պատրաստման եղանակը՝
1. Միսը խառնել բրնձի, մանր կտրատած սոխի և համեմունքների հետ
2. Խաղողի տերևները լվանալ և թողնել տաք ջրի մեջ 10 րոպե
3. Լցոնել տերևները և դասավորել կաթսայի մեջ
4. Եփել դանդաղ կրակի վրա 45-60 րոպե""", # Detailed tolma recipe
"""passage: Սպիտակուցի օրական չափաբաժինը կախված է մարդու քաշից, սեռից և ֆիզիկական ակտիվությունից:
Միջին հաշվով, կանանց համար խորհուրդ է տրվում 46-50 գրամ սպիտակուց օրական:
Մարզիկների համար այս թիվը կարող է հասնել մինչև 1.6-2 գրամ մարմնի քաշի յուրաքանչյուր կիլոգրամի համար:
Հղիների համար պահանջվում է լրացուցիչ 25 գրամ սպիտակուց:
Սպիտակուցի հարուստ աղբյուրներ են՝
- Հավի միս (31գ/100գ)
- Ձու (13գ/100գ)
- Ոսպ (25գ/100գ)
- Մածուն (3.5գ/100գ)"""] # Detailed protein intake advice
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[83.96063232421875, 30.283924102783203], [32.504661560058594, 82.4246826171875]]
✨ Features
- Retrieval-augmented generation (RAG)
- Semantic search in Armenian
- Document similarity computation
- Cross-lingual text understanding
- Text classification tasks
- Information retrieval
💻 Usage Examples
Basic Usage
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('Metric-AI/armenian-text-embeddings-1')
model = AutoModel.from_pretrained('Metric-AI/armenian-text-embeddings-1')
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = [
'query: Ինչպե՞ս պատրաստել տոլմա', # How to make tolma
'query: Քանի՞ գրամ սպիտակուց է հարկավոր օրական', # How many grams of protein needed daily
"""passage: Տոլմայի բաղադրատոմս՝
Բաղադրիչներ՝
- 500գ աղացած միս
- 1 բաժակ բրինձ
- Խաղողի տերևներ
- 2 գլուխ սոխ
- Համեմունքներ՝ աղ, սև պղպեղ, քարի
Պատրաստման եղանակը՝
1. Միսը խառնել բրնձի, մանր կտրատած սոխի և համեմունքների հետ
2. Խաղողի տերևները լվանալ և թողնել տաք ջրի մեջ 10 րոպե
3. Լցոնել տերևները և դասավորել կաթսայի մեջ
4. Եփել դանդաղ կրակի վրա 45-60 րոպե""", # Detailed tolma recipe
"""passage: Սպիտակուցի օրական չափաբաժինը կախված է մարդու քաշից, սեռից և ֆիզիկական ակտիվությունից:
Միջին հաշվով, կանանց համար խորհուրդ է տրվում 46-50 գրամ սպիտակուց օրական:
Մարզիկների համար այս թիվը կարող է հասնել մինչև 1.6-2 գրամ մարմնի քաշի յուրաքանչյուր կիլոգրամի համար:
Հղիների համար պահանջվում է լրացուցիչ 25 գրամ սպիտակուց:
Սպիտակուցի հարուստ աղբյուրներ են՝
- Հավի միս (31գ/100գ)
- Ձու (13գ/100գ)
- Ոսպ (25գ/100գ)
- Մածուն (3.5գ/100գ)"""] # Detailed protein intake advice
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[83.96063232421875, 30.283924102783203], [32.504661560058594, 82.4246826171875]]
Advanced Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('Metric-AI/armenian-text-embeddings-1')
embeddings = model.encode(input_texts, normalize_embeddings=True)
📚 Documentation
Model Details
Property | Details |
---|---|
Model Name | Armenian-Text-Embeddings-1 |
Model Type | Text Embeddings for Armenian Language |
Base Model | intfloat/multilingual-e5-base |
Version | 1.0.0 |
License | Apache 2.0 |
Last Updated | November 2024 |
Model Architecture | Transformer-based embeddings model |
Input | Armenian text |
Output | Dense vector embeddings |
Training Data
Dataset Details
- Source: Reddit dataset with English-Armenian translations
- Size: 1.08M pairs of rows
- Content Type: Title and body text pairs
- Token Statistics:
- Training Set:
- Translated Title Tokens: 23,921,393
- Translated Body Tokens: 194,200,654
- Test Set:
- Translated Title Tokens: 242,443
- Translated Body Tokens: 1,946,164
- Training Set:
- Split Ratio: 99% train, 1% test
Training Procedure
Training Details
- Weight Averaging:
- Base model (multilingual-e5-base): 0.6 weight
- Fine-tuned model: 0.4 weight
- Training Duration: 2 days
- Hardware: 4 x NVIDIA A100 40GB GPUs
- Training Parameters:
- Epochs: 5
- Batch Size: 256 per GPU, (256*4 in total)
- Learning Rate: 5e-5
- Weight Decay: 0.01
- Warmup Steps: 1000
- Maximum Sequence Length: 128 tokens
- FP16 Training: Enabled
- Gradient Clipping: 1.0
Optimization Configuration
- Framework: DeepSpeed Stage 2
- Optimizer: AdamW with auto weight decay
- Mixed Precision: FP16 with dynamic loss scaling
- ZeRO Optimization: Stage 2 with:
- Allgather partitions
- Overlap communications
- Contiguous gradients
- Additional Features:
- Gradient checkpointing
- Tensor parallelism (size: 2)
Performance and Limitations
Capabilities
- Effective for semantic similarity tasks in Armenian
- Suitable for document classification and clustering
Limitations
- Performance may vary on domain-specific terminology
- May not capture Armenian-specific cultural contexts effectively
- Limited by the quality of training data translations
Known Biases
- May exhibit biases present in Reddit content
Environmental Impact
- Training Hardware: 4 x NVIDIA A100 40GB
- Training Duration: 48 hours
- Estimated Energy Consumption: 384 kWh (estimated based on A100 power consumption)
Ethical Considerations
- Data Privacy: Training data from public Reddit content
- Potential Misuse: Could be misused for content manipulation or spam
- Bias: May perpetuate social biases present in Reddit content
- Recommendations:
- Monitor system outputs for harmful content
- Implement content filtering for production use
- Regular bias assessment recommended
Technical Specifications
- Model Size: ~278M parameters (based on e5-base)
- Embedding Dimension: 384
- Max Sequence Length: 128 tokens
- Framework Compatibility:
- PyTorch
- Hugging Face Transformers
- DeepSpeed
🔧 Technical Details
The model is based on the intfloat/multilingual-e5-base model and fine-tuned on a Reddit dataset with English-Armenian translations. It uses a Transformer-based architecture to generate dense vector embeddings for Armenian text. The training process involves weight averaging, specific training parameters, and optimization configurations to achieve good performance on various NLP tasks.
📄 License
This model is licensed under the Apache 2.0 license.
📚 Citation
@misc{armenian-text-embeddings-1,
author = {Spartak Bughdaryan, Zaruhi Navasardyan, Bagrat Minasyan, Hrant Davtyan},
title = {Armenian-Text-Embeddings-1: Enhanced Armenian Language Embeddings},
year = {2024},
howpublished = {\url{https://metric.am/blog/announcing-armenian-text-embeddings/}}
}
Additional Information
Base Model References
- multilingual-e5-base: https://huggingface.co/intfloat/multilingual-e5-base
Acknowledgments
- intfloat for the original multilingual-e5-base model
- Reddit community for the source content
- DeepSpeed team for optimization toolkit
Version History
- 1.0.0 (November 2024): Initial release
Jina Embeddings V3
Jina Embeddings V3 is a multilingual sentence embedding model supporting over 100 languages, specializing in sentence similarity and feature extraction tasks.
Text Embedding
Transformers Supports Multiple Languages

J
jinaai
3.7M
911
Ms Marco MiniLM L6 V2
Apache-2.0
A cross-encoder model trained on the MS Marco passage ranking task for query-passage relevance scoring in information retrieval
Text Embedding English
M
cross-encoder
2.5M
86
Opensearch Neural Sparse Encoding Doc V2 Distill
Apache-2.0
A sparse retrieval model based on distillation technology, optimized for OpenSearch, supporting inference-free document encoding with improved search relevance and efficiency over V1
Text Embedding
Transformers English

O
opensearch-project
1.8M
7
Sapbert From PubMedBERT Fulltext
Apache-2.0
A biomedical entity representation model based on PubMedBERT, optimized for semantic relation capture through self-aligned pre-training
Text Embedding English
S
cambridgeltl
1.7M
49
Gte Large
MIT
GTE-Large is a powerful sentence transformer model focused on sentence similarity and text embedding tasks, excelling in multiple benchmark tests.
Text Embedding English
G
thenlper
1.5M
278
Gte Base En V1.5
Apache-2.0
GTE-base-en-v1.5 is an English sentence transformer model focused on sentence similarity tasks, excelling in multiple text embedding benchmarks.
Text Embedding
Transformers Supports Multiple Languages

G
Alibaba-NLP
1.5M
63
Gte Multilingual Base
Apache-2.0
GTE Multilingual Base is a multilingual sentence embedding model supporting over 50 languages, suitable for tasks like sentence similarity calculation.
Text Embedding
Transformers Supports Multiple Languages

G
Alibaba-NLP
1.2M
246
Polybert
polyBERT is a chemical language model designed to achieve fully machine-driven ultrafast polymer informatics. It maps PSMILES strings into 600-dimensional dense fingerprints to numerically represent polymer chemical structures.
Text Embedding
Transformers

P
kuelumbus
1.0M
5
Bert Base Turkish Cased Mean Nli Stsb Tr
Apache-2.0
A sentence embedding model based on Turkish BERT, optimized for semantic similarity tasks
Text Embedding
Transformers Other

B
emrecan
1.0M
40
GIST Small Embedding V0
MIT
A text embedding model fine-tuned based on BAAI/bge-small-en-v1.5, trained with the MEDI dataset and MTEB classification task datasets, optimized for query encoding in retrieval tasks.
Text Embedding
Safetensors English
G
avsolatorio
945.68k
29
Featured Recommended AI Models