Gbert Large Paraphrase Euclidean

🚀 German BERT large paraphrase euclidean

This is a sentence-transformers model that maps sentences and paragraphs (text) into a 1024-dimensional dense vector space. It's designed to be used with SetFit to enhance German few-shot text classification. There's also a sibling model named deutsche-telekom/gbert-large-paraphrase-cosine. This model is based on deepset/gbert-large, and we're very grateful to deepset!

🚀 Quick Start

This model is a sentence-transformers model. It maps sentences & paragraphs (text) into a 1024 dimensional dense vector space. The model is intended to be used together with SetFit to improve German few-shot text classification.

✨ Features

• Maps sentences and paragraphs into a 1024-dimensional dense vector space.
• Intended for use with SetFit to enhance German few-shot text classification.
• Has a sibling model deutsche-telekom/gbert-large-paraphrase-cosine.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples for using the model are provided in the original document, so this section is skipped.

📚 Documentation

Training

Loss Function

We have used BatchHardSoftMarginTripletLoss with eucledian distance as the loss function:

    train_loss = losses.BatchHardSoftMarginTripletLoss(
       model=model,
       distance_metric=BatchHardTripletLossDistanceFunction.eucledian_distance,
   )

Training Data

The model is trained on a carefully filtered dataset of deutsche-telekom/ger-backtrans-paraphrase. We deleted the following pairs of sentences:

• min_char_len less than 15
• jaccard_similarity greater than 0.3
• de_token_count greater than 30
• en_de_token_count greater than 30
• cos_sim less than 0.85

Hyperparameters

Evaluation Results

We use the NLU Few-shot Benchmark - English and German dataset to evaluate this model in a German few-shot scenario.

Qualitative results

• Multilingual sentence embeddings provide the worst results.
• Electra models also deliver poor results.
• German BERT base size model (deepset/gbert-base) provides good results.
• German BERT large size model (deepset/gbert-large) provides very good results.
• Our fine-tuned models (this model and deutsche-telekom/gbert-large-paraphrase-cosine) provide best results.

🔧 Technical Details

The model is based on deepset/gbert-large. The training process involves using BatchHardSoftMarginTripletLoss with euclidean distance as the loss function and training on a filtered dataset of deutsche-telekom/ger-backtrans-paraphrase.

📄 License

Copyright (c) 2023 Philip May, Deutsche Telekom AG. Copyright (c) 2022 deepset GmbH.

Licensed under the MIT License (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License by reviewing the file LICENSE in the repository.