Albert Small Kor Cross Encoder V1

🚀 albert-small-kor-cross-encoder-v1

• This is a model fine - tuned as a cross - encoder by training the albert - small - kor - v1 model.
• This model was trained using SentenceTransformers Cross - Encoder class.

🚀 Quick Start

Pre - trained models can be used like this:

from sentence_transformers import CrossEncoder
model = CrossEncoder('bongsoo/albert-small-kor-cross-encoder-v1')
scores = model.predict([('오늘 날씨가 좋다', '오늘 등산을 한다'), ('오늘 날씨가 흐리다', '오늘 비가 내린다')])
print(scores)

[0.45417202 0.6294121 ]

The model will predict scores for the pairs ('Sentence 1', 'Sentence 2') and ('Sentence 3', 'Sentence 4').

You can use this model also without sentence_transformers and by just using Transformers AutoModel class

✨ Features

• This model is fine - tuned as a cross - encoder by training the albert - small - kor - v1 model.
• It was trained using the SentenceTransformers Cross - Encoder class.

🔧 Technical Details

Training

• The model was trained in the sequence of sts(10) - nli(3) - sts(10) - nli(3) - sts(10) (No distil training).
• STS Training:
  • Parameters: seed = 111, epoch = 10, lr = 1e - 4, eps = 1e - 6, warm_step = 10%, max_seq_len = 128, train_batch = 128 (32 for small model) (albert 13m/7G).
  • Training code
• NLI Training:
  • Parameters: seed = 111, epoch = 3, lr = 3e - 5, eps = 1e - 8, warm_step = 10%, max_seq_len = 128, train_batch = 64, eval_bath = 64 (albert 2h/7G).
  • Training code
• Evaluation code, Test code

Performance

📄 License

This model is released under the Apache - 2.0 license.