Language Detection

🚀 Language Detection Model

A BERT-based language detection model trained on hac541309/open-lid-dataset, which includes 121 million sentences across 200 languages. This model is optimized for fast and accurate language identification in text classification tasks.

✨ Features

• Multilingual Support: Supports languages such as English, French, German, Spanish, Arabic, and Greek.
• High Performance: Achieves high precision, recall, F1-score, and accuracy in language detection.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("alexneakameni/language_detection")
model = AutoModelForSequenceClassification.from_pretrained("alexneakameni/language_detection")

language_detection = pipeline("text-classification", model=model, tokenizer=tokenizer)

text = "Hello world!"
predictions = language_detection(text)
print(predictions)

This will output the predicted language code or label with the corresponding confidence score.

📚 Documentation

Model Details

Training Process

• Dataset:
  • Used the open-lid-dataset.
  • Split into train (90%) and test (10%).
• Tokenizer: A custom BertTokenizerFast with special tokens for [UNK], [CLS], [SEP], [PAD], [MASK].
• Hyperparameters:
  • Learning Rate: 2e-5.
  • Batch Size: 256 (training) / 512 (testing).
  • Epochs: 1.
  • Scheduler: Cosine.
• Trainer: Leveraged the Hugging Face Trainer API with Weights & Biases for logging.

Data Augmentation

To improve model generalization and robustness, a new text augmentation strategy was introduced. This includes:

• Removing digits (random probability).
• Shuffling words to introduce variation.
• Removing words selectively.
• Adding random digits to simulate noise.
• Modifying punctuation to handle different text formats.

Impact of Augmentation

Adding these augmentations improved overall model performance, as seen in the latest evaluation results:

Evaluation

Updated Performance Metrics

Detailed Evaluation (~12 millions texts)

Comparison with Previous Performance

After introducing text augmentations, the model's performance improved on the same evaluation dataset, with accuracy increasing from 0.9695 to 0.9733, along with similar improvements in average precision, recall, and F1 score.

Conclusion

The integration of new text augmentation techniques has led to a measurable improvement in model accuracy and robustness. These enhancements allow for better generalization across diverse language scripts, improving the model’s usability in real-world applications.

A detailed per-script classification report is also provided in the repository for further analysis.

📄 License

This model is licensed under the MIT license.

⚠️ Important Note

The model’s performance may vary depending on text length, language variety, and domain-specific vocabulary. Always validate results against your own datasets for critical applications.

💡 Usage Tip

For more information, see the repository documentation.

Thank you for using this model—feedback and contributions are welcome!