Compoundpiece

🚀 Compound Normalization Model

This is a compound normalization model from CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models, which can be used to handle compound words in text.

🚀 Quick Start

The model can be used directly through the transformers library's pipeline function.

💻 Usage Examples

Basic Usage

from transformers import pipeline

pipe = pipeline("text2text-generation", "benjamin/compoundpiece")

pipe("Hauswirtschaftslehre", max_length=32)
# [{'generated_text': 'Haus-Wirtschaft-Lehre'}]

📚 Documentation

Supported Languages

The model supports the following languages:

• af, az, be, bg, bn, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gl, gu, he, hi, hu, hy, id, is, it, ka, kk, ky, la, lt, lv, mg, mk, ml, mt, nl, pa, pl, pt, ro, ru, sk, sq, sv, ta, te, th, tr, uk, yi, yo

Datasets

The model is trained on the dataset benjamin/compoundpiece.

📄 License

This project is licensed under the MIT license.

📖 Citation

If you use this model in your research, please cite the following paper:

@article{minixhofer2023compoundpiece,
  title={CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models},
  author={Minixhofer, Benjamin and Pfeiffer, Jonas and Vuli{\'c}, Ivan},
  journal={arXiv preprint arXiv:2305.14214},
  year={2023}
}