Gliner Model Merge Large V1.0

license: apache-2.0 language:

• en metrics:
• f1
• recall
• precision tags:
• NER pipeline_tag: token-classification library_name: gliner datasets:
• knowledgator/GLINER-multi-task-synthetic-data
• EmergentMethods/AskNews-NER-v0
• urchade/pile-mistral-v0.1
• MultiCoNER/multiconer_v2
• DFKI-SLT/few-nerd base_model: knowledgator/gliner-multitask-large-v0.5

The xomad/gliner-model-merge-large-v1.0 model is developed from the pretrained model knowledgator/gliner-multitask-large-v0.5 to explore the capabilities of model merging techniques, resulting in a significant performance boost of 3.25 points, elevating the model's capability from 0.6276 to 0.6601 F1-score.

The model is trained exclusively on datasets with commercial-friendly licenses to ensure broad applicability under the Apache-2.0 license. The following datasets were used in the training process:

• knowledgator/GLINER-multi-task-synthetic-data
• EmergentMethods/AskNews-NER-v0
• urchade/pile-mistral-v0.1
• MultiCoNER/multiconer_v2
• DFKI-SLT/few-nerd

⚙️ Finetuning process

The process begins with the base model knowledgator/gliner-multitask-large-v0.5. Our model xomad/gliner-model-merge-large-v1.0 is fine-tuned separately on each of the above datasets , and we save multiple checkpoints along the fine-tuning process. We put all these checkpoints together into a pool and then we apply the Model soups technique to produce different merged models:

• uniform_merged
• greedy_on_random
• greedy_on_sorted

Following this, we apply WiSE-FT merging technique to pairs of models selected from a group of the above 3 models and the original model to produce the wise_ft_merged model. This concludes the 1st finetuning phase.

The process is then repeated in the 2nd finetuning phase, using the wise_ft_merged as the new starting point, to produce the final model. The whole finetuning flow is illustrated in the following figure:

The performance of the pool of fine-tuned models and the merged models are evaluated on the CrossNER, TwitterNER benchmarks, and plotted in the following 2 figures (as crossner_f1 and other_f1 respectively).

The 1st finetuning phase plot:

The 2nd finetuning phase plot:

🛠️ Installation

To use this model, you must install the GLiNER Python library:

pip install gliner

Once you've downloaded the GLiNER library, you can import the GLiNER class. You can then load this model using GLiNER.from_pretrained.

💻 Usage

from gliner import GLiNER

model = GLiNER.from_pretrained("xomad/gliner-model-merge-large-v1.0")

text = """
Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800. During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014.
"""

labels = ["founder", "computer", "software", "position", "date", "company"]

entities = model.predict_entities(text, labels)

for entity in entities:
    print(entity["text"], "=>", entity["label"])

Output:

Microsoft => company
Bill Gates => founder
Paul Allen => founder
April 4, 1975 => date
BASIC => software
Altair 8800 => computer
Microsoft => company
chairman => position
chief executive officer => position
president => position
chief software architect => position
May 2014 => date

📊 Benchmarks:

Performance on different zero-shot NER benchmarks (CrossNER, mit-movie and mit-restaurant), numbers reported from https://huggingface.co/knowledgator/gliner-multitask-large-v0.5:

Detailed performance on different datasets:

Authors

Hoan Nguyen, at xomad.com

Citations

@misc{wortsman2022modelsoupsaveragingweights,
      title={Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time}, 
      author={Mitchell Wortsman and Gabriel Ilharco and Samir Yitzhak Gadre and Rebecca Roelofs and Raphael Gontijo-Lopes and Ari S. Morcos and Hongseok Namkoong and Ali Farhadi and Yair Carmon and Simon Kornblith and Ludwig Schmidt},
      year={2022},
      eprint={2203.05482},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2203.05482}, 
}

@InProceedings{Wortsman_2022_CVPR,
    author    = {Wortsman, Mitchell and Ilharco, Gabriel and Kim, Jong Wook and Li, Mike and Kornblith, Simon and Roelofs, Rebecca and Lopes, Raphael Gontijo and Hajishirzi, Hannaneh and Farhadi, Ali and Namkoong, Hongseok and Schmidt, Ludwig},
    title     = {Robust Fine-Tuning of Zero-Shot Models},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {7959-7971}
}

@misc{stepanov2024gliner,
      title={GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks}, 
      author={Ihor Stepanov and Mykhailo Shtopko},
      year={2024},
      eprint={2406.12925},
      archivePrefix={arXiv},
      primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}
}

@misc{zaratiana2023gliner,
      title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer}, 
      author={Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
      year={2023},
      eprint={2311.08526},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}