ModernBERT-Morocco Open Source Model - The uses are yet to be revealed. Enjoy potential diverse applications for free.

Modernbert Morocco

Developed by BounharAbdelaziz

This model is a fine-tuned version of ModernBERT-base on an unspecified dataset. Specific use cases and performance metrics require further details

Large Language Model

Transformers

Open Source License:Apache-2.0 #BERT fine-tuning #High learning rate optimization #Gradient accumulation training

Downloads 23

Release Time : 2/20/2025

Model Overview

A fine-tuned model based on the ModernBERT-base architecture, with functionality dependent on the fine-tuning task

Model Features

Based on ModernBERT architecture

Uses ModernBERT-base as the foundation model, potentially incorporating modern BERT architectural improvements

Efficient fine-tuning

Trained with relatively large batch sizes (total training batch size 128) and gradient accumulation

Model Capabilities

Text understanding

Text representation learning

Use Cases

Natural Language Processing

Text classification

Applicable for text classification tasks

Performance metrics to be added

Question answering systems

Potentially suitable for question answering applications

Performance metrics to be added

🚀 Model

This model is a fine - tuned version of [answerdotai/ModernBERT - base](https://huggingface.co/answerdotai/ModernBERT - base) on an unknown dataset. It achieves the following results on the evaluation set:

Loss: nan

📚 Documentation

Training and Evaluation Data

More information needed

Model Description

More information needed

Intended Uses & Limitations

More information needed

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.005
train_batch_size: 8
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 128
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon = 1e - 08 and optimizer_args = No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.07
num_epochs: 2

Training Results

Training Loss	Epoch	Step	Validation Loss
125.0167	0.0109	100	nan
112.5359	0.0219	200	nan
109.896	0.0328	300	nan
107.4339	0.0437	400	nan
103.1406	0.0546	500	nan
99.518	0.0656	600	nan
98.1404	0.0765	700	nan
96.4646	0.0874	800	nan
95.7492	0.0983	900	nan
93.8507	0.1093	1000	nan
93.7277	0.1202	1100	nan
92.1633	0.1311	1200	nan
91.0273	0.1420	1300	nan
90.2002	0.1530	1400	nan
89.4479	0.1639	1500	nan
89.1879	0.1748	1600	nan
86.8561	0.1857	1700	nan
86.2537	0.1967	1800	nan
86.8297	0.2076	1900	nan
84.6928	0.2185	2000	nan
83.4784	0.2294	2100	nan
83.5887	0.2404	2200	nan
83.9307	0.2513	2300	nan
81.3527	0.2622	2400	nan
81.4105	0.2731	2500	nan
81.1048	0.2841	2600	nan
79.4346	0.2950	2700	nan
80.1727	0.3059	2800	nan
80.3314	0.3169	2900	nan
79.3279	0.3278	3000	nan
78.772	0.3387	3100	nan
77.1061	0.3496	3200	nan
77.3927	0.3606	3300	nan
77.128	0.3715	3400	nan
77.3792	0.3824	3500	nan
76.9679	0.3933	3600	nan
75.4298	0.4043	3700	nan
76.2873	0.4152	3800	nan
75.4714	0.4261	3900	nan
75.3966	0.4370	4000	nan
75.2704	0.4480	4100	nan
74.7007	0.4589	4200	nan
74.1831	0.4698	4300	nan
73.9942	0.4807	4400	nan
74.2908	0.4917	4500	nan
73.3644	0.5026	4600	nan
73.0533	0.5135	4700	nan
72.1435	0.5244	4800	nan
71.8705	0.5354	4900	nan
73.3312	0.5463	5000	nan
72.0031	0.5572	5100	nan
70.9734	0.5682	5200	nan
71.031	0.5791	5300	nan
71.2214	0.5900	5400	nan
70.7596	0.6009	5500	6.3902
71.2633	0.6119	5600	nan
70.3307	0.6228	5700	nan
70.0143	0.6337	5800	nan
70.7308	0.6446	5900	nan
69.6832	0.6556	6000	nan
69.295	0.6665	6100	nan
69.426	0.6774	6200	nan
69.9395	0.6883	6300	nan
68.4942	0.6993	6400	nan
69.5833	0.7102	6500	nan
68.3381	0.7211	6600	nan
68.4515	0.7320	6700	nan
68.0571	0.7430	6800	nan
68.1398	0.7539	6900	nan
67.5816	0.7648	7000	nan
66.0035	0.7757	7100	nan
67.7892	0.7867	7200	nan
67.9904	0.7976	7300	nan
65.9595	0.8085	7400	nan
66.0176	0.8194	7500	nan
66.3258	0.8304	7600	nan
65.9997	0.8413	7700	nan
67.0377	0.8522	7800	nan
66.2209	0.8632	7900	nan
66.2458	0.8741	8000	6.0199
65.5858	0.8850	8100	nan
65.111	0.8959	8200	nan
64.9051	0.9069	8300	nan
65.771	0.9178	8400	nan
65.3083	0.9287	8500	nan
65.3556	0.9396	8600	nan
64.592	0.9506	8700	nan
65.2071	0.9615	8800	nan
64.3542	0.9724	8900	nan
65.0919	0.9833	9000	nan
64.5229	0.9943	9100	nan
63.9692	1.0051	9200	nan
63.5139	1.0161	9300	nan
63.5847	1.0270	9400	nan
63.8988	1.0379	9500	nan
62.3398	1.0488	9600	nan
63.8375	1.0598	9700	nan
63.8011	1.0707	9800	nan
62.4506	1.0816	9900	nan
62.933	1.0925	10000	nan
62.813	1.1035	10100	nan
62.0427	1.1144	10200	nan
63.0628	1.1253	10300	nan
61.3597	1.1362	10400	nan
61.9852	1.1472	10500	nan
62.4618	1.1581	10600	nan
61.7416	1.1690	10700	nan
61.8847	1.1800	10800	nan
62.2208	1.1909	10900	nan
62.0095	1.2018	11000	nan
60.6946	1.2127	11100	nan
61.4203	1.2237	11200	nan
61.7838	1.2346	11300	nan
61.991	1.2455	11400	nan
61.5899	1.2564	11500	nan
59.8005	1.2674	11600	nan
60.7846	1.2783	11700	nan
60.5796	1.2892	11800	nan
61.5156	1.3001	11900	nan
60.3144	1.3111	12000	nan
60.2115	1.3220	12100	nan
60.368	1.3329	12200	nan
60.7462	1.3438	12300	nan
61.1936	1.3548	12400	6.0033
60.9203	1.3657	12500	nan
59.5265	1.3766	12600	nan
59.978	1.3875	12700	nan
60.6729	1.3985	12800	nan
60.7364	1.4094	12900	nan
59.8604	1.4203	13000	nan
60.1816	1.4312	13100	nan
61.0396	1.4422	13200	nan
59.6997	1.4531	13300	nan
59.7544	1.4640	13400	nan
60.2458	1.4750	13500	nan
59.4263	1.4859	13600	nan
60.1375	1.4968	13700	nan
59.4983	1.5077	13800	nan
58.9182	1.5187	13900	nan
59.2961	1.5296	14000	nan
58.4649	1.5405	14100	nan
58.5321	1.5514	14200	nan
58.7082	1.5624	14300	nan
59.5857	1.5733	14400	nan
59.2364	1.5842	14500	nan
58.8255	1.5951	14600	nan
60.2955	1.6061	14700	nan
58.1949	1.6170	14800	nan
59.6096	1.6279	14900	nan
58.7729	1.6388	15000	nan
58.2987	1.6498	15100	nan
58.6004	1.6607	15200	nan
58.4145	1.6716	15300	nan
58.9517	1.6825	15400	nan
58.9631	1.6935	15500	nan
58.2923	1.7044	15600	nan
58.7865	1.7153	15700	nan
58.2494	1.7262	15800	nan
58.7492	1.7372	15900	nan
57.9321	1.7481	16000	nan
58.8437	1.7590	16100	nan
58.5637	1.7700	16200	nan
58.5184	1.7809	16300	nan
57.9655	1.7918	16400	nan
58.9973	1.8027	16500	nan
57.7771	1.8137	16600	nan
58.8119	1.8246	16700	nan
58.2166	1.8355	16800	nan
58.9727	1.8464	16900	nan
58.1561	1.8574	17000	nan
58.7419	1.8683	17100	nan
59.0596	1.8792	17200	nan
57.1149	1.8901	17300	nan
59.1509	1.9011	17400	nan
58.8787	1.9120	17500	nan
58.0355	1.9229	17600	nan
58.4026	1.9338	17700	nan
58.0197	1.9448	17800	nan
57.3607	1.9557	17900	nan
58.7545	1.9666	18000	5.6811
57.3768	1.9775	18100	nan
58.3111	1.9885	18200	nan
58.4388	1.9994	18300	nan

Framework Versions

Transformers 4.49.0
Pytorch 2.6.0+cu124
Datasets 2.21.0
Tokenizers 0.21.0

📄 License

The model is licensed under the apache - 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご