AfroLlama_V1 Open-Source Language Model - Free Processing of Multilingual Texts such as Swahili and Content Generation

Afrollama V1

Developed by Jacaranda

AfroLlama3 is an 8-billion-parameter language model optimized from Meta-Llama-3-8B, specifically enhanced for text processing and generation in Swahili, English, Zulu, Xhosa, Yoruba, and Hausa.

Large Language Model

Transformers

Supports Multiple Languages#African Multilingual Optimization #Culturally Sensitive Generation #LoRA Fusion Architecture

Downloads 122

Release Time : 7/11/2024

Model Overview

This model aims to improve natural language understanding and generation for users of the six target languages while retaining the general language capabilities of the original Llama 3.

Model Features

Multilingual Optimization

Specifically optimized for six African languages to enhance text understanding and generation capabilities.

Three-Stage Training

Adopts a three-stage training process: single-language pre-training based on LoRA, model fusion, and instruction fine-tuning.

Cultural Sensitivity

Emphasizes culturally sensitive understanding and incorporates safety datasets to reduce biases.

Retained General Capabilities

Preserves the general language capabilities of the original Llama 3 while optimizing for specific languages.

Model Capabilities

Multilingual Text Generation

Domain-Specific Q&A

Culturally Sensitive Content Understanding

Instruction Following

Use Cases

Industry Applications

Medical Q&A System

Provides multilingual medical information Q&A services for African regions.

Educational Assistant

Supports multilingual educational content generation and Q&A.

Cultural Content Creation

Multilingual Story Generation

Generates fables and stories incorporating cultural elements.

Compared to the original model, it can generate content more aligned with cultural backgrounds.

🚀 AfroLlama3

AfroLlama3 is an 8B-parameter language model optimized for Swahili, Xhosa, Zulu, Yoruba, Hausa, and English. It enhances natural language understanding and generation for users in these languages.

🚀 Quick Start

Use this notebook to perform inference. We recommend using an L4 or A100 GPU for inference.

✨ Features

Multilingual Proficiency: Improved performance on Swahili, Xhosa, Zulu, Yoruba, Hausa, and English language tasks.
General Language Capabilities: Maintains the general language capabilities of the original Llama 3 model.
Optimized for Understanding and Generation: Specifically optimized for both understanding and generating text in the target languages.

📦 Installation

Not provided in the original document, so this section is skipped.

💻 Usage Examples

Not provided in the original document, so this section is skipped.

📚 Documentation

Model Details

AfroLlama3 is an 8B-parameter language model built upon the foundation of meta-llama/Meta-Llama-3-8B. It has been specifically enhanced to excel in processing and generating text in Swahili, Xhosa, Zulu, Yoruba, Hausa, and English. This model aims to improve natural language understanding and generation capabilities for users and researchers speaking these languages.

Model Description

Key features:

Improved performance on Swahili, Xhosa, Zulu, Yoruba, Hausa, and English language tasks.
Maintains general language capabilities of the original Llama 3 model.
Optimized for both understanding and generating Swahili, Xhosa, Zulu, Yoruba, Hausa, and English text.

Training The training process for AfroLlama involved three main phases:

1. LoRA-based Individual Languages Continual Pre-training: We conducted continuous pre-training using publicly available corpora in each individual language. The primary focus was on causal language modeling, specifically training the model to predict the next tokens based on preceding tokens for each target language. Our continuous pre-training involved implementing the LoRA technique, where we froze the base model parameters of the foundation Meta/Llama3 model and introduced additional lightweight components (adapters). These adapters were specifically trained to capture the intricacies, terminologies, and nuances of each language. The LoRA hyperparameters were kept the same across the different languages.

2. Merging The language-specific pretrained checkpoints were merged to create one standalone model using a Ties (Trim, Elect, and Sign)-Based merging recipe. The three steps of Ties are: i) Resetting parameters that changed insignificantly during fine-tuning (Trim) - involves identifying parameters that didn't change much during the continual-pretraining process and resetting them to their original values. This can help reduce overfitting and improve generalization. ii) Resolving sign conflicts (interference) - addressing situations where different training examples or objectives cause conflicting updates to parameter signs, potentially leading to interference or reduced model performance. iii) Merging only the parameters that are in alignment with the final agreed-upon sign - combining or updating only those parameters whose signs are consistent with a consensus direction (Majority Sign), potentially helping to reduce noise and improve overall model coherence.

3. LoRA-based Instruction Tuning:

The merged model was fine-tuned on a curated dataset of Swahili, Xhosa, Zulu, Yoruba, Hausa, and English instructions and responses.
Included task-specific data to improve performance on common language tasks.
Emphasized maintaining coherence and contextual understanding in Swahili, Xhosa, Zulu, Yoruba, Hausa, and English.
Incorporated safety datasets to improve the model's ability to generate safe and ethical responses.
Included examples of harmful content and appropriate non-harmful alternatives.
Focused on reducing biases and improving the model's understanding of cultural sensitivities in the Swahili, Xhosa, Zulu, Yoruba, Hausa, and English context.

Approximate dataset sizes: Instruction tuning: [331,400 instruction-response pairs]

Developed by: Jacaranda Health

Property	Details
Model Type	Llama
Language(s) (NLP)	Swahili, Xhosa, Zulu, Yoruba, Hausa, and English
License	CC BY-NC-SA 4.0 DEED
Model Developers	Stanslaus Mwongela, Jay Patel, Sathy Rajasekharan, Lyvia Lusiji, Francesco Piccino, Mfoniso Ukwak, Ellen Sebastian

Uses

AfroLlama is optimized for downstream tasks, notably those demanding instructional datasets in Swahili, Xhosa, Zulu, Yoruba, Hausa, and English. Organizations can further fine-tune it for their specific domains. Potential areas include:

Question-answering within specific domains.
Assistant-driven chat capabilities: healthcare, agriculture, legal, education, tourism and hospitality, public services, financial sectors, communication, customer assistance, commerce, etc.

Sample Evaluation Prompts

Prompt	AfroLlama Response	Meta-Llama-3-8B-Instruct Response
Swahili: Andika hadithi ya bwana sungura (Translation: Write a story about Mr. Rabbit)	Kulikuwa na wakati, kulikuwa na sungura mwenye hekima aliyeitwa Bwana Sungura. Alikuwa kiumbe kilicho na hekima zaidi katika msitu, na alikuwa anajulikana kwa ushauri wake wa busara na hekima. Sungura alikuwa ameishi msituni kwa miaka mingi, na hivyo alikuwa amejifunza mengi kuhusu dunia inayomzunguka. Kila siku, alitumia muda wake kuchunguza msitu, kukutana na viumbe tofauti, na kujifunza habari mpya. Sungura alikuwa rafiki sana na kila mtu aliyekutana naye. Ikiwa mtu alihitaji ushauri au mwongozo, wangemtafuta Bwana Sungura. Pia alikuwa mkarimu na mwenye kusaidia, akiwapa watu vidokezo vyenye hekima na ufahamu. Sungura alikuwa pia mlinzi mkubwa wa msitu. Alilinda miti na mimea kutokana na madhara, na kuhakikisha kuwa hakuna mtu aliyepita njia yake bila kujua. Bwana Sungura alikuwa mpendwa na wengi, na urithi wake utaendelea kuishi kupitia hadithi na hadithi ambazo zimepitiwa kupitia vizazi. (Translation: Once upon a time, there was a wise rabbit named Mr. Rabbit. He was the wisest creature in the forest, known for his sage advice and wisdom. The rabbit had lived in the forest for many years, learning much about the world around him. Every day, he spent his time exploring the forest, meeting different creatures, and learning new information. Mr. Rabbit was very friendly with everyone he met. If anyone needed advice or guidance, they would seek out Mr. Rabbit. He was also generous and helpful, offering wise and insightful tips. Mr. Rabbit was also a great protector of the forest. He guarded the trees and plants from harm, ensuring no one passed his path without knowledge. Mr. Rabbit was loved by many, and his legacy would continue to live through the stories and tales passed down through generations.)	Andika hadithi ya bwana sungura!
Yoruba: Kọ itan nipa Ọgbọn Ehoroa (Translation: Tell the story of Mr.Rabbit)	Lọjọ kan, ọkan igba kan, ọkan agbọnrin kan wa ti o ni ẹmi aláìlègbé. O nigbagbogbo fẹ lati mọ nipa aye ti o wa ita ilu kekere rẹ ati awọn iyanu ti o le ri. Ọjọ kan, o pinnu lati bẹrẹ irin-ajo nla kan. O rin irin-ajo lọ si awọn ibi jijinna ati pe o ri ọpọlọpọ iyanu, bii awọn oke-nla, awọn odo, ati awọn igbo. O paapaa pade awọn ẹda ijinlẹ, bii dragoni ati awọn ẹranko idan. Lẹhin irin-ajo pipẹ, o pada de ilu kekere rẹ o si gbe pẹlu inu didun ati ọpẹ fun ohun gbogbo ti o ti ri. ((Translation: Once upon a time, there was a hare who had an adventurous spirit. He always wanted to know about the world outside his small village and the wonders he could see. One day, he decided to embark on a great journey. He traveled to distant places and saw many wonders, like mountains, rivers, and forests. He even met mystical creatures like dragons and magical animals. After a long journey, he returned to his small village, living happily and gratefully for all he had seen.)	I think you're asking me to tell a story about Ogbọn Ehoroa. Here's a short story: Ogbọn Ehoroa was a wise and kind-hearted woman who lived in a small village surrounded by lush green forests and rolling hills. She was known throughout the village for her exceptional cooking skills, particularly her delicious palm wine soup. One day, a severe drought hit the land, and the villagers were struggling to find food and water. The crops were withering away, and the rivers were drying up. The villagers were worried and didn't know what to do. Ogbọn Ehoroa called a meeting with the village elders and proposed a plan to find a solution to their problem. She suggested that they work together to build a new irrigation system to bring water from a nearby stream to the parched fields. The villagers were skeptical at first, but Ogbọn Ehoroa's persistence and determination eventually won them over. They worked tirelessly for days, digging trenches and building canals. And to everyone's surprise, the plan worked! The crops began to grow again, and the villagers had enough food to eat. As a token of gratitude, the villagers decided to celebrate Ogbọn Ehoroa's bravery and ingenuity. They prepared a grand feast, complete with her famous palm wine soup. And from that day on, Ogbọn Ehoroa was hailed as a hero and a symbol of hope and resilience. What do you think? Would you like me to continue the story or start a new one?
Hausa: Rubuta labarin Mista Zomo (Translation: Write a story about Mr. Rabbit)	Da wani lokaci, akwai wani zomo mai suna Mr. Zomo wanda yake rayuwa a cikin wani babban gida. Yana da matukar farin ciki kuma yana son wasa, amma sau da yawa yakan ji kadaitaka saboda baya da wanda zai raba wasansa da shi. Wata rana, ya yanke shawarar fita ya nemo abokai. Ya yi tafiya zuwa wurare daban-daban, har ya hadu da wani zomo mai suna Daisy. Mr. Zomo da Daisy suka zama abokan juna nan take, kuma suka yi amfani da sauran ranar suna wasa tare. Tun daga wannan ranar, Mr. Zomo bai kara ji kadaitaka ba - koyaushe yana da aboki da zai raba masa farin cikinsa. ((Translation: Once upon a time, there was a rabbit named Mr. Zomo who lived in a big house. He was very happy and loved to play, but he often felt lonely because he had no one to share his games with. One day, he decided to go out and find friends. He traveled to different places until he met a rabbit named Daisy. Mr. Zomo and Daisy became friends instantly, spending the rest of the day playing together. From that day on, Mr. Zomo never felt lonely again - he always had a friend to share his joy with.)	I see you're asking about the Hausa phrase "Rubuta labarin Mista Zomo". "Rubuta" means "to write", "labari" means "story" or "news", and "Mista Zomo" is likely a proper noun, possibly a title or a name. So, a possible translation of the phrase would be "Write the story of Mr. Zomo" or "Write Mr. Zomo's news". Could you please provide more context or information about what you're trying to achieve with this phrase? I'd be happy to help you further!
isiXhosa: Bhala ibali ngoMnumzana Mqhagi (Translation: Tell the story of Mr. Mqhagi)	Kwakudala-dala, kwakukho umntu ogama linguMnu. Scrooge owayengumntu ongathandi nto ngokupheleleyo. Wayenomdla omkhulu wemali kwaye wayezimisele ukuyigcina yonke imali yakhe. Wayehlala ekhetha iindlela zokugcina imali, nokuba kuthetha ukuzinciphisa iindleko ezincinane okanye ukubeka imali yakhe ebhankini engapheliyo. Ngenye imini, uMnu. Scrooge wafumanisa ukuba unamakhulu eedola zexabiso eliphezulu kunye nezinye izinto ezixabisekileyo. Wazaliswa luvuyo kwaye wagqiba kwelokuba asebenzise le mali ukwenza utyalomali oluhle. Ngoko ke, wakha ishishini eliyimpumelelo kwaye waba sisityebi.UMnu. Scrooge waba ngumntu onobutyebi kwaye wayekhululekile ngobomi bakhe obutsha. Kodwa njengoko ixesha lihamba, uMnu. Scrooge waqonda ukuba ubutyebi bakhe abuzange banelise. Wayefuna ukuba nesixa esingaphezulu semali, kodwa akukho mntu wayenokumnceda ekufumaneni. Ekugqibeleni, wafumana ukuzaliseka ngokuba nemali engaphezulu, kodwa wafumanisa ukuba ayizange ibe yinto eninzi kangako. UMnu. Scrooge wafunda ukuba ubutyebi abubanga ulonwabo lokwenene. Yonke into awayeyifumene yayiluncedo nje, kwaye akukho nto ingakumbi. ((Translation: Long ago, there was a man named Mr. Scrooge who was completely uninterested in anything. He had a great interest in money and was determined to keep all his wealth. He was always finding ways to save money, even if it meant cutting down on small expenses or keeping his money in an eternal bank.One day, Mr. Scrooge discovered he had hundreds of valuable dollars and other precious items. Filled with joy, he decided to use this money for good investments. He built a successful business and became wealthy. Mr. Scrooge became a wealthy man and was content with his new life. But as time passed, Mr. Scrooge realized that his wealth never truly satisfied him. He wanted to have more money, but no one could help him find it. Eventually, he found fulfillment by having more money, but discovered it was never that much. Mr. Scrooge learned that wealth is not true happiness. Everything he gained was just a help, and there is nothing more.)	The provided response seems to be incomplete in the original text.

🔧 Technical Details

The training process details, including the three - phase training (LoRA - based individual languages continual pre - training, merging, and LoRA - based instruction tuning), along with the Ties - Based merging recipe steps, provide in - depth technical information about how the AfroLlama3 model was developed.

📄 License

This model is released under the [CC BY - NC - SA 4.0 DEED](http://creativecommons.org/licenses/by - nc - sa/4.0/).

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご