language:

• sw
• en
• zu
• xh
• yo
• ha metrics:
• perplexity pipeline_tag: question-answering

Model Card for AfroLlama3

Use this notebook to inference. We recommend you inference on L4 or A100 GPU.

Model Details

AfroLlama3 is a 8B Parameters language model that builds upon the foundation of meta-llama/Meta-Llama-3-8B.It has been specifically enhanced to excel in processing and generating text in Swahili, Xhosa, Zulu, Yoruba, Hausa and English language. This model aims to improve natural language understanding and generation capabilities for Swahili, Xhosa, Zulu, Yoruba, Hausa and English speaking users and researchers.

Model Description

Key features:

• Improved performance on Swahili, Xhosa, Zulu, Yoruba, Hausa and English language tasks
• Maintains general language capabilities of the original Llama 3 model
• Optimized for both understanding and generating Swahili, Xhosa, Zulu, Yoruba, Hausa and English text

Training The training process for AfroLlama involved three main phases:

1. LoRA-based Individual Languages Continual Pre-training: We conducted continuous pre-training using publicly available corpi in each individual language, the primary focus was on causal language modeling,specifically training the model to predict the next tokens based on preceding tokens for each target language. Our continuous pre- training involved implementing the LoRA technique, where-in we froze the base model parameters of the foundation Meta/Llama3 model and introduced additional lightweight components(adpaters). adapters. These adapters were specifically trained to capture the intricacies, terminologies, and nuances of the each language. The LoRA hyperparams were kept same across the different languages.

2. Merging The language specific pretrained checkpoints were merged to create one standalone using a Ties(Trim, Elect and Sign)- Based merging recipe. The three steps of Ties are: i) Resetting parameters that changed insignificantly during fine-tuning(Trim)- involves identifying parameters that didn't change much during the continual-pretraining process and resetting them to their original values. This can help reduce overfitting and improve generalization. ii)Resolving sign conflicts (interference)- addressing situations where different training examples or objectives cause conflicting updates to parameter signs, potentially leading to interference or reduced model performance. iii) Merging only the parameters that are in alignment with the final agreed-upon sign-combining or updating only those parameters whose signs are consistent with a consensus direction(Majority Sign), potentially helping to reduce noise and improve overall model coherence.

3. LoRA-based Instruction Tuning:

• The merged model was fine-tuned on a curated dataset of Swahili, Xhosa, Zulu, Yoruba, Hausa and English instructions and responses
• Included task-specific data to improve performance on common language tasks
• Emphasized maintaining coherence and contextual understanding in Swahili, Xhosa, Zulu, Yoruba, Hausa and English
• Incorporated safety datasets to improve the model's ability to generate safe and ethical responses
• Included examples of harmful content and appropriate non-harmful alternatives
• Focused on reducing biases and improving the model's understanding of cultural sensitivities in the Swahili, Xhosa, Zulu, Yoruba, Hausa and English context Approximate dataset sizes:

Instruction tuning:[331,400 instruction-response pairs]

• Developed by: Jacaranda Health
• Model type: Llama
• Language(s) (NLP): Swahili, Xhosa, Zulu, Yoruba, Hausa and English
• License: CC BY-NC-SA 4.0 DEED
• Model Developers: Stanslaus Mwongela, Jay Patel, Sathy Rajasekharan, Lyvia Lusiji, Francesco Piccino, Mfoniso Ukwak, Ellen Sebastian

Uses

AfroLlama is optimized for downstream tasks, notably those demanding instructional datasets in Swahili, Xhosa, Zulu, Yoruba, Hausa and English. Organizations can further fine-tune it for their specific domains. Potential areas include:

• Question-answering within specific domains.
• Assistant-driven chat capabilities: healthcare, agriculture, legal, education, tourism and hospitality, public services, financial sectors, communication, customer assistance, commerce, etcpublic services, financial sectors, communication, customer assistance, commerce, etc.

Sample Evaluation Prompts

--

Out-of-Scope Use

The use of the developed Large Language Model (LLM) capabilities is for research,social good and internal use purposes only. For commercial use and distribution, organisations/individuals are encouraged to contactJacaranda Health. To ensure the ethical and responsible use of AfroLlama, we have outlined a set of guidelines. These guidelines categorize activities and practices into three main areas: prohibited actions, high-risk activities, and deceptive practices. By understanding and adhering to these directives, users can contribute to a safer and more trustworthy environment.

1. Prohibited Actions:

• Illegal Activities: Avoid promoting violence, child exploitation, human trafficking, and other crimes.
• Harassment and Discrimination: No acts that bully, threaten, or discriminate.
• Unauthorized Professions: No unlicensed professional activities.
• Data Misuse: Handle personal data with proper consents.
• Rights Violations: Respect third-party rights.
• Malware Creation: Avoid creating harmful software.

2. High-Risk Activities:

• Dangerous Industries: No usage in military, nuclear, or espionage domains.
• Weapons and Drugs: Avoid illegal arms or drug activities.
• Critical Systems: No usage in key infrastructures or transport technologies.
• Promotion of Harm: Avoid content advocating self-harm or violence.

3. Deceptive Practices:

• Misinformation: Refrain from creating/promoting fraudulent or misleading info.
• Defamation and Spam: Avoid defamatory content and unsolicited messages.
• Impersonation: No pretending to be someone without authorization.
• Misrepresentation: No false claims about HauaLlama outputs.
• Fake Online Engagement: No promotion of false online interactions.

Bias, Risks, and Limitations

AfroLlama is a cutting-edge technology brimming with possibilities, yet is not without inherent risks. The extensive testing conducted thus far has been predominantly in Swahili, Xhosa, Zulu, Yoruba, Hausa and English however leaving an expansive terrain of uncharted scenarios. Consequently, like its LLM counterparts, AfroLlama outcome predictability remains elusive, and there's the potential for it to occasionally generate responses that are either inaccurate, biased, or otherwise objectionable in nature when prompted by users. With this in mind, the responsible course of action dictates that, prior to deploying AfroLlama in any applications, developers must embark on a diligent journey of safety testing and meticulous fine-tuning, customized to the unique demands of their specific use cases.

Contact-Us

For any questions, feedback, or commercial inquiries, please reach out at ai@jacarandahealth.org