Sasando-1-7M Open-Source Indonesian Text Generator - Free Deployment of Experimental Miniature Model

Sasando 1 7M

Developed by afrizalha

A miniature experimental Indonesian text generator based on the Phi-3 architecture, available in 7M and 25M parameter versions

Large Language Model

Transformers

OtherOpen Source License:MIT #Indonesian text generation #Low-resource optimization #Miniature language model

Downloads 58

Release Time : 6/8/2024

Model Overview

Sasando-1 is a compact language model focused on Indonesian, designed to achieve text generation capabilities with limited resources, suitable for low-resource language preservation research.

Model Features

Low-resource optimization

Designed for resource-scarce environments, demonstrating that miniature language models trained on strictly controlled datasets can still generate fluent text

Multi-parameter versions

Offers both 7M and 25M parameter versions to accommodate different computational needs

Language preservation focus

Tailored for Indonesia's multilingual context, exploring generative AI applications in language preservation

Model Capabilities

Indonesian text generation

Short text generation

Use Cases

Language preservation

Endangered language digitization

Provides a generative model prototype for resource-scarce Indonesian languages

Demonstrates the feasibility of miniature models with limited data

Educational research

Low-resource NLP education

Serves as a low-cost case study for language model instruction

🚀 Descaling low-resource language models

Sasando-1 is a highly experimental short - sequence text generator built with the Phi - 3 architecture, aiming to leverage limited resources for generative model development in the context of low - resource languages.

🚀 Quick Start

If you want to experience Sasando-1 directly, you can ❕Go straight to the gradio demo❕.

✨ Features

Tiny Sizes: Sasando-1 comes in two microscopic sizes with 7M and 25M parameters.
Phi - 3 Architecture: It is built using the Phi - 3 architecture.
Limited Vocabulary Training: Trained on a tightly - controlled Indo4B dataset with only 18000 unique words.
Low - Resource Adaptation: Demonstrates that thinly - available resources can be used to develop generative models with cheap compute.

📚 Documentation

🇮🇩 Context

Indonesia has over 700 languages, many of which are dying at an alarming rate. Generative AI can play a significant role in language preservation. However, Indonesia faces several challenges:

Low - Volume Digital Resources: Many languages, even those with millions of speakers, have limited digital resources.
High Cost of Large Models: Running large models can be costly, and Indonesia is a middle - income country with limited funding.

Sasando-1 is a prototype that shows it's possible to develop generative models using limited data and budget.

✨ Specs

Parameter Sizes: Available in 7M and 25M parameter variations.
Architecture: Based on the Phi - 3 architecture.
Embedding Vocab: Has an embedding vocab of 4096.
Training: Trained on approximately 257M tokens over 4 epochs.

🔭 Out-of-Scope Use

This is a research preview base model. It is not instruction - tuned and has minimal safety curation. It is not intended for commercial or practical applications. Also, you are expected to have fun when using this model.

📄 License

This project is developed by Afrizal Hasbi Azizy and is licensed under the MIT license.

Training log

Sasando-1 is a tiny, highly experimental short-sequence text generator built using the Phi-3 architecture.

This repo contains the 7M version

Preliminary research preview

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご