# Multi-domain pre-training

A Korean-optimized large language model developed by Saltlux AI Lab based on Meta Llama-3-8B, featuring an extended Korean tokenizer and pre-trained on 100GB of curated Korean data

A Vietnamese-optimized large language model based on Llama-2-7B, enhanced through continual pre-training on 124GB of multi-domain Vietnamese and English data for improved language understanding

MPT-7B is an open-source commercial large language model trained by MosaicML. It is pre-trained on 1 trillion tokens of English text and code, and uses an improved Transformer architecture to optimize training and inference efficiency.

Kaz-RoBERTa is a transformers model pre-trained in a self-supervised manner on a large-scale Kazakh corpus, primarily designed for masked language modeling tasks.

ElhBERTeu is a BERT model developed for the Basque language, trained on multi-domain corpora and demonstrating excellent performance in the BasqueGLUE benchmark.

DistilBERT is a lightweight distilled version of BERT, retaining 97% of BERT's performance while being 40% smaller and 60% faster.

DistilBERT is a lightweight distilled version of BERT, retaining most of BERT's performance but with fewer parameters and faster inference speed.

CPM is a Transformer-based 2.6 billion-parameter Chinese pre-trained language model, trained on 100GB of Chinese corpus, supporting various natural language processing tasks.

Indonesian generation model pre-trained based on causal language modeling objectives, trained using the Flax framework