Replit Code V1 5 3b

🚀 Replit Code V-1.5 3B

Replit Code v1.5 is a 3.3B parameter Causal Language Model specialized in code completion, trained on a vast code dataset.

🚀 Quick Start

This section provides a quick overview of the Replit Code V-1.5 3B model and how to get started with it.

✨ Features

• Code Completion Focus: Replit Code v1.5 is a 3.3B parameter Causal Language Model centered on Code Completion.
• Diverse Training Data: Trained on 1T tokens of code for 30 programming languages from Bigcode's Stack Dedup dataset, a filtered natural - language sample, and a dev - oriented sample from RedPajama's StackExchange dataset.
• Custom Tokenizer: Uses the GPTNeoX tokenizer with a custom - trained and optimized vocabulary of 32768 tokens, achieving compression with maintained or improved coverage.
• Powerful Training Platform: Trained on the MosaicML platform using [LLM Foundry](https://github.com/mosaicml/llm - foundry) and Composer on 128 H100 - 80GB GPUs.

📦 Installation

You will need to install the latest versions of the following dependencies:

einops
torch
transformers

💻 Usage Examples

Basic Usage

You can generate code using the transformers library as follows:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('replit/replit-code-v1_5-3b', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('replit/replit-code-v1_5-3b', trust_remote_code=True)

x = tokenizer.encode('def fibonacci(n): ', return_tensors='pt')
y = model.generate(x, max_length=100, do_sample=True, top_p=0.95, top_k=4, temperature=0.2, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)

# decoding
generated_code = tokenizer.decode(y[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(generated_code)

Experiment with different decoding methods and parameters to get the best results for your use case.

Advanced Usage

Using Triton Implementation of Flash Attention:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

config = AutoConfig.from_pretrained(
    "replit/replit-code-v1_5-3b",
    trust_remote_code=True
)
config.attn_config['attn_impl'] = 'triton'

# load model
tokenizer = AutoTokenizer.from_pretrained('replit/replit-code-v1_5-3b', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('replit/replit-code-v1_5-3b', config=config, trust_remote_code=True)
model.to(device='cuda:0', dtype=torch.bfloat16)

# forward pass
x = tokenizer.encode('def fibonacci(n): ', return_tensors='pt').to(device='cuda:0')
x = x.to(device='cuda:0')
y = model.generate(x, max_length=100, do_sample=True, top_p=0.95, top_k=4, temperature=0.2, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)

# decoding
generated_code = tokenizer.decode(y[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(generated_code)

Experiment with different decoding methods and parameters to get the best results for your use case. We recommend experimenting with temperature and reptition_penalty for optimal performance on your use case!

📚 Documentation

Model Description

The model is trained in bfloat16 on 1T tokens of code (~200B tokens over 5 epochs, including linear cooldown) for the following 30 programming languages:

Java, JavaScript, C, PHP, Python, C++, C#, TypeScript, Go, CSS, HTML, Rust, Ruby, Swift, Scala, Shell, Lua, Perl, Haskell, JSX, Julia, Common Lisp, OCaml, Solidity, Scheme, R, Zig, SQL, Racket, D

The context size of the model is 4096 tokens.

Intended Use

Replit intends this model be used by anyone as a foundational model for application - specific fine - tuning without strict limitations on commercial use. The model is trained specifically for code completion tasks.

Limitations

The pre - training dataset may have contained offensive or inappropriate content even after applying data cleansing and toxicity and profanity filters, and such content may be reflected in model generated text. We recommend that users exercise reasonable caution when using in production systems. Do not use for any applications that may cause harm or distress to individuals or groups.

📄 License

This project is licensed under the Apache 2.0 license.

📋 Information Table

⚠️ Important Note

The pre - training dataset may have contained offensive or inappropriate content even after applying data cleansing and toxicity and profanity filters, and such content may be reflected in model generated text. Exercise reasonable caution when using in production systems. Do not use for any applications that may cause harm or distress to individuals or groups.

💡 Usage Tip

Experiment with different decoding methods and parameters such as temperature and reptition_penalty to get the best results for your use case.