đ CodeLlama-7B-KStack Model
This repository hosts the CodeLlama-7b model, fine - tuned on the KStack dataset with rule - based filtering, presented in the Hugging Face Transformers format. KStack, being the largest collection of permissively licensed Kotlin code, enables the model to perform better with Kotlin code.
đ Quick Start
The following steps and code example show you how to use the model.
đģ Usage Examples
Basic Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = 'JetBrains/CodeLlama-7B-KStack'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
input_text = """\
This function takes an integer n and returns factorial of a number:
fun factorial(n: Int): Int {\
"""
input_ids = tokenizer.encode(
input_text, return_tensors='pt'
).to('cuda')
output = model.generate(
input_ids, max_length=60, num_return_sequences=1,
pad_token_id=tokenizer.eos_token_id,
)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
Advanced Usage
As with the base model, we can use FIM. To do this, the following format must be used:
'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
đ§ Technical Details
Training Setup
The model was trained on one A100 GPU with following hyperparameters:
Hyperparameter |
Value |
warmup |
5% |
max_lr |
1e-6 |
num_epochs |
1 |
attention_dropout |
0.1 |
scheduler |
cosine |
total_batch_size |
128 (~65K tokens per step) |
num_epochs |
1 |
More details about fine - tuning can be found in the technical report (coming soon!).
Fine - tuning Data
For tuning the model, we used the KStack dataset, the largest collection of permissively licensed Kotlin code. To increase the quality of the dataset and filter out outliers, such as homework assignments, we filter out the dataset entries according to the following rules:
- We filter out files, which belong to low - popular repos (the sum of stars and forks is less than 6).
- Next, we filter out files, which belong to repos with less than 5 Kotlin files.
- Finally, we remove files which have fewer than 20 SLOC.
We clean the content of the remaining dataset entries according to the following rules:
- We remove all non - ASCII entries.
- We remove all package lines, such as package kotlinx.coroutines.channels.
- We remove half of the import lines.
We removed half of the imports to avoid potential hallucinations by the model, where it might attempt to import unnecessary libraries. Additionally, packages were removed because this information is only useful at the project level and may introduce additional noise during the learning process.
Evaluation
For evaluation, we used the Kotlin HumanEval dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre - processing necessary to obtain our results, including the code for running, on the datasets's page.
Here are the results of our evaluation:
Model name |
Kotlin HumanEval Pass Rate |
CodeLlama-7B |
26.09 |
CodeLlama-7B-KStack |
29.19 |
đ License
The model is licensed under the Apache - 2.0 license.
â ī¸ Important Note
CodeLlama-7B-KStack is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine - tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack, developers should perform safety testing and tuning tailored to their specific applications of the model.