đ Russian o1 / GigaChat 20B-A3B Instruct GGUF
This project is a LoRA adapter for the GigaChat-20B-A3B model. It is trained on the Egor-AI/Russian_thinking_dataset, which is a machine - translated Russian version of the BintangFortuna/OpenO1-SFT-EN-SY dataset.
The trained model can mimic logical thinking in Russian, similar to how o1
from OpenAI
does.
đ Documentation
System Prompt
You need to use the following system prompt:
You are an AI assistant. Format your responses as follows: <Thought> Your thoughts (understanding, reasoning) </Thought> <output> Your answer </output>
W&B Report
The W&B report can be found at: https://api.wandb.ai/links/evilfreelancer/nlec8bt8
Training Details
The training was conducted using the impruver utility with the configuration GigaChat/20B-A3B_lora_o1.
It took approximately 117 hours on an RTX 4090, and 23GB of video memory was required.
Configuration File
output_dir: ./models/GigaChat_20B-A3B_lora_thinking
train_path: ./train.GigaChat_20B-A3B_lora_thinking.jsonl
val_path: ./val.GigaChat_20B-A3B_lora_thinking.jsonl
datasets:
- name: Egor-AI/Russian_thinking_dataset
converter: impruver.instruction_to_messages
mapping:
system: system
instruction: prompt
output: response
model:
class: custom.gigachat.DeepseekForCausalLM
name: ai-sage/GigaChat-20B-A3B-instruct-bf16
attn_implementation: flash_attention_2
load_in_4bit: true
load_in_8bit: false
dtype: bf16
lora:
r: 8
lora_alpha: 32
lora_dropout: 0.1
bias: none
target_modules: [ q_proj, v_proj, k_proj, o_proj, gate_proj, down_proj, up_proj ]
task_type: CAUSAL_LM
tokenizer:
class: transformers.AutoTokenizer
name: ai-sage/GigaChat-20B-A3B-instruct
max_tokens_count: 1500
special_tokens:
pad_token_id: 1
pad_token: <s>
bos_token_id: 1
bos_token: <s>
eos_token_id: 128001
eos_token: <|message_sep|>
chat_template: >
{% if messages[0]['role'] == 'system' -%}
{%- set loop_messages = messages[1:] -%}
{%- set system_message = bos_token + messages[0]['content'] + additional_special_tokens[1] -%}
{%- else -%}
{%- set loop_messages = messages -%}
{%- set system_message = bos_token + '' -%}
{%- endif -%}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{ system_message -}}
{%- endif -%}
{%- if message['role'] == 'user' -%}
{{ message['role'] + additional_special_tokens[0] + message['content'] + additional_special_tokens[1] -}}
{{ 'available functions' + additional_special_tokens[0] + additional_special_tokens[2] + additional_special_tokens[3] + additional_special_tokens[1] -}}
{%- endif -%}
{%- if message['role'] == 'assistant' -%}
{{ message['role'] + additional_special_tokens[0] + message['content'] + additional_special_tokens[1] -}}
{%- endif -%}
{%- if loop.last and add_generation_prompt -%}
{{ 'assistant' + additional_special_tokens[0] -}}
{%- endif -%}
{%- endfor %}
trainer:
eval_strategy: steps
save_strategy: steps
eval_steps: 100
save_steps: 100
per_device_train_batch_size: 1
per_device_eval_batch_size: 1
gradient_accumulation_steps: 8
logging_steps: 1
learning_rate: 0.0004
num_train_epochs: 2
lr_scheduler_type: cosine
warmup_steps: 16
optim: adamw_torch_4bit
metric_for_best_model: eval_loss
load_best_model_at_end: true
save_total_limit: 2
seed: 42
remove_unused_columns: false
max_grad_norm: 1.0
weight_decay: 0.08
torch_compile: false
đ License
The project is licensed under the MIT license.
đ Information Table
Property |
Details |
Model Type |
LoRA adapter for GigaChat-20B-A3B |
Training Data |
Egor-AI/Russian_thinking_dataset |
Base Model |
evilfreelancer/o1_gigachat-20b-a3b_lora |
Pipeline Tag |
question-answering |
Tags |
chat, o1, cot, thinking, reflection |
Language |
ru, en |