๐ LLaMA 3 Youko QLoRA Fine-tune for Japanese Visual Novel Translation
This project is a QLoRA fine-tune of LLaMA 3 Youko, leveraging a new version of the VNTL dataset. Its core objective is to enhance the performance of large language models (LLMs) in translating Japanese visual novels into English.
๐ Quick Start
This fine-tuned model is designed to improve the translation of Japanese visual novels to English. It uses the LLaMA 3 prompt format and offers better accuracy and stability compared to the previous version.
โจ Features
- Enhanced Dataset: The new version of the VNTL dataset has been rebuilt and expanded from the ground up, leading to better performance in terms of accuracy and stability.
- Default Prompt Format: Switched to the default LLaMA 3 prompt format, which resolves issues users had with the custom one.
- Multi - line Translation Support: Added proper support for multi - line translations, while the old version only handled single lines.
- Improved Accuracy: Overall better translation accuracy, although the translations tend to be more literal.
๐ฆ Installation
No installation steps are provided in the original document, so this section is skipped.
๐ป Usage Examples
Basic Usage
This fine - tune uses the LLaMA 3 prompt format. Here is an example for translation:
<|begin_of_text|><|start_header_id|>Metadata<|end_header_id|>
[character] Name: Uryuu Shingo (็็ ๆฐๅพ) | Gender: Male | Aliases: Onii-chan (ใๅ
ใกใใ)
[character] Name: Uryuu Sakuno (็็ ๆกไน) | Gender: Female<|eot_id|><|start_header_id|>Japanese<|end_header_id|>
[ๆกไน]: ใโฆโฆใใใใ<|eot_id|><|start_header_id|>English<|end_header_id|>
[Sakuno]: ใ... Sorry.ใ<|eot_id|><|start_header_id|>Japanese<|end_header_id|>
[ๆฐๅพ]: ใใใใใใใ่จใฃใกใใชใใ ใใฉใ่ฟทๅญใงใใใฃใใใๆกไนใฏๅฏๆใใใใใใใใๅฟ้
ใใกใใฃใฆใใใ ใไฟบใ<|eot_id|><|start_header_id|>English<|end_header_id|>
[Shingo]: "Nah, I know itโs weird to say this, but Iโm glad you got lost. Youโre so cute, Sakuno, so I was really worried about you."<|eot_id|>
Advanced Usage
The Metadata section isn't limited to character information. You can also add trivia and teach the model the correct way to pronounce words it struggles with. Here's an example:
<|begin_of_text|><|start_header_id|>Metadata<|end_header_id|>
[character] Name: Uryuu Shingo (็็ ๆฐๅพ) | Gender: Male | Aliases: Onii-chan (ใๅ
ใกใใ)
[character] Name: Uryuu Sakuno (็็ ๆกไน) | Gender: Female
[element] Name: Murasamemaru (ๅข้จไธธ) | Type: Quality<|eot_id|><|start_header_id|>Japanese<|end_header_id|>
[ๆกไน]: ใโฆโฆใใใใ<|eot_id|><|start_header_id|>English<|end_header_id|>
[Sakuno]: ใ... Sorry.ใ<|eot_id|><|start_header_id|>Japanese<|end_header_id|>
[ๆฐๅพ]: ใใใใใใใ่จใฃใกใใชใใ ใใฉใ่ฟทๅญใงใใใฃใใใๆกไนใฏๅข้จไธธใใใใใใใใๅฟ้
ใใกใใฃใฆใใใ ใไฟบใ<|eot_id|><|start_header_id|>English<|end_header_id|>
The generated translation for that prompt, with temperature 0, is:
[Shingo]: "Nah, I know itโs not the best thing to say, but Iโm glad you got lost. Sakunoโs Murasamemaru, so I was really worried about you, you know?"
๐ Documentation
Sampling Recommendations
For optimal results, it's highly recommended to use neutral sampling parameters (temperature 0 with no repetition penalty) when using this model.
Notes
This new version of VNTL 8B has been rebuilt and expanded. It outperforms the previous version in accuracy and stability, making far fewer mistakes even at high temperatures (though temperature 0 is still recommended for the best accuracy). The translations are more accurate but tend to be more literal compared to the previous version.
๐ง Technical Details
This fine - tune was done using similar hyperparameters as the previous version. The only difference is the dataset, which is a brand - new one.
Property |
Details |
Rank |
128 |
Alpha |
32 |
Effective Batch Size |
45 |
Warmup Ratio |
0.02 |
Learning Rate |
6e - 5 |
Embedding Learning Rate |
1e - 5 |
Optimizer |
grokadamw |
LR Schedule |
cosine |
Weight Decay |
0.01 |
Train Loss |
0.42 |
๐ License
The license for this project is llama3.
โ ๏ธ Important Note
While the translations are more accurate, they tend to be more literal compared to the previous version.
๐ก Usage Tip
For optimal results, it's highly recommended to use neutral sampling parameters (temperature 0 with no repetition penalty) when using this model.