Zenz-v2.5-medium Open-source Model - Achieving Context-aware Conversion between Japanese Kana and Kanji

Zenz V2.5 Medium

Developed by Miwa-Keita

A GPT-2 architecture conditional language model designed specifically for Japanese kana-kanji conversion tasks, supporting context-aware conversion

Large Language Model

Safetensors

Japanese#Japanese Kana Conversion #Character-level BPE #Context-aware

Downloads 25

Release Time : 1/13/2025

Model Overview

A character-level + byte-level BPE tokenizer model based on the GPT-2 architecture, primarily used for the neural network kana-kanji conversion system 'Zenzai', excelling in kana-kanji conversion tasks

Model Features

Character-level + Byte-level BPE Tokenizer

Utilizes advanced character-level and byte-level BPE tokenization techniques to improve Japanese text processing efficiency

Context-aware Conversion

Capable of understanding contextual information for more accurate kana-kanji conversion

Multiple Size Variants

Offers small, medium, and micro versions to meet different application scenario requirements

Model Capabilities

Japanese Kana-Kanji Conversion

Context-aware Text Generation

Japanese Text Processing

Use Cases

Input Method Systems

Zenzai Input Method

Serves as the core engine for the AzooKey kana-kanji converter

Delivers high-quality Japanese input conversion experience

Japanese Text Processing

Japanese Text Normalization

Converts kana text into standardized kanji-kana mixed text

Enhances readability and standardization of Japanese text

🚀 zenz-v2.5-small

Zenz-v2.5-small is a conditional language model based on the GPT-2 architecture, specialized for the kana-kanji conversion task and intended for use in the neural kana-kanji conversion system "Zenzai".

✨ Features

Character-level + byte-level BPE tokenizer.
High performance in the kana-kanji conversion task.
Demonstrates high performance in context-aware conversion.

Zenz-v2.5 is available in three model sizes:

zenz-v2.5-medium: A large-scale model with 310M parameters.
zenz-v2.5-small: A medium-scale model with 91M parameters.
zenz-v2.5-xsmall: A small-scale model with 26M parameters.

📚 Documentation

Model Details

Model Description

This model uses ku-nlp/gpt2-medium-japanese-char as the base model, which is provided under the CC-BY-SA 4.0 license.

This model is also provided under the CC-BY-SA 4.0 license.

Developed by: Keita Miwa (𝕏)
Model type: GPT-2
Language(s) (NLP): Japanese
License: CC-BY-SA 4.0
Finetuned from model: ku-nlp/gpt2-small-japanese-char

Model Sources

This model is designed to be used in conjunction with Zenzai (AzooKeyKanaKanjiConverter).

Repository: https://github.com/ensan-hcl/AzooKeyKanaKanjiConverter

Data Sources

This model was built using the zenz-v2.5-dataset.

🔧 Acknowledgements

During the construction of this model, we received computational resource support from Sakura Internet Inc. We would like to express our sincere gratitude.

We also utilized the following libraries, tools, and language resources to build this model:

MeCab (https://taku910.github.io/mecab/)
ipadic-NEologd (https://github.com/neologd/mecab-ipadic-neologd)
torch (https://pypi.org/project/torch/)
transformers (https://pypi.org/project/transformers/)
datasets (https://pypi.org/project/datasets/)
jaconv (https://pypi.org/project/jaconv/)
llama.cpp (https://github.com/ggerganov/llama.cpp)
llm.c (https://github.com/karpathy/llm.c)

📄 License

This model is licensed under CC-BY-SA 4.0.

Property	Details
Model Type	GPT-2
Training Data	zenz-v2.5-dataset
License	CC-BY-SA 4.0
Finetuned from model	ku-nlp/gpt2-small-japanese-char

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご