Cool Japan Diffusion 2 1 2 Beta

🚀 Cool Japan Diffusion 2.1.2 Beta Model Card

Cool Japan Diffusion is a model fine - tuned from Stable Diffusion, specializing in expressing Cool Japan elements such as anime, manga, and games.

Note: China will implement legal restrictions on image - generating AI. (Warning for people in China)

The English version is here.

🚀 Quick Start

Cool Japan Diffusion is a model that fine - tunes Stable Diffusion and is specialized in expressing Cool Japan elements like anime, manga, and games. Note that it has no particular relation to the Cool Japan Strategy of the Cabinet Office.

📄 License

Regarding the license, it's just an addition of commercial use prohibition (except for exceptions) to the original CreativeML Open RAIL++ - M License. The reason for adding the commercial use prohibition (except for exceptions) is the concern that it might have a negative impact on the creative industry. If this concern is dispelled, the next version will revert to the original license and allow commercial use. By the way, the Japanese translation of the original license is here. People in for - profit enterprises should consult with their legal department. Those using it for personal interest generally don't need to worry too much as long as they follow common sense. As stated in the license, if you modify this model, you need to inherit this license.

📚 Documentation

Legal and Ethical Considerations

This model was created in Japan, so Japanese laws apply. The author claims that the training of this model is legal based on Article 30 - 4 of the Copyright Act. Also, regarding the distribution of this model, the author claims that it does not fall under the category of principal offenders or abettors in the context of the Copyright Act and Article 175 of the Criminal Code. For more details, please refer to the opinion of lawyer Kakinuma. However, as stated in the license, the products of this model should be handled in accordance with various laws and regulations.

The author believes that the act of distributing this model is not ethically good because the permission of the copyright holders of the training materials was not obtained. However, legally, the permission of the copyright holders is not required for training, just like search engines, there is no legal problem. Therefore, please consider that this distribution also serves the purpose of investigating the ethical aspects rather than just the legal ones.

Usage

If you want to have a quick and easy experience, please use this Space. The detailed instructions on how to handle this model are written here. The model can be downloaded here.

Model Details

Model Usage Examples

It's used in the same way as Stable Diffusion v2. There are many methods, and here are two patterns:

• Web UI
• Diffusers

Web UI

⚠️ Important Note

It is recommended to install xformers and turn on the --xformers --disable - nan - check options. Otherwise, turn on the --no - half option. Please create according to this instruction manual.

Diffusers

Use 🤗's Diffusers library. First, run the following script to install the library:

pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy

Then, run the following script to generate an image:

from diffusers import StableDiffusionPipeline, EulerAncestralDiscreteScheduler
import torch

model_id = "aipicasso/cool-japan-diffusion-2-1-2-beta"

scheduler = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, torch_dtype=torch.float32)
pipe = pipe.to("cuda")

prompt = "anime, masterpiece, a portrait of a girl, good pupil, 4k, detailed"
negative_prompt="deformed, blurry, bad anatomy, bad pupil, disfigured, poorly drawn face, mutation, mutated, extra limb, ugly, poorly drawn hands, bad hands, fused fingers, messy drawing, broken legs censor, low quality, mutated hands and fingers, long body, mutation, poorly drawn, bad eyes, ui, error, missing fingers, fused fingers, one hand with more than 5 fingers, one hand with less than 5 fingers, one hand with more than 5 digit, one hand with less than 5 digit, extra digit, fewer digits, fused digit, missing digit, bad digit, liquid digit, long body, uncoordinated body, unnatural body, lowres, jpeg artifacts, 3d, cg, text, japanese kanji"
images = pipe(prompt,negative_prompt=negative_prompt, num_inference_steps=20).images
images[0].save("girl.png")

💡 Usage Tip

• Using xformers seems to speed up the process.
• If you have limited GPU memory when using a GPU, please use pipe.enable_attention_slicing().

Intended Use

• Contests
  • Submissions to AI Art Grand Prix. Make sure to disclose all the data used for fine - tuning and let the judges determine if the review criteria are met. If you have any requests for the contest, please contact me on Hugging Face's Community.
• Reporting on Image - generating AI
  • It's possible not only for public broadcasters but also for for - profit enterprises. This is because it is judged that the "right to know" information about image - synthesizing AI will not have a negative impact on the creative industry. Also, freedom of the press is respected.
• Introduction of Cool Japan
  • Explain what Cool Japan is to people from other countries. Alfred Increment often feels that many international students in Japan are attracted by Cool Japan but are often disappointed to find that what they thought was "cool" in Japan is considered "uncool" here. People from other countries should be more proud of their own cultures that others admire.
• Research and Development
  • Model usage on Discord
    • Prompt engineering
    • Fine - tuning (also known as additional learning), such as DreamBooth
    • Merging with other models
  • Compatibility between the Latent Diffusion Model and Cool Japan
  • Investigating the performance of this model using metrics like FID
  • Checking the independence of this model from models other than Stable Diffusion using checksums or hash functions.
• Education
  • Graduation projects of art college students and vocational school students
  • Graduation theses and assignment projects of university students
  • Teachers explaining the current situation of image - generating AI
• Self - expression
  • Expressing one's emotions and thoughts on SNS
• Uses described in Hugging Face's Community
  • Please ask questions in Japanese or English.

Unintended Use

• Representing things as facts.
• Using it in monetized content on platforms like YouTube.
• Directly providing it as a commercial service.
• Doing things that would trouble teachers.
• Other actions that would have a negative impact on the creative industry.

Prohibited or Malicious Use

• Do not publish digital forgeries (Digital Forgery) (risk of violating the Copyright Act).
  • Especially, do not publish existing characters (risk of violating the Copyright Act). It seems that characters that were not trained can also be generated. (This tweet itself is permitted for research purposes.)
• Do not perform Image - to - Image on others' works without permission (risk of violating the Copyright Act).
• Do not distribute pornographic materials (risk of violating Article 175 of the Criminal Code).
  • Do not violate the so - called industry etiquette.
• Do not state non - facts as facts (risk of being subject to the crime of interfering with business).
  • Avoid spreading fake news.

Model Limitations and Biases

Model Limitations

• Not well - understood.

Biases

It has the same biases as Stable Diffusion. Please be careful.

Training

Training Data

The following data and models are mainly used to fine - tune Stable Diffusion:

• Regarding VAE
  • Data that complies with Japanese domestic laws, excluding Danbooru and Danbooru datasets: 650,000 types (an infinite number of images can be created through data augmentation).
• Regarding U - Net
  • Data that complies with Japanese domestic laws, excluding Danbooru and Danbooru datasets: 2 million pairs.
  • Merged model: 1 (Open RAIL license).

Training Process

The VAE and U - Net of Stable Diffusion were fine - tuned.

• Hardware: RTX 4090, A6000
• Optimizer: AdamW
• Gradient Accumulations: 1
• Batch Size: 1

Evaluation Results

Not provided.

Environmental Impact

It has little environmental impact.

• Hardware Type: RTX 4090, A6000
• Usage Time (in hours): 500
• Cloud Provider: None
• Training Location: Japan
• Carbon Emissions: Not much.

References

@InProceedings{Rombach_2022_CVPR, author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj"orn}, title = {High - Resolution Image Synthesis With Latent Diffusion Models}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {10684 - 10695} }

*This model card was written by Alfred Increment based on Stable Diffusion v2.