Cool Japan Diffusion 2 1 0

🚀 Cool Japan Diffusion 2.1.0 Model Card

Cool Japan Diffusion is a model fine - tuned from Stable Diffusion, specialized in expressing Cool Japan elements such as anime, manga, and games.

⚠️ Important Note

Note: From January 10, 2023, China will implement legal restrictions on image - generating AI. (Warning for people in China)

The English version is here.

🚀 Quick Start

Cool Japan Diffusion (for learning) is a model fine - tuned from Stable Diffusion, specifically designed to represent Cool Japan elements in anime, manga, games, etc. Note that it has no particular relation to the Cool Japan Strategy of the Cabinet Office.

📄 License

Regarding the license, it's just an addition of a commercial use prohibition (excluding exceptions) to the original CreativeML Open RAIL++ - M License. The reason for adding the commercial use prohibition (excluding exceptions) is the concern that it may have a negative impact on the creative industry. If this concern is dispelled, the next version will revert to the original license and allow commercial use. By the way, the Japanese translation of the original license is here. If you're from a for - profit company, please consult your legal department. If you're using it for personal interest, you should be fine as long as you follow general common sense. As stated in the license, even if you modify this model, you need to inherit this license.

🔧 Legal and Ethical Considerations

This model was created in Japan, so Japanese laws apply. We claim that the training of this model is legal based on Article 30 - 4 of the Copyright Law. Also, regarding the distribution of this model, we claim that it does not fall under the category of principal offenders or accessory offenders in light of the Copyright Law and Article 175 of the Criminal Code. For more details, please refer to the opinion of lawyer Kakinuma. However, as stated in the license, please handle the products of this model in accordance with various laws and regulations.

However, the author believes that distributing this model is not ethically good because the permission of the copyright holders of the works used for training was not obtained. But legally, obtaining the permission of the copyright holders is not necessary for training, just like search engines, there is no legal problem. Therefore, please consider that this distribution also serves the purpose of investigating ethical aspects rather than just legal ones.

💻 Usage Examples

Basic Usage

If you want to have a quick and easy experience, please use this [Space](https://huggingface.co/spaces/alfredplpl/cool - japan - diffusion - 2 - 1 - 0). The detailed usage instructions of this model are here. You can download the model [here](https://huggingface.co/aipicasso/cool - japan - diffusion - 2 - 1 - 0/resolve/main/v2 - 1 - 0.ckpt).

Advanced Usage

Web UI

Please create according to this instruction manual.

Diffusers

Use 🤗's Diffusers library. First, run the following script to install the library:

pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy

Then run the following script to generate an image:

from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
import torch

model_id = "aipicasso/cool-japan-diffusion-2-1-0-beta"

scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "anime, a portrait of a girl with black short hair and red eyes, kimono, full color illustration, official art, 4k, detailed"
negative_prompt="low quality, bad face, bad anatomy, bad hand, lowres, jpeg artifacts, 2d, 3d, cg, text"
image = pipe(prompt,negative_prompt=negative_prompt).images[0]

image.save("girl.png")

⚠️ Important Note

• It seems that using xformers can speed up the process.
• If you have limited GPU memory when using a GPU, please use pipe.enable_attention_slicing().

Expected Use Cases

• Contests
  • Submission to AI Art Grand Prix. We will disclose all the data used for fine - tuning and ensure that the judging criteria are met. Also, we will apply in advance for confirmation. If you have any requests for the contest, please let me know on Hugging Face's Community.
• Reports on Image - generating AI
  • It's possible not only for public broadcasters but also for for - profit companies. This is because we believe that the "right to know" information about image - synthesizing AI will not have a negative impact on the creative industry, and we respect the freedom of the press.
• Introduction of Cool Japan
  • Explain what Cool Japan is to people from other countries. Alfred Increment has noticed that many international students in Japan are attracted by Cool Japan but often get disappointed when they find that what they thought was "cool" in Japan is not really considered "cool" here. People from other countries should be more proud of their own cultures that others admire.
• Research and Development
  • Model Usage on Discord
    • Prompt engineering
    • Fine - tuning (also known as additional learning), such as DreamBooth
    • Merging with other models
  • Compatibility between Latent Diffusion Model and Cool Japan
  • Investigating the performance of this model using metrics like FID
  • Checking the independence of this model from models other than Stable Diffusion using checksums or hash functions.
• Education
  • Graduation projects of art college students and vocational school students
  • Graduation theses and project works of university students
  • Teachers can use it to convey the current situation of image - generating AI.
• Self - expression
  • Express your emotions and thoughts on SNS.
• Use Cases on Hugging Face's Community
  • Ask questions in Japanese or English.

Unexpected Use Cases

• Representing things as facts without basis.
• Using it in monetized content on platforms like YouTube.
• Directly providing it as a commercial service.
• Causing trouble for teachers.
• Other actions that may have a negative impact on the creative industry.

Prohibited or Malicious Use Cases

• Do not publish digital forgeries (Digital Forgery) (it may violate the Copyright Law). Especially, do not publish existing characters (it may violate the Copyright Law). It seems that characters that were not used for training can also be generated. (This tweet itself is permitted for research purposes.)
• Do not perform Image - to - Image on others' works without permission (it may violate the Copyright Law).
• Do not distribute pornographic materials (it may violate Article 175 of the Criminal Code). Do not violate the so - called industry etiquette.
• Do not spread misinformation as facts (it may be subject to the crime of obstructing business by force), such as fake news.

🔧 Model Limitations and Biases

Model Limitations

• Not well - understood yet.

Biases

This model has the same biases as Stable Diffusion. Please be careful.

🔧 Technical Details

Training Data

We mainly used the following data to fine - tune Stable Diffusion:

Training Process

We fine - tuned the VAE and U - Net of Stable Diffusion.

• Hardware: RTX 3090
• Optimizer: AdamW
• Gradient Accumulations: 1
• Batch Size: 1

📚 Evaluation Results

There is little impact.

• Hardware Type: RTX 3090
• Usage Time (in hours): 300
• Cloud Provider: None
• Training Location: Japan
• Carbon Emissions: Not much

📚 References

@InProceedings{Rombach_2022_CVPR,
    author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
    title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {10684-10695}
}

This model card was written by Alfred Increment based on [Stable Diffusion v2](https://huggingface.co/stabilityai/stable - diffusion - 2/raw/main/README.md).