Rwkv Raven 1b5

🚀 Model card for RWKV-4 | 1B5 parameters chat version (Raven)

RWKV is a project led by Bo Peng. It combines the advantages of RNN and Transformer, offering great performance, fast inference, VRAM savings, fast training, an "infinite" context length, and free sentence embedding. You can learn more about the model architecture in the blogposts from Johan Wind here and here. Join the RWKV discord server to learn more about the project.

🚀 Quick Start

This model card provides detailed information about the RWKV-4 model, including its details, usage, and citation.

✨ Features

• Transformer-level LLM performance: RWKV is an RNN that can achieve performance comparable to Transformer-based LLMs.
• Parallelizable training: It can be directly trained like a GPT.
• Combination of RNN and Transformer advantages: Great performance, fast inference, VRAM savings, fast training, "infinite" ctx_len, and free sentence embedding.

📦 Installation

Datasets

• EleutherAI/pile

Convert the raw weights to the HF format

You can use the convert_rwkv_checkpoint_to_hf.py script by specifying the repo_id of the original weights, the filename and the output directory. You can also optionally directly push the converted model on the Hub by passing --push_to_hub flag and --model_name argument to specify where to push the converted weights.

python convert_rwkv_checkpoint_to_hf.py --repo_id RAW_HUB_REPO --checkpoint_file RAW_FILE --output_dir OUTPUT_DIR --push_to_hub --model_name dummy_user/converted-rwkv

💻 Usage Examples

Basic Usage

The "Raven" models needs to be prompted in a specific way, learn more about that in the integration blogpost.

Running the model on a CPU

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5")
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Running the model on a single GPU

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5").to(0)
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Running the model in half-precision, on GPU

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5", torch_dtype=torch.float16).to(0)
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Running the model multiple GPUs

# pip install accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

📚 Documentation

The details of the architecture can be found on the blogpost mentioned above and the Hugging Face blogpost of the integration.

📄 License

No license information provided in the original README.

📄 Citation

If you use this model, please consider citing the original work, from the original repo here