RWKV - Raven - 3B Open - source Large Language Model - Free Deployment, Supports Infinite Context Length Processing

Rwkv Raven 3b

Developed by RWKV

RWKV is a large language model combining the advantages of RNN and Transformer, supporting efficient training and fast inference with unlimited context length processing capability.

Large Language Model

Transformers

#RNN-Transformer Hybrid Architecture #Unlimited Context Length #Efficient GPU Memory Utilization

Downloads 273

Release Time : 5/4/2023

Model Overview

RWKV-4 Raven is a 3-billion-parameter large language model featuring a unique RNN-Transformer hybrid architecture, suitable for chat and text generation tasks.

Model Features

Efficient Architecture

Combines RNN and Transformer advantages, supporting parallelized training while maintaining RNN's efficient inference characteristics

Long Context Processing

Theoretically supports unlimited context length processing capability

Resource Efficient

More GPU memory-efficient during training and inference compared to traditional Transformer models

Model Capabilities

Chinese Text Generation

Multi-turn Dialogue

Long Text Comprehension

Creative Writing

Use Cases

Dialogue Systems

Intelligent Chatbot

Build fluent and natural dialogue systems

Capable of coherent multi-turn conversations

Content Creation

Story Generation

Generates coherent long-form stories from brief prompts

Produces logical and creative content

🚀 RWKV-4 | 3B parameters chat version (Raven)

RWKV is a project led by Bo Peng. It combines the best of RNN and transformer, offering great performance, fast inference, VRAM savings, fast training, "infinite" ctx_len, and free sentence embedding.

⚡ TL;DR

Below is the description from the original repository

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). It's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

📖 Model Details

The details of the architecture can be found on the blogpost mentioned above and the Hugging Face blogpost of the integration.

💻 Usage

Convert the raw weights to the HF format

You can use the convert_rwkv_checkpoint_to_hf.py script by specifying the repo_id of the original weights, the filename and the output directory. You can also optionally directly push the converted model on the Hub by passing --push_to_hub flag and --model_name argument to specify where to push the converted weights.

python convert_rwkv_checkpoint_to_hf.py --repo_id RAW_HUB_REPO --checkpoint_file RAW_FILE --output_dir OUTPUT_DIR --push_to_hub --model_name dummy_user/converted-rwkv

Generate text

You can use the AutoModelForCausalLM and AutoTokenizer classes to generate texts from the model. Expand the sections below to understand how to run the model in different scenarios: The "Raven" models needs to be prompted in a specific way, learn more about that in the integration blogpost.

🖥️ Running the model on a CPU

Click to expand

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-3b")
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-3b")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

�GPU Running the model on a single GPU

Click to expand

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-3b").to(0)
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-3b")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

🎯 Running the model in half-precision, on GPU

Click to expand

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-3b", torch_dtype=torch.float16).to(0)
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-3b")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

🛠️ Running the model multiple GPUs

Click to expand

# pip install accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-3b", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-3b")

prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=40)
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

📄 Citation

If you use this model, please consider citing the original work, from the original repo here

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご