Reformer Crime And Punishment

🚀 Reformer Model trained on "Crime and Punishment"

This project presents a Reformer language model trained on the classic novel "Crime and Punishment". It offers a unique way to generate text based on the rich narrative of this literary work.

🚀 Quick Start

The ReformerLM model can generate text as demonstrated in the following code:

model = ReformerModelWithLMHead.from_pretrained("google/reformer-crime-and-punishment")
tok = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
tok.decode(model.generate(tok.encode("A few months later", return_tensors="pt"), do_sample=True,temperature=0.7, max_length=100)[0])

# gives:'A few months later on was more than anything in the flat. 
# “I have already.” “That’s not my notion that he had forgotten him. 
# What does that matter? And why do you mean? It’s only another fellow,” he said as he went out, as though he want'

📦 Installation

No specific installation steps are provided in the original README. If you want to use this model, you can install the necessary libraries according to the Hugging Face's official documentation.

💻 Usage Examples

Basic Usage

model = ReformerModelWithLMHead.from_pretrained("google/reformer-crime-and-punishment")
tok = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
input_text = "A few months later"
input_ids = tok.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, do_sample=True, temperature=0.7, max_length=100)
generated_text = tok.decode(output[0])
print(generated_text)

📚 Documentation

Data Source

"Crime and Punishment" is a novel written by Fyodor Dostoevsky and translated into English. The training data was taken from gs://trax-ml/reformer/crime-and-punishment-2554.txt, which contains roughly 0.5M tokens.

Model Training

The ReformerLM model was trained in flax using the colab notebook proposed by the authors: https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb. The weights were then converted to Hugging Face's PyTorch ReformerLM model ReformerModelWithLMHead.

| Property | Details |
|----------|---------|
| Model Type | ReformerLM |
| Training Data | Taken from `gs://trax-ml/reformer/crime-and-punishment-2554.txt`, containing roughly 0.5M tokens |