Latent Recurrent Depth Lm

🚀 Latent Recurrent Depth Language Model

The Latent Recurrent Depth Language Model (LRD - LM) is an experimental text - generation architecture. It captures deeper contextual information through iterative, latent processing. Instead of generating long chain - of - thought sequences, it refines its internal state over multiple recurrent iterations. This approach improves text generation quality while keeping the parameter count in check.

🚀 Quick Start

The model can be used for text generation via its integrated generate() method. You can control parameters such as the maximum sequence length, number of recurrent iterations, temperature, and top - k filtering.

Example: Direct Inference

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoModel

# Load the model and tokenizer from the hub
model = AutoModelForCausalLM.from_pretrained("codewithdark/latent-recurrent-depth-lm")
tokenizer = AutoTokenizer.from_pretrained("codewithdark/latent-recurrent-depth-lm")

prompt = "In the realm of language modeling"
input_ids = tokenizer(prompt, return_tensors='pt').input_ids

# Generate logits using a specified number of recurrent iterations
logits = model(input_ids, num_iterations=3)

# Sample from logits to produce generated text
import torch
probs = torch.softmax(logits[:, -1, :], dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
generated_ids = torch.cat([input_ids, next_token], dim=1)
generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
clean_text = generated_text.replace('Ġ','')
print(generated_text)

Alternative: Using the generate() Method

from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("codewithdark/latent-recurrent-depth-lm")
model = AutoModel.from_pretrained("codewithdark/latent-recurrent-depth-lm", trust_remote_code=True)

prompt = "In the realm of language modeling"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
generated_ids = model.generate(input_ids, max_length=50, num_iterations=10, temperature=0.5, top_k=50)
generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
clean_text = generated_text.replace('Ġ','')
print(clean_text)

✨ Features

• Iterative Latent Processing: Captures deeper contextual information through multiple recurrent iterations without generating verbose chain - of - thought sequences.
• Modest Parameter Count: Keeps the parameter count in check while improving text generation quality.

📚 Documentation

Architecture

The model is built around three key components:

• Prelude Block: Handles the initial processing by embedding input tokens and applying self - attention with positional encodings.
• Recurrent Block: A core, weight - shared block that iteratively refines a latent state. It “thinks” over the input without outputting intermediate tokens.
• Coda Block: Decodes the refined latent state into output token probabilities.

Applications & Limitations

Intended Uses:

• Text Generation: Generate creative text, dialogue, code, or other natural language content.
• Research: Serve as a testbed for exploring novel architectures and techniques in language modeling.

Limitations:

• Data Constraints: Trained on a small subset (first 1000 samples) of the Wikitext - 2 - raw - v1 dataset, which may limit its performance compared to models trained on larger corpora.
• Performance: Its overall performance is experimental and may not match state - of - the - art models.
• Computational Overhead: The iterative processing introduces extra computation.
• Bias: Generated outputs may reflect biases present in the training data.

Training Details

The model was fine - tuned on a subset of the Wikitext - 2 - raw - v1 dataset (first 1000 samples) using the AdamW optimizer and a cosine annealing learning rate scheduler. The training configuration and hyperparameters are provided in the accompanying code, and adjustments may be needed for improved performance.

🔧 Technical Details

The model's architecture is designed to capture deeper contextual information. The recurrent block refines the latent state over multiple iterations, allowing the model to “think” about the input without outputting intermediate tokens. This iterative approach helps in improving the text generation quality while keeping the parameter count under control.

📄 License

This project is licensed under the MIT License.

⚠️ Important Note

This model is intended for research and experimental use. Users must ensure ethical application and carefully consider potential biases and misuse when deploying or further developing this technology.