Theta 35

🚀 Theta-35

Theta-35 is an advanced reasoning model in the Theta series developed by SVECTOR. Specialized in complex thinking and reasoning, it outperforms conventional instruction - tuned models in downstream tasks, especially for challenging problems that demand deep logical analysis and multi - step reasoning.

🚀 Quick Start

This repo contains the Theta-35 model. Here is a code snippet showing how to load the tokenizer and model, and how to generate content:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer directly
model_name = "SVECTOR-CORPORATION/Theta-35"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Prepare prompt
prompt = "How many planets are in our solar system? Explain your reasoning."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True  # This will automatically add "<reasoning>" tag
)

# Generate response
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768,
    temperature=0.6,
    top_p=0.95,
    top_k=30
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

# Decode and print response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

✨ Features

• Training Stage: Pretraining & Post - training (Supervised Finetuning and Reinforcement Learning)
• Architecture: Transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
• Number of Parameters: 33B
• Number of Parameters (Non - Embedding): 33B
• Number of Layers: 64
• Number of Attention Heads (GQA): 40 for Q and 8 for KV
• Context Length: Full 131,072 tokens
• Sliding Window: 32,768 tokens

📦 Installation

Theta-35 requires the latest version of Hugging Face transformers. We advise you to use version 4.43.1 or newer.

With older versions of transformers, you may encounter the following error:

KeyError: 'theta'

💻 Usage Examples

Basic Usage

The code in the Quick Start section shows the basic way to load the model and generate responses.

Advanced Usage

To achieve optimal performance with Theta-35, we recommend the following settings:

1. Enforce Thoughtful Output: Ensure the model starts with "<reasoning>\n" to promote step-by-step thinking, which enhances output quality. If you use apply_chat_template and set add_generation_prompt=True, this is automatically implemented.

2. Sampling Parameters:

   • Use Temperature = 0.6 and TopP = 0.95 instead of Greedy decoding to avoid repetitions.
   • Use TopK between 20 and 40 to filter out rare token occurrences while maintaining diversity.

3. Standardize Output Format: We recommend using prompts to standardize model outputs when benchmarking.

   • Math Problems: Include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.
   • Multiple - Choice Questions: Add "Please show your choice in the answer field with only the choice letter, e.g.,\"answer\": \"C\"." to the prompt.

4. Handle Long Inputs: For inputs exceeding 32,768 tokens, enable sliding window attention to improve the model's ability to process long sequences efficiently.

For supported frameworks, you could add the following to config.json to enable extended context handling:

{
  ...,
  "use_sliding_window": true,
  "sliding_window": 32768
}

📚 Documentation

Theta-35 demonstrates exceptional performance across various reasoning tasks, including:

• Mathematical reasoning
• Logical deduction
• Multi - step problem solving
• Code understanding and generation
• Scientific reasoning

Detailed evaluation results are reported in our documentation.

Note: For the best experience, please review the usage guidelines before deploying Theta models.

For more details, please refer to our documentation.

📄 License

This project is licensed under the Apache-2.0 license.

📖 Citation

If you find our work helpful, feel free to give us a cite.

@misc{theta35,
    title = {Theta-35: Advanced Reasoning in Large Language Models},
    url = {https://www.svector.co.in/models/theta-35},
    author = {SVECTOR Team},
    month = {March},
    year = {2025}
}

@article{theta,
      title={Theta Technical Report}, 
      author={SVECTOR Research Team},
      year={2025}
}