SOLAR 0 70b 16bit

🚀 SOLAR-0-70b-16bit Model

Solar, a new bot created by Upstage, is a fine - tuned Llama 2 model and ranks high on the HuggingFace Open LLM leaderboard, demonstrating the power of open - source progress.

🚀 Quick Start

Solar, a new bot developed by Upstage, is now accessible on Poe. As a top - ranked model on the HuggingFace Open LLM leaderboard and a fine - tuned version of Llama 2, Solar showcases the advancements enabled by open source. Try it now at https://poe.com/Solar-0-70b

✨ Features

• Capable of handling up to 10k+ input tokens due to the rope_scaling option.
• Tested on A100 80GB hardware.

📦 Installation

No specific installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

tokenizer = AutoTokenizer.from_pretrained("upstage/Llama-2-70b-instruct-v2")
model = AutoModelForCausalLM.from_pretrained(
    "upstage/Llama-2-70b-instruct-v2",
    device_map="auto",
    torch_dtype=torch.float16,
    load_in_8bit=True,
    rope_scaling={"type": "dynamic", "factor": 2} # allows handling of longer inputs
)

prompt = "### User:\nThomas is healthy, but he has to go to the hospital. What could be the reasons?\n\n### Assistant:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
del inputs["token_type_ids"]
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

output = model.generate(**inputs, streamer=streamer, use_cache=True, max_new_tokens=float('inf'))
output_text = tokenizer.decode(output[0], skip_special_tokens=True)

📚 Documentation

Model Details

Dataset Details

Used Datasets

• Orca - style dataset
• Alpaca - style dataset
• No other dataset was used except for the dataset mentioned above
• No benchmark test set or the training set are used

Prompt Template

### System:
{System}

### User:
{User}

### Assistant:
{Assistant}

Hardware and Software

Evaluation Results

Overview

• We conducted a performance evaluation following the tasks being evaluated on the Open LLM Leaderboard. We evaluated our model on four benchmark datasets, which include ARC - Challenge, HellaSwag, MMLU, and TruthfulQA. We used the [lm - evaluation - harness repository](https://github.com/EleutherAI/lm - evaluation - harness), specifically commit [b281b0921b636bc36ad05c0b0b0763bd6dd43463](https://github.com/EleutherAI/lm - evaluation - harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463).
• We used [MT - bench](https://github.com/lm - sys/FastChat/tree/main/fastchat/llm_judge), a set of challenging multi - turn open - ended questions, to evaluate the models.

Main Results

Scripts for H4 Score Reproduction

• Prepare evaluation environments:

# clone the repository
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
# check out the specific commit
git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463
# change to the repository directory
cd lm-evaluation-harness

📄 License

The fine - tuned checkpoints of this model are licensed under the Non - Commercial Creative Commons license ([CC BY - NC - 4.0](https://creativecommons.org/licenses/by - nc/4.0/)).

📞 Contact Us

About Upstage

• Upstage is a company specialized in Large Language Models (LLMs) and AI. We will help you build private LLMs and related applications. If you have a dataset to build domain - specific LLMs or make LLM applications, please contact us at click here to contact.
• As of August 1st, our 70B model has reached the top spot in openLLM rankings, marking itself as the current leading performer globally.