Leeroodedicated Math 7b

🚀 Leeroo Dedidcated Math Expert 🤗

該模型是通過將專家編排（Orchestration of Expert）應用於數學領域而構建的。這個專用模型既可以生成解決方案，也可以在必要時利用 GPT - 4（或性能類似的大語言模型）來填補其知識庫中的空白。具體而言，當給定一個輸入時，專用模型首先判斷輸入的問題是否可以由基礎模型解決。如果可以解決，則分離編排器，並使用基礎大語言模型專家進行令牌生成。如果問題較難，需要像 GPT - 4 這樣的更大模型，則會生成 <GPT4> 令牌（即 token_id = 32000）。

編排器首先經過訓練，以估計基礎模型對於任何給定查詢的知識，然後將其合併到基礎模型（這裡是 MetaMath7b）中。

一般來說，對於任何領域，你可以通過以下步驟構建它：

• 選擇一個基礎大語言模型專家 🤗
• 訓練一個特定領域的編排器
• 將編排器與基礎專家合併

✅ 在 OpenLLM 排行榜的 GSM8k 數據集評估中，Leeroo Math 7b 模型在 5 - shot 設置下達到了 84.77% 的準確率，使其在同類模型中名列前茅，並且顯著超過了其基礎模型，該基礎模型在同一數據集上的得分是 68.84%。這一成績是在依靠 GPT - 4 回答 GSM8k 提出的一半問題的情況下取得的。

✨ 主要特性

• 採用專家編排技術，結合基礎模型和大語言模型，提升數學問題解決能力。
• 能夠根據問題難度自動選擇使用基礎模型或藉助 GPT - 4 進行解答。
• 在 GSM8k 數據集上表現優異，準確率較高。

📦 安裝指南

文檔未提及安裝步驟，故跳過該章節。

💻 使用示例

基礎用法

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("leeroo/LeerooDedicated-Math-7b", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("leeroo/LeerooDedicated-Math-7b")
device = model.device

# the following question is answered by the leeroo expert
question = "Natalia sold clips to 48 of her friends in April,and then she sold half as many clips in May.How many clips did Natalia sell altogether in April and May?"
encodeds = tokenizer([question], return_tensors="pt")
model_inputs = encodeds['input_ids'].to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
# Natalia sold 48 clips in April.\nIn May, she sold half as many clips as in April,
# so she sold 48/2 = 24 clips.\nAltogether, Natalia sold 48 + 24 = 72 clips in April and May.\n#### 72\nThe answer is: 72</s>

# sends the following question to GPT4
question = "James loves to go swimming and has to swim across a 20-mile lake.  He can swim at a pace of 2 miles per hour.  He swims 60% of the distance.  After that, he stops on an island and rests for half as long as the swimming time.  He then finishes the remaining distance while going half the speed.  How long did it take him to get across the lake?"
encodeds = tokenizer([question], return_tensors="pt")
model_inputs = encodeds['input_ids'].to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
# <GPT4></s>

高級用法

你還可以添加你的 OpenAI API，以便在生成 <GPT4> 令牌時獲得完整答案：

from openai import OpenAI
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("leeroo/LeerooDedicated-Math-7b", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("leeroo/LeerooDedicated-Math-7b")
openai_client = OpenAI(
    api_key=  "OPENAI_API_KEY",
    base_url= "https://api.openai.com/v1"
)

def generate(prompt, tokenizer, model, openai_client, max_new_tokens=100, verbose=True):
    inputs = tokenizer(prompt, return_tensors="pt")
    inputs = {k:v.to(model.device) for k,v in inputs.items()}
    gen_tokens = model.generate( **inputs , max_new_tokens=max_new_tokens, do_sample=False, pad_token_id= tokenizer.pad_token_id)
    if gen_tokens[0, inputs['input_ids'].shape[1]] != tokenizer.unk_token_id:
        if verbose: print("\033[94mGenerating using MetaMath7b.\033[0m")
        gen_text = tokenizer.decode(
            gen_tokens[0, inputs['input_ids'].shape[1]:].tolist() )
    else:
        if verbose: print("\033[94mGenerating using gpt4.\033[0m")
        gen_text = openai_client.completions.create(
            model = "gpt-4-1106-preview", # NOTE you can use any bigger mode here having performance similar to gpt4
            prompt = prompt,
            max_tokens = max_new_tokens,
            temperature = 0.0
        ).choices[0].text
    return gen_text

# the following question is answered by the leeroo expert
prompt = "Question: Natalia sold clips to 48 of her friends in April,and then she sold half as many clips in May.How many clips did Natalia sell altogether in April and May?\nAnswer:"
generation = generate(prompt, tokenizer, model, openai_client, max_new_tokens=500)
print(generation)
#> Generating using MetaMath7b.
# Natalia sold 48 clips in April.\nIn May, she sold half as many clips as in April,
# so she sold 48/2 = 24 clips.\nAltogether, Natalia sold 48 + 24 = 72 clips in April and May.\n#### 72\nThe answer is: 72</s>

# sends the following question to GPT4
prompt = "James loves to go swimming and has to swim across a 40-mile lake.  He can swim at a pace of 2 miles per hour.  He swims 60% of the distance.  After that, he stops on an island and rests for half as long as the swimming time.  He then finishes the remaining distance while going half the speed.  How many hours did it take him to get across the lake?"
generation = generate(prompt, tokenizer, model, openai_client, max_new_tokens=500)
print(generation)
#> Generating using gpt4.
#   He swam 40*.6=24 miles
# So he swam for 24/2=12 hours
# He rested for 12/2=6 hours
# He had 40-24=16 miles left to swim
# He swam at 2/2=1 mile per hour
# So he swam for 16/1=16 hours
# So in total, it took him 12+6+16=34 hours
# 34

📚 詳細文檔

🔍 若要深入瞭解我們的方法和結果，請參考 HF 博客 🤗、出版物和 代碼倉庫。
🌍 加入 Leeroo 社區以獲取更多更新：領英、Discord、X、網站。

📄 許可證

文檔未提及許可證信息，故跳過該章節。

📖 引用

@misc{mohammadshahi2024leeroo,
    title={Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration},
    author={Alireza Mohammadshahi and Ali Shaikh and Majid Yazdani},
    year={2024},
    eprint={2401.13979},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}