Leeroodedicated Math 7b

🚀 Leeroo Dedidcated Math Expert 🤗

该模型是通过将专家编排（Orchestration of Expert）应用于数学领域而构建的。这个专用模型既可以生成解决方案，也可以在必要时利用 GPT - 4（或性能类似的大语言模型）来填补其知识库中的空白。具体而言，当给定一个输入时，专用模型首先判断输入的问题是否可以由基础模型解决。如果可以解决，则分离编排器，并使用基础大语言模型专家进行令牌生成。如果问题较难，需要像 GPT - 4 这样的更大模型，则会生成 <GPT4> 令牌（即 token_id = 32000）。

编排器首先经过训练，以估计基础模型对于任何给定查询的知识，然后将其合并到基础模型（这里是 MetaMath7b）中。

一般来说，对于任何领域，你可以通过以下步骤构建它：

• 选择一个基础大语言模型专家 🤗
• 训练一个特定领域的编排器
• 将编排器与基础专家合并

✅ 在 OpenLLM 排行榜的 GSM8k 数据集评估中，Leeroo Math 7b 模型在 5 - shot 设置下达到了 84.77% 的准确率，使其在同类模型中名列前茅，并且显著超过了其基础模型，该基础模型在同一数据集上的得分是 68.84%。这一成绩是在依靠 GPT - 4 回答 GSM8k 提出的一半问题的情况下取得的。

✨ 主要特性

• 采用专家编排技术，结合基础模型和大语言模型，提升数学问题解决能力。
• 能够根据问题难度自动选择使用基础模型或借助 GPT - 4 进行解答。
• 在 GSM8k 数据集上表现优异，准确率较高。

📦 安装指南

文档未提及安装步骤，故跳过该章节。

💻 使用示例

基础用法

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("leeroo/LeerooDedicated-Math-7b", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("leeroo/LeerooDedicated-Math-7b")
device = model.device

# the following question is answered by the leeroo expert
question = "Natalia sold clips to 48 of her friends in April,and then she sold half as many clips in May.How many clips did Natalia sell altogether in April and May?"
encodeds = tokenizer([question], return_tensors="pt")
model_inputs = encodeds['input_ids'].to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
# Natalia sold 48 clips in April.\nIn May, she sold half as many clips as in April,
# so she sold 48/2 = 24 clips.\nAltogether, Natalia sold 48 + 24 = 72 clips in April and May.\n#### 72\nThe answer is: 72</s>

# sends the following question to GPT4
question = "James loves to go swimming and has to swim across a 20-mile lake.  He can swim at a pace of 2 miles per hour.  He swims 60% of the distance.  After that, he stops on an island and rests for half as long as the swimming time.  He then finishes the remaining distance while going half the speed.  How long did it take him to get across the lake?"
encodeds = tokenizer([question], return_tensors="pt")
model_inputs = encodeds['input_ids'].to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
# <GPT4></s>

高级用法

你还可以添加你的 OpenAI API，以便在生成 <GPT4> 令牌时获得完整答案：

from openai import OpenAI
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("leeroo/LeerooDedicated-Math-7b", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("leeroo/LeerooDedicated-Math-7b")
openai_client = OpenAI(
    api_key=  "OPENAI_API_KEY",
    base_url= "https://api.openai.com/v1"
)

def generate(prompt, tokenizer, model, openai_client, max_new_tokens=100, verbose=True):
    inputs = tokenizer(prompt, return_tensors="pt")
    inputs = {k:v.to(model.device) for k,v in inputs.items()}
    gen_tokens = model.generate( **inputs , max_new_tokens=max_new_tokens, do_sample=False, pad_token_id= tokenizer.pad_token_id)
    if gen_tokens[0, inputs['input_ids'].shape[1]] != tokenizer.unk_token_id:
        if verbose: print("\033[94mGenerating using MetaMath7b.\033[0m")
        gen_text = tokenizer.decode(
            gen_tokens[0, inputs['input_ids'].shape[1]:].tolist() )
    else:
        if verbose: print("\033[94mGenerating using gpt4.\033[0m")
        gen_text = openai_client.completions.create(
            model = "gpt-4-1106-preview", # NOTE you can use any bigger mode here having performance similar to gpt4
            prompt = prompt,
            max_tokens = max_new_tokens,
            temperature = 0.0
        ).choices[0].text
    return gen_text

# the following question is answered by the leeroo expert
prompt = "Question: Natalia sold clips to 48 of her friends in April,and then she sold half as many clips in May.How many clips did Natalia sell altogether in April and May?\nAnswer:"
generation = generate(prompt, tokenizer, model, openai_client, max_new_tokens=500)
print(generation)
#> Generating using MetaMath7b.
# Natalia sold 48 clips in April.\nIn May, she sold half as many clips as in April,
# so she sold 48/2 = 24 clips.\nAltogether, Natalia sold 48 + 24 = 72 clips in April and May.\n#### 72\nThe answer is: 72</s>

# sends the following question to GPT4
prompt = "James loves to go swimming and has to swim across a 40-mile lake.  He can swim at a pace of 2 miles per hour.  He swims 60% of the distance.  After that, he stops on an island and rests for half as long as the swimming time.  He then finishes the remaining distance while going half the speed.  How many hours did it take him to get across the lake?"
generation = generate(prompt, tokenizer, model, openai_client, max_new_tokens=500)
print(generation)
#> Generating using gpt4.
#   He swam 40*.6=24 miles
# So he swam for 24/2=12 hours
# He rested for 12/2=6 hours
# He had 40-24=16 miles left to swim
# He swam at 2/2=1 mile per hour
# So he swam for 16/1=16 hours
# So in total, it took him 12+6+16=34 hours
# 34

📚 详细文档

🔍 若要深入了解我们的方法和结果，请参考 HF 博客 🤗、出版物和 代码仓库。
🌍 加入 Leeroo 社区以获取更多更新：领英、Discord、X、网站。

📄 许可证

文档未提及许可证信息，故跳过该章节。

📖 引用

@misc{mohammadshahi2024leeroo,
    title={Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration},
    author={Alireza Mohammadshahi and Ali Shaikh and Majid Yazdani},
    year={2024},
    eprint={2401.13979},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}