Goku 8x22B V0.1

🚀 Goku-8x22B-v0.1 (Goku 141b - A35b)

Goku-8x22B-v0.1是基于philschmid/guanaco-sharegpt-style数据集对v2ray/Mixtral-8x22B-v0.1模型进行微调后的版本。该模型共有1410亿个参数，其中只有350亿个处于激活状态。

🚀 快速开始

使用pipeline进行高级辅助

from transformers import pipeline

pipe = pipeline("text-generation", model="MaziyarPanahi/Goku-8x22B-v0.1")

直接加载模型

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")

通过适配器加载

如果你已经下载了以下模型之一：v2ray/Mixtral-8x22B-v0.1 或 mistral-community/Mixtral-8x22B-v0.1（它们是相同的），你也可以使用PEFT仅加载适配器。

# 假设你已经下载了 
# 调整词表大小
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id="v2ray/Mixtral-8x22B-v0.1"
peft_model_id = "~/.cache/huggingface/hub/models--MaziyarPanahi--Goku-8x22B-v0.1/adapter"

tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
# 我为ChatML模板添加了2个新标记
# 如果你使用PEFT/适配器，此步骤是必需的
model.resize_token_embeddings(len(tokenizer))
model.load_adapter(peft_model_id)

# 你甚至可以使用TextStreamer和文本生成pipeline与你的适配器一起使用
streamer = TextStreamer(tokenizer)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=750,
    temperature=0.6,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    return_full_text=False,
    add_special_tokens=False,
    streamer=streamer
)

💻 使用示例

基础用法

Goku-8x22B-v0.1已在文本生成、基于长上下文回答问题、编码和一些推理任务中进行了测试。在下一个版本中，我将使用更多与数学和编码相关的数据集。

以下是由MaziyarPanahi/Goku-8x22B-v0.1生成的一个示例故事：

Goku had heard a commotion from his house but when he went to check he saw nothing. He thought to himself, "I'll let it go, it was probably just a bird or something. I'm sure it will be fine." But that was when he heard the commotion again, so he went outside and this time he saw two figures on the horizon. One of the figures was a giant pinkish-purple creature, while the other was small, pink, ball-shaped thing.

As the figures approached, Goku realized the large creature was his former enemy, the powerful Majin Buu. And the smaller creature was Kirby, a powerful Star Warrior from the planet Popstar. Goku couldn't believe his eyes.

The two creatures approached Goku menacingly. "Kirby and I have teamed up," said Majin Buu. "We're going to destroy the world!"

Goku was taken aback by the statement. He had never considered the possibility of these two powerful creatures joining forces. He knew he had to put a stop to them, before they could cause any more damage.

He took a deep breath and faced the two creatures. "You two won't get away with this," Goku said firmly. "I won't let you destroy the world."

Majin Buu scoffed, "You can't stop us! Kirby and I are too powerful!"

Goku quickly formed an energy ball in his hands and faced the two creatures. "We'll see about that," he said.

The battle that ensued was intense. The two creatures worked together, using their powerful energy attacks to try to overcome Goku. But Goku kept fighting, using his own powerful energy attacks to counter their moves.

After what seemed like an eternity, Goku managed to get the upper hand. He used a powerful energy attack to defeat the two creatures. After they were defeated, Goku looked around and saw the damage that had been caused by the battle. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

He summoned all of his power and focused it into a powerful energy attack. The energy spread throughout his body and he felt his power grow stronger. With a battle cry, he launched the attack at the two creatures.

The energy hit them both, sending them flying back, stunned for a moment. Goku continued to pressure them with his energy attacks, but they soon recovered and began to counter-attack with their own energy blasts.

Goku knew he had to act quickly if he was going to defeat them. He focused his energy into one powerful attack, and launched it at Kirby. The attack hit and the Star Warrior was sent flying away.

Goku then focused his attention on Majin Buu. He launched a series of energy attacks, using his signature technique, the Kamehameha, and managed to defeat the powerful creature.

After the battle, Goku looked around at the destruction that had been caused by the two creatures. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

With the two creatures defeated, Goku knew he still had a job to do. He took a deep breath and set out to repair the damage that had been caused by the two powerful creatures. He worked for hours, using his energy to put everything back in order and ensuring that the world was safe once again.

Goku's journey was long and hard but, in the end, he was successful. He defeated two powerful enemies and saved the world from destroyers. Thanks to his hard work, the world was able to heal and once again become a place of peace and prosperity.

📄 许可证

本项目采用Apache-2.0许可证。