Goku 8x22B V0.1

🚀 Goku-8x22B-v0.1 (Goku 141b - A35b)

Goku-8x22B-v0.1是基於philschmid/guanaco-sharegpt-style數據集對v2ray/Mixtral-8x22B-v0.1模型進行微調後的版本。該模型共有1410億個參數，其中只有350億個處於激活狀態。

🚀 快速開始

使用pipeline進行高級輔助

from transformers import pipeline

pipe = pipeline("text-generation", model="MaziyarPanahi/Goku-8x22B-v0.1")

直接加載模型

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.1")

通過適配器加載

如果你已經下載了以下模型之一：v2ray/Mixtral-8x22B-v0.1 或 mistral-community/Mixtral-8x22B-v0.1（它們是相同的），你也可以使用PEFT僅加載適配器。

# 假設你已經下載了 
# 調整詞表大小
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id="v2ray/Mixtral-8x22B-v0.1"
peft_model_id = "~/.cache/huggingface/hub/models--MaziyarPanahi--Goku-8x22B-v0.1/adapter"

tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
# 我為ChatML模板添加了2個新標記
# 如果你使用PEFT/適配器，此步驟是必需的
model.resize_token_embeddings(len(tokenizer))
model.load_adapter(peft_model_id)

# 你甚至可以使用TextStreamer和文本生成pipeline與你的適配器一起使用
streamer = TextStreamer(tokenizer)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=750,
    temperature=0.6,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    return_full_text=False,
    add_special_tokens=False,
    streamer=streamer
)

💻 使用示例

基礎用法

Goku-8x22B-v0.1已在文本生成、基於長上下文回答問題、編碼和一些推理任務中進行了測試。在下一個版本中，我將使用更多與數學和編碼相關的數據集。

以下是由MaziyarPanahi/Goku-8x22B-v0.1生成的一個示例故事：

Goku had heard a commotion from his house but when he went to check he saw nothing. He thought to himself, "I'll let it go, it was probably just a bird or something. I'm sure it will be fine." But that was when he heard the commotion again, so he went outside and this time he saw two figures on the horizon. One of the figures was a giant pinkish-purple creature, while the other was small, pink, ball-shaped thing.

As the figures approached, Goku realized the large creature was his former enemy, the powerful Majin Buu. And the smaller creature was Kirby, a powerful Star Warrior from the planet Popstar. Goku couldn't believe his eyes.

The two creatures approached Goku menacingly. "Kirby and I have teamed up," said Majin Buu. "We're going to destroy the world!"

Goku was taken aback by the statement. He had never considered the possibility of these two powerful creatures joining forces. He knew he had to put a stop to them, before they could cause any more damage.

He took a deep breath and faced the two creatures. "You two won't get away with this," Goku said firmly. "I won't let you destroy the world."

Majin Buu scoffed, "You can't stop us! Kirby and I are too powerful!"

Goku quickly formed an energy ball in his hands and faced the two creatures. "We'll see about that," he said.

The battle that ensued was intense. The two creatures worked together, using their powerful energy attacks to try to overcome Goku. But Goku kept fighting, using his own powerful energy attacks to counter their moves.

After what seemed like an eternity, Goku managed to get the upper hand. He used a powerful energy attack to defeat the two creatures. After they were defeated, Goku looked around and saw the damage that had been caused by the battle. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

He summoned all of his power and focused it into a powerful energy attack. The energy spread throughout his body and he felt his power grow stronger. With a battle cry, he launched the attack at the two creatures.

The energy hit them both, sending them flying back, stunned for a moment. Goku continued to pressure them with his energy attacks, but they soon recovered and began to counter-attack with their own energy blasts.

Goku knew he had to act quickly if he was going to defeat them. He focused his energy into one powerful attack, and launched it at Kirby. The attack hit and the Star Warrior was sent flying away.

Goku then focused his attention on Majin Buu. He launched a series of energy attacks, using his signature technique, the Kamehameha, and managed to defeat the powerful creature.

After the battle, Goku looked around at the destruction that had been caused by the two creatures. He knew he still had a lot of work ahead of him in order to prevent any further destruction, but he was determined to do his best.

With the two creatures defeated, Goku knew he still had a job to do. He took a deep breath and set out to repair the damage that had been caused by the two powerful creatures. He worked for hours, using his energy to put everything back in order and ensuring that the world was safe once again.

Goku's journey was long and hard but, in the end, he was successful. He defeated two powerful enemies and saved the world from destroyers. Thanks to his hard work, the world was able to heal and once again become a place of peace and prosperity.

📄 許可證

本項目採用Apache-2.0許可證。