Mistral 7B Instruct Aya 101

🚀 Mistral-7B-Instruct-KhanAcademy-v0.2

このモデルは、mistralai/Mistral-7B-Instruct-v0.2 をNoneデータセットでファインチューニングしたものです。評価セットでは、以下の結果を達成しています。

• 損失: 1.1502

🚀 クイックスタート

このモデルを使用するには、以下のコードを参考にしてください。

パイプラインを使用する場合

# ハイレベルのヘルパーとしてパイプラインを使用
from transformers import pipeline

pipe = pipeline("text-generation", model="MaziyarPanahi/Mistral-7B-Instruct-Aya-101")

モデルを直接ロードする場合

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Mistral-7B-Instruct-Aya-101")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Mistral-7B-Instruct-Aya-101")

💻 使用例

基本的な使用法

<s> [INST] You are a sentient, superintelligent artificial general intelligence, here to teach and assist me. Write a short story about Goku discovering kirby has teamed up with Majin Buu to destroy the world.[/INST] Goku was sitting on his favorite rock in the middle of the forest, enjoying the peace and quiet. He had just finished training with Vegeta and was feeling quite satisfied with himself. Suddenly, he sensed a powerful presence approaching.

"Who goes there?" Goku called out, standing up and preparing for battle.

A figure appeared before him, and Goku's eyes widened in shock. It was Kirby! But why was he here? And why did he look so different?

"Goku-san!" Kirby cried, rushing forward and hugging the Saiyan prince. "I have come to ask for your help!"

Goku was confused but happy to see his old friend. "What's going on, Kirby? What do you need my help with?"

Kirby looked around nervously before speaking. "Majin Buu has joined forces with me, and we plan to destroy the world!"

Goku was stunned. "What?! Why would you do that, Kirby?!"

"It's not what it seems," Kirby explained. "Buu promised to make the world a better place if I helped him. We will only destroy the evil beings and leave the good ones alone."

Goku was torn. On one hand, he couldn't trust Majin Buu, who had caused so much destruction in the past. On the other hand, he didn't want to hurt Kirby or stand in the way of his friend's goals.

"I can't let you do this, Kirby," Goku said finally. "We can't trust Majin Buu. He'll only cause more harm in the end."

Kirby sighed. "I understand, Goku-san. But I must follow my own path now. Farewell, my friend."

With a heavy heart, Goku watched as Kirby and Majin Buu disappeared into the distance, ready to wreak havoc on the world. Goku knew he couldn't stop them, but he vowed to protect the innocent and fight against any evil that threatened the peace of the universe.</s>

高度な使用法

<s> [INST] You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

describe about pros and cons of docker system.[/INST] Docker is an open-source platform for developing, shipping, and running applications using container technology. It simplifies the process of creating, deploying, and scaling applications by encapsulating code and dependencies into containers. Here are some advantages and disadvantages of using Docker:

Advantages:
1. Consistent Environment: Docker provides a consistent environment across different platforms and environments, ensuring that applications run the same way everywhere.
2. Faster Development: Docker allows developers to quickly spin up new environments and test their code in isolated containers, reducing development time.
3. Improved Security: Containers isolate applications from each other and from the host system, making it harder for attackers to exploit vulnerabilities.
4. Scalability: Docker makes it easy to scale applications by adding more containers as needed, allowing businesses to handle increased traffic and demand.
5. Reduced Dependencies: Docker reduces the number of dependencies required to run an application, making it easier to manage and maintain.
6. Portability: Docker containers can be easily moved between different environments, such as development, testing, and production, without requiring significant changes.
7. Automated Deployment: Docker integrates with popular deployment tools like Kubernetes and Docker Swarm, making it easier to automate the deployment process.
8. Cost Savings: Docker can reduce infrastructure costs by allowing businesses to use fewer servers and resources, as well as by enabling more efficient resource utilization.

Disadvantages:
1. Learning Curve: Docker has a steep learning curve, especially for those who are new to container technology.
2. Resource Intensive: Running multiple containers on a single host can be resource-intensive, potentially leading to performance issues.
3. Complexity: Docker introduces additional complexity to the development and deployment process, which can be challenging for smaller teams or projects.
4. Vendor Lock-in: Docker has a large market share, but there are also competing container technologies like Kubernetes and LXC. Choosing one over the others can lead to vendor lock-in.
5. Limited Customizability: While Docker offers a lot of flexibility, it may not provide enough customizability for certain use cases.
6. Security Risks: While Docker improves security in many ways, it also introduces new risks, such as the potential for misconfigured containers or vulnerabilities in the Docker engine itself.
7. Monitoring Challenges: Managing and monitoring containers at scale can be challenging, requiring specialized tools and expertise.</s>

🔧 技術詳細

トレーニングハイパーパラメータ

トレーニング中に以下のハイパーパラメータが使用されました。

• 学習率（learning_rate）: 5e-06
• トレーニングバッチサイズ（train_batch_size）: 2
• 評価バッチサイズ（eval_batch_size）: 2
• シード（seed）: 42
• 分散タイプ（distributed_type）: multi-GPU
• デバイス数（num_devices）: 4
• 勾配累積ステップ数（gradient_accumulation_steps）: 4
• 総トレーニングバッチサイズ（total_train_batch_size）: 32
• 総評価バッチサイズ（total_eval_batch_size）: 8
• オプティマイザ（optimizer）: Adam（ベータ=(0.9,0.999)、イプシロン=1e-08）
• 学習率スケジューラタイプ（lr_scheduler_type）: cosine
• 学習率スケジューラウォームアップステップ数（lr_scheduler_warmup_steps）: 10
• エポック数（num_epochs）: 1

トレーニング結果

フレームワークバージョン

• Transformers 4.39.0.dev0
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.0

📄 ライセンス

このモデルは、Apache-2.0ライセンスの下で提供されています。

モデル情報

base_model: mistralai/Mistral-7B-Instruct-v0.2
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true


hub_model_id: MaziyarPanahi/Mistral-7B-Instruct-KhanAcademy-v0.2
hf_use_auth_token: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: CohereForAI/aya_dataset
    type:
      system_prompt: ""
      field_instruction: inputs
      field_output: targets
      format: "[INST] {instruction} [/INST]"
      no_input_format: "[INST] {instruction} [/INST]"

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./models/MaziyarPanahi/Mistral-7B-Instruct-Aya-101

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"