Starcoder2 7b

🚀 StarCoder2

StarCoder2-7B 模型是一個擁有 70 億參數的模型，在 17 種編程語言的代碼數據上進行訓練，可用於代碼生成等任務。

🚀 快速開始

安裝依賴

首先，確保從源代碼安裝 transformers：

pip install git+https://github.com/huggingface/transformers.git

運行模型

在 CPU/GPU/多 GPU 上運行模型

• 使用全精度

# pip install git+https://github.com/huggingface/transformers.git # TODO: merge PR to main
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/starcoder2-7b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 29232.57 MB

• 使用 torch.bfloat16

# pip install accelerate
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

checkpoint = "bigcode/starcoder2-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# for fp16 use `torch_dtype=torch.float16` instead
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 14616.29 MB

通過 bitsandbytes 使用量化版本

• 使用 8 位精度 (int8)

# pip install bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

# to use 4bit use `load_in_4bit=True` instead
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

checkpoint = "bigcode/starcoder2-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
# load_in_8bit
Memory footprint: 7670.52 MB
# load_in_4bit
>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 4197.64 MB

✨ 主要特性

• 多語言支持：在 17 種編程語言的代碼數據上進行訓練。
• 先進架構：採用分組查詢注意力、滑動窗口注意力和中間填充目標等技術。
• 長上下文處理：支持 16384 個標記的上下文窗口。

📦 安裝指南

確保從源代碼安裝 transformers：

pip install git+https://github.com/huggingface/transformers.git

💻 使用示例

基礎用法

# pip install git+https://github.com/huggingface/transformers.git # TODO: merge PR to main
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/starcoder2-7b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

高級用法

# pip install accelerate
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

checkpoint = "bigcode/starcoder2-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# for fp16 use `torch_dtype=torch.float16` instead
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

📚 詳細文檔

預期用途

該模型在 GitHub 代碼以及 Arxiv 和 Wikipedia 等其他選定數據源上進行訓練。因此，它不是一個指令模型，像“編寫一個計算平方根的函數”這樣的命令效果不佳。

生成代碼

你可以在 StarCoder2 的 GitHub 倉庫 中找到微調腳本。

歸屬與其他要求

該模型的預訓練數據集僅過濾了許可性許可證和無許可證的代碼。然而，模型可以逐字生成數據集中的源代碼。代碼的許可證可能要求歸屬和/或其他特定要求，必須予以遵守。我們提供了一個 搜索索引，可讓你搜索預訓練數據，以確定生成的代碼來自何處，並對你的代碼進行適當的歸屬。

🔧 技術細節

模型

• 架構：具有分組查詢和滑動窗口注意力以及中間填充目標的 Transformer 解碼器
• 預訓練步驟：100 萬步
• 預訓練標記：3.5 萬億以上
• 精度：bfloat16

硬件

• GPU：432 個 H100

軟件

• 框架：nanotron
• 神經網絡：PyTorch

📄 許可證

該模型遵循 BigCode OpenRAIL - M v1 許可協議。你可以在 此處 找到完整協議。

📚 引用

@misc{lozhkov2024starcoder,
      title={StarCoder 2 and The Stack v2: The Next Generation}, 
      author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},
      year={2024},
      eprint={2402.19173},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

模型指標

注意事項

⚠️ 重要提示

該模型生成的代碼不能保證按預期工作，可能效率低下，包含錯誤或漏洞。請參考 論文 深入瞭解模型的侷限性。

💡 使用建議

由於該模型不是指令模型，建議提供一定的上下文來生成代碼。同時，使用我們提供的 搜索索引 對生成的代碼進行歸屬。