Gemma 2 2b Crosscoder L13 Mu4.1e 02 Lr1e 04

🚀 跨編碼器模型（CrossCoder）

本跨編碼器模型可用於特徵提取，它基於Gemma 2 2B和Gemma 2 2B IT模型在特定數據集上進行訓練，能有效提取特徵並進行重構。

🚀 快速開始

此跨編碼器在Gemma 2 2B和Gemma 2 2B IT的第13層並行激活上進行訓練，使用的數據集為fineweb和lsmsy - chat - 1m的子集。

你可以使用我們的dictionary_learning庫分支來加載該模型：

!pip install git+https://github.com/jkminder/dictionary_learning
from dictionary_learning import CrossCoder
from nnsight import LanguageModel
import torch as th

crosscoder = CrossCoder.from_pretrained("Butanium/gemma-2-2b-crosscoder-l13-mu4.1e-02-lr1e-04", from_hub=True)
gemma_2 = LanguageModel("google/gemma-2-2b", device_map="cuda:0")
gemma_2_it = LanguageModel("google/gemma-2-2b-it", device_map="cuda:1")
prompt = "quick fox brown"

with gemma_2.trace(prompt):
    l13_act_base = gemma_2.model.layers[13].output[0][:, -1].save() # (1, 2304)
    gemma_2.model.layers[13].output.stop()

with gemma_2_it.trace(prompt):
    l13_act_it = gemma_2_it.model.layers[13].output[0][:, -1].save() # (1, 2304)
    gemma_2_it.model.layers[13].output.stop()


crosscoder_input = th.cat([l13_act_base, l13_act_it], dim=0).unsqueeze(0).cpu() # (batch, 2, 2304)
print(crosscoder_input.shape)
reconstruction, features = crosscoder(crosscoder_input, output_features=True)

# print metrics
print(f"MSE loss: {th.nn.functional.mse_loss(reconstruction, crosscoder_input).item():.2f}")
print(f"L1 sparsity: {features.abs().sum():.1f}")
print(f"L0 sparsity: {(features > 1e-4).sum()}")

📄 許可證

本項目採用MIT許可證。

📋 模型信息