Gemma 2 2b Crosscoder L13 Mu4.1e 02 Lr1e 04

🚀 跨编码器模型（CrossCoder）

本跨编码器模型可用于特征提取，它基于Gemma 2 2B和Gemma 2 2B IT模型在特定数据集上进行训练，能有效提取特征并进行重构。

🚀 快速开始

此跨编码器在Gemma 2 2B和Gemma 2 2B IT的第13层并行激活上进行训练，使用的数据集为fineweb和lsmsy - chat - 1m的子集。

你可以使用我们的dictionary_learning库分支来加载该模型：

!pip install git+https://github.com/jkminder/dictionary_learning
from dictionary_learning import CrossCoder
from nnsight import LanguageModel
import torch as th

crosscoder = CrossCoder.from_pretrained("Butanium/gemma-2-2b-crosscoder-l13-mu4.1e-02-lr1e-04", from_hub=True)
gemma_2 = LanguageModel("google/gemma-2-2b", device_map="cuda:0")
gemma_2_it = LanguageModel("google/gemma-2-2b-it", device_map="cuda:1")
prompt = "quick fox brown"

with gemma_2.trace(prompt):
    l13_act_base = gemma_2.model.layers[13].output[0][:, -1].save() # (1, 2304)
    gemma_2.model.layers[13].output.stop()

with gemma_2_it.trace(prompt):
    l13_act_it = gemma_2_it.model.layers[13].output[0][:, -1].save() # (1, 2304)
    gemma_2_it.model.layers[13].output.stop()


crosscoder_input = th.cat([l13_act_base, l13_act_it], dim=0).unsqueeze(0).cpu() # (batch, 2, 2304)
print(crosscoder_input.shape)
reconstruction, features = crosscoder(crosscoder_input, output_features=True)

# print metrics
print(f"MSE loss: {th.nn.functional.mse_loss(reconstruction, crosscoder_input).item():.2f}")
print(f"L1 sparsity: {features.abs().sum():.1f}")
print(f"L0 sparsity: {(features > 1e-4).sum()}")

📄 许可证

本项目采用MIT许可证。

📋 模型信息