Mamba-1.4B-HF Open-Source Language Model - Effortlessly Achieve High-Efficiency Text Generation Tasks

Mamba 1.4b Hf

Developed by state-spaces

Mamba is an efficient language model based on the State Space Model (SSM) architecture, with 1.4B parameters, supporting text generation tasks

Large Language Model

Transformers

#Efficient Text Generation #Lightweight Fine-tuning #Long Sequence Processing

Downloads 5,431

Release Time : 3/5/2024

Model Overview

Mamba is a novel sequence modeling architecture that achieves efficient long sequence processing through selective state space mechanisms, making it particularly suitable for generative tasks

Model Features

Efficient Sequence Modeling

Utilizes selective state space mechanisms for more efficient long sequence processing compared to traditional Transformers

Optimized Inference Speed

Supports CUDA kernel optimization, providing faster inference speeds than standard implementations

Lightweight Fine-tuning Support

Compatible with the PEFT library, supporting parameter-efficient fine-tuning methods like LoRA

Model Capabilities

Text Generation

Dialogue Systems

Content Creation

Use Cases

Text Generation

Dialogue Response Generation

Used for generating responses in chatbots or dialogue systems

Examples demonstrate the ability to generate coherent dialogue responses

Content Continuation

Automatically continues content based on given text prompts

Maintains contextual coherence for multi-round generation

Personalized Applications

Personalized Fine-tuning

Domain adaptation of the model through techniques like LoRA

Examples showcase the fine-tuning process on a dataset of famous quotes

🚀 Mamba

This repository offers the mamba-2.8b model compatible with the transformers library. The model checkpoints remain unchanged, while the complete config.json file and the tokenizer have been uploaded to this repository.

🚀 Quick Start

📦 Installation

You need to install transformers from main until transformers=4.39.0 is released.

pip install git+https://github.com/huggingface/transformers@main

We also recommend you to install both causal_conv_1d and mamba-ssm using:

pip install causal-conv1d>=1.2.0
pip install mamba-ssm

If any of these two is not installed, the "eager" implementation will be used. Otherwise the more optimised cuda kernels will be used.

💻 Usage Examples

Basic Usage

You can use the classic generate API:

>>> from transformers import MambaConfig, MambaForCausalLM, AutoTokenizer
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-1.4b-hf")
>>> model = MambaForCausalLM.from_pretrained("state-spaces/mamba-1.4b-hf")
>>> input_ids = tokenizer("Hey how are you doing?", return_tensors="pt")["input_ids"]

>>> out = model.generate(input_ids, max_new_tokens=10)
>>> print(tokenizer.batch_decode(out))
["Hey how are you doing?\n\nI'm doing great.\n\nI"]

Advanced Usage

In order to finetune using the peft library, we recommend keeping the model in float32!

from datasets import load_dataset
from trl import SFTTrainer
from peft import LoraConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-1.4b-hf")
model = AutoModelForCausalLM.from_pretrained("state-spaces/mamba-1.4b-hf")
dataset = load_dataset("Abirate/english_quotes", split="train")
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    logging_dir='./logs',
    logging_steps=10,
    learning_rate=2e-3
)
lora_config =  LoraConfig(
        r=8,
        target_modules=["x_proj", "embeddings", "in_proj", "out_proj"],
        task_type="CAUSAL_LM",
        bias="none"
)
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    peft_config=lora_config,
    train_dataset=dataset,
    dataset_text_field="quote",
)
trainer.train()

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご