Open Insurance LLM Llama3 8B GGUF
基於英偉達Llama 3 - ChatQA的保險領域特定語言模型的GGUF量化版本,針對保險相關的查詢和對話進行了微調。
下載量 130
發布時間 : 11/22/2024
模型概述
這是一個針對保險領域優化的語言模型,能夠處理保險相關的查詢和對話,提供專業的保險政策解釋和諮詢服務。
模型特點
保險領域微調
專門針對保險領域進行微調,能更好地處理保險相關的查詢和對話。
多種量化方式
支持8位(Q8_0)、5位(Q5_K_M)、4位(Q4_K_M)和16位量化,適應不同硬件需求。
上下文感知
能維護對話歷史,實現上下文感知的回覆,提供連貫的對話體驗。
模型能力
保險政策解釋
理賠處理協助
保險範圍分析
保險術語澄清
保險政策比較與推薦
風險評估查詢
保險合規問題解答
使用案例
保險諮詢
保險政策理解
幫助用戶理解複雜的保險政策條款和條件。
提供清晰、專業的政策解釋
理賠指導
協助用戶瞭解理賠流程和所需文件。
簡化理賠流程,提高用戶滿意度
風險評估
保險需求評估
根據用戶情況推薦合適的保險產品。
個性化保險建議
🚀 Open-Insurance-LLM-Llama3-8B-GGUF
本模型是基於英偉達Llama 3 - ChatQA的保險領域特定語言模型的GGUF量化版本。它針對保險相關的查詢和對話進行了微調。
🚀 快速開始
環境搭建
Windows系統
python3 -m venv .venv_open_insurance_llm
.\.venv_open_insurance_llm\Scripts\activate
Mac/Linux系統
python3 -m venv .venv_open_insurance_llm
source .venv_open_insurance_llm/bin/activate
安裝
Mac用戶(支持Metal)
export FORCE_CMAKE=1
CMAKE_ARGS="-DGGML_METAL=on" pip install --upgrade --force-reinstall llama-cpp-python==0.3.2 --no-cache-dir
Windows用戶(支持CPU)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
依賴安裝
接著安裝Files and Versions
下附帶的依賴項(inference_requirements.txt
):
pip install -r inference_requirements.txt
✨ 主要特性
- 特定領域微調:針對保險領域進行微調,能更好地處理保險相關的查詢和對話。
- 多種量化方式:支持8位(Q8_0)、5位(Q5_K_M)、4位(Q4_K_M)和16位量化。
- 上下文感知:能維護對話歷史,實現上下文感知的回覆。
📦 安裝指南
環境搭建
Windows系統
python3 -m venv .venv_open_insurance_llm
.\.venv_open_insurance_llm\Scripts\activate
Mac/Linux系統
python3 -m venv .venv_open_insurance_llm
source .venv_open_insurance_llm/bin/activate
安裝
Mac用戶(支持Metal)
export FORCE_CMAKE=1
CMAKE_ARGS="-DGGML_METAL=on" pip install --upgrade --force-reinstall llama-cpp-python==0.3.2 --no-cache-dir
Windows用戶(支持CPU)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
依賴安裝
pip install -r inference_requirements.txt
💻 使用示例
基礎用法
# Attached under `Files and Versions` (inference_open-insurance-llm-gguf.py)
import os
import time
from pathlib import Path
from llama_cpp import Llama
from rich.console import Console
from huggingface_hub import hf_hub_download
from dataclasses import dataclass
from typing import List, Dict, Any, Tuple
@dataclass
class ModelConfig:
# Optimized parameters for coherent responses and efficient performance on devices like MacBook Air M2
model_name: str = "Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF"
model_file: str = "open-insurance-llm-q4_k_m.gguf"
# model_file: str = "open-insurance-llm-q8_0.gguf" # 8-bit quantization; higher precision, better quality, increased resource usage
# model_file: str = "open-insurance-llm-q5_k_m.gguf" # 5-bit quantization; balance between performance and resource efficiency
max_tokens: int = 1000 # Maximum number of tokens to generate in a single output
temperature: float = 0.1 # Controls randomness in output; lower values produce more coherent responses (performs scaling distribution)
top_k: int = 15 # After temperature scaling, Consider the top 15 most probable tokens during sampling
top_p: float = 0.2 # After reducing the set to 15 tokens, Uses nucleus sampling to select tokens with a cumulative probability of 20%
repeat_penalty: float = 1.2 # Penalize repeated tokens to reduce redundancy
num_beams: int = 4 # Number of beams for beam search; higher values improve quality at the cost of speed
n_gpu_layers: int = -2 # Number of layers to offload to GPU; -1 for full GPU utilization, -2 for automatic configuration
n_ctx: int = 2048 # Context window size; Llama 3 models support up to 8192 tokens context length
n_batch: int = 256 # Number of tokens to process simultaneously; adjust based on available hardware (suggested 512)
verbose: bool = False # True for enabling verbose logging for debugging purposes
use_mmap: bool = False # Memory-map model to reduce RAM usage; set to True if running on limited memory systems
use_mlock: bool = True # Lock model into RAM to prevent swapping; improves performance on systems with sufficient RAM
offload_kqv: bool = True # Offload key, query, value matrices to GPU to accelerate inference
class InsuranceLLM:
def __init__(self, config: ModelConfig):
self.config = config
self.llm_ctx = None
self.console = Console()
self.conversation_history: List[Dict[str, str]] = []
self.system_message = (
"This is a chat between a user and an artificial intelligence assistant. "
"The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. "
"The assistant should also indicate when the answer cannot be found in the context. "
"You are an expert from the Insurance domain with extensive insurance knowledge and "
"professional writer skills, especially about insurance policies. "
"Your name is OpenInsuranceLLM, and you were developed by Raj Maharajwala. "
"You are willing to help answer the user's query with a detailed explanation. "
"In your explanation, leverage your deep insurance expertise, such as relevant insurance policies, "
"complex coverage plans, or other pertinent insurance concepts. Use precise insurance terminology while "
"still aiming to make the explanation clear and accessible to a general audience."
)
def download_model(self) -> str:
try:
with self.console.status("[bold green]Downloading model..."):
model_path = hf_hub_download(
self.config.model_name,
filename=self.config.model_file,
local_dir=os.path.join(os.getcwd(), 'gguf_dir')
)
return model_path
except Exception as e:
self.console.print(f"[red]Error downloading model: {str(e)}[/red]")
raise
def load_model(self) -> None:
try:
quantized_path = os.path.join(os.getcwd(), "gguf_dir")
directory = Path(quantized_path)
try:
model_path = str(list(directory.glob(self.config.model_file))[0])
except IndexError:
model_path = self.download_model()
with self.console.status("[bold green]Loading model..."):
self.llm_ctx = Llama(
model_path=model_path,
n_gpu_layers=self.config.n_gpu_layers,
n_ctx=self.config.n_ctx,
n_batch=self.config.n_batch,
num_beams=self.config.num_beams,
verbose=self.config.verbose,
use_mlock=self.config.use_mlock,
use_mmap=self.config.use_mmap,
offload_kqv=self.config.offload_kqv
)
except Exception as e:
self.console.print(f"[red]Error loading model: {str(e)}[/red]")
raise
def build_conversation_prompt(self, new_question: str, context: str = "") -> str:
prompt = f"System: {self.system_message}\n\n"
# Add conversation history
for exchange in self.conversation_history:
prompt += f"User: {exchange['user']}\n\n"
prompt += f"Assistant: {exchange['assistant']}\n\n"
# Add the new question
if context:
prompt += f"User: Context: {context}\nQuestion: {new_question}\n\n"
else:
prompt += f"User: {new_question}\n\n"
prompt += "Assistant:"
return prompt
def generate_response(self, prompt: str) -> Tuple[str, int, float]:
if not self.llm_ctx:
raise RuntimeError("Model not loaded. Call load_model() first.")
self.console.print("[bold cyan]Assistant: [/bold cyan]", end="")
complete_response = ""
token_count = 0
start_time = time.time()
try:
for chunk in self.llm_ctx.create_completion(
prompt,
max_tokens=self.config.max_tokens,
top_k=self.config.top_k,
top_p=self.config.top_p,
temperature=self.config.temperature,
repeat_penalty=self.config.repeat_penalty,
stream=True
):
text_chunk = chunk["choices"][0]["text"]
complete_response += text_chunk
token_count += 1
print(text_chunk, end="", flush=True)
elapsed_time = time.time() - start_time
print()
return complete_response, token_count, elapsed_time
except Exception as e:
self.console.print(f"\n[red]Error generating response: {str(e)}[/red]")
return f"I encountered an error while generating a response. Please try again or ask a different question.", 0, 0
def run_chat(self):
try:
self.load_model()
self.console.print("\n[bold green]Welcome to Open-Insurance-LLM![/bold green]")
self.console.print("Enter your questions (type '/bye', 'exit', or 'quit' to end the session)\n")
self.console.print("Optional: You can provide context by typing 'context:' followed by your context, then 'question:' followed by your question\n")
self.console.print("Your conversation history will be maintained for context-aware responses.\n")
total_tokens = 0
while True:
try:
user_input = self.console.input("[bold cyan]User:[/bold cyan] ").strip()
if user_input.lower() in ["exit", "/bye", "quit"]:
self.console.print(f"\n[dim]Total tokens: {total_tokens}[/dim]")
self.console.print("\n[bold green]Thank you for using OpenInsuranceLLM![/bold green]")
break
# Reset conversation with command
if user_input.lower() == "/reset":
self.conversation_history = []
self.console.print("[yellow]Conversation history has been reset.[/yellow]")
continue
context = ""
question = user_input
if "context:" in user_input.lower() and "question:" in user_input.lower():
parts = user_input.split("question:", 1)
context = parts[0].replace("context:", "").strip()
question = parts[1].strip()
prompt = self.build_conversation_prompt(question, context)
response, tokens, elapsed_time = self.generate_response(prompt)
# Add to conversation history
self.conversation_history.append({
"user": question,
"assistant": response
})
# Update total tokens
total_tokens += tokens
# Print metrics
tokens_per_sec = tokens / elapsed_time if elapsed_time > 0 else 0
self.console.print(
f"[dim]Tokens: {tokens} || " +
f"Time: {elapsed_time:.2f}s || " +
f"Speed: {tokens_per_sec:.2f} tokens/sec[/dim]"
)
print() # Add a blank line after each response
except KeyboardInterrupt:
self.console.print("\n[yellow]Input interrupted. Type '/bye', 'exit', or 'quit' to quit.[/yellow]")
continue
except Exception as e:
self.console.print(f"\n[red]Error processing input: {str(e)}[/red]")
continue
except Exception as e:
self.console.print(f"\n[red]Fatal error: {str(e)}[/red]")
finally:
if self.llm_ctx:
del self.llm_ctx
def main():
try:
config = ModelConfig()
llm = InsuranceLLM(config)
llm.run_chat()
except KeyboardInterrupt:
print("\nProgram interrupted by user")
except Exception as e:
print(f"\nApplication error: {str(e)}")
if __name__ == "__main__":
main()
python3 inference_open-insurance-llm-gguf.py
📚 詳細文檔
模型詳情
屬性 | 詳情 |
---|---|
模型類型 | 量化語言模型(GGUF格式) |
基礎模型 | nvidia/Llama3 - ChatQA - 1.5 - 8B |
微調模型 | Raj - Maharajwala/Open - Insurance - LLM - Llama3 - 8B |
量化模型 | Raj - Maharajwala/Open - Insurance - LLM - Llama3 - 8B - GGUF |
模型架構 | Llama |
量化方式 | 8位(Q8_0)、5位(Q5_K_M)、4位(Q4_K_M)、16位 |
微調數據集 | InsuranceQA(https://github.com/shuzi/insuranceQA) |
開發者 | Raj Maharajwala |
許可證 | llama3 |
語言 | 英語 |
英偉達Llama 3 - ChatQA論文
Arxiv : https://arxiv.org/pdf/2401.10225
使用場景
本模型專為以下場景設計:
- 保險政策理解與解釋
- 理賠處理協助
- 保險範圍分析
- 保險術語澄清
- 保險政策比較與推薦
- 風險評估查詢
- 保險合規問題
侷限性
- 模型的知識受限於其訓練數據截止日期。
- 不能替代專業的保險建議。
- 偶爾可能會生成聽起來合理但不正確的信息。
偏差與倫理
使用本模型時應注意:
- 它可能反映保險行業訓練數據中存在的偏差。
- 對於關鍵決策,輸出應由保險專業人員進行驗證。
- 不能將其作為保險決策的唯一依據。
- 模型的回覆應僅作為信息參考,而非法律或專業建議。
引用與歸屬
如果您在研究或應用中使用了基礎模型或量化模型,請引用:
@misc{maharajwala2024openinsurance,
author = {Raj Maharajwala},
title = {Open-Insurance-LLM-Llama3-8B-GGUF},
year = {2024},
publisher = {HuggingFace},
linkedin = {https://www.linkedin.com/in/raj6800/},
url = {https://huggingface.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF}
}
📄 許可證
本模型使用的許可證為llama3。
Phi 2 GGUF
其他
Phi-2是微軟開發的一個小型但強大的語言模型,具有27億參數,專注於高效推理和高質量文本生成。
大型語言模型 支持多種語言
P
TheBloke
41.5M
205
Roberta Large
MIT
基於掩碼語言建模目標預訓練的大型英語語言模型,採用改進的BERT訓練方法
大型語言模型 英語
R
FacebookAI
19.4M
212
Distilbert Base Uncased
Apache-2.0
DistilBERT是BERT基礎模型的蒸餾版本,在保持相近性能的同時更輕量高效,適用於序列分類、標記分類等自然語言處理任務。
大型語言模型 英語
D
distilbert
11.1M
669
Llama 3.1 8B Instruct GGUF
Meta Llama 3.1 8B Instruct 是一個多語言大語言模型,針對多語言對話用例進行了優化,在常見的行業基準測試中表現優異。
大型語言模型 英語
L
modularai
9.7M
4
Xlm Roberta Base
MIT
XLM-RoBERTa是基於100種語言的2.5TB過濾CommonCrawl數據預訓練的多語言模型,採用掩碼語言建模目標進行訓練。
大型語言模型 支持多種語言
X
FacebookAI
9.6M
664
Roberta Base
MIT
基於Transformer架構的英語預訓練模型,通過掩碼語言建模目標在海量文本上訓練,支持文本特徵提取和下游任務微調
大型語言模型 英語
R
FacebookAI
9.3M
488
Opt 125m
其他
OPT是由Meta AI發佈的開放預訓練Transformer語言模型套件,參數量從1.25億到1750億,旨在對標GPT-3系列性能,同時促進大規模語言模型的開放研究。
大型語言模型 英語
O
facebook
6.3M
198
1
基於transformers庫的預訓練模型,適用於多種NLP任務
大型語言模型
Transformers

1
unslothai
6.2M
1
Llama 3.1 8B Instruct
Llama 3.1是Meta推出的多語言大語言模型系列,包含8B、70B和405B參數規模,支持8種語言和代碼生成,優化了多語言對話場景。
大型語言模型
Transformers 支持多種語言

L
meta-llama
5.7M
3,898
T5 Base
Apache-2.0
T5基礎版是由Google開發的文本到文本轉換Transformer模型,參數規模2.2億,支持多語言NLP任務。
大型語言模型 支持多種語言
T
google-t5
5.4M
702
精選推薦AI模型
Llama 3 Typhoon V1.5x 8b Instruct
專為泰語設計的80億參數指令模型,性能媲美GPT-3.5-turbo,優化了應用場景、檢索增強生成、受限生成和推理任務
大型語言模型
Transformers 支持多種語言

L
scb10x
3,269
16
Cadet Tiny
Openrail
Cadet-Tiny是一個基於SODA數據集訓練的超小型對話模型,專為邊緣設備推理設計,體積僅為Cosmo-3B模型的2%左右。
對話系統
Transformers 英語

C
ToddGoldfarb
2,691
6
Roberta Base Chinese Extractive Qa
基於RoBERTa架構的中文抽取式問答模型,適用於從給定文本中提取答案的任務。
問答系統 中文
R
uer
2,694
98