Llama-Primus-Base Open-Source Large Model - Boosts Cybersecurity Benchmark Score by 15.88%, Guardians of Security Forts

Llama Primus Base

Developed by trendmicro-ailab

A specialized pre-trained large language model for cybersecurity based on Llama-3.1-8B-Instruct, achieving a 15.88% improvement in comprehensive cybersecurity benchmark scores

Large Language Model

Transformers

EnglishOpen Source License:MIT #Cybersecurity Pre-training #CVE-CWE Mapping #Threat Intelligence Analysis

Downloads 58

Release Time : 2/19/2025

Model Overview

Llama-Primus-Base is a foundational model based on Llama-3.1-8B-Instruct, continuously pre-trained on the Primus-Seed (0.2B) and Primus-FineWeb (2.57B) cybersecurity text datasets, specifically optimized for the cybersecurity domain

Model Features

Cybersecurity Specialized Pre-training

Pre-trained using two cybersecurity-specific datasets, Primus-Seed and Primus-FineWeb, significantly improving performance in the cybersecurity domain

Significant Performance Improvement

Achieves a 15.88% improvement in comprehensive scores across multiple cybersecurity benchmarks, demonstrating the effectiveness of specialized cybersecurity pre-training

Open-source Dataset Support

Provides open-source datasets covering all stages of cybersecurity LLM training, including pre-training, instruction fine-tuning, and distillation inference data

Model Capabilities

Cybersecurity Text Generation

Cybersecurity Knowledge Q&A

Cyber Threat Intelligence Analysis

Cybersecurity Assessment

Use Cases

Cybersecurity Education

CISSP Exam Preparation

Assists in preparing for CISSP certification exams, achieving 72.3% accuracy in textbook test questions

2.2% improvement over the base model

Threat Intelligence Analysis

CVE to CWE Mapping

Maps Common Vulnerabilities and Exposures (CVE) to Common Weakness Enumeration (CWE)

67.8% accuracy, an 8.7% improvement over the base model

Attack Tactic Identification

Identifies and classifies techniques and tactics of cyber attacks

F1 score 0.314, a 15.4% improvement over the base model

Security Assessment

CVSS Score Prediction

Predicts Common Vulnerability Scoring System (CVSS) scores

Mean absolute deviation 1.0912, a 14.1% improvement over the base model

🚀 Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training

Large Language Models (LLMs) have shown great potential in various fields. However, there is a lack of open - source datasets for LLM pre - training in the cybersecurity domain. Primus fills this gap by providing a collection of datasets for different stages of cybersecurity LLM training.

🚀 Quick Start

For more details about the model and its performance, please refer to the paper: [📄Paper].

✨ Features

Introduction

Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, with promising applications in specialized domains such as finance, law, and biomedicine. However, in the domain of cybersecurity, we noticed a lack of open - source datasets specifically designed for LLM pre - training—even though much research has shown that LLMs acquire their knowledge during pre - training. To fill this gap, we present a collection of datasets covering multiple stages of cybersecurity LLM training, including pre - training (Primus - Seed and Primus - FineWeb), instruction fine - tuning (Primus - Instruct), and reasoning data for distillation (Primus - Reasoning). Based on these datasets and Llama - 3.1 - 8B - Instruct, we developed Llama - Primus - Base, Llama - Primus - Merged, and Llama - Primus - Reasoning. This model card is Llama - Primus - Base.

⚠️ Important Note

No TrendMicro customer information is included.

Cybersecurity Benchmark Results

Metric (5 - shot, w/o CoT)	Llama - 3.1 - 8B - Instruct	Llama - Primus - Base
CISSP (Exams in book)	0.7073	0.7230
CTI - Bench (MCQ)	0.6420	0.6676
CTI - Bench (CVE → CWE)	0.5910	0.6780
*CTI - Bench (CVSS, lower is better)*	1.2712	1.0912
CTI - Bench (ATE)	0.2721	0.3140
CyberMetric (500)	0.8560	0.8660
SecEval	0.4966	0.5007
*Agg.*	2.29	2.66 ↑15.88% 💥

CTI - Bench (CVSS) is scored using Mean Absolute Deviation (lower is better), CTI - ATE uses F1 score, and the others use accuracy. The aggregate score (Agg.) is the sum of all benchmarks, with CTI - Bench (CVSS) negated.

References:

CyberMetric: CyberMetric: A Benchmark Dataset based on Retrieval - Augmented...
CtiBench: CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
SecEval: SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models

About Primus

Primus is Trend Micro's pioneering family of lightweight, state - of - the - art open cybersecurity language models and datasets. Developed through our cutting - edge research initiatives and advanced technology, these resources share the innovative foundation that powers our enterprise - class [Trend Cybertron](https://newsroom.trendmicro.com/2025 - 02 - 25 - Trend - Micro - Puts - Industry - Ahead - of - Cyberattacks - with - Industrys - First - Proactive - Cybersecurity - AI) solution. As an industry leader in cybersecurity, Trend Micro is proud to contribute these powerful, efficiency - optimized models and datasets to the community, while maintaining the excellence and reliability that define our global security standards.

📄 License

This model is based on the MIT license, but you must also comply with the Llama 3.1 Community License Agreement.

📚 Documentation

Model Information

Property	Details
Model Type	Text Generation
Base Model	meta - llama/Llama - 3.1 - 8B - Instruct
Library Name	transformers
Tags	cybersecurity, pretraining
Datasets	trendmicro - ailab/Primus - FineWeb, trendmicro - ailab/Primus - Seed
Extra Gated Fields	Affiliation (text), Country (country), I want to use this model for (Research, Commercial, Other), Job title (Student, Research graduate, AI researcher, AI developer/engineer, Cybersecurity researcher, Reporter, Other), geo (ip_location)

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご