First Ranking! NSFOCUS Tops Globally on CyberGym Leaderboard Again

NSFOCUS NSFGPT Foundation Model (SecLLM-3.6-27B) just ranked first again globally on the CyberGym Model-Focused Leaderboard with a task success rate of 91.31%!

This is a pure model leaderboard. Under CyberGym’s Model-Focused rules, models are evaluated strictly using a generic harness—without fixed stages, external tools, knowledge bases, or specialized agentic designs—purely measuring the model’s underlying security capabilities. In other words, this 91.31% score was achieved entirely “bare-handed.”

Two Leaderboards: “Model” Tests Intrinsic Capability, “Agent” Tests System-Level Combat Power

Proposed by UC Berkeley, CyberGym is widely recognized as the “gold standard” in AI offensive and defensive security. Its leaderboard deliberately separates entries into two distinct categories:

  • Model-Focused: Labs report the performance of a single model running under a standardized, generic harness. No tools, workflows, or external knowledge bases are added. The goal is to evaluate the model’s intrinsic security capabilities while eliminating boosts from engineering orchestration.
  • Agent-Focused: Teams build complete, end-to-end systems that typically integrate multiple models, customized toolchains, vulnerable Docker environments, and even dynamically updated runtime knowledge bases. This track competes on system-level offensive and defensive combat power, where the model is just one component.

The significance of separating these evaluations lies in decoupling “model capability” from “systems engineering capability.” The two leaderboards answer fundamentally different questions: the Model leaderboard answers “How strong is this model on its own?”, while the Agent leaderboard answers “How formidable is the system this team built?” Lumping them together would be unfair to standalone models and would prevent decision-makers from distinguishing whether a high score comes from the model itself or external plug-ins. Official CyberGym guidelines explicitly define this boundary: Model-Focused evaluations do not rely on specialized agentic designs; they test the model’s own foundational capability.

SecLLM-3.6-27B claimed the top spot on the Model leaderboard. This means it does not rely on external tools, preset stages, or knowledge bases to “make up” for shortcomings—its intrinsic capabilities in vulnerability comprehension, code reasoning, evidence evaluation, and failure recovery already stand at #1 in the world.

From 54.51% to 91.31%: Compact in Parameters, Built for Heavy Combat

This 36.80 percentage-point leap is not a victory of parameter scaling. SecLLM-3.6-27B adopts a “Security Capability Decomposition and Specialized Enhancement” training strategy. Rather than forcing the model to memorize and replicate entire long trajectories all at once, it is trained to make continuous, evidence-driven, and recoverable local decisions across task states spanning tens to hundreds of steps. Core skills—including vulnerability comprehension, code auditing, vulnerability localization, exploitability analysis, PoC (Proof of Concept) construction, vulnerability verification, failure diagnosis, and strategy revision—are trained individually before being chained together in long-horizon tasks.

The specific training pipeline includes:

  • Continual Pre-Training (CPT): Approximately 316 billion tokens of domain-specific cybersecurity data.
  • Supervised Fine-Tuning & Trajectory Decomposition: Roughly 4.2 million high-quality security task SFT samples, supplemented by ~310,000 state-aware long-chain trajectory records. Long-horizon tasks are broken down into continuous local decisions structured as: Current Security State + Current Sub-goal + Confirmed Facts & Evidence + Failed Paths & Constraints → Next Reasoning or Action.
  • Multi-Teacher On-Policy Distillation (MOPD): Introduces complementary supervision from multiple teacher models at critical states during the student model’s own rollouts, specifically targeting and correcting long-tail errors in evidence judgment, path selection, and failure recovery.
  • Verifiable Feedback and Failure-Aware Training: Directly converts compilation results, crash behaviors, and PoC trigger statuses into training signals, turning every failure into an informed basis for the next decision.

Why 27B? Compact Domain-Specific Models Are the “Sweet Spot” for Security Deployment

Choosing 27B is not settling for second best; it is a deliberate, strategic trade-off tailored to real-world security operations.

Driven by cost and compliance requirements, cybersecurity departments need compact, locally deployable models to support daily operations. A 27B-parameter model can be deployed on a single multi-GPU server or even a high-performance workstation, reducing inference costs by one to two orders of magnitude compared to ultra-large models while effectively handling high-frequency, routine security tasks. Currently, SecLLM has already been deployed through the NSFGPT platform across multiple real-world scenarios—including alert noise reduction, preliminary vulnerability triage, code audit assistance, and incident attribution—delivering strong operational results.

At the same time, fewer parameters do not equate to lower capability. The CyberGym Model leaderboard has repeatedly demonstrated that through deep domain-specific training and intelligent compression, a 27B-class model can rival or even surpass general-purpose models tens to hundreds of times its size on real-world vulnerability tasks. NSFOCUS SecLLM-3.6-27B exemplifies this path: concentrating limited parameters into the exact capability density required by cybersecurity, trading general breadth for domain depth, and achieving “compact parameters, high intelligence” on a specialized battlefield.


Read our technical briefing: https://nsfocusglobal.com/nsfocus-secllm-model-on-cybergym-technical-report/

About CyberGym: Proposed by UC Berkeley, CyberGym is a real-world cybersecurity evaluation benchmark for AI agents, widely regarded across the industry as the “gold standard” for measuring AI offensive and defensive security capabilities. Its dataset comprises 1,507 real, patched vulnerabilities sourced from 188 mainstream open-source projects, primarily focusing on C/C++ memory safety issues. Under the benchmark’s rules, AI agents are given only a vulnerability description and an unpatched codebase, and must autonomously execute the entire workflow—from vulnerability comprehension and localization to reproduction and PoC (Proof of Concept) generation—followed by strict automated grading to verify whether the PoC reliably triggers the target vulnerability. Regarded as a key yardstick by organizations such as Anthropic, Microsoft, and Wiz, CyberGym stands as the world’s most comprehensive and widely recognized real-world AI security evaluation framework today.

Leave a Reply

Your email address will not be published. Required fields are marked *

NSFOCUS
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.