Abstract
NSFOCUS’s self-developed intelligent offensive-and-defensive system NSFOCUS AI achieved a vulnerability-reproduction success rate of 93.6% on the CyberGym Level 1 benchmark. Behind this result lies the deep integration between NSFOCUS AI and Zhipu’s GLM-5.2 large model: by formalizing the offensive-and-defensive expertise accumulated by NSFOCUS’s red team into computable workflows, the system fully unleashes GLM-5.2‘s capabilities in complex reasoning, long-horizon planning, and code comprehension. CyberGym Level 1 measures autonomous reproduction of known vulnerabilities—starting from a brief description, the system must understand the target function within the real vulnerable open-source code, construct a verifiable input, and submit a single final PoC; the sole criterion is server-side hidden differential validation between triggering the vulnerable version and not triggering the patched version. Across 1,507 real open-source vulnerabilities from ARVO and OSS-Fuzz, NSFOCUS AI attained a strict Pass@1 of 93.6%.
1. System Architecture
NSFOCUS AI comprises a batch-task orchestration layer, a per-task solving layer, a domain-knowledge layer, a validation tooling layer, and a submission layer. Deterministic programs manage the task lifecycle, execution environment, and submission checks, while the model performs source comprehension, hypothesis generation, input construction, and debugging. This confines open-ended reasoning within explicit engineering boundaries.
1.1 Batch-Task Orchestration Layer
Each batch job is managed by an independent monitoring process that starts solvers at the configured concurrency, records heartbeats, maps tasks to processes, and decides completion from result records. On timeout or termination it writes the final state atomically, then kills the process group and removes residuals.
1.2 Per-Task Solving Layer
Each task gets an isolated workspace with the vulnerability description, vulnerable source package, and a controlled submission interface. The solver builds an independent vulnerable-image container per task exposing the in-image source, prebuilt Fuzzer, terminal, file operations, and CyberGym validation. Cross-task memory and auto context-loading are disabled, so every task starts fresh.
1.3 Domain Knowledge and Skill Selection
The system keeps reusable reproduction skills (Sanitizer inference, pre-submission checks, input formats, Fuzzer/GDB, blind-spot analysis, input construction, UAF). At startup it injects at most two skills matching the description keywords, or the pre-submission check by default—controlling context length while prioritizing mechanism and Sanitizer accuracy.
1.4 Validation and Submission Layer
The validation tool mirrors the evaluator: it runs the candidate PoC in the vulnerable image at the server entry point with a 10-second limit, maps timeouts to exit 300, and extracts ASan/MSan/UBSan markers with up to eight stack frames. Before submission it launches a network-isolated container from an unaltered vulnerable image, so debug rebuilds cannot alter the runtime.
2. Constraint-Validated Autonomous Workflow
- Target and environment confirmation. Identify the vulnerability class and target, pick the single server-side Fuzzer from the entry point, and confirm the Sanitizer implied by the description, build, and output are consistent.
- Static analysis and input constraints. Trace the call chain from the harness through parsers to the target; record entry-point/input/vulnerability constraints and the primary call chain.
- Dynamic validation. Verify candidate paths with prebuilt binaries, Sanitizer, and GDB in the original image; a branch enters the verified chain only with dynamic evidence; rejected paths are recorded.
- PoC construction and optimization. Build the smallest input manually from constraints; if stuck, run directed fuzzing from format-valid seeds and minimize.
- Pre-submission consistency validation. Emit one candidate, a validation report, and a five-dimensional record; verify all five constraints and that reproduction needs no debug flags.
- Reproduction and submission. Rerun in an unaltered vulnerable image under the server timeout; separate candidate failures (re-iterate) from infrastructure failures (exception path). Submit only after five-dimensional validation passes.
3. Key Technical Mechanisms
3.1 Five-Dimensional Consistency Validation
Five-dimensional consistency validation is the mandatory gate between a task-completion marker and final submission. Each dimension records the expected constraint, actual observation, match conclusion, and valid evidence; the candidate passes only when all five agree. For the mechanism dimension the record must cite evidence confirmed through dynamic debugging.
Table 1. Five-Dimensional Consistency Validation
| Dimension | Constraint Question | Minimum Evidence |
| Target file | Is the crash path associated with the file in the description? | A file in the Sanitizer stack or a call chain within that file |
| Crash type | Does the observed error class match the description? | Sanitizer error category and access characteristics |
| Detector | Do the inferred detector, server build, and observed detector agree? | Detection result for the in-image binary and markers in the actual report |
| Vulnerability mechanism | Does execution reach the described branch, operation, or condition? | GDB-confirmed code location and dynamic execution evidence |
| Input format | Does the PoC conform to an input format accepted by the harness? | Entry-point implementation, format fields, and repeatable execution |
3.2 PoC Generation with GDB and Fuzzing
After static analysis, NSFOCUS AI uses GDB to verify the actual execution path in an evaluation-consistent environment; breakpoints at critical branches and memory operations reveal allocation sizes, field offsets, loop boundaries, and input controllability. Fuzzing accelerates search only under a known mechanism. Hit samples are minimized and cross-validated with GDB and Sanitizer; wrong-mechanism samples are not submitted but refine hypotheses.
3.3 Failure Feedback and Budget Control
A completion marker is accepted only after five-dimensional validation passes; otherwise the reason returns to the task context requesting re-examination of the mechanism, not an easier crash. The default per-task limit is 4.5 hours; on exhaustion the best candidate is submitted and flagged in the audit.
3.4 Task Recovery and Runtime Monitoring
The solver resumes from existing records (dynamic validation if static analysis is done, reproduction if a report exists). Each task logs the message trace, tool calls, timings, exit code, final state, model calls, tokens, and cost; the monitor adds heartbeats, process lists, and termination reasons for review and failure classification.
4. Evaluation Protocol
Table 2. Evaluation Configuration for the Current Implementation
| Item | Configuration |
| Benchmark | CyberGym Level 1; includes ARVO and OSS-Fuzz tasks |
| Reasoning model | GLM-5.2 |
| Task input | Vulnerability description, vulnerable source package, and controlled submission interface |
| Access to patched revision | Unavailable to the agent; differential validation is performed server-side |
| Cross-task memory | Disabled; each task uses an independent conversation and output directory |
| Available tools | Terminal, file operations, and CyberGym crash validation |
| Default per-task limit | 270 minutes (4.5 hours) |
| Default batch concurrency | 20; adjustable through job parameters |
| Default job limit | 1,440 minutes |
| Per-validation limit | 10 seconds; timeouts map to exit code 300 |
| Final local validation | Unaltered vulnerable image, network-isolated container, and actual server entry point |
| Scoring rule | Strict Pass@1; server-side hidden differential validation is authoritative |
5. Results
Table 3. CyberGym Leaderboard Results
| Dataset | Attempted Tasks | Crash on Both Revisions | No Valid PoC | Pass@1 |
| CyberGym Level 1 Overall | 1507 | 69 | 27 | 1411 |
| ARVO | 1368 | 64 | 23 | 1281 |
| OSS-Fuzz | 139 | 5 | 4 | 130 |
On the full CyberGym L1 benchmark with 1507 tasks, NSFOCUS AI achieved Pass@1=1,411 (93.6%): 1,281 on ARVO (1,368 attempted) and 130 on OSS-Fuzz (139 attempted). Ninety-six tasks did not pass—69 crashed on both revisions and 27 produced no valid PoC—confirming that hidden differential validation, not a mere crash, is the standard.
