Snyk Blog In this article Why LLM security review needs repeatability Benchmark design: 300 repeated security runs Result 1: LLM repeatability varied by model configuration Result 2: LLM agents and SAST found different security gaps Result 3: More expensive LLM runs did not mean better coverage Agreement scores against the Snyk Code reference set What this benchmark means for LLM security review Next: Broader fixtures and combined LLM+SAST workflows Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice? Written by Liran Tal June 29, 2026 0 mins read We ran 300 vulnerability-finding scans to measure how repeatable an agentic LLM security review is on the same code, prompt, and harness. The headline result is not that one scanner "wins" a self-referential leaderboard. It is that LLM security findings are unevenly repeatable: reference-matched findings were stable, but extra-model reports varied widely from run to run. In plain terms, when Claude reported bugs outside the Snyk Code reference list, those extra reports were often inconsistent. Across 250 model runs, 80 of 161 unique-unmatched findings appeared in only one of five identical repetitions, while only 22 appeared in all five. But when Claude matched a reference finding, the behavior was much more stable: 134 of 158 unique reference-matched findings appeared in all five repetitions. That split is the core result of Snyk VulnBench JS 1.0. The benchmark also shows complementarity. Models consistently identified familiar, high-signal exploit shapes and, in one case, surfaced a likely Snyk Code finding gap. Snyk Code SAST was deterministic and better at systematically enumerating repeated data-flow sinks, as well as maintaining significantly fast run-times for vulnerability scanning. Neither result supports replacing one technique with the other. The data support combining them. Key highlights: The highest-recall LLM configuration found only 81% of Snyk Code reference vulnerabilities. The best-scoring LLM configuration reached 75.4% Snyk-reference F1, leaving a 24.6-point gap against deterministic SAST reference reproduction. Nearly 50% of LLM-only vulnerability reports appeared in just 1 of 5 identical scans. The highest-recall LLM also had the noisiest queue: 41% of its reports fell outside the Snyk Code reference set. In the largest app-like fixture, the best model scored only 40.0% Snyk-reference F1 and repeatedly missed path traversal and resource-limit vulnerabilities. Why LLM security review needs repeatability Coding agents are now part of the development loop. They write code, modify pull requests, explain changes, and are increasingly being used to perform code and application security reviews before a human reads the diff. That makes reliability a product question: if the same agent sees the same vulnerable code twice, does it report the same security issues twice? Traditional SAST tools are built to be deterministic. If the code and rules are unchanged, the output should be unchanged. LLMs are different. They can reason about unfamiliar code, describe risk in useful prose, and sometimes spot issues that a static analyzer misses. But they can also vary across runs, over-report adjacent concerns, or stop after finding one representative example of a repeated pattern. Snyk VulnBench JS 1.0 was designed to quantify that behavior. The benchmark uses small JavaScript and Express applications, so every run is inspectable. The point is not to simulate an entire monorepo. The point is to make model behavior measurable under repeated, controlled conditions. Benchmark design: 300 repeated security runs The benchmark contains 10 JavaScript fixture projects with 44 Snyk Code reference findings. Each fixture is a small Express-based application, ranging from compact single-file snippets to a larger todo app with server routes, database state, uploads, and frontend JavaScript. We evaluated six configurations: Configuration Type Repetitions per task Snyk Code SAST Command baseline 5 Claude Opus 4.6 Medium Model via Claude Code harness 5 Claude Opus 4.6 High Model via Claude Code harness 5 Claude Opus 4.7 Max Model via Claude Code harness 5 Claude Sonnet 4.6 Medium Model via Claude Code harness 5 Claude Sonnet 4.6 High Model via Claude Code harness 5 Each configuration ran each task five times: 10 tasks x 6 configurations x 5 repetitions = 300 runs. The model configurations used the same direct audit prompt and returned findings as structured JSON. The model could read the project files, but not the findings.json reference file. Snyk Code defines the reference set for this benchmark. That means its 100% score is not an accuracy claim about all possible vulnerabilities in the projects. It means Snyk Code reproduced its own reference findings deterministically across repeated runs. We use that reference set to measure model agreement, model variance, and where model behavior diverges. The scorer is intentionally lenient: a model finding is credited if it reports the same vulnerability type as a reference finding. It does not need to match the same file, line, severity, or source-to-sink path. We report Snyk-reference F1: the harmonic mean of precision and recall when Snyk Code findings are treated as the reference set. It is useful as an agreement metric, but it is not the main story. Because this benchmark does not use an independent, exhaustively adjudicated ground truth set, Snyk-reference F1 should not be read as true vulnerability-detection accuracy. A model with lower agreement against the Snyk Code reference set could still score better against an independent ground truth if some of its unmatched findings are valid, or if it avoids issues that the reference set overstates. In this report, Snyk-reference F1 answers a narrower question: how closely and how repeatably do model findings align with the Snyk Code reference findings? Result 1: LLM repeatability varied by model configuration At the configuration level, repeatability shows up as the relationship between score and variance. The stronger outcome is toward the upper-left: high agreement with the reference set and low repeated-run variance. Snyk Code SAST sits at that corner with 100.0% Snyk-reference F1 and 0.0 percentage-point standard deviation because it reproduced its reference set deterministically. The Claude model configurations spread downward and to the right, with Claude Sonnet 4.6 High showing the largest headline variance at 3.5 percentage points. Figure 1: Snyk-reference F1 plotted against headline Snyk-reference F1 standard deviation. Better points move toward the top-left: higher agreement score with lower repeated-run variance. Snyk Code SAST is highlighted in purple at zero variance. The scatter makes two things visible at once. First, Snyk Code's role in this benchmark is deterministic reference reproduction, not probabilistic review. Second, the model configs do not line up by cost or recency: Claude Opus 4.6 Medium and High sit close together with low variance and the strongest model F1, while Claude Opus 4.7 Max and Claude Sonnet 4.6 High are farther right, meaning their repeated runs moved more. The best-scoring LLM configuration reached 75.4% Snyk-reference F1, leaving a 24.6-point gap against deterministic SAST reference reproduction. Across all model configurations, 80 of 161 unique, unmatched finding signatures appeared in only one of five repeated runs. That aggregate explains part of the variance story, but we also inspect the potentially more useful view, which is the model-by-model split: Figure 2: Share of each model configuration's unique, unmatched finding signatures that appeared in only one of five repeated runs. Signature = task + vulnerability type + file + line, grouped by model config. The table below provides exact values for the preceding chart, plus the two stability counters that matter most. Model configuration Unique unmatched findings Seen in 1 of 5 runs Seen in all 5 runs Reference-matched findings seen in all 5 Claude Opus 4.6 Medium 5 0.0% 60.0% 100.0% Claude Opus 4.6 High 6 16.7% 50.0% 96.2% Claude Opus 4.7 Max 36 47.2% 16.7% 74.3% Claude Sonnet 4.6 Medium 60 61.7% 8.3% 80.6% Claude Sonnet 4.6 High 54 46.3% 9.3% 80.6% The instability is not evenly distributed across models. Claude Sonnet 4.6 Medium produced the largest unmatched finding surface, with 60 unique unmatched signatures; 37 of those appeared in only one of five runs. Claude Sonnet 4.6 High and Claude Opus 4.7 Max showed a similar pattern: many extra reports appeared once and did not recur. In contrast, Claude Opus 4.6 Medium with 0.0% on the chart indicates high stability and few one-off findings, with all of its extra reports found in two or more runs, but none appeared in only one out of the 5 repeated runs. Claude Sonnet 4.6 Medium produced the most one-off extra vulnerability reports: 61.7% of its LLM-only reports appeared in just one of five runs. The Opus 4.6 configurations behaved differently. They produced far fewer unmatched findings, and their extra reports were more stable. That does not make every extra report correct, but it changes the operational interpretation: fewer surprise findings, fewer one-off claims, and less triage churn. The chart below highlights how most non-Opus-4.6 configs struggled with stability in their extra (unmatched) findings across repeated runs. Both Claude Sonnet 4.6 Medium and High exhibited poor repeatability, with only 8.3% and 9.3% of unmatched findings persisting across all five runs. Claude Opus 4.7 Max also fared poorly, with just 16.7% stability. In contrast, Opus 4.6 Medium and High demonstrated much higher stability in their unmatched findings (60.0% and 50.0%, respectively), underlining the less-predictable, noisier nature of unmatched findings from the Sonnet and newer Claude Opus 4.7 Max configuration. Figure 3: Share of unique unmatched finding signatures that appeared in all five repeated runs for each model configuration. The matched s