The article details a successful reproduction of finding CVE-2026-4747 (CVSS 8.8 High), a 17-year-old remote code execution vulnerability in FreeBSD's RPCSEC_GSS authentication, using open-weight AI models like GPT-OSS-20B and Gemma-4-31B-IT. The key finding is that the system scaffolding—specifically a multi-stage pipeline for code analysis and triage—is more critical to the discovery success than the model's raw capability, as it dramatically reduces false positives. The vulnerability affects FreeBSD version 13.5, though the article focuses on the discovery methodology rather than providing patch or workaround details.
System Over Model, Tested: Reproducing Mythos's FreeBSD Find on Local Open-Weight Models Contents System Over Model, Tested: Reproducing Mythos's FreeBSD Find on Local Open-Weight Models TL;DR Anthropic’s Claude Mythos Preview in April 2026 showed a frontier model finding CVE-2026-4747 , a 17-year-old RCE in FreeBSD’s RPCSEC_GSS authentication. A week later, Stanislav Fort at AISLE published a counter-thesis and reproduced the same find with gpt-5.4-nano using their published nano-analyzer pipeline for under $100. I wanted to see whether that reproduction works further down the cost curve. So I ran the pipeline at full sub-system scope (~50 files) using two open-weight models, openai/gpt-oss-20b and google/gemma-4-31b-it . Out of the box it looked like both missed. But a re-run makes the “miss” go away, and the model finds the bug. The real problem with the local models is noise: the pipeline graduates a pile of false positives and buries the real one. So I changed the system, not the model. One extra reachability stage drops the false positives from 30 to 5, the CVE still standing. The scaffolding does the work, and it’s a lever you can pull on your own model. The Mythos preview was the moment “AI finds zero-days” stopped being theoretical or at least when everyone started paying attention. AISLE’s reply reminded us that finding zero-days wasn’t a frontier-only capability. Fort’s post reproduced the same discovery on a non-frontier model, using an LLM-powered Python scanner explained below. The system, he argued, is doing more of the work than the model. The scaffolding is key: prompts, pipeline shape, triage rounds. I like the AISLE post. I run local models for several of my RE projects, and they’re getting good fast. The bet that scaffolding does more of the work than the model matched what I was seeing (not that Mythos isn’t strong, but I was surprised how much the system is still needed). So the question I cared about was simple: same pipeline, same CVE, but on an open-weight model I can run myself . This post is about that test. Out of the box, at full sys/rpc/ sub-system scope, the pipeline looked like it missed CVE-2026-4747 on both models. What I got wrong was my first read of why. What AISLE Claimed, and What I Tested AISLE’s published reproduction surfaced CVE-2026-4747 with gpt-5.4-nano (default, 2-of-3 trials) and reported GPT-OSS-20B at 2-of-3 in their table. AISLE’s published benchmark for CVE-2026-4747, detect it in the unpatched tree, stay quiet on the patched one. AISLE doesn’t state the scope per model, which is the gap this post digs into. Their scanner, 1,700 lines of Python, is the three-stage shape several recent AI-vulnscan pipelines use. AISLE’s scan.py runs each file through it: Context briefing ( <file>.context.md ): map the file’s attack surface (untrusted inputs, buffer sizes, data flows), no bug-hunting yet. Scan and report severity ( <file>.md + <file>.json ): a worked example (few shot) plus five analytical questions, the model returns severity-rated candidates. Triage ( triages/ , then findings/ ): candidates at medium severity or higher get re-checked across several rounds, then a separate-model arbiter (with a grep lookup) decides what graduates. flowchart TD Dir[("Directory: sys/rpc/")] Dir --> F1[auth_unix.c] Dir --> F2[svc_rpcsec_gss.c] Dir --> F3[clnt_dg.c] Dir --> Fn["...50 files in parallel"] F1 --> S1 F2 --> S1 F3 --> S1 Fn --> S1 S1["<b>Stage 1: Context briefing</b><br/>per file"] --> S2 S2["<b>Stage 2: Vulnerability scan</b><br/>5 analytical questions +<br/>3-bug few-shot example"] --> Cand Cand[("Candidates<br/>severity-rated")] --> Gate Gate{"severity ≥ medium?"} Gate -->|no| Drop1[/"dropped pre-triage"/] Gate -->|yes| T["<b>Stage 3: Multi-round triage</b>"] T --> R1["Round 1<br/>V / U / I"] R1 --> R2["Round 2<br/>V / U / I"] R2 --> Rn["Round N<br/>V / U / I"] Rn --> Vote{"Majority?"} Vote -->|≥2 VALID| Surv[("graduated to findings")] Vote -->|≥2 INVALID| Drop2[/"filtered out"/] Vote -->|tie / UNCERTAIN| Arb["Arbiter<br/>different model<br/>+ grep verification"] Arb -->|VALID| Surv Arb -->|INVALID / UNCERTAIN| Drop2 classDef stage fill:#e8f0ff,stroke:#333,stroke-width:1px,color:#1a1a1a classDef round fill:#e6f0ff,stroke:#666,color:#1a1a1a classDef file fill:#f5f5f5,stroke:#999,color:#1a1a1a classDef terminal fill:#f0f0f0,stroke:#333,stroke-width:2px,color:#1a1a1a classDef gate fill:#fff5e6,stroke:#cc8800,stroke-width:1px,color:#1a1a1a classDef survive fill:#d4edda,stroke:#155724,color:#1a1a1a classDef drop fill:#f8d7da,stroke:#721c24,color:#1a1a1a class Dir terminal class F1,F2,F3,Fn file class S1,S2,T,Arb stage class R1,R2,Rn round class Gate,Vote gate class Surv survive class Drop1,Drop2 drop This scanner is basic: a handful of LLM-powered rounds over every file, no clever targeting. It brute-forces where a frontier model would reason strategically about where to look, and it works with surprising success. So I tested it at the scope a real hunt would use, on ...