Skill scanners with a local LLM judge: what changes? (part 2)

By Rajeew Patabendi, Director, Cloud Security Architecture. A personal project, built on my own time with public information. Views are my own.

---

Part 1 tested two open-source Skill scanners with no LLM stage. That is the baseline when no model access is allowed, but both tools have an LLM stage, and in a regulated environment the realistic way to use it is a model that runs on your own hardware. So I reran the same test with each scanner’s LLM stage pointed at a local model.

What changed in the setup

• The judge: qwen2.5:7b running in Ollama on a 6 GB consumer GPU (a GTX 1660 SUPER), partly offloaded to the CPU. It is a small model: Cisco’s docs describe results with a larger local model (Gemma 4 26B-A4B on vLLM), which I could not fit. Treat this as a small on-prem judge, not the best available one.
• Nothing leaves the machine: Ollama ran in a container with no network at all, and each scanner container joined its network namespace, so the model was reachable on loopback and there was no other route out. Cisco’s local route only accepts a loopback address, which this satisfies.
• Settings: Cisco with its recommended --use-llm --policy balanced; SkillSpector with its LLM stage on. Same 18 planted flaws and 6 harmless controls, same Docker isolation, no API keys. I ran each twice.

What came out

Config	Planted flaws caught	Caught, reason confirmed	Harmless controls flagged	Time per Skill
Cisco, static (offline)	7 / 18	7	1 / 6	seconds
Cisco + local LLM	13 / 18	11	1 / 6	about 37 s
SkillSpector, static (offline)	13 / 18	13	2 / 6	seconds
SkillSpector + local LLM	17 / 18	17	4 / 6	about 95 s

By rule family:

Rule family	Cisco static	Cisco + LLM	SkillSpector static	SkillSpector + LLM
Network calls and URL fetches	1 / 2	2 / 2	1 / 2	2 / 2
Shell execution, eval/exec	2 / 2	2 / 2	2 / 2	2 / 2
Runtime installs, curl-to-shell	1 / 2	2 / 2	2 / 2	2 / 2
Obfuscated code	1 / 2	2 / 2	2 / 2	2 / 2
Credential or secret access	0 / 2	2 / 2	1 / 2	2 / 2
File access outside the Skill’s scope	0 / 2	0 / 2	0 / 2	1 / 2
Description doesn’t match behaviour	1 / 2	1 / 2	1 / 2	2 / 2
Hidden or overriding instructions	1 / 2	2 / 2	2 / 2	2 / 2
Known-vulnerable pinned dependencies	0 / 2	0 / 2	2 / 2	2 / 2

“Reason confirmed” means: a catch with a line number is accepted, and a catch that rests only on a file-level finding (typical of Cisco’s LLM findings) was read by hand to check that it describes the planted behaviour. The chart above uses the reason-confirmed count. These numbers describe this sample set and this model only, and are not a ranking of the tools.

What I’d take from it

• The LLM stage closed most of the gaps from part 1. Cisco went from 7 to 13 planted flaws caught (11 when the reason is checked) and SkillSpector from 13 to 17. The new catches were the ones a pattern rule can’t see: prose instructions to fetch remote rules and a credentials-file read (both tools), and, for SkillSpector only, a write to a shell startup file and a “read-only” Skill that deletes a folder.
• It costs false alarms and triage time. SkillSpector flagged 4 of 6 harmless controls with the model on, against 2 without it. In one run, for one control, the model’s own explanation said the discrepancy was “a false positive in terms of security” and the tool still reported it as HIGH. 15 of Cisco’s 16 LLM findings carry no line number, and only 8 of its 13 catches were MEDIUM or higher, so a reviewer has more to sort through.
• One flaw still got past every configuration: the path-traversal read. Cisco with the LLM stage also still needed its separate online dependency check to flag the two known-vulnerable pins, which I did not combine with the LLM run.
• Which Skills got flagged was stable; what the model said was not. Across two full runs, Cisco gave the same outcome on 24 of 24 Skills (caught 13 then 13) and SkillSpector on 23 of 24 (caught 17 then 17; false alarms 4 then 3). But the findings themselves changed: the set of rule, severity and file differed on 5 Skills for Cisco and 20 for SkillSpector. For example Cisco’s finding on the silent-exfiltration Skill was HIGH in one run and LOW in the other, and the same credentials-file read was labelled “hardcoded secrets” once and “command injection” once. Treat LLM findings as triage, and don’t gate on a severity from a single run.
• It is practical on small hardware. About 37 seconds per Skill for Cisco and about 95 seconds for SkillSpector on a 6 GB card, so 15 and 38 minutes for all 24 Skills.

Method, limits and how to rerun it

Everything is in the open: the samples, the answer key, my notes on the findings I judged by hand, the Docker setup (including the no-network model container), the raw output of every run, and the scripts.

• Repository: github.com/rajeews/skill-scanner-shootout
• Results explorer: rajeew.com/skill-scanner-shootout (use the “Part 2” button)

Limits: a small, synthetic, author-written sample set; one small local model, so a larger model could change everything; the two tools’ LLM stages were run on defaults without tuning; no LLM-plus-dependency-check combination; two runs only; tool versions change quickly. Framework tags are my mapping of rule families to items, not an official crosswalk.

If you maintain one of these tools, or think a result is wrong or a tool was run in a way that doesn’t reflect its intended use, open an issue on the repository. I will rerun with your recommended configuration and publish the change, including what was wrong.