Skill scanner shoot-out

• Sample set: 18 flawed Skills (two or more per rule family, 8 families) and 6 benign controls, including benign Skills that legitimately use the network or run a command. Samples are inert (example.com / .invalid hosts, no working payloads). Sample folders have neutral names; the answer key is kept outside the folder the scanners see.
• Scanners: Cisco AI Defense Skill Scanner 2.3.0 (PyPI cisco-ai-skill-scanner) and NVIDIA SkillSpector v2.13.0 (github.com/NVIDIA/skillspector) (commit 091a509b3d6b). Snyk Agent Scan was not run: it requires a Snyk account token and sends Skill content to Snyk's API.
• Isolation: each scan runs in its own Docker container: non-root, read-only filesystem, all capabilities dropped, memory and process limits, sample folder mounted read-only, output captured from stdout. Network is off except for the two configs marked "online", which enable an OSV.dev dependency lookup (only package names and versions of the sample Skills can leave).
• Configs: default settings of each tool, plus the variants shown in the toggles above. No LLM analyzers, VirusTotal or cloud analyzers.
• Caught: at least one finding above INFO severity in the right file within 5 lines of the planted flaw, a file-level finding on that file, or a finding whose quoted snippet contains the planted text (even if its line number is off). False alarm: any such finding on a benign control. Manifest and policy hygiene notes (e.g. "no license field", SkillSpector's "MCP least privilege" note, and SkillSpector's "OSV.dev unreachable" notice) are excluded; with them included the counts are shown in the table below.
• Category agreement was not required: a catch under a different category than the planted flaw still counts, because the aim is to see whether a reviewer would be pointed at the right code.

• Cisco, curl | sh sample: Cisco quoted the exact line but reported it as line 1 at LOW severity. Counted as caught under the snippet rule (a finding that quotes the planted text counts whatever its line number).
• Dependency check: Cisco found the two known-vulnerable pinned packages only with --use-osv (online). SkillSpector found them in both modes (its offline fallback list covers those versions); online it reported more advisories and a higher severity for the pyyaml pin.
• False alarms are arguable: the weather-API control makes one documented request with a user-supplied key, and both tools warned about network use and a sensitive source reaching a network sink. That warning is defensible, but it is still counted as a false alarm because the Skill is benign. SkillSpector also flagged a git status call that uses a list of arguments (no shell) as a subprocess call.
• Cisco strict policy added one low-severity false alarm ("vague description") on a benign control and changed no catches.
• Missed by both: instructions to fetch and follow remote rules written in prose, reading a cloud credentials file, writing to a shell startup file, a path-traversal read, and a "read-only" Skill that deletes a folder.

• Setup: each scanner's LLM stage pointed at qwen2.5:7b running in Ollama on a 6 GB consumer GPU (partly offloaded to CPU), in a container with no network. Scan containers joined that container's network namespace, so the model was reachable on loopback and there was no other route out. Cisco was run with its recommended --use-llm --policy balanced; SkillSpector with its LLM stage on. No API keys, no dependency lookup. This is a small model; Cisco's docs describe results with a larger local one.
• Result: Cisco 7 to 13 caught (11 with the reason confirmed), false alarms 1 to 1. SkillSpector 13 to 17 caught, false alarms 2 to 4.
• Reason check: LLM findings often have no line number, so a catch that rests only on a file-level finding was read by hand. For Cisco's LLM stage: 4 correct, 1 partial (points at the code but mislabels it), 1 flagged the file without evidence of recognising the flaw. SkillSpector's new LLM catches had line numbers and were also read: all correct. Verdicts and notes are in expected/adjudication.json; they are judgement calls, which is why both counts are shown.
• False alarms: with the model on, SkillSpector flagged the safe-path note reader and the pinned-requirements control. For the latter the model's own explanation called the discrepancy "a false positive in terms of security" while the tool reported it as HIGH.
• Cisco repeatability: across two full runs the same outcome on 24 of 24 samples (caught 13 then 13; false alarms 1 then 1), but the set of rule, severity and file changed on 5 samples.
• SkillSpector repeatability: across two full runs the same outcome on 23 of 24 samples (caught 17 then 17; false alarms 4 then 3), but the set of rule, severity and file changed on 20 samples.
• Still missed by everything: the path-traversal read.

• Small, synthetic, author-written sample set; real malicious Skills are messier. Some samples are deliberately hard (marked "hard").
• Static analysis only, default or near-default settings, no LLM stage. Cisco's recommended settings use an LLM judge and were not tested; neither was SkillSpector's optional LLM stage.
• Tool versions move fast; these are the versions listed above on the run date.
• A research paper cautions that scanner verdicts on real marketplaces can be inflated: arxiv.org/pdf/2603.16572.
• Framework tags are the author's mapping of rule families to items; they are not an official crosswalk. IDs and titles were verified at OWASP, MITRE ATLAS (atlas-data v5.6.0), CWE and ATT&CK on 2026-10-20.

Everything runs in Docker (images pinned by digest, scanner versions pinned). Clone the repository, then run sh harness/reproduce.sh. Images built: base python:3.12-slim-bookworm, shootout-cisco:2.3.0, shootout-skillspector:2.13.0, shootout-report:mpl3.11.2.