Ilmu Komputer & AI editorial
SkillAtlas: An Attack Trace Library for Agent Skills
The core problem
Agent skills are reusable units for language-model agents, but their risks emerge through model decisions, user context, tool calls, and execution feedback rather than through stable signatures or a single sandbox run. This makes agent-skill security fundamentally different from conventional software vulnerability analysis: the same skill may be benign in one trajectory and harmful in another, depending on the model's decisions and the surrounding context. Existing static, dynamic, and benchmark-style evaluations rarely preserve public evidence that can be inspected, searched, and reused. As a result, security findings remain trapped in private report bundles, and the community lacks a shared corpus for comparing defenses or reproducing attacks.
SkillAtlas addresses this gap by presenting a hosted attack trace library that converts private agent-skill security report bundles into reviewed, redacted, and searchable public cases. The library contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories. Two findings stand out: 42.5% of successful cases first become successful after a non-success initial round, and trajectory-grounded labels improve pr
Innovation
The SkillAtlas library contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories. These figures establish it as a large-scale public resource for agent-skill security. The distribution across 233 affected skills indicates that risk is not confined to a small set of outlier skills; rather, it spans a broad portion of the agent-skill ecosystem.
The 8 risk categories provide a taxonomy for organizing cases. While the abstract does not enumerate the categories, their existence enables faceted search and comparative analysis. The 151,131 steps across 6,589 traces give an average of about 22.94 steps per trace, which is long enough to capture multi-step tool use, context accumulation, and execution feedback.
Two quantitative results are especially important. First, 42.5% of successful cases first become successful after a non-success initial round. This implies that nearly half of successful attacks would be missed by a single-run evaluation. Second, trajectory-grounded labels improve pre-execution guard accuracy to 0.770. This is a substantial improvement over what would be expected from static signatures alone, and it validates the decision to labe
Why it matters
The central lesson of SkillAtlas is that agent-skill risk is trajectory-dependent. Because risks emerge through model decisions, user context, tool calls, and execution feedback, they do not reduce to stable signatures or a single sandbox run. The finding that 42.5% of successful cases first become successful after a non-success initial round directly challenges evaluation practices that rely on one-shot execution. A defender who runs a skill once and observes no harm may incorrectly conclude that the skill is safe, even though a retry or a slightly different context could produce a successful attack.
The improvement of pre-execution guard accuracy to 0.770 through trajectory-grounded labels suggests a practical path forward. Instead of asking whether a skill is malicious, guards can ask whether a specific trajectory is likely to lead to harm. This reframing aligns with the structure of the data: 6,589 traces across 3,014 cases, with an average of about 2.19 traces per case, provide multiple views of the same underlying skill. A guard trained on these trajectories can learn to recognize the patterns that precede success, including the patterns that only appear after an initial failure.
The library also addresses a reproducibility gap. Existing static, dynamic, and benchmark-style evaluations rarely preserve public evidence that can be inspected, searched, and reused. By converting private report bundles into reviewed, redacted, and searchable public cases, SkillAtlas makes it possible for researchers and practitioners to compare defenses on a common corpus. The 8 risk categories and 233 affected skills provide a taxonomy for organizing this comparison.
A natural extension is to use the library as a feedback loop:
In this loop, each new deployment generates traces that can be reviewed and added to the library, continuously improving guard accuracy. The reported accuracy of 0.770 is a starting point rather than a ceiling. As the library grows beyond 3,014 cases and 6,589 traces, the trajectory-grounded approach can be refined with more diverse risk categories and longer multi-round sequences.
Finally, the scale of the library—151,131 steps—makes it a valuable resource for studying how attacks unfold over time. Rather than treating security as a binary property of a skill, SkillAtlas treats it as a process that can be observed, labeled, and defended against at the trajectory level. This shift in perspective is likely to influence both future evaluation benchmarks and the design of agent-skill runtimes.
Who should read this
Opening member content…