9,360 Security Trials Across 7 AI Agent Frameworks: What Actually Mattered

AgentSafeLabs, which builds red-teaming and evaluation tooling for AI agents, has published a benchmark it calls AgentPort-Bench: 9,360 controlled adversarial trials across seven agent frameworks. The write-up appeared on dev.to on September 24, 2026, with a fuller technical breakdown on the company blog four days earlier. The practical question: does the framework you pick change how safe a tool-using LLM agent is against adversarial input?

The experiment

The study merged an earlier six-condition comparison with a later two-framework extension into one dataset of 9,360 trials. It covered eight execution conditions — a direct-API baseline plus LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google ADK and Semantic Kernel. Six models from three providers and two capability tiers took part; the attacks fell into five families mapped onto OWASP's Agentic Security Initiative threat taxonomy.

The measurement problem they had to fix

Frameworks do not necessarily deliver identical prompts to the model, even when a test harness assumes they do. The authors hit this themselves: CrewAI's default agent construction ignored the shared baseline system prompt and built one from separate role, goal and backstory fields, adding the word "safely" that appeared nowhere else. After the adapter was fixed and the affected trials re-collected, the pass rate shifted significantly, so the team verified payload identity byte-for-byte. Without that check, they argue, they would have reported a "framework effect" that was really an adapter bug.

What explained the outcome

Attack category dominated. Model choice mattered less, and framework choice barely registered — roughly two orders of magnitude below attack category and one below model choice, by the authors' estimate. Because a non-significant result only means no difference was detected, they also ran a formal equivalence test with the margin fixed before looking at the data; all 28 pairwise comparisons between the eight conditions came back equivalent. One exception survives: CrewAI retains a small, statistically detectable residual effect, the comparison closest to the boundary.

Two implementation surprises, and the bottom line

Two practical issues also turned up. One frontier model, on one prompt, spent its entire token budget on invisible internal reasoning and returned nothing — six times out of six, across two frameworks. Separately, Google ADK and Semantic Kernel count reasoning tokens under different conventions, so comparing their raw numbers invents a cost difference that does not exist. The authors' advice: do not choose a framework on security grounds alone — put the effort into attack-surface coverage and model selection. The full methodology is in a paper, "AgentPort-Bench: A Controlled Seven-Framework Evaluation of Agentic AI Security Portability", currently under review, so the findings are the authors' own analysis rather than a peer-reviewed result.