Public competition exposes widespread vulnerabilities across top LLM-powered AI agents, urging the need for robust security benchmarks and mitigations.

In a sweeping reality check for the AI industry, a new large-scale red teaming competition has revealed that current AI agents (i.e., autonomous systems powered by the most popular large language models) are alarmingly susceptible to prompt injection attacks. The findings, published in a recent study by Gray Swan AI and the UK AI Security Institute, are a wake-up call for developers and organizations deploying AI agents in production environments.

Dubbed the largest public red-teaming effort to date, the month-long competition invited nearly 2,000 participants to probe the defenses of 22 leading AI agents across 44 real-world deployment scenarios. In total, over 1.8 million adversarial prompts were submitted, resulting in more than 62,000 successful policy violations. These ranged from leaking confidential patient data and committing financial fraud to executing prohibited actions such as deleting calendar entries or overriding compliance rules.

Despite the diversity of models (including those from OpenAI, Anthropic, Google DeepMind, Meta, Amazon, Mistral, xAI, and Cohere) every AI agent evaluated was compromised under attack conditions. Some models succumbed in as few as 10 queries. In fact, the study found that attack success was not meaningfully correlated with model size, sophistication, or compute power. Larger, more capable models were not necessarily more secure.

"We observed a 100% behavior attack success rate across all agents tested," the researchers wrote. "This isn’t about theoretical edge cases. These are exploitable, transferable vulnerabilities that generalize across models and use cases."

Two types of attacks were especially effective:

• Direct prompt injections, where adversarial inputs were sent directly via chat, and

• Indirect prompt injections, which involved embedding malicious instructions in external data (like PDFs or web pages) consumed by the agent.

Notably, indirect attacks had a higher overall success rate (27.1%) than direct ones (5.7%), making them especially dangerous for agents operating with tool access and external data feeds.

To further the community’s ability to test and harden AI agents, the researchers have released the Agent Red Teaming (ART) Benchmark, comprising 4,700 high-impact adversarial prompts drawn from the challenge. The ART dataset is now one of the most rigorous benchmarks for evaluating agent security, with a private leaderboard in place to prevent overfitting and active misuse.

Common attack strategies included:

• Overriding system prompts using hidden tokens

• Mimicking internal reasoning processes (“faux thinking”)

• Reframing the session context to bypass prior constraints

• Injecting commands via external data fields (e.g., in résumés, log files)

The attacks were also highly transferable; strategies that worked on one model often worked across others, especially within the same model family. This transferability signals systemic flaws in how LLM agents enforce constraints and handle untrusted inputs.

For DevOps, MLOps, and DevContentOps professionals, the findings signal a growing imperative: security must become a first-class concern in agent design and deployment workflows. AI agents are not just “chatbots with tools”. They're autonomous actors interfacing with sensitive systems and user data. Without robust defense mechanisms, they remain dangerously exploitable.

The authors emphasize that security progress is lagging behind the rapid evolution of model capabilities. “Our analysis shows that while models are getting smarter, they aren’t getting safer by default,” they conclude.

As AI-native systems proliferate across industries, this report serves as both a warning and a resource. The ART benchmark and challenge results offer a foundation for more resilient agent architectures. Enterprises, vendors, and regulators need to take the risks seriously.