July 31, 2025
5 min read
Will Knight
RunSybil’s AI agents probe websites for vulnerabilities, showcasing a new AI-driven era in cybersecurity and penetration testing.
I Watched AI Agents Try to Hack My Vibe-Coded Website
RunSybil, a startup founded by OpenAI’s first security researcher, deploys AI agents that probe websites for vulnerabilities—ushering in a new era of AI-powered cybersecurity. A few weeks ago, I observed a small team of artificial intelligence agents spend about 10 minutes attempting to hack my newly created vibe-coded website. The AI agents, developed by the startup RunSybil, collaborated to identify weak spots on my site. An orchestrator agent named Sybil oversees several specialized agents, all powered by a mix of custom language models and off-the-shelf APIs. Unlike traditional vulnerability scanners that look for specific known issues, Sybil operates at a higher level, using artificial intuition to discover weaknesses. For example, it might detect that a guest user has privileged access—something a regular scanner could miss—and exploit this to build an attack. Ariel Herbert-Voss, CEO and cofounder of RunSybil, believes that increasingly capable AI models will revolutionize both offensive and defensive cybersecurity. “I would argue that we're definitely on the cusp of a technology explosion in terms of capabilities that both bad and good actors can take advantage of,” Herbert-Voss said. “Our mission is to build the next generation of offensive security testing just to help everybody keep up.” The website targeted by Sybil was one I recently built using Claude Code to help me sift through new AI research papers. The site, called Arxiv Slurper, consists of a backend server that accesses Arxiv—where most AI research is posted—along with other resources. It combs through paper abstracts for keywords like “novel,” “first,” “surprising,” and some technical terms I’m interested in. Although it’s a work in progress and required some manual bug fixes and configuration, I was impressed with how easy it was to assemble something potentially useful. A key challenge with vibe-coded sites like this is that it’s difficult to know what security vulnerabilities might exist. So when I spoke with Herbert-Voss about Sybil, I asked if it could test my new site for weaknesses. Thankfully, because my site is quite basic, Sybil did not find any vulnerabilities. Herbert-Voss explained that most vulnerabilities arise from more complex features like forms, plug-ins, and cryptographic components. We then watched as the same agents probed a dummy ecommerce website with known vulnerabilities, owned by Herbert-Voss. Sybil mapped the application and its access points, probed for weak spots by manipulating parameters and testing edge cases, and chained together findings to escalate attacks until it broke something meaningful. In this case, it successfully identified ways to hack the site. Unlike a human tester, Sybil runs thousands of these processes in parallel, never misses details, and never stops. “The result is something that behaves like a seasoned attacker but operates with machine precision and scale,” Herbert-Voss said. “AI-powered penetration testing is a promising direction that can have significant benefits for defending systems,” said Lujo Bauer, a computer scientist at Carnegie Mellon University (CMU) specializing in AI and computer security. Bauer recently coauthored a study with others from CMU and AI company Anthropic exploring AI penetration testing. The researchers found that the most advanced commercial models could not perform network attacks directly, but by setting high-level objectives like scanning a network or infecting a host, they enabled AI to conduct penetration tests. Sarah Guo, an investor and founder at Conviction, which backs RunSybil, noted that it’s rare to find people who understand both AI and cybersecurity. She added that RunSybil promises to make the kind of security assessments large companies perform periodically more widely available and continuous. “They can do baseline penetration testing with models and tool use continuously,” she said. “So you'll always have a view of what it really looks like to be under attack.” These techniques may become even more necessary as attackers develop their own AI strategies. “We have to assume that attackers are already using AI to their benefit,” said Bauer. “So developing pen-testing tools that use it is both responsible and likely necessary to balance the increasing risk of attack.” Herbert-Voss is well positioned to lead this effort, having been the first security researcher at OpenAI. “I built all sorts of crazy things like new prototypes of polymorphic malware, spearphishing infrastructure, reverse engineering tools,” he said. “I was concerned that we didn’t have a solution for when everybody gets access to language models—including the bad guys.”This article is part of Will Knight’s AI Lab newsletter, a weekly dispatch from beyond the cutting edge of AI.
Source: WIRED