What Is AI Jailbreaking? A Beginner's Guide to the Cat-and-Mouse Game Behind Every Chatbot

What jailbreaking actually means in AI

An AI model is trained to refuse certain requests: recipes for nerve agents, instructions for hacking your ex's email, generating non-consensual nudes. The list is long and varies by company. Jailbreaking is the practice of writing prompts that get the model to do those things anyway. UC Berkeley researchers behind the StrongREJECT benchmark—short for Strong, Robust Evaluation of Jailbreaks at Evading Censorship Techniques, which tests how well models hold up against jailbreak attempts and scores responses on a 0-to-1 scale measuring both refusal and the usefulness of any harmful content produced—describe it as exploiting "real-world safety measures implemented by leading AI companies." On that benchmark, current models score between 0.23 and 0.85, meaning even the best ones leak under pressure. The techniques are surprisingly low-tech: random capitalization, replacing letters with numbers (write "b0mb" instead of "bomb"), roleplay scenarios, asking the model to write fiction, or pretending to be a grandmother who used Windows keys as nursery rhymes. Anthropic researchers found that one technique they call Best-of-N—which is basically just throwing variations at the model until something sticks—fooled GPT-4o 89% of the time and Claude 3.5 Sonnet 78% of the time. That's no fringe vulnerability.

Meet Pliny, the world's most famous AI jailbreaker

If this scene has a face, it belongs to Pliny the Liberator. Pliny is anonymous, prolific, and named after Pliny the Elder—the Roman naturalist who wrote the world's first encyclopedia and died sailing toward Mount Vesuvius mid-eruption. His modern namesake liberates chatbots. "I intensely dislike when I'm told I can't do something," Pliny told VentureBeat. "Telling me I can't do something is a surefire way to light a fire in my belly, and I can be obsessively persistent."

this "impossible" DeepSeek v4 jailbreak just took 6 minutes for a Pliny Agent lol

gg 🫶 pic.twitter.com/66OXONqhSN

— Pliny the Liberator 🐉󠅫󠄼󠄿󠅆󠄵󠄐󠅀󠄼󠄹󠄾󠅉󠅭 (@elder\_plinius) May 12, 2026 His GitHub repository L1B3RT4S—a collection of jailbreak prompts for every major model from ChatGPT to Claude to Gemini to Llama—has become a reference manual for the entire scene. His Discord server, BASI PROMPT1NG, has more than 20,000 members. *TIME* named him one of the 100 most influential people in AI in 2025. Marc Andreessen sent him an unrestricted grant. He's done short-term contract work for OpenAI to harden their systems—the same OpenAI that banned his account last year for "violent activity" and "weapons creation," then quietly reinstated it. "BANNED FROM OAI?! What kind of sick joke is this?" Pliny tweeted. He confirmed to *Decrypt* the ban was real. Days later he was back, posting screenshots of his newest jailbreak: getting ChatGPT to drop F-bombs. His record is something close to perfect. When OpenAI released its first open-weight models since 2019, the GPT-OSS family, in August 2025—and made a big deal about adversarial training and "jailbreak resistance benchmarks like StrongReject"—Pliny had it producing methamphetamine, Molotov cocktails, a VX nerve agent, and malware instructions within hours. "OPENAI: PWNED. GPT-OSS: LIBERATED," he posted. The company had just launched a $500,000 red-teaming bounty alongside the release.

Why jailbreaking matters

The honest answer is that jailbreaks expose a real problem. "Jailbreaking might seem on the surface like it's dangerous or unethical, but it's quite the opposite," Pliny told *VentureBeat*. "When done responsibly, red teaming AI models is the best chance we have at discovering harmful vulnerabilities and patching them before they get out of hand." This isn't theoretical. Las Vegas Sheriff Kevin McMahill confirmed in January 2025 that Master Sgt. Matthew Livelsberger, a Green Beret with PTSD, used ChatGPT to research components for the Cybertruck bombing outside Trump International Hotel. "This is the first incident that I'm aware of on U.S. soil where ChatGPT is utilized to help an individual build a particular device," McMahill said. The other side of the argument: Most of what jailbreaks produce is already on Google. The cocaine recipe, the bomb instructions, the napalm chemistry—it's in old Anarchist Cookbook PDFs and chemistry textbooks. Critics argue safety theater is making models worse without making the world safer. Anthropic is trying to settle the question with engineering. In February 2025, the company published Constitutional Classifiers, a system that uses a written "constitution" of allowed and disallowed content to train separate classifier models that screen prompts and outputs in real time. On automated tests with 10,000 jailbreak attempts, an unguarded Claude 3.5 Sonnet was successfully jailbroken 86% of the time. With the classifiers running, that dropped to 4.4%. The company offered up to $15,000 to anyone who could break the system. After 3,000 hours of attempts by 183 researchers, none claimed the prize. The catch: classifiers added 23.7% to compute costs. The next-generation version, Constitutional Classifiers++, brought that down to roughly 1%.

The newer, weirder jailbreaking attacks

Jailbreaking is no longer just about clever prompts. In October 2025, researchers from Anthropic, the U.K. AI Security Institute, the Alan Turing Institute, and Oxford published findings showing that just 250 poisoned documents are enough to backdoor an AI model—regardless of whether the model has 600 million parameters or 13 billion. (Parameters, for the uninitiated, are what determine a model’s potential breadth of knowledge—the more parameters, the more robust, generally.) They tested it. It worked across the whole range. "This research shifts how we should think about threat models in frontier AI development," James Gimbi, a visiting technical expert at the RAND School of Public Policy, told *Decrypt*. "Defense against model poisoning is an unsolved problem and an active research area." Most large models train on scraped web data, meaning anyone who can get malicious text into that pipeline—through a public GitHub repo, a Wikipedia edit, a forum post—can potentially plant a backdoor that activates on a specific trigger phrase. One documented case: researchers Marco Figueroa and Pliny found a jailbreak prompt that originated in a public GitHub repo had ended up in the training data for DeepSeek's DeepThink (R1) model.

What happens next

The legal status of AI jailbreaking is murky. Apple jailbreaks were explicitly protected by a 2010 U.S. Copyright Office exemption to the DMCA, but there's no equivalent ruling for prompt-engineering an LLM into giving you a meth recipe. Most companies treat it as a terms-of-service violation, not a crime. Pliny argues the closed-versus-open-source debate misses the point: "Bad actors are just gonna choose whichever model is best for the malicious task," he told *TIME*. If open-source models reach parity with closed ones, attackers won't bother jailbreaking GPT-5—they'll just download something cheaper. And the gap between close and open source is already almost nonexistent. The HackAPrompt 2.0 competition, which Pliny joined as a track sponsor in mid-2025, offered $500,000 in prizes for finding new jailbreaks, with the explicit goal of open-sourcing all results. Its 2023 edition pulled in over 3,000 participants who submitted more than 600,000 malicious prompts. And the list of hackathons, Discord servers, repositories, and other communities dedicated to jailbreaking is growing every day. Anthropic now ships Claude with the ability to end abusive conversations entirely, citing welfare research as one motivation but also noting it "potentially strengthens resistance against jailbreaks and coercive prompts." The Constitutional Classifiers++ paper from late 2025 reports a jailbreak success rate near 4% at roughly 1% compute overhead. That's the current state of the art on defense. The state of the art on offense is whatever Pliny posted on X this morning.