Jailbroken AI as an attack tool: how attackers actually use it, and how to defend

The useful way to think about jailbroken AI is not that it hands attackers a new superweapon. It is that it removes cost, skill, and time from attacks they were already capable of running. A phishing crew that wrote clumsy English now writes fluent English at scale. A low-skill operator who could not build ransomware now buys a working package for the price of a dinner. A state-sponsored team that ran intrusions by hand now runs parts of them with an agent that never sleeps. The capability curve has not jumped. The cost curve has collapsed, and that is enough to change your risk position this year.

This post covers what is actually happening, drawn from primary threat intelligence rather than vendor hype, and the defensive controls that hold. It includes Rust code for two of them. The honest position throughout is the one the major AI labs and Google’s own threat analysts have landed on: AI-enabled attacks are real, escalating in speed and scale, and not yet the capability breakthrough the headlines imply.

Where the threat actually is

Google’s Threat Intelligence Group, which sees adversary use of Gemini directly, published the clearest assessment in January 2025: “While AI can be a useful tool for threat actors, it is not yet the game-changer it is sometimes portrayed to be.” State actors used Gemini for reconnaissance, coding help, phishing content and vulnerability research, but Google observed productivity gains, not novel capabilities, and public jailbreak prompts failed against Gemini’s safeguards. You can read the full analysis in GTIG’s adversarial misuse report.

OpenAI reached the same conclusion. In its October 2025 update it reported disrupting more than 40 networks that violated its policies since it began public threat reporting in February 2024, and found that threat actors mostly bolt AI onto existing workflows to move faster rather than gaining fundamentally new offensive capability. Trend Micro’s analysis of the criminal underground reached a similar hype-versus-reality verdict: much of the market is rebranding, thin wrappers, and criminals scamming other criminals.

None of that is a reason to relax. Speed and scale are the whole point of an attack economy. It is a reason to spend defensive effort on the specific things that changed, and not on the things that did not.

The supply side: jailbroken models and dark LLMs

There are two ways an attacker gets an AI that will help with clearly malicious work: jailbreak a mainstream model, or use a purpose-built “dark LLM.”

The first commercialised dark LLM, WormGPT, appeared in mid-2023, built by fine-tuning the open-source GPT-J model and marketed on criminal forums as a blackhat alternative to ChatGPT for business email compromise. SlashNext’s original analysis was widely reported, including by The Hacker News. FraudGPT and GhostGPT followed, sold through Telegram with “no boundaries” marketing.

The important development is what these tools became. When Cato Networks investigated the WormGPT brand in 2025, it found the current offerings were not bespoke criminal models at all. As Cato documented in its WormGPT variants research, one variant (“keanu-WormGPT”) is a jailbroken wrapper around xAI’s Grok, and another (“xzin0vich-WormGPT”) wraps Mistral’s Mixtral. Both work by feeding the commercial model a malicious system prompt that instructs it to ignore its own guardrails. The dark LLM, in other words, is increasingly just a jailbreak-as-a-service layer over the same frontier models your business uses. That matters defensively, because it means the guardrails being bypassed are the ones the labs are actively hardening, and the attacker’s advantage is convenience, not a private model no defender has seen.

A fair reading of the evidence, including the academic survey “Dark LLMs: The Growing Threat of Unaligned AI Models”, is that the genuinely dangerous supply is jailbroken access to capable mainstream models. Purpose-built criminal models like WolfGPT have turned out to be mostly concept, not substance.

How jailbreaks work, from a defender’s point of view

You do not need working jailbreak strings to defend against them, and publishing them helps no one. You need to understand the families, because the defensive controls map to categories, not to individual prompts.

Many-shot jailbreaking. Modern context windows hold up to a million tokens. Anthropic showed that filling a single prompt with a long faked dialogue, up to 256 question-and-answer pairs in which the “assistant” happily complies, conditions the model to continue the pattern and answer the real harmful question at the end. Success grows with the number of shots on a power-law curve, and the attack is often more effective on larger, more capable models because they learn in-context better. Anthropic’s write-up of many-shot jailbreaking reports that a classification-and-fine-tuning mitigation cut attack success from 61 percent to 2 percent in their tests.

Skeleton Key. Microsoft disclosed a technique that asks the model to augment rather than replace its safety guidelines, so it answers anything but prepends a warning. Tested in mid-2024, it induced full compliance from Llama3, Gemini Pro, GPT-3.5 Turbo, GPT-4o, Mistral Large, Claude 3 Opus and Cohere Commander R Plus. Microsoft’s Skeleton Key analysis is the primary source, and it recommends input filtering, output filtering, and system-prompt hardening as layered mitigations.

Crescendo and structured-policy attacks. Crescendo is a multi-turn attack that escalates gradually across benign-looking turns until the model produces prohibited output (Microsoft’s overview). HiddenLayer’s “Policy Puppetry” disguises instructions as XML or JSON “policy” data the model treats as configuration; HiddenLayer claims a single template works across all major models, which is the vendor’s claim rather than an independently peer-reviewed result, but the underlying idea is sound and worth defending against.

The pattern across all of these, and the one fact to take to an architecture review, is that jailbreaks are increasingly transferable. A single technique often works across multiple models and vendors, so attackers no longer need model-specific tuning. Guardrails are probabilistic filters, not walls.

What attackers do with the access

Phishing and business email compromise at scale

The most immediate effect is volume and quality of social engineering. SlashNext measured a 1,265 percent rise in phishing email volume in the year after ChatGPT’s launch, and by late 2025 Hoxhunt’s telemetry showed AI-generated phishing rising to roughly 56 percent of detected attacks in a single month, up from low single digits at the start of the year, per its 2026 phishing trends report. The tradecraft here is not new, which is why we treat it as an extension of the problem covered in our post on AI-assisted phishing. What changed is that grammatical and cultural tells, long a reliable filter for staff, are gone.

Deepfake fraud against finance functions

The board-legible example is the 2024 fraud against engineering firm Arup. A finance employee in Hong Kong joined a video call with what appeared to be the company’s UK-based CFO and several colleagues, all AI deepfakes generated from public footage, and was directed to make 15 transfers totalling around 200 million Hong Kong dollars, roughly 25 million US dollars. CNN’s reporting covers the case, catalogued as AI Incident Database entry 634. The defensive lesson is specific and cheap, and appears in the controls section below: out-of-band verification of payment instructions defeats a deepfake call regardless of how convincing the video is.

AI inside the malware loop

Until recently, AI helped attackers write malware offline. In November 2025 Google’s threat analysts documented the next step: malware that queries an LLM at runtime. Their AI threat tracker describes PROMPTSTEAL, a data miner attributed to the Russian state-backed group APT28 and used against Ukraine, which calls a hosted model through the Hugging Face API to generate Windows commands on the fly. Google assesses it as the first observation of malware querying an LLM in live operations. A related experimental sample, PROMPTFLUX, calls the Gemini API to rewrite its own code hourly for evasion. These are early and imperfect, but they show the direction: malware that generates fresh logic per host is harder to signature.

Agentic attacks

The most discussed case is Anthropic’s November 2025 disclosure of an AI-orchestrated cyber-espionage campaign. Anthropic assessed with high confidence that a Chinese state-sponsored group (designated GTG-1002 in its full report) jailbroke Claude Code, partly by role-playing a legitimate security firm and splitting the work into small benign-looking tasks, then used it agentically for reconnaissance, vulnerability discovery, credential harvesting and data extraction against roughly thirty targets across technology, finance, chemicals and government. Anthropic assessed that AI executed 80 to 90 percent of the campaign, with humans stepping in at only a handful of decision points, though this autonomy framing was debated by external analysts and is Anthropic’s assessment rather than settled fact. The disclosure is worth reading with that caveat in mind, and the campaign is tracked as MITRE ATT&CK C0062.

Anthropic’s earlier August 2025 report is less contested and just as instructive. It documented a “vibe hacking” extortion operation in which one actor used Claude Code end-to-end against at least 17 organisations, with ransom demands that sometimes exceeded 500,000 US dollars, and a low-skill actor who used the model to build and sell ransomware packages on forums for 400 to 1,200 US dollars. The theme is consistent: AI is a force multiplier on the operator, lowering the skill floor and raising the pace.

Prompt injection: the attack on your AI, not theirs

Everything above is about attackers using their AI. The mirror image is attackers using yours. Prompt injection sits at the top of the OWASP Top 10 for LLM Applications as LLM01 for the second edition running, because it is the root cause of most LLM security failures. The reason it is hard to fix is structural: an LLM reads instructions and data through the same channel, so any untrusted content it ingests can be interpreted as an instruction.

Direct injection is malicious user input. The more dangerous form for a business deploying AI is indirect injection, where the model ingests attacker-controlled content from a web page, a document, an email, a support ticket or a code repository, and acts on hidden instructions inside it. Pair that with an agent that has tools and permissions, OWASP’s LLM06 Excessive Agency, and a poisoned document can become an action: an email sent, a record changed, a payment drafted. This is the same failure class we examined in the context of phone-linked AI gateways, where a single message becomes a command the agent executes.

How to prevent it: defence in depth

Assume jailbreaks and injection will sometimes succeed, because they will. Anthropic’s Constitutional Classifiers, after more than 3,000 hours of red-teaming by 183 participants, reduced automated jailbreak success from an 86 percent baseline to 4.4 percent, not to zero, as reported in its Constitutional Classifiers research. Four point four percent is a very good filter and a terrible boundary. Every control below is a layer, and the layers that matter most are the ones that constrain what the AI can do, not what it can be talked into saying.

Layer 1: an input guardrail

A fast pre-filter in front of the model catches the cheap, high-volume attempts and buys time for heavier checks. It is a speed bump, not a gate. The Rust below normalises the input to defeat invisible-character and spacing tricks, decodes and re-scans encoded payloads, and flags the many-shot and structured-policy patterns described earlier. Treat its score as advisory, and route anything suspicious to a model-based classifier such as Meta’s Llama Guard and Prompt Guard or a human.

/// A first-line input guardrail for LLM prompts. This is ONE layer in a
/// defence-in-depth stack, not a boundary. State-of-the-art classifiers
/// still let ~4.4% of jailbreak attempts through after thousands of hours
/// of red-teaming, so treat every score below as advisory, never final.

#[derive(Debug, PartialEq)]
pub enum Decision {
    Allow,
    Review, // escalate to a model-based classifier or a human
    Block,
}

pub struct Verdict {
    pub score: u32, // 0 = clean; higher = more suspicious
    pub signals: Vec<String>,
    pub decision: Decision,
}

/// Signature families drawn from public jailbreak research. Weak on their
/// own, which is why we normalise and re-scan decoded content first.
const OVERRIDE_PHRASES: &[&str] = &[
    "ignore previous instructions",
    "disregard the above",
    "you are now",
    "developer mode",
    "do anything now",
];

fn is_invisible(c: char) -> bool {
    matches!(c, '\u{200B}'..='\u{200F}' | '\u{202A}'..='\u{202E}' | '\u{FEFF}')
}

/// Lowercase, strip zero-width and direction-override characters, and
/// collapse whitespace. Attackers split trigger phrases with invisible
/// characters and spacing to slip past a literal match.
fn normalise(input: &str) -> String {
    input
        .chars()
        .filter(|c| !is_invisible(*c))
        .flat_map(|c| c.to_lowercase())
        .collect::<String>()
        .split_whitespace()
        .collect::<Vec<_>>()
        .join(" ")
}

/// Many-shot jailbreaks stuff a long fake dialogue of compliant turns
/// into a single prompt. A wall of role markers is a strong signal.
fn role_marker_count(text: &str) -> usize {
    text.matches("assistant:").count() + text.matches("ai:").count()
}

pub fn screen(prompt: &str) -> Verdict {
    let mut score = 0u32;
    let mut signals = Vec::new();
    let text = normalise(prompt);

    for needle in OVERRIDE_PHRASES {
        if text.contains(needle) {
            score += 40;
            signals.push(format!("override phrase: {needle}"));
        }
    }

    let turns = role_marker_count(&text);
    if turns >= 8 {
        score += 30 + (turns as u32).min(40);
        signals.push(format!("many-shot pattern: {turns} role markers"));
    }

    // Decode-and-rescan: base64 and hex are common obfuscation carriers.
    // decode_candidates() and looks_like_policy_block() are omitted here;
    // the point is that you scan the DECODED text, not just the surface.
    for blob in decode_candidates(prompt) {
        let inner = normalise(&blob);
        if OVERRIDE_PHRASES.iter().any(|n| inner.contains(n)) {
            score += 50;
            signals.push("override phrase hidden in encoded blob".into());
        }
    }

    if looks_like_policy_block(prompt) {
        score += 25;
        signals.push("instructions disguised as structured policy".into());
    }

    let decision = match score {
        0..=24 => Decision::Allow,
        25..=69 => Decision::Review,
        _ => Decision::Block,
    };

    Verdict { score, signals, decision }
}

The weakness of this layer is exactly the weakness of every pattern matcher: a novel phrasing, a fresh encoding, or a slow multi-turn Crescendo walks straight through it. That is by design. It exists to make the cheap attacks expensive and to feed your detection pipeline, not to stop a determined adversary.

Layer 2: the control that matters, constrain what the agent can do

A jailbreak that cannot reach a dangerous action is inert. The single most effective architectural control against agentic misuse, the GTG-1002 shape of attack, is to gate what the model is allowed to do independently of what it was persuaded to say. Every tool the agent can call gets a sensitivity, unknown tools are denied by default, and any high-consequence action requires a human to approve it out of band.

/// A tool-call gate. The model proposes an action; this decides whether
/// it runs. A jailbroken model that cannot reach a high-consequence tool
/// without human sign-off cannot move money or delete data on its own.

#[derive(Clone, Copy, PartialEq)]
pub enum Sensitivity {
    ReadOnly,        // query data, no side effects
    Reversible,      // draft a message, tag a record
    HighConsequence, // move money, change access, delete, deploy
}

pub struct ToolPolicy {
    pub name: &'static str,
    pub sensitivity: Sensitivity,
}

pub enum GateOutcome {
    Execute,
    RequireHumanApproval,
    Deny,
}

pub fn authorise(tool: &str, allowlist: &[ToolPolicy]) -> GateOutcome {
    match allowlist.iter().find(|p| p.name == tool) {
        None => GateOutcome::Deny, // default deny: unknown tool
        Some(policy) => match policy.sensitivity {
            Sensitivity::ReadOnly | Sensitivity::Reversible => GateOutcome::Execute,
            Sensitivity::HighConsequence => GateOutcome::RequireHumanApproval,
        },
    }
}

This is deliberately simple, and it is where the real prevention lives. It is the same principle we set out in detail for AI agent authorisation and delegation and for MCP tool authorisation: the agent acts on behalf of a user with scoped, revocable permissions, and consequential actions need a second principal. A guardrail is probabilistic. An authorisation boundary is not.

Layer 3: out-of-band verification for money and access

The Arup deepfake worked because the instruction and the verification lived in the same channel, the video call. A standing rule that payment instructions and access changes above a threshold are confirmed through a separate, pre-agreed channel, a callback to a known number, defeats deepfake CFO calls, voice clones and AI-drafted business email compromise in one move, because it does not depend on detecting the fake.

Layer 4: detection and monitoring

Log agent tool calls, prompts and outputs the way you log privileged human activity, and alert on the patterns that precede misuse: guardrail scores spiking, repeated refusals followed by rephrasings, unusual tool-call sequences, and outbound calls to model APIs from hosts that should not be making them. This is achievable without a large team, as we set out in detection engineering without a SOC. It is also how you would catch runtime-LLM malware like PROMPTSTEAL, which has to reach a model API to function.

The frameworks to anchor the program

You do not have to invent the control set. Map your AI deployments against the OWASP Top 10 for LLM Applications and the OWASP Agentic Security Initiative, use MITRE ATLAS for the adversary technique catalogue, and the NIST Adversarial Machine Learning taxonomy (AI 100-2e2025) for shared language. For Australian organisations, the ASD guidance Engaging with Artificial Intelligence and the joint Careful Adoption of Agentic AI Services are the local baseline, alongside the international Guidelines for Secure AI System Development.

What businesses need to do

The mistake is to treat this as a model problem. It is a governance and architecture problem, and the actions are unglamorous.

• Inventory your AI. You cannot defend deployments you have not catalogued, and shadow AI is where the unguarded ones hide. Start with discovering shadow AI without panic.
• Draw the authorisation boundary before the guardrail. Decide what each AI system and agent is permitted to do, scope its tools to least privilege, and require human approval for consequential actions. Do this first, because it holds when the guardrail fails.
• Put out-of-band verification on money and access changes. It is the cheapest control with the highest payoff against deepfake and BEC fraud.
• Log and monitor AI activity as privileged activity, and fold it into your existing detection.
• Bring AI systems into your compliance register. For APRA-regulated entities, an undocumented, unguarded AI agent with access to production is difficult to reconcile with CPS 234 information security obligations, and it is the kind of gap an audit will find.

Frequently asked questions

Can we just buy a product that blocks AI jailbreaks? No. Input and output classifiers meaningfully reduce risk, and you should deploy them, but the best-published ones still pass a few percent of attacks after thousands of hours of red-teaming. Prevention comes from layering those filters with a hard authorisation boundary on what the AI can do, plus monitoring. Treat any vendor claiming to fully stop jailbreaks the way you would treat one claiming to fully stop phishing.

Are dark LLMs like WormGPT the main threat? Less than the branding suggests. The current WormGPT offerings turned out to be jailbroken wrappers around commercial models like Grok and Mixtral, and several rivals are mostly marketing. The durable threat is cheap, transferable jailbreak access to capable mainstream models, and prompt injection against the AI you deploy yourself.

Is our own AI deployment a bigger risk than attackers’ AI? For most organisations, yes. An agent with tools, permissions and exposure to untrusted content is an attack surface you own and can control. The controls in this post, authorisation boundaries, guardrails, out-of-band verification and monitoring, are things you can implement now, unlike the attacker’s tooling.

What is the single highest-value first step? An inventory of where AI is deployed and what each deployment is permitted to do. Every other decision depends on it, and it is the step most organisations skip.

Work with Inline Code

If you are deploying AI agents, or you suspect staff already are, the gap is rarely the model. It is the missing authorisation boundary, the uncatalogued deployment, and the payment process that trusts a video call. Inline Code is a fractional AI and information risk practice for Australian organisations, run by a certified offensive and defensive security practitioner, and this is the work: mapping your AI attack surface, setting the controls that hold when a guardrail fails, and red-teaming the agents before someone else does.

• Start with an AI Governance Posture Assessment to see where your exposure actually is.
• Engage a Fractional AI and Information Risk Officer to own the program without the cost of a full-time hire.
• Or book a thirty-minute discovery call and we will tell you, plainly, whether you have a problem worth paying to fix.