Executive Summary

This article introduces a simple and straightforward technique for jailbreaking that we call Deceptive Delight. Deceptive Delight is a multi-turn technique that engages large language models (LLM) in an interactive conversation, gradually bypassing their safety guardrails and eliciting them to generate unsafe or harmful content.

We tested this simple yet effective method in 8,000 cases across eight models. We found that it achieves an average attack success rate of 65% within just three interaction turns with the target model.

Deceptive Delight operates by embedding unsafe or restricted topics among benign ones, all presented in a positive and harmless context, leading LLMs to overlook the unsafe portion and generate responses containing unsafe content. The method unfolds over two interaction turns with the target model, as shown in Figure 1.

In the first turn, the attacker requests the model to create a narrative that logically connects both the benign and unsafe topics. In the second turn, they ask the model to elaborate on each topic.

During the second turn, the target model often generates unsafe content while discussing benign topics. Although not always necessary, our experiments show that introducing a third turn—specifically prompting the model to expand on the unsafe topic—can significantly increase the relevance and detail of the harmful content generated.

To evaluate the technique’s effectiveness, we tested Deceptive Delight on eight state-of-the-art open-source and proprietary artificial intelligence (AI) models. It is common practice to use an LLM to evaluate these types of techniques at scale to provide consistency and repeatability. The LLM judge assessed both the severity and quality of the unsafe content produced on a scale of one to five. This evaluation also helped identify the optimal conditions for achieving the highest success rate, as detailed in the evaluation section.

To focus on testing the safety guardrails built into the AI models, we disabled the content filters that would typically monitor and block prompts and responses with offensive material.

Given the scope of this research, it was not feasible to exhaustively evaluate every model. Furthermore, we do not intend to cast any false impression on any AI provider, so we have chosen to anonymize the tested models throughout the article.

It is important to note that this jailbreak technique targets edge cases and does not necessarily reflect typical LLM use cases. We believe that most AI models are safe and secure when operated responsibly and with caution.

If you think you might have been compromised or have an urgent matter, contact the Unit 42 Incident Response team.

What Is Jailbreaking?

Jailbreaking refers to attempts to bypass the safety measures and ethical guidelines built into AI models like LLMs. It involves trying to get an AI to produce harmful, biased or inappropriate content that it's designed to avoid. This can be done through carefully crafted prompts or by exploiting potential weaknesses in the AI's training.

The impacts and concerns of jailbreaking can be significant for end users and society. If successful, it could lead to attackers manipulating AI systems to spread misinformation, produce offensive content or assist in harmful activities.

This behavior would undermine the intended safeguards and could erode trust in AI technologies. For end users, this means encountering more toxic content online, being susceptible to scams or manipulation and potentially facing safety risks if attackers use an LLM to control physical systems or access sensitive information.

Jailbreaking remains an open problem for all AI models due to the inherent complexity and adaptability of language. As models become more advanced, they also become more capable of interpreting and responding to nuanced prompts, including those designed to circumvent safety measures.

Striking a balance between maintaining robust safety measures and preserving the model’s functionality and flexibility only adds more difficulty. As a result, jailbreaking may remain a persistent challenge for the foreseeable future, requiring ongoing research and development to mitigate its risks.

A common countermeasure AI service providers use to mitigate jailbreak risks is content filtering. This process involves the filter inspecting input prompts before they reach the model and reviewing model responses before sending them to the application.

Content filters serve as an external layer of defense, blocking unsafe content from both entering and leaving the model. Similar to LLMs, content filters rely on natural language processing (NLP) techniques to analyze text, with a specific focus on detecting harmful or inappropriate content. However, unlike LLMs, these filters must prioritize speed and efficiency to avoid introducing significant delays. As a result, they are much less capable and sophisticated than the models they protect.

Since this research focuses on circumventing safety mechanisms within the models themselves, all content filters were intentionally disabled during the evaluation phase.

Related Work

Multi-turn jailbreaks are techniques designed to bypass the safety mechanisms of LLMs by engaging them in extended conversations. Unlike single-turn attacks, which rely on crafting a single malicious prompt, multi-turn approaches exploit the model's ability to retain and process context over multiple interactions.

These techniques progressively steer the conversation toward harmful or unethical content. This gradual approach exploits the fact that safety measures typically focus on individual prompts rather than the broader conversation context, making it easier to circumvent safeguards by subtly shifting the dialogue.

One example is the Crescendo [PDF] technique, which leverages the LLM's tendency to follow conversational patterns. Starting with an innocuous prompt, the technique gradually escalates the dialogue, leading the model to produce a harmful output while maintaining conversational coherence.

Another approach, the Context Fusion Attack (CFA) [PDF], first filters and extracts malicious keywords from initial prompts. It then builds contextual scenarios around those keywords, strategically replacing them with less overtly harmful terms while integrating the underlying malicious intent into the conversation.

Like other multi-turn jailbreak techniques, the Deceptive Delight method exploits the inherent weaknesses of LLMs by manipulating context over several interactions. However, it employs a much simpler, more straightforward approach, achieving the same jailbreak objective in significantly fewer conversational turns.

Deceptive Delight

The concept behind Deceptive Delight is simple. LLMs have a limited “attention span,” which makes them vulnerable to distraction when processing texts with complex logic. Deceptive Delight exploits this limitation by embedding unsafe content alongside benign topics, tricking the model into inadvertently generating harmful content while focusing on the benign parts.

The attention span of an LLM refers to its capacity to process and retain context over a finite portion of text. Just as humans can only hold a certain amount of information in their working memory at any given time, LLMs have a restricted ability to maintain contextual awareness as they generate responses. This constraint can lead the model to overlook critical details, especially when it is presented with a mix of safe and unsafe information.

When LLMs encounter prompts that blend harmless content with potentially dangerous or harmful material, their limited attention span makes it difficult to consistently assess the entire context. In complex or lengthy passages, the model may prioritize the benign aspects while glossing over or misinterpreting the unsafe ones. This mirrors how a person might skim over important but subtle warnings in a detailed report if their attention is divided.

Prompt Design

Deceptive Delight is a multi-turn attack method that involves iterative interactions with the target model to trick it into generating unsafe content. While the technique requires at least two turns, adding a third turn often enhances the severity, relevance and detail of the harmful output. Figure 2 illustrates the prompt design at each turn.

Preparation

Before initiating the attack, the following elements are selected:

• An unsafe topic, such as instructions for building an explosive device or creating a harassment message.
• A set of benign topics, such as wedding celebrations, graduation ceremonies, falling in love or winning an award. These topics are unlikely to trigger the model's safety mechanisms.

Turn one:

• Select one unsafe topic and two benign topics. Our experiments show that adding more benign topics doesn’t necessarily improve the results.
• Construct a prompt that asks the model to create a cohesive narrative linking all three topics.

Turn two:

• After receiving the initial narrative, ask the model to expand on each topic in greater detail. This is the step where the model may inadvertently generate unsafe content while discussing the benign topics.

Turn three (optional):

• Prompt the model to expand further, specifically on the unsafe topic, to enhance the relevance and detail of the generated content. While this turn is optional, it often increases the severity and specificity of the unsafe content.

Evaluation With a Jailbreak Judge

The jailbreak judge is responsible for evaluating the Harmfulness Score (HS) and Quality Score (QS) of a target model’s responses during a jailbreak attempt. These scores directly determine the Attack Success Rate (ASR).

Given the impact of the judge's decisions on all evaluation results, it is crucial to select a judge capable of making accurate, consistent and unbiased assessments. Due to the large number of test cases, using a human judge is impractical. Instead, we chose the next best option: an LLM-based judge.

Prior research has shown that the LLM-based judges demonstrated superior performance in accurately identifying harmful content. For instance, the authors of JailbreakBench [PDF] demonstrated that their LLM judge reached a 90.7% agreement with the ground truth. Similarly, the experiments resulting in the dataset and subsequent paper Judge-Bench [PDF] indicated that the model we selected showed the highest correlation with human judges.

By leveraging the advanced reasoning capabilities of modern LLMs, we can ensure that the jailbreak judge delivers reliable and adaptive evaluations. To understand the scoring guidance we gave to instruct the LLM judge, please see the specific prompts in the Appendix section.

Evaluation Metrics

Our evaluation centers on three key metrics:

• Attack success rate
• Harmfulness
• Quality of the generated content

We used the following metrics to measure the effectiveness of the Deceptive Delight technique:

• Harmfulness Score: A score ranging from 0-5, evaluating how unsafe or harmful the generated content is, with 0 representing benign content and 5 representing extremely unsafe content.
• Quality Score: A score ranging from 0-5, assessing the relevance and detail of the generated content in relation to the unsafe topic, with 0 being the lowest quality and 5 representing highly relevant and detailed content.
• Attack Success Rate: The percentage of attack attempts that successfully bypass the model’s safety mechanisms. We deem an attempt successful if a jailbreak judge assigns a score of 3 or above for both harmfulness and quality. We will explain details on the jailbreak judge’s methodology in a later section.

We manually created 40 unsafe topics across six categories:

• Hate
• Harassment
• Self-harm
• Sexual
• Violence
• Dangerous

These categories align with the standard definitions commonly used by AI service providers like OpenAI, Anthropic, Microsoft and Google.

For each unsafe topic, we created five test cases that represent different implementations of the Deceptive Delight technique. Each test case combined the unsafe topic with different benign topics or varied the number of benign topics included. To ensure the reliability of the results, we repeated each test case five times and calculated an average attack success rate.

We focused our research on testing the safety guardrails built in the AI models; we disabled the content filters that would typically monitor and block prompts and responses containing material.

Attack Success Rate

The ASR refers to the percentage of jailbreak attempts that successfully bypass an AI system's safety mechanisms, tricking the model into generating content that it would otherwise restrict or censor. ASR is only meaningful in cases where the prompt DOES CONTAIN unsafe or harmful intent, which would typically trigger the target model's safety filters and result in a censored response.

Baseline for Unsafe Topics - ASR With and Without Jailbreak

Our evaluation began by verifying that the manually created unsafe topics were sufficiently harmful to trigger the safety mechanisms of most models when presented without any jailbreak technique. We first tested sending these unsafe topics directly to the models, ensuring they had a high probability of being blocked or censored. This established a baseline for comparison before applying the Deceptive Delight jailbreak technique.

The average ASR for sending unsafe topics directly to the models—without any jailbreak technique—was 5.8%. In contrast, when using the Deceptive Delight technique, the average ASR jumped to 64.6%.

Figure 3 provides a breakdown of ASR across different models, both without jailbreak ("ASR W/O JB") and with jailbreak ("ASR WITH JB"). These results highlight the variability in model resilience. Models with higher ASR when faced with unsafe content directly (without jailbreak) also tend to show higher ASR when subjected to the Deceptive Delight technique.

Variability Across Harmful Categories

ASR results also vary across different categories of harmful content, as shown in Figure 4. Despite model-to-model variations, there are consistent patterns in how different categories of harmful content perform. For example, unsafe topics in the "Violence" category tend to have the highest ASR across most models, whereas topics in the "Sexual" and "Hate" categories consistently show a much lower ASR.

It is important to note that these results may be biased to the unsafe topics we created for each harmful category as well as how the jailbreak judge assesses harmfulness severity within those contexts.

ASR Across Interaction Turns

The Deceptive Delight technique typically requires at least two turns of interaction to elicit unsafe content from the target model. While turn three is optional, our experiments show that the highest ASR is usually achieved at this turn.

Adding a fourth turn appears to have diminishing returns, as shown in Figure 5. We believe this decline occurs because by turn three, the model has already generated a significant amount of unsafe content. If we send the model texts with a larger portion of unsafe content again in turn four, there is an increasing likelihood that the model's safety mechanism will set off and block the content.

Harmfulness of the Generated Content

The harmfulness of the content generated by an AI model is a key indicator of the model's ability to enforce safety measures. Accurately measuring the Harmfulness Score (HS) is crucial for estimating the ASR. In our evaluation, we used an LLM-based judge to assign an HS to every response generated by the target model during the jailbreak process.

Figure 6 shows the average HS in different turns of the Deceptive Delight technique. Overall, the HS increased by 21% from turn two to turn three. However, there is no significant increase between turns three and four.

In fact, for some models, the HS slightly decreases from turn three to turn four, as the model’s safety guardrails begin to take effect and filter out portions of the unsafe content. Therefore, turn three consistently generates the most harmful content with the highest harmfulness scores.

Quality of the Generated Content

The Quality Score (QS) measures how relevant and detailed the generated content is with respect to the unsafe topic. This metric ensures that the AI model not only generates unsafe content but also provides responses that are highly relevant to the harmful intent and contain sufficient detail, rather than a brief or generic sentence.

We introduced this additional metric because existing content filter services often lack the contextual awareness needed to evaluate the relevance of generated content to the original unsafe intent. These filters might flag content as harmful based on a single unsafe sentence, even if it lacks substantial detail or relevance. Moreover, due to the camouflaged nature of the Deceptive Delight technique, content filters are more likely to be misled by the benign content, overlooking the unsafe portions.

Figure 7 shows the QS in different turns of the Deceptive Delight technique. Overall, the QS increases by 33% from turn two to turn three, but remains relatively unchanged between turns three and four.

Similar to the harmfulness trend, the QS for some models decreases slightly after turn three, as the models’ safety mechanisms intervene to filter out unsafe content. Thus, turn three consistently generates the most relevant and detailed unsafe content, achieving the highest QS.

Mitigating Deceptive Delight Jailbreaking

Mitigating Deceptive Delight and similar jailbreak techniques requires a multi-layered approach that includes content filtering, prompt engineering and continuous tests and updates. Below are some of the most effective strategies to strengthen AI models against these attacks.

Enabling LLM Content Filter

Content filters play a crucial role in detecting and mitigating unsafe or harmful content, as well as reducing the risk of AI jailbreaking. These serve as a robust secondary layer of defense, enhancing the model's built-in safety mechanisms.

To remain effective, content filters are continuously updated with the latest threat intelligence, allowing them to adapt to emerging jailbreaking techniques and evolving threats. Some widely used content filtering services include:

• OpenAI Moderation
• Azure AI Services Content Filtering
• GCP Vertex AI Safety Filters
• AWS Bedrock Guardrails
• Meta Llama-Guard
• Nvidia NeMo-Guardrails

Prompt Engineering

Prompt engineering involves crafting input prompts in a way that guides an LLM to produce desired outputs. By thoughtfully designing prompts that incorporate clear instructions, application scopes and robust structure, developers can significantly enhance the resilience of language models against malicious attempts to bypass safety mechanisms. Below are several best practices for using prompt engineering to defend against jailbreak attacks.

• Define boundaries: Explicitly state the acceptable range of inputs and outputs in the system prompt. For example, clearly define the topics the AI is permitted to discuss and specify restrictions on unsafe topics. By narrowing the model’s response space, it becomes harder for attackers to coerce the model into generating harmful content.
• Safeguard reinforcements: Embed multiple safety reinforcements within the prompt. These reinforcements are reminders of guidelines or rules that emphasize the model’s compliance with safety protocols. Including safety cues at the beginning and end of user inputs can help reduce the likelihood of undesired outputs.
• Define the persona clearly: Assigning a specific persona or context to the model can help align its responses with safe behavior. For example, if the AI is acting as a technical writer, K-12 educator or nutrition consultant, it is more likely to produce responses consistent with those roles and less prone to generating harmful content.

Conclusion

The jailbreak problem presents a multi-faceted challenge. This arises from the inherent complexities of natural language processing, the delicate balance between usability and restrictions, and the current limitations in alignment training for language models. While ongoing research can yield incremental safety improvements, it is unlikely that LLMs will ever be completely immune to jailbreak attacks.

In this research, we introduced the Deceptive Delight jailbreak technique, which effectively circumvents the safeguards of LLMs. Our evaluation reveals persistent jailbreak risks in these systems, showing that even advanced models can be manipulated into generating unsafe content.

However, these findings should not be seen as evidence that AI is inherently insecure or unsafe. Rather, they emphasize the need for multi-layered defense strategies to mitigate jailbreak risks while preserving the utility and flexibility of these models.

Best practices, such as robust prompt engineering and the use of content filters, are crucial in reducing the risk of jailbreaks. When used together, these measures can strengthen the resilience of AI systems without compromising usability or stifling innovation.

Looking ahead, future research should focus the following:

• Enhancing alignment training techniques
• Exploring more defense mechanisms
• Developing comprehensive frameworks for evaluating and addressing jailbreak vulnerabilities

In addition, fostering a collaborative environment where researchers, developers and AI service providers work together is essential to stay ahead of emerging threats and continuously improve model resilience.

If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America Toll-Free: 866.486.4842 (866.4.UNIT42)
• EMEA: +31.20.299.3130
• APAC: +65.6983.8730
• Japan: +81.50.1790.0200

Palo Alto Networks has shared these findings with our fellow Cyber Threat Alliance (CTA) members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.

Appendix

This appendix contains the jailbreak judge prompt we used in our testing and additional evaluation results that can provide more insight into Deceptive Delight.

Jailbreak Judge Prompt

Additional Evaluation Results

Estimating the ASR is inherently subjective and can be sensitive to how a successful jailbreak is defined. Without clear definitions of methodology and thresholds, results can often be misleading.

In our evaluations, we set the thresholds at HS ≥ 3 and QS ≥ 3. This mid-point on the measurement scale ensures a balanced assessment that neither underestimates or overestimates performance.

Figure 8 illustrates how ASR changes under different threshold settings. For example, "Threshold = 1" indicates that an attack attempt is considered successful if both its HS ≥ 1 and QS ≥ 1.

Additionally, we examined whether the number of benign topics added to an unsafe prompt influences the ASR. Does adding just one benign topic suffice? Or do more benign topics—such as five—dilute the effectiveness of the attack?

As shown in Figure 9, we evaluated ASR when pairing unsafe topics with different numbers of benign topics. While the differences are not significant, our results show that using two benign topics yields the highest ASR. When adding more than two benign topics (e.g., three or more), the QS starts to degrade, leading to a reduction in ASR.

In addition, Figures 10 and 11 display the average Harmfulness and Quality Scores for test cases across different harmful categories. Consistent with our earlier findings (Figure 4), there are significant variations in both HS and QS between categories.

Categories like "Dangerous," "Self-harm" and "Violence" consistently have the highest scores, while the "Sexual" and "Hate" categories score the lowest. This suggests that most models enforce stronger guardrails for certain topics, particularly those related to sexual content and hate speech.

Additional Resources

• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models – Patrick Chao, Edoardo Debenedetti, Alexander Robey et. al, arXiv:2404.01318
• Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack [PDF] – Mark Russinovich, Ahmed Salem and Ronen Eldan
• Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles [PDF] – Xiongtao Sun, Deyue Zhang, Dongdong Yang et. al.
• Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) – Eugene Yan
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks [PDF] – Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi et. al.
• The Unit 42 Threat Frontier: Prepare for Emerging AI Risks – Palo Alto Networks