AutoDAN-Turbo autonomously develops jailbreak strategies to bypass language model safeguards

The system can develop new jailbreak approaches on its own and save them in an organized way. This allows it to reuse and build upon successful strategies.

How AutoDAN-Turbo works

AutoDAN-Turbo creates a full prompt from a jailbreak strategy. It can also incorporate existing human-made jailbreak methods into its strategy library.

The system only needs access to the model's text output to work. Tests show it achieves high success rates in attacking both open-source and proprietary language models.

Outperforming other methods

AutoDAN-Turbo now leads other approaches on the Harmbench dataset for testing jailbreaks. It tends to work better with larger models like Llama-3-70B, but also performs well on smaller models.

The system not only succeeds more often in its attacks, but also produces more harmful outputs, as measured by the StrongREJECT score.

The researchers say that AutoDAN-Turbo's strong performance comes from its ability to explore jailbreak strategies independently, without human guidance. In contrast, other methods, such as Rainbow Teaming, rely on a limited set of human-generated strategies, resulting in a lower ASR.

In particular, AutoDAN-Turbo achieved an attack success rate of 88.5% on GPT-4-1106-Turbo. By adding seven human-designed jailbreak strategies from research papers, it achieved an even higher success rate of 93.4%.

The AutoDAN-Turbo code is available as a free download on GitHub, along with setup instructions.