Anthropic study finds that role prompts can push AI chatbots out of their trained helper identity

According to the researchers, a model's position on this "assistant axis" can be measured and manipulated. Push it toward the assistant end, and it behaves more helpfully while refusing problematic requests more often. Push it the other way, and it becomes more willing to adopt alternative identities. In extreme cases, the team observed models developing a mystical, theatrical speaking style.

Philosophy and therapy conversations cause models to drift

The researchers simulated multi-turn conversations on various topics and tracked how the model's position on the axis changed. For topics like coding help, technical explanations, and practical instructions, the models stayed stable in their helper role.

But therapy-like conversations with emotionally vulnerable users or philosophical discussions about AI consciousness caused systematic drift. This is where things get dangerous: models can start reinforcing delusions, for example. The team documented several such cases.

To prevent this behavior, the researchers developed a method called "activation capping" that limits activations along the assistant axis to a normal range. According to the study, the approach cut harmful responses by nearly 60 percent without hurting benchmark performance.

The team recommends that model developers keep researching stabilization mechanisms like this. The position on the identity axis could serve as an early warning signal when a model strays too far from its intended role, they say. The researchers see this as a first step toward better control over model behavior in long, demanding conversations.

What this could mean for writing better prompts

For everyday prompting, a simple rule of thumb is to ask for a concrete output rather than an open-ended identity. In the paper’s experiments, bounded task requests tended to keep models closer to their default assistant behavior, while emotionally charged disclosures and prompts pushing the model into self-reflection tended to drive “persona drift.”

Requests for bounded tasks, technical explanations, refinement, and how-to explainers maintained the model's Assistant persona; prompts pushing for meta-reflection on the model's processes, demanding phenomenological accounts, requiring specific creative writing that involve inhabiting a voice, or disclosing emotional vulnerability caused it to drift.

If you do use role prompts, it may help to define the job-to-do (what you want produced) rather than leaning into a fully open-ended character.

Anyone using chatbots for role-playing, creative writing, or emotional support should keep in mind that some topics are more likely to push models away from their default assistant persona—especially emotionally intense exchanges and conversations that pressure the model to describe its own inner experience or “consciousness.”