Honest LLaMA: New method could make ChatGPT more truthful

The team uses linear probes to identify sections in the neural network that have high accuracy in factuality tests using parts of the TruthfulQA benchmark. Once the team has identified these sections in some of the transformer's attention heads, ITI shifts model activations along these attention heads during text generation.

ITI significantly increases Alpaca's truthfulness

The researchers show that with ITI, the accuracy of the open-source Alpaca model in the TruthfulQA benchmark increases from 32.5 to 65.1 percent, with similar jumps for Vicuna and LLaMA. However, too large a shift in model activations can also have negative consequences: The model denies answers and thus becomes less useful. This trade-off between factuality and helpfulness can be balanced by adjusting the intervention strength of ITI.

ITI has some overlap with reinforcement learning, where human feedback can also increase factuality. However, RLHF can also encourage misleading behavior as the model tries to match human expectations. ITI does not have this problem and is also minimally invasive, requiring little training data and computational power, the researchers say.

Studies of large language models could lead to a better understanding of "truth"

The team now wants to understand how the method can be generalized to other datasets in a real-world chat setting, and develop a deeper understanding of the trade-off between factuality and helpfulness. In addition, it may be possible in the future to learn the manually identified network segments in a self-supervised manner to make the method more scalable.

Finally, the researchers point out that the topic could also make a wider contribution: "From a scientific perspective, it would be interesting to better understand the multidimensional geometry of representations of complex attributes such as 'truth'."

The code and more information are available on GitHub.