A study by researchers from Korea University, Upstage AI, and AIGEN Sciences has identified specialized components in large language models that handle time-dependent information. These "temporal heads" play a crucial role in how AI systems process facts that change over time.
The researchers found these temporal heads exist across multiple language models, though their exact locations can vary between systems. Their responses also differ depending on the type of knowledge being processed and the specific year in question.
These specialized components don't just understand straightforward date references like "In 2004" - they can also process more complex temporal phrases like "The year when the Olympics were held in Athens." This suggests the models have developed a more nuanced understanding of time that goes beyond basic number processing.
Discovery could enable targeted LLM updates
When the researchers disabled these temporal heads, the models lost their ability to recall time-specific information while keeping their other capabilities intact. This selective disruption had no impact on how well the models handled time-independent knowledge or answered general questions.
The team also found they could modify temporal knowledge simply by adjusting the values of these specialized heads. This could reduce the cost of keeping AI systems up to date. Instead of retraining entire models - an expensive and time-consuming process - developers might be able to update time-sensitive information by targeting just these temporal heads.
However, the researchers acknowledge important limitations in their work. Smaller models like Phi-3-mini, with only 3.8 billion parameters, don't respond as well to targeted manipulation of temporal heads. These models likely require more complex mechanisms that have yet to be discovered and understood.
