Researchers have released a preview of LongLLaMA, a large language model capable of handling long contexts up to 256.000 tokens or more. Built on the open-source OpenLLaMA and fine-tuned using the Focused Transformer (FoT) method, it permits some attention layers to access a memory cache of key-value pairs to extend their context length.
According to the researchers, the model retains performance on tasks that don't require long contexts, and can be used as a drop-in replacement for shorter context LLaMA implementations. The team has released their smaller 3B variant under the Apache 2.0 license, with inference code supporting longer contexts on Hugging Face. More information and examples of LongLLaMA can be found on their GitHub repository.
Over 150 executives from European companies, including Airbus and Siemens, have criticized the proposed EU artificial intelligence regulation in an open letter, arguing that the rules could harm competitiveness and technological sovereignty without adequately addressing challenges.
The concerns focus on the regulations' heavy emphasis on foundation models, which underpin chatbots, and the potential for disproportionate compliance costs and liability risks. The companies have called for a regulatory body of industry experts to monitor the implementation of the law instead of focusing on rigid compliance.
In our assessment, the draft legislation would jeopardise Europe’s competitiveness and technological sovereignty without effectively tackling the challenges we are and will be facing. [...] Europe cannot afford to stay on the sidelines.
Excerpt from the open letter sent to the EU Commission, the parliament and member states