Content
summary Summary

Giraffe is an open-source LLM with a context window of 32,000 tokens, making it useful for many applications in business contexts.

Ad

Large language models like GPT-4 show impressive capabilities but often have a limited context window, which limits their use in tasks where they would have to process dozens of pages. Variants such as GPT-4-32k or Anthropic's Claude with a context window of 100,000 tokens provide a much larger "memory" and are therefore more powerful in such use cases.

Now, researchers have extended the context window of the open-source LLaMA model by up to 10 times using interpolation techniques that reach about 32,000 tokens. The resulting LLM, called Giraffe, comes in a 13 billion parameter version and has one of the largest context windows of any open-source LLM.

Open Source Giraffe provides insight into scaling context windows

Being open-source, the research also provides some important insights into the inner workings of LLMs and different scaling techniques for enlarging the context window. According to the Abacus.AI team, Liner scaling of position embeddings was the most effective at increasing context length, with others also having some effect.

Ad
Ad

They also found that accuracy on long context tasks decreased with increasing length, demonstrating the limitations of current techniques, and showed that perplexity, commonly used to measure LLM performance, alone is insufficient to measure long context performance, highlighting the need for custom testing.

More information and data are available on the project GitHub, with the Giraffe-v2-13b-32k model being hosted on Hugging Face.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers extended the context window of the open-source LLaMA model, creating Giraffe, a large language model (LLM) with a context window of 32,000 tokens that's useful for various business applications.
  • Giraffe, which comes in a 13 billion parameter version, provides important insights into LLMs and different scaling techniques for enlarging the context window. Linear scaling of position embeddings was found to be the most effective.
  • The research also highlights limitations of current techniques and the need for custom testing since perplexity alone is insufficient to measure long context performance. Giraffe-v2-13b-32k is available on Hugging Face and the project GitHub.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.