Content
summary Summary

Stanford's Alpaca was just the beginning: In recent weeks, several AI models have been unveiled using training data generated by ChatGPT.

In mid-March, Stanford researchers unveiled the Alpaca language model, a variant of Meta's LLaMA 7B that was fine-tuned with AI-generated data. The team trained the LLaMA model with 52,000 example statements generated by OpenAI's GPT-3.5 (text-davinci-003).

Alpaca achieved partially comparable results in the team's benchmarks, and was so cheap at $600 that alignment researcher Eliezer Yudkowsky, for example, sees it as a real challenge for companies like OpenAI.

Alpaca formula is open source, but may not be used commercially

However, the LLaMA model used for Alpaca is not released for commercial use, and the OpenAI GPT-3.5 terms of use prohibit using the model to develop AI models that compete with OpenAI. Stanford has therefore not yet released the model, only the training data and the code to generate the data and fine-tune the model.

Ad
Ad

Shortly after Alpaca, several open-source models were released that build directly on or are heavily inspired by this work. Alpaca-LoRA uses the resource-efficient low-rank adaptation (LoRA) method, also widely used in Stable Diffusion, with Meta's LLaMA to achieve results comparable to Alpaca.

Startup Nomic AI released GPT4All, a LLaMA variant trained with 430,000 GPT-3.5-turbo outputs selected from a dataset of one million outputs in total.

ChatDoctor, on the other hand, is a LLaMA model specialized for medical chats. The authors of the scientific paper trained LLaMA first with the 52,000 Alpaca training examples and then with 5,000 real conversations between medical professionals and patients.

The startup Databricks relied on EleutherAI's GPT-J-6B instead of LLaMA for its chatbot Dolly, which also used the Alpaca training dataset. "We find that even years-old open source models with much earlier architectures exhibit striking behaviors when fine tuned on a small corpus of instruction training data," Databricks said of training with the GPT-generated data.

ChatGPT clones and data gold

Stanford has provided a recipe for better open-source models with Alpaca, usually based on the powerful LLaMA models with special datasets generated by ChatGPT. Now that the larger LLaMA models have been leaked, we can expect much better models - but without a commercial license for now.

Recommendation

That ChatGPT's output is suitable as first-class training data is also shown by a report that Google employees wanted to use ChatGPT dialogs as training data for Bard. The process was stopped when an employee brought it to management's attention.

However, high-quality, human-generated data remains relevant for high-performance models-at least for companies like OpenAI that want to improve their own models. This is evidenced by the fact that OpenAI reportedly employs many human experts to verify or create specialized data, such as for code tasks. However, for the open-source community looking to create a free and efficient alternative to ChatGPT, the AI results may be enough for now.

Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Stanford's Alpaca has started a trend: High-quality data generated by ChatGPT is being used to train open-source AI models.
  • These include Alpaca-LoRaA, GPT4All, ChatDoctor, and Dolly, which is based on an EleutherAI model.
  • AI-generated data is useful for fine-tuning and has great potential for open-source models. However, commercial use is hampered by LLaMA's licenses and OpenAI's policies.
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.