To make OpenAI's GPT-3 language model better, AI researchers on Twitter are using Python interpreters.
When OpenAI unveiled GPT-3 in May 2020, the large language model's performance sent waves through news and social media. No AI system before could write such compelling text, such beautiful poetry, and even fill out Excel spreadsheets.
In the weeks that followed, researchers discovered more uses for GPT-3, such as generating simple HTML code. OpenAI later refined this capability with Codex.
But with the hype came criticism: GPT-3 quickly failed Kevin Lacker's Turing test, among other examples.
OpenAI's GPT-3 generates a lot of nonsense
OpenAIs language model can't reason, can't perform more complex mathematical operations, and doesn't have access to databases on the Internet to check facts. As a result, the model generates a lot of nonsense - hallucinating is the technical jargon - and reproduces stereotypes. Even OpenAI co-founder Sam Altman warned against hyping GPT-3, noting that "it still has serious weaknesses and sometimes makes very silly mistakes".
Since then, OpenAI has been researching ways to address some of those limitations. The InstructGPT models released in January 2022, for example, are less toxic and follow instructions better. The WebGPT project is exploring how GPT models can interrogate the Internet.
Twitter community gives GPT-3 access to Python interpreter
On Twitter, AI researcher Sergey Karayev has now unveiled a different experiment: He has linked OpenAI's language model to a Python interpreter. For this, Karayev uses a text prompt for GPT-3, which teaches the model to formulate answers to questions as a Python statement or a small Python program.
The code is then run by a Python interpreter in the cloud and shown to the user.
Here's a brief glimpse of our INCREDIBLE near future.
GPT-3 armed with a Python interpreter can
- do exact math
- make API requests
- answer in unprecedented waysThanks to @goodside and @amasad for the idea and repl!
Play with it: https://t.co/uY2nqtdRjp pic.twitter.com/JnkiUyTQx1
- Sergey Karayev (@sergeykarayev) September 12, 2022
With this simple addition, GPT-3 can solve much more difficult mathematical tasks or make API requests. In one example, Karayevs gives GPT-3 limited access to the Internet and allows it to output current stock market prices.
Another Twitter user shows how GPT-3 can display a sphere with Matplotlib thanks to the Python interpreter.
"show me a sphere using matplotlib but no other external libraries" pic.twitter.com/biARbDcI1J
- kenneth cassel (@KennethCassel) September 12, 2022
GPT-3 gets a bit "neurosymbolic" with Python interpreter
Karayev calls this a "brief glimpse of our INCREDIBLE future". Soon, GPT-3 could be using this method to search the web by processing HTML or drawing on numerous freely available APIs.
Since the program can also generate code that, say, wipes a hard drive, he strongly advises against running the experiment on a local computer.
Despite expected limitations, cognitive researcher Gary Marcus calls the experiment on Twitter a "nice step towards neurosymbolic, with a neural net calling a symbolic system to fill in on something it can’t itself reliably do."
Karayev has published his code on the free collaborative browser IDE Replit. Those who want to test the project need a Replit account and an OpenAI API key. This still needs to be added to the main.py file.
