summary Summary

To make OpenAI's GPT-3 language model better, AI researchers on Twitter are using Python interpreters.

When OpenAI unveiled GPT-3 in May 2020, the large language model's performance sent waves through news and social media. No AI system before could write such compelling text, such beautiful poetry, and even fill out Excel spreadsheets.

In the weeks that followed, researchers discovered more uses for GPT-3, such as generating simple HTML code. OpenAI later refined this capability with Codex.

But with the hype came criticism: GPT-3 quickly failed Kevin Lacker's Turing test, among other examples.


OpenAI's GPT-3 generates a lot of nonsense

OpenAIs language model can't reason, can't perform more complex mathematical operations, and doesn't have access to databases on the Internet to check facts. As a result, the model generates a lot of nonsense - hallucinating is the technical jargon - and reproduces stereotypes. Even OpenAI co-founder Sam Altman warned against hyping GPT-3, noting that "it still has serious weaknesses and sometimes makes very silly mistakes".

Since then, OpenAI has been researching ways to address some of those limitations. The InstructGPT models released in January 2022, for example, are less toxic and follow instructions better. The WebGPT project is exploring how GPT models can interrogate the Internet.

Twitter community gives GPT-3 access to Python interpreter

On Twitter, AI researcher Sergey Karayev has now unveiled a different experiment: He has linked OpenAI's language model to a Python interpreter. For this, Karayev uses a text prompt for GPT-3, which teaches the model to formulate answers to questions as a Python statement or a small Python program.

The code is then run by a Python interpreter in the cloud and shown to the user.


With this simple addition, GPT-3 can solve much more difficult mathematical tasks or make API requests. In one example, Karayevs gives GPT-3 limited access to the Internet and allows it to output current stock market prices.

Another Twitter user shows how GPT-3 can display a sphere with Matplotlib thanks to the Python interpreter.

GPT-3 gets a bit "neurosymbolic" with Python interpreter

Karayev calls this a "brief glimpse of our INCREDIBLE future". Soon, GPT-3 could be using this method to search the web by processing HTML or drawing on numerous freely available APIs.

Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.

Since the program can also generate code that, say, wipes a hard drive, he strongly advises against running the experiment on a local computer.

Despite expected limitations, cognitive researcher Gary Marcus calls the experiment on Twitter a "nice step towards neurosymbolic, with a neural net calling a symbolic system to fill in on something it can’t itself reliably do."

Karayev has published his code on the free collaborative browser IDE Replit. Those who want to test the project need a Replit account and an OpenAI API key. This still needs to be added to the file.

Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
  • OpenAI's GPT-3 language model can process text, but doesn't handle symbolic tasks well.
  • AI researcher Sergey Karayev shows on Twitter how GPT-3 can be linked to a Python interpreter.
  • This allows GPT-3 to formulate mathematical queries or ask for stock quotes as Python code, for example, and answer them that way.
  • Cognitive researcher Gary Marcus sees the experiment as a small step towards neurosymbolic systems.
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.