OpenAI introduces a new plugin feature for ChatGPT that connects it to the web. This makes the model more accurate and up-to-date, and greatly expands its capabilities. Publishers can block the model from accessing their content.
The first plugins come from third-party providers like Expedia or Kayak for travel and route planning, Wolfram for complex mathematical tasks, or Zapier, which integrates numerous other apps like Google Sheets, Trello, Gmail, and more into ChatGPT.
OpenAI also integrates up-to-date product search with Klarna and Shop, and even a local store ordering service with Instacart. ChatGPT has access to up-to-date and accurate information via the plugins, and can thus (largely) eliminate two of the most common points of criticism.
"By integrating explicit access to external data—such as up-to-date information online, code-based calculations, or custom plugin-retrieved information—language models can strengthen their responses with evidence-based references," writes OpenAI.
We are adding support for plugins to ChatGPT — extensions which integrate it with third-party services or allow it to access up-to-date information. We’re starting small to study real-world use, impact, and safety and alignment challenges: https://t.co/A9epaBBBzx pic.twitter.com/KS5jcFoNhf
— OpenAI (@OpenAI) March 23, 2023
Similar to Meta's Toolformer, ChatGPT can automatically call APIs through plugins when needed. For example, if a user asks, "Where should I spend a few nights in Paris?" the AI model will call the hotel reservation plugin API, retrieve the API data, and generate a natural language response from it.
In its development documentation, OpenAI lists other use cases for ChatGPT plugins, such as retrieving the latest sports scores, stock quotes and news, retrieving information from a company's knowledge base, or retrieving personal notes.
OpenAI introduces two of its own plugins and wants to prioritize security
OpenAI itself provides a web browsing plugin and a plugin for running code in a sandbox. It also provides open source code for a retrieval plugin that allows you to pull information from different sources or systems.
"Plugins are very experimental still, but we think there's something great in this direction; it's been a heavily requested feature," writes OpenAI chief Sam Altman.
The code plugin for ChatGPT is suitable for solving quantitative and qualitative mathematical problems, analyzing and visualizing data, and converting files between formats, according to OpenAI.
OpenAI compares ChatGPT with its Code plugin to a "very eager junior programmer working at the speed of your fingertips" who can make new workflows easy and efficient and introduce more people to programming.
OpenAI welcomes feedback to save the online ecosystem
OpenAI's web browsing plugin is based on work done with WebGPT, which was first introduced in late 2021. It can retrieve information from the web in real time via Microsoft's Bing and "discuss" it, as OpenAI calls it.
At this point at the latest, OpenAI begins to compete openly and fiercely with conventional Internet searches, even revealing its search process with every click. The search results can then be used as a basis for further discussions with ChatGPT.
Publishers who do not want to provide ChatGPT with their content - after all, the AI is very likely to prevent a click on the corresponding website - can block the ChatGPT crawler. In this case, the ChatGPT user will see a "click failed" message in the chat interface. The startup emphasizes that the crawler only becomes active upon a chat request and does not automatically crawl pages for content.
OpenAI acknowledges that ChatGPT with web browsing plugin could affect the entire content ecosystem on the web, but has no solution for this issue for now - other than the crawler blocker.
"We appreciate that this is a new method of interacting with the web, and welcome feedback on additional ways to drive traffic back to sources and add to the overall health of the ecosystem," OpenAI writes.
OpenAI wants to prioritize safety and describes major job market implications of language models that use tools
OpenAI will gradually roll out the plugin feature to developers and users, with only a small group getting access first. ChatGPT Plus users will be given priority. After an alpha phase, plugins will also be available to API users. You can sign up for a waitlist here.
OpenAI cites new security risks posed by plugins as the reason for the slow rollout. These could lead to harmful or unintended actions, and would expand the options available to malicious actors. OpenAI says it has been working with external red-teaming partners to develop new security measures.
In a recent paper, OpenAI worked with research partners to explore the potential impact of large language models on the job market. In that paper, the company described the far-reaching implications of language model-based software, which it now mentions in its plugin announcement. Language models with access to tools "will likely have much greater economic impacts than those without", OpenAI writes.
