TikTok's AI Meme Maker filter shows the damage that unregulated AI can do: It particularly offends overweight people. The filter is currently disabled after sparking controversy.
In mid-August 2023, the social media platform TikTok introduced a new filter called "AI Meme Maker," which allows users to upload a selfie, for example. The AI then creates a so-called macro meme based on the image, i.e. it adds text over the image to give it additional meaning.
Normally, this can be fun, and memes can be a bit cheeky. However, it quickly became clear that the text generated by the AI meme generator was often very offensive, mocking users based on their appearance or weight. The effect became known as "AI Roast". One of the most famous examples comes from TikToker @mrmustachesmom, with more than 14 million views.
Video: @mrmustachesmom/TikTok
After the complaints piled up, TikTok temporarily turned off the filter, though the background remains unclear in the absence of an official statement from the platform.
In September 2023, the filter was reintroduced without much notice, now with the comment that the AI "does not always provide appropriate or relevant suggestions."
Currently, on October 24, 2023, the filter is no longer in TikTok's list of filters.
Most effects on TikTok are created by third parties known only by their TikTok name. This is the case with the "AI Meme Maker". Who exactly is behind the filter is currently unknown.
According to TikTok, however, all filters are reviewed by a safety team, which apparently was not bothered by the offending AI meme texts.
"But a lot of filters fall through the cracks because they’re only discriminatory or prejudiced toward a specific group of people," writes Tricia Crimmins in her column at Daily Dot. AI Meme Maker is one of them, she says, and it discriminates against fat people. That the filter is down again may have something to do with the backlash.
There is no way to be sure that all the memes purportedly created by the AI Meme Maker were actually written by an AI. After applying the effect, users can edit the text as they wish.
Several studies in the past have suggested that AI models can reinforce stereotypes.
ChatGPT can also be offensive
A condition for tools like AI Meme Generator is that the AI can analyze the image. ChatGPT can do this since GPT-4V. OpenAI has taken steps to ensure that the chatbot does not offend the people depicted, primarily by blocking descriptions of people.
However, these safeguards can be easily circumvented by simple prompt engineering - for example, by claiming that it is not a photo, but a painting with fictional people.
Some users have shown that GPT-4V is as easy to hack as adding barely visible text to images, giving an additional prompt that the user who uploaded the image doesn't know about.
