Google researchers studied how creative professional writers think about AI as a writing tool.
For an experiment, the Google research team asked 13 professional writers from different creative writing fields to write their stories using the AI writing program Wordcraft. They wrote texts of 1000 to 1500 words with no content guidelines.
Wordcraft is a text editor with built-in AI writing assistance based on Google's dialog-optimized AI model LaMDA. Wordcraft is specifically designed for writing creative and fiction texts. The tool is not yet publicly available.
The same goes for LaMDA, which is considered one of the most powerful language models currently available. It received special attention in the summer of 2022, when former Google engineer Blake Lemoine claimed that the language system was sentient. He fell for LaMDA's dialog capability, which is based on what the conversation partner says and implies.
The following video shows a demo of Wordcraft to give you an idea. The demo is based on an older version of LaMDA.
AI is not yet a substitute for professional writers, say professional writers
The writing test ran for eight weeks. The team interviewed the writers in two 45-minute sessions, one at the beginning of the test to check expectations and one afterward to ask about the experience.
In addition, all participants kept an AI diary in which they recorded their experiences based on predetermined guiding questions. The researchers evaluated these logs in detail together with the interviews.
The research team's findings:
- None of the 13 professionals see themselves career-threatened by AI writing tools.
- They describe the technology's greatest potential as making creative writing easier, faster and more fun for professionals and amateur writers alike.
- The language tool is also useful for brainstorming ideas and details in stories.
- Developers of AI language tools should focus on the parts of writing that are the least fun and involve the target audience in the development.
Lack of consistency and boring suggestions
Study participants described LaMDA's lack of narrative consistency as an obstacle to text generation - a problem that affects all current major language models. In the currently known architectures, their attention is limited to short paragraphs, which makes them unsuitable for long, coherent texts with a clear dramaturgical sequence and recurring protagonists.
Participants also criticized boring suggestions without character and clear handwriting, as well as shortcomings in Wordcraft's user interface. The researchers see the boredom critique as a dilemma because large AI language models like LaMDA are deliberately limited in their generative abilities to avoid crossing ethical or moral boundaries.
"A system that always errs on the side of avoiding transgression is hamstringing itself from ever achieving human-level creativity, which is often grounded in a rejection of tropes and norms," the researchers write.
The AI copyright problem also affects writing assistants
Three study participants also expressed concern about the source of Wordcraft text suggestions being unknown. If Wordcraft were to take text excerpts unchanged from other novels, for example, it could damage the author's image or even have legal consequences.
One author described Wordcraft suggestions from the worlds of Doctor Who and Avatar. The AI software also tried to include well-known characters like the Wizard of Oz and Megatron (Transformers) in the plot.
One possible solution to this problem could be for Wordcraft to use the Internet to check whether a text might be copyrighted. However, the researchers concede that language models could mimic the work of other authors in a more subtle way than a direct copy. Similar copyright discussions exist for images and for code.
One of the major successes of our study was in revealing the heterogeneous set of needs and wants writers have for an AI assistant. Just as different writers need different roles to be performed by their beta readers and co-authors, we found that participant opinion about the proper role of AI varied widely.
From the paper
For a detailed evaluation, see the paper "Creative Writing with an AI-Powered Writing Assistant."