Harvard University student Maya Bodnick wanted to see if GPT-4 could pass Harvard's first-year home essays in the humanities and social sciences.
Bodnick had GPT-4 write seven essays on topics such as economic concepts, presidentialism in Latin America, and a literary analysis of a passage from Proust.
She then gave the essays to the professors to grade, indicating for bias purposes that the essay had been written either by her or by GPT-4. In fact, GPT-4 wrote all of the essays using the essay assignment as a prompt.
The result: A. A. A. A-. B. B-. Pass. GPT-4 earned a "respectable" 3.57 GPA, according to Bodnick.
The student said she submitted GPT-4's answers unedited, but with two caveats: She pieced together the essay from multiple answers to meet the word limit, since GPT-4 generates a maximum of 750 words at a time.
She also asked the reviewers to ignore the missing citations that GPT-4 could not provide. These would normally be part of the grade.
Great grades with special praise and little criticism
In addition to some excellent grades, reviewers gave special praise such as "beautifully written" or "well articulated paper." In one paper on conflict resolution, the reviewer criticized a flowery writing style with too many adjectives and metaphors.
In terms of content, the reviews were also positive, for example, praising the high level of detail or the structured approach to each question. The lowest grade of "B-" was given to the paper on Latin American presidentialism, where the reviewer criticized the paper for ignoring positive assessments of presidentialism as well as economic aspects.
"I think we can extrapolate from ChatGPT-4’s overall solid performance that AI-generated essays can probably get passing grades in liberal arts classes at most universities around the country," Bodnick writes. At Princeton or UC Berkeley, these grades might have been B's and C's instead of A's and B's. GPT-4 would still have passed. The seven essays can be viewed here.
A new learning and teaching paradigm
AI will entirely change the way the humanities and social sciences are taught, Bodnick believes. It has never been easier to cheat on a homework assignment, she said, and the technology is likely to get even better at things like citation.
Recently, major AI companies volunteered to develop a watermarking system. However, OpenAI withdrew its detector due to a lack of accuracy, and OpenAI CEO Sam Altman has consistently expressed doubt that an AI text recognition system will work in the medium to long term.
So testing will have to return, at least in part, to face-to-face interviews rather than essays, says Bodnick. She sees the fact that GPT-4 was able to write her essays as an indication of possible developments in her future profession.
That's why, she says, it's important for the education system to use her findings to think about how careers are changing and how to prepare for them - rather than trying to identify and punish AI essays.
"My gut reaction is that liberal arts majors — who spend most of their academic career writing essays — are going to face even greater difficulties in a post-AI world. AI isn’t just coming for the college essay; it’s coming for the cerebral class."