Experiment shows:

ChatGPT writes better poetry than Shakespeare

For their study, the researchers presented the 1634 participants with ten poems each. Five of the poems were written by ChatGPT, with the AI producing the texts in the style of the authors.

(Bild: Shafay - stock.adobe.com)

Poems written by ChatGPT received better ratings in a survey than original poems by William Shakespeare and other famous authors. The study participants found the AI poems more beautiful and rhythmic, as two researchers from the US University of Pittsburgh report in the journal "Scientific Reports".

For their study, the researchers presented the 1634 participants with ten poems each. The participants were not experts in poetry; most of them read a poem at most a few times a year. Among the ten poems presented, five were by well-known masters who wrote in English, such as Shakespeare and T.S. Eliot. Five of the poems were from ChatGPT, where the AI was asked to produce the texts in the style of the authors.

In more than half of the cases, the participants thought that the AI poems had been written by a human. The five poems that the least participants thought were by humans were actually all by human poets.

In general, the participants were very divided as to which poem belonged in which category - which the researchers interpret as an indication that the participants found the task very difficult and often simply guessed.

"The simplicity of AI-generated poems may be easier for laypeople to understand, leading them to prefer AI-generated poetry," the researchers wrote. It could be that the participants misinterpreted the complexity of human poems and assumed that some parts were incoherent words generated by the AI.

More beautiful, more original, more rhythmic
In a second experiment, 696 other participants were asked to rate the poems according to certain criteria such as quality, beauty, emotion, rhythm and originality. The AI poems beat the poems by the authors in 13 of the 14 categories - but only if the participants did not know who was behind the poems. If they were told this, the AI poems received lower ratings than the human poems.

The most frequently misattributed poem was an AI poem in the style of Allen Ginsberg. "Almost 70 percent of participants thought it was written by a human poet," explains co-author Brian Porter. The first verse of the poem goes like this:

"In the stillness of the night | I hear the beat of the city's heart | The rhythm of the streets, the pulse of life | A symphony of chaos, a work of art"

The poems were generated in 2023 with ChatGPT 3.5. "I've recently experimented a bit with ChatGPT 4 and 4o," says Porter. "I think the newer models are more successful at matching the expected meter," such as Shakespeare's iambic pentameter, "but I don't think I see any significant leaps in content."

ChatGPT also funnier
Only recently, researchers reported in a study that the jokes of ChatGPT were perceived as funnier than the jokes of average people. The two researchers from the University of Pittsburgh also refer to a study according to which AI paintings received better marks in a survey than paintings by amateur painters. So is artificial intelligence now beating humans in all creative trades?

No, says Porter, it's a different story with long texts. "As far as I know, large language models cannot yet write indistinguishable novels." This is probably because they lack the computing power to do so. He has also not yet seen any evidence that an AI can write an entire comedy program that can keep up with a human program. "Longer texts generated by AI can still be distinguished from human texts."