Putting AI to the Test: Can It Write Like a Human?

If you work with written content, you are quite aware of the role artificial intelligence (AI) plays in the field. But if you think the conversation started with ChatGPT, I’m afraid I have to pause you there. Although the hype around AI has been especially present in 2023, the concept has been around as early as the 1980s.

For decades content creators have been using AI tools for generating and editing written pieces. I remember using grammar proofing in Microsoft Word and what a game changer Grammarly became when my professor at Carnegie Mellon suggested using it religiously for our papers. 

The content team of Drofa Comms is not new to AI tools. However, we decided to do something special for this Insider – conduct an experiment and let our colleagues from different departments analyse three incognito texts on the same topic: the role of AI in PR. Here is a twist: one text is written by a human, a PR professional to be precise, and two are generated by AI.

The Experiment and Criteria

So, now that you are hooked let me explain the purpose of this experiment. We are not trying to decide who is better for the job. We are simply trying to objectively analyse and give feedback to three texts written on the same request. Here is what we asked our humans and machines to write about:

Prepare a comment of up to 150 words on the topic of the rise of AI in PR. You should answer the following questions:

• How do you feel about the rise of AI in PR? 

• How is it being used increasingly for content and measurement? 

• Do you think this is a good or bad thing, and to what extent will people be replaced with machines, or more likely, creatives be replaced with techies?

And here are the criteria we evaluated them by on a scale from 1 (does not meet the criteria) to 5 (meets the criteria):

• Originality (how original and not plagiaristic the text seems to you; are there any new ideas in this text)

• Coherency (does the text have a structure and understandable logic)

• Relevance (how relevant the text is to the topic and assignment)

• Clarity of stance and opinion (how clear is the author's opinion on this topic)

• Use of evidence in arguments and its validity (is the opinion supported by arguments and relevant examples; is all information accurate)

• Final comment (what catches your attention; how do you rate the text as a whole; guessing who wrote this text: AI or human)

  
  

Disclaimer: later on, I will reveal the authors of all three pieces. Before continuing reading, I suggest you carefully study the texts. We placed them at the end of this article in a slide format. You can scroll down, read them, and return to this paragraph.

As for the respondents, we got 40% content editors, 40% PR specialists, and 20% from non-PR or content-related departments. And I have to tell you: there was no consensus on the experiment results among participants.

Revealing the Results

Are you ready to check your abilities? Here we go: the first text was written by a PR professional, Notion AI created the second text, and ChatGPT generated the third one.

It came as no surprise that content editors were correct in guessing the “human touch” – all of them identified a text written by a human. Half of the PR professionals were also correct in guessing whose text was written by a human. The rest identified the second text (Notion AI) as written by a human being. 

Let’s dive deep into the criteria. We calculated each text on a 100-point scale. The third place goes to the text generated by Notion AI: 62.4 points out of 100. The ChatGPT text comes second with 76 points out of 100. And the first place goes to a text written by a human being, with 80 out of 100 points. Although 60% of respondents had the highest evaluation of the first text written by a human, the rest of the participants gave personal preferences for different pieces.

AI and Human: Peculiarities in Writing

Most notably, ChatGPT exceeded the word count criterion, although it was explicitly mentioned in the requirements. Overall, participants noted that AI-generated texts have word-for-word repetitions from a given assignment and wordy sentences. On the other hand, machine-generated texts had a clear structure but lacked smooth, logical transitions from paragraph to paragraph. 

Content editors agreed that the first text written by a human had the highest originality score. They likewise stated that AI-generated texts provided a purely theoretical stand, looking like an explanation rather than an opinion. However, several respondents noted that a text created by a human lacked factual argumentation compared to AI versions.

Our AI experiment does not end here. In part II, we will share 5 tips we identified during our research on how to effectively use AI tools in written content creation to get the most out of working with advanced technologies.