How Good Is AI, Really? Part 1: Critiquing an AI’s Critique

I’ll be honest with you: I don’t like generative AI.

I’ve done my research on the technology and written about a dozen pieces on it for various clients, and through it all I’ve tried to keep an open mind. But even ignoring the environmental concerns and the fact that it is, in a sense, a professional competitor, I just don’t think it offers much value.

But I’m willing to be proven wrong.

So, I’m starting a new series of blogs testing different forms of writing or editing AIs. I’ll try out a few AI tools, then analyze the results.

I picked some low-hanging fruit for this first installment: I host this blog on a WordPress platform, and WordPress has recently introduced an AI-powered editor. So, I ran the editor on a few of my recent blogs.

Here’s how it did.

Blog 1: What Content Editors Wish Writers Knew (and Vice Versa)

My most recent blog explored some communication breakdowns that can happen between writers and editors. This is what WordPress’s AI had to say about it:

“The content offers valuable insights from both the editor’s and the freelance writer’s perspectives, providing practical tips for better collaboration. To enhance the post, consider adding subheadings to break down the sections for better readability. Additionally, incorporating specific examples or anecdotes could further engage the audience by illustrating the communication challenges described. Lastly, including a concluding paragraph summarizing the key takeaways would provide a cohesive ending to the piece.”

I thought this was a bit of a mixed bag of advice. To take each suggestion individually:

1. I have subheadings. I’m a little confused why it suggests adding subheadings when I have eight throughout the piece. Does it want even more subheadings?
2. Adding specific examples and anecdotes is generally good advice, but I chose not to incorporate any in this blog because doing so would risk painting former clients or colleagues in a negative light.
3. I have a concluding paragraph that summarizes the piece’s overall main idea. I’m not sure what more the AI wants.

Article 2: Why I Won’t Use AI for My Writing

The next blog I tried evaluating was one on why I don’t use generative AI in my work (the irony entertains me).

The AI’s feedback was:

The post presents a clear perspective on ChatGPT and its implications for the writing industry. It effectively communicates the author’s decision not to integrate ChatGPT into their business. To improve the post:

1. Consider providing specific examples or statistics to support the points made.

2. Engage the reader by including personal anecdotes or experiences related to the topic.

3. Break up long paragraphs to enhance readability and engagement.

In my opinion:

1. Tip #1 good advice, and something I would definitely do if I were to write the piece today. At the time of writing, gen AI was only about six months old, so trustworthy statistics were a little sparse.
2. This is an interesting point because the whole blog is summarizing my personal experiences. That being said, I don’t get very specific in it, and it’s true that some more vivid anecdotes could have brought the piece to life.
3. My longest paragraphs in this blog are only five lines long on my desktop. That might be too long for an SEO article, but I think it’s fine for a personal blog targeting other people interested in writing.

Article 3: What I Learned from My First Two Years Freelancing

Finally, I ran the editor on my most popular blog, “What I Learned from My First Two Years Freelancing.” Here’s what the AI thought:

The content provides a personal and insightful perspective on the freelancing journey, highlighting the importance of communication, embracing individual work rhythms, dealing with client interactions, and the value of networking. Consider adding subheadings to break down the content for easier navigation. Emphasize the key takeaways to create a stronger conclusion. Ensure a clear call-to-action for readers to engage with your writing services.

1. This one actually doesn’t have subheadings, so points to the AI there. It’s true that I included more content between headings than I generally do, but in the context of this blog, additional subheadings would make it choppy.
2. I honestly don’t think emphasizing takeaways is the right move for this blog. While I do give direct advice at some points, I’m ultimately inviting readers to hear my perspective and decide if it’s helpful for them, not being prescriptive.
3. The final sentence of this blog is, “Contact me to learn how we can work together.” I’m not sure calls to action get any clearer than that.

The Takeaways

So, what did I learn from this experiment, other than the fact that AIs apparently don’t know what calls to action are?

The suggestions were pretty hit or miss. Some were genuinely helpful; others were so off the mark I’m wondering where they came from.  

I also noticed that the AI only seemed to offer variations on a couple different themes. It called out conclusions, subheadings, and the absence of specific examples and/or statistics.

These are all important elements, but the lack of variety is concerning. In the infinite world of writing, are those the only elements the AI is aware of? What about clarity, conciseness, even grammar?

Ultimately, this AI editor isn’t awful, but it definitely isn’t worth using unless you’re 1) pretty new to writing and 2) in a pinch and unable to ask anyone else for help. In general, it’s probably best to work with a human.