Experiment: Does AI write Twitter better than humans?
Everyone is talking and tweeting about AI. You’ve probably heard it all by now: AI can help you write better. AI can help you tweet faster. AI will come to your work (let’s just drop that theory – AI is a tool, not a replacement).
But can AI help your social signatures perform better?
As a busy social media marketer, I love anything that promises to simplify and speed up the content creation process. But we wanted to find out if the use of AI actually affects engagement and reach, especially on Twitter.
Do human-written signatures have more engagement and reach than AI-written signatures? Does the Twitter algorithm penalize tweets written by AI? I did an experiment to find out.
Hypothesis: Human-written Twitter captions will get more attention and reach than AI-written captions.
Our educated guess is that human-written Twitter captions will get more engagement and reach than those written by AI.
Twitter is a conversational platform. People like tweets with a clear voice, and Twitter users like to connect with other people. We suspect that people won’t be attracted to a tweet that doesn’t sound like it’s coming from a person.
But will people really know if a tweet was written by AI or not? And if so, how will it affect reach and engagement?
Let’s find out.
Methodology
To test whether human-written Twitter captions generate more engagement and reach than AI-written ones, I shared three sets of tweets on my personal Twitter account. Each set of tweets used the same format so they could be fairly compared.
For this experiment, I posted two text-only tweets, two link tweets, and two image tweets.
To compose AI-generated tweets, I used Lately, a tool that repurposes long content into bite-sized social captions.
(Bonus: you can connect the Lately app to Hootsuite to fine-tune, schedule, and auto-post your posts right on your Hootsuite dashboard)
It will then parse the content and create captions for viewing and editing. Once edited, you can submit them directly to Hootsuite for planning.
Some things to remember about Lately and any other AI content creation tools: First, you have to train it. For AI tools to generate content that is most like you or your brand, they need to understand your voice, content, and audience. That’s why Lately is asking you to submit the long part before it can generate captions.
Another thing to note about AI-generated content is that it still has to be edited by a human.
Once Lately generates tweets based on the content you’ve submitted, you can edit them for sound, clarity, and context. Recently uses these ideas to create better content next time. The more you use it, the more it will look like what your brand will actually publish.
After I created the human captions, I used the Hootsuite Lately integration to create similar tweets using the same long pieces of content. I then sketched them out in Hootsuite and used the suggested posting time feature to schedule them.
Results
I tested the performance of each tweet about 24 hours after it was posted (Twitter’s life cycle moves fast). Here are the results:
I collected the engagement rate for each tweet to understand how they work. Below I will consider specific indicators for each of them.
Tweet #1: Text-only tweet
Because I’m using the Lately AI tool for this experiment, I had to focus on rephrasing long pieces of content for these captions.
Here is the first tweet I wrote for this experiment. In this tweet, I paraphrased a quote from an article about a freelancer’s chore.
And here are the results of this tweet, according to Hootsuite Analytics. The engagement rate was 14.14% with a total of 28 interactions and 198 impressions.
In terms of style, this is a normal tweet for me. I love adding voice through asterisks and I love using parentheses in my emails. Bracketed statements (statements that clarify or clarify something and are usually enclosed in parentheses) are a great way to convey additional information and provide more context (see what I did there?). Also, parenthesized sentences just add an extra human element, if you ask me.
All this to say is that when the AI generated a similar tweet from the same article, I didn’t edit it to add my usual human touches because I wanted to see how it would work on its own.
Here are the results of the AI generated signature. The engagement rate was 4.06% with only 5 interactions and 123 impressions.
For this first comparison of tweets, a human-written caption fared better. Is it because of the missing brackets? Maybe, but probably not.
There are several factors that may have affected the effectiveness of this tweet, so let’s try again with a different type of tweet.
Tweet #2: Tweet with a link
For my next comparison, I wanted to see how a tweet with a link would perform. I paraphrased an article I wrote about the types of career paths for people working in content marketing (particularly freelance and in-house positions).
Here is a tweet I wrote:
With all the job insecurities that many have been dealing with lately, this article I wrote for @superpathco seems like a timely reminder.
Freelancing should not be an all-or-nothing decision! Many of the people I spoke to for this article do it on the side or between jobs. https://t.co/EI8rePLBuS
And here are the results of this tweet. The engagement rate was 8.47% with 32 interactions and 378 impressions.
This tweet included a link to an article and a mention of the brand I wrote it for. This tweet had more reach than my previous tweet and I suspect it’s because the brand I mentioned retweeted it. I checked Twitter’s built-in analytics for additional metrics, and this tweet received 21 detailed disclosures and five link clicks. Interesting!
For an AI-written caption, I recently paraphrased the same article and created a tweet for it. It also contained a link and a mention (which I had to add), but I didn’t change anything in the text it generated.
“It’s nice to know that freelancing exists just in case my career ever takes me down this path again!”It is not uncommon for FT content marketers to take on freelance work on the side or move between FT and freelance throughout their career. @superpathco
Here are the final results of this tweet. It generated an engagement rate of 7.5% with only 6 interactions and 80 impressions. I also checked Twitter analytics for this tweet and it received one detailed disclosure and two link clicks.
My human-written tweet outperformed the AI-generated tweet in this round. While the engagement rate was about the same, the tweet I wrote got four times as many impressions, probably because it was retweeted.
Would an AI-generated caption work better if it was also retweeted? May be. Would he get more link clicks if he had more reach? Maybe. But you could also argue that it wasn’t retweeted or clicked on as much because the headline itself wasn’t as engaging.
Let’s do another test.
Tweet #3: Visual Link Tweet
For my last tweet comparison, I wanted to include something visual to see how it affected engagement and reach. For comparison, I used two articles that I wrote for Hootsuite. (Again, since I’m using Lately for this experiment, the tweets should have been based on a long piece of content.)
They each have a similar thumbnail image and will appear the same in the feed. In addition, both articles cover a similar topic – social accessibility – so their content could be fairly compared.
Here is the first tweet I shared with the caption I wrote.
If you have worked in social media for the past few years, you know that digital accessibility and inclusion are top priorities.
In my latest post for @hootsuite, I explain why Facebook alt text is one way to make your content more accessible and discoverable. https://t.co/huQ5AksuCg
This tweet generated an engagement rate of 5.71% with a total of 8 interactions and 140 impressions. It got a couple of likes and one comment. According to Twitter’s own analytics, this tweet also received one link click and five details.
I recently created a tweet for another article I wrote on a similar topic about TikTok auto captions. I should note that the AI paraphrased the article for the first part of the tweet, but I had to add a second line for some context and include a mention, as it did in my first tweet.
Do you want more people to watch your TikTok videos? Make the viewing experience as accessible and enjoyable as possible by adding automatic subtitles.
Read more about automatic signatures in my first article for @hootsuite: https://t.co/JhSjH0TQ0g.
All in all, this tweet didn’t work. It had almost no reach or engagement—no likes, no comments, no retweets, no link clicks. But it received several detail extensions.
Once again, a human-written tweet outperformed AI-generated text.
What do the results mean?
Ultimately, all of my human-written tweets performed better than AI-written captions. I admittedly don’t have a lot of followers on my personal Twitter account, so none of these numbers are too exciting. But I think they demonstrate (on a very small scale) that more human tweets resonate better on the platform.
Here are some of my findings from this experiment:
Subtitles written by artificial intelligence can save time
To be honest, I may be a writer, but it takes me a long time to write a tweet, especially one that paraphrases a long article. I was amazed at how quickly the AI tool was able to come up with captions even if they weren’t ready to be published.
Whether you’re naturally good at writing or not, coming up with creative captions every week takes time and energy away from your other tasks, like strategizing or engaging with your followers. So when your job is to write dozens of social captions in any given day or week, using an AI tool can save you a lot of time.
A tool like Lately is especially useful if you use Twitter to share long content like blog posts. This tool can create dozens of captions from a single piece, which means you can create a month-long Twitter caption job in minutes.
AI-written lettering still needs human contact
Because Lately generated dozens of captions for every article I gave it, there were a lot of captions that didn’t make the list. Some of them were decent, but others were just confusing and required a lot of editing.
If you’re using a caption tool, you’ll still need to review and edit them, at least initially – especially for adding voice.
AI tools usually don’t understand the full context of a tweet and won’t immediately recognize your brand’s voice. Most AI tools need to be trained to better understand what you want them to do and to set the right tone for your brand. These tools may take time to catch on, but the more you use them and tweak what they write, the better they adapt.
How directly do AI-written captions affect reach? The jury hasn’t come out yet. In my experiment, all of my human-written captions had more reach than AI-written captions, but this could be because the content itself resonated more. It’s hard to tell if the platform knew that the tweets written by AI were actually written by AI. But if I had to guess, I’d say Twitter doesn’t penalize you for using AI to write your captions.
Ultimately, if you’re going to use AI to write your Twitter captions, think of them more as a starting point. You’ll have to use your social experience to make them attractive and publish-ready, but you’ll be able to take them down a lot faster than if you were starting from scratch.
