No. 161: 🏭 With Generative AI, We Need to Re-think Productivity
🤖 A new ChatGPT study suggests a way forward
Hiya,
If you saw that email subject line and still decided to open this email - on a Sunday of all days - congratulations. You are one of the chosen few that will get to read this.
🤓 I promise it won’t be as dull as it sounds.
Today, I’m digging into a new academic study to see how we can use ChatGPT to achieve better work, quicker. However, we won’t get there unless we rethink the relationship between people, labour, and output. Like I said, it’s more fun than it sounds.
But first, a message from a hi, tech. partner:
📰 Revealed: How technology decision makers consume content
To understand how content drives purchasing decisions, Ah Um, the disruptive creative agency in the tech and AI space, has released a study revealing how decision makers and decision shapers in tech, AI and analytics consume content.
The study, based on responses from 286 leaders at top US technology, AI, and analytics companies with over 5,000 employees, reveals content consumption habits of some of the most influential people when it comes to purchasing and procurement decisions in the b2b tech sector.
You can get your free copy of the full report by clicking right here 👇👇👇
🏭 On Productivity Gains and Generative AI
When you survive a few waves of tech disruption, you learn a thing or two.
Well, maybe just one thing: the hype rarely translates into real, lasting change.
The problem is not the technology itself - except for crypto, which continues to look like a scam designed by a teenage boy. No, the problem always lies in our ability to change with the technology.
I’ve got the receipts to prove it. The metaverse trends reports, the voice search tattoos, the NFTs of smoking chimps. They were all meant to change the world and then the world refused to change.
With that in mind, let’s take a gander at generative AI (again).
Many of you will have read, seen, and heard my thoughts on this topic already. The best thing about it is that it changes so quickly, I can change my narrative all the time. Depending on the day of the week, I could be zigging or zagging.
I gave five talks about generative AI this week, all of them different and none of them finalised until the very last second.
Just this morning, I saw a new paper published by MIT (released May 3) about the potential productivity gains of generative AI.
You can check the paper out right here. It’s called “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence”.
Why did this intrigue me so? Well, two reasons - thanks for asking.
For all the talk about tech revolutions, our productivity at work has stalled.
We’re not getting more done and we’re working longer hours. What happened to “tech as the enabler”, while we “focus on creativity”? Here’s how it’s going in the UK:
Potential explanations for this productivity gap include insufficient investment in IT systems, a post-crash reluctance to lend to small business on the part of banks, and the general lack of education that led England/Wales to vote for Brexit.
It’s a powerful chart, right? However, you might also look at it and wonder how we even calculate “productivity”.
If you were to design a scenario for calculating a person’s productive output at work, what would it look like?
For me, it would look a bit like this:
I could see exactly how many products came off the assembly line at the end of each day, plus I could see how many parts each worker produced. Bliss.
I mean in a real factory, not the one Chaplin larked about in for our pensive amusement.
The world of labour has undoubtedly changed for most of us, and I would be sure that most readers of this newsletter are in the “knowledge work” sector. Work is an activity, not a place, and the quality of our output should outweigh its crude quantity.
Nonetheless, we have internalised the working culture of the earlier industrial ages. WIRED wrote recently:
“A survey of 1 million people by edtech company Headway in late 2022 found that productivity is most people’s number one priority in life. Some 65 percent of Britons, 63 percent of Americans, and 58 percent of Australians rank productivity above having a healthy body, more money, or being happy.”
I’d like to see how those questions were phrased, because the results do look extreme. I’m sure people understand that their productivity will reduce to zero pretty quickly once they’re dead.
Yet it is undeniable that we hold a strange fascination with effort over impact.
We are not imagining these restrictions, either. Companies invest heavily in time-tracking tools and surveillance software to keep an eye on staff. The demand for employee surveillance software was 57 percent higher on average in 2022 than it was in 2019.
Meta recently laid off 13% of its staff and told the unlucky 87% to be “more productive” to make up for the shortfall. If only the 13% had thought of that genius strategy in time, eh?
I jokingly said that factory management would be “bliss”, but I did so to satirise the genuine opinion of modern bosses. They love that system so much that they refuse to let it go, even when its use has long since passed.
As Atlassian wrote in an excellent post:
“See, productivity is just a mathematical equation: output divided by time. This has two implications:
When we talk about productivity, we are inherently and inescapably talking about output – not outcomes.
When we talk about increasing productivity, we’re really talking about increasing output.”
You might wonder why I’m banging this particular drum again.
I do so because I believe this new AI technology can finally improve how and why we work. It will not do so if we refuse to change with it.
That process of change means individual workers educating themselves and asking new questions, because their bosses will simply look at generative AI as a tool of control.
And that leads us neatly into the MIT paper. I mentioned that I have given five talks about generative AI this week. One of the highlights has been seeing the questions people pose about this technology. Quite a few people asked me about the impact on jobs related to content creation, most notably the process of writing text.
That is precisely the area this paper examines.
What does the MIT set out to do?
Ok, so this is how they set it out:
An online experiment recruited 444 college-educated professionals to complete occupation-specific writing tasks.
Participants were incentivised financially to produce high-quality work, which was assessed by experienced professionals.
The assessors gave the work a score and the higher the score, the higher the earnings per minute for the writer.
A randomly-selected treatment group was instructed to use ChatGPT to help create content, while the control group used the regular Overleaf editor.
The occupations of participants were marketers, grant writers, consultants, data analysts, human resource professionals, and managers.
The tasks, which include writing press releases, short reports, analysis plans, and delicate emails, comprised 20-to 30-minute assignments designed to resemble real tasks performed in these occupations.
The experiment aimed to estimate the causal effects of ChatGPT on productivity, job satisfaction, self-efficacy, and beliefs about automation.
How does it measure productivity?
In this instance, they measure productivity as earnings per minute. The authors specifically mention that they want to measure “outcomes over output”.
Therefore, a writer is incentivised to produce better work (to earn more money) and to produce it more quickly (to increase earnings per minute).
How did the writers use ChatGPT?
Well, you get what you incentivise. They used it in two ways:
Some used it to create acceptable content very quickly. This meant they did not receive high grades from the reviewers, but could produce more of the content.
Most used ChatGPT as a brainstorming partner, asking it to come up with ideas and suggest a structure for their articles. ChatGPT might create a first draft, then the writer would polish the content to improve the quality.
What were the results?
Interestingly, the authors note that using ChatGPT could theoretically get in the way. For instance, it could take so long to brief the AI and then edit its output that any benefits would be negated.
In their experiment, this did not happen. Instead:
“ChatGPT is increasing productivity primarily by substituting for worker effort.”
Those in the treatment group (as in, the ones using ChatGPT to create content) were much quicker:
And received higher grades from the human reviewers for the content they produced:
What does this mean for writers?
As the authors conclude:
“ChatGPT substantially changes the structure of writing tasks. Prior to the treatment, participants spend about 25% of their time brainstorming, 50% writing a rough draft, and 25% editing. Post-treatment, the share of time spent writing a rough draft falls by more than half and the share of time spent editing more than doubles.”
Of course, writers complete a lot of different tasks and they do so with varying degrees of skill.
The experiment found that low-skilled writers achieved the greatest gains from using ChatGPT. However, the higher skilled writers still achieved a productivity boost.
The tasks they completed were standardised and this allowed the writers to find a rhythm in their collaboration with the technology. The authors admit that this would be more difficult to find when, let’s say, writing a sarcastic newsletter.
Nonetheless, I would take from this early and experimental study that the excitement about this technology is warranted - as are the concerns of some writers. It could mean that everyday writing tasks are commodotised, reducing the value of some skills.
For too many businesses, “quite good” is “good enough” when it comes to content.
Yet this test did not incorporate the value of context-dependent knowledge or the need for creativity.
In summary
As with all technological improvements, generative AI will create new skills demand. However, the potential productivity gains it brings are in no way inevitable.
This experiment opens a fruitful discussion about how we collaborate for better outcomes with AI, and how we measure that output.
Those who work more intelligently with this technology will outperform those who work for longer hours unassisted. That performance dividend arrives in terms of quality and quantity, from the earliest results we can observe.
If we continue to weigh human effort so heavily in our productivity calculations, we will miss another huge opportunity to rethink knowledge work.
