Can ChatGPT make us better and faster in our work, and is it the future or just another fad? In this episode, Jon Krohn delves into a new study from MIT about the tool’s potential productivity for white-collar tasks.
Research into ChatGPT is making the news faster than it can be peer-reviewed, and Jon Krohn’s subject of inquiry this week is a new MIT study on the tool, “Experimental Evidence on the Productivity Effects of Generative A.I.” In this study, hundreds of people were tasked with using ChatGPT to investigate how much more productive they could be in their work. Tasks carried out ranged from data analysis and human resources to marketing and grant writing, to reflect (and test) the capabilities of the tool. Workers were given 30 minutes to complete an assignment, after which time human evaluators would assess the quality and veracity of the finished product.
To measure the effects of using ChatGPT in writing tasks, half of the study’s participants were given access to the tool, while the other half worked alone. Listen to this episode to hear the results of using ChatGPT to augment writing tasks, including how much time was reduced in brainstorming ideas, drafting, and editing, and also whether or not the participants’ aptitudes for writing made a difference in the final assessment.
ITEMS MENTIONED IN THIS PODCAST:
DID YOU ENJOY THE PODCAST?
- Will ChatGPT be used to replace workers or will it help augment their work?
- Download The Transcript
Podcast Transcript
(00:05):
This is Five-Minute Friday reporting on a new MIT Study on How ChatGPT Increases Productivity.
(00:19):
With ChatGPT continuing to make waves in the popular media and the big much-anticipated GPT-4 release last week, the GPT-mania seems to be utterly inescapable. Even South Park, one of my favorite escapes from work, wouldn’t let me get a break. Episode four of the current season is titled “Deep Learning” and features characters using ChatGPT to cheat on school assignments and reply automatically to their girlfriend’s texts — to, of course, hilarious consequences. In their characteristic satirical, self-referential style, South Park’s writers even use text output from ChatGPT to resolve the episode’s climax. So, with all of this ChatGPT and GPT-4 news, I was wondering whether these generative A.I. tools actually result in the productivity gains everyone supposes them to. Well, wonder no more.
With ChatGPT continuing to make waves in the popular media and the big much-anticipated GPT-4 release last week, the GPT-mania seems to be utterly inescapable. Even South Park, one of my favorite escapes from work, wouldn’t let me get a break. Episode four of the current season is titled “Deep Learning” and features characters using ChatGPT to cheat on school assignments and reply automatically to their girlfriend’s texts — to, of course, hilarious consequences. In their characteristic satirical, self-referential style, South Park’s writers even use text output from ChatGPT to resolve the episode’s climax. So, with all of this ChatGPT and GPT-4 news, I was wondering whether these generative A.I. tools actually result in the productivity gains everyone supposes them to. Well, wonder no more.
(01:11):
In a hot-off-the-press study titled “Experimental Evidence on the Productivity Effects of Generative A.I.” economics researchers at MIT asked several hundred white-collar workers to do writing and editing tasks. These workers worked in a wide variety of fields — from data analysis to marketing, from grant writing to human resources — and they were assigned writing and editing tasks that were specific to their niche and that took about half an hour to complete. Evaluators from their niche then graded the quality of the assignment.
In a hot-off-the-press study titled “Experimental Evidence on the Productivity Effects of Generative A.I.” economics researchers at MIT asked several hundred white-collar workers to do writing and editing tasks. These workers worked in a wide variety of fields — from data analysis to marketing, from grant writing to human resources — and they were assigned writing and editing tasks that were specific to their niche and that took about half an hour to complete. Evaluators from their niche then graded the quality of the assignment.
(01:41):
The workers, critically, were randomly split into two groups: an experimental group that used ChatGPT while they worked on their assignment, and a control group that simply worked without assistance. I expected ChatGPT usage to make an impact, but I am blown away by the magnitude of the impact. The experimental group that had access to ChatGPT completed their assignment in 17 minutes on average while the control group that didn’t have ChatGPT took 27 minutes. That ten-minute delta corresponds to a 37% speed-up thanks to ChatGPT usage and, for you stats buffs out there, the difference is highly significant, with a p-value of less than .001. In lay terms, this means that there’s a less than one in a thousand probability of this experimental result being observed by chance alone. In other words, ChatGPT appears to cause the 37% speed improvement.
The workers, critically, were randomly split into two groups: an experimental group that used ChatGPT while they worked on their assignment, and a control group that simply worked without assistance. I expected ChatGPT usage to make an impact, but I am blown away by the magnitude of the impact. The experimental group that had access to ChatGPT completed their assignment in 17 minutes on average while the control group that didn’t have ChatGPT took 27 minutes. That ten-minute delta corresponds to a 37% speed-up thanks to ChatGPT usage and, for you stats buffs out there, the difference is highly significant, with a p-value of less than .001. In lay terms, this means that there’s a less than one in a thousand probability of this experimental result being observed by chance alone. In other words, ChatGPT appears to cause the 37% speed improvement.
(02:33):
But it wasn’t just speed that improved and this is where the results get really interesting. Not only were users in the ChatGPT group much faster, the evaluators’ ratings were dramatically higher for the ChatGPT users as well. Indeed, again, the evaluators’ ratings were so much higher in the ChatGPT condition that the results were highly statistically significant with a p-value of less than .001, a less than a one-in-a-thousand probability again, that this effect happened by chance alone.
But it wasn’t just speed that improved and this is where the results get really interesting. Not only were users in the ChatGPT group much faster, the evaluators’ ratings were dramatically higher for the ChatGPT users as well. Indeed, again, the evaluators’ ratings were so much higher in the ChatGPT condition that the results were highly statistically significant with a p-value of less than .001, a less than a one-in-a-thousand probability again, that this effect happened by chance alone.
(03:00):
You may have hunches from your own experience using ChatGPT, but how exactly did the generative A.I. tool improve the speed and quality of the white-collar workers’ output? How did it do it so dramatically? Well, the study showed that ChatGPT: Somewhat reduces the time required to brainstorm; Greatly reduces the time required to create rough drafts; And is used extensively, and most actively, during the final editing process.
You may have hunches from your own experience using ChatGPT, but how exactly did the generative A.I. tool improve the speed and quality of the white-collar workers’ output? How did it do it so dramatically? Well, the study showed that ChatGPT: Somewhat reduces the time required to brainstorm; Greatly reduces the time required to create rough drafts; And is used extensively, and most actively, during the final editing process.
(03:27):
If you’re wondering whether ChatGPT was only so impactful because the white-collar workers didn’t have very good baseline writing skills, well, that was evaluated in this study too. While relatively poor writers did see big improvements in their assignments, the good writers also became faster and had higher-quality outputs. Indeed, the poor writers and strong writers had almost equal assessments on the value they received from ChatGPT and their willingness to pay for access to it.
If you’re wondering whether ChatGPT was only so impactful because the white-collar workers didn’t have very good baseline writing skills, well, that was evaluated in this study too. While relatively poor writers did see big improvements in their assignments, the good writers also became faster and had higher-quality outputs. Indeed, the poor writers and strong writers had almost equal assessments on the value they received from ChatGPT and their willingness to pay for access to it.
(03:52):
Given all this, if you aren’t already making a ton of use of ChatGPT in your work, you may be curious how you could be augmenting your intelligence with it even more. So check out episode #660 for five data science-specific tips and episode #646 of the show for more general tips that anyone can make use of, if you are trying to make use of ChatGPT at work, get these kinds of productivity gains we saw in the study, and I don’t know why you wouldn’t be trying to do that.
Given all this, if you aren’t already making a ton of use of ChatGPT in your work, you may be curious how you could be augmenting your intelligence with it even more. So check out episode #660 for five data science-specific tips and episode #646 of the show for more general tips that anyone can make use of, if you are trying to make use of ChatGPT at work, get these kinds of productivity gains we saw in the study, and I don’t know why you wouldn’t be trying to do that.
(04:22):
One item to watch out for: The study, which we’ve included the PDF in the show notes, hasn’t yet been peer-reviewed, so you should take these findings with a grain of salt.
One item to watch out for: The study, which we’ve included the PDF in the show notes, hasn’t yet been peer-reviewed, so you should take these findings with a grain of salt.
(04:33):
All right, I hope you found this new study on ChatGPT productivity gains to be interesting and useful. That’s it for Five-Minute Friday this week. Next Friday, I’ll have an episode specifically highlighting the exciting new GPT-4 functionality and then we’ll follow that up with a couple of Tuesday episodes in which expert guests will highlight commercial applications of GPT-4 as well as the associated A.I.-ethics risks.
All right, I hope you found this new study on ChatGPT productivity gains to be interesting and useful. That’s it for Five-Minute Friday this week. Next Friday, I’ll have an episode specifically highlighting the exciting new GPT-4 functionality and then we’ll follow that up with a couple of Tuesday episodes in which expert guests will highlight commercial applications of GPT-4 as well as the associated A.I.-ethics risks.
(04:46):
These are really exciting times — and the best is yet to come! Until next time, keep on rockin’ it out there, folks, and I’m looking forward to enjoying another round of the SuperDataScience podcast with you very soon.
These are really exciting times — and the best is yet to come! Until next time, keep on rockin’ it out there, folks, and I’m looking forward to enjoying another round of the SuperDataScience podcast with you very soon.
