đ¨We have a new working paper full of experiments on how AI effects work, and the results suggest a big impact using just the technologies available todayđ¨
Over the past months, I have been working with a team of amazing social scientists on a set of large pre-registered experiments to test the effect of AI at Boston Consulting Group, the elite consulting firm.
The headline is that consultants using the GPT-4 AI finished 12.2% more tasks on average, completed tasks 25.1% more quickly, and produced 40% higher quality results than those without. And low performers had the biggest gains.
But we also found that people who used AI for tasks it wasnât good at were more likely to make mistakes, trusting AI when they shouldnât. It was often hard for people to know when AI was good or bad at a task because AI is weird, creating a âJagged Frontierâ of capabilities. But some consultants navigated the Frontier well, by acting as what we call âCyborgsâ or âCentaurs,â moving back and forth between AI and human work in ways that combined the strengths of both. I think this is the way work is heading, very quickly.
All of this was done by a great team, including the Harvard social scientists Fabrizio Dell'Acqua, Edward McFowland III, and Karim Lakhani; Hila Lifshitz- Assaf from Warwick Business School and Katherine Kellogg of MIT (plus myself). Saran Rajendran, Lisa Krayer, and François Candelon ran the experiment on the BCG side.
There is a lot more in the paper: https://lnkd.in/eZUp34CW
And in the summary: https://lnkd.in/eASQ_CVr