Basing one’s whole argument on an irrelevant metric

Posted by Matt Birchler
— 2 min read

Ted Gioia: Ugly Numbers From Microsoft and ChatGPT Reveal That AI Demand Is Already Shrinking

What’s wrong? Everybody was supposed to prefer AI over conventional search. And it turns out that nobody cares.

Not everybody said that, but yes, I’ve agreed from the jump that search engines are better at finding facts that ChatGPT.

If AI really delivered the goods, visitors to ChatGPT should be doubling every few weeks.

Can we just take a second and recognize that a service that goes from zero to 100 million users in a few months shouldn’t be expected to double in users every few weeks anymore? I think it’s pretty clear that school being out dropped usage, as did the simple fact that thousands of apps and services (including their own app) have cropped up over the past 6 months that use ChatGPT’s API, but don’t contribute to the metric the author links to.

Hell, I never to to the ChatGPT site anymore, but I use it everyday in Raycast, I generate images pretty regularly with MidJourney, I am experimenting with Claude, and I’m keeping my eye on Llama models that I can one day run on my personal computer. If all we care about is ChatGPT’s website traffic, we’re not measuring the reach of text/image generators, we’re measuring one service that did indeed have a massive cultural moment that it couldn’t maintain 100% forever.

This is not how consumers respond to transformative technology. The current demand pattern resembles, instead, what we would call a fad or craze. And this is just one warning sign among many. Everywhere we look, the situation is the same.

We’ll see, but again, I think the author is over-indexing on a single service’s web traffic and not on the larger market. “Everywhere we look, the situation is the same”? Where? In businesses that are adopting GitHub Copilot all over? Sure, there are flash-in-the-pan apps that come and go, but let me take you back to year 2008 when fart and flashlight apps were topping the charts. They eventually crashed, but they didn’t invalidate the idea of apps.

Who is buying all those AI-written books? Who prefers AI-made songs to human music? Who wants to rely on AI journalism to keep up on the news? Who trusts AI in any mission critical job? I don't know anybody doing this.

Indeed, although I would say that this is setting unrealistic standards for what this is useful for in the first place. Yes, it can do those things, but those aren’t the use cases that are moving the needle.

Again, see my last paragraph about real life use cases, or think about this one someone posted to Hacker News, in which they are generating transcripts using MacWhisper (which uses OpenAI’s tech), fixing up those transcripts in Claude, and using that to create much better captions than you would get from YouTube. That’s improving accessibility, it’s contributing nothing to the ChatGPT web analytics Gioia is hanging their argument on, but it’s a good use case.

As I wrote 5 months ago in March:

ChatGPT is very fun, but I think the real useful versions of this will come from specialized rollouts of the technology in more constrained environments.

I still think this is true. ChatGPT got the world’s attention, but the big user-facing wins long term will be in more focused integrations into the software we use to do our work.