I really appreciate a sceptical(ish) piece that is neither Gary Marcus-ing about capabilities, nor extrapolating economic impact on the basis of only moderate improvements to Chatbots. I will be interested to see if your robotics piece changes any of your conclusions - my gut instinct is that more sectors of the economy will be impacted than 30%, if drop in workers scale from late 2020s and robots through the early 2030s
> The arguments for the Intelligence Explosion are premised on the idea that we will run the instances of the AI researcher on the cluster they were trained on.
No? This is mostly not a crux for those arguments. Those arguments use the training cluster size because it's a useful measure for how much compute would be available, but the specific numbers don't matter so much here. Whether you can run 1,000,000 automated researchers or 20,000 does matter, but it's often not the crux of the argument.
A couple times you quote the line “I think the Gemini program would probably be maybe five times faster with 10 times more compute or something like that”. I'm curious if there's a similar amount that might change your mind if it was "the Gemini program would probably be maybe X times faster with 10 times more intellectual labor at my level or something like that"?
>> Those arguments use the training cluster size because it's a useful measure for how much compute would be available, but the specific numbers don't matter so much here.
I think I'd dispute the usefulness of the measure -- if you want to train recursively, you need training cluster to be a training cluster (or at least, if you think about your total compute budget over time, if you want to do 1 month all inference on research and 1 month doing new post-training or something, this is effectively having a 50% training compute budget).
This R&D compute (experiments and training) has to be paid for by inference compute for customers or subsidized by investors, and at least right now, public markets keep asking where is the revenue.
>> I'm curious if there's a similar amount that might change your mind if it was "the Gemini program would probably be maybe X times faster with 10 times more intellectual labor at my level or something like that"?
My question would be: what are those additional researchers doing?
If they're actually doing experiments, you'd have to divide the experimental compute between researchers by 10; and I don't know what the 'returns to scale' of centralising experiment compute budget behind a few researchers are, but I'd imagine it's pretty high. On the other hand, with 10 times more researchers thinking of ideas, you could sample from each researchers' best ideas more selectively.
I think there's some benefit to having these researchers fixing bugs before running experiments, and I've heard at one lab, every time you want to run an experiment it takes 10 minutes to start. Then there's other things like kernel optimisations for the architectural change you're trying, which don't make sense for researchers to do every time, but cheap (and very smart) models which have the context could be able to do this.
Final thing I'll add is that the inference cost of increasing intellectual labour by 10 would be pretty high (though of course this comes down over time), and would eat into your experimental budget, so the more instances that you run, the more extreme the 'boundedness by compute' would seem to become too. So I do think there'd be some returns to adding more intellectual labour, though they'd have to tail off at some point (in the limit, all compute is spent on thinking of experiments and not doing experiments). Unclear what this looks like, though would be keen to see someone survey lab researchers on this!
Sorry to hear you thought this! We tried hard to avoid buying into the hype and to come to a grounded view. Would be great to hear more about where you think we're incorrect!
Rereading it. Alternates between bombastic and speculative (could be faster by 10 times! ) with sensible specification that say basically - nah… machine learning has much to offer in specific fields with known physical rules. AÍ more generally - unclear . And no I one talks about what is the research and development that will NOT be done to generate new discovery because all the money, energy and brains is being sucked into generating LLMs for suckers who will have to pay the monopolising companies serving it for outrageous fees .
Oh, we expressed skepticism that R&D output could be accelerated by a factor of 10! We were merely reporting what people at the labs were saying, and proposing the bottlenecks to this happening.
Agree impact of AI is pretty unclear, seems important however to take seriously what the people who are seeing internal deployments at the labs are saying about trajectory.
Re "the R&D not done because of funding for LLMs" ->
Interesting point; do you know to what extent there is substitution for AI vs just more money for funding R&D? Any sense for which things aren't being funded to fund AI research?
Also at present, open-source seems to be only just lagging frontier capabilities (c.f. Deepseek's release of r1 last week), so there is no margin here.
In the long run, I share your market structure concerns. I think what this ends up looking like depends on a) the returns to being first in unlocking the new research paradigm (i.e. people will train the next-gen chain of thought model on chains from the previous generation, so there's a compounding return; and b) when it happens, the decisiveness of the AI researcher -- if this can increase research output a lot, quickly; whoever gets there first could plausibly unlock much better capabilities much more quickly. Also feel uncertain about how this shakes down!
But good points made now. No I do not have detailed data on what is not being done to provide hundreds of millions for AI LLMs . But all money comes from somewhere, whether is is apple charging 30 pc profit for hosting apps, or Microsoft overcharging for software licences, or google ai misdirecting you to their clients. No free lunch. And so far - appear from deep mind’s efforts in the life sciences - nothing really useful to humans.
I will be persuaded when AI does really unpleasant tasks. Handling human waste, picking vegetables, handling animals, filling potholes (not just imaging software to tell you where rhymes are).
One interesting factor is the deflationary impact of AI and robotics, and its effect on GDP. A $10,000 robot doing what formerly required three workers at $70k per worker, makes output very cheap. Cheap things mean the economy declines in dollars. People’s standard of living will explode but actual GDP will decline. Need a new measure!
Real GDP is calculated with renormalised prices so if all prices in the economy dropped by 10x tomorrow and output of all goods grew by 25%, then real GDP would still record 25% growth.
Will it? Affording food, rent, and healthcare is more important than automating bureaucracy and automating production of goods you can no longer afford (and most of which are crappy gadgets you don't need anyway). All this "progress" has only made us society more isolated, more lonely, more depressed, more addicted, and less fertile and social
I really appreciate a sceptical(ish) piece that is neither Gary Marcus-ing about capabilities, nor extrapolating economic impact on the basis of only moderate improvements to Chatbots. I will be interested to see if your robotics piece changes any of your conclusions - my gut instinct is that more sectors of the economy will be impacted than 30%, if drop in workers scale from late 2020s and robots through the early 2030s
> The arguments for the Intelligence Explosion are premised on the idea that we will run the instances of the AI researcher on the cluster they were trained on.
No? This is mostly not a crux for those arguments. Those arguments use the training cluster size because it's a useful measure for how much compute would be available, but the specific numbers don't matter so much here. Whether you can run 1,000,000 automated researchers or 20,000 does matter, but it's often not the crux of the argument.
A couple times you quote the line “I think the Gemini program would probably be maybe five times faster with 10 times more compute or something like that”. I'm curious if there's a similar amount that might change your mind if it was "the Gemini program would probably be maybe X times faster with 10 times more intellectual labor at my level or something like that"?
Few thoughts:
>> Those arguments use the training cluster size because it's a useful measure for how much compute would be available, but the specific numbers don't matter so much here.
I think I'd dispute the usefulness of the measure -- if you want to train recursively, you need training cluster to be a training cluster (or at least, if you think about your total compute budget over time, if you want to do 1 month all inference on research and 1 month doing new post-training or something, this is effectively having a 50% training compute budget).
This R&D compute (experiments and training) has to be paid for by inference compute for customers or subsidized by investors, and at least right now, public markets keep asking where is the revenue.
>> I'm curious if there's a similar amount that might change your mind if it was "the Gemini program would probably be maybe X times faster with 10 times more intellectual labor at my level or something like that"?
My question would be: what are those additional researchers doing?
If they're actually doing experiments, you'd have to divide the experimental compute between researchers by 10; and I don't know what the 'returns to scale' of centralising experiment compute budget behind a few researchers are, but I'd imagine it's pretty high. On the other hand, with 10 times more researchers thinking of ideas, you could sample from each researchers' best ideas more selectively.
I think there's some benefit to having these researchers fixing bugs before running experiments, and I've heard at one lab, every time you want to run an experiment it takes 10 minutes to start. Then there's other things like kernel optimisations for the architectural change you're trying, which don't make sense for researchers to do every time, but cheap (and very smart) models which have the context could be able to do this.
Final thing I'll add is that the inference cost of increasing intellectual labour by 10 would be pretty high (though of course this comes down over time), and would eat into your experimental budget, so the more instances that you run, the more extreme the 'boundedness by compute' would seem to become too. So I do think there'd be some returns to adding more intellectual labour, though they'd have to tail off at some point (in the limit, all compute is spent on thinking of experiments and not doing experiments). Unclear what this looks like, though would be keen to see someone survey lab researchers on this!
I can't find the actual growth model, do you have a link to it anywhere?
Yawn. Hype.
Sorry to hear you thought this! We tried hard to avoid buying into the hype and to come to a grounded view. Would be great to hear more about where you think we're incorrect!
Thanks!
Rereading it. Alternates between bombastic and speculative (could be faster by 10 times! ) with sensible specification that say basically - nah… machine learning has much to offer in specific fields with known physical rules. AÍ more generally - unclear . And no I one talks about what is the research and development that will NOT be done to generate new discovery because all the money, energy and brains is being sucked into generating LLMs for suckers who will have to pay the monopolising companies serving it for outrageous fees .
Oh, we expressed skepticism that R&D output could be accelerated by a factor of 10! We were merely reporting what people at the labs were saying, and proposing the bottlenecks to this happening.
Agree impact of AI is pretty unclear, seems important however to take seriously what the people who are seeing internal deployments at the labs are saying about trajectory.
Re "the R&D not done because of funding for LLMs" ->
Interesting point; do you know to what extent there is substitution for AI vs just more money for funding R&D? Any sense for which things aren't being funded to fund AI research?
Microsoft's R&D budget has grown from ~$20b in 2020, to ~$29b in 2024; so it seems principally incremental here. On the contrary, UKRI have announced £7 million in Jan '25 for AI research -- https://www.ukri.org/news/over-7-million-awarded-to-help-ai-boost-growth-in-the-uk/; and £12 million to address 'rapid AI advances' in May 2024 -- https://www.ukri.org/news/12-million-for-uk-projects-to-address-rapid-ai-advances/. Aside from whether this is sufficient, the magnitudes are tiny!
Re monopolising companies serving it for outrageous fees ->
I am quite sceptical that margins will be very good on models. Apparently, OpenAI are losing money on o1-pro subscriptions -- https://x.com/sama/status/1876104315296968813.
Also at present, open-source seems to be only just lagging frontier capabilities (c.f. Deepseek's release of r1 last week), so there is no margin here.
In the long run, I share your market structure concerns. I think what this ends up looking like depends on a) the returns to being first in unlocking the new research paradigm (i.e. people will train the next-gen chain of thought model on chains from the previous generation, so there's a compounding return; and b) when it happens, the decisiveness of the AI researcher -- if this can increase research output a lot, quickly; whoever gets there first could plausibly unlock much better capabilities much more quickly. Also feel uncertain about how this shakes down!
Uncertainty large. Money poured in: enormous. Hype: off the scale.
But good points made now. No I do not have detailed data on what is not being done to provide hundreds of millions for AI LLMs . But all money comes from somewhere, whether is is apple charging 30 pc profit for hosting apps, or Microsoft overcharging for software licences, or google ai misdirecting you to their clients. No free lunch. And so far - appear from deep mind’s efforts in the life sciences - nothing really useful to humans.
I will be persuaded when AI does really unpleasant tasks. Handling human waste, picking vegetables, handling animals, filling potholes (not just imaging software to tell you where rhymes are).
> The view in San Francisco is that AI will far exceed the pace and depth of change in all previous technological revolutions.
The said the same shit, with the same confidence for every technology they hyped, hit a market peak, and was forgotten after a few years
Agree -- important to think for ourselves and decide when it might be right/wrong!
One interesting factor is the deflationary impact of AI and robotics, and its effect on GDP. A $10,000 robot doing what formerly required three workers at $70k per worker, makes output very cheap. Cheap things mean the economy declines in dollars. People’s standard of living will explode but actual GDP will decline. Need a new measure!
Real GDP is calculated with renormalised prices so if all prices in the economy dropped by 10x tomorrow and output of all goods grew by 25%, then real GDP would still record 25% growth.
>People’s standard of living will explode
Will it? Affording food, rent, and healthcare is more important than automating bureaucracy and automating production of goods you can no longer afford (and most of which are crappy gadgets you don't need anyway). All this "progress" has only made us society more isolated, more lonely, more depressed, more addicted, and less fertile and social