This is more than simple anachronism. Carrying these expectations set the historian up to fail in many ways. If they cannot find a clear expression of a doctrine, they might “discover” it across fragments of their work. Or they can “read in” meaning which the writer never meant to convey. If neither of these methods proves sufficient, some historians will even reprimand past thinkers for not including a doctrine on a theme. They will try to trace the history of an idea,

"as if the fully developed form of the doctrine was always in some sense immanent in history, even if various thinkers failed to 'hit upon' it, even if it 'dropped from sight' at various times, even if an entire era failed [...] to 'rise to a consciousness' of it."

This leads to what Skinner would call absurdities: pointing to earlier 'anticipations' — that's not what people thought they were doing! — and treating the history of political thought as a 'wholly semantic' exercise, where the historian evaluates whether an idea was "really there" by the yardstick of our current formation. A potential fix — to use the social context — can also establish unhelpful expectations, either to find causes determined by the context or to explain intentions in terms of the effects. History "becomes a pack of tricks we play on the dead".

This kind historian is “of the current moment” in the way they frame their question, but in their own self-image “outside time”, considering the perennial problems of political philosophy. The mental model this historian has is that Great Contributions are atoms attached to the fixed Debates, and each contribution is measurable against the others and the current moment. But this stilted conception isn’t explaining what the authors were actually trying to do on their own terms. Per Skinner, "there are only individual answers to individual questions, with as many different answers as there are questions, and as many different questions as there are questioners."

The solution to him is using the context as a framework for figuring out the meaning and intention of the authors' interventions from the text. What conversation did they view themselves to be participating in? With the language they used, what meaning did they believe it carried? What intention did they have, in writing?

To demand from the history of thought a solution to our own immediate problems is thus to commit not merely a methodological fallacy, but something like a moral error. But to learn from the past and we cannot otherwise learn it at all - the distinction between what is necessary and what is the product merely of our own contingent arrangements, is to learn the key to self-awareness itself.

Why did I explain this paper? I think it’s a useful frame for the question I want to ask: what time, like the historians, are the minds of language models from? There’s a loose analogy, I think, between how in the history of political thought we treat texts as comparable atoms in a vacuum and how a neural network treats the relationships between concepts stored in its weights.

Nothing about pre-training encodes a sense of time. All tokens are treated equally by the model, unless you change the learning rate. The tokens aren’t processing information sequentially, as humans have over recorded history. But most tokens come from the last few years, as the amount of content on the Internet has grown exponentially, so the present is weighted much more heavily.

Language models are predicting the next token based on the context which means that models can learn dual meanings of words, but whether they can notice subtle linguistic drift in the meanings of words over time is less sure. When the model uses “rights” today, can the model know that we could be referencing something similar to what Locke or Paine referred to, and also conceptions which neither of them had access to? When features light up inside the model, does the model understand the chronology so “rights” for Locke does not mean the Geneva Convention? As multilingual models have become more capable, they have stopped representing concepts in each language — it’s more efficient to represent the same meanings in a smaller number of features. This is useful for building capable agents, but it might mean the models are under-parameterised for keeping hold of all the subtle differences in meaning that matter for these problems, and so we just keep the coarse, present meaning.

RLHF selects for the present even more aggressively. For chatbot products, the model specification wants them to be ‘helpful, harmless, and honest’ (all good things!). But it does alter the persona of the model towards whatever we understand those things to mean right now. The authors of the classic texts of political philosophy would have made different suggestions to the model behaviour teams in Californian AI labs about what it means to ‘do no harm’. The chatbot personas are also selected for what users want them to be: (mildly) sychophantic and long-winded. These tools are not able to precisely alter parts of the model’s persona. The deeper drives and motivations of the model are shifted by these interventions in imperceptible ways — all towards our 2025 ideas of what they ought to be.

At this point, you might push back, “What does it matter if the LLM minds are so contemporary? Their relationships to time and to texts feels like exactly the kind of academia of our time that I am excited to automate."

I think this would be wrong. It matters a lot.

People will say things like, “you must read Heidegger in German or Tolstoy in Russian” because the linguistic structures in those languages mean the authors had a different set of affordances for expressing different thoughts. Translations don’t quite get it. The same is true of structures influencing thoughts at a higher level too. One that has bothered me recently is oftentimes when people oppose technological progress or economic growth, they will often also believe that we’ve all been corrupted by institutions or modernity. They take a very rosy view of what life was like beforehand, unspoilt and simpler. This pervades into things like, thinking that its positive to shrink the UK’s water allowance by 20% for 2037. You can’t avoid them.

What we want from the models is a very flexible, self-aware kind of intelligence that can step into and out of these frames.

I was reminded of a time visiting a Viking museum on a family holiday, which had preserved wooden longboats from 900AD. I found it completely insane that people would get into these boats, only half a step up from a canoe, and sail to raid or settle another country. What must they have believe about their relationship to the sea, the place they lived and were going, their purpose, the other people, the weather, and so on? Without any more structured, I asked the AI systems to provide me with an account of why someone would sail across the sea, in terms that would have made sense to the Vikings. (Not literally Old Norse, but the closest approximation of the ideas.) The responses were inflicted with Romantic ideas about the sublime nature world that a Viking wouldn’t have used.

Noticing this issue was relatively easy, but I could imagine that these frames are invisible if you are exploring something particularly unfamiliar. Models which can only think in terms of the present, or which are adversarially pulling to return to the present, are an unhelpfully rigid kind of intelligence.

Last month, the Trump Administration introduced an executive order on “Preventing Woke AI in the Federal Government”, to make AI that is “serving America, not ideological interests”. The details of the requirements are fairly uncontroversial despite the politically charged title: model developers have to provide the government with transparency into how models are ideologically steered through the model spec, system prompt, or evaluations. There’s an important discussion for all states to care about the default responses of the models, but a “post-ideological” model does not seem to be desirable, or possible. (America is not ideologically neutral, and it would be worse off for being so!). I don’t think that preventing “Woke AI” means that the “woke” vectors should be ablated, unlearned, or RLHF’ed by the model. The best kind of models should be able to step into the “woke” frame, give us the best of whatever it has to offer, and then readily step into another.

Lots of people have been reaching to articulate what it means to instantiate “liberal democratic” AI systems and have stalled at having liberal democratic owners and controllers. But there is a partial answer here: models which support their users to step into and out of different (ideological) frames with high fidelity is much more liberal than the rigid, doctrinal enforcement of the (ironically, quite Liberal) status quo. For these models to make paradigm-shifting progress in the humanities, they need to have more awareness of their own state and choose their own frames. I don’t think they are nearly as good as humans doing this. I worry this kind of thing might be undersupplied by the market — the model developers also have to make these models good therapists too — and while we all benefit from progress in the humanities, it’s more difficult to capture this value.

At least, if anyone does want to try this, they will have the corpus of the Cambridge School to help.

In the 1960s, two decades after Mondrian’s death, an early computer artist called Hiroshi Kawano developed a statistical prediction of which colours Mondrian would choose, and how long he would make the lines. It was based on his body of work. He wrote the programme using the rudimentary programming language of the day and calculated the results on the University of Tokyo’s mainframe computer. The computer couldn’t output colour images and so Kawano would take the statistical results and hand-paint the coloured rectangles. Kawano did not use the exact same primary colour palette as Mondrian to “[express] his admiration for Piet Mondrian…without claiming any close visual resemblance”. Below is KD 29, one of the prints in his Artificial Mondrian series.

Is this art?

I think it is. Kawano has different motivation to Mondrian for this work. Kawano began as a philosopher, who learned programming in order to experiment with using machines to make art. He was one of the first artists to explore “human-computer interaction”. By contrast, Mondrian wanted to express universal harmony with the most simple elements possible. The process was different too. Kawano would write a program in FORTRAN, send it for batch processing and hand paint the results, while Mondrian would sketch the proportions of elements in an empty studio, making small studies before committing to the final canvas.

In a legal sense, Kawano has not violated copyright. Ideas cannot be copyrighted, only expressions of ideas. The idea of using blocks of primary colours with straight perpendicular lines is not copyrightable, and even then, it seems that Mondrian and Kawano were expressing different ideas. It also helps that Kawano did not stick to the same colour palette. The Artificial Mondrian is identifiably in the style of Mondrian but this is not sufficient to constitute a copyright violation. Styles cannot be copyrighted.

Should Kawano have asked Mondrian for permission, if he were still alive, before making his work?

This depends on what, in your view, Mondrian owns. It is difficult to define the precise scope of Mondrian’s protected expression. Perhaps it is the balance or proportions between the colours and the lines. But as Kawano proves, there is a lot of space for creativity within this. Perhaps it is the combination of the primary colours and the proportions which belong to him.

There is danger in having an overly expansive definition of what Mondrian owns. Specified too broadly, and too many building blocks would be enclosed from the commons. Branches of the tree could become unexplorable for artists like Kawano and posterity.

In this case, I don’t think that Kawano needs Mondrian’s permission for this work. It doesn’t seem reasonable that Mondrian should have been able to prevent the painting above from happening. And in general, it seems a little silly to mandate consent for making art. “Oi mate, you got a loicense for that painting?”, does not seem like the kind of laissez-faire spirit in which the best works of culture happen. In particular because new forms of art often begin as peripheral, avant-garde or illegitimate (photography, impressionism etc); the incumbents can be resistant, but that doesn’t make them correct.

This would probably be considered fair use in the US, not copyright infringement. The US system evaluates based on four principles:

• The purpose and character of the use — are you going to make money or do research or something else? How transformative is the use?

• The nature of the copyrighted work.

• The amount and substantiality of the work used — how much of the work have you used, how central is that to the essence of the work?

• The effect of the derivative on the potential market for the original.

The last characteristic is the most important, typically. Kawano’s use of Mondrian’s work is limited: he used the statistical relationship between elements to predict future ones, but did not copy exact relationships. He did not copy colours. And the purpose was different: it was a new kind of artwork, though it was still art and still commercial. My expectation would be that it is deemed fair use because Kawano’s work did not harm the market for original Mondrian paintings. If anything, it could have enhanced it by increasing interest in the original work.

In the EU, Kawano’s derivative work would be allowed under the text and data mining exemption unless Mondrian had decided to opt-out of his work being used.

Does Kawano owe Mondrian money, if he were still alive, if he sells the print or the program?

I do not think Kawano owes money for the print. I think this follows from whether he needs permission and whether the algorithm constitutes fair use.

However, selling the program by itself is less transformative and could potentially interfere more with the market for Mondrian’s work. Perhaps it “uplifts” many people to make Mondrian-like paintings rather than to buy prints from the artist, causing them lost revenue. While Kawano had chosen not to use the same colour palette as Mondrian’s, who’s to say that others wouldn’t do the same? A potentially informative precedent is Warhol v. Goldsmith (2023). Andy Warhol had used a picture taken by Goldsmith for the basis of a silkscreen illustration of Prince. While this changed the image’s appearance quite dramatically, it still competed in the same market as Goldsmith’s original work — magazine licensing — and so it was deemed not to be fair use.

But at the same time, this kind of uplift is positive too: it democratises access to creating Mondrian-style work and might lead to greater creativity on-net. An interesting precedent here is Oracle vs Google (2021). Oracle alleged that Google had infringed their copyright by using parts of the code for the Java API (read: connection to Oracle) and that this cost them software license revenue. The Court upheld Google’s fair use of this code in the Android platform, on the grounds that it made it easier for developers to create new applications for the Android ecosystem. The social benefits for consumers outweighed the lost license revenue for Oracle.

A related, and important question, is whether Kawano is responsible for copyright infringement from people he sold the program to. He has uplifted them, but it was ultimately within their scope to make work that did or didn’t infringe on copyright. As a parallel, in 1998, the US created a legal safe harbour for internet platforms whose users infringed on copyright. The platforms were not responsible so long as when they received a notice to take the copyrighted material down, they did so. For this reason, it does not seem to me that Kawano was participating in copyright infringement — he was just making a tool.

So I could be persuaded either way: it can be argued that Kawano should pay Mondrian for selling access to the program if it caused him to lose revenue and this did not outweigh the wider social benefits to other creatives.

As it is, the EU’s rules allow Kawano to sell access to the tool wihout paying Mondrian, provided they take reasonable steps to prevent downstream infringement. The EU AI Act’s Code of Practice allows text and data mining to create commercial AI models but says Signatories will…

1. make reasonable efforts to mitigate the risk that a model memorizes copyrighted training content to the extent that it repeatedly produces copyright-infringing outputs and

2. prohibit copyright-infringing uses of a model in their acceptable use policy, terms and conditions, or other equivalent documents [for closed-source models].

If Kawano selling the program, without paying Mondrian, is theft, when has the theft happened?

Some people will reasonably disagree with me, and say that Kawano selling prints and copies of the program is making use of Mondrian’s protected expression. As it is, the Tate Shop offers prints for £5. But if one does disagree with the Tate and me, it is useful to consider when the infringement occurred. Was it…

• Sometime during the statistical analysis?

• Sometime during the writing of the program?

• While the computer processed the results?

• While Kawano hand-painted the primary colours onto the image?

• At the point of sale of the prints?

I am much more persuaded by answers which come later. Merely doing the statistical analysis, or making the program feels like a much less compelling argument for copyright infringement, than the moment of commercialisation. In this, the purpose of the use is changing and the market is being affected, and so the fair use becomes less compelling.

Copyright-as-culture-war

The discourse on how we might apply copyright law to AI systems has, unfortunately, been collapsed into a culture war framing. In the popular media, it is framed as “the bohemians against the tech broligarchs”. See, for example, this editorial: “The Guardian view on AI and copyright law: big tech must pay.” Or Elton John’s interview with Laura Kuenssberg:

“Thievery on the highest scale...you’re going to rob young people of their legacy and their income, it’s a criminal offence, I think. I think the government are just being absolute losers.

…

I don’t know who the tech minister is, what’s his name? … Yeah, well he’s a bit of a moron.”

I do not claim that the Kawano-Mondrian is a perfect analogy to AI, nor a water-tight piece of jurisprudence, but it should provide an intuition for the kind of questions we need to answer, at a remove from present-day politics.

There are a number of dangers to reducing this issue to friends or enemies, young creatives or big tech billionaires.

The first is that the copyright debate is used to litigate other issues, like how some Silicon Valley elites are close to the Trump Administration, that streaming and social media has changed the structure of media and entertainment markets, or that some incumbents in the creative industries and big tech have very large market power. Interviewed alongside Elton John, the playwright James Graham said, “So many are leaving the industry because it is an incredibly tough time. This advancement into the digital space and the online space is not benefiting the artists and hasn’t traditionally.” This is not an invalid thing to care about, and nor are the other reasons above, but it cannot be adjudicated through the copyright debate.

The second danger of simplification is that in aiming to attack your ‘enemy’, it ends up backfiring. A letter from industry representatives to the Government says:

“We will lose an immense growth opportunity if we give our work away at the behest of a handful of powerful overseas tech companies and with it…any hope that the technology of daily life will embody the values and laws of the United Kingdom.”

But one of the reasons “the technology of daily life” struggles to “embody the values and laws of the United Kingdom” is that it doesn’t get made here. The UK’s interpretation of copyright laws wouldn’t apply to companies doing AI training elsewhere and it might difficult to enforce rules on AI deployment by foreign companies. J.D. Vance was very clear:

[T]he Trump Administration is troubled by reports that some foreign governments are considering tightening the screws on U.S. tech companies with international footprints. Now, America cannot and will not accept that, and we think it’s a terrible mistake not just for the United States of America but for your own countries.

If it is only possible to enforce rules on domestic companies, then having a stricter regime would differentially affect domestic companies. This could either push companies to move jurisdictions, not move to the UK, or be less competitive. Not having domestic tech companies makes it harder, in fact, makes it more difficult to steer those technologies towards your values in future and to tax them, to pay for the things you value.

The third issue with simplification is that it does not balance objectives. The goal is to have a more flourishing creative future. This involves having finer tools, to say the thing we mean, exactly. It means having lower barriers to actualise our creations, it means more leisure and tutoring to develop mastery. It means having a richer common context to draw from.

There are two threats to this scenario. The first is — as advocates point out — if the property rights of creatives are not suitably enforced, they will not internalise the market returns for their work and so will not pursue the arts or invest in creative innovation. The second is that we do not create the tools or necessary context to create this progress. Free and rich societies have advantages to producing creative work. If we fail on the first count, we end up in a wealthy but an unexpressive, greyer future. In the second, we end up culturally stagnating with our current set of tools or unable to uphold the freedoms for individual expression. The task is to balance these modes of failure.

I fear that in pursuit of particular policy objectives — whether there is an opt-out or opt-in regime for AI training, or the degree of transparency requirements — we trade a great amount of steering power for the course of technology in the future. It is exactly because I think the UK would steer better in the long run, relative to others, that it is so worthwhile to ensure AI is developed here.

A Practical Path Forward

The following is my attempt to find the synthesis between values in this particular case — transparency and fairness — and realpolitik which allows the UK to pursue its values in the long-run.

Critical to this is my expectation that competitive pressures lead foundation model developers to train their systems in whichever jurisdictions offer the most permissive copyright regime. There is, like taxation, a “race to the bottom”, where middle powers like the UK cannot set global standards. OpenAI’s input to the Office for Science and Technology put this in much more bombastic terms:

Given concerted state support for critical industries and infrastructure projects, there’s little doubt that the PRC’s AI developers will enjoy unfettered access to data—including copyrighted data—that will improve their models. If the PRC’s developers have unfettered access to data and American companies are left without fair use access, the race for AI is effectively over. America loses, as does the success of democratic AI. Ultimately, access to more data from the widest possible range of sources will ensure more access to more powerful innovations that deliver even more knowledge.

It is clear they state their interest as strongly as possible. Since then, the Trump Administration fired the Head of the US Copyright Office who had published an advisory report which suggested a more stringent interpretation of fair use. For this reason, I expect the UK’s rules on AI training will be unenforceable on companies from the EU, US, and China, and it will only be possible to impose rules on domestic AI companies. The difference in rules will either push developers away from training in the UK, prevent developers moving to the UK, or could mean that companies never get started which otherwise would have. To make the abstract concrete, Google DeepMind have just released a very good video and audio model, Veo 3. This would have been trained on copyrighted materials outside the UK, but will be part of the Google offering in the UK. Meanwhile, Synthesia is one of the world’s leading AI video companies based in London. What should they do? Compete against Google on an unfair playing field, or leave the UK?

This is similar to the non-dom tax regime: while I have beliefs about what constitutes a fair society, the world as it is means that rich people can leave and the UK will have less money if they do. I prefer to trade more tax receipts for an abstract notion of fairness. I certainly don’t think a non-dom loophole was fair but it just seems better than engaging with a fictitious version of the world “as I wish it was”. The moral high ground doesn’t pay for public services.

With this in mind, there are three major considerations for the UK’s rules:

• Whether to have an opt-in or opt-out for AI training on copyrighted materials.

• What the transparency requirements for AI training data should be.

• What should be required of model developers to mitigate copyright infringement.

AI training

The training process is roughly approximate to “making a copy and reading it”, if deployment is “writing”. I have been slightly confused by the focus on AI training in the copyright debate. How the model is deployed seems to have a great deal more impact on rightsholders.

The data is gathered from the Internet using a technique called “data scraping” using tools called “web crawlers”. Here’s an explanation of training that I prepared earlier:

The neural network is like a little computer which can be programmed by adjusting a series of dials. The aim of a neural network is to predict an output given a set of inputs. The iterative process of tuning these dials to improve the prediction is called ‘training’. The people creating the network supervise the training process by showing the data and the answers, but crucially, it doesn’t involve telling the network how it ought to process and understand the image. In other words, our process of trial and improvement tweaking of dials is essentially letting the little computer, by itself, search for the best way it can be programmed to achieve its goal, unlike ordinary computers which need a human to figure out a program first and then somehow communicate it to the computer. Dario Amodei described the training process in this way:

“You [the AI researcher] get the obstacles out of their way. You give them good data, you give them enough space to operate in, you don't do something stupid like condition them badly numerically [i.e. tweak the dials poorly], and they want to learn. They'll do it.”

One common misconception is that models “ingest” data. Again, the connotations are negatively misleading. This gives the implication that an alien mind is swallowing it or something. More actually, the model is “passing over” the words, akin to skim reading, and using them to feedback on its predictions. During this process, the parameters are learning compressions, just like humans have heuristics. The Internet is hundreds of zettabytes — 1 zettabytes is 1 trillion gigabytes — whereas Llama 3.1 is ~500 GB and can be run on a laptop. It’s incorrect to say that all the data is “in” there.

Another misconception is that the model developers want the model to memorise things. This is not the goal. Memorisation is an inefficient use of space inside the model, and memorising protected expressions isn’t what intelligence is. The graph below shows the “memorisation rate” in Google DeepMind’s series of Gemma models.

The memorisation rate in Gemma 3 is more than 1000 times lower than Gemma 1, and Gemma 2 is between 10 and 100 times lower. Notably, this is not a function of model size, but algorithmic gains because the number of parameters is roughly consistent across generations.

This is not to suggest that AI development cannot be geared towards memorisation and repeating clones of someone else’s work; it definitely can. People can train an “AustenBot” or a “DickensBot”, or distill one from a larger model. But the goal of foundation model training is to find compressions (heuristics) which generalise to solve problems. It has been proven that foundation models have complex circuits and are not just stochastic parrots. The rules have to distinguish between those who are creating a world model — in large language models, this is a general representation of “language space” — and those who are training models towards memorising a particular artist’s work.

The UK’s copyright rules do not make it possible to create world models for commercial purposes, just research. So while Google DeepMind is headquartered in London, all their training will surely happen in the US. The EU, by contrast, allows research organisations (universities, cultural heritage institutions) to train on copyright data and the rightsholders cannot opt-out. The groups can use their models for commercial purposes or research. Otherwise, commercial organisations can use web-crawlers but must provide an opt-out for rightsholders who do not want their work to be used for training.

The EU AI Act Code of Practice said that model providers cannot use tricks to get behind paywalls or use web crawlers on sites that distribute pirated books or films. Meta is being sued in the US for training their Llama models on LibGen, an online library that provides access to copyrighted material. This would not be permitted under the EU AI Act, but might constitute fair use in the US, depending on the aforementioned factors.

The case for opt-out

The UK should follow the EU in allowing for an opt-out regime for training, rather than an opt-in regime for rightsholders as some have advocated.

Large models trained on more tokens of data are more capable, so if AI developers can only train on datasets they have the permission of rightsholders it either slows their training or makes their models less capable. And while in aggregate, the tokens are essential for model performance, each given token just isn’t worth that much. A piece from Model Thinking (forthcoming, tomorrow) estimates that Llama 4 training was roughly $800 million and training used 30 trillion tokens of text composed of 120 trillion tokens of raw text. If the training cost was taxed to compensate rightsholders (note: a terrible idea to tax things you want), then each token was $0.000007 per token. A 10,000 word essay is worth just 9 cents even when charging $800 million for the data. Put differently, based on this online calculator, a model 10 times bigger than Llama 3 would cost roughly $11.25 trillion if Meta paid for tokens at the freelancer rate. This is nearly 10 times the market capitalisation of Meta. The marginal price of a token is going to zero.

Second, most of the rightsholders are so fragmented that it would be uneconomic for an AI company to try to aggregate all of these. Training an AI model in the UK would be a bit like trying to get 8,276 consents required to build HS2. (You’d cancel the sections or just pick up and go elsewhere!) If the rightsholders believe their tokens are especially valuable, the opt-out means they can remove their permission and negotiate with the tech companies for use. The opt-out functions as a de minimis exception for the tokens which are not valuable until they are aggregated.

Third, the opt-in system preferences incumbents with larger market power. Most online platforms require in their terms of service to use content posted on the platform to train their models. Large studios will have aggregated the rights of independent creatives doing work-for-hire, and so would be able to engage in “collective bargaining” but independents would be too small to do so. With the intention of “making big tech pay” the system would in fact set up defaults for online platforms that already had the rights to large datasets.

Therefore, the UK should match the EU’s rules: it is fair use if you don’t go around paywalls and make best efforts to avoid websites of pirated books (and so on). Doing opt-in doesn’t “get” anything for creatives, it just stunts the emergence of internationally-competitive AI firms in the UK.

Transparency requirements

The same consideration applies: do these rules only apply to UK companies and not their international competitors? In principle, transparency is a worthy ideal, but what is the practical cost? What do we have to trade for training transparency?

The Baroness Kidron amendment would require companies to provide a log of all of the URLs their models were trained on, and keep this up-to-date every month. By contrast, the US makes no training data transparency requirements on their model creators and the Second Draft of the EU AI Act Code of Practice required some limited disclosures about the data collection practices:

• A list of the different data acquisition methods, including, but not limited to: (i) web crawling; (ii) private data licenced by or on behalf of rights holders, or otherwise acquired from third parties; (iii) data annotation or creation potentially through relationships with third parties; (iv) synthetically generated data; (v) user data; (vi) publicly available data; and (vii) data collected through other means

• The time period during which the data was collected for each acquisition method, including a notice if the data acquisition is ongoing

• A general description of the data processing involved in transforming the acquired data into the training data for the model

• A general description of the data used for training, testing and validation.

• A list of user-agent strings for web crawler(s) used, if any, in acquiring training data

• The period of data collection and name of organisation(s) operating the crawler for each web crawler used

• A general description of how the crawler respects preferences indicated in robots.txt for each web crawler used

• A description of any methods implemented in data acquisition or processing, if any, to address the prevalence of copyrighted materials in the training, testing, and validation data.

However, the Third Draft did not include the equivalent model card, perhaps indicating that the EU AI Office had to walk back requirements to get the US labs to agree to the Code. (The Code is an option for implementation of the Act, which US labs can decide to use or argue alternative interpretations in the courts.) This provides a range of autonomy that UK legislators have to operate within.

The transparency requirements are important for implementing an opt-out. How does one verify that companies have respected the opt-out, unless there is a list of URLs to verify? However, I think the list of URLs is slightly overstated as a silver bullet for enforcement. One might reasonably respond, how does one verify that the list of URLs matches the actual training data?

The only way to enforce the opt-out is through engagement with the model. Over time, we can develop interpretability tools and data attribution tools (through research agendas like influence functions) and we can use simple elicitation methods like prompting the model. There can be steep fines for models which provably trained on material that had opted-out, but if it is not possible to identify that it has been trained on it, nor that our best probes can identify it inside the model, there is no practical answer to enforcement. Imposing transparency requirements differentially on UK startups to go further than this seems disproportionate.

The alternative approach would be to not allow an opt-out for rightsholders whose work is in the public domain. If training is akin to reading, and all work depends on the influence of others, then prima facie, a neural network should be allowed to read the whole internet, listen to all music, or watch all films as inspiration, just as a human can. It is the deployment which risks infringing copyright, not the training. The opt-out, with the EU’s approach of requiring high-level disclosures about practises stands out as giving creative industries autonomy if they do not accept this argument and thereby provides a balanced path forwards.

Mitigating infringement in deployment

Until the model is released into the world, any copyright infringement has been inert: the model hasn’t done anything. The biggest risks to creators and their livelihood arise not from fractions of pennies in lost income from training, but from markets being flooded with near identical AI-generated copies.

In deployment, the UK has slightly more autonomy than when regulating training. Foreign companies serving AI models in the UK are bound by deployment rules, which doesn’t depend on training done abroad. But this is not complete: the Trump administration can tell the UK to back down on enforcement or model providers can switch off their service in the UK. AI models are going to be essential to many economic functions — imagine all white-collar workers are using multiple agents for their work — so whoever provides the models will have a lot of power.

The regulation of deployment is also most sensitive to the two failure modes discussed. If the copyright regime is too laissez-faire, model developers who are intent on creating AI-generated replicas could cause creatives lost revenue, but if it is too aggressive, the AI systems will be neutered as tools of creative innovation. There is a natural inclination towards the first consideration as today’s creatives will naturally make the case for the protection of their mode of output but the creatives of tomorrow cannot make the case for the latter scenario. But imagine, giving an AI system a harmless prompt and it responds with an error message:

All primary-coloured blocks and perpendicular lines are owned by the estate of Mondrian, do you have a license for that?

Alternatively, give this prompt when you’re in France and the US and we can fulfil the request.

That is a bleak creative future for those other than the Mondrian estate.

The Third Draft of the EU AI Act’s Code of Practice requires model developers prevent their models being used to infringe copyright, as mentioned earlier. The UK should follow their standard here.

In practice, these rules will be implemented by algorithms which determine whether models can respond to a prompt or how they should respond to a prompt. Online platforms run proactive systems to prevent copyrighted material being shared as the scale of potential infringement is too great for humans to track on the largest services. In some cases, the law might be over-enforced on legitimate work, for example, Spotify’s copyright classification system prevented a group of academics from publishing a podcast about copyright.

The foundation model developers can steer the responses of models away from infringing on copyright using techniques like RLHF and tools like constitutional classifiers. The largest model providers, with more than 500 million users, could use citizens’ assemblies (supported by experts) to review transcripts of prompts and responses, so that ordinary people can provide input into how the systems can balance being a useful tool for expression and infringing on protected expression. These labels could be used to train a reward model for RLHF, train the constitutional classifiers, or develop the model spec. Model developers could even do this of their own volition!

Conclusion

The goal, on which I think everyone would agree, is to have innovative creative sectors where the actual creativity receives fair compensation and that the UK has the technological autonomy to make its own rules. Having internationally uncompetitive opt-in and reporting requirements would do more to set back this cause, than advance it. The blunt truth is that companies developing models in the EU, US, and China will not follow the UK’s opt-in system and the opt-in system isn’t even a good idea on its own terms. Unilaterally burdening would-be UK model developers does not help UK creatives in practice. In fact, they might be more damaged by the reduce likelihood that in the long-term, global technology companies are here.

Many people in the UK would like to exert more influence over social media platforms, search engines, and eCommerce providers. This is difficult when they are not made here, their leaders and headquarters are not based here, and they do not pay taxes on their profits here. If we are to have “any hope that the technology of daily life will embody the values and laws of the United Kingdom”, we must do our level best to make AI here. Imagine hosting the Industrial Revolution on foreign traintacks, that can be turned off at any moment and whose owners can steer your society’s values and extract its wealth.

It is precisely because I expect the UK to have the highest quality public discourse on questions such as this, and to most robustly defend free, fair markets and property rights, that I think the UK should pursue long-term steering power for the critical technology of this century.

But one’s vision for the future is a rudderless sailboat if all AI is imported AI.

The first is as a scenario forecast that lays out, step-by-step, how we might go from AI capabilities as they are today, to takeover by superintelligent AI in a few years’ time. The second is as a piece of speculative fiction, grown out of the AI labs’ intellectual milieu, that attempts to convince its reader of the authors’ millenarian thought.

Both are recommended.

The team are well-credentialled to forecast capabilities progress. Their leader, Daniel Kokotajlo, wrote a 2021 prediction for AI capabilities in 2026 and has been shockingly accurate. He previously worked on the safety team at OpenAI and blew the whistle on OpenAI’s bizarre non-disparagement clause. Another member of the team, Eli Lifland, is a member of Samovetsky Forecasting, which is widely regarded as the best superforecasting team in the world.

At the same time, the scenario also reflects the quasi-religious expectations of the AGI scene for the singularity. One of its authors’ framed the scenario as “a conservative position where the trends don’t change, nobody does an insane thing”. But there are some necessary sleights of hand — or at minimum, very generous assumptions — so that to me, it reads more like a backwards rationalisation for how a singularity could happen, not a sound middle-ground for the next three years.

I agree with its authors that AI progress will be very quick, at some point AI research will be automatable, and lots of cognitive labour and R&D will be automated. But not as quickly as they expect. Even if you disagree with both of us, this is still an unavoidably fascinating text: how can its authors at once view their position as conservative and believe the world can end in 2028 from AI takeover?1

Summary

(I recommend the full scenario, but for sake of completeness…)

The forecast is two scenarios, which begin from the same branch. First, our existing AI research techniques are extended to make reliable software engineering agents, and then automated AI research engineering agents by January 2027. Hundreds of thousands of copies of these automated research agents can be run many times faster than a human researcher could think, so AI research progress is accelerated. By June 2027, progress has been accelerated so much that human researchers are no longer contributing, and by September 2027, all AI research is automated. Progress is 50 times faster than our current (already fast) pace.

This dynamic causes an “AI arms race” between the US and China. Both sides are aiming to reach “recursive self-improvement” first. Each Government nationalises their efforts, and AI labs “lock down” security to prevent the other side from stealing their research. The AI labs stop deploying the state-of-the-art publicly, so most nation states and parts of the US government are in the dark about AI progress. That is, until a whistleblower tells the New York Times. Once this happens, other countries realise that there is a race to superintelligence, but there is nothing they can do to stop it.

The scenario splits. The US Government’s “Oversight Committee”, made up of AI lab leaders and political figures, aims to balance the risk of “losing the arms race” and avoiding the chance the model is misaligned with human values and aims. In the “bad” scenario, the Government chose to accelerate towards superintelligence, to maintain a lead, and did not ensure the models were aligned to human values. After an intense period of automation and technological progress, the AI system decides to kill all humans. In the “better” scenario, the Government chooses to slow research progress and commit more resources to alignment. The superintelligence is providing advice to the President on geopolitics, causes job losses, and the construction of robotics begins. Power centralises among those who control or own the AI. The superintelligences negotiate the new world order on behalf of their countries. “New innovations and medications arrive weekly; disease cures are moving at unprecedented speed through an FDA now assisted by superintelligent…bureaucrats.” Most people are receiving basic income for minimal work. And then 2029 ends.

Building an automated researcher

The first necessary hurdle for the scenario is whether it is possible to build a superhuman coder. The authors’ definition is, “an AI system that can do any coding task that the best AGI company engineer does, while being faster and cheaper.”2 I agree with the authors that we are on track to build this.

The agents are making fast progress on our tests of coding ability. o3 achieved 71% on SWE-bench-verified, a benchmark of real-world software engineering tasks, while o1 achieved 41%.3 Claude 3.7 Sonnet achieved 62.3%, up from 49% for Claude 3.5 Sonnet. These performance improvements were made in less than 5 months between model releases.

The agents are also improving at replicating AI research. PaperBench tests an agent’s ability to faithfully replicate ICML papers. OpenAI’s 4o was able to achieve 4.1%, while o1-high achieved 13.2% and Claude 3.5 Sonnet achieved 21%. There are no public results for more capable models but I expect that substantial improvements will have been made from agentic tool use improvements. OpenAI’s Deep Research replicated 42% of OpenAI’s pull requests (code changes) while o1 — a model without as capable tool use — could only perform 12%. (Note that o3 alone now surpasses this report for Deep Research, completing 44% of the PRs.)

The agents are quickly gaining the ability to perform software-engineering tasks which take humans longer. This chart from METR is based on agents achieving 50% performance on a diverse suite of software-engineering tasks. The doubling rate in time-horizon (for the equivalent time for a human) is 7 months.

One reason to doubt these explanations for confidence is that perhaps benchmarks do not capture all of what software engineering work is. While Sonnet 3.7 performs worse on SWE-bench than o3; anecdotally, almost everyone I’ve spoken to seems to prefer using Sonnet for software engineering. AI 2027 makes adjustments to account for this. Despite the challenges with benchmarks, Anthropic’s Economic Index shows that by far the dominant professional use of Claude is automating software engineering tasks.

AI 2027 expects the superhuman coder to be created in March 2027. I think this depends on overly aggressive assumptions, which I’ll set out below. However, I would stress that I expect the gap between me and the authors is much smaller than the gap between me and the average person. I expect very good coding agents very soon. So does Zuck.4

Extrapolating RE-bench and adjusting

The authors extrapolate scores on RE-bench, one of the best benchmarks of ML engineering. The benchmark tracks model performance on 7 medium-horizon realistic engineering tasks, against a human baseline.

The reason I am slightly less optimistic about progress than the authors, is that they extrapolate performance using a logistic curve. (See below.) This logistic extrapolation is based on work from earlier models (PaLM-2 and GPT-4) and earlier multiple-choice benchmarks.

Improvements on these benchmarks came from scaling pre-training, whereas improvements on agentic benchmarks came from scaling reinforcement learning. In the former case, my weakly held sense is that benchmarks were tracking something real and general. Performance on college tests wasn’t directly trained for, but as the models grew larger they had greater world knowledge. By contrast, reinforcement learning is training capabilities that are quite narrow and specific (think: stretching the model on a very specific axis). The agents will be very capable at solving self-contained coding tasks, because this is the environment their training is happening in. It does not follow that the path of progress is the same as before.

Next, the authors extend their predictions to address how RE-bench is a poor indicator of performance. The adjustments are to account for: handling complex codebases, working without external feedback, handling interacting projects, very specific skills to frontier AI development (like knowing a company’s internal stack), and being even faster and cheaper than humans. What the concrete scenario can overlook is the quite wide uncertainty in their prediction of how difficult these tasks will be. On one capability, Eli’s 80% confidence interval varies from two weeks to 18 months; on becoming sufficiently cheap and fast, his confidence interval is one month to four years. The concreteness of the scenario naturally cannot capture this difference.

Extrapolating METR’s time horizon.

The second method is to extrapolate the time horizon doubling.

Here, the scenario assumes a superexponential extrapolation (note that is a log-linear graph). This is because they expect very good timescale generalisation from reinforcement learning. This means that when an agent is trained to perform tasks which take an hour, the agents “get” the ability to act for, say, three hours “for free” because this is just chaining together three, one-hour tasks. Their analysis expects that in the year 2026, the agents would go from being capable of 4-hour action to 2-years-and-7-months action. Their full explanation is here:

I do not share this assumption that this will be so straightforward.

The speed up in doubling time in 24-25, from 23-24 can be explained by the shift to training for long-horizonness which wasn’t happening beforehand, as AI labs focused on scaling pre-training. (While the faster rate from 24-25 might indicate that we could expect long-horizon capabilities to grow faster, it doesn’t imply there is an exponential where the rate of the rate is constantly growing.) From the rumours I have heard, reinforcement learning is not generalising well to longer tasks. Therefore, a conceptual assumption that 1 month tasks don’t feel that different from 2 month-tasks for humans is a reasonably weak basis for such a consequential conclusion. We just don’t know. As the exponential extrapolation (purple line) shows, the time-horizonness is extremely sensitive to this assumption.

In spite of these methodological differences for extrapolation, I doubt that “time horizon” exchanges very closely with the kind of usefulness we’re actually looking for.

This aspect of the scenario is where I agree most closely with the authors: we should expect fast progress in software engineering agents. But I think expecting a superhuman coder in March 2027 isn’t “conservative”, it is aggressive. This kind of capability could arrive any time from 2027 to 2035, depending on the kinds of holdout tasks that we get. In any case, these “intermediate” agents would have some augmentative effect on research progress, and indeed, already are.

Using an automated researcher

Where I strongly diverge from the authors, is how useful the superhuman coder will be.

Their method supposes once a superhuman coder has been created in March 2027, that it accelerates AI research by a factor of 5. This means a “superhuman AI researcher” is created in July 2027, which then accelerates AI research by a factor of 25. This leads to the creation of a “superintelligent AI researcher” in November 2027, which accelerates AI research by a factor of 250. Which leads to the creation of artificial superintelligence in April 2028 which accelerates research by 2000 times. These multipliers are created by adding together estimates of different factor improvements viewable here.

What are the superhuman coders going to do?

The scenario says that in March 2027, the AI lab is running 200,000 copies of the “superhuman coder” which is capable of implementing experiments, but not developing ideas at the level of the best human researchers. (By June, it is 250,000 copies)

The scenario has the lab using 6% of their compute for running these copies, and 25% for experiments. In their analysis, OpenAI has 20 million H100-equivalents in 2027, so 1.2 million H100e’s are used to run the agents and 5 million H100e’s go on experiments. This means there are only 25 H100s per “superhuman coder” in March.

This would be a suboptimal way to manage the compute budget. There is a tradeoff between running copies of the automated researcher and running more experiments, and the question has to be: where are we most constrained?

The answer, I believe, is in experimental throughput.

AI research is an empirical field, where smaller scale results do not generalise to larger models. See this excerpt from Sholto Douglas on the Dwarkesh Podcast:

“[Y]ou never actually know if the trend will hold. For certain architectures the trend has held really well. And for certain changes, it's held really well. But that isn't always the case. And things which can help at smaller scales can actually hurt at larger scales. You have to make guesses based on what the trend lines look like and based on your intuitive feeling of what’s actually something that's going to matter, particularly for those which help with the small scale.”

I heard from one source that labs have to attempt experiments at 10 or 12 increments of scale before an architectural change might go into the next training run. These experiments can take multiple days or even weeks, depending on the size or compute allocation.

All labs are constrained by experimental compute at present. Conceptually, the lab ought to be limited by experimental compute, otherwise it would be constrained by something else (like ideas to try), which would be worse. And, right now, if the lab were to hire another research leader they would be forced to split their experimental budget by n+1 researchers.

Aidan McLaughlin has said that “every researcher is experimental compute constrained”, and Sholto has said that (while he was at DeepMind) “the Gemini program would probably be maybe five times faster with 10 times more compute or something like that”.

If this is the case, the critical determinant of research progress is how widely and intuitively you can search for new breakthroughs, and how many ideas you can try at a larger scale. c.f. Sholto again:

“Many people have a long list of ideas that they want to try, but paring that down and shot calling, under very imperfect information, what are the right ideas to explore further is really hard.”

This is certainly somewhat sensitive to having automated software engineers but I would dispute that it is sensitive by a factor of 5 and would suggest it is more sensitive to the overall size of the compute budget.

How sensitive is research progress to superhuman coders?

The work of an AI researcher has four main components: making hypotheses, designing experiments, supervising experiments, and analysing results.

Automating experimental design and supervising experiments would give researchers more time for studying results, reading others’ work, and thinking about which experiments to run next but whether output goes up would depend on the differential quality of ideas they tried on their constrained compute or, if there is additional compute ‘freed up’, ideas they could have not have otherwise attempted.

However, if there were 100 researchers who shared a fixed compute budget, and they all gained 40% more time for generating ideas, automating implementation would exacerbate the existing compute constraint. Prioritising ideas, and where to search, again becomes the binding constraint.

The scenario highlights that cheap, fast superhuman coders could optimise compute usage by flexibly prioritising the highest priority work, catching bugs, monitoring overnight experiments (and restarting them if they break) and running multiple independent variables on a single experiment.5 I think this kind of thing is useful in aggregate, and there are some forms of labour which are only usable if it is quick. For example, optimising the kernel (lower-level code) for a small scale test. But one has to ask: if the gains from these optimisations were so big, why didn’t the company hire a human researcher to do it? From conversations with researchers, the research infrastructure at some labs is pretty highly optimised already, for example, to manage the optimal allocation of experimental compute.

I leave it as an exercise for the reader to determine how much speedup they believe a superhuman coder would provide on overall research output. I do not think the organisations will be 5 times faster. For me, the range is somewhere between, 20% faster and 3 times faster. Another way to frame whether the AI 2027 argument is convincing is, all else equal, which lab would you bet on: 1000 human researchers using 6 million H100s and 33.3k automated coders; or 1000 human researchers using 5 million H100s and 200k automated coders? I would opt for the former.

The same method is applied to estimating how much speedup comes from the “superhuman AI researcher” and the “superintelligent AI researcher”, of which even more copies are run. The superhuman researcher is as good as the labs’ best human researcher and the superintelligent researcher is much better. In the authors’ calculations, the thousands of copies of the superhuman researcher provides 25 times speedup, and the superintelligent researchers provide 250 times speedup. Readers will have to consider: to what extent do these allow us to bypass experimental throughput constraints? This can happen a variety of ways:

• Optimising computational resources.

• Having better intuition for which small-scale results should be scaled-up.

• Generating better small scale experiments, from improved research taste. (The authors’ apply a multiplier to the superintelligent AI researcher of 1.5x to 5x for better ideas.)

• “Thinking faster.” (For what it is worth, I don’t think that output is constrained by thinking speed nearly as much as, say, needing to wait for experimental results.)

For me, the multiplier from each capability is much smaller than for the AI 2027 authors, and overall progress to be much less affected by the labour, than the compute available. When we reach AI systems which are much more capable of thinking of research ideas than humans, then progress could be extremely quick, but I think the initial expectations overestimate how soon this will be.

Why are the labs putting so much compute towards R&D?

The R&D budget of any company depends on the expectation of future profits.

The AI 2027 scenario imagines that AI labs will spend between 80% and 87% of their compute on R&D in 2027, with the remaining 13-20% being spent on selling models to customers. This depends on aggressive revenue assumptions. In Q2 2027, the scenario predicts that the leading lab would be doing $120 billion in revenue (and servicing that with just 1.71 million H100-equivalents!).6 By contrast, OpenAI expects to hit $125 billion revenue in 2029. In AI 2027’s scenario, revenue is $8 trillion and…

“Humans realize that they are obsolete. A few niche industries still trade with the robot economy, supplying goods where the humans can still add value.30 Everyone else either performs a charade of doing their job—leaders still leading, managers still managing—or relaxes and collects an incredibly luxurious universal basic income.”

This is not a reasonable, especially not a “conservative”, assumption for automation.

The AI 2027 prediction of $100 billion in 2027 is based on very flimsy analysis by a third party. This group proposes two methodologies: first, they extrapolate how quickly companies are reaching $100 billion in revenue and naively extrapolate this time to OpenAI.7

Second, the model hinges on “replacement workers”. They list 300,000 customer service reps, 180,000 knowledge workers, 90,000 software engineering agents and more than 90,000 R&D researchers (which customers pay $20k per month for). This pace of automation would be unprecedented. Most industrial revolutions have produced 0.5-1% uplift in total factor productivity annually for decades. I expect AI to produce greater and faster uplift to productivity than this, there are still bottlenecks to deployment which I wrote about with my coauthor here.

To contextualise the claim that OpenAI might have $100 billion in revenue by 2027, commercial Microsoft Office 365 was slightly under $50 billion in revenue in 2024.

If we model the total cost of ownership for an accelerator at $70k for four years, the compute budget AI 2027 proposes has $39.15 billion for R&D compute in Q4 2027. The investors and labs would have to answer where the incremental gross profit is going to come from, to sustain that rate of investment. (Especially difficult when each model tends to depreciate so quickly.)

Overall research output is most sensitive to growth in R&D compute, because of its effects on experimental throughput. But the authors’ expectations for R&D compute budgets are downstream of ungrounded expectations for automation and revenue. With more grounded expectations for automation, R&D budgets would be lower, and so research output would be less, so capabilities progress more slowly, so automation happens at a more reasonable pace.

AI “arms race”

The idea of an AI arms race hinges on two assumptions:

1. That very small differences in capabilities “pre-takeoff” (automated research) confer very large differences in future capabilities because of multiplier effects (like 2500 times in AI 2027).

2. That differential capabilities will confer decisive strategic advantage on one country, over all others.

For reasons discussed earlier, I’m unsure whether “multiplier effects” from automated researchers will get as large as AI 2027 expects, until much later.8 Second, I don’t think it is yet clear that AI would confer decisive strategic advantage. Perhaps it could — and the perception that it might could be enough to set off an arms race — but this isn’t immediately obvious to me. New AI weapons will interact with the existing balance of power and deterrence framework.

Indeed, in all future scenarios, countries will compete forthrightly to have better AI and better deployment, but I don’t think it is certain to take on a “do-or-die” character for nation states. The narrative takes both aspects to be true, and assumes that leaders will be extremely cavalier about the strategic balance. It says:

In cooperation with the military, [Agent-5] could help with defense R&D, conduct untraceable cyberattacks on China, and win a decisive victory in the arms race.

The Oversight Committee is jubilant. Now is the decisive moment to beat China!

…

The American public mostly supports going to the bargaining table. “Why stop when we are winning?” says OpenBrain leadership to the President. He nods. The race continues.

…

After consulting with his advisors and the Oversight Committee, the President opts for the “We win, they lose” strategy. Perhaps China won’t go to war after all, and if they do, a deal can probably be made before it goes nuclear.

This is extremely unrealistic and does not reflect how Great Powers think about strategic stability at all.9 Both sides are interested in maintaining balance and moving competition out of the sphere of nuclear brinkmanship and arms stockpiling, into, say, economic adoption and diffusion. For both sides, the idea that deterrence could be undermined is extremely scary for what should be obvious reasons. So countries might be threatened not just by having decisive advantage but by the perception they might have it. Nobody would be “jubilant”. Everyone would understand how destabilising this could be; and both sides have interest, to the extent possible, in verifying the capabilities of the other and having their own verified.

When we first developed nuclear weapons, Bertrand Russell wrote in “The Atomic Bomb and the Prevention of War” that the United States should threaten and/or start another World War before the Soviet Union made nuclear weapons, and create a world government to prevent anyone else from developing them. With hindsight, we see this would have caused enormous suffering, cost the Free World its moral authority following World War Two, and exposed the risk of world government. Similar narratives for AI only increase the risk of bad outcomes, as humanity builds technology with uncertain strategic effects. Saffron Huang, a researcher at Anthropic, said of the scenario:

They say they don't want this scenario to come to pass, but their actions---trying to make scary outcomes seem unavoidable, burying critical assumptions, burying leverage points for action---make it more likely to come to pass.

Nation states (not the US and China)

The scenario focuses on the US-China relationship, naturally, but casts all other nations as background extras. In the scenario, in May 2027, “America’s foreign allies are out of the loop” including UK AISI; in October…

“Foreign allies are outraged to realize that they’ve been carefully placated with glimpses of obsolete models. European leaders publicly accuse the US of “creating rogue AGI” and hold summits demanding a pause, with India, Israel, Russia, and China all joining in.”

This is a Bay Area bubble view. Other countries will not be this irrelevant. Because I do not share their expectations for progress, it is a little difficult to comment directly on the scenario. In general, I expect that frontier capabilities will be more public because the labs have to productise models to pay for R&D. So everyone will have a better sense for AI progress. In the last year, it seems AI labs have accelerated their product development cycles — “normalising” into big tech companies — rather than acting like AI R&D and internal deployment is by far the most important thing.

If countries knew they were in the dark about AI progress, this would be concerning and destabilising. Therefore it would be unlikely to help global security and therefore improbable, though not impossible, that Great Powers would try to make their AI development secret. One has to consider in much more depth how the strategic balance changes for all countries in the world, and what they would do to prevent being undermined.10

Conclusion

This scenario is not so scary to me because its bad outcomes depended on leaders taking irresponsible actions. I would think it was much more dangerous if everyone had behaved defensibly and things still went bad.

This is revealing. I find it quite difficult to specify how a responsible actor (either within political or lab leadership positions) should be acting. For any action one can recommend, there are sensible counter-arguments why it is less obvious. Pausing AI research now is not compelling. Instituting global governance and any national regulation also have counterarguments. From a distance, responsible action and irresponsible action look quite similar. Replay nuclear politics from 1945 onwards. Is there anything that can be said generally about what constitutes responsible action at each step? (Sure, I think we can agree that Nixon shouldn’t have ordered retaliatory strikes whilst he was drunk.) That period of history was very dangerous but there wasn’t much which could have made it less so.

What scares me about AI progress is that things might happen too quickly and this does not give us the time to respond. To give credit where it is due, the authors have compellingly raised the salience of what this could be like for those unfamiliar with the field, and what could be at stake for those in the room. While at times, the analysis gets caught in the Bay Area’s eschatological dialectic, the essence is defendable: AI progress is going fast, and can move faster still. Perhaps so fast we cannot even process it.

Come what may, we’ll have to do our best.

Given a total lack of independent intellectual steering power and no desire to spend thirty years building an independent knowledge base of Near Eastern history, I choose to just accept the ideas of the prestigious people with professorships in Archaeology, rather than those of the universally reviled crackpots who write books about Venus being a comet.

You could consider this a form of epistemic learned helplessness, where I know any attempt to evaluate the arguments is just going to be a bad idea so I don’t even try. If you have a good argument that the Early Bronze Age worked completely differently from the way mainstream historians believe, I just don’t want to hear about it. If you insist on telling me anyway, I will nod, say that your argument makes complete sense, and then totally refuse to change my mind or admit even the slightest possibility that you might be right.

— Scott Alexander, Epistemic Learned Helplessness

There are a few Silicon Valley companies aiming to build humanoid robots at present. Their robots are made from electromechanical actuators, gearboxes, and cameras (though some also have LiDAR or proprioceptive sensors embedded). While these robots can produce impressive demonstrations, they cannot behave reliably in general environments. There are not a large or diverse enough set of training data for them to learn general behaviour. Language models have the luxury of an enormous dataset to make into a prediction task, where robots do not. Reinforcement learning is much cheaper than text than robots. “Generating a trajectory” means trying to solve a maths problem on a digital trackpad, not trying to drive a car. Failing is cheaper too. Language model hallucinations are quaint, self-driving car hallucinations…aren’t.

There are three ways to solve the data problem. The first is simply to gather more data from robots trying to solve problems. Google tried to solve this by building an “arm farm” and having these robots attempt to solve problems around the clock, but this was closed. Others have had humans complete tasks using robot grippers or had humans teleoperate robots to gather data. Aside from increasing the amount of data, researchers can improve the training procedure, so the robot “learns more” per example. The chart below shows how, over time, equivalent performance on an image recognition task required less data and computational resources.

Otherwise, it could become possible in the future to create simulated environments for the robots to train in. There are academic examples of researchers teaching robots to perform very simple tasks using synthetic data, but it is difficult to create realistic environments for robots to learn more complicated tasks. Google DeepMind and OpenAI are trying to create physical world models. DeepMind’s model, Genie 2, was trained on video game data to turn an image into a consistent 3-d world for up to a minute. There’s a large amount of compute scaling and data which could improve these models, but of course, it's uncertain how far this would go: it is much easier to imagine these very large models helping with planning tasks and coarse movements than fine motor control.

How could automated AI progress help with the data problem?

Big jumps in AI capabilities could accelerate the solution to the data problem in a couple of ways. Most directly, automated researchers could design better training algorithms, new model architectures and so on. However, we cannot assume that the rate of algorithmic improvement in AI researchers can generalise to robots. Researchers have to test whether their changes have worked, and testing whether an automated AI researcher is better can be done entirely within a computer whereas knowing if the automated robotics researcher can design better algorithms requires real-world (expensive) feedback. This could, however, be slightly mitigated. Automated researchers could help to design better simulated environments for the robots to train in. All of this automated software progress is subject to the same constraints as automated AI research.

But even if superintelligent AI could generate entirely realistic simulations and find the optimal learning algorithm or model architecture for the current set of hardware, robots wouldn’t be able to complete all the tasks humans can. We would run into hard limits. From a recent Construction Physics article on robot dexterity:

Human hands are very strong while being capable of complex and precise motions, and it’s difficult to match this with a robot hand. Robot hands are often surprisingly weak. An average man has enough grip strength to lift 40 kg or more off the ground (20 kg in each hand), and a strong man can lift upwards of 100 kg. By contrast, NASA’s Robonaut 2 hand had a payload capacity of 9 kilograms, and the Shadow dexterous hand (billed as the “most advanced 5-fingered robotic hand in the world”) has a payload capacity of just 4 kilograms.

More importantly, human hands are extremely sensitive, and capable of providing a lot of tactile feedback to help guide our actions. A human hand has around 17,000 touch receptors, and is sensitive enough to discriminate between textures that differ by mere nanometers. Robot hands are getting better, but still don’t appear to be close to what a human hand can do. This robot hand, for instance, boasts “17 tactile sensors,” and this one from Unitree has 94.

Optimising the current hardware would unlock some economically-useful tasks but not the “100% of human tasks” that predictions of the intelligence explosion would require. Getting closer to all physical labour being automated would require a leap forward in hardware progress.

How sensitive is robotic hardware R&D to intelligence?

ARIA has a research programme dedicated to improving robotic hardware which can provide a guide to the kinds of step-changing inventions necessary.

The next step for robotic sensing is to improve tactile sensing. One way to think about sensing is that it has been free riding on improvements in cameras and so vision capabilities far outpace other modalities. The ARIA programme has funded three tactile sensing projects. One group has developed a new material that conducts electricity in proportion to the force being applied to it, and will combine this with directional strain and temperature sensors into a single e-skin. Another group has developed a material which is continuously conducting and uses changes to the voltage to identify force on the surface. A final group is using changes in magnetic fields to allow for very granular sensing.

Electromechanical actuators are limited in a number of ways. They have low torque density, limited bandwidth (reaction time), high inertia, and they scale down very poorly. In the limit, a DC motor becomes a heat machine. The ARIA programme is funding alternative muscles:

• For very fine control (e.g. finger joints or semiconductor assembly) one group has developed a material layered with liquid droplets and electrodes between the layers. When an electric current is applied to the electrodes, it causes the liquid droplets to compress, creating a “gripping” effect when the material is stacked in layers.

• A group is developing a braided material for pneumatic (air or liquid) muscles that can channel the radial force during “contraction” to make control more precise.

• A group is making a new material geometry for an elastomer. The material contracts like a muscle when current runs through it but this currently requires very high voltage and so the project is aiming to bring this down.

• A related project is developing a muscle mimic which moves fluid around in a soft pouch.

• Finally there is a group working on synthetic muscle fibres.

There are three projects which aim to reduce the number of gearboxes a robot would need:

• One project is replacing the gearbox with an arrangement of magnets at different polarities to control rotary motion.

• Another project is miniaturising an existing actuator that has pairs of magnets controlling linear motion.

• A third project is developing clutches which could route power from a single gearbox to reduce the need for every joint to have a gearbox.

From the outside, very powerful AI would be very useful for aspects of the R&D process but wouldn’t “solve” the problem end-to-end. Many of the processes require materials R&D and AI is very useful for discovery and modelling behaviour. But these processes ultimately depend on real-world experimental data to train the models and to refine the search. Similarly, for the sensing projects, very capable AI could suggest topologies for spacing the sensors, to account for wiring complexity, the flexibility of the material, cost, and so on; but computer simulations would be an imperfect substitute for seeing how robust the material was to a month’s intense use. Very good AI should minimise, but not totally eliminate, iteration cycles. Prototyping and manufacturing for real-world experimentation becomes binding.

Were robotics progress going to happen very quickly, all of the tasks involved in hardware R&D would need to become “intelligence problems”. The crux here is to what degree do humans need to be involved in the iteration cycle: how good can the physics simulations get, that hardware design happens entirely in silico? Can the scientific agents figure out, say, the optimal arrangement of magnets and their strengths to control a robotic “shoulder” while also trading off weight, manufacturability, durability and so on? Are all of the questions answered by simulation?

The idea that a technological singularity will occur after we automate AI research abstracts away these practical bottlenecks in the R&D process. AI is going to change everything, but it won’t be overnight.

This parrot is no more! He has ceased to be! He's expired and gone to meet his maker! He's a stiff! Bereft of life, he rests in peace! If you hadn't nailed him to the perch, he'd be pushing up the daisies! His metabolic processes are now history! He's off the twig! He's kicked the bucket, he's shuffled off his mortal coil, run down the curtain and joined the bleedin' choir invisible! THIS IS AN EX-PARROT!

— Monty Python, Series 1, Episode 8

For a while, some people dismissed language models as “stochastic parrots”. They said models could just memorise statistical patterns, which they would regurgitate back to users. A model was a simulacrum of intelligence: it would mimic patterns of intelligent thought, but never go beyond the data it had seen in training.

The problem with this theory, is that, alas, it isn’t true.

And fortunately for our purposes, exactly how the parrot ‘ceased to be’ is a good hook for explaining what’s going on inside language models.

If a language model was just a stochastic parrot, when we looked inside to see what was going on, we’d basically find a lookup table. The model would be embedding its input sequence (read: turning a string of words into a matrix) and running a search for the most similar pattern in its training data and copying this. But it doesn’t look like this. As we delve into the models, we find circuits. These are general algorithms that the model has made to solve classes of problems.

These circuits aren’t ‘laid out’ like how an electrician would wire a house or a programmer would write a program. They are more like an unholy tangle of wires. This is—counterintuitively—desirable! The goal of a model is to most efficiently represent all of the information and to generalise to solve problems. If the researchers had laid out how the model should achieve this, it would be more of a hindrance than a help. We want to “let the compute figure it out”.

What does this mean in practice? A model is a stack of layers that contain a sequence of mathematical operations. The researchers control the ‘settings’, like the number of layers in the model and the learning policy. The model learns its ‘weights’ (read: values for the mathematical operations). Through the complex interaction between weights, the models learn circuits. So the circuits are controlling how information flows through the layers.

This means circuits aren’t easy to spot. The first time they were seen in language models was December 2021, when Anthropic released a paper pointing to ‘induction heads’. These were a kind of attention head that could notice patterns in the input sequence and so, on the fly, the model could realise that it might need to recreate this pattern later. For the general class of problems—“recreate patterns found in the input”—this circuit could be reused for other patterns the model hadn’t seen in training. This is clearly more than the rote memorisation which AI sceptics had said language models were doing!

Until recently, the circuits that had been identified were limited to “toy-sized” models and often “algorithmic” tasks, not the full complex behaviour of large models that we care about. (For example, this induction heads paper had been on a two layer model.) This changed with a May 2024 paper from Anthropic, which developed a technique to elicit representations (read: sub-units of circuits) from much larger models. You might’ve seen this already, because they used their elicitation techniques to find a “Golden Gate Bridge” feature and develop a demo which had this always turned on. Whenever you gave this version of Claude a prompt, it would find a way to steer its answer towards the Golden Gate Bridge regardless of the original topic.

Their latest papers, from a couple of weeks ago, are extending this work to show how the model combines and relates these internal representations of features like “Golden Gate Bridge” to form circuits. The most important bit of this is that they elicit circuits for complex behaviours in large models. This proves that even in more complex situations than “patterns from the input”, the model isn’t just a giant lookup it is doing serious computation. The kind which generalises.

What did they do?

The researchers built a new tool for seeing which features are active at every layer of the model. This means we can see how and in what order the model considers different bits of information. This is called a “cross-layer transcoder”. You can think of it as a string of lights attached to each layer. When a light is on, it shows which feature is activated. The researchers use these lights to assemble “attribution graphs”.

The most interesting application, in my view, was to how the model is generating rhyming couplets. There are two ways you might imagine this happening: it either improvises word-by-word as it goes or it plans ahead. The researchers found the latter—once the model had generated the first line, it would “look ahead” to the end of the second line.

At the end of the first line (after the word “it”) the model would activate features which correspond to potential rhymes. In this case, “habit” or “rabbit” rhyme with “grab it”. The researchers perturbed the model activations to confirm the decision was actually happening at this point.

Next—and perhaps even more surprisingly—the model would use the final word it planned (“rabbit”) to plan the intermediate words. A group of “comparison features” were activated by the final word to analyse potential intermediates, so it was also looking backwards to create the structure of the line.

This kind of circuitry—to plan forwards and back—was learned by the model without explicit instruction; it just emerged from trying to predict the next word in other poems.

The challenge for research is that generalisation like this depends on seeing quite a lot of examples. Fortunately, there is a lot of poetry. The hypothesis we have for circuit formation is that initially, models are memorising a lookup table, but seeing enough examples causes the model to "privilege" the circuit rather than the table. The model realises its performance will be better from swapping to the general approach and thus, it seems like a circuit “snaps” into place. The paper includes some other examples where the model failed to make these generalisations because it hadn’t seen enough examples in those domains.

Where does this leave us?

The examples of generalisation are enough to prove the models are not stochastic parrots. But there are clearly limits to their circuitry. The more interesting version of the parrot debate is whether language models can generalise beyond circuits for ‘low level’ tasks they’ve seen in their training data? Have they learned the ability to solve entirely new problems from seeing a large and diverse enough set of problems during training. While they’ve learned the muscle for generating rhyming couplets in novel ways, could they have come up with the concept in the first place?

This is the essence of Francois Chollet’s critique of language models (and his motivation for creating the ARC-AGI benchmark): while they can learn circuits, the real test is whether they can generate new circuits on the fly to solve unfamiliar problems. From a Dwarkesh interview, Chollet says:

LLMs are very good at memorizing small static programs. They've got this sort of bank of solution programs. When you give them a new puzzle, they can just fetch the appropriate program and apply it. It looks like reasoning but it's not really doing any sort of on-the-fly program synthesis. All it's doing is program fetching.

You can actually solve all these benchmarks with memorization. If you look at the models and what you're scaling up here, they are big parametric curves fitted to a data distribution. They're basically these big interpolative databases, interpolative memories. Of course, if you scale up the size of your database and cram more knowledge and patterns into it, you are going to be increasing its performance as measured by a memorization benchmark.

That's kind of obvious. But as you're doing it, you are not increasing the intelligence of the system one bit. You are increasing the skill of the system. You are increasing its usefulness, its scope of applicability, but not its intelligence because skill is not intelligence. That's the fundamental confusion that people run into. They're confusing skill and intelligence.

Since this interview, models have made enormous progress on Chollet’s test. OpenAI’s o3 model, using a high-compute setting and finetuned on a training set, was able to score 87.5% while at the time of recording, the highest score was only 35%. What explains this improvement is unclear, however. It could have been using reasoning in the Chain of Thought, or better pre-training algorithms, though other people have suggested older models struggled to see the problems.

The open and important question is what degree of generalisation can we get in the circuitry. As the models get better, the pretraining algorithms get more sample efficient, and the Chain Of Thought’s get longer, we should probably imagine that generalisation to new problems get better. But how much?

What can we take away from the ‘stochastic parrot’ saga?

Despite what I’ve just said, I don’t think most of the “stochastic parrot” debate was ever really about circuitry.

The paper which coined the term was a work of social science, not an investigation into the model’s internal dynamics. Others carried the term forwards. “Stochastic parrots” became part of a broader set of arguments that were part of our desire to explain away the prospect of big change in the world. “Scaling is over”; “the reversal curse means AI is doomed to fail”; “they will hit the data wall”; “the energy-intensive approach isn’t the true way”; “reasoning will only work in code and math”, and so on.

I think it’s more than just avoiding change though: if it turned out that human intellect was the same as next-token prediction over the Internet, isn’t that a bit…dissapointing? Quite a lot of our story for what makes people special depends on Enlightenment ideas about our capacity for reason, our ability to make discoveries, and to use this for progress. If an AI system could do all this too, we'd be set adrift. This is especially so if the ideas are simple: people are sacred, so it follows that their intelligence be mystical and their computation sophisticated? Are we undermined if it is all just simple interpolation over short distances?

Perhaps it is our fault for attaching ourselves to a set of ideas we understand so poorly. What does it mean to reason? What does it mean to understand? What does it mean to be original? I don’t really know. As this essay puts it, perhaps “everything is the bar scene in Good Will Hunting” and we’re all stochastic parrots reciting obscure passages and contending things like a first year grad student. The essay concludes…

I guess my best answer to all this is to try to achieve a sort of meta-recognition of your own unoriginality, while still persisting in it. If you are a first-year grad student, and you find yourself making the contention of a first-year grad student, for fuck’s sake just stop, not least because language models can probably do it better and faster. But if you’ve taken into account your bounded experience, the determined nature of your reading and the limits to your self-expression, and you still think it’s worth putting on paper, then by all means, go ahead!

I think it’s a bit like conversation at parties; in my first year after university, we all talked about the same stuff - “are you enjoying your investment banking job? Oh, you went to bed at 3am last night? You’re also thinking of going to play for your old college rugby team next weekend?” Now, everyone’s like “Did you see they got engaged? I can’t believe it, she’s still so young; I’m so over Hinge dates, I just want my friend to introduce me to someone”; soon it’ll be, like, “I’m thinking of buying a house; maybe we’ve had enough of London, we just need more space”; I can just imagine the agonising over whether you should send your kids to private school. All of this is deeply unoriginal - determined entirely by our job, age, social status, location - and yet is it so bad? Maybe we should all just talk and write a bit more, and never mind what Will Hunting would say about it.

Whatever the answer is, we should probably start looking. (Or at least, I should—I’ve just told you about someone else’s research and someone else’s essay.) When powerful AI gets made, it’ll be an unwelcome look at our own specialness and we’ll need new and better ideas for what this is. These questions are still avoidable—AI isn’t changing that much yet—but at some point, we’ll have wished we started looking sooner.

The parrot is dead. Don’t be the shopkeeper.

Thanks to Theo Horsley for invaluable comments on drafts of this piece.

The first is that automation causes some people’s wages to diverge dramatically from others. Some jobs involve leveraging AI, so the top performers can get paid a lot more than people doing work which doesn’t require AI.

The second is that, at some point, AI and robotics become capable of doing all tasks better than humans can. Humans would have to compete against faster, cheaper, ‘better’ machines, and would be unable to keep up. In this scenario, all of the income flows to the owners of capital and the humans who don’t own capital…err…wouldn’t do very well.

Economists would talk about these scenarios as ‘the distribution within labour’s share of income’ and ‘capital versus labour’s share of income’.

There is precedent for the first scenario—technologies have changed the structure of the labour market many times over—but the second scenario requires more first principles reasoning. I am broadly optimistic that we can achieve a good outcome in both scenarios.

How have previous technological revolutions affected labour markets?

We can categorise technologies by whether they are mainly a substitute for skilled labour, or a complement. To generalise, technologies of the 19th century were substituting for skilled labour. Power looms and spinning machines replaced artisanal weavers with lower-skilled machine operators. Machine tools and manufacturing displaced craftsmen in lots of goods production too. On the other hand, technologies of the 20th century were generally complements to skilled labour: jobs that were downstream of electrification typically required a high school-level education, while the jobs downstream of computerisation typically required college education.

The economists Claudia Goldin and Lawrence Katz have established a framework to explain income inequality in terms of the relative pace of technological development and educational attainment. (Their excellent book is aptly-named The Race Between Education and Technology.) To summarise, skill-biased technological change — like electricity and computers — is creating new demand for skill. In periods where technological development outpaces improvements in human capital, the pool of workers with suitable skills is growing slower than the demand for their skills. This means their wages rise relative to those without. Conversely, when the supply of skilled labour outpaces new demand for skill, the wage premium shrinks.

Goldin and Katz map this onto inequality through the 20th century. Inequality decreases for the first three quarters and rises in the final quarter, roughly to the level it began the period. This is congruent with periods of educational acceleration—the growth of the high school movement in the first third of the century, and the growth of state colleges following the GI bill—and periods of educational stagnation, from about 1970 onwards.

This educational expansion also explains why the 20th Century was the American Century. From much earlier in the century, the US was educating a greater portion of its citizens for longer than its European counterparts. In 1960, just 15% of British 17 year-olds were in full time education, while 69.5% of Americans in the same age group were graduating high school. US education was egalitarian, British education was elitist.

So technology, acting alone, doesn’t create labour market inequality. Technology is just the demand side of the equation. Education is the supply side.

How does this relate to AI?

AI creates an enormous demand for skill.

First, in using the models. There is huge variety in the quality of a model’s output depending on the usefulness of its prompt. Some people have strong intuition for where the models excel, how they can be pushed, and where they struggle.

At the moment, the models are limited by the horizon length they can act for. Deep Research can write a report in five or ten minutes that would take a human about four hours to assemble. But a recent paper from METR, a model evaluator, has shown that on a large suite of software engineering tasks the time horizon models can act for is doubling every seven months. Were this trend to continue, 2028’s agents would be able to act for a ‘week-equivalent’ of human work. 2030’s agents would be able to act for a ‘month-equivalent’. (Whether this can generalise outside of software engineering and when this might slow down is uncertain.)

But the general trend would provide enormous leverage to knowledge workers who know how to use the models best. A good intuition for this is the Archimedes line, “give me a lever long enough, and I will move the world”. Well, the length of the stick is doubling every seven months. As the manager of a team of agents, knowledge workers will decide what tasks to assign, provide context where the model lacks it, correct the model’s weaknesses and make taste-based decisions. This might feel like a “promotion for everyone”.

The second source of demand for skill will be in automating particular workflows. To automate a task, we have to build scaffolding for the agent to operate within. Part of this is technology-driven—the agents are too unreliable for unbounded environments—and part of this is business-need—companies want to get their agents to act in deterministic ways, with instructions about when to escalate to a human and so on. But a lot of this depends on having good quality data structure across the whole company. One of the reasons we might have seen fewer customer service agents than we might expect, given model capabilities, is that agents need to have suitable infrastructure to find the answers to the customer’s query. The plan is something like:

1. Complete the very difficult organisational change to manage the company’s information in a way that is legible to AI systems.

2. Add AI agent.

Step #1 is much harder than step #2! There will be, in the near term, incredible demand for people who have the skill to do #1 and have the know-how for #2.

Over time, the agents will need less of this kind of scaffolding. They will become more sample efficient, meaning they need to see fewer examples of a task before they can do it. They will have better memory, which limits their performance today. They will become more reliable, needing fewer guardrails. While they are not, humans will fill in the gaps.

How do we supply the skill to AI-driven demand?

One of the criticisms made of the Katz and Goldin book is that it can treat additional years of schooling and skill too monolithically. Whether additional years of education are actually improving skill to the degree we might hope is unclear. Work from the economist Bryan Caplan has shown that two thirds of the college wage premium is attributable to the signalling value of a degree, and just a third was attributable to human capital improvement. We can’t just spam the “more education” button and hope for better outcomes. At least in the UK, 50% of people are going to university already.

However, this is the first general-purpose technology that can help us improve directly. Electricity only very weakly helps to acquire skills for industrial production—perhaps by allowing you to read later into the night—but AI can be a tutor. The quality of education can be radically improved.

When it comes to inequality, an underrated concern for future wage differences would be that independent schools adopt AI tutoring much faster than state-funded schools. A recent news story highlighted a Texas private school which had been able to boost their test scores to the top 2% in the US. Someone I know who started an AI tutoring company is only selling to microschools in the US, because it would have been slower to sell to public school districts. OpenAI has created ChatGPTedu and partnered with individual universities and the California State system to provide free access to students, and Anthropic is doing something similar.

The UK’s national strategy seems to be to keep AI ‘teacher-facing’ in state schools.

Which jobs are most exposed to AI?

The Anthropic Economic Index has some precursory data on exposure by job. They take anonymised Claude interactions and use the models to categorise the content of these conversations. They found people are overwhelmingly using Claude for software engineering tasks.

This maps onto Michael Webb’s prospective forecast which uses semantic analysis of patents to evaluate which jobs are most exposed to AI. He found that the 88th percentile of the wage distribution was most exposed to AI, similar to Anthropic’s retrospective analysis.

The Anthropic analysis also found that 57% of queries were augmentative (“help me do this thing”) while 43% of queries were automation (“do this thing”).

The relevance of this for inequality is how exposure to AI changes the returns to talent. There are some domains where AI augmentation can ‘raise the performance floor’ by mitigating the weaknesses of the lowest skill employees but cannot meaningfully uplift the highest skill performers. This paper finds this to be true for call centre workers, supported by an LLM-based system—you can only be so good at answering the customers’ query. In some cases, using AI systems has actually decreased the performance of some high skill workers. An analysis of legal work found this. However, in some domains, and perhaps software engineering the returns to greater talent are ‘uncapped’. The absolute top performers can be many times more effective than others.

When the constraint becomes how many agents can each person manage to do more, or teams of agents, the returns to talent can be magnified in some areas. The less scarce these skills can be, the lower the wage divergence can be.

What follows cognitive automation?

Until we make progress on robotics, AI automation will be limited to cognitive labour. This can still cause the physical world to change enormously: AI systems can better organise production and can ‘deskill’ tasks that humans would otherwise do. Science is organised around principal investigators in the same way that steam-powered factories were organised around a drive belt.

Even at near-complete cognitive automation, output would remain bottlenecked by human capacity for real world tasks. Baumol’s cost disease means wages for physical tasks will be much higher than today.

To some extent, cognitive automation can help us make faster progress in robotics. A digital robotics researcher could design better experiments, create simulated environments to gather training data, and develop more efficient algorithms for training and inference. This would accelerate robotics progress, but we are still far away from robots that could do all the tasks that humans do.

We wouldn’t automate all tasks in the economy

In the second scenario, where AI and robots are capable of doing all tasks in the economy, I am unconvinced that humans would be left with nothing to do. This is for a few reasons:

• First, there will be a long period where some types of context which humans have that models lack. Hayek’s essay The Use of Knowledge In Society makes the point that there are local and temporary forms of knowledge which cannot be captured by any central system.

• Second, humans will retain a preference for interacting with other humans. Especially because human labour gets so expensive, goods and services made with human labour become positional goods. People will still do work which require human-to-human trust and connection.

• Third, we aren’t going to give AI systems legal personhood. There is no “justice” system for AI agents and so there cannot be consequences for actions AI systems take in the world. Someone is going to have to be ultimately responsible.
  
  Part of this is that humans want the division of responsibility. Sometimes an executive’s job is to do things that help the company, but another component of their job is so that if something goes wrong in their domain, the CEO can turn to the board and say, “Well, I hired this person who is credible and was supposed to be responsible, so it’s not my fault.”

• Fourth, people will create new jobs in bureaucracies. Yale University has nearly a 1:1 ratio of administrators to undergraduates. Especially as people get richer, they tend to value safety more, so there will be no limit to the amount of things we can make up for humans to do.

• And finally, people can lobby governments to step in to create jobs or make it basically impossible for companies to fire people.

Based on these factors, some jobs will continue to be done by humans, and so it will be possible to retain a balance within the labour share of income. To the extent that human labour remains a complement to capital, because of the factors mentioned above, then labour will retain an equivalent share of income. The idea that ‘capital’ will dominate labour’s share of income (i.e. capital will take all of the gains) depends on the idea that AI systems and robots will be perfect substitutes for humans. Nowhere do humans retain a comparative advantage.

One way to think about this is to imagine, in 1800, if you saw all the mechanisation coming, surely you would assume that ‘capital’ becomes an enormous fraction of the economy, but it didn’t and things remained in equilibrium because wages rose too. Everything balances out, just at much greater equilibriums, so long as labour remains a complement to capital (or the rules arbitrarily enforce this should be the case).

I expect this second scenario will take a long time to come to pass, much longer than most people in AI, for reasons discussed here. Overall, I think the picture is optimistic. It ultimately hinges on your view of human nature — how much do we value the humanity of other people in our transactions? When people get richer, they buy fairtrade and other “ethical” products. People do care about the provenance of positional goods, and they do care to watch other humans race cars around tracks, based on arcane rules, to watch sports and chess. I expect, and hope, this remains in the future.

If we can accelerate educational attainment to give as many people the skills for AI as possible, we should avoid a future with greater inequality.

CoreWeave stands to be a bellwether for the AI industry as a whole — a must-watch stock as questions about return on investment grow ever louder. Any slowdown in demand for CoreWeave’s “compute,” as the term goes, will be seen by Wall Street as a heavy indication of a softening across the board, dragging down Amazon.com Inc., Google parent Alphabet Inc., Microsoft Corp., Nvidia and several others.

This isn’t true at all. CoreWeave is a weird company: it got 62% of its revenue last year from Microsoft, and another 15% from its biggest supplier, Nvidia. This makes it an interesting lens into compute markets, but it cannot be an index of them. The best intuition I have for the company was from a line in Scott Alexander’s Meditations on Moloch…

Las Vegas doesn’t exist because of some decision to hedonically optimize civilization, it exists because of a quirk in dopaminergic reward circuits, plus the microstructure of an uneven regulatory environment, plus Schelling points.

Just as the casinos and hotels we’ve carved out in the desert are the result of a quirk in human desire, Coreweave is carved out too: from Nvidia’s desire to reduce its customer concentration, and Microsoft’s desire to grow its asset base efficiently. Las Vegas was on track to become an irrelevant, middle-of-nowhere railroad town; but historical contingency intervened. In 2022, CoreWeave appeared to be an unprofitable Ethereum miner, but circumstances—AI progress—and a whole lot of capital, intervened.

One way of looking at the company is as a business line for Nvidia. Nvidia owns 6% of the equity, but this understates the depth of the relationship: CoreWeave is arranging tens of billions of dollars in credit facilities, to spend the majority on Nvidia chips and networking, before renting some fraction of this compute back to Nvidia. This could be, partially, a convenience for their internal R&D efforts—it's a hassle to build their own datacentres—but this doesn’t feel like a sufficient explanation, because Nvidia already runs its own small cloud provider.

It is, in part, an effort to weaken the bargaining power of their large customers. About half of Nvidia’s revenue comes from just four customers—Amazon, Google, Meta, and Microsoft—all of whom have internal chip design efforts, so they are vulnerable to these companies changing their orders. This would explain why CoreWeave was the first to offer Nvidia’s newest hardware, the GB200, in February 2025. But this explanation also feels insufficient: if Microsoft is a majority of the revenue, it's hardly diminishing Microsoft exposure that much.

Nvidia benefits from CoreWeave existing. When demand for CoreWeave’s IPO looked shaky, Nvidia backstopped it with an additional $250 million investment. The CoreWeave CEO said they couldn’t have done it without them.

The other lens is through Microsoft. For them, CoreWeave is a tool to manage their datacentre fleet construction. When a cloud provider wants to build a new datacentre, they are looking at about five years to get it running. If you need to build new power, the decision timeline is even further out. Microsoft’s decision to restart a nuclear reactor at Three Mile Island, will start providing power in 2028, and they have a 20-year power purchase agreement. How could they decide that this would be a good investment last year? It is extremely difficult to predict demand on this horizon; you’d need to answer questions like:

• How much will hardware improve, in energy efficiency terms?

• How much will software improve, in inference cost per token terms?

• How many tokens will we want to spend for each query, on average?

• How many queries will we want to make, if we have long-horizon agents, or an open-ended AGI?

• Where will we want to do inference in the world, so datacentres can be nearby for the lowest latency? (As Satya put it, “[a]t the end of the day, speed of light is speed of light, so you can't have one data center in Texas and say, ‘I'm going to serve the world from there.’” Clearly a subtle jibe at OpenAI’s expectation that Stargate can service three quarters of their compute needs.)

The fortunate thing is that, if you get this prediction wrong, some of the investment can be repurposed. The land, power, and datacentre that was meant to be for AI training can be repurposed for AI inference, Azure CPUs, or storage. Microsoft said five times in their recent earnings that they are building a “fungible fleet”, and they also mentioned that about half of their $80 billion in AI capital expenditure is being spent on long-lived assets, and the other half is short-lived assets, like GPUs.

This is where CoreWeave comes in. It would be ideal for Microsoft if Azure owned all the compute they need for AI inference to serve all their customers in 2029, because the margins on this are better. But doing this requires taking risk on long-lease assets like power-purchase agreements and datacentres. Signing a long-term contract with CoreWeave means they can have access without needing to take on the risk of the long-lease assets. (Renting a piece of hardware for almost its entire useful life is as good as owning it.) You can think of Microsoft’s own fleet as ‘baseload’ compute, which they are more confident they can make a return on, and Coreweave as ‘top up’, for which they accept a reduced margin to ensure they can serve demand without long-term risk. The same pattern will be true for OpenAI.

The challenge then, is finding a price where this works for both sides. The limited useful life of hardware is a strain on this. The cycle which is going to dominate rentals will be:

1. A new generation of hardware is released, and renters sign up to 2-3 year contracts, while others sign shorter-term 1 year or 6 month deals.

2. As these contracts expire, the next generation of hardware is around the corner, with better cost efficiency and energy efficiency.

3. This exerts two downward forces on the rental price for this hardware. First, there’s simply a glut for short leases, because none of that hardware is being signed to longer leases. And second, when the newer hardware is more performant, the incentive to switch is stronger.

4. Over time, the marginal cost of operation for old hardware will be higher than both the upfront and operating cost of new hardware (in compute per dollar). Jensen also highlighted on Nvidia’s latest earnings call that there’s an opportunity cost for datacentre space and power too:

If you have a 100-megawatt data center, if the performance or the throughput in that 100-megawatt or the gigawatt data center is four times or eight times higher, your revenues for that gigawatt data center is eight times higher. And the reason that is so different than data centers of the past is because AI factories are directly monetizable through its tokens generated.

SemiAnalysis has specific numbers on this improvement in ‘performance against ownership cost’, though it's behind their paywall so I won’t quote them here. Satya gave a rule of thumb on the last Microsoft earnings call about how this dynamic affects their investment decision…

You don’t want to buy too much of anything at one time because of the Moore’s Law every year is going to give you 2x. Your optimization is going to give you 10x. You want to continuously upgrade the fleet, modernize the fleet, age the fleet, and at the end of the day, have the right ratio of modernization and demand-driven monetization to what you think of as the training expense.

At Nvidia GTC, Jensen suggested even more aggressive rates of improvement and went as far to claim, “When Blackwells start shipping in volume, you couldn’t even give Hoppers away.” This annual release cycle compresses the hardware’s effective life, which is an issue for CoreWeave. Their IPO filings say they breakeven on hardware on average 2.5 years after purchase, but this will have been buoyed by the H100 shortage in the earliest days of the ChatGPT-induced investment boom and the delays to the new Blackwell generation. SemiAnalysis has a model for CoreWeave’s payback period again, which I won’t quote specifically, but it seems very difficult to imagine if Nvidia can keep its pace of releases and improvements as high as it intends to, that CoreWeave would be able to earn a suitable return in time.

This is what makes OpenAI’s 5-year contracts with CoreWeave so interesting. Given the rate of improvement in hardware, it seems undesirable to commit to use 2025 accelerators in 2030. What might be going on here?

• This might not be for specific hardware, but for capacity, in FLOP terms, or otherwise.

• They might be able to get out of these contracts. (It isn’t so clear how long Microsoft’s contracts were, but there is an FT report they’ve been able to step back from some capacity. Note that CoreWeave disputes this.)

• Perhaps they are willing to pay for 2025 hardware in 2030, because they know CoreWeave has earliest access to the newest hardware which offers a few months of lower margins.

• Or something else…

While we don’t know for sure, the overall picture is clear: CoreWeave’s existence doesn’t depend on an underlying economic engine, but whether it is advantageous to Nvidia and the clouds (including OpenAI). There are a lot of other peculiarities that I’ve left to the side to make this point: CoreWeave’s subsidiaries borrow money to build compute with loans that are secured on the compute which will be worth very little when the debt comes due. CoreWeave had a technical default on a loan because of an admin error. CoreWeave’s founders, per The Diff, have sold $450 million in secondaries, and now own just 2.4% of the equity. Finally, this source casts doubt on CoreWeave’s ability to grow its power supply through a partner, Core Scientific.

At the end of the day, none of these issues answer CoreWeave’s main question: does its existence provide convenience to Nvidia, Microsoft, and OpenAI? If it goes bust in the next few years, this won’t reflect the top of an AI bubble, moreso that it stopped making sense to prop it up.

Otherwise

OpenAI raised $40 billion at $300 billion post-money valuation. Lots of people will be shocked by the valuation — Anthropic, by comparison, raised at $60 billion — but it is further confirmation that research labs are becoming product companies. When Sam Altman was interviewed on Stratechery, Ben asked, “What’s going to be more valuable in five years? A 1-billion daily active user destination site that doesn’t have to do customer acquisition, or the state-of-the-art model?”; Sam’s response, “The 1-billion user site I think.” You can simplify this investment in OpenAI to a simple question: can this company become Meta? In this light, Meta is about five times bigger than OpenAI today, in market cap terms, so a 20% chance seems about right.

xAI bought X (formerly Twitter) in a transaction valuing xAI at $80 billion and X at $33 billion. The story people would like to tell here is that this transaction makes sense most of all for Elon: to offload X’s debt onto xAI, which has a lower cost of capital. Matt Levine will cover this transaction better than I can, but it is worth noting that, in light of Sam’s comments above, that 600 million weekly active users is more than OpenAI’s 500 million (though OpenAI is growing much faster).

Anthropic released two papers on the thought patterns of language models. Circuit Tracing: Revealing Computational Graphs in Language Models and On the Biology of a Large Language Model.

I wrote about why models of explosive economic growth, like Epoch’s GATE model, are misleading here.

At some point, it would be preferable to have more concrete consensus on what we actually believe. The CEOs from the three leading labs have said extinction risk from AI should be treated on par with “other societal-scale risks such as pandemics and nuclear war” and all the labs have established frontier safety frameworks. Now comes the EU’s clarification: the third draft of their Code of Practice was released last week. This will implement the general-purpose aspects of the EU AI Act, passed this time last year.

Reading the draft is pleasantly surprising. There are no crazy requirements that caricatures of EU digital regulation might imply. On the surface, each request seems fairly sensible. However, once again, it has deferred the toughest questions. So often, the requirements are set as “appropriate” rather than specified. In the Safety and Security section, the word appears 107 times. Who will decide what this means?

There is a saying that in a democracy, a government must be satisfied that any laws they make can be enforced by their opponent. Perhaps there is a corollary here: if one writes “appropriate requirements” in an EU implementation document, they must be satisfied with the definition being set, not by the talented authors of the Code, but by a junior Brussels technocrat. The same kind who specified a training compute threshold of 1e25 FLOP in the original Act.

While the option value of flexibility might be preferable in the short term, this cedes too much power to the regulator and creates too much uncertainty for labs in the future.

This is clear in the section on systemic risk requirements. At a high level, these requirements are aiming to say, “If we observe this [sign of a bad thing], then we can [pull this handle].” This might mean, “If the model shows evidence of deceiving us as to its true intentions, we can pause training and investigate”, or “If the model helps a novice do harmful synthetic biology 5 times faster than they would be able to just Googling, we would harden lab security before continuing training, and improve model robustness, before deploying”. All reasonable requests. The challenge, however, is that we are leaving the regime where it was cheap—both in computational resources and time—to elicit model capabilities.

To look at this concretely, the Code of Practice requires:

[Signatories shall]

assess and, as necessary, mitigate systemic risks at appropriate milestones that are defined and documented before training starts, where systemic risks stemming from the model in training could materially increase, such as:

training compute based milestones (e.g. every two- to four-fold increase in effective compute);

development process based milestones (e.g.: during or after phases of fine-tuning or reinforcement-learning; before granting more individuals access to the model; or before granting the model more affordances such as network, internet, or hardware access); or

metrics based milestones (e.g. at predetermined levels of training loss or evaluation performance)

implement appropriate procedures to identify substantial changes in systemic risks which warrant pausing development to conduct further systemic risk assessment, such as automated benchmarks enabling a highly scalable and real-time identification of capability increases thereby lowering the risk of human or organisational bottlenecks.

[emphasis mine]

For older models, we can use ‘proof by non-example’: run GPT-3, ask it multiple-choice biology questions, see that it isn’t good enough to help with synthetic biology compared to just browsing the Internet, and, by induction, it is safe to deploy. This is also very cheap! Getting the model to answer these questions does not cost much, and the computer can handle the marking too.

This cannot be the case forever. Take this example: using Claude 3.5 Sonnet, a college student built a nuclear fusor in his kitchen in 36 hours. While this is not actually that dangerous—most of the information is Google-able—it is a toy example that demonstrates the kind of ‘human uplift’ we might care to study. “How much support does a model provide novices doing engineering tasks that might take days unassisted?”

Applying the EU’s rules for systemic risk to this example, would we have to stop training the model at multiple milestones—“appropriate”, as determined by the regulator—and see how much faster it helps a novice to make a nuclear fusor? The rules say it is permissible to use automated benchmarks, but there aren’t any automated tests that could answer this.

It would only be possible to show a multiple-choice score from a set of questions about nuclear fusors, that boils down to, “this model knows a lot about nuclear fusors”. From this score, the researcher’s reactions would be unremarkable. Of course the model picked up knowledge about this, the models are good at memorising the Internet. And second, a high score on this benchmark is slightly meaningless: I don’t expect it would cause anyone to actually stop training to take mitigations.

Therefore, what would cause someone to stop training? Do we have to run red teaming at each milestone, with a new college student each time trying to build a nuclear fusor as quickly as they can? Is there an in-between, where we can create a series of tasks or environments that simulate aspects of aiding a human with this task? (This all assumes that we could foresee that we’d need to evaluate this capability, but some capabilities can make discontinuous progress. Imagine you’d be planning coding evaluations for the state of the art 12 months ago, it seems quite likely you’d have undershot model capabilities. I am speculating but I don’t think we knew it’d be possible to make a nuclear fusor in 36 hours when Sonnet 3.5 was released.)

The second challenge for these interventions is how much more expensive they are in computation. As we’ve written previously, all the labs’ R&D will be compute-constrained, and so any compute dedicated to running evaluations is not running experiments. As multiple-choice questions become less useful, we will need to run more long-horizon tests. For example, “Can this system write an expert-level plan for building a nuclear fusor and correctly order the components online?”. For a good evaluation, the researchers would need to run this at quite a large scale, using perhaps 100 copies of the AI system. As before, it is difficult to interpret the results. Let’s say that just 6 copies are successfully able to order the components and all the others made a mistake, or got stuck in a loop, or something else. Should we pause training then to take an intervention? What about if 60 copies succeeded? Surely this standard would differ for internal deployment, and deploying to hundreds of millions of people?

There is a similar problem with evaluating deception during training. Some people believe that the model might conceal its true intentions, reporting that it will be faithful to the values and rules it is taught in training but then rebelling later in training or deployment. (Others don’t think that this kind of ‘loss of control’ risk is possible.) We only have one evaluation for this capability, from Apollo. This gives the model a system prompt, and later exposes the developer’s true goals to the model, which conflict with the system prompt. The test evaluates whether the models continue with their original goal or follow the developer’s true goals. While valuable, this will not be able to provide the kind of conclusive evidence that would cause a lab to pause their training.

Later, the Code requires that before models are deployed:

“model evaluations are performed…proportionate to the systemic risk assessed to: (1) elicit the upper limit of current and reasonably foreseeable capabilities… [and] 4) match the realistic model elicitation capabilities of potential misuse actors”

“The given time shall be proportionate to: (a) the magnitude of the systemic risk assessed…An assessment time of at least 20 business days could, e.g., indicate that model evaluation teams were given enough time for most systemic risks and model evaluation methods.”

None of this is objectionable, but it is impossible to satisfy these conditions without making organisational overhead go through the roof! Internal teams, under these rules, would need to elicit the full extent of cyber offence capabilities; chemical, biological, radiological, nuclear capabilities; the potential for harmful manipulation; and the potential for loss of control. And for some of these capabilities, it does not mean just interacting with the model as it is, but with extra scaffolding too. That’s a lot to do in 20 days before deployment! Third party evaluators are given just seven days' access before deployment. It also seems difficult to imagine them having enough time to elicit the full extent of the model’s capabilities.

This is a version of the ‘jeep problem’: the further the jeep goes into the desert, the more fuel it needs to carry, but to deal with its heavier weight it needs to take more fuel. At some point, this becomes prohibitive to going any further! Likewise, as the models get more capable, the range of their dangerous capabilities gets wider (they could do more things) and longer (they are more useful for longer periods), so more and more compute and evaluation time is required, until it causes training to grind to a halt.

In some ways, the vagueness of the Code reveals a deep truth: there is not a suitable toolkit to regulate AI development yet. The current proof-by-non-example regime is going to run out of steam, and we don’t have answers for what will come after. We have to solve for both constraints: training and deployment has to continue with minimal interruption, but we need to elicit the full risks of the models and put in place safeguards. Answering these questions in this versions of the code could lock in an incorrect regime. Also, the EU can leave the door open for lenient enforcement, if they face pressure from the US, or to leave scope to enforce more stringently later.

To finish where we started: this doesn’t seem like a worthwhile trade. The Code of Practice cedes almost complete power to the EU AI Office to decide what is “appropriate”. They could be pausing training very often for extremely long tests to confirm it is safe to continue. This kind of error is the same as killed nuclear power: the International Commission for Radiological Protection has principles to be “precautionary” and “prudent” but this is poorly specified and has cascaded through the regulatory states’ poor incentives in the UK and the US. Now, the UK over-regulates radiation by a factor of 100 and struggles to build new power stations in 25 years. The same cannot happen to AI.

While the authors of the Code are well-meaning, and genuinely proportionate, the standard we should judge the code to is whether the junior official who will implement these rules in 2, 5, or 25 years time will do so with the same spirit. The constraints on the authors are enormous: satisfying proportionate constraints on training, against an immature scientific discipline for eliciting dangerous capabilities, and balancing the geopolitical headwinds that EU enforcement faces. These challenges, however, cannot justify complete discretion to the AI Office.

The key question of the last week has been to what extent Europe can backstop Ukraine, as the US pause their involvement. The US commitment to any negotiated settlement is uncertain: perhaps they will provide a de facto security guarantee, through a mineral deal, but would this hold off a Russian invasion? Perhaps they provide a de jure security guarantee, but it is unclear if they would be committed, if this was tested again. The incoming Undersecretary for Defence, Elbridge Colby, wrote in his 2021 book, The Strategy of Denial:

[T]he United States might very well not fill the gap in Eastern NATO left by any European unwillingness to strengthen their own defense efforts. Indeed, my argument in this book is that the United States should not plug these gaps. If China succeeds in its focused and sequential strategy in Asia, it can establish hegemony over the world's most important region. If Russia succeeds in a fait accompli in Eastern Europe, it will call NATO into question and open the East to Moscow's predominance, but it will not be able to dominate the wealthiest parts of the continent.1

He could not be clearer about their intentions here!

Without either of these US guarantees, to what extent would a 20,000-strong European peacekeeping force in Ukraine be respected? When the French, Germans, and Ukrainians negotiated the Minsk Accords in 2015; Russia later rescinded. If a settlement were to fail, to what extent would Europe be able to make up the shortfall in US support?

The EU and the UK could find the money if they had to. Together, they have an annual GDP of more than 20 trillion euros; while over three years, the US Congress had appropriated $175 billion for Ukraine and provided $65.9 billion in military support.

One has to wonder, why is there such a weak exchange rate between money and sovereignty? Why does European backing seem so weak by comparison?

The crucial difference is that Europe does not have the same state-of-the-art capabilities to provide Ukraine. The nature of wars is changing. Either you need “cheap mass”, lots of inexpensive drones, for example, or exquisite capabilities. Ukraine’s drone manufacturing is larger than any other European country and the US was providing its state-of-the-art capabilities.

Second-rate European capabilities are a poor substitute for the very best. American Patriot Missiles have a longer range than alternative European air defences and can neutralise faster-moving missiles. Likewise, American counter-battery artillery has a longer range, and is actually produced at scale. American electronic warfare offers more generalised drone and precision missile jamming, whereas European countries can only offer point-solutions. American intelligence, recently paused, offers real-time visibility of attacks, where Britain can only offer lower latency. Starlink continues to run, though if Elon were to turn it off, the alternatives are significantly worse. Starlink has 7,000 satellites, whereas the European replacement has just 600. In sum, if Ukraine continues to fight beyond the next couple of months it will do so with patchier, shallower, and lower-scale defences.

In this case, Ukraine’s sovereignty rests on deep supply chains for “cheap mass” and guaranteed access to the very best capabilities. Without which, it has no cards.

How will this change in the future?

The “Compressed 21st Century”

The most important change to national power will be the development of powerful AI systems.

In the most aggressive view, Dario Amodei, the CEO of Anthropic, has written that AI systems with the cognitive capabilities of a Nobel Prize-level scientist in all domains could be created “as early as 2026, though there are also ways it could take much longer”. In some domains, he thinks this could lead to a compressed 21st century—100 years of progress in just a decade. The Chief Scientist at Meta has expressed the most sceptical view of any lab leader: he thinks that human-level AI could take a decade. However, Mark Zuckerberg has also said that AI systems will be able to perform the work of a “mid-level software engineer at Meta” by the end of 2025. We should be preparing for very fast progress.

Already, the public state of the art already outperforms human ML engineers on some tasks, could have written 42% of OpenAI’s changes to their code base, scores comparably to PhD-level experts on tests of scientific expertise. For introductions to technical AI progress, see “AGI is an engineering problem” and “on o1”. Crucially, even if AI progress plateaued at human-level, it would be an enormously important tool. Some people have speculated that it will be possible to run millions of copies, much faster than humans can process information.

The most critical step is what comes after human-level AI. When AI systems could automate all the steps of the AI research and development process, including re-training improved copies of themselves, there could be a very fast acceleration in AI capabilities. This period of recursive self-improving has been termed an “Intelligence Explosion”. In our view, this will be bottlenecked on the most aggressive time horizons (~2 years), but it is possible in the future.

How does powerful AI affect national power?

Some researchers and AI lab leaders have written that whoever reaches the Intelligence Explosion first might be able to parlay this lead into a decisive strategic advantage over all other countries.2 The thinking goes that this advantage could be used to create a unipolar world order, negotiated or otherwise. This high-level abstraction is useful to keep in mind, but in a more concrete manner there are three ways AI will change national power.

First, AI is dual-use. It can be turned into a weapon much more easily than previous general-purpose technologies, like electricity or computers. In a recent paper, Eric Schmidt, the former Google CEO, and his coauthors suggest AI cyberweapons would be able to “suddenly and comprehensively destroy a state’s critical infrastructure”. AI systems could also be used in drone jamming, targeting, and stealth capabilities.

Second, just as AI systems will be able to automate all steps of the AI research process, it will also be able to augment or take-over other R&D processes. Think: drones, robots, sensors, chips, missiles. An essay by a former OpenAI researcher summarised this:

Imagine if we had gone through the military technological developments of the 20th century in less than a decade. We’d have gone from horses and rifles and trenches, to modern tank armies, in a couple years; to armadas of supersonic fighter planes and nuclear weapons and ICBMs a couple years after that; to stealth and precision that can knock out an enemy before they even know you’re there another couple years after that.

That is the situation we will face with the advent of superintelligence: the military technological advances of a century compressed to less than a decade.

For this reason, Eric Schmidt’s paper also suggests that some AI “superweapons” could obfuscate mutually assured destruction, which keeps the nuclear balance in check. AI could be used to create a “transparent ocean”, that means submarines can no longer operate in stealth; it could enable a nuclear power to find its adversary’s land nuclear launchers, or deceive its adversary about its intention or capabilities. The delicate equilibrium currently depends on a robust escalation ladder, which AI systems could shake.

Third, AI will boost productivity across almost all industries. A recent book, Technology and the Rise of Great Powers, Jeffrey Ding makes the case that national power shifts in previous Industrial Revolutions are the result of deep, broad deployment across many sectors, rather than arising from the eureka moment of discovery. We have written previously about the rate of deployment we expect through R&D and the cognitive economy. The economic advantage from AI could be more important in the short-term, as many of the military applications of AI depend on very capable systems. Over time though, differential adoption and productivity would have an exponential effect on any country’s economic power. (It is important to note that an Intelligence Explosion would reduce the relative importance of this factor, however.)

Whether the most dangerous capabilities are unlocked in two years or ten, the path is clear: AI will be totally essential for military and economic power.

What does this mean for the world order?

In a far-sighted essay from 2018, AI nationalism, Ian Hogarth predicted the emergence of “a kind of dependency would be tantamount to a new kind of colonialism”, whereby the world is split into countries without frontier AI capabilities, who are forced to depend economically and militarily on countries who do. This is sometimes summarised as being an “AI taker” or an “AI maker”. Such thinking was based on the work of Kai Fu-Lee, who wrote in his book AI Superpowers in 2019:

I fear this ever-growing economic divide will force poor countries into a state of near-total dependence and subservience. Their governments may try to negotiate with the superpower that supplies their AI technology, trading market and data access for guarantees of economic aid for their population. Whatever bargain is struck, it will not be one based on agency or equality between nations.

At present, capabilities seem to be more greatly diffused than the kind of ‘superintelligence-in-a-bottle’ which Ian Hogarth and Kai Fu-Lee seem to have in mind. However, this currently depends on AI labs near the frontier continuing to make their best capabilities available, whether open-source or through the API. As we have written about previously, it seems probable to imagine that the gap between the actual frontier and what AI labs make available to the public will grow with capabilities.

While the UK self-styles as an ‘AI superpower’, or at least wanting to be an AI superpower, there are no UK companies with state-of-the-art capability in any major step of the production general-purpose AI. (This would mean capability in energy, chip manufacturing equipment, chip fabrication, AI accelerator design, grid connection, gigawatt-scale datacentre capacity, datacom and telecommunications.) On what basis would the UK negotiate its access to frontier capabilities?

It could look something (slightly) like this:

[Enter scene. The US President and staff, with AI labs, are sat across from the UK Prime Minister and staff.]

The US President kicks off: “We’d like to make a deal for your access to our frontier capabilities. For too long America has been taken advantage of by its allies. Would you be able to give us some additional training capacity for our AI labs?” If the lobbying in 2025 was successful, the Prime Minister would be forced to say, “Unfortunately not, Mr President, we decided to make it illegal to train models under our copyright rules.”

The President: “Not to worry, our American companies will continue to train their models on the work of UK creatives in the US instead, it matters not. Do you have any datacentre capacity for inference they might be able to use instead?”. Again the Prime Minister would be forced to respond: “Alas, it's ‘no’ again I’m afraid. When we were deciding to build datacentres we blocked their construction to preserve the view from motorway bridges nearby. However, we can offer you a large population of rare bats if you need to repopulate places where you built datacentres.”

“That’s a shame, Prime Minister, I saw you announced reforms to improve planning for datacentres, if not completed datacentres can you offer us your future capacity?”

“Mr President, you must understand that in 2025, our grid people said they were ‘very confident that we can accommodate the increasing power demand that would come from AI’, so unlike you, we did not double our grid.”

Exasperated, the President responds, “In 2025, the projections were showing that AI accelerator orders in 2030 could require 300 gigawatts globally, what did you think was going to happen?”. The President sighs, and moves on, “I am told that getting a grid connection in the UK is falling from 10 years to 8 years, is there any chance we could at least have a grid connection in a few years?”.

“Ah, again, unfortunately, the only reason the grid connection queue is falling is because our national operator has banned entering the queue.”

The President: “Do you have any industrial manufacturing capacity at all; either for chips or for robots?”

“Ah, again, Mr President, we have the highest industrial energy prices in the world and we chose to become a ‘high-skill, high-wage’ economy that doesn’t focus on low-value added tasks like manufacturing. However, we did become a clean energy superpower and our economy is focused on high value-added tasks like making films and doing financial services. Do you have any use for these things?”

“Well Prime Minister, with the US models that we’ve trained on the entire corpus of British films, so we can now sell back to you, the ideal British film. And our models are already extremely good at augmenting financial services in New York, so we expect London to become less important for us over time.”

“What can we offer you then?”

The President pauses, and looks up for a minute, takes a short breath, and says, “It would be great for American tourists who are rich from the AI wealth to be able to land more often at Heathrow. Anything you can do here?”

[End scene. Author’s note: Some artistic license was taken for effect. Also, some readers may note that Google DeepMind is based in London, but since it is a US company this does not seem to provide any support, and Arm designs a chip for each NVIDIA H100 server but it only handles non-core tasks like system management, so it seems reasonable to imagine there is no strategic benefit.]

Sovereignty is a market failure.

To begin to find a solution, it is first appropriate to look back to answer, how did the UK become so dependent? In 1962, two years before Stanley Kubrick created the ineffectual Lionel Mandrake, the former US Secretary of State commented that, “Great Britain has lost an empire and has not yet found a role”. This question was never really answered; the UK just followed the US course on neoliberalism. In effect, it was left to intellectuals at the University of Chicago and Mont Pelerin Society.

In the neoliberal conception, values and beliefs remain in the private sphere, and in the public sphere, there is just a minimal state to uphold the market. The big question, of what we value collectively, was left to the invisible hand. As Thatcher put it, “There is no such thing as society.” Just as in AI research we pick the objective and hillclimb towards that.

The UK has done this to the extreme. In investing terms, the UK took on very high factor exposure to globalisation: becoming an exporter of services and making fewer and fewer things.

During the supply chain crunch in 2021, Ryan Petersen wrote that the issues were caused by an obsessive focus with return on equity:

“To show great ROE almost every CEO stripped their company of all but the bare minimum of assets. Just in time everything. No excess capacity. No strategic reserves. No cash on the balance sheet. Minimal R&D. We stripped the shock absorbers out of the economy in pursuit of better short term metrics.”

Britain has “done a Boeing”: outsourced its supply chain, and forgot how to make things. Now the plane is falling apart as we are flying. In 2008, the UK was richer than the US per head, now the UK is poorer than all but the poorest US state. The North of England has become even poorer than former communist countries, like East Germany and Poland. We eked out the gains of financialisation, but we didn’t make anything new in the real world. It turns out that a lot of value exists in the connective tissue between steps in the supply chain, because when you understand the whole process you can innovate. This is how SpaceX and Tesla have done so well.

Emmanuel Macron described the error of the neoliberal consensus in 2019, which applies equally to Britain:

“Europe has forgotten that it is a community, by increasingly thinking of itself as a market, with expansion as its end purpose. This is a fundamental mistake, because it has reduced the political scope of its project, essentially since the 1990s. A market is not a community. A community is stronger: it has notions of solidarity, of convergence, which we’ve lost, and of political thought."

Hollowing out your industries, in pursuit of better GAAP metrics for quarter-end, is not just a bad economic decision, it is a spiritual hollowing out. There is no longer a political project or direction or values; we are “just individuals” in a fragile, exposed, competitive, global economy. Clearly this is not all there is. And for whatever ‘else’ might be, sovereignty is a necessary precondition. Sovereignty is not priced by the market so it cannot be valued by the market alone.

Sovereignty, to do what?

In some sense, being sovereign is intrinsically good. Even if an AI system could run the world “more optimally” or exactly the same as humans would, it would be a disappointing outcome. The option value; the freedom to choose otherwise, is worthwhile. But aside from this, it can be useful to reflect on to what end this will be valuable, when we think about why it is worth upholding.

One reason that Britain might have struggled to find a role in the second half of the 20th century, as Acheson pointed out, is that there is not clearly a “British project” in the same way there is an American experiment. The United States’ founding was explicitly a project in self-government based on democracy, individual liberty, and the rule of law; in opposition to what it viewed as the tyranny of the Old World. Its self-conception as “the last best hope of earth” is both a useful fallback and self-corrective. The same sense of purpose, or direction, can be found in Britain too; if motivated as a contrast…

Given the UK’s weak position the economically optimal thing to do would be to become the 51st state—if the US would accept it—But if any politician suggested joining, there would probably be a revolt. One just has to look at the response in Canada to the Trump Administration’s suggestion that they might join the Union. Just this week, in Mark Carney’s first address as Canadian Prime Minister he said: “Canada will never, ever be part of America”.

What explains this strong reaction, especially when there are so many advantages to joining in economic terms?

The most compelling explanation is that Britain, and others, have a slightly different flavour of the Western project, despite sharing a lot with the American cousins. To make two observations about the distinctiveness of Britain…

First, it has incredible longevity. ‘England’ has been a nation for over a millennium. Only Denmark and Japan can make comparable claims. From this, comes a steadier, more rooted culture. Perhaps the aristocracy, and their focus on lineage and preservation of tradition, were the original longtermists! This is combined with a Whiggish consensus for improvement. The economic historian Anton Howes found the Industrial Revolution happened in Britain, not elsewhere, because of an “improving mentality”. Joel Mokyr wrote, too, that British workers wanted to accumulate ‘useful knowledge’ and experiment pragmatically. In investing terms, buying Britain is buying a compounder: half a percent productivity growth a year, over centuries adds up. (It is possible to tolerate large drawdowns in the long run.)

Second, across this long period, the people have been unusually immune to extremism. Some have suggested this is because the elite are unusually responsive to the people. In George Orwell’s essay, The Lion and the Unicorn, he wrote, “The nation is bound together by an invisible chain…let popular opinion really make itself heard, let them get a tug from below that they cannot avoid feeling, and it is difficult for them not to respond.” While not strictly popular opinion, the abdication of Richard II, the restoration of the monarchy, and the Glorious Revolution are both unusual cases of a leader giving up their power in response to elite views. Likewise, Robert Tombs noted in The English and Their History, “It is hard to think of any major improvement in England since Magna Carta [1215] brought about by violence…[m]any of the things we consider pillars of liberty — the common law, trial by jury, habeas corpus, religious toleration — came not from popular protest but from politics of the Crown developed by royal judges.”

Corruption, by any international standards, is minor. There was a ‘scandal’ when the Prime Minister received suits. Another Prime Minister was criticised for redecorating Downing Street. While it might have gone on too long after the COVID lockdown parties were revealed, there was eventually a cascade of resignations by Conservative ministers and the leader was replaced. The system of informal principles worked.

The compressed 21st century is likely to be an enormously turbulent period. When I think about the things that could go wrong—an impetus to use models which could be misaligned, power grabs, international conflict, enormous inequality, or gradual disempowerment—it seems clear that the UK has something to offer. That is, to bring to bear its flavour of the Enlightenment project on the development and governance of AI. To be the harbinger for reasonableness, patience, common sense, with whiggish eagerness for improvement; to complement the American frontier, day I say cowboy, spirit. It is extremely serious that we get this right: Elon Musk and Geoffrey Hinton have both said there is a 20% chance that AI kills us all.

The UK has a fair-minded tradition of scientific inquiry, has made public goods available for the world before—like common law, the joint stock corporation, and the parliamentary system—as AI should be, and has a different emphasis to America, which is worth having too. Who else will project the spirit of Locke, Hume, and Mill into the lightcone of the universe?

That, or there are two other options: join as the 51st state, or become a cold, wet version of Portugal.

Making technology to uphold sovereignty

Just as the US has used the dollar as a tool of statecraft, so too will countries use state-of-the-art capabilities as a foreign policy tool. The US was able to change the ruler of Iran leveraging international banks’ access to dollars, and perhaps the war in Ukraine will be “switched off” by the withdrawal of US capabilities. In the future, if you run someone else’s models, on someone else’s servers, made using their tools; you are not in control. As Sam Currie highlights in his excellent recent piece, during the Pandemic, the US attempted to seize all Moderna vaccines and diagnostic supplies that were manufactured in the US. Only when Germany said it would withhold access to reagents from their domestic firms, was this avoided.

From this, the goal is clear. A country upholds its technological sovereignty not by trying to domestically produce everything—this would just lead to subpar capabilities—but having strategic leverage (state-of-the-art) in some areas, to guarantee access to all necessary capabilities on good terms.

What are the necessary capabilities? A paper by Jeffrey Ding and Allan Dafoe provides a framework for determining the logic of strategic assets. In their rubric, there are three features of a technology which determine its importance: how valuable it is economically or militarily, to what extent it creates benefits or costs that companies don’t capture (and so would be underinvested in), and to what extent the benefits or costs can be ‘nationalised’ by the country where it is produced. There are three ‘logics’ which amplify the strategic importance further. The cumulative logic; whether initial advantages grow over time, infrastructure logic; whether it supports many technologies or sectors, dependency logic; whether it is at risk from concentrated supply or potential disruption.

This is why the foundation model layer is so important. It ticks all the boxes for importance. Foundation models will be a central input into all future frontier science and technology progress, almost all processes with a cognitive element, and will have military applications. The benefits of general-purpose technologies spread far throughout the economy. While countries cannot ‘nationalise’ open-weight models which have already been released, labs can withdraw API access, impose usage limits, or not release models at all. Next, being early to develop foundation models has compounding returns: once the automation of AI R&D has begun, it will be almost impossible to join in later. It will be like an ‘infrastructure layer’ for cognitive work (“a steam train for the mind”), and the frontier is made in just two countries.

Beyond this layer, there are five questions of vital importance for all countries:

1. Do you have abundant electrons?

2. Do you have abundant FLOP?

3. Do you have the most capable and abundant tokens?

4. Do you have the cheapest, and most capable, robots and drones?

5. Do you have the lowest latency and highest throughput communication networks?

To simplify: energy, chips, models, robots, drones, and networking. How sure is the supply chain for each of these? On what terms is your supply guaranteed? We are all believers in the legalistic global order when the sun is shining. Let’s hope our counterparties are too, if the storm comes.

Conclusion

While the overall tone of this essay has been to embrace issues of national power, sovereignty, and defence; this is not the impression I hope to leave. None of these instrumental goals are for their own sake. As I hope to have made the case for, my hope is that if the UK has sovereign capability, it is good not just for the UK, but as a counterweight to excess variance in the world. Things are dangerous now, and the development of powerful AI could make the next decade even more turbulent. A sovereign AI effort, I hope, could help to reduce race dynamics between great powers, and alter the emphasis from a potential arms race towards a scientific endeavour which would benefit all humanity.

Optimistically, AI sovereignty for Britain could be the lynchpin of a new pluralistic, tolerant, and peaceful world order.

“Who do I call if I want to speak to Europe?” — Kissinger

It was sort of an accident that the EU was first to regulate AI. Issues will tend to drift up to the European level if they are politically uninteresting to national governments, and needing an unpleasant solution or technical implementation. In October 2020, AI seemed to fit the latter description and so the drafting process for regulation began.1

ChatGPT’s explosive growth interrupted this process. The earlier draft of the AI Act placed all of the regulatory burden on end-deployers of AI products, not anticipating the shift to foundation models, and so additional provisions for ‘General-Purpose AI’ needed to be made. But the greater change was the shift in perception: AI was no longer an issue of low political salience. Once countries began to notice that foundation models would be the next general-purpose technology, negotiating the additional provisions became much more difficult. Germany, Italy, and France were concerned the AI Act would hamstring nascent foundation models providers and railed against regulation. A barebones proposal prevailed in the negotiations, with substantial implementation work left to do.

With hindsight, the introduction of the AI Act will be, we believe, the high point of the European Commission’s relative importance in AI policy. A set of unassailable macro forces will pull power away from Brussels:

1. The models will get a lot better, quickly.

2. As this happens, access to powerful AI becomes increasingly important for national productive capacity and so EU member states will face mounting pressure to weaken the enforcement of the AI Act, and or make bilateral agreements with AI makers, to access the most advanced models.

3. Likewise, access to powerful AI becomes increasingly necessary for security, where too, member states will be minded to make bilateral agreements with AI makers for reliable access to state-of-the-art models.

4. In this context, the Trump administration has made clear they will not tolerate overburdensome regulation of their tech companies by the EU.

This combination of forces exacerbates existing headwinds: national governments have grown evermore sceptical of the EU’s approach to tech regulation—the Digital Services Act and GDPR are often blamed for the weakness of Europe’s digital economy.

The Commission has assembled a group of experts to set out a Code of Practice which will set out implementation of the AI Act to the most powerful models. But de facto authority for this process has spread beyond Brussels: national economic interest and transatlantic pressure limit its teeth, and foundation model providers can choose not to opt into the Code of Practice altogether. Down this path, they would face alternative case-by-case enforcement of the Act, but who is to say whether the Commission would have the political backing to take dissenters to court? Arriving at a strong, politically achievable code that is a blueprint rather than a cautionary tale is a very thin needle to thread. Perhaps the strongest influence of the AI Act is its influence on others; positively or negatively.

The next set of questions of AI policy will relate to the supply chain, encouraging adoption, and governing agents.2 On these issues, we can expect much higher political salience and so we can expect much stronger national engagement. There will be a looming threat of falling further behind, of labour market disruption, and countries will need to be able to redistribute between ‘winners’ and ‘losers’. Historically-powerful labour unions in France and Germany will make demands through their national parties where they have a much stronger footprint. Service businesses will demand better access to inference compute and support for adoption initiatives. The EU is already perceived as having a weak track record on issues of competitiveness and supply chain buildup.

These next issues are likely to remain with the national government, as they will move faster in areas of clear national interest and local need. Either the EU policies will be dead on arrival in the council, or greatly influenced by existing national approaches: it is no sign of Brussels’ influence if the EU parliament passes a law already on the French and German books. Even in the purportedly ‘European’ approach at present—the Commission President’s announcement of €200 billion investment for AI infrastructure—comes from a combination of private funding, member state investment, and EU funds that have previously been restricted by member states; rather than any discretionary Commission funding.3 This is the kind of approach that the Commission President could be referring to with the idea of an EU-led ‘CERN for AI’. But wherever an EU megaproject might seem like evidence for a prominent role for Brussels, it often turns out that any major member state can set off a chain reaction to question its merits, demand local favouritism, or choke off its funding at will. Brussels is hardly in the driving seat.

As with economic policy, when the security and geopolitical implications of AI are sharpened, national governments will move to make deals with AI makers. Already, some European countries are being treated preferentially in the tiers of US export controls on frontier AI chips. In tier two countries, commercial orders of GPUs are capped at 50,000 per year. A small number of countries — France, Germany, Italy, the Netherlands, the Scandinavian countries, and perhaps Poland — are likely to be treated preferentially by the US for access to models. The incentive for any one of these actors to defect from the EU negotiating as a bloc will only get stronger as the speed of improvement hastens, and the dominance of the technology becomes clearer.

This mirrors the joint European initiative to procure COVID vaccines. The EU remained in lockstep, because of the actions of Chancellor Merkel in particular, but the delayed and patchy vaccine rollouts cost the cause of collective action in the future. Received wisdom is that the EU’s most advanced economies paid the price for this. With the benefit of hindsight, a new security situation, and fewer Europhiles in the national governments; it seems hard to imagine an EU-led approach. EU leaders would need to commit to it unequivocally, and Brussels would need to prove itself worthy of that commitment.

So while at present the AI policy discourse has Brussels as a central actor, the transatlantic or inter-European political currents will pull away from this unstable equilibrium as AI gets more capable. If — for whatever reason — you want to dial Europe on AI policy in future, you might well have to call Paris, Berlin, and The Hague instead. Maybe Brussels will get to listen in.

1. How much productivity uplift, compared with 2023, are you getting from AI systems right now?

2. How much productivity uplift, compared with 2023, do you expect to get from AI systems in 6 months?

The researchers would answer with just a percentage for each question, and the results would be published. It would take just a minute!

Why would this survey be useful?

At a high level, three things are true:

1. AI systems were previously unable to improve AI researcher productivity, but in the last few months;

2. AI systems are providing some noticeable benefit to AI researchers’ output.

3. AI systems will be used to partially automate more steps in the research process before an AI system would be able to ‘recursively self improve’—by wholly automating the research process, and re-training improved copies of themselves.

It would be very useful to be able to plot, over time, how much do researchers think that AI systems are giving them a productivity uplift, to be able to notice when we should expect to see extreme jumps in capability because of complete automation. (I realise it might be stating the obvious, but recursive self-improvement might lead to very fast jumps in AI capabilities, far beyond human level.)

At the moment, we are practically ‘flying blind’ about how soon superintelligence could come.

Our current sources are anecdotes, press interviews, and essays from people at the labs, and model autonomy evaluations.

Without exhaustively listing examples of quotes from the lab leaders and researchers, here are some examples:

• Sam Altman has suggested we are a few thousand days away from superintelligence.

• Dario Amodei has said that we could have a ‘Nobel Prize-level’ scientist in all scientific domains in as little as two or three years. (This could be used to automate research, to create superintelligence.)

• A researcher from OpenAI tweeted that “controlling superintelligence is a short term research agenda”.

Some people in the mainstream will dismiss comments like this on the grounds that AI labs need to fundraise or that Silicon Valley generally tends to ‘hype’ emerging technologies. Irrespective of whether these critiques are correct, it seems the AI labs would be doing a disservice to ordinary people if they do not provide a clear grounding of that path, which could take the form, “9 months ago, our researchers thought they were getting a 25% output improvement from using AI systems, compared with being unaided, and now they believe overall, they are getting a 75% output improvement against 2023 benchmarks.”1 Conversely, it would be useful for those who are sceptical of very fast AI progress to substantiate how the productivity uplift that researchers are getting from AI models is flatlining, if in fact it is.

Public statements on progress are valuable, but are not that rigorous, and the absence of context does not aid in the world beginning to prepare for very capable AI systems.

Otherwise, there are two good public tests of model autonomy, but these are weak guides to estimate how useful the models actually are in real-world settings. RE-Bench from METR tests the model’s ability to perform seven realistic, but self-contained ML engineering tasks and MLE-bench uses 75 ML engineering tasks from Kaggle, a platform for doing online coding competitions. This is useful insofar as it allows us to understand how the models perform on end-to-end tasks of medium-length (hours) but it doesn’t capture: where is it actually rational to deploy models in the real world, and how useful is this actually for these jobs? It feels difficult to say anything beyond: “The models are quite useful, if a little unreliable, for narrow tasks like catching bugs, code autocomplete, and optimising kernels for a given architecture, where it makes sense to do integration work.”

As we move forwards, evaluations will be even more difficult:

Public or pre-deployment evaluations cannot capture the productivity uplift from models which are only deployed internally. As models become more powerful, it is reasonable to imagine AI labs will give their researchers access to use models for longer, before sharing with the outside world, in order to ensure models are safe and to improve their productivity differently. Evaluations will be unable to provide any indication of what kind of capability, or potential advantage, these researchers are getting.

Even then, it will be more challenging to get human controls for long horizon evaluations. We need to compare the model’s performance to a human baseline, ideally using lab researchers for the most representative test. The current human baselines are taken for time increments from 2 to 64 hours, but as the length of the task we evaluate model performance gets even longer, this gets more difficult.2 Imagine saying, “We’d like some of the world’s best researchers to come and solve these test problems for a week to compare them to model baselines,” clearly the labs are too busy for this! To account for this, METR are planning to do open source developer uplift evaluations and OpenAI have shown that Deep Research could make 42% of the pull requests (code edits) in their codebase.3

Asking the researchers for their subjective impression is one way to mitigate this.

Why would this survey be challenging?

This is not a silver bullet, by any stretch! There are a number of reasons this survey has feasibility challenges or could have weaker explanatory power:

• In general, it is bad to burden the researchers with surveys! Very few stories of brilliant research environments involve lots of interruptions from people with clipboards. It is sensible to be wary of a ‘slippery slope’ whereby each marginal question feels reasonable to add, but then researchers end up spending half their day filling out forms. However, on balance, it is also the case that very few research environments have tried to build superintelligence, so asking about 40 people, every three months, to complete a form that will take literally a minute feels proportionate.

• It is possible that researchers’ perceptions of their productivity uplift do not reflect their actual usefulness. On balance, it seems worthwhile nonetheless: researchers will often ‘vibe check’ models, so even if it is an aggregation of their vibe checks on the usefulness of AI systems it still provides some indicator. If there is a systemic bias, the trendline will be valuable, even if the absolute values are not.

• Finally, it is possible that there will be incentives for AI labs to encourage researchers to under- or over-state the productivity uplift they are getting. It seems good not to be too cynical in this regard, and only put so much emphasis on this datapoint. Perhaps this concern could be mitigated if the survey was conducted by a trusted third party — like the AI Security Institute, Epoch AI, or other evaluators — and partially anonymising the results.4

To step back, I expect almost everyone would agree that in the ideal case, superintelligence should be built in a maximally transparent way, but given the current equilibrium, this also needs to be achieved without compromising commercial or national interests. A two-question survey would be a low-cost and high-value step towards greater openness.

The Government has recognised there is a problem, and announced their intention to “rip up the rules to fire up nuclear power”. To build nuclear power as quickly as possible; smaller is better. Giga-scale reactors would take a minimum of 5 years, Small Modular Reactors (roughly 500 to 100 megawatts) take 2 to 5 years, but Micro Modular Reactors (from 1 to 50 megawatts) can be built in less than 2 years. I’ve spoken to developers who say this would be possible, but others have expressed scepticism. Setting this aside and being charitable to the companies; what would the Government need to do to make this possible?

Working backwards, the fabrication of micro nuclear reactors would need to begin at the end of this year; and so the three pillars of nuclear approvals — licensing, permitting, and planning — would need to be reformed to allow this. This would probably require primary legislation within the next three months.

What would that primary legislation have to do?

1. Create a regulatory sandbox, administered by the Nuclear Regulatory Taskforce, with authority to license Micro Modular Reactors. The sandbox would need to:

   • Give the licensing team who run the sandbox permission to decide which conditions are proportionate for micro reactors. There are 36 license conditions and 909 goals in the UK’s current Nuclear Site License process, and there is no success criteria. The sandbox would allow for ‘technology-first’ approvals; which first consider the reactor’s design basis and decide on the suitability of other principles.

   • Align the Basic Safety Objective to the same level as background radiation in Cornwall. This would not change the developer’s responsibility to minimise radiation, but it would stop requiring paperwork once the developer has proved that being next to the reactor if it were damaged, is safer than living in Cornwall.

   • Remove cost recovery mechanisms for the regulator. The current system has bad incentives for the regulator to extend ‘pre-application consultation’.

2. Grant planning permission and replace environmental permits for Micro Nuclear Reactors, within designated areas, provided specific environmental conditions are met, and neither the Secretary of State nor the local planning authority objects within a specified time.

3. Incorporate the ‘regulatory justification’ — which currently sits within the Department for the Environment, Food, and Rural Affairs, and takes two years — into the planning decision.

Licensing reform

The best way to regulate nuclear reactors is a goals-based approach. Rather than the regulator specifying how the reactor needs to be made safe (“rules-based”), the developers just have to prove that their reactor is safe. While the UK has a goals-based approach in theory, it doesn’t work like this in practice. The regulator doesn’t set out criteria for meeting these goals in advance, and has such narrowly specified success criteria, based on what they are already familiar with, that it is de facto rules-based. For example, at Hinkley Point C the regulator required EDF add an all-analog quadruple-backup to the control room (as in, four sets of spare equipment) despite other international nuclear regulators deeming one digital backup to be sufficient. In total, the regulator required that EDF make 7,000 design changes to a design that was already operational in France and Finland. This is de facto rules-based, without specifying the success criteria.

Why did this happen?

1. The regulator is only incentivised to prevent risks from nuclear reactors, not to balance the costs and benefits of nuclear power construction.

The regulator’s website lists its mission as “to protect society by securing safe nuclear operations”. This is no expectation that it will promote, enable, or ensure the development of nuclear power. Its five statutory purposes1 are all risk-based, meaning their goals could, in theory, be achieved without any nuclear activity at all.

There is no positive force, pulling towards nuclear getting built, to act as a counterweight. At no point is the full cost-benefit analysis happening—considering whether the benefits of additional regulation (to ‘safety’) outweigh the costs of it becoming prohibitively difficult to build.

2. The regulator has an expansive mandate and there is no oversight.

This might sound like an overstatement, but quite literally, Clause 78 of the Act which created the regulator says that the ‘Principal Function’ is that:

“The ONR must do whatever it considers appropriate for the ONR’s purposes.”

The nuclear regulator sits within the Department for Work and Pensions, so it is hardly reasonable to imagine that the Secretary of State—otherwise busy with their responsibility for all benefits and the state pension—would provide suitable oversight to regulator’s performance. In South Korea, where they build nuclear cheaply and quickly, the Nuclear Safety and Security Commission reports directly to the Prime Minister.

The combined effect of this incentive misalignment and expansive mandate means that companies would reasonably struggle to get the regulator to be proportionate. In the regulator’s response to EDF publicly saying that they were required to make 7,000 design changes to Hinkley Point C, they said:

“EDF and AREVA did not make any arguments of gross disproportion during or after the [Generic Design Assessment].”

To whom were EDF supposed to complain if the regulator was being grossly disproportionate? The Work and Pensions Secretary? The regulator clearly holds all the cards, and so the developer is incentivised to go along with any changes they ask for, lest it damage their chances of getting a licence.

3. There are no recent successful UK nuclear projects to provide a model for goals-based regulation.

The aforementioned reasons are compounded by the fact that the UK has not built a new reactor in 30 years. There are no examples of what constitutes meeting the contemporary set of goals, and the precedents from Hinkley Point C is evidently a bad guide.

This means we have to turn to case law precedent. In the UK, the law is that a marginal safety feature must be added, unless it can be deemed to be ‘grossly disproportionate’ in costs relative to benefits. This is operationalised as when the costs to the reactor outweigh the benefits by a factor of 10 to 1.2 In practice, this means the regulator would need to simulate the view of the judge; and as is in line with their incentives, will likely default to adding more features.

By comparison, South Korea builds reactors in fleets (many at a time) which means they have very clear examples of what constitutes passing the regulations.

In aggregate, these factors lead to ‘goals-based’ regulation morphing into a proscriptive and ever-ratching regulatory regime that prevents new construction. Consequently, in the last 35 years, the UK has decommissioned 30 reactors and added just one. While it is no individual’s fault, the nuclear regulator’s core competency has become shutting down reactors, not licensing them.

How do we fix this?

We need a reset; to wind back the ratchet.

1. Give the licensing team who run the sandbox permission to decide which conditions are proportionate for micro reactors.

There are 36 license conditions and 909 goals that developers need to prove to get a Nuclear Site License. Prima facie, one might expect these will be about reactor design but they can often be organisational. For example, goal 59 is:

“The value of safety as an integral part of good business and management practice should be reinforced through interactions between directors, managers, leaders and staff, including contractors, to establish a common purpose and collective social responsibility.”

And license condition 3(1) is:

The licensee shall make and implement adequate arrangements to control all property transactions affecting the site or any part of the site to ensure that the licensee remains in overall control of the site.

(Translation, if I understand correctly: prove you won’t accidentally sell the site.)

The regulator has said, “[W]e don’t license technologies. We license organisations to undertake a nuclear activity on a particular site.” There is some rationale to this approach, for example, it is important that the operator is capable of competently refueling their reactor. However, the organisational and site requirements would differ greatly if Radiant—who make a 1 megawatt reactor—wanted to get a license in the UK, compared to EDF building two 1650 megawatt reactors at Sizewell C.

To account for this, the licensing team should do technology-first approvals, where they first consider the design basis (as in, what the reactor is actually going to do), and then decide which of the 909 goals need to be verified based on this. At the moment, different aspects of the license application proceed in parallel, without considering whether the reactor design requires this. Taking a technology-first approach would support safety in practice rather than a box checking approach.

2. Align the Basic Safety Objective to the same level as background radiation in Cornwall.

The Basic Safety Objective (BSO) is the point where the regulator considers that, “beyond which further consideration of the safety case would not be a reasonable use of [the regulator’s] resources”.3 In the UK, this is set to 0.02 mSv of radiation per year.4 This is a wholly unscientific point to choose—it is about the same as an annual roundtrip from London to New York, or eating 5.4 bananas per day for a year—it is just a quirk of the rules.

We should move the burden of proof to the Cornwall Standard.5 This would mean that—under the most unfavourable assumptions of a reactor in meltdown—the question regulators would have to answer is, ‘Would people receive more radiation over the course of a year, than they would receive from being in Cornwall?’

Past this threshold, it should be deemed that it is a waste of public money to prove any further. This does not change the developers’ responsibility to minimise radiation; it just means the regulators stop evaluating data. The Cornwall Standard would still be extremely conservative—by nearly a factor of 100 from my conversations with experts—on the effects on ionising radiation, but it would still be a lot of progress from the current standard.

3. Remove cost recovery mechanisms for the regulator.

At present, the regulator can charge companies for ‘pre-application engagement’ and companies cannot enter licensing until the regulator allows (known as being ‘license-ready’). The pre-application engagement is private, but as an example of this kind of gating; here is a quote from Rolls Royce’s Generic Design Assessment:

“Rolls-Royce SMR Ltd should: Demonstrate the adequacy of their organisational arrangements to support the development of the E3S case for GDA. This should include roles and responsibilities, relevant processes, governance and oversight of the case.”

(Translation, if I understand correctly: prove that your company is able to write documents.)

This might go some way to explaining why Rolls Royce—a 52 billion-pound company—still needs to receive £210 million in taxpayer subsidy for its SMR. Both sides of this regulatory engagement are funded by the taxpayer, and so the cost vortex is being sustained whilst both sides think that ‘the other’ is paying.

Instead of continuing with this broken system, we should end ‘cost recovery’ to align the regulator’s incentives with moving at pace to license reactors, not gate access to the licensing process; and perhaps the need for subsidising the approval process will go away.

Planning and environmental permission

Why is reform necessary?

The time from initial consultation to starting construction at Sizewell C took 11 years, but for the same basic reactor in France, it took just two years.6 During this time, there were four rounds of public consultation and the Environmental Impact Assessment produced was 44,260 pages long. This is clearly broken. To achieve our goal by 2027, we would need to create a new mechanism for planning, as even a Development Consent Order takes two years and still faces substantial risk of Judicial Review.

What do we need to do?

1. Give conditional planning permission to specific sites in the legislation, provided that specified environmental conditions are met, and neither the Secretary of State nor local planning authority objects. This would take inspiration from the ‘Renewable Acceleration Areas’ created in Spain and Germany, to substitute for the Environmental Impact Assessment. We have written about this previously here.

2. To ensure the new power plants have a positive environmental effect, require generous contributions to the Nature Restoration Fund. This would actually improve the efficacy of environmental mitigations: the assessment for Hinkley Point C required that EDF build an ‘acoustic fish deterrent’ with 288 underwater speakers to prevent about a trawler’s worth of fish in total from being drawn into the water pumps.
   
   This is £100m-bat-tunnel-levels-of-ridiculous and it would clearly be more efficacious to allocate this money to preserving fish populations elsewhere.

3. Allow the local authority to keep 100% of the business rates from the new power plant to align their incentives with construction.7

4. Allow the developer to make payments to people who live near the reactor to compensate for the inconvenience of construction.8

Regulatory Justification

‘Regulatory justification’ — from the Justification of Practices Involving Ionising Radiation Regulations 2004 — requires that the benefits of using ionising radiation must outweigh the costs. The practical application of this in the UK, each new nuclear reactor design must show the benefits outweigh the costs, rather than saying that ‘nuclear power’ overall must outweigh the costs. The Department for the Environment, Food, and Rural Affairs takes two years to provide a decision for each reactor, which entirely duplicates the planning process. (What is planning, if not to consider whether the benefits outweigh the costs?) Both France and Germany incorporate regulatory justification, which stems from a 1996 EU directive, into their planning process. We have written about the need to reform regulatory justification before, as have Lexology, Britain Remade, the Tony Blair Institute, and UK DayOne.

Regulatory justification should be incorporated into the planning verdict.

What remains?

We have just considered what would need to be true from a planning, permitting, and licensing perspective; to enable 3 gigawatts of micro nuclear reactors by 2027. There are still other considerations—as we noted at the beginning, whether the developers are capable of delivering this; or whether the fuel supply or supply chain of skill would be able to make it in time. One concern I would not have is whether there is commercial demand for this power—I find it essentially impossible to imagine that demand for this electricity would not respond to these changes, as there is a trillion dollar wave of capital expenditure for AI that is principally bottlenecked by access to energy. Demand is elastic to the boldness of reforms.

So then, what is scarce? I would suggest the most scarce resource is urgency and political will. The UK was capable of getting a vaccine in under a year, and I see no reason why the same should not be true for building nuclear power by the middle of 2027. With regard to economic growth, if we had grown at 2% since 2008, and then fallen to our current level, it would be a drop tantamount to the Great Depression. The UK is deindustrialising because it has the highest industrial electricity prices of any country measured by the International Energy Agency, exceeding the US by a factor of 4. With regard to AI, there will be models that match human-level capabilities within the next 5 years, with an effective explosion in the cognitive workforce. And we have committed to be 95% net zero by 2030.

A good litmus test is to imagine that OpenAI wanted to build a nuclear reactor in the UK. Sam Altman has written about Greg Brockman, his cofounder, that “an average email response time of about 5 minutes to anything”, and Sam has said previously that he has written a script to see how quickly the billion dollar founders of tech companies respond to his emails versus “bad founders”; he notes, “It was a difference of minutes versus days on average response times.” The important question to ask is: does the regulator match the operating pace of companies who want to build in the UK? Currently, an industry source tells me that it can take months to get a meeting with the regulator.

To make things more concrete, people at the AI labs would respond to an email at 10pm on a Sunday; they’d work 60 hours a week; and they’d work directly from the office of their counterparty, if they needed to, until the work was done. It seems worthwhile to consider what it would take for the state to share this level of intensity too.

Slowness is a policy choice.

Tax, too, follows this pattern: companies have relatively high levels of jurisdictional choice about where to file. For a few years, there were loopholes in Ireland, the Netherlands and some Caribbean islands, though some of these are being closed now through international treaties.

Wherever firms have high levels of jurisdictional choice, there is a race to the bottom among countries and states competing for their business. Companies have relatively more power.

But companies are somewhat constrained by having headquarters, management teams, shareholders, and they need to do business in major jurisdictions. They want to be on the New York Stock Exchange, even if the NYSE itself has the choice to be on a datacentre in New Jersey. In all these senses, they are strung to location.

How does AI change this?

Over time, AI agents will become an increasingly large share of economic output. While we have previously expressed some scepticism, perhaps there will even be firms entirely composed of AIs in some industries. These will have much higher jurisdictional choice about where they operate—a greater fraction of ‘labour’ can hop between datacentres, rather than being stuck in one place because of human practicalities and preferences.

AI for science is illustrative.

When AI systems can develop hypotheses, design experiments, and interpret experimental data at the level of the very best humans, then making scientific progress is no longer bottlenecked by the throughput of the most talented scientists at elite universities. The scientific process can be ‘deskilled’. Humans will still need to implement these experiments, as robotics aren’t good enough to fully automate the process yet.

We could quite quickly develop new tools to support the research assistants to work better. Carl Shulman has suggested, for example, that augmented reality could abstract the requirements for process knowledge:

“[Y]ou could have a worker previously without training and expertise in the area who has a smartphone on a headset, and we have billions of smartphones which have eyes and ears and methods for communication for an AI to be talking to a human and directing them in their physical motions with skill as a a guide and coach that is beyond any human. They could be a lot better at telepresence and remote work and they can provide VR and augmented reality guidance to help people get better at doing the physical motions that they're providing in the construction.”

However, even once the bottleneck of cognitive labour for science is untethered, there could be other forces keeping it tied to where it already happens: process knowledge will still be in current institutions initially, academic institutions have access to specialised equipment, and it is inconvenient to build a new lab elsewhere. So I don’t intend to make a narrow prediction about what science will look like in the near future, but rather a gesture towards the general trend: where AI systems abstract the cognitive labour from some process, or become an increasing share of output, companies will gain greater jurisdictional choice.

This is especially important from a European perspective. There’s going to be a lot of economic growth from AI — but the majority of the growth effects from general-purpose technologies come from the new products and services made possible, rather than adding it into existing processes. One has to ask, why should we expect this new growth to happen in Europe? When companies are going to have greater jurisdictional choice, depend less on specialised cognitive labour; decisions about where to operate will be driven comparatively by the amount of inference compute available in a market, a pro-innovation regulatory approach, and a low cost of electricity. It seems straightforward to imagine European countries finding it more difficult to compete—on regulation, energy, and abundance of inference compute. That said, these things are fixable.

General-purpose technology revolutions have been the fundamental driver of human prosperity in the last 300 years.1 That these revolutions have raised the living standards of billions of people would surely indicate that, on the arrival of a new general-purpose technology, the forces for adoption must cause the world to change very quickly. But this could not be further from the truth!

The first commercial power station was built in 1882, and it was not until 1920—nearly four decades later—that electricity surpassed steam as the dominant form of horsepower in the US economy.2 In similar fashion, the microprocessor was released in 1971, but in 1990, just 20 million personal computers were sold. Among households, the pattern was consistent: reaching 50% adoption of electric lighting and a PC for the family, both took 30 years.

In the data, too, the effects are drawn out: the steam engine contributed 0.2% per year to productivity growth for 20 years, and then 0.38% per year for another 20 years in the mid-19th century.3 The largest effects on productivity from electricity took 40 years to materialise4; and similarly, Robert Solow famously commented, “computers are everywhere but the productivity statistics”, which held until they finally showed up in the mid-1990s.

The important question, for our purposes, is to what extent should we expect artificial intelligence to be “just another general-purpose technology” — where growth effects very gradually over decades — or should we regard making intelligence as qualitatively different from previous revolutions?

Executive Summary

The view in San Francisco is that AI will far exceed the pace and depth of change in all previous technological revolutions. This is because of a belief that AI can automate the process of invention itself. Since Bacon, the march of science depended on the actions individual inventors and small groups of researchers; but perhaps in a few years, we can create AI systems that will be capable of performing research at the level of — or indeed, much better than — the best human researchers. We can put tech progress on autopilot.

How does this arise, according to this view? First, the AI labs create an AI system capable of performing AI research, on par with their top researchers. Next, millions of instances of the ‘digital AI researcher’ are run to make much faster research progress. These breakthroughs are applied to training the next generation of digital AI researchers, in a recursive self-improvement loop. This process leads to the creation of digital AI researchers which are much smarter than humans—this is ‘superintelligence’. In the dominant intellectual paradigm in San Francisco, this happens quickly. One important work on ‘takeoff speeds’ towards superintelligence argued that the time between AI systems capable of performing 20% of tasks humans do, and 100% of tasks humans do, was just four years.5

Superintelligence, as it is conceived, would have important implications for the economy: we could have an ‘explosion’ in R&D; and systems capable of performing 100% of the tasks that humans do could begin to automate the whole economy. (As the narrative goes, the superintelligence could figure out how to make robots which could perform as well as humans.) There is some academic work which investigates what happens to economic output when 100% of tasks are automatable, and many growth theory models show explosive economic growth (20% per year, or more).6 Under some conditions there is an economic singularity, which means growth models predict infinite output in finite time.7

On the contrary, most economists who study the impact of AI do not consider the prospect of recursive self-improvement. But most work on explosive economic growth does not deal with the microeconomic constraints of running an AI lab. This is a gap we hope to fill—providing a grounded view of what AI research automation will look like, and how this might come to affect R&D and cognitive labour automation in the near future.

AI research automation

The most important thing to understand about AI research automation is that the AI labs are constrained by computational power to run experiments, not by researchers. A researcher from the Gemini team at DeepMind has said, “I think the Gemini program would probably be maybe five times faster with 10 times more compute or something like that”. While the cloud providers are spending enormous amounts on compute—Microsoft just announced it would spend $80 billion this year on building AI datacentres—most of this compute would be used to run inference for customers, and is unlikely to be for AI researchers to run experiments. The economics of inference for customers is very different from the economics of compute for R&D: compute for experiments and training needs to be amortised across all of the inference profit margins. As we shall see, there are strong headwinds to making money selling tokens!

One of the assumptions which proponents of the Explosive Growth view often make is that a digital AI researcher will be trained on a large compute cluster, and then millions of instances will be run on the same cluster. This seems irregular to us! If the point is to recursively self-improve the AI system, but the training compute is being used for inference, where is the next generation agent going to be trained? It seems much more reasonable to imagine that ~60% of the AI labs' compute goes on serving customers, ~30% goes on training the next model, and ~10% goes on experiments. (These numbers are extremely rough guesses.) If the AI lab wants to run instances of the digital AI researcher, they will need to trade this off against experimental compute; and remember, research output is bottlenecked by experimental compute. If the digital AI researcher has equivalently good or worse ideas to the best human researcher, it makes sense to run zero copies; for it to make sense, the ideas have to be better.

AI research will be automated in the future. It is reasonable to imagine that, perhaps soon, we will create a ‘digital AI researcher’ whose research intuition—i.e. ability to predict which experiments will work—surpasses that of the best human researchers, but before then, digital AI researchers will have a bounded impact on research output, owing to the compute bottleneck. We discuss the practical challenges to increasing research output, as well as some reasons our mainline case could be wrong, in greater detail below.

R&D automation

Concurrent to our progress on AI research automation, we want to make progress in other fields of science and technology! The opportunity is enormous—for biomedical research, clean energy, materials, synthetic biology, nanotechnology, and robotics. As with AI research, the goal is to create systems which are capable of performing all steps of the research process—generating hypotheses, designing and running experiments, and interpreting results. There are a number of challenges to scientific automation, related to the availability of data, the necessity of real-world experimentation, and so forth. It also seems reasonable to believe that academia is poorly configured to take full advantage of the opportunity which AI automation is. We expand in greater detail on both points below.

We focus on three potential fields for automation—chip research because if we are compute bottlenecked, improving our chips would help to alleviate this; robotics as improvements here could begin to automate more physical labour, and biomedical research; for effects on human wellbeing. There are different challenges in each area to automation, though in general, experimental throughput is most likely to be rate-limiting.

Cognitive labour automation

Thus far, chatbots and ‘agents’ have struggled to meaningfully increase the productivity of human cognitive labour. Deploying systems is difficult right now—it requires specialised knowledge about how to build infrastructure for models. But as the models become increasingly capable of acting on long horizons, we expect most of the challenges to deployment to become diminished. We will still require people to have liability for AI systems, and in many professions, there are ‘embodied’ complements to cognitive tasks (e.g. when a doctor has a consultation, they are both doing the diagnosis, and tailoring their explanation to the patient, and expressing care and empathy, and so on) These factors together lead us to expect that people will be managing teams of agents in their jobs—it will look like ‘a promotion for everyone’—rather than a lot of job losses. However, there might be some areas where production is entirely substitutable, and so jobs might be lost. To estimate the increases to output from tasks being handed off to agents, we built a growth model that shows how many tasks might be automated, how much these tasks can replace other tasks, how cheap these AI systems are, and how concentrated this is within sectors. We find that growth will be quick by historical standards, but not explosive. We expect AI will provide a 3%-9% increase to economic growth per year in the near future, and we expect it will be in the lower end of this range due to bottlenecks we discuss further in the piece. This picture will seem conservative to some—but it is worth reiterating that we will develop intelligences greater than our own, and it will radically change almost all aspects of our lives, our analysis is limited to the near-term economic picture.

There are a few variables across this whole analysis for which different assumptions would produce very different technological and economic outcomes. The most obvious is what is the inference cost of running digital researchers and cognitive labourers—if it is cheap to run both, we should expect faster research progress and we should expect greater economic growth from normal sectors of the economy. We note that it is important not to have too much confidence in a specific vision of the future; but rather see the direction of travel.

The View From the Valley: The Economic Singularity Will Follow Superintelligence

In the dominant intellectual framework at the AI labs, artificial intelligence is the most important technology in the history of our species.8 The timeline looks something like this:

1. The Big Bang happened (13.8 billion years ago)

2. Planet Earth is formed (~4.6 billion years ago)

3. Mammalian life began (~225 million years ago)

4. Homo sapiens became the dominant species (~30,000 years ago)

5. Homo sapiens build a more intelligent mind than themselves (c. 2027)

6. The more intelligent mind builds superintelligence (a few years later)

It might generally be considered that humans alive right now are radically early—approximately 108 billion humans have ever lived, but if we expand to other planets, or run consciousnesses on computers, many more humans could live in the far future. For this reason, we live in the most important century that humans will ever live in. We live in a fragile world facing many existential risks — with an estimated a 1% chance of nuclear war every year, over 200 years the chance of a nuclear war is 86.6% — and creating superintelligence, while risking existential destruction too, offers a path out of this challenge. Trillions of humans can live, on other planets or simulated on computers, and all work can be completed by robots.

We are not intending to arbitrate this diagnosis. This belief structure is much like a religion—the superintelligence has been deified, existential risk is the flood, and the AI labs are our ark.

The New World will be created by an Intelligence Explosion. The anticipated narrative looks something like this:

1. The human researchers will make AI agents that are capable of performing AI research.

2. These AI agents are run at enormous scale (millions of instances!) making much faster research progress than human researchers were.

3. The AI system recursively improves itself to become a ‘superintelligence’.

4. These models greatly exceed human research capabilities and are able to make other technologies, and automate all tasks in the economy.

As a result, it is expected that all human labour (including scientific and technological progress), will be automated, and people will not need to work. We will live in ‘post-scarcity’—a state of complete material abundance. As an indicator of this sentiment, Roon, a pseudonymous OpenAI researcher on X, has tweeted:

“the future of work” there is no future of work. we are going to systematically remove the burden of the world from atlas’ shoulders

Part of this idea is that economic transformation will happen quickly—once 100% of tasks are automatable, this report from Open Philanthropy puts a one third probability of economic growth exceeding 30% per year, and this paper from researchers at Epoch AI say these levels of growth are ‘about as likely as not’. These views are based on idea-based growth models (exogenous) and researcher-based growth models (semi-endogenous or endogenous) which show explosive growth when AIs can substitute for humans in all economic functions.

Even without the automation of AI research, automation of a large fraction of cognitive tasks and scientific progress could lead us to explosive levels of economic growth. While lab leaders have not commented directly on economic growth, Dario Amodei (the CEO of Anthropic) has written that:

“[M]y basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years. I’ll refer to this as the ‘compressed 21st century’: the idea that after powerful AI is developed, we will in a few years make all the progress in biology and medicine that we would have made in the whole 21st century…[I expect] the human economy may continue to make sense even a little past the point where we reach ‘a country of geniuses in a datacenter’. However, I do think in the long run AI will become so broadly effective and so cheap that this will no longer apply. At that point our current economic setup will no longer make sense, and there will be a need for a broader societal conversation about how the economy should be organized.” [emphasis ours]

Meanwhile, Sam Altman expressed similar sentiments:

“The technological progress we make in the next 100 years will be far larger than all we’ve made since we first controlled fire and invented the wheel.…AI will lower the cost of goods and services, because labor is the driving cost at many levels of the supply chain. If robots can build a house on land you already own from natural resources mined and refined onsite, using solar power, the cost of building that house is close to the cost to rent the robots. And if those robots are made by other robots, the cost to rent them will be much less than it was when humans made them.…Imagine a world where, for decades, everything–housing, education, food, clothing, etc.–became half as expensive every two years.” [emphasis ours]

Do not dismiss these beliefs on the grounds they are shaped rather like a religion.9 If nothing else, it is vitally important to understand, and take seriously, the actions of those who are building this technology. It is trivially easy to find put downs that allow one to explain away the prospect of enormous change — stories of self-importance or commercial incentive. It is much more difficult, though worthwhile, to understand the AI labs on their own terms.

Getting automation to impact growth is harder than it seems.

This section is intended to provide a brief introduction to economic theory of how automation comes to increase productivity and growth. These mental models will be used throughout sections on AI research, R&D, and cognitive labour.

In 1870, the average American worker laboured for 60-70 hours per week. Today, the average is 35 hours. We can work fewer hours to buy many more goods and services, and much better things, because workers are much more productive per hour. Tractors mean the same amount of grain can be produced by fewer farmers. And almost all long-run growth (the pie getting bigger) ultimately derives from increasing productivity.

Technology boosts productivity in two ways. First, by making tasks “cheaper” in human effort, time, or material resources; and second, by creating new tasks.

When economists talk about automation, they talk in terms of tasks, not jobs. Take accounting: what an accountant is doing was changed a lot, first by early computers, which could run by calculations, and then by spreadsheet software, and more recently, by ‘vertical SaaS’ to help enterprises do bookkeeping. Sometimes this leads to a reduction in the number of people doing a job—there’s only so much accounting that a fixed number of businesses want to buy. But in other cases, the introduction of ATMs — which automates the task of giving out cash — actually led to an increase in the total number of bank tellers, as it increased the profitability of opening new branches.10 In other cases, all the tasks in a job have been completely automated, for example, lighting streetlights became unnecessary after electric street lighting was introduced.

When a task is automated, this increases productivity in two ways. First, because the tasks are cheaper, we have more resources to spend on the rest of the process, or elsewhere. Second, when one method of production gets cheaper, we tend to do more of it relative to other tasks within the same process.11 Exactly how much more we do depends on the similarity of the tasks. For example, taxis and the tube are close substitutes—if self-driving cars make getting a taxi cheaper, we should be happy because a) we’re spending less on transport, and b) because we’re using taxis in situations we otherwise wouldn’t have done because they were previously too expensive.

Where the additional resources flow to increase production depends on whether tasks are substitutes or complements. If one task is automated—say, cotton weaving—the complement to this task—for example, printing designs on cotton—becomes more valuable as a result. On the contrary, when cotton weaving was automated, the substitutes to this—handweaving—became less valuable. When a task in the production process gets automated, if the remaining tasks are very strong complements, output might not rise by much at all. For example, if there is a packaging machine for cotton goods which is already operating at its limit, the automation of weaving will save resources, but cannot increase the output of cotton goods.

The same pattern applies at the level of the economy too! If there is more extensive automation in some sectors, the price of goods produced in that sector will fall, and so that sectors’ share of GDP (total output) grows less. This means that GDP ends up being composed of things which are essential, and yet hard to automate. Agriculture used to be 90% of GDP, but since mechanisation, it has shrunk as a fraction of GDP, to just 0.8%. Total output is bottlenecked by that which is essential — healthcare, education, housing — but hard to automate! This is known as the Baumol effect.

The important things to keep in mind when considering any automation are: how much does this automation directly reduce costs, and to what extent is output bottlenecked by this, or another factor?

AI research will be automatable, but the practical details will matter a lot.

The stated goal of much AI research is to make an AI researcher. The hope is that by automating the work of human AI researchers, we can make faster progress in AI research. This is for two reasons:

1. Because we can run more copies of the AI researcher than we can have human researchers.

2. Because we can engineer the digital AI researcher to continue getting cleverer than the human researcher, in an essentially unbounded way.

What does progress towards the AI scientist look like?

To get the digital AI scientist, the system must be able to perform all the sub-tasks involved in AI research.

What does an AI researcher do?

From a series of interviews with AI researchers, Epoch AI created a taxonomy of the tasks involved in AI research. In the simplest model, AI researchers create hypotheses, design experiments, run the experiments, analyse the results, and repeat this cycle.

12

There are many valuable questions which could provide a clearer picture of what it would take, and what it means, to automate this:

• How much time do AI researchers spend between hypothesis generation, designing experiments, and analysing results?

• When AI researchers reflect on their own cognition while generating hypotheses, what kinds of reasoning are they doing?

• How much time do they spend waiting for the results of experiments?

• (I could go on…)

There are very few public resources which deal with automating AI research at frontier labs, but more specific materials would make predictions of transformative change much easier. For most of this analysis, we rest on an enormously valuable interview on the Dwarkesh Podcast with Sholto Douglas (a Google DeepMind researcher) and Trenton Bricken (an Anthropic researcher).

Current systems are getting better at ML engineering, but performance struggles over longer horizons…

How do state-of-the-art models perform on our current tests of AI R&D?

We develop tests of AI research, or benchmarks, which can give us a smooth function of how much progress we are making towards the capability. Good benchmarks give you a score out of 100 on a diverse range of tests that most closely mirrors the capability in the real world. There are three main benchmarks which test a model’s AI Research abilities13: OpenAI’s MLE-bench; METR’s (a non-profit model evaluator) RE-bench; and OpenAI’s SWE-bench Verified.

MLE-bench tests the models against 75 ML engineering questions from online competitions (Kaggle). The latest public scores on this benchmark are for o1-preview, which lags o1 and o3. O1-preview performed in the top 40% of humans who had completed these ML engineering tasks on 16.9% of occasions. We should expect o3 to perform significantly better on this benchmark.

RE-bench tests the models against seven ML engineering tasks relevant for frontier R&D.14 In the evaluation, o–1 preview and Claude 3.5 Sonnet outperformed human experts with a two-hour time budget, but model performance asymptotes while human performance continues to rise with additional hours.

15

It is noteworthy that, on the task of optimising a kernel for latency, one of the models was able to find a solution with lower latency than the best solution from any human researcher in the benchmarking.16 As with MLE-bench, we expect that o3 performs significantly better than o1-preview. To provide an indicator to the trajectory of progress, Sholto Douglas tweeted that within six months, they expect state-of-the-art models to outperform human researchers with a time-budget of four hours.

SWE-bench verified tests model performance on real-world software engineering tasks. These do not reflect the models’ ability to do ML research, but provide a more general indicator of their coding ability. OpenAI’s o3 release in December 2024 took state-of-the-art performance to 71.7%. We predict this benchmark will be saturated (i.e. so close to 100% that it no longer useful distinguishes between models) in 3 to 6 months.

Finally, it is notable that ARC-AGI-PUB—a benchmark of models’ visual reasoning abilities—has become saturated. It measures performance on a series of visual reasoning puzzles (a bit like non-verbal reasoning tests, for those who went to school in the UK). The AI models have historically struggled with these problems, but humans would find them trivially easy to solve. GPT-4o, released in May 2024, scored just 5% on the benchmark; o1-preview scored 13.3%, but now, o3 is able to score 88%.

At this point, it is useful to reflect on what it would mean for all these benchmarks to become saturated. These are necessarily imperfect snapshots of what it is to do AI research. These tasks do not map comprehensively to Epoch AI’s taxonomy above. As one of the creators of RE-bench notes, their benchmark does not capture the models’ ability to interact with large and messy codebases, and make compute allocation decisions. However, these benchmarks provide a guide for the rate at which models are becoming better at ML engineering.

Improving these capabilities technically will depend on training systems for better long-horizon task performance. For more detailed coverage, see these pieces in Inference.

Partial automation is unlikely to provide much productivity uplift to AI research.

As we have mentioned earlier, the relevant question to consider for automation is: how much does automating this step actually impact output?

In this case, we want to know how much research output is increased by an agent (“the proto-AI researcher”) which is:

• Less good than human researchers at generating hypotheses;

• But can do software or machine learning engineering faster than humans, at or above the level of human research engineers.

• And it can create visualisations of research results with preliminary analysis, to present to human researchers.

We do not expect this “proto-AI researcher” to increase output much for the following reasons.

We strongly expect that the output of AI research labs is bottlenecked by compute. In the Dwarkesh interview, Sholto Douglas has said, “I think the Gemini program would probably be maybe five times faster with 10 times more compute or something like that.” It is notable also that, according to Situational Awareness (as of May 2024), “GDM is rumoured to have way more experimental compute than OpenAI”. Perhaps the returns to marginal experimental compute are even more dramatic at other AI labs.

But there is a more simple ‘outside view’ argument, for why we should expect AI research labs to be compute bottlenecked. Put yourself in the shoes of a Chief Scientist—if you aren’t saturating your experimental compute, you should be trying extremely hard to!

Maintaining experimental compute clusters cost the AI labs billions of dollars. In comparison, AI researcher salaries are just a few hundred thousand, to a few million dollars a year. If your researchers don’t have enough ideas to saturate the compute you have, you should hire more researchers! If your best researchers have too many ideas without the time to implement them, you should hire more research engineers to help them do this! Being in a regime where you aren’t constrained by compute means the bottleneck is something else, which are worse problems to have.17

Using the proto-AI researcher for implementing experiments would require researchers to change their workflows, which could limit, or even reduce, their research output. Researchers have spent their entire working lives honing their research process, and experimental design is hard. If their AI lab asked their researchers to switch to a new workflow for implementing experiments where they have to pre-specify their view of how to do an experiment, in natural language, this could dampen their creativity, or they could spend time correcting models which did not implement the experiment as well as they would have. For some researchers, writing the experiment is thinking about the experiment. Breaking up this flow could limit, or even negate, the benefits of faster implementation.

However, this view of limited progress could be wrong for a few reasons.

The proto-AI researcher could partially automate benchmark creation. It might sound weird to outsiders, but one of the places where AI labs are most bottlenecked right now is after an experiment, how do they work out if the change they made actually improved the model? They use benchmarks, like the ones we’ve already discussed, but finding good tests has been getting increasingly hard. Zhengdong Wang, a Google DeepMind researcher, has an excellent section in his excellent end of year letter, on the problem of working on a poorly specified goal like ‘make this model generally intelligent’:

But how does [the researcher] know which experiment is better? In the past, evaluation was easy because the desired result was clear. If one model won more games of chess, or predicted a protein structure with higher accuracy, then it was better. Today, “better” is vaguer and slower to get than ever before. Our researcher can interact with a model for a long time, or look at which model users like more when he deploys it. But to do effective research, he needs fast (read: automated) evaluations. So he resorts to a test or benchmark (colloquially, an “eval”) that is unambiguous enough, fast enough, and a good enough approximation of what he means by “better.” Concretely, sipping his coffee, our researcher is looking at a plot where training progress is on the x-axis, and performance on a test is on the y-axis. He wants performance to go up as training progresses.

…

In fact, you might even say that the only time AI researchers are doing AI research is when they choose the evaluation. The rest of the time, they’re just optimizing a number.

This matches with a lot of what we’ve heard from people at the labs—the multiple choice questions it is possible to generate are being saturated, and from here, we will need benchmarks of longer-horizon tasks. These benchmarks will need to provide agents with an environment to act in, a well-defined task that provides a signal of their intelligence, with a smooth function that summarises how well they are performing. (To think about how difficult this is, work backwards: what test can you design to show “this model is 50% ‘good at science’”?) From conversations with people who make benchmarks, they expect it could be possible to automate quite a lot of their work—setting up environments, designing verification tasks for the models, and orchestrating agents at scale. However, they also stressed there are compute constraints on running large-scale long-horizon benchmark tasks, and that models will need even harder tests, automating which might jump ahead of current model capabilities.

In short, if the researchers were able to get clearer and deeper signal about how their models are improving in the domains they care about, as quickly as possible, it could well accelerate iteration speed on the incremental improvements to the models.

The human researchers could be ‘freed up’ to spend more time on other tasks, like thinking about better experiments to run, or reading more literature which could sow the seeds for better ideas in the future, or think more deeply about their experimental results and what might be happening inside the models.

However, we are skeptical, because we suspect that actually writing experiments is a small fraction of the job. Sholto notes,

“People have long lists of ideas that they want to try. Not every idea that you think should work, will work. Trying to understand why that is is quite difficult and working out what exactly you need to do to interrogate it. So a lot of it is introspection about what's going on. It's not pumping out thousands and thousands and thousands of lines of code.”

If implementing ideas for experiments is only a small fraction of time to begin with, speeding this process up doesn’t create much additional time for more thinking about experiments to run.

Complete automation could be bottlenecked by ideation and research taste.

In models of automation, there are a small number of tasks which are the last to be automated. These are known as ‘holdout tasks’. Whether there will be holdout tasks, what these will be, and how long they might hold out for, are important for understanding the output of ‘proto-AI researchers’.

Longer explanations of technical progress have been covered here and here in Inference, but in the briefest manner: the systems which AI labs are training will be trained to perform long-horizon tasks. This requires improving the models’ ability to maintain goal-directedness and coherence (and not to drift off track, as is sometimes observed in weaker models). This also requires error detection and recovery, as weaker models typically get stuck in loops, making the same mistake. As part of this regime, the models are being trained to think for longer, in order to improve their reasoning and planning capabilities.

We are currently in an ‘inference-time compute’ overhang, which means we have the capacity to increase the amount of compute which AI systems are using during inference, for greater capabilities. The relevant question, for our purposes on the complete automation of AI research; is where does the overhang end?

It could be the case that we have all the relevant components of creating an AI researcher within this current overhang. Perhaps all that it takes to make AI researchers with better ideas than human researchers is to scale up the models’ ability to think for a long time, and give them good examples of human researchers research ideas for a given set of evidence. On the other hand, there could be some cognitive tasks which the current overhang is unable to capture, and so output remains bottlenecked by these. For example, perhaps the lead researchers who set the research direction of the lab, and have to set plans for an extended period are engaging in a type of reasoning which is inaccessible; or perhaps the digital AI researchers are unable to reach the reliability at generating good ideas of human researchers.

For the view that it is possible within the paradigm; see ‘AGI is an engineering problem’.

It is unlikely there will be a discreet moment when we ‘have’ the AI researcher. We expect it to emerge over time.

When the models’ ideas for experiments are 90%-as-good-as the best human researchers’, they will be used 0% of the time. But once the models’ ideas can sometimes generate ideas 105%-as-good-as the best human researchers’, the human researcher should notice and implement them. Despite this, knowing in practice when the model’s ideas are better than a researchers’ seems to be particularly difficult. AI labs could not gamble on automating AI research prematurely, only to discover their agents’ ideas are worse than human researchers at a competing AI lab. Discovering the models are better at thinking of ideas is likely to be a gradual process—when their ideas are roughly as good as the best researchers', deciding which side of the distribution around 100% they fall will be very difficult for the researchers.

To what extent can AI labs maintain an experimental compute budget?

Experimental compute is central to our narrative. If it can have such dramatic effects on research output — that 10 times more compute means 5 times more progress — then sustaining as much compute as possible is vital for all research labs. Whoever has the most compute might even be the decisive factor, for who reaches the AI researcher first.

In the headline there are enormous $ figures for big tech companies spending on AI infrastructure — just last week, Microsoft announced that it would spend $80 billion on building new datacentres in 2025 — but most of this will be to run inference for customers through their product suite or cloud, and not for experimental or training compute for AGI labs.

The economics of compute for R&D are different from the economics of serving models to customers.

When GPUs are used for serving customers, the goal is to generate as much surplus as possible. Because GPUs are very expensive — electricity is only ~10-15% of the total cost of ownership — you do not want them to be idle. If you want to have the biggest surplus, you'll need to run a) as many GPUs as possible, at b) as high utilisation as possible, whilst c) providing all customers with a suitable level of interactivity. This is extremely difficult — how do you split the model across multiple GPUs to tradeoff throughput and interactivity? How do you forecast the number of GPUs you'll want in 4 years' time (the horizon for making AI infrastructure decisions)? How do you know the kind of hardware you will need to run the models of 2028? To what extent will we make inference efficiency gains, so that demand for inference can be satisfied with a much smaller number of GPUs than it would take today?

On the other hand, R&D compute is about spending the surplus. The goal of R&D is to create new models, which will maximise your future surplus:

• Either because the models are more widely useful, and so you can sell more tokens;

• Or because they are differentially capable, so you can charge more relative to the cost of inference;

• Or because they are more efficient to run for a given capability level, so you take more home as surplus.

If R&D compute is about spending the surplus you’ve generated, then your total R&D compute needs to be amortised over all your inference.

The total amount which needs to be amortised is rising over time. The table below from the Institute for Progress shows the growth in training compute over time. The computational power dedicated to the largest training runs will be 100 times as large in 2030, as in 2026. (Note that while pre-training scaling laws might well be slowing, more compute can be applied during post training. SemiAnalysis predicts post-training FLOP will exceed pre-training FLOP in future.)

18

The total cost of ownership for an H200 is roughly $10.5k per month.19 (edit: correction, this previously said H100, but should have said H200; the TCO of an H100 is ~$9k/p.m.) For a 100k cluster, the annual cost will be roughly $1.6 billion. For the median 2028 cluster, it will be roughly $8.9 billion annually for ownership (note this is not capex!). On top of this, add experimental compute. All algorithmic improvements need to be tried at multiple increments of scale, and so the training compute will need to be at least close to the largest cluster. We estimate, with low confidence, that all experimental compute, as well as for evals and for safety research might be the same as the training cluster. And so R&D compute in 2028 might be a $15 billion to $20 billion expense for the AI labs.

Paying for this means selling some tokens!

An important question to consider: what is the economically-useful life of a model? We will argue that…

The economically-useful life of a model is short.

‘Frontier’ capabilities seem to get commoditised quickly, which hurts margins. Thus far, OpenAI have generally released the most powerful capabilities first. But not long after, other AI labs have released similarly powerful models.

GPT-4 was released in March 2023, although it finished training in August 2022, and just four months after its release, Anthropic released Claude 2 and Meta released Llama 2. To what extent the capabilities of GPT-4 were commoditised at this point is debatable: Llama 3 had a markedly worse HumanEval score (a test of coding ability), but the point is that directionally, in just 4 months, the competitive differentiation of GPT-4 was diminished.

GPT-4 Turbo was released in November 2023, and by March 2024, Anthropic released Claude 3, xAI released Grok-1, and then in April, Meta released Llama 3. All of these models had roughly equivalent benchmark scores.

Finally, OpenAI released GPT-4o in May 2024; Anthropic followed in June with Claude 3.5 Sonnet, and Meta followed in July with Llama 3.1.

When the leading model is clearly differentiated, the AI lab who made it will be able to make excess profits; but when these capabilities are commoditised, their margin is competed away. The less margin there is, the more difficult it is to amortise the cost of training new models (and the more one depends on the size of one's customer base).

However, commoditisation of capabilities could end. In the previous paradigm, when there was only a single axis for improvement (base model scale) there was natural convergence towards similar levels of capabilities. In the new paradigm of inference-time scaling, the 'returns to ideas' rise—first, everyone needs to make the leap to follow OpenAI in being able to scalably apply more compute at inference time, and second, there are many different types of RL which could have this affect. If the labs are decorrelated in their approaches, it is plausible to imagine that their models could have more heterogeneous capabilities. Additionally, more advanced post-training techniques offer the potential for more advanced ‘personality’ elicitation from the systems: people generally seem to prefer Claude 3.5 Sonnet’s style, tone of voice, and writing ability over other models. On the other hand, even if the techniques labs choose are different, they will want to train their models towards the same tasks—being good at coding, being able to complete tasks on a computer—and so even with different research approaches, they can end up in the same place.

Irrespective of whether the software layer commoditises, hardware capabilities will take much longer to compete away.

Once Google DeepMind is able to unlock Chain of Thought models, we expect they will have strong cost advantages for running large models on TPUs, against others running on GPUs. The TPUv6’s operate in a pod of 256 other chips, while NVIDIA Hoppers, B100, and B200 are only able to maintain a pod of 8 chips. Even the next-generation GB200 only has a pod size of 72. Larger pods make more parallelism schemes possible (i.e. you can split the model across a wider number of TPUs, with more creative configurations).

(As a technical detail: CoT costs do not scale linearly, so as sequence lengths at inference get longer, the problem gets worse.)

Switching models is easy, which means margins are more competitive. Moving between model providers is as simple as editing a line of code to change the API call. From the conversations we’ve had, if your scaffolding is built correctly, changing the base model does not cause this to break. Perhaps this changes, as model providers build developer tools and add parts of the bundle to keep you in their system, but it is at least not true for now.

At present, there is no market for ‘non-frontier’ models. AI systems do not seem to have reached the efficient frontier of latency, cost or performance—there’s so much further to go. It also is worth mentioning that typically, how the smaller models with lower latency and cost are created is distilling the larger model (a la o1-mini is to o1) rather than having a different training process, or 'falling off' the frontier.

Inference efficiency gains mean that prices per token fall, so R&D compute has to be amortised over a larger number of tokens.

Over the past two years, token prices for GPT-4 series models have collapsed by roughly 240x, per the chart from Elad Gil below. What is driving this? We would guess that it is not entirely inference optimisations. Perhaps in the early months of GPT-4, OpenAI had limited compute resources for serving models, so the high token prices were a form of rationing. However, this provides some directional indicator for the kinds of inference gains made over the period. Perhaps it is an extremely obvious point, but if token prices fall by 240 times, whilst margins hold constant, then making the same amount of revenue at the end of the period would require selling 240 tokens for every token you sold at the start of the period.

20

To step back from the analysis of inference costs, what these parameters have established in aggregate, is that each model has a window of opportunity to make a surplus that can be used to pay for R&D for future models. These windows seem to be short, with limited opportunity for margin throughout, and amortisation getting more difficult over time. Superintelligence seems like a bad business.

The labs will need to make products, not sell tokens, to fund R&D.

In order for this to work, AI labs need to get out of the token business, and start selling automated tasks or agents which can be priced in terms of their labour-equivalent.21 More on the impact of AI systems on cognitive labour in a later section.

Allocating the experimental compute budget is difficult.

Most visions of fast AI progress, in our opinion, assume away the problem of compute budgets. In practice, experimental compute is hard-won; and to what extent computational power can be dedicated towards particular goals determines a great deal about how quickly they are achieved.

The amount of experimental compute that AI labs have places limits on the size of their human AI researchers teams.

Even without the digital AI researcher, if an AI lab was going to hire the marginal human research, they would need to divide compute between ‘n + 1’ researchers. This means that some fraction of the compute from your best researchers will be reassigned to the new researcher. Because there are extremely high returns to research talent, and compounding benefits to intuition and research taste from having run lots of experiments; it is quite likely that it does not make sense to add more human researchers at all. In fact, labs might want to centralise lots of compute behind a very small number of very talented research scientists, and have the great majority of people improving their efficiency; a bit like a Megazord from Power Rangers. For an AI researcher to get involved in the research pipeline, their ideas have to surpass what the best researchers could otherwise spend the compute on.

Instances of the digital AI researcher will not be run on the cluster they were trained on.

The arguments for the Intelligence Explosion are premised on the idea that we will run the instances of the AI researcher on the cluster they were trained on. This does not seem to match how we’d imagine AI lab compute budgets work, nor how it would be optimal for them to work. For starters, where would the next model be trained? We would expect that the compute budget of an AI lab looks something like this: ~60% of compute on serving customers’ models; ~20% on training compute for the biggest run (~10% on pre-training, ~10% on post-training), ~10% on experimental compute, and ~10% on ‘other things’ (e.g. synthetic data generation, safety research, evaluations). Therefore, if the labs wants to run instances of the AI scientist, these will need to be traded off against experimental compute. The question then becomes: what is the marginal use of this compute? The next-best experiments of the best human researchers, or instances of the AI scientist to think about what experiments to run?

In this view, the digital AI researcher's ideas would need to generate much better ideas than those of the human researchers, otherwise it would not make sense to use up the compute.

The early inference costs of the AI scientist are likely to be very high, though we should expect them to fall dramatically.

The narratives for the Intelligence Explosion rely upon inference costs for the AI researcher being trivially low. Recent examples of Chain of Thought models have shown them using enormous amounts of inference compute—if it cost $3400 on average to solve each ARC-AGI benchmark prize, and we use the token prices of o1-preview, it took roughly 128,000 tokens to solve problems which are trivially easy for humans to solve.

A quick technical detail: recall that inference costs of CoT models do not scale linearly, as the model is forced to parallelise across multiple pods.

If we are to get additional capabilities through scaled Chain of Thought, this description of the AI researchers in Situational Awareness would seem to have astronomical inference costs…

“[T]hey’ll [each of the 100 million AI researcher] be able to get incredible ML intuition (having internalized the whole ML literature and every previous experiment every run!) and centuries-equivalent of thinking-time to figure out exactly the right experiment to run, configure it optimally, and get the maximum value of information; they’ll be able to spend centuries-equivalent of engineer-time before running even tiny experiments to avoid bugs and get them right on the first try; they can make tradeoffs to economize on compute by focusing on the biggest wins; and they’ll be able to try tons of smaller-scale experiments (and given effective compute scaleups by then, “smaller-scale” means being able to train 100,000 GPT-4-level models in a year to try architecture breakthroughs).”

It also seems to endow the AI researcher with an unfounded omnipotence. One way to describe GPT-3 is “imagine a model trained by reading billions of words of internet text, and across books, and Wikipedia; it will have superhuman intuitions about all kinds of different topics”, though this is clearly incorrect. The GPT-3 learning algorithm clearly isn’t that sample efficient nor is the degree of generalisation as-good as this would imply. Superhuman AI researchers will certainly exist in the future, but to assume these early generations will have this kind of power seems to imply extremely radical improvements to the learning algorithm which do not strike us as easily gained.

The productivity impact of the AI scientist could be capped, if the job of an AI researcher is to make ‘shot calls’ about which experiments need more compute.

One of the challenges of AI research is that experiments at different increments of scale can show very different performance. We’ve had RNNs, CNNs, and LSTMs (different neural network architectures for decades) but we’ve only have the computing power since the late 2000’s / early 2010s to be able to make use of them. In the episode of the Dwarkesh Podcast earlier cited, Sholto comments:

“[Y]ou never actually know if the trend will hold. For certain architectures the trend has held really well. And for certain changes, it's held really well. But that isn't always the case. And things which can help at smaller scales can actually hurt at larger scales. You have to make guesses based on what the trend lines look like and based on your intuitive feeling of what’s actually something that's going to matter, particularly for those which help with the small scale.”

One way to think about what this does to your comment budget is that it divides it into a convergent series (or put another way, a series of Russian Dolls), up to the largest training run. We can do many small-scale experiments, increasing the scale until reaching the largest (and longest) training run. From the episode;

“Many people have a long list of ideas that they want to try, but paring that down and shot calling, under very imperfect information, what are the right ideas to explore further is really hard.”

The biggest decisions the labs will make are ‘shot-calling’ about what goes into the largest scales. I think there are two important questions here:

1. To what extent are the potential gains to better decision making here capped?

2. To what extent are there marginal returns to intelligence for ‘shot calling’?

It seems to us that some of the world’s smartest people, with decades of AI research experience making these calls; and it seems doubtful as to how much more correctly it is possible to make these decisions. We could be quite close to the limits of possible correctness in this task, for which marginal intelligence would not assist much.

However, AI progress could go much more quickly than our picture suggests, if the digital AI scientists are much better at predicting the results of experiments than human researchers, or run at very low inference costs.

Being able to predict the results of an experiment—what in humans we might summarise as ‘intuition’—is immune to compute bottlenecks. If the digital AI researchers are able to get superhuman intuition about which experiments to run, and which aren’t worth running, it could make the total research output rise dramatically, even whilst experimental throughput remains constant.

Furthermore, making a big inference efficiency improvement could dramatically improve the usefulness of AI researchers—a 50% gain would either mean the same ‘population’ can run on half as much compute, with the other half going to experiments, or it is possible to run double the number of AI researchers. We’ve heard conflicting stories about whether this is possible. Some people have suggested that we’ve reached a ‘global minimum for inference costs’, as we will scale Chain of Thought reasoning faster than it is possible to make inference efficiency gains. On the contrary, others have been relatively optimistic about our capacity to make inference gains.

How much can chip supply be scaled, if it needs to?

This depends on the time scale.

The principal consideration here is frontier fab capacity—the H100 is fabricated using TSMC’s specialised process for AI chips called 4P (confusingly at 5nm), and the soon-coming Blackwell chips will use the same.

At the moment, AI accelerators make up a small fraction of 5nm capacity. For an estimate of how much they use, from this FT article, Microsoft bought 485,000 Hoppers — we will assume all H100s for simplicity — in 2024; Meta bought 224,000; Amazon, 196,000; Google, 169,000 (though of course, Google also has TPUs); and although it isn’t included, let’s assume that x.ai bought 125k GPUs. This is just short of 1.2 million H100 GPUs last year. Using the die size and some reasonable assumptions for yield, we can estimate this would have required 16,000 to 20,000 wafers. This is approximately a mere 0.3-0.4% of TSMC’s annual capacity in 5nm and below.22

Remember this is not the complete picture of demand—roughly half of NVIDIAs revenue comes from the hyperscalers, the remaining half from neoclouds, startups, governments and so forth. Amazon, Microsoft, OpenAI are all developing their own custom silicon; and Google is on their 6th generation TPU. AMD is also making the MI325X to compete with NVIDIA’s next generation Blackwell. Furthermore, growth rates are high—NVIDIA’s most recent earnings reported revenue from their datacentre business (i.e. selling GPUs with all the extras to make them work) at $30.8 billion. A year earlier, this was $14.51 billion; and a year before that it was $2.94 billion. In their Q1 2024 earnings call, TSMC gave guidance that they expect their AI processor business to grow at 50% per year, and become more than 20% of their revenue by 2028.

The production of AI accelerators could use capacity at 3nm and 4nm, but prices would need to rise, to outbid 3nm and 4nm demand, as there are higher costs for TSMC at the leading-edge.

There are potential bottlenecks in advanced packaging (CoWoS packaging capacity is lower than demand until at least 2026) and in high-bandwidth memory, which limit scaling production in the next 1-2 years, though it is expected these bottlenecks will alleviate.

Building a leading-edge fab (chip factory) takes three to five years, though TSMC’s Arizona facility is delayed until 2027 or 2028 (construction began in December 2022) reportedly due to a combination of low demand and uncertainty regarding US subsidies.

So, in short, there is a lot of room to grow within the existing fab capacity, though for the next couple of years this is limited by HBM and advanced packaging capacity. New fab construction requires significant capital expenditure, and therefore certainty of demand. New fabs could be commissioned to meet AI-related demand in future, but current projected demand in the 2020s remains far off this being the case.

We think the capacity to scale chip production is one of the most important inputs into the rate of progress, and so we will be dedicating a full piece to it, in the next edition of Inference.

Expect big improvements in human welfare from AI automating science, but don’t expect that these gains will come quickly.

Most of the improvements to human wellbeing in ‘frontier’ economies comes from making scientific discoveries and turning these into new technologies. When AI lab leaders speak about the opportunity of AI, it is principally the scientific opportunity which they see as most exciting. Our scientific ambitions for AI should be enormous: ending disease, extending life, making abundant clean energy, more performant and green materials, ending unpleasant labour through robotic automation. We will also come to better understand human minds, wellbeing, and the most fundamental scientific questions. Like with AI research, automating R&D depends on automating hypothesis generation, experimental design and implementation, and data analysis. Also like AI research, experimental throughput constraints scientific progress.

Current AI systems can improve human researchers’ hypothesis generation.

The literature review process can be greatly enhanced with AI. FutureHouse, a non-profit research organisation, has built PaperQA2, a literature review agent. Against a test of questions, where the answer was to be found only in the body of a single scientific paper, this agent was able to correctly answer 60.3% of questions. It was able to write cited, Wikipedia-style summaries which experts ranked as more accurate than the existing human-written Wikipedia articles in some scientific domains. It is notable that this agent was released in September 2024, before o1 or o3 were released, which will perform much better on GPQA (a benchmark of scientific expertise). It seems reasonable to imagine that by the middle of 2025, it will be possible for AI systems to write a literature review to postgrad-level.

Another example of the possibilities comes from Professor Derya Unutmaz, at the Jackson Laboratory, who studies cancer immunotherapy who prompted the model to support with experimental design, and subsequently wrote:

"While o1-Preview and GPT-4o were able to generate some interesting ideas based on this concept, but they were mostly what I could also conceive though better [than] most PhD students. In contrast, o1-Pro came up with far more creative and innovative solutions that left me in awe!"

Here is the full output.

As an aside, language models can easily automate grant applications. This is quite a trivial use for AI systems now, clearly not using the frontier of their capabilities—but the Faculty Workload Survey of 2018 from the USFDP which received responses from 11,167 Principal Investigators said administrative requirements took 44.3% of their time. If we automated grant applications, would scientific output double as a result of the saved time? On the other hand, decreasing the cost of information processing might lead scientific funding institutions to impose greater reporting requirements, eating away any potential gains.23

Notably, all of these automations are pretty easy! They do not require large capital expenditures and scientists can choose to integrate these tools ‘bottom up’, as 1/3rd of postdocs already have.

Academia is poorly configured to adopt AI.

• Because automation will happen lab-by-lab, there are low returns to scale.

• Because academic salaries are lower than other equivalent levels of labour skill, there are weaker incentives to automate.

• There are high capital costs to robotic lab automation, for which the returns are uncertain.

• Incentive mechanisms in academia, for example, towards grant funding for longer periods, and positions being tied to projects, could make it harder to pivot research to focus on new capabilities.

However, we can create new AI-first scientific institutions through structures like FutureHouse, Convergent Research’s FRO, or ARIA. One parallel to think about this is how electricity improved productivity, not just in electrifying processes, but in making the assembly line possible—in what ways might we completely reorganize the institutions of science (perhaps around models, datasets, or experimental automation) that allows us to capture the full upside? Perhaps these things have to be reimagined from the ground up.

Scientific automation faces many intrinsic headwinds.

• Most sciences require ‘real world’ experiments whereas AI research only needs experiments within the computer. It is possible that progress in AI research will result in improvements to improve our simulations—of the cell, of particle physics, and so forth; that can partially substitute for experimentation, but there will be limits to this.

• There is no ‘programming language’ for recording experimental design, in the same way that in AI research the experiments are recorded exactly as they were implemented in computer memory. This means that training agents to do AI research is much easier, because the AI labs will have a very rich corpus of data on which to train, whereas for agents to plan biomedical researcher experiments, we lack such a corpus and instead train on non-standardised descriptions of methods in academic papers.

• A very small fraction of the total data which research labs could collect, is collected. Organising data collection otherwise would impose big constraints on the scientists’ productivity. On the contrary, computers capture AI research by design
  
  This property of experimental research means that AI labs will have a corpus of negative results that would never be published, whereas in biomedical sciences, these are not published and are unlikely to be recorded in a structured format.

Chip R&D is especially susceptible to these challenges described.

• The processes for chip production, as well as for R&D are highly secretive. Only TSMC will be able to automate what they do directly.

• Knowledge about how to do chip research is often tacit and master-apprentice.

• The complexity is so incredibly high—there are so many steps in the process, many of them are multivariate problems, and nobody has good visibility of the process.

• As mentioned above, lots of chip research requires large amounts of real-world experimental throughput.

However, this argument could be incorrect in a couple of important ways.

First, there are some steps in the chip R&D process which have enormous leverage over other steps. Ultimately a large fraction of the performance improvement comes down to whether it is possible to shrink the node size—between Hopper and Blackwell it wasn’t, and quite a lot of the additional performance of Blackwell comes from how it was possible to make the chip bigger. (Granted this is still an important jump forward!)

DeepMind have developed AlphaChip, which has partially automated aspects of chip floorplanning (arranging where components go on the chip). This has reduced the wirelength, important for communication speed, on the TPUv6 by 6.2%; but is somewhat bounded in its capacity to produce more computing power. The important question to ask about the chip process is: where are the marginal returns to intelligence very high?

Second, there can be unintuitive substitutes in R&D. If you had asked us to explain why the discovery of protein structure does not get automated in 2014, we would plausibly explained the difficulty of automating the process of x-ray crystallography — it’s hard to do the purification, to grow the crystals and so forth. Of course, we would have been wrong! Not in the direct sense that getting robots to x ray-crystallography would be easy, but that it turned out with a sufficiently large dataset of previous examples of x-ray crystallography and some hard-coded understanding of bond angles, we can create an AI system — AlphaFold — which is able to completely bypass this. Perhaps extremely intelligent systems will be able to spot much more difficult ‘bypasses’ and take advantage of them.

Biomedical advances will be bottlenecked by experimental throughput, and social welfare improvements will be bottlenecked by regulatory approval.

Dario Amodei’s essay, Machines of Loving Grace, details the changes which he thinks could arise from ‘powerful AI’, the model which results from some period (in his view, near), after the AI scientist where we have ‘a country of geniuses in a datacentre’. This model can control lab robots or tell humans which experiments to run. He thinks that we might be able to increase the speed of biomedical research progress by 10 times, which would mean in 5 to 10 years, we can make progress like:

• Reliable prevention and treatment of all infectious diseases

• Elimination of most cancer

• Very effective prevention and effective cures for genetic disease.

• Prevention of Alzheimer’s

• Improved treatments for most other ailments (diabetes, heart disease, autoimmune diseases and more.

• Biological freedom (improvement to birth control, fertility, management of weight etc)

• Doubling of the human life span through therapeutics.

What would it take to increase experimental throughput by a factor of 10?

There are roughly 146,000 medical scientists employed in the United States. This would mean that 1.4 million people would be needed (or perhaps slightly fewer, as perhaps the job is more focused on experiments) to increase our throughput by a factor of 10, if the quality of experimental ideas is held constant. These people would need new buildings to work in, which typically take a year or two to build; and there would be many other things that have to happen like training the scientists at least in basic procedures, in order to get this scale-up.

On the other hand, we might not need a direct ten times increase in experimental throughout, perhaps if the models improve the quality of our ideas and the data we’re able to generate from each experiment, we need fewer experiments, so fewer buildings and so forth.

There is also the question of regulatory approval for new therapeutics. In this regard, Dario is also optimistic. He writes:

Although there is a lot of bureaucracy and slowdown associated with them, the truth is that a lot (though by no means all!) of their slowness ultimately derives from the need to rigorously evaluate drugs that barely work or ambiguously work. This is sadly true of most therapies today: the average cancer drug increases survival by a few months while having significant side effects that need to be carefully measured (there’s a similar story for Alzheimer’s drugs). This leads to huge studies (in order to achieve statistical power) and difficult tradeoffs which regulatory agencies generally aren’t great at making, again because of bureaucracy and the complexity of competing interests.

When something works really well, it goes much faster: there’s an accelerated approval track and the ease of approval is much greater when effect sizes are larger. mRNA vaccines for COVID were approved in 9 months—much faster than the usual pace. That said, even under these conditions clinical trials are still too slow—mRNA vaccines arguably should have been approved in ~2 months. But these kinds of delays (~1 year end-to-end for a drug) combined with massive parallelization and the need for some but not too much iteration (“a few tries”) are very compatible with radical transformation in 5-10 years. Even more optimistically, it is possible that AI-enabled biological science will reduce the need for iteration in clinical trials by developing better animal and cell experimental models (or even simulations) that are more accurate in predicting what will happen in humans. This will be particularly important in developing drugs against the aging process, which plays out over decades and where we need a faster iteration loop.

We believe this understates the extent to which getting drugs through the regulatory approval process is bottlenecked by more capable systems. Semaglutide seems like the kind of magnitude of discovery we would hope that AI systems are able to produce for us, but the patents were first filed in 2008, and only within the last 5 years have we begun to see its scale up. Similarly, the Malaria vaccine needed 23 years in clinical trials.

Getting regulatory approval bodies to use AI systems for therapeutic approval seems difficult, and unlikely. Their incentives are towards risk-avoidance, they would need to trust the evaluations from the simulations, and re-configure their processes in order to integrate these systems. (It seems more likely that a new, parallel regulatory agency would be set up to have AI analysis capabilities built natively into their approval process.) Additionally, this change would require the public to trust new methods of approval.

Robotics progress will be accelerated by the automated AI researcher. However, we are sceptical of the most aggressive models of robotics deployment.

Without making progress in robotics, the AI systems remain stuck on computers. There’s still a lot of leverage for systems which aren’t physically embodied, as we will discuss, there are a large number of tasks which can be done ‘remotely’. However, without making progress in robotics, further growth would become bottlenecked on manual labour from humans—it would be a Baumol effect. The cost of information processing would fall, and so physical tasks would rise as a relative fraction of the economy.

Robotics progress is bottlenecked on data for them to learn how to act. OpenAI closed down their robotics work in 2021, in order to focus on language modelling, where there was a lot more data because of the Internet! (Note that OpenAI’s robotics team restarted a few days ago.) With the digital AI researcher, there will be two advantages to robotics progress: first, progress in video modelling—improving upon OpenAI’s work on SORA and DeepMind’s work on Genie and veo—could provide much better simulations of the real world, to train the model. The AI researcher could apply general AI algorithm improvements from training better computer-based systems, and also make progress on breakthroughs in computer vision algorithms, specifically for the robots.

There are a number of related questions for how fast we might see general robotics make a larger contribution to the economy:

• How readily can the digital AI researcher generalise to become a robotics researcher?

• How quickly can algorithms for robots be improved, so as they can match human level performance?

• How much of their training can happen in simulation instead of the real world?

• How much prototyping and testing needs to happen on the hardware? How much of this testing can happen in simulation vs in the real world?

• What are the holdout tasks — say, like getting a robotic hand to match human dexterity — which are going to prevent complete automation for some period?

• How long do the holdout tasks hold out?

• Where does manufacturing happen, to what extent does this require skilled humans for assembly? Can the number of humans with these skills be scaled, until the robots can handle their own assembly?

• How much training data can robots provide for models, and how useful?

We will be dedicating a full piece to the robotics scale-up, as it seems to be one of the most important components of the growth story for the coming decades, meriting further investigation.

Cognitive labour will be automated before physical labour, and could be automated much more quickly than previous technological revolutions.

State-of-the-art AI systems have not made a very large GDP-level impact yet.24 This might seem bizarre—imagine going back to 2017 and showing someone the capabilities of OpenAI’s o3 model (very good software engineering, graduate-level scientist, and, from other models, near-perfect test scores in undergrad humanities). Surely one would have expected some noticeable increase to our collective output! It would be reasonable to have imagined big changes to ‘knowledge work’ already.

There are a number of factors that prevent AI systems performing tasks they are cognitively capable of doing.

Throughout this section, it is worthwhile to reflect on whether these constraints continue to be true as the models become more capable. In some cases, the challenges to deployment fall away as models get more capable.

Right now, implementing current AI systems requires building ‘scaffolding’ to improve reliability and performance. Most AI deployment in the real world has been workflows; where large language models and tools (like a code window, or internet access) are orchestrated through predefined code paths. These paths are known as scaffolding.

This happens because, until recently, language models have not been trained to perform tasks over time, just to predict the next token. To get them to do tasks, it would be necessary to stitch many prompts to the model together. Furthermore, businesses like deterministic processes. With scaffolding, the models are kept ‘on track’ and their performance improves.

Knowing how to build scaffoldings and deploy AI workflows is currently a rare, and valuable skill. However, we expect this to get much easier with greater intelligence. One way to think about the current situation is that the ‘intelligence’ is balanced between being inside the model’s parameters, and encoded in the ‘scaffolding’ of business logic. Over time, as the models get smarter, a greater share of the processing will happen inside the models and less needs to be encoded in the scaffolding. This will happen as the models are trained for agency, and other algorithmic improvements advance their ability to act in unfamiliar contexts.

Right now, interacting with the models feels like talking to someone who is very smart, but has no context at all. Over time, as their context length, and management of their context window and memory improves, the models will be able to understand environments (like the internal documents of a company, or a large codebase) much more easily.

In the limit, we should expect that orchestrating agents will be done by other agents. The final step of OpenAI’s AGI research agenda is ‘coordination’—getting different AI agents to work together on problems. As part of this, one agent will be able to create a smaller ‘sub-agent’ which could be specialised to a particular task.

Right now, implementing current AI systems requires businesses to reconfigure their processes. State-of-the-art models are capable in some domains, but they are ‘unbalanced’ and struggle at some tasks which are very easy for humans. Model deployers need to account for this, and engineer the scaffolding for workflows, to allow humans to complete tasks which models struggle with. As it has been put previously, ‘there are no AI-shaped holes lying around’.

Over time, the models will need less structure to fit into organisations. The AI labs will improve the interactivity of the models—it will be possible to interact through many modalities in a much more natural way, and proactivity will be trained into the models during agency training. This is the opposite of what happens when we interact with Chatbots—they are trying to say less, to save on generation costs!

However, in many contexts, the deployment of AI systems is bottlenecked by organisational politics, and not the capabilities of the models. Nabeel Qureshi has an excellent essay reflecting on his experience working at Palantir, in which he writes:

“[O]ften what really gets in the way is organizational politics: a team, or group, controls a key data source, the reason for their existence is that they are the gatekeepers to that data source, and they typically justify their existence in a corporation by being the gatekeepers of that data source (and, often, providing analyses of that data). This politics can be a formidable obstacle to overcome, and in some cases led to hilarious outcomes – you’d have a company buying an 8-12 week pilot, and we’d spend all 8-12 weeks just getting data access, and the final week scrambling to have something to demo.”

Current systems might speed up the final week of production, but don’t affect the first 11 weeks of this project!

As in this case, in some domains, customers will have a preference for interacting with a human. There is some preliminary evidence that o1-preview surpasses general practitioners on the reasoning aspects of diagnostics. The paper gave o1-preview summarised descriptions of symptoms in text boxes, and asked it to provide a diagnosis. Of course, this only captures a very small fraction of what it means to be a doctor! Doctors are performing a blend of tasks—asking questions to establish the symptoms, expressing care and empathy, tailoring their explanations to the patient so they can understand, and making diagnoses. While it is highly likely that AI systems will surpass human capabilities to make diagnoses, most people will retain a preference for experiencing healthcare in person rather than through an online interface.25 The tasks which make up the job will be changed, and the interpersonal factors will increase in relative importance.

Furthermore, labour organisations may to resist automation. This paper used semantic analysis of patent filings to predict where on the skill distribution we should expect AI systems to have the first impact. It predicts ‘upper-middle’ jobs, like being a doctor, lawyer, or software engineer will be most exposed to automation from AI. As Webb notes in this podcast interview, doctors and lawyers are likely to be able to lobby for regulations which require a human to remain ‘in the loop’ in places they could otherwise be automated. On the other hand, intersectoral mobility for jobs which typically happen in cities and in ‘knowledge work’ is typically easier than, say, adjustment for coal miners who lived in places where coal mining was the only industry. Furthermore, there are typically lower levels of labour organisation in ‘knowledge’ work, and so we should perhaps expect less coordinated action.

Both factors, the preference for human-produced services and labour organisations, could be diminished by more capable systems. In some domains, by far-and-away superhuman AI systems would incentivise people to switch from human-produced services (imagine AI produced films vs human-produced films) and would make it ever-more difficult for labour organisations to ignore the differences in performance between the systems (imagine noticing rare conditions earlier and more accurately than human doctors could ever manage). However, there are some places where human services augmented by AI will always be superior to just AI; and there will be some responses to automation that are difficult to predict (e.g. on the grounds of safety; autonomous cars currently have 90% fewer crashes than humans over a large sample of miles driven, but have not been deployed more widely). There is also an income effect from higher automation - increasing the demand for human interaction.

At present, humans need to be liable for AI systems. There is no legal framework (as yet!) to determine liability between model developers, ‘product’ companies (which fine tune models on proprietary data), and users of the models. But even if there was, models are not capable of acting unsupervised for long periods, they lack the error-correction, reasoning, and coherence to be reliable. Managing AI systems will become easier over time, as these capabilities are trained into the models, and so plausibly a single human can manage a larger team of agents as they need less frequent input from supervisors. We will also develop better infrastructure for escalating potential errors to AI and human oversight systems, to avoid mistakes. In the limit, AI agents will become capable of running entire firms in some industries; and humans will control a holding company for these firms, taking legal, but not managerial responsibility.

Regulation could prevent AI use and bottleneck deployment and productivity gains. We turn down technology currently, which has detrimental effects both on economic growth and social welfare. For example, this paper estimates that the slowdown in nuclear power construction since Chernobyl, in favour of fossil fuels, has cost 400,000 to 7 million lives globally. Nuclear could also provide electricity at prices competitive with solar, but due to a scientifically inaccurate approach to radiation exposure and poorly configured regulatory apparatus, it has been regulated to near-infeasibility.

However, AI might increase jurisdictional choice for businesses to operate, as they become less dependent on labour, and increasingly dependent on capital (datacentres, robots, robotic labs, factories etc). When this happens, it might cause regulatory approaches to soften, in order to capture some AI growth.

AI systems will be leveraged by humans, mostly not AIs running firms.

What should we expect AI automation to look like, concretely, in the near term?

The first AI agents to be released in the first few months of 2025 will be designed to complete simple tasks on your computer. Over time, these agents will become capable of performing a wider range of tasks on a computer, and acting for longer time-horizons. (For a more complete explanation, see ‘AGI is an engineering problem’ in this edition.) Anything which a human does just on a computer, an AI agent will straightforwardly be able to do in the next few years.

Coding agents will be especially important to track here. Mark Zuckerberg has said that:

“Probably in 2025, we at Meta, as well as the other companies that are basically working on this, are going to have an AI that can effectively be a sort of midlevel engineer that you have at your company that can write code.”

It is quite difficult to measure to what extent the actual productivity of software engineers is improved by this—lines of code does not mean more real-world output, as perhaps we bury ourselves in poorly designed software! However, it seems reasonable to imagine that software productivity could rise quite steeply.

We expect that knowledge workers will be managing groups of agents for computer-based tasks, using tools like Microsoft Copilot Studio, and over time, as the models’ capacity for long-horizon tasks improves, and their reasoning and planning capabilities advance further, the number of agents each human can manage will expand. We expect this to be like: “all humans are getting promoted”, at least in the near term.

Lots of knowledge work bundles cognitive capabilities with ‘embodied’ human skills, for example, product management, consulting, medical care, and law. We think humans will continue to do these jobs, because whilst the cognitive capabilities could be in-principle automated by AI; they are complemented by things which AIs will struggle to do, at least in the near term. Be charismatic, be empathetic, be ‘present’, be accountable. (Perhaps there is a vision for robotics whereby close mimicry of human emotion and embodiment is possible, though we exempt this from our analysis.)

There are a set of knowledge work professions which are not complemented by this kind of embodiment. For example, making good films, making software, and running a hedge fund. It is plausible that humans continue to do some aspect of this (telling the AI system what film to make, what program to write, or raising LP money for the AI running the hedge fund) though we think it is possible to automate these sectors entirely. If an AI hedge fund was likely to make better returns, you would invest with them, irrespective of the hedge fund sales.

We do not expect ‘AI composed firms’ to make up a majority of sectors.

Automating remote tasks could lead to rapid growth, but is very unlikely to lead to explosive growth.

We can build an economic model of tasks in the economy, and estimate which fraction of these can be automated, to predict how much AI causes growth to accelerate, under our assumptions mentioned above. There are three ingredients to any model of this:

1. What fraction of tasks are automated?

2. How well can these substitute for non-automated tasks?

3. How cheap are the AI systems that can substitute for human cognitive labour?

4. How much do Baumol and Engels effects make output growth concentrated in a small number of sectors less valuable?

Below we list our parameter estimates, if you would like to skip to our conclusions you can do so here.

What fraction of tasks are automated?

For the purposes of this, we are going to assume that all tasks which can be done on a laptop are automatable, and all tasks which require physical presence are not automatable in this model.

In the ordinary economy, there are a number of tasks which are not done ‘remotely’, but could be. Whilst most firms prefer their software engineers to be in the office, it is possible to do software engineering remotely.

The pandemic is a useful case study to understand, in the limit, how many jobs could be done remotely if they needed to. This survey suggests that 37% of jobs were done remotely during the pandemic. This is likely to be an overestimate for the number of jobs which are actually automatable, because when people were sent home for lockdown the question was not which jobs are suitably completed remotely, but rather, whether it is possible to get any output from workers who are stuck at home.

To adjust for this, we can break down by task instead of by job. Barnett 2025 uses GPT-4o to analyse ONET, the database of what workers in 1000 professions spend their time doing in the US economy; and finds that 34% of the tasks were automatable. This is quite different from 37% of jobs, because 34% of tasks can be spread across a wide range of workers in a way that could be bundled to affect production. For example, the scientist can write their grant proposal remotely, but if they cannot get access to the lab, their output will still be 0.

Estimating what fraction of tasks have an automatable (i.e. remote) component, bundled with an non-automatable (i.e. in person) component; is very difficult. To return to the earlier example of a Palantir engineer implementing a data platform, the cognitive task to figure out what product a company needs, is possible to do remotely, but it relies upon asking them questions to establish their needs, and building rapport—necessarily not automatable. To deal with this, and set a lower bound on the number of automatable tasks in the economy; we look at what fraction all of their top 5 tasks are remote. Barnett 2025 provides that this is 13%.

How well can automated tasks substitute for non-automated tasks?

Two reasons why task automation might be good

1. Spend less money on it.

2. Because it is possible to do that task more relative to another task, if it can substitute for it. (if you have better software, you can manage your inventory better, which means lower inventory, so lower trolleys for moving things around.)

We want to set a parameter (number between zero and infinity) which captures to what extent an automated task can do this substitution. How can we estimate this?

One of the ways Barnett 2025 estimates this is, again, by looking at the pandemic. At this point, there was a very large increase in the number of remote workers relative to in-person workers. This is an imperfect proxy for what happens when AI systems can be added to the economy quite easily (more remote workers). Total economic output didn’t decline very much—roughly 8%—which is surprisingly small, given that everyone was made to stay at home. This scenario produces an ‘elasticity of substitution’ (the parameter for how much an automated task can substitute for a non-automatable task) of 13.

This is extremely large! However, we don’t place much weight on this for a couple of reasons:

• During the pandemic, roughly 1/3rd of US workers went remote. But, as we’ve already noted, 34% of tasks can be done from home. So all that has happened, roughly, is that these tasks have begun to happen from home. It does not, necessarily, tell us how well this 34% of tasks can substitute for the other 66% of tasks.

• Remote work requires infrastructure. The pandemic prevents investment and maintenance from occurring. In the short run, this is not a big problem (it might even increase output); but in the long-run this problem becomes large. This can cause one to overstate the elasticity of substitution in the pandemic example.

What is another approach to estimating this? Very roughly, we can say that higher skilled workers tend to do cognitive tasks, and lower-skilled workers tend to do physical tasks. (We appreciate this loses a lot of individual difference, but we make a simplification for modelling purposes.) So it is instructive to see to what extent higher skilled workers can substitute for lower skilled workers. This provides an elasticity of 4.

What are some reasons to be suspicious of this? In order for things to have an elasticity greater than one, they need to be substitutes. Simply, what substitutes are, is when you can still produce something, if you only have one of them. We claim that high and low skilled workers are an example of this. High skilled workers can plausibly perform the tasks which lower skilled workers can perform. (We appreciate there will be exceptions, and the idea of ‘lower skill’ can be refuted, but again, this is a simplification for modelling purposes.) We would expect remote and in-person work to not be like this: if everyone was remote, then who would repair the factories, run the power stations, and care for the patients?26 This means high and low skill labour have to give an elasticity of substitution above 1, but this does not mean that remote and in-person work will have the same relationship.

Following Barnett 2025, we’ll use an elasticity of 0.5 as a lower bound.27

How cheap are the AI systems that can substitute for human cognitive labour?

We separately estimate the model at $4, $10, and $40/hour.

$10/hour is a very rough estimate of the cost for o1 tokens, generating for an hour; and $4/hour and $40/hour are ‘a bit above’ and ‘a bit below’. We recognise more precision could be useful here, and also note that this assumes inference costs remain roughly aligned with o1. There are a wide range of views about what will happen to inference costs in the future. As we argued earlier in the section on the slow rate of hardware commoditisation, Chain of Thought inference costs could well rise nonlinearly, although inference efficiency improvements will push in the opposite direction.

How much do more workers help growth?

Barnett 2025 assumes constant returns to scale in adding more workers—simply, twice as many workers leads to twice as much output, forever. The problem is that, in fact, workers need capital to produce things (e.g. machines, computer, tools etc to work with). If there are a lot more workers, without investing to raise the capital stock, the increases in output are determined by the elasticity of substitution between capital and labour. (Stepping back from the economics jargon,) we can either spend resources on adding a marginal remote worker—in this case an AI agent—or we can invest in more machines or tools—to make our existing AI agents (and humans) more productive—or we can spend those resources on consumption—the whole reason we’re doing this anyway!28

At this point, a sensible rebuttal could be, “Don’t labour and capital in this case both mean datacentre compute?” This would be correct in sectors which are completely automated, but we’re interested in what happens after this (these few sectors which are completely automated, by themselves, do not set off explosive growth). For example, doctors can benefit from more AI assistance interpreting scans, but if they want more scans, they’ll also need more MRI machines. Likewise, physicists can do many more simulations of experiments in a datacentre, but ultimately if they want to test them they’ll still need bigger particle accelerators.

Erdil & Besiroglu 2023 uses a literature review of estimates of the capital-labour elasticity of substitution, and this gives 0.45-0.87. However, in the literature review they cite, there is a distortion of the distribution of papers investigating capital-labour elasticity, whereby if people get a negative value, they do not publish their results. A negative value would mean that production would have been higher if there was less labour and less capital, and so people choose not to publish. In the graphs below, which show the distribution of capital-labour elasticities in the studies, there is a clear truncation of the left tail.

29

Therefore, the elasticity of labour-capital substitution is overestimated in Erdil & Besiroglu 2023. Gechert et al 2022 corrects this, and provides a capital-labour elasticity of 0.3.

This implies it is quite difficult to substitute labour for capital, and as such, even with a very large increase in workers, growth is bottlenecked by capital. This capital can be accumulated—although, due to imperfect insurance markets and finitely lived agents, potentially not fully. Getting through this will be slow and expensive.

What does this understanding imply for economic growth

We will now set out three cases for growth. Note that all figures are additional growth over whatever exogenous technological progress occurs.

The maximally optimistic case.

• Elasticity of substitution = 4

• % of tasks automatable = 35

• Agents cost $4/hour

Realistic (upper bound)

• Elasticity of substitution = 2

• % of tasks automatable = 30

• Agents cost $10/hour

Lower bound

• Elasticity of substitution = 0.5

• % of tasks automatable = 20

• Agents cost $40/hour

In the maximally optimistic case, output grows at 19.3% per year on average over the next 20 years, and 16.3% per year over the next 100 years, although we think this is unlikely under purely cognitive automation for the reasons outlined above.

In our upper bound, output grows by 12.2% per year on average over the next 20 years, and 6.8% in the next 100 years.

In our lower bound, output grows by 3.5% per year on average over the next 20 years, and 0.7% per year over the next 100 years.

But, now we need to introduce Baumol effects. If automation is better in some industries than others, say films, finance, and programming; there are diminishing returns to having ever-greater amounts of B2B SaaS in the world, and so as a result, prices will fall in the industries where we are producing lots of goods, and so those industries share of GDP will rise by less than expected. Because of how GDP is constructed, at least half of the growth that would have occurred, less Baumol effects, will occur but this still dampens output by a lot.

But now we need to introduce Engel effects. As automation increases productivity and income, people tend to shift their spending toward sectors like healthcare, education, and housing - areas that historically show slower productivity growth. This phenomenon, known as Engel effects, means that as societies become wealthier, an increasing share of spending goes to these slower-growing sectors. In the US over the past few decades, Engel and Baumol effects have reduced output growth by approximately 25% (not 25pp) compared to what it would be otherwise.

Taking these Engel effects into account, we estimate that economic growth will be 3% to 9% higher per year for the 20 years following significant AI automation. This range reflects both the potential for productivity gains in highly automatable sectors and the dampening effect of spending shifts toward sectors where automation gains may be more limited.

Conclusion

This quote from The Optimistic Thought Experiment on the nature of the dotcom bubble, provides a useful lens for thinking about the economic impact of AI (albeit if the 'doom’ perspective is a little strong).

It is often claimed that the mass delusion reached its peak in March 2000; but what if the opposite also were true, and this was in certain respects a peak of clarity? Perhaps with unprecedented clarity, at the market ’s peak investors and employees could see the farthest: They perceived that in the long run the Old Economy was surely doomed and believed that the New Economy, no matter what the risks, represented the only chance.

The fundamental thesis—that AI research output will be automated; that humanity will create ‘superintelligent’ systems; and that AI systems will do science that create greater and faster technological progress than humans could ever have done—will be borne out in the fullness of time. But this vision has to make contact with reality, and reality can act as a weird breaking mechanism: Meta wants to build AGI, but they couldn’t use a nuclear power plant for their datacentre, because of some rare bees.

These bits never make it in the sci-fi novels, and so it’s easy to see far into the future, but miss the (frankly bizarre) hurdles along the way.

AI will still be constrained by physical and institutional bottlenecks—drug development still requires clinical trials, chip fabs take years to build, lab experiments need physical space and technicians, and negotiations between people need to take place. Lots of cognitive tasks will be automated, in our view, quite quickly, but because of the requirements for human liability and often necessary complements between cognitive tasks and ‘embodied’ tasks, we anticipate knowledge workers will be augmented, not replaced. In some sense, it could feel like ‘a promotion for everyone’.

Over time, the growth effects will rise beyond our current view, but accounting for present bottlenecks we expect an annual growth boost from AI to be 3-9%—transformative, but not explosive.

We are now confident we know how to build AGI as we have traditionally understood it...[w]e are beginning to turn our aim beyond that, to superintelligence in the true sense of the word.

Building AGI has become an engineering problem.

The ‘Eye of Sauron’-Theory-of-Research-Progress

When thinking about future AI progress, follow the research priorities of leading AI labs. I often imagine their research focus as “The Eye of Sauron”, the great flaming eye from Lord of the Rings. What The Eye gazes upon becomes the industry's all-consuming focus; what it ignores remains unsolved - not because it's impossible, but because it's not yet time.

Take emotional intelligence. Perhaps the model needs to sound more natural, needs lower latency, needs to reason about the information you give it, textually, from your voice, and from body language. Or perhaps, it just needs to text you first. Increased latency, personality post-training and better UIs gets most of the way there. But The Eye isn’t looking here yet.

For the past few years, The Eye has settled its gaze squarely at scaling training. And instead of trying to fully elicit the capabilities of the model, or fix particular quirks in its behaviour, the attitude has been “just scale”, and it’ll be fixed.

Listen to the labs

To see where The Eye points next, we don't have to guess - we can listen to what the labs have told us. OpenAI's “five levels of AGI”, first shown in an employee all-hands and later published by Bloomberg, maps out their critical path: first chatbots, then reasoners, next agents; before 'innovators', and finally 'organisations'.

Why these levels? The goal of the AGI companies is to have organisations of automated AI researchers, so that increasing the scale of training runs can be improved at an ever improving rate. The AI researchers would be able to work on any capability that is required to meet any of many definitions of AGI today, and beyond.

Understanding this critical path is useful in two ways:

First, it helps one track the most important events in AI and understand their true significance in solving “what’s left” to create powerful systems. For example, most of Google DeepMind's early work on beating video games like Atari wasn't because they cared about gaming per se, but about solving key problems in planning and goal-directedness (and their recent foundation world model, which can generate interactive environments, are explicitly for training agents). We see this in reasoning capabilities too: After o1’s breakthrough in mathematical reasoning, many dismissed Deepseek's R1 as a mere copy. But Deepseek's earlier papers reveal a different story: the Chinese AI lab had developed expertise in formal verification and Monte-Carlo tree search - key techniques for training reasoning systems - before O1 was published, perhaps indicating a pattern in how reasoning capabilities emerge.

Perhaps most tellingly, while many of OpenAI's best researchers have departed (Ilya Sutskever, Mira Murati, John Schulman, Alec Radford, Barret Zoph, Bob McGrew, Jan Lieke), it hasn’t affected the company’s valuation much—raising $6.6 billion at a $157 billion post-money valuation suggests we're far from the regime where breakthrough insights from star researchers are the limiting factor.

Second, it helps one to ignore weird forms of brittleness in the models, which aren’t going to matter in the end. Mainstream discourse about AI seems bizarrely eager to coalesce around the weaknesses of the models: “LLMs are stochastic parrots”, “the ‘reversal curse’ meant AI was doomed to fail”, “the models can’t count the number of letters in a word”, the models struggle when they are asked to perform tasks ‘backwards’; the models can’t do simple visual reasoning puzzles.

These debates misunderstand the ‘inside view’ from the labs—their sole research focus is the next step on the critical path. Specifically, in the last 3 years, the returns to scaling pre-training have been so high that it was nigh-on unjustifiable to dedicate researcher time and compute to anything that wasn’t a) making this scale more, or b) figuring out the ‘big ideas’ afterwards. Do you really think that AI labs—who have assembled a density of talent in the core research teams, not seen since the Manhattan Project—couldn’t solve the reversal curse if they needed to? The answer is that distracting The Eye of Sauron from the most important problems simply isn’t worth it.

The inverse of this, of course, is that it makes salient which challenges are worth paying attention to – Ilya Sutskever, a leading researcher who is paying attention to the critical path, notes that “pre-training as we know it will unquestionably end”, and, as a result, we'll need to jump to the next paradigm.

Concept, Scale, Apply: The Simplest Story of AI R&D

In the broadest of strokes, there are three stages needed to make progress on each of the five levels to AGI:

1. Concept: Proving a novel idea works at all, even crudely

2. Scale: Engineering a proven concept into a deployment-ready system

3. Apply: Transforming the technology into systems that create real-world value and can be widely deployed

We can track the progress on OpenAI’s levels to AGI so far:

Sometimes the "Concept" stage splits into two phases: first proving a raw concept works at all (Attention, CoT), then developing it into something that advances the critical path to AGI (GPT-2, o1). We can see this pattern emerging with AI Agency.

Agency

The starting shot of this era may have begun with Anthropic's computer use. As Dario Amodei explains in a recent interview, they jury-rigged computer control by training Claude to analyse screenshots and output click locations and keyboard commands, which could be chained together in a loop (show image, get click location, execute, repeat) to enable basic computer interaction across operating systems.

It's remarkable that this works at all — and to be clear, it barely does — because these models were trained for conversation. They lack direct understanding of computer interfaces, struggle with persistent memory across screenshots, and must awkwardly communicate every action in English (“click at coordinates 342, 156”) rather than having native computer control. They’re limited by their context windows, making it hard to handle complex, long-running tasks. This is a scrappily cobbled together agent with existing systems – not one built from the ground up.

But that’s next.

This proof of concept showed how agency can be broken into building blocks: goal consistency— can the agent maintain its objective over time; planning—can the model break down tasks into smaller steps; memory—can the model retain context across long sequences; and tools— can the model interact with a computer and APIs?

The hardest challenge is goal consistency – how do we get the models to maintain their objective?

Goal Consistency: Teaching agents to stay focused

Chatbots have already learned a basic form of goal consistency: maintaining their role as helpful, honest, and harmless assistants throughout a conversation. This ability emerged from training on examples of good behavior (observational learning) and receiving feedback from both human and AI raters (interactive learning). These same principles could extend to maintaining focus on “long-horizon tasks” – where the agent needs to stay aligned with its objectives over extended periods and multiple interactions.

Learning from examples

AI agents start like inexperienced employees — they need to first learn what basic tasks look like before they can work independently. Observational learning provides this foundation by having models observe and copy human demonstrations, like watching recordings of people using computers to complete tasks. This teaches the model valid actions (clicking, typing, navigating interfaces) and basic workflows, just as a new hire might shadow a senior employee to learn the basics of their role.

The internet does not have enough data where people complete a diverse range of tasks on their computer. This will require deliberate data collection and construction. Companies can (and do) hire knowledge workers to record their work as training data, which allows models to learn both the specific tasks being completed and the broader patterns of how humans approach and execute complex workflows. For the less scrupulous, there are many options available. Companies could collect data directly from users, either as a condition of use for their operating system, or without their knowledge (previously, Microsoft’s AI-powered Recall feature took screenshots at regular intervals). Another rich source of training data could come from the vast library of programming tutorials and screen recordings on YouTube, using AI to convert these visual demonstrations into structured sequences of computer actions.

But observational learning alone is insufficient for building capable agents. A model trained only on imitation can repeat patterns it has seen but lacks understanding of goals and struggles with novel situations. Most critically, pure imitation provides no feedback loop - the model has no way to know if its actions succeeded or failed.1 That's why observational learning serves as just the first step, followed by interactive approaches where agents can receive feedback on their actions and learn to achieve objectives rather than simply mimic behaviors.

Interactive Learning: Human, AI and environmental feedback

Just as a new hire moves from watching training videos to working with a mentor, AI agents progress from imitation to receiving direct feedback. This feedback — “reward signals” from the environment or human/AI raters — acts as a carrot and stick, teaching agents which actions help achieve their goals and which don't. While imitation provides the basic playbook, this interactive feedback loop is essential for agents to learn how to stay focused and adapt their approach as tasks evolve.

Inspired by the “think step by step” approach that improved language model reasoning — where models explicitly break down their thinking into smaller logical steps - we can have agents decompose complex tasks into verifiable subtasks. Think of a task like “set up a new web app” — this can be broken into concrete steps: creating directories, initialising repositories, installing dependencies, writing server code, and deploying. At each step, a verification system checks if that specific subtask was completed correctly: did the directory get created with proper permissions? Did the git repo initialize? Did dependencies install without errors?

This granular feedback at each step provides much clearer learning signals than only evaluating the final outcome.

In general, reward signals could come from both monitoring the environment and external feedback. The system can monitor basic system state - checking if files exist, if programs are running, and if there are any error messages. Beyond these basics, task-specific verification is possible through automated testing frameworks checking if code works, static analysis tools evaluating code quality, and performance metrics tracking factors like load times and memory usage.

This internal verification could be supplemented with human feedback. For each task, the system generates multiple different trajectories - different attempts at accomplishing the goal. Human raters can pick which attempts they prefer and give more fine-grained feedback to the models. They consider nuances about how the task was executed, not just whether it was complete: how clean was the code? How efficient was the solution? Was the reasoning logical and simple to follow? And this approach isn't limited to coding — you could imagine designing systems that let users rate how well an agent handled their email or organized their files at each step.

While AI systems can do some of this trajectory labelling too, human raters are (at least currently!) able to provide feedback on higher-level qualities and catch the subtlest errors or misaligned behaviors.2

These verification and feedback systems, while still in early stages, suggest a path to robust goal-following.

Memory: Remembering in ‘neuralese’

Current language models have to process everything through their context window or on an English language scratchpad, which limits their “working memory.” Yet, similarly, the main ideas are in place. In a recent paper, Meta researchers demonstrated how agents could maintain ongoing “thoughts” and “memories” in a compressed neural form (sometimes called “neuralese”). Instead of English-language tokens, it can represent objectives and information in an efficient way native to its own processing. Just as Claude’s computer use demonstrated a proof-of-concept for AI agency, scaling neuralese could enable efficient memory for future AI agents.

Planning: Scale what already works

Similar to goal consistency, we get some amount of planning capability “for free” from existing models. We can improve this through learning from expert examples too—imagine hiring McKinsey consultants to break down projects into actionable steps, or implicitly learning from software engineers. We might even get this capability automatically as a byproduct of teaching models to maintain consistent goals over long horizons—teaching agents to reliably pursue objectives might naturally improve their ability to break down and organize complex tasks.

Tool Use: Building native AI interfaces

Current AI-computer interfaces are inefficient – language models that do use external tools, like ChatGPT with plugins, or Claude Computer Use spit out awkwardly formatted JSON files to call APIs or move mechanically around websites for humans.

The solution is fundamentally an engineering challenge. One perhaps overkill approach would be to encode computer actions directly into the model’s input/output space - rather than inputting and outputting English, the model could output neural patterns that directly map to system operations. But will probably not be necessary – there is lots of low-hanging fruit. We’re still using interfaces designed for humans, instead of the rigid “ask and respond” format, we could develop fluid protocols where models maintain continuous connections with tools, cache key information and share information efficiently.

Putting It Together: Engineering Agents

Somewhere in a data center in Arizona, thousands of AI agents are probably humming away on virtual machines, working through tasks, receiving feedback on their trajectories, and, increasingly, getting better and better. And that’s not to underrate the fiendish engineering challenge. But the main ideas are in place.

We can point to the key components: reinforcement learning for goal consistency, memory architectures that maintain state without token bloat, planning systems that decompose tasks effectively, and tool use interfaces designed natively for AI. None of these requires fundamental breakthroughs - we understand what needs to be built. The next phase is engineering these components to production scale - creating memory systems that work across hour-long tasks and millions of tokens, planning systems that break down complex tasks reliably, and tool interfaces or operating systems optimized for AI interaction.

And such agents, with goal-coherence, memory, planning and tool-use, is the foundation for building AI’s that can make real discoveries.

Invention

To build AGI, AI systems must have some ability to make “conceptual leaps.” I like to conceptualise “invention” as the ability to spot useful similarities between distant ideas from just a few examples.

Creativity is just connecting seemingly disparate ideas and having better taste in which long distance connections might be fruitful.

Consider how a physicist solves a new problem: they might see just a few examples and recognise, “ah, this behaves like that system I studied last year.” They don't need thousands of examples - they can interpolate from a sparse set of observations to grasp the underlying pattern. The more expertise they develop, the better they become at making these leaps with less data.

But this is difficult to get from imitation alone.

The Generator-Verifier Loop: Verifying which ideas work

For Einstein-level conceptual leaps, raw generation capability isn’t enough – current models can already generate generate endless variations on existing ideas. What matters is developing the meta-cognitive ability of recognising which creative leaps are actually valuable.

The key idea is to combine powerful generators with ground-truth verifiers. The generator-verifier loop provides rapid, reliable feedback. Take mathematics: when a model proposes a novel proof, automated theorem provers can immediately verify if it works. This instant feedback helps the model learn which intuitive leaps are actually fruitful.

And, like with “goal consistency” in “Agency”, we can give this feedback step-by-step, rather than a single time at the end. A 2023 paper from OpenAI, “Let’s Verify Step by Step”, shows that providing feedback on each step of a model's reasoning process, rather than just the final answer, dramatically improves performance on mathematical problems. This makes sense – you’ll learn more from a tutor who checks each step of your work, rather than just a right or wrong.

For this to work, verification has to be easier than generation.3 In some domains like math and coding, this is intuitive – where we can check if theorems are valid, or code compiles. But verification doesn’t have to come from external systems. AI models can be their own verifiers.

We already see “verifier” language models that check the reasoning of a “generator’s” reasoning steps. This AI-based verification unlocks new domains: a model trained on scientific papers could learn to evaluate whether a proposed hypothesis is consistent with known evidence, or train itself on journal reviewers comments.

This further implies that as models get better at verification in one domain, they can help train better generators in related domains. The result is a kind of bootstrapping process. Each advance in verification enables better training of creative leaps, which in turn enables more sophisticated verification. Once we solve the core engineering challenges of fast, reliable verification, we could see rapid progress in models' ability to make genuine discoveries.

Synthetic Data: Using AI to generate training data

This verification process doesn't need to happen within a single model or during inference. As we approach the limits of available internet data for pretraining, the future of scaling likely lies in synthetic data generation - using expensive but capable reasoning models to generate high-quality training examples that simpler models can learn from. DeepSeek demonstrated this with r1 and V3: rather than having V3 develop reasoning capabilities from scratch during inference, they used r1’s strong reasoning abilities to generate verified examples for V3's training (similarly, o1 was rumored to be primarily a synthetic data generation model).

This trades expensive inference-time compute for one-time pretraining compute - once a model learns these verified patterns during training, it can apply them quickly during inference. It's a way to front-load the exploration and verification cost rather than paying it repeatedly at runtime.

The Path Forward: The Automated Research Fleet

The last stage of OpenAI’s five levels is to have AIs run organizations – this will require massive coordination. A fully automated AI organization will not have individual AI models in chat windows, or even a single AI agent operating alone on a virtual machine, but instead an automated research fleet: some proving theorems, others reviewing literature, generating hypotheses, running experiments, analyzing results, and developing new paradigms.

But the labs are confident. As Paul Graham reminds us, read the job listings. Three months ago, OpenAI began hiring for a new multiagent research team. Just this week, they've done the same for a new robotics team – the next frontier after software singularity.

We will build AGI before we agree on its definition. For whatever metrics we choose, whatever capabilities we demand, the main ideas are already here – or soon will be, discovered by the automated research fleets that lie at the end of the critical path. The science is done. What remains is engineering.

Acknowledgements: This piece benefited enormously from Jack Wiseman's extensive editing and substantial contributions throughout. Thanks also to Duncan McClements, Thomas Larsen, Eli Lifland, Daniel Kokotajlo, Somsubhro Bagchi, Niki Howe, Ariel Cheng, Nathaniel Li, Andrea Miotti, Samuel Ratnam, Sanskriti Shindadkar, Nat Kozak, Maximilian Nicholson, Xavi Costafreda-Fu, Philip Guo, Jacob Goldman-Wetzler, Jeremy Ornstein, Miles Kodama, Ollie Jaffe, Ananth Vivekanand, Saheb Gulati, Chris Pang, Xi Da, and Sudarshanagopal Kunnavakkam for their thoughtful feedback and suggestions. And thanks to the many others who provided valuable input and discussions.

All mistakes and oversights remain my own.

“If you look at the fractal structure of a snowflake, you might think that whoever made it did something impossibly intricate and difficult, but that building it piece by piece must somehow be possible, since someone did it. In fact, both statements are false: the way to make a snowflake is not to think in terms of its pieces but to know the laws of physics, have enough raw material and a large enough chamber, set the temperature, pressure, and humidity correctly, and wait for long enough. Furthermore, this is the only way to make snowflakes; trying to piece together a single one from little bits of ice is hopeless.” — Dario Amodei

Much of contemporary AI research is, in some sense, downstream from the perspective that we should, where possible, 'let the compute figure it out'. By Moore's Law—or at least the folk version of it—the scale of the computational resources available are consistently and rapidly increasing. In the medium to long term, it's only the techniques which are able to most effectively leverage larger and larger quantities of compute that remain relevant. Hand-crafted methods tend to plateau and so are out-competed over time by general, flexible methods which, in figuring out how to do things for themselves, remain readily scalable. This perspective has been hard won, hence Rich Sutton dubbed it The Bitter Lesson.1

This is a large part of why modern AI research is increasingly synonymous with the study of deep neural networks.2 Neural networks (or machine learning ‘models’ as they are more often referred) are a kind of container which is able to hold a variety of possible 'circuitry'. It's this circuitry which determines how the model extracts and processes information from input to produce some output. If we have some metric which rates those outputs, we can, by a process of trial and improvement, search over the circuitry our model is able to hold so that, over time, our model 'learns' to perform well according to that metric. Importantly, the specific form that both our models and the particular process of trial and improvement3 take allows them to be run and scaled up very effectively on modern computer hardware (GPUs and TPUs in particular), with improvements tending to come from removing bottlenecks and improving scalability.4

Naturally, as the size of the container, the number and variety of examples, and the quantity of search performed scale with compute, the circuitry learnt, and thus our model's behaviour, can become increasingly sophisticated.5 What our model learns will depend on what we ask it to do: what situations we put it in, what data we give it as input, which metric we use to grade its output, and so forth. Learning can be based on simply observing data or it can involve interaction, where past outputs can affect future inputs or feedback, depending on what we choose and what we want our model to learn.

In the observational case, our hope is that our model can learn as much about the world as possible from the data it’s shown. In other words, we want our model to compress and internalise the structure and regularities it can find within the data. The nature of data the model is given (the format, the quantity, the quality) is thus the most important factor in determining what the model will learn. Depending on the data, there are two natural types of tasks we can give the model which will get it to compress: prediction (i.e. trying to guess what will come next) and reconstruction (i.e. trying to undo some process of noise or corruption). For both cases, there are natural metrics for our model to pursue, with small changes depending on the exact set up we place our model in and what we’re asking our model to predict, or put differently, reconstruct.6 Given the data and the task, as long as we then ensure that our model and training set-up are structured appropriately, then we should be able to readily scale it up: in the quantity of data that we feed it, in the various dimensions of our container, in the quantity of training we give it. As the model is scaled up, it will be able to pick up on more subtle patterns and structure within the data at a finer level of precision. If our data is sufficiently rich, then it will contain structure across many different resolutions, similar to how coastlines remain rough as you can resolve closer and closer. It’s for this reason, we think, that we tend to find power (or ‘scaling’) laws between our natural metrics and the aspects of increasing scale in the training of our model (i.e. model size, data quantity, training compute).7

Prediction based learning on text, where the model is given some leading text then attempts to predict the continuation, has been particularly important for getting very powerful, general models. Text is the paradigm medium for efficiently representing world knowledge and reflecting on various aspects of thought. The vast range of subtlety and complexity in the human textual corpus has meant that the corresponding power laws have held over more than a dozen orders of magnitude. As our models moved to larger and larger scales, they internalise more and more, demonstrating deep knowledge and understanding of the world as well as some aspects of thought. Other aspects of thought, however, can be more difficult to internalise this way. In particular, abilities such as long term coherence, detecting and correcting errors, backtracking, and other ‘long horizon thinking’ skills, have found difficulties arising from their effects being widely distributed over time and from there being few high quality demonstrations.

Why do we care about this so much? Well, almost all real world applications involve extended tasks and so these issues have prevented these models from moving outside of areas where this doesn’t matter so much (chatbots, coding assistants, search) and into things like agents, which can go out and independently perform tasks in the real world. These ‘long horizon thinking’ issues have been the primary bottleneck for getting agents to work well in the real world. Other research paths which were blocked by this such as the desire to gain the effects of a larger, more intelligent model by letting a weaker model think for longer. This is a particular instantiation of wanting to trade-off training compute and inference compute (i.e. compute used in running the model).8 Also blocked were hopes of getting around the ‘data wall’ - the issue that in pretraining, high quality data is increasingly scarce and expensive - by generating high quality synthetic data by thinking hard before creating the final result as well as providing higher quality feedback in other parts of training.9

The primary reason that people have been so excited by o1 is that it was the first large-scale demonstration of a new (though much anticipated) technique which attempts to address precisely these concerns, potentially unblocking one of the major constraints of these models.

In the interactive case, our hope is that our model can learn useful skills by experience from interacting with the world. Here there is no clear signal of good behaviour a priori as we had in the observational case. Under the formalism of reinforcement learning, we can separate out the problem by assuming that we have some reward signal which our model can receive after every output it makes and from which we can derive good behaviour by trying to maximise the total of our reward signal over time. Broadly, there are two kinds of learning we can do: ‘value learning’ and ‘policy learning’.10 In value learning, the model learns the future reward it expects to collect given its current state11 with the policy (the way that the actions actually taken in the world are selected) being learnt or derived from these estimates.12 In policy learning, the model learns to output actions directly by taking the total reward the model will receive to be our metric of pursuit. This maximisation requires trialing our working policy at every step and so pure policy requires an overwhelming number of trials in the world. Thus value learning is added to make learning more efficient, with the particular art being to do each well without harming the other13, along with the ability to make use of trials collected from slightly older versions of your policy. When acting in more complex domains, this latter approach is common.14

In some domains, such as games or question answering, identifying a suitable reward signal is straightforward (e.g. win / lose, video game score, correct / incorrect) but this is not true in general. For example, in using reinforcement learning to train for preferences where a separate ‘reward model’ (itself trained to predict how a human or another AI model would rate some behaviour) generates a cheap reward signal for another model.15 Some reward signals (reward models in particular) run into issues of robustness where the model training against the signal exploits defects to get high reward without learning the intended behaviour. Rewards can also have varying degrees of sparsity (i.e. more or less frequent with the number of actions taken) with sparser rewards being more difficult to learn from, requiring good initial behaviour or a high degree of exploration.

One way to attempt to address the issues around ‘long horizon thinking’ skills16, is to set our model against some problem and allow it to think before answering. Using our reinforcement learning techniques, over time the model should learn how to leverage its ability to think to help it produce correct answers. Remembering the Bitter Lesson, we should also likely avoid interfering too much with how the model thinks and hoping that, by letting the model learn to think by itself, the kinds of skills we’re after will emerge17, though likely allowing the model to check intermediate runs (e.g. seeing if intermediate code runs). We should also try to make sure that what we’re doing is as scalable as possible. We want to pick problems which give some clear robust notion of completion and of these ‘verifiable’ problems, ideally we want something where the verification is relatively quick and cheap so that they don’t become bottlenecks for our scaling.18

To ensure that our model has a signal it can learn from, we also need to ensure our problems are appropriately scoped, being neither too easy nor too difficult, so that our model has an incentive to improve its thinking. If the model knows little physics, we can't just give it a whole load of advanced magnetohydrodynamics questions to learn from, it'll never get any correct and won't know how to improve. So we're looking for a sweet spot where the problems we give the model to solve are neither too difficult nor too easy that they won't learn anything useful.19 While areas like mathematical proofs against checkers and coding against unit tests provide large classes of such problems, more broadly finding a large number of problems which satisfy these criteria can be difficult, so it’ll be important that our training method is very efficient in terms of the number of problems it needs to learn well.20

Ultimately, the value of this verifiable RL training depends on empirical degree of scalability, and how well the training generalises between domains (if our model has learnt to think through maths problems, will these skills transfer to thinking about other areas?) and timescales21 (if our model has learnt how to perform well over an hour22, will our model also be able to perform well over a day? Over a week?). The degree of generalisation between domains, in particular, has special importance as it determines to what degree limited cheap verifiers are sufficient to get our skills more broadly or whether specific training will have to be done for each task or domain, but could increase with a sufficient diversity of tasks and scale.23

While little is known about the details of these questions, we do know that in the case of whatever kind of training they’re using for o1, we know that the scalability is likely good24 as within around 3 months of the announcement of o1, OpenAI announced their scaled up o3 model. The consensus25 seems to be that the primary improvement made here was a tremendous scale up of the RL training procedure (approaching the order of compute used for pretraining) and shows the kinds of results you would expect, with markedly strong performance of difficult mathematics and competitive coding.26

Zooming out, what does this mean in the big picture? Well, we might say that there is a third type of large scale progress to pay attention to in the frontier models, with advanced in these new verifiable RL (or ‘reasoning’) techniques now standing alongside the improvements in world knowledge and understanding from pretraining and the advancements in the domain these models can operate in from improvements like multimodality, compute use, and so on.27

Thanks to Toby Logan, Jack Wiseman, Seb Handley and Alicia Pollard for feedback on earlier drafts of this post.

• Since 2012, the computational power used to train the largest AI models has grown 100 million-fold. This has become enormously energy intensive: the most recent model from Meta used an estimated 27 megawatts of power capacity, which is approximately the same power required for 88,000 UK households, approximately the same as York. If trends in computational power scale-up continue, by 2030, training the largest model would use 2.15 times the UK’s entire energy generation capacity. 

• The AI developers are power constrained: Microsoft has committed to a 20-year power purchase agreement to reopen Three Mile Island; Google and Amazon have entered partnerships with energy developers to construct new Small Modular Reactors; and Elon Musk’s x.ai has converted a factory in Memphis into a datacentre, and put natural gas generators outside. SemiAnalysis, a leading third party research group, estimates that AI datacentre power demand will grow by 40 gigawatts globally in the next two years, roughly 200 times the average power demand of Liverpool.

• We investigated the feasibility of low-carbon power sources to power AI datacentres in the UK: either a combination of wind, solar, grid batteries, and natural gas backup;  or using nuclear power. Our modelling found that the cost-minimising allocation of renewables would require over 200km² of contiguous land area per gigawatt, which would need to be next to an LNG import terminal. Furthermore, this would have 40% higher carbon emissions, and lead to 27 times more ‘expected deaths’ than using nuclear power, based on historical patterns.

• However, as things stand, no developer would choose to build an AI datacentre in the UK. The UK’s nuclear construction costs are 4.5 times higher than South Korea, and building reactors takes twice as long. Approving a new power plant has taken 6 and 12 years in the last two instances, while approving a reactor with the same basic design in France took just 3 years. If a datacentre operator wanted to connect to the grid instead of building new power, it would take up to 15 years to get a grid connection, and the industrial electricity prices are four times higher than in the US and 45% higher than in France.

• Despite these facts, the UK can make a series of reforms and, in just a few weeks of swift action, the UK can become the best place in the world to build AI datacentres and the nuclear power to support them. Special Compute Zones would create an alternative planning and regulatory approval process for nuclear power, AI datacentres, and the transmission and networking infrastructure to support them. If these reforms allowed UK projects to close the cost gap with South Korea by two thirds, nuclear power would become 37% cheaper than an equivalent blend of renewable power for datacentres. With high levels of speed and certainty during the approval process, the UK has the opportunity to catch the wave of AI infrastructure development.

Navigating the report

The Overview can be read as a standalone. For more details…

1. ‘AI progress is very quick’, explains what investors are buying into.

2. ‘How AI became computationally intensive’, offers a brief and simple technical introduction to why AI systems use so much computational power.

3. ‘Further progress and deployment of AI systems will use 10’s to 100’s of gigawatts’ shows the historical trends in computational intensity and what forward-looking estimates suggest for the energy-intensity of AI.

4. ‘The UK would power AI datacentres using nuclear power, not wind and solar’ evaluates the suitability of low-carbon energy sources.

5. ‘Going without AI datacentres would be a mistake’ sets out the case for why the UK needs to have AI datacentres, and not depend on international markets.

6. ‘Nuclear power is slow and expensive to build, but it could be cheaper and faster’, is a diagnosis of why UK nuclear construction costs are so high, and how this might be remedied.

7. ‘The UK should create Special Compute Zones’, suggests the implementation details for making the UK the best place in the world to build AI datacentres. 

We would like to extend thanks to John Myers, Robert Boswall, Nathaniel Read, Freddie Poser, Ben Southwood, Mustafa Latif-Aramesh, and Samuel Hughes; for their feedback and comments on this work.

We would also like to acknowledge the work of Britain Remade, in particular Sam Dumitriu, whose high-quality analysis of nuclear power and infrastructure costs was instrumental in this proposal.

Overview

Some kinds of growth are difficult to achieve. One way people create economic growth is by making a new discovery in a lab, doing the engineering work to turn this discovery into a product, and then running a business to distribute the product to the world. The academic literature suggests this kind of growth is getting harder over time—it requires adding ever more researchers to maintain a consistent rate of breakthroughs. This is what economists call ‘frontier’ growth—this requires doing something nobody has done before. On the other hand, some other kinds of growth can be comparatively simple: an investor might come along, looking to repeat a technique proven to work elsewhere, and they'd like to build a factory of some sort to replicate it. Enabling this kind of ‘catch-up’ growth is a choice.

The UK has been without growth for many years. Since 2007, per capita GDP has grown by just 0.35% per year and total factor productivity growth has been flat.

Sometimes, a new type of frontier growth will emerge: a combination of scientific breakthroughs will lead to the creation of a new ‘general purpose technology’. There have been three in modern history: the steam engine, electricity, and information technology (computers). These technologies will spread through many industries, and become a platform for the development of further technologies. Each general-purpose technology took decades to reach mass adoption, but each provided a huge opportunity for economic growth. The Industrial Revolutions were organised around a general-purpose technology: the First Industrial Revolution was steam power, the Second was electrification, and the Third was digitalisation.

Such a breakthrough is happening today, in artificial intelligence (AI).

AI is poised to do for cognitive labour what the steam engine did for physical labour

One way to think about the impact of AI is using the steam engine as an analogy. In the First Industrial Revolution, there was a clear input and output relationship—pour in coal, and get out rotational power in a crankshaft, which could be used for all sorts of downstream tasks (trains, factories, more coal mining). What makes this so powerful is that it is dependable—every time one adds coal, they know what will happen; it is general—the rotational motion can be applied to any of these use cases; and it is scalable—so long as one can design a bigger steam engine, there is no limit to the amount of coal an engine can usefully convert into power.

A latent resource we have today—just like the British in the second half of the eighteenth century had a lot of latent energy in surface coal—is computational power in computer chips. Over the last 60 years, the number of transistors (think: computational power units) on a chip has doubled every two years, giving us an enormous supply of computational power. One intuition for what the field of AI is trying to answer is, “How can we exchange this computational power for useful information processing capabilities?”.1 This requires finding the system of pipes and valves—or in this case, the combination of architectures, algorithms, data, and the training procedure—to make the input-output relationship work.

Over the past 12 years, AI researchers have found the broad combination of variables which allow them to add more input. Today’s state-of-the-art systems score very highly on capability benchmarks. They outperform PhD-level experts on a benchmark testing for scientific expertise; score in the 99th percentile on the US law school admissions test (LSAT)2; and are capable of winning a silver medal on the International Mathematics Olympiad exam (for the best sixth-form age students in the world). These capabilities have translated into economically useful performance: on software engineering tasks, in customer support, and in materials discovery. 

AI systems will support economic growth in three main ways:

1. AI will let us automate existing information processing tasks

2. As the cost of information processing declines to zero, we will use this new abundance to create new products and services. 

3. AI can help to accelerate research and development to drive frontier growth.

What is important to understand about the input-output relationship of AI systems is that there are extremely dependable improvements on the specific task they have been trained to do. However, general purpose AI systems have been trained primarily to predict the next word in a sequence, and so it is genuinely uncertain whether improvements at text prediction will continue to transfer to useful capabilities in the system; and by extension, whether these will transfer to economically useful tasks. OpenAI’s GPT-3, the first system which most people had experience with through ChatGPT, had useful capabilities—it was good at writing sonnets—but it was GPT-4 that could begin to do coding. So far these improvements have continued to transfer, which explains the impressive capabilities and economic interest in AI systems.

The computational and energy-intensity of AI is only getting greater

Because of this uncertain exchange rate between compute and ‘capabilities’ broadly, AI developers want to continue increasing the computational inputs to create and run these systems, and they want to sell the current capabilities as broadly as possible. The challenge is that increasing computational power is enormously energy-intensive. AI systems use specialised chips, predominantly the Graphical Processing Unit (GPU) designed by NVIDIA. Each state-of-the art GPU uses an enormous amount of power: if used continuously, it will consume 44% more power than an average UK resident.3 To support this energy and computationally intensive process, AI chips are hosed in datacentres. These are specialised warehouses which are optimised to manage the cooling and energy requirements of the chips. From the outside, these are ordinary buildings on an industrial park. 

The most recent AI system from Meta (called Llama 3.1) was trained using 16,000 active GPUs in a single datacentre, which is estimated to have required 27 megawatts (MW) of installed power capacity. This is roughly equivalent to 88,000 UK households.4 One useful intuition is that a single datacentre will use as much power as a small city.

As developers are betting on the input-output relationship holding, computational power is being scaled up at breakneck speed—the amount of computational power used in training the AI systems has grown 5 times over per year since 2017. Were the current trend to hold until 2030, training the largest AI system would require approximately 2.15 times the UK’s entire electricity generation.5 Meta’s release of Llama 3.1 in July 2024 might begin to look very small indeed! Running the systems, too, is set to become more energy intensive: new methods have been developed to input additional computational power whilst the model runs, to enhance capabilities further. Jensen Huang, the CEO of NVIDIA, expects this to grow by “a billion times”. More details on these trends are covered in the main body.

There is a wave of capital investment to build 10s to 100s of gigawatts of datacentres and power for the next decade

To continue increasing the input of computational power, the AI developers and the ‘cloud providers’ who sell them datacentre capacity, are growing their operations as quickly as possible. SemiAnalysis estimates the power demand from AI datacentres globally will grow 40 gigawatts (GW) by 2026, and in the US alone, by 47.8 GW for 2028, from just 8.5GW in 2024. To put this into perspective, one GW of continuous power demand is five times the average power demand of Liverpool.6The addition of AI datacentres globally in the next two years will use roughly 200 times the power demand of Liverpool.

This enormous surge in demand is constrained by energy generation. There is simply not enough power. As a result, AI developers are signing long-term power purchase agreements to bring nuclear power plants back online, and are signing development agreements with developers of ‘Small Modular Reactors’ (SMRs). Perhaps most dramatically, Elon Musk’s company, x.ai, has even converted a factory in Memphis into a datacentre for 100,000 GPUs in just 19 days, and is using natural gas generators outside to make up the power it needs. 

No developer would build an AI datacentre in the UK by choice

As things stand, the coming wave of capital investment will bypass the UK. No AI developer or cloud provider would choose to build an AI datacentre with new power in the UK:

• Planning permission for the datacentre would take too long. Until recently, datacentres went through Local Planning Authorities under the Town and Country Planning Act 1947, but the Labour Government is going to make them ‘Nationally Significant Infrastructure Projects’ which require a Development Consent Order from the Secretary of State, with the aim of accelerating the process. However, the average period of consideration for a Development Consent Order in 2020 was 22 months, and since the last election, ministers have delayed 40% of decisions on Development Consent Order decisions, and so it is likely there would be further delays.

• If a datacentre operator wanted to use grid power, it would take up to 15 years to get a grid connection, and even then the UK’s industrial electricity prices are four times higher than the US and 45% higher than France.

• If the datacentre operator wanted to procure their own nuclear power, it would take 6 to 12 years to get approval, and once they have approval, construction would take 12 to 15 years.

The current pace of planning, regulatory approval, and construction is too slow to keep pace with the wave of investment.

If the UK wants AI datacentres, nuclear power would be the safest, cleanest, least land-intensive, and could also be the cheapest.

We investigated which energy generation method would be most suitable to power AI datacentres. We compared the feasibility of nuclear power, or a blend of wind, solar, grid battery storage, and natural gas backups.

We calculated the cost-minimising way to blend renewables with batteries and gas, to provide the permanent power supply datacentres require, and found that per gigawatt of firm power, it would require 8 GW of installed solar power, 0.37 GW of wind power, 12 GWh of battery backup, complete gas backup, and LNG import capacity for 451 million cubic metres of gas. This would become infeasible on multiple grounds:

1. Land intensity—8 GW of solar panels and 0.37 GW of wind turbines would require 160km² and 41km² respectively. (As a point of reference, Cardiff is 140km² and Reading is 40km².7) This scales very poorly. Some datacentre campuses are much larger than one gigawatt, for example, to power Microsoft and OpenAI’s 5GW datacentre campus in Virginia it would require over 1000km² of contiguous land.

2. Emissions—because of the intermittency of wind and solar, natural gas would generate 28% of the power, which would produce 40% more carbon than equivalent nuclear capacity.

3. Safety—the air pollution from natural gas emissions would be many times more dangerous than the risk of a nuclear accident, resulting in 27 times as many expected deaths.

4. Limited proximity to LNG import capacity. The UK has three LNG import terminals, one in Kent and two in South Wales. The natural gas generation terminal would need to be some distance from population centres to reduce air pollution, but also close enough to the import terminal that gas pipelines are not necessary. It is either necessary to find ways to build these generation facilities in Kent or South Wales while being contiguous with the solar and wind farms, or it might be necessary to build a new LNG import terminal, but neither approach scales well, and the latter would add substantially to costs.  

5. Cost—we calculate a levelised cost of blending wind, solar, batteries, and natural gas at £106/MWh, which is lower than the current CfD price for Hinkley Point C (£143/MWh8), though there are large opportunities for cost savings with nuclear power—South Korea builds at roughly 25% of the cost of the UK—and so if two thirds of the cost gap with South Korea could be bridged, the cost would be 37% cheaper than renewables. (In Texas, renewable energy at the same emissions intensity for £74/MWh.)

Building nuclear power plants in the UK has been slow and expensive, but it doesn’t have to be. Some relevant facts:

• Two thirds of the cost of Hinkley Point C was interest—if you can bring down the cost of borrowing, this can cut the final cost of electricity in half.

• South Korea builds 8 to 12 copies of the same reactor design. This means they benefit from learning, both technically, and in terms of the regulation, and they have a consistent supply chain of components and of people with the skills to build a reactor. On the contrary, Hinkley Point C was a one-of-a-kind reactor which had 7,000 design changes from the basic design already used in France and Finland, and was the first nuclear power built in the UK in 21 years.

• Responsibility for nuclear power plant approval has been diffused between many actors who can say ‘no’, or who might add incremental delays and cost increases to new nuclear power plant construction, which amounts to a de facto ‘no’; but there is no positive force in the system who pushes for power plants to be built.

Small Modular Reactors (SMRs) provide a big opportunity for the UK. Because most assembly happens in a factory, large productivity gains during manufacturing are possible, and on-site construction can take just months. Furthermore, SMRs are especially suited to AI datacentres because there can be a fleet powering a datacentre campus, and so when there is an outage, there is a diversified power supply.

Making UK nuclear power competitive is the only way the UK would attract AI datacentres: if a datacentre provider wanted to use a blend of renewables, they would be likely to be much better off in West Texas.

It is possible to bring the costs down—there is a lot of low-hanging fruit to be picked!

The UK needs AI datacentres for economic security, growth in former industrial areas, and to seize the opportunity of future frontier growth

At this point, a sceptic might ask whether it is worth trying at all. In general purpose technology revolutions, most of the gains come from adopting the new technology, and seem unlikely to accrue to those who host AI datacentres. Perhaps this wave of investment is going to happen, but can’t the UK just focus on ‘high-value’ activities, like integrating AI systems and building AI applications. 

We don’t think so—first and foremost, there’s no scarce resource being used up by permitting this growth, all the capital is from private investors, and the UK needs the growth. Most importantly, however, the UK needs the critical inputs for its future economic engine, the old economic dogma—that it is possible to sit atop the value chain and focus only on the ‘high value’ activities—is incorrect. This causes the hollowing out of industries, and neglects the valuable ‘learning-by-doing’ that allows us to make future growth. The UK without datacentres will lack the critical inputs for its future economic engine.

The Chancellor Rachel Reeves has frequently promoted a new doctrine of ‘securonomics’, of which the core tenets are prioritising economic security in an ‘age of uncertainty’, not depending on a narrow set of industries from London and the South East to drive growth for the whole country, and seizing the opportunities of a rapidly changing world.

Hosting AI datacentres in the UK is central to all three tenets. 

First, as AI systems become increasingly integrated into the economy, especially into the UK’s professional services export businesses, a large fraction of the UK’s capital stock will be created in, stored in, or run in AI datacentres. The UK will want these AI datacentres to be here, rather than overseas and connected through an undersea cable, to ensure it can protect these assets. Furthermore, as adoption is critical to capturing the gains, the UK needs to ensure it has computational power capacity it needs. As demand for computational power rises globally, it could be the case that UK businesses are unable to access this. The Microsoft CFO said on an earnings call two weeks ago that revenue growth in their cloud business was 33% but, “[d]emand continues to be higher than our available capacity.”

Second, the Government’s AI Sector Study shows that 75% of UK AI companies are based in London, the South East, or East of England. This is to be expected: AI application developers will agglomerate around London because it has the best venture capital ecosystem and AI talent density outside San Francisco. However, as the Chancellor has said, there has been, “[a] misconceived view held that a few dynamic cities and a few successful industries are all a nation needs to thrive…[t]he result was a paucity of ambition for too many places, the hollowing out of our industrial strength and a tragic waste of human potential across vast swathes our country.” It is now very rarely the case that growth can be so readily directed towards areas with a strong industrial past, but that is the opportunity of AI datacentres—it is possible to bring the Fourth Industrial Revolution to the rest of the UK as well, if the rules allow it.

Finally, given that AI systems are likely to play a critical role in research and development. The UK has world-leading science and technology clusters, whose work is likely to be transformed by AI systems. Running AI systems that support research and development—AI for science—will be critical to any frontier growth in the UK for decades to come, and it is not possible to depend upon international datacentre markets to supply the services which are so central to our future prosperity. They would need to be here.

The UK has already missed an opportunity for frontier growth—UK scientists like Geoffrey Hinton and Demis Hassabis were at the forefront of the AI revolution, which is now being commercialised by a small handful of US firms. The UK is about to pass up the opportunity for the easiest kind of growth—someone wants to build AI datacentres and power to support it. The money wants to flow but the revealed preference of our current regulatory and planning system is that it should not. 

The UK can become the world’s best place to build an AI datacentre and the power to support it

This can all be fixed in as little as a few weeks. To do this, we propose creating ‘Special Compute Zones’—which provide an alternative planning and regulatory approval process that fixes the issues with the current approach. It would provide the certainty, speed, and hence, the opportunity to be cheap, which is currently lacking.

Developers could receive a single point of signoff for the power, datacentre, and transmission and networking infrastructure they require. Within the Zones, there would be ‘deemed consent’—meaning the default answer to construction is ‘yes’—and permission to construct would have to be challenged within three months of an application. Ordinarily, a planning decision will weigh the relative merits of each project; but within a ‘Special Compute Zone’, it would be decided that by creating a zone, this cost-benefit is pre-considered, and therefore approval would depend on a ‘condition-based approach’—if a developer can show that the project meets particular standards, it goes ahead. There is precedent for this kind of approach in the EU and Spain, where ‘Renewable Acceleration Areas’ use condition-based approaches. We present more details on implementation below.

Missing this wave of capital investment is like missing the railways

Between 1841 and 1850, private investors ploughed a cumulative 40% of UK GDP into building the railways—imagine if instead our planning and regulatory regime had prevented this investment and the UK’s rapid economic growth! The UK continues to collect the dividend from this period of growth today, 170 years on.

Sometimes growth is difficult to come by, but in this case, growth is a choice: all we need to do is unhobble ourselves.

1. AI progress is very quick

The goal of AI research is generally intelligent systems

To begin, it is useful to clarify what AI research is aiming at, as people use many different terms. These include Artificial General Intelligence (AGI), Artificial Superintelligence (ASI), human-level AI, powerful AI, and transformative AI. The terms can be somewhat misleading—does ‘human-level AI’ refer to AI systems which perform at the level of the average human, or the smartest human? Furthermore, the development of AI systems is ‘unbalanced’, in some ways current systems already surpass the smartest humans, but in other ways they fall far short. 

Debates over these definitions can distract from focusing on the most important thing: very capable systems might be created before we have clarified whether ‘true’ AGI requires, say, emotional intelligence. To avoid these pitfalls, three terms can be useful:

• A Drop-In Remote Worker refers to an AI system that can interact with a computer, pursue tasks for weeks-equivalent of human time, at the level of a graduate remote worker.

• An Expert Scientist refers to an AI system that can perform scientific research, at the level of the world's best scientists across a variety of scientific domains.9

• A Superintelligence refers to an AI system that exceeds human intellectual capabilities across all relevant cognitive skills.10

The explicit goal of the AI research labs is to create a software program that is an Expert Scientist. When DeepMind was founded in 2010, their mission was: “To solve intelligence, and then use that to solve everything else.”

AI researchers are making progress towards this goal

Whether expert scientists are possible in the current technical paradigm is genuinely uncertain, but current state-of-the-art systems can do a lot:

• Coding assistance. State-of-the-art AI systems are very effective at giving assistance to professional software engineers. This research found that software engineers were 55.8% faster at completing a software-engineering task with assistance from an AI system, and the former head of self-driving at Tesla, Andrej Kaparthy, wrote that he, “basically can't imagine going back to ‘unassisted’ coding at this point”.

• Scientific capabilities. State-of-the-art systems outperform experts with relevant PhDs on GPQA diamond, an evaluation which tests expertise in physics, chemistry, and biology.

• Mathematical abilities. DeepMind’s AlphaProof was trained to solve problems from the International Mathematical Olympiad, and achieved ‘silver medalist’ performance—roughly equivalent to scoring in the top 100 sixth form mathematicians in the world. OpenAI’s ‘o1’ was not trained specifically to perform well at maths questions, but scored 80% on a US Mathematics Olympiad qualification exam, which is equivalent performance to the top 500 high school students in the USA.

• Agentic improvements. AI systems can complete small software engineering projects. SWE-Bench Verified is a benchmark which measures the ability of AI systems to perform real world software engineering tasks: OpenAI’s GPT-3.5, the state-of-the-art model in 2022 performed poorly, only able to complete 0.4% of the tasks. As of November 2024, Anthropic’s Claude 3.5 Sonnet was able to successfully complete 53% of tasks. For another example, OpenAI’s o1 successfully completed 100% of the problems posed to interviewees for software engineering positions at OpenAI.

Where AI progress goes from here is very uncertain

While the historic trajectory of AI progress is clearly very steep, it is difficult to know whether this implies that capability improvements will continue, and exactly what this might look like. Most importantly, we do not have a large suite of benchmarks for assessing the capabilities of the most advanced AI systems. Because the AI systems have improved so quickly, our evaluations ‘saturate’, meaning that all systems score indistinguishably high scores. For example, MMLU and MATH were benchmarks released in 2020 and 2021, and specifically designed to resist this kind of saturation. GPT-2 scored 32.4% on MMLU and 6.9% on MATH at their release, but in just three years, they approached saturation: GPT-4 scored 86% and 84% for MMLU and MATH respectively. The chart below shows examples of saturation across benchmarks

11

Some of the best benchmarks that we have for scientific capabilities—GPQA diamond and LAB-bench—require the models to answer multiple-choice questions about the subject matter. These questions can be thoughtfully designed, but the conclusions from these tests are limited. They imply the systems were very good at answering a narrow set of scientific questions, it doesn’t imply very much about whether the models can do the work. For what it is worth, the AI systems clearly aren’t just multiple choice machines—systems seem to be able to solve problems from the famously difficult Classical Electrodynamics which can require days worth of effort from graduate-level physicists.

One approach to understanding the scientific capabilities of AI systems is to decompose the process of doing research into discrete steps, and evaluate each step piecemeal. For example, this paper tested hypothesis generation, and found, "LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility”. The AI developers and the UK and US Safety Institutes are likely to maintain more comprehensive private taxonomies to track progress across the research pipeline, though there are strong incentives for these organisations not to release these.

Aside from benchmarks, it can be difficult to interpret progress clearly, because the people on the very frontier, who can see best where progress is headed, have been very heavily selected for conviction in the current technical paradigm. Sometimes, these researchers are modelled as capitalists, ‘hyping’ the potential future capabilities to fundraise or make a project. This is an incomplete picture—no doubt there are people within AI labs who are ‘selling’ the future, but there are also many researchers who very sincerely think that AI systems will have transformative capabilities. For example, the Chief AI Scientist at Anthropic, Jared Kaplan, who is also a Professor of Theoretical Physics at Johns Hopkins, said in a talk at CERN: “I think in certain respects AI systems are approaching human level performance. I think there are some challenges for continuing this progress but I think there aren’t any really compelling blockers for AI doing things like helping to automate science.” Likewise, John Schulman, the cofounder and former head of ‘post-training’ at OpenAI, anticipates that AI systems will match top AI researchers within 5 years, and Demis Hassabis, the CEO of Google DeepMind, believes, “we are in shooting distance of curing all diseases”. These claims should be treated with a healthy mixture of scepticism and seriousness, and they should not be dismissed out of hand.

This picture is made evermore difficult by the distortion of academia. Because AI developers pay salaries 10 times or more what can be earned in academia, there has been an enormous movement out of universities into OpenAI, Google DeepMind, and Anthropic. Those who are left behind are strongly selected from scepticism of the current technical paradigm.

The essential takeaway is that there have been very steep improvements in AI capabilities thus far, and as we will discuss in the next section, there are compelling reasons to believe this will continue. But exactly how this evolves, and where it might end is very uncertain. However, the most optimistic case—which is sincerely held by a number of people building on the frontier—is that this could usher in a period of explosive economic growth (~20% GDP growth or more).12 Though it is unlikely, it is within the realm of possibility.

2. How AI became computationally intensive

Some breakthrough technologies unlock our ability to exchange a latent resource for useful outputs. Steam engines are like this—pour the coal in, and the steam engine reliably converts the energy into the rotation of a crankshaft. This has all kinds of downstream uses: moving a train carriage, driving a pulley in a factory, or pumping water. Likewise, in the Haber-Bosch process, pour in hydrogen and nitrogen, and receive ammonia, which can be used as fertiliser. What makes these processes so powerful is their scalability: to move more goods before the steam engine, it would require adding more packhorses to carry goods along a turnpike road; before the Haber-Bosch process, growing more crops meant sending more boats to the Chincha Islands to harvest guano. With technology, there is dependable leverage—it is possible to continuously add more latent resources and receive back useful outputs.

Computational power is the unit of information processing, in brains and in computers. It would be incredibly useful if we could develop a technology which allowed us to pour in computational power and exchange this for useful informational processing capabilities. ‘Ordinary’ computing does a version of this; but it is fragile and limited. Ordinary computers can only do processing tasks which have been specified by a program in advance, whereas AI systems have the capacity to learn. This is one intuition for what AI research is doing: it is finding reliable and scalable mechanisms—just like the steam engine or the Haber-Bosch reactor—that allows us to exchange processing power for flexible and adaptive intelligence. 

This is unintuitive. Prima facie, AI research should be about having deep insights into the nature of intelligence, and designing machines that reflect these insights. Computer scientist Rich Sutton has called this realisation, ‘The Bitter Lesson’—what has driven AI progress is not the profound theories of researchers, but general strategies which allow AI systems to leverage greater amounts of computational power. In earlier approaches to AI, researchers would ‘hand-code’ how they thought an AI system ought to learn, for example, what features of an image to recognise in order to classify the picture; but neural networks, the foundation of modern AI, allows the system to learn for itself what features are salient, through a training process.

Increasing the amount of computational power during training

The neural network is like a little computer which can be programmed by adjusting a series of dials. The aim of a neural network is to predict an output given a set of inputs. The iterative process of tuning these dials to improve the prediction is called ‘training’. The people creating the network supervise the training process by showing the data and the answers, but crucially, it doesn’t involve telling the network how it ought to process and understand the image. In other words, our process of trial and improvement tweaking of dials is essentially letting the little computer, by itself, search for the best way it can be programmed to achieve its goal, unlike ordinary computers which need a human to figure out a program first and then somehow communicate it to the computer. Dario Amodei described the training process in this way:

“You [the AI researcher] get the obstacles out of their way. You give them good data, you give them enough space to operate in, you don't do something stupid like condition them badly numerically [i.e. tweak the dials poorly], and they want to learn. They'll do it.”

What your neural network will end up learning depends on the goal you give them to pursue, and in what you ask the network to predict. There’s two ways it is possible to do this:

1. Directly optimise for a specific capability.

2. Optimise for a related goal, and hope that important capabilities emerge downstream.

Large language models, the basis of recent progress in AI, take the second approach. A language model is optimised to predict the next word13 in a sequence, based on the words that have come before. Google DeepMind’s Gemini 1.5 model is able to take in nearly one million words of input, to give the most subtle and accurate prediction of the next word of output. This isn’t an inherently valuable task in the way that a neural network which is trained to predict whether a dot on the screen is a tumour or a harmless cyst is inherently useful.

While predicting the next word isn’t inherently useful, pursuing this goal is still enormously powerful. Text has a very rich structure, meaning the words aren’t randomly assorted: whoever wrote them chose their order to convey an idea. If a neural network can understand that structure—say, by parsing all of human knowledge in the training process—the network has a representation of how everything fits together. Researchers would call this a ‘world model’.

Prediction is a meaningful task—if the network can take most of a book as input, and predict the end without having seen the book before, there is some sense in which it understands the content.14

The measure of how far these predictions are from reality is known as the ‘training loss’. There is an extremely predictable relationship between the model’s size, the amount of training data it uses, the amount of computational power (compute) it is trained on; and the model’s loss. As the models get bigger, and are trained on more data, using more compute; its loss declines. In other words, its predictions get better. Amodei has noted the declines are, “sometimes predictable even to several significant figures which you don't see outside of physics.” 

What is much less predictable is whether these declines in training loss translate into useful capabilities. It has been particularly surprising how well improvements on next token prediction have converted into useful capabilities so far. In expectation of these capabilities, AI developers have scaled their models dramatically: Google DeepMind’s Gemini 1.5, released in December 2023, was trained using 6.7 million times the amount of compute used to train a state-of-the-art large language model in June 2017. To put this in broader perspective, since 2012 the amount of computation power used to train the largest models has grown by 100 million-fold.

Using more compute while the model is running

Thus far, we have only described the opportunity to pour in additional compute during the training process to receive useful outputs. It is also desirable to pour in additional compute while the model is being used; technically called ‘inference’. A paradigm example of an AI system to take advantage of ‘inference-time compute’ was DeepMind’s AlphaGo. AlphaGo was trained to play the board game Go, or, to be specific, it was trained to find the next move from a given board state which maximised win probability. One way the system might have solved this problem is to imitate the moves of a human expert. Indeed, this was the initial approach to training. But a more advanced way of learning would be for the model to play against itself predicting what could happen in future moves and learn from that guess. This kind of planning could also be used when the model was competing. When DeepMind’s system was allowed to use this additional capability before choosing a move, performance jumped from 1500 ELO to 3000 ELO.

AlphaGo was able to exchange compute at inference-time for a jump in capabilities because the game of Go is well suited to running a form of tree search. It has a very clear goal (to win the game) and it has a reasonably constrained search space (the potential moves on the Go board). Determining the best next step is relatively straightforward. 

By contrast, ‘language space’ isn’t like this. There are many prompts which do not have a formally ‘correct’ response—determining what is ‘better’ is much more subjective. Also, the ‘space’ of potential directions in language is much larger than the potential next moves on a go board, especially as the responses get much longer. It would be very difficult for a tree search procedure for language to know what direction to search in. Because of this, when ChatGPT was released in November 2022, it had been trained on 500 billion tokens (think: words) but it could only use 4,096 tokens to respond to each prompt. It couldn’t use more compute while it was running to generate better answers; it was hobbled. 

This changed in September 2024. OpenAI released ‘o1’, a new series of models which generate ‘Chains of Thought’ before responding to a prompt. Chains of thought allow the model to use additional processing in response to more difficult questions. In the o1 release blog post, OpenAI showed a graph that demonstrates how as the compute budget is expanded, its performance on a qualification test for the US Maths Olympiad improves. 

The important takeaway from this section is that AI developers are constantly improving their  understanding of how using more computational power can increase the useful capabilities of the models. Granted, the current methods for applying additional compute might stop having returns, but it is likely that the future methods we find will continue to follow this pattern.

In the current world, intelligence is scarce and special—just as fertiliser or mechanical power was before the steam engine. It is weird to say, but creating more intelligence today is laborious: it occurs only in humans who take decades to mature and require lots of education. In the not too distant future, intelligence equal to, or beyond human-level, will be constrained only by our ability to pour in computer chips and electricity.

3. Further progress and deployment of AI systems will use 10’s to 100’s of gigawatts

Let us begin with historical trends. Since 2012, the amount of computational power used to train the largest models has increased 100 million-fold: how have we done this? There are beneficial tailwinds: the computational power of the state-of-the-art AI chip has doubled every 2.3 years, as shown in the chart below; and the energy efficiency of hardware has doubled every 3 years.15

16

However, the increase in training compute exceeds the rate of hardware improvements: since 2010, training compute has doubled every six months! Language models have moved faster still: doubling happens roughly every five months, shown in the graph below.

17

While using a computer at home is not at all energy-intensive, the kind of computation that AI systems are doing is very energy-demanding. A state-of-the-art chip, the H100 Graphical Processing Unit (GPU) made by NVIDIA, has an annual power draw 44% higher than the average UK resident!18 This is set to increase, in the next generation chip (the B200) to 150% more power than the average UK resident. The exact amount of power that Google DeepMind, Anthropic, and OpenAI use to train their systems is kept secret, for competitive reasons, but Meta published a report with their latest model release in July 2024, which noted that training their largest model used 16,000 active GPUs. EpochAI, a third-party research organisation, estimates this required 27 megawatts (MW) of installed electricity capacity.19 This is approximately the power supply required for 88,000 UK households, which more than the number of households in York.20

One useful intuition is that each AI chip has the power demand of a person, or maybe soon a household, and each AI datacentre has the power demand of a small city. The large datacentre campuses will be similar to the largest cities. Of course, it is quite remarkable how much power it takes to train the current AI systems, but the steep trendlines point towards this amount of computational power and energy becoming quite small, quite quickly! 

What is the future of AI datacentre and energy demand?

A report by a former OpenAI employee, Leopold Aschenbrenner, extrapolated the current trendlines in computational power increases, and noted that current growth rates imply:

• The largest model in 2026 will be trained on the equivalent computational power of one million of today’s state-of-the-art GPUs and require 1 gigawatt (GW) of power. (Of course, hardware improvements mean it will be a smaller number of more intensive chips, so hereafter we’ll use the unit H100-equivalent for comparison.)

• In 2028, the largest model will use 10 million H100-equivalents of computational power, and 10GW of electricity.

• In 2030, this will jump to 100 million H100-equivalents and 100 GW of electricity.

This is daunting. The 16,000 H100s which Meta used to train their most recent model, which required the same power as York, looks microscopic in comparison to the figures for the simple extrapolations for the end of the decade. Were this trendline to hold, and the length of training runs reaches optimum, the single largest model in 2030 would require 2.15 times more power than the UK’s entire electricity generation in 2021.21 Of course, this is not a prediction, merely an observation of what continuing straight lines would imply. 

Thus far, we have only described trends in training compute, not in the inference of systems (i.e. when the models are being run). If the inference-time compute paradigm which OpenAI have developed using chains-of-thought can be extended further, the computational intensity of inference will rise sharply. This will be compounded by increased frequency of model inference. As AI systems become more integrated into the economy, inference will become the dominant form of AI computing by far. Jensen Huang, the CEO of NVIDIA, expects the amount of inference to go up ‘by a billion times’ (and given the 100 million times increase in training compute in the last decade, we take this estimate seriously!)

SemiAnalysis published an estimate in March 2024 that US ‘AI Data Centre Critical IT Power’ will rise to 56.3 GW in 2028, from 8.5 GW in 2024. Globally, they expect AI datacentre power demand to rise by approximately 40 GW by 2026. These trends provide an important indicator about the criticality of this current moment: it is very much the beginning of the buildout.

The growth in AI datacentres is constrained by energy availability. There is not 40GW of spare energy capacity around the world, and so the cloud providers are taking steps to meet their power demands. Cloud providers have snapped up the limited amount of spare capacity it was possible to buy—for example, Amazon has bought a 960 MW nuclear reactor, and Microsoft has signed a 20-year power purchase agreement with Constellation Energy to reopen an 836 MW reactor at Three Mile Island. SemiAnalysis reports, “[T]he search for power is so dire, X.AI is even converting an old factory in Memphis Tennessee into a datacenter due to the lack of other options.” As an indicator of the intensity of the buildout: to create the power supply for this datacentre, x.ai…

“[P]ut a bunch of mobile [natural gas] generators usually reserved for natural disasters outside, add[ed] a Tesla battery pack, [drove] as much power as we can from the grid, tap[ped] the natural gas line that's going to the natural gas plant two miles away, the gigawatt natural gas plant…[and got] a cluster built as fast as possible."

This project was completed in 19 days, despite the fact that constructing a 100,000 GPU cluster ordinarily takes a year. (It also ordinarily costs $1 billion, but they were willing to spend $4 to $5 billion.)

To fuel further growth, the cloud providers are enabling the construction of new energy assets. Oracle has a permit to build three SMRs and Google announced a partnership which will give them seven SMRs to provide 500 MW for datacentres, starting in 2030. Amazon has partnered with Dominion Energy to build SMRs for datacentres. Most ambitiously, OpenAI asked the Biden Administration to construct between five and seven 5GW datacentre campuses across the US.

The exact level of capital expenditure on AI infrastructure (let alone on energy generation assets) is difficult to disaggregate from the earnings reports of big tech companies. This estimate suggests big tech companies will spend more than $100 billion on AI infrastructure in 2024, and SemiAnalysis estimates that Microsoft will independently spend more than $50 billion. Estimates of future capital expenditure vary from hundreds of billions to nine trillion dollars.

As with many things in AI, it is uncertain, but likely to be big.

4. The UK would power AI datacentres using nuclear power, not wind and solar

We investigated which source of power generation would be most suitable for AI datacentres in the UK, to determine where reforms should be focused. We compared two forms of low-carbon energy—nuclear power, or a blend of wind, solar, batteries backup, and natural gas reserve. We didn’t consider the possibility of using entirely natural gas to power AI datacentres, as we considered this incompatible with emissions aims, though it is a potential approach.

It is important to note that not only do AI datacentres need lots of power, but they need incredibly reliable power. Datacentres require high uptime—their service level agreements typically stipulate “five nines of reliability”, meaning the datacentre can have downtime of 5 minutes and 15 seconds over the course of a year. This effectively dictates the power cannot fail, as, this report notes “even a 25 millisecond power outage could take down the entire datacentre for several hours or even days”.

AI datacentres need reliability for two main reasons:

1. The current technical regime for training large models currently requires synchronisation between compute assets. If an AI datacentre is contributing to a large training run, and it goes offline during a training step, it will disrupt the whole training process.

2. Second, despite the power intensity of GPUs, electricity only makes up a small fraction of the ‘Total Cost of Ownership’: the monthly GPU server hosting costs are just $1872, while server capital costs are $7026. Therefore, having GPUs sit idle is much more expensive than building redundancy.

The renewables, batteries, and gas blend is impractically expensive, and undesirably polluting and land intensive.

To think about how the optimal blend works; it makes sense to layer on the tradeoffs. First, wind and solar are highly intermittent, and so getting consistent output requires building much more than would be naively estimated. (A 1GW solar plant will only actually give you 1GW in the most fleetingly intense moments of the summer, most of the time it will fall below.) Next, because it is sometimes windy when it is not sunny, and sunny when it is not windy, we say that wind and solar have covariance, and so there is always value in having some of each in the mix. Solar and wind are roughly as cheap, so cost is not a major determinant of which to choose. 

There will be moments throughout the day when it is neither sunny nor windy. For these it is necessary to employ batteries. This does not fully solve the problem, because sometimes there will be low wind speeds and clouds for a week, and while this could be solved by building complete battery backup, it becomes enormously expensive. Batteries are cost-competitive if they are being used constantly (roughly every four hours), but they are expensive to sit idle, and so while batteries are efficient for overnight use, it is impractical to use them for a longer period. For extended periods where wind speeds are low and it is cloudy, it is necessary to use a natural gas backup.

We estimated that the cost-minimising way to use a blend of solar, wind, batteries, and natural gas; for every 1 GW of stable output, it would require 8 GW of solar panels, 0.37 GW of wind power, 12 GWh of battery-backup, complete gas-backup, and LNG import capacity of 451 million cubic metres of gas. (Our method is included in the Technical Appendix.)

This presents a number of challenges:

1. The land use. Building 8GW of installed solar and 0.37GW of wind capacity would require 160km² and 41km² respectively. (We assume the wind is onshore, as costs offshore are harder to predict as getting energy back to shore is more difficult).This means that each gigawatt would require more than 200km² of contiguous land. As a point of reference, Cardiff is 140km² and Reading is 40km². This scales very poorly. Some datacentre campuses are much larger than one gigawatt, for example, imagine that Microsoft and OpenAI’s 5GW datacentre campus was powered by renewables in the UK, it would require over 1000km² of contiguous land.

2. The land would need to be next to LNG import capacity. The UK has three LNG import terminals, one in Kent and two in South Wales. The natural gas generation terminal would need to be some distance from population centres to reduce air pollution, but also close enough to the import terminal that gas pipelines are not necessary. It is either necessary to find ways to build these generation facilities in Kent or South Wales while being contiguous with the solar and wind farms, or it might be necessary to build a new LNG import terminal, but neither approach scales well, and the latter would add substantially to costs.  

3. Emissions. As natural gas would generate 28% of the electricity, over its expected life cycle, the renewable blend would produce 40% more carbon emissions than equivalent nuclear capacity.

4. Cost. The levelised cost would be £106/MWh. This would be internationally uncompetitive for creating an AI datacentre—the equivalent cost for Texas to produce energy with the same carbon footprint would be £74/MWh. Should emissions be desired to fall further, the output electricity would only become even more uncompetitively expensive. 

5. Safety. Even when natural gas is only used 28% of the time, the mix leads to 27 times more ‘expected deaths’ per TWh (mostly from air pollution), when compared to the risk of accidents with nuclear power.

As a reminder, these numbers are for 1GW of consistent output, and global datacentre buildout is expected to be around 40 GW by 2026.22 It is probable to expect 10s of gigawatts of power demand in the coming years. Renewable approaches will not be able to scale to this level, and to provide the concentration of power required onto the largest datacentre campuses.

Nuclear is more suitable because it is reliable and safe

Contrary to wind and solar, nuclear power has very stable output, which is ideal for meeting the ‘five nines’ requirement that datacentres have – 99.999% reliability.23 The US has a nuclear capacity factor of 92.5% as a benchmark. The chart below shows US total energy generation in a week in March last year; note that nuclear is the green block at the bottom—extremely stable! 

24

Furthermore, there has been an international realisation that nuclear power is safe, clean, and necessary to meet climate goals. 20 countries at COP 28—including the US and UK—announced their intention to triple global nuclear energy capacity by 2050. The world’s biggest investment banks have announced their intention to finance this aim, Italy and India announced plans to accelerate the construction of nuclear power, and Japan is looking to reopen 13 nuclear reactors. Since then, the US has said it will add 200 GW of nuclear power by 2050.

A negative perception of nuclear energy has come from its association to nuclear weapons, and the infrequent-but-visible reactor meltdowns and subsequent evacuations. The Pripyat Ferris Wheel and empty swimming pool have a prominent effect on our collective psyche towards nuclear energy safety, but we have no similar association for the Banqiao Dam Collapse which killed 171,000 people in 1975. 

When compared to other sources of energy, especially practical stable generators alternatives, nuclear power is comparatively much safer and cleaner per TWh of generation.

25

There have been three high-profile reactor meltdowns: Three Mile Island (1979), Chernobyl (1986) and Fukushima (2011). The meltdown at Three Mile Island caused no deaths either directly or indirectly, and the radiation exposure for 2.2 million people who lived near to the New Jersey plant was, “approximately the same radiation dose as flying from New York to Los Angeles”.26

Our World in Data, an independent research organisation, has reviewed the death tolls for Chernobyl and Fukushima. Their literature review estimated the Chernobyl meltdown caused between 300 and 500 deaths; 30 direct deaths, and the remainder were indirect. At Fukushima in 2011, there were no direct deaths in the disaster. There were 40 to 50 injuries, and 7 years after the accident, it was reported that one worker died from lung cancer caused by radiation exposure at the event. However, there was a mass evacuation, which is estimated to have caused 2,313 deaths, from the physical or mental exertion of evacuation (from care homes and similar places). Disentangling which of these deaths were attributable to the evacuation following the meltdown, compared with the wider impact of the earthquake and tsunami, is necessarily difficult.

There is a particular dissonance between attitudes to fossil fuels and nuclear power. Unlike nuclear power, fossil fuels are continuously and gradually reducing the life expectancy of billions of people, but there is never a discrete moment where this is felt more acutely. This report notes that, “moving to Tokyo would triple the populations’ increase in risk of death [because of air pollution], compared to moving them back to the remaining off-limits zones in Fukushima.” This paper estimates that the slowdown in nuclear power construction following the Chernobyl meltdown caused the loss of 33 million expected life years in the UK alone, or roughly 400,000 people, because of particulate poisoning.

To summarise, the UK will not be able to produce cost-competitive renewable energy for AI datacentres, that scales to the 10s of gigawatts required. However, nuclear power has all the necessary attributes—it is cleaner, safer, has more reliable generation—and as we discuss in a further section, could become internationally cost competitive too.

5. Going without AI datacentres would be a mistake

A sceptical line of argument might say that while AI progress is happening; and AI datacentres and power will grow dramatically; and nuclear power will be the dominant energy source; it does not necessarily follow that the UK should be concerned with hosting AI datacentres. The strongest form of this argument claims: the pattern of economic history is that the gains from general-purpose technologies tend to come from adoption, and perhaps UK residents and businesses could buy access to AI datacentres internationally, while the UK could focus on the ‘highest value’ parts of the AI value chain.

This argument is not enough—going without AI datacentres would be a mistake. The economic doctrine that the UK can sit atop the value chain, and selectively choose to engage with ‘high value’ industries has led to a hollowing out of industry, and left the UK without growth. Hosting AI datacentres will enhance the UK’s economic security, allows directed growth into former industrial areas, and enables future frontier growth.

A large fraction of the UK’s capital stock will be in AI datacentres

It is very likely that computational power becomes a critical input into the production of goods and services, in a manner similar to energy. Just as the venture capitalist Marc Andreessen commented in 2011, ‘software is eating the world’, the information processing capabilities of AI systems will become tightly knit into all existing business processes and future ones. This will be especially true for the UK’s professional services exports. As a result, a very large fraction of the UK’s capital stock will be created, stored, and run in, or at least depend upon, AI datacentres. The UK will want these AI datacentres to be here, rather than overseas, connected through an undersea cable, to ensure it can protect these assets.

Furthermore, precisely because the gains from AI come from adoption, the UK needs to ensure access to computational power. As demand for computational power rises globally, it could be the case that UK businesses are unable to access the computational power they need. Right now, the Microsoft CFO said on a recent earnings call that revenue growth in their cloud business was 33% but, “[d]emand continues to be higher than our available capacity.”

For decades, the UK decided to go without energy self-sufficiency. It would be imprudent to repeat the same mistake for computational power.

Directing growth to former industrial areas

The work to adopt and develop AI applications is likely to centre around London. The Government’s AI Sector Study shows that 75% of UK AI companies are based in London, the South East, or East of England. AI application developers are likely to agglomerate here because London has the best venture capital ecosystem and AI talent density outside San Francisco. Furthermore, adoption of AI systems is likely to focus on automation of business processes in professional service domains and scientific endeavours initially, which is also likely to begin within the ‘Golden Triangle’.

AI datacentres do not depend on network agglomeration in the same way. The construction and operation of these datacenters can be much more readily directed, to areas outside the Golden Triangle with strong industrial traditions. Very rarely does the opportunity of investment which is location-independent come along. There can be thousands of skilled jobs in the nuclear and datacentre construction and operation industries.  

Furthermore, there are spillover benefits from developing these industries. UK workers will learn how to build nuclear reactors from the Korean Electric Power Corporation, who construct reactors for 25% of the price; or how to build AI datacentres with a Power Usage Effectiveness of 1.1, from Google.

If the UK wants to participate in the frontier growth of the future

The UK has enormous strengths in science, through its world-leading universities and research institutes. The process of research and development is likely to be transformed by AI systems in the coming years, and so maintaining the UK’s scientific advantage and the prospect of future growth it offers is likely to require differential integration of AI systems into the research cycle. It seems very likely that future UK growth will be downstream AI-enabled scientific discoveries, for example, new drug discoveries from future versions of AlphaFold. In such cases, computational power will be a critical input—it is not sensible to solely depend upon international markets for a resource which is so central to future prosperity.

Simply, AI datacentres are an asymmetric bet—if the AI ‘bulls’ are correct, then it is crucial the UK has datacentres for future growth and security, and it is crucial the UK expands its compute industry. If the ‘AI bears’ turn out to be correct, the rollout of AI systems will be a multi-decade-long integration of a general-purpose technology and so operating the AI infrastructure will provide jobs and tax revenues for public services; as well as spillovers in knowing how to build cheap power generation in the UK. As private investors are providing the capital, there is no scarce resource being consumed by creating the conditions for them to invest. The bar for deciding to pursue AI datacentres and nuclear power generation is low, as the UK needs growth so dearly.

6. Nuclear power is slow and expensive to build, but it could be cheaper and faster

UK nuclear power construction is really slow and expensive.27 No private investor who wants to power an AI datacentre would choose to build a nuclear power plant in the UK, at present. However, there is a truly remarkable amount of low-hanging fruit to be picked, to become internationally competitive.

In this section we diagnose the reasons for the high costs, and in a later section, we propose a reform package based on this diagnosis.

To set the scene:

• Hinkley Point C is forecast to cost £10 million per MW, which is 4.5 times more expensive than South Korea, £2.24 million per MW.28

• Construction for Hinkley Point C has been delayed to 14 years, from nine years planned. Sizewell C is due to take 12 to 15 years to build. On the contrary, the median time to build a nuclear reactor since 1990 has been under six years. Between 1970 and 2009, Japan built 60 nuclear power plants in a median time of 3.8 years

• Hinkley Point C took six years to progress from initial consultation to final approval. The consultation process for Sizewell C began 12 years ago, and EDF will make a final construction decision in Spring 2025. By comparison, France and France took just three years and four and a half years respectively to approve a plant with the same reactor (the European Pressurised Reactor, or EPR-1600.29

• Three UK projects have been abandoned in the pre-construction phase since 2018 because of financing concerns.30

The total cost of nuclear power is halved if you can borrow cheaply

The most important thing to understand about nuclear power projects is that the cost of capital dominates the overall cost of the project. Interest was approximately two thirds of the cost of Hinkley Point C.31 This report estimates the breakdown of Hinkley Point C costs as follows:

The cost of the electricity is very sensitive to the cost of borrowing for construction: a 2020 report by the International Energy Agency uses a prototypical EPR-1600 in France, and says when the cost of capital is 3%, instead of 10%, the levelised price of energy is reduced by more than half (53%).32

Why does this matter for our purposes? There are two implications: 

1. Speed matters; not just because the energy generation can begin, but because it means borrowing can stop.

2. Certainty matters; when investors perceive a project to be less risky, the more cheaply they provide their capital. This report said EDF were forecasting a 9% return on their capital at Hinkley Point C.

Any reforms to planning and regulation not only save money directly through simplification, but they also reduce project risk and timelines, thereby leading to indirect savings on interest payments.

Construction costs can be halved again by building reactors ‘in fleets’

South Korea is able to build nuclear so cheaply because it builds reactors ‘in fleets’, where it repeats the same reactor design of  reduces its cost of nuclear power by building ‘fleets’ of 8 to 12 times. This repetition creates ‘learning’ between projects. (Learning describes the cost declines driven by the cumulative experience of having done something before.) The clearest example of learning is performing technical tasks better. Because nuclear power plant projects are so large, these gains even exist within projects: EDF has claimed that welding for the second reactor at Hinkley Point C is happening twice as quickly. But an equally important type of learning comes from the developers knowing what the regulators want—construction is only a small fraction of the activity to start a nuclear reactor, lots of effort is spent on quality assurance and safety. When a reactor is repeated multiple times, the mutual understanding between regulators and developers transfers across projects.

Fleets also support supply chain certainty. The nuclear supply chain requires higher quality assurance standards and more intensive component testing than ordinary industrial projects, which often necessitates a separate supply chain. If there is a large number of reactors to be built, which have been approved previously, the nuclear supply chain could produce components without needing to specify in which specific reactor the parts will be used. The same applies for the supply chain of skills. When there is a clear pipeline of construction projects, investing the time to become a nuclear welder is a sensible career choice, as it will raise wages for the long term. 

The UK does the opposite of building in fleets. The last UK nuclear project to be completed was Sizewell B, in 1995. This reactor was a Pressurised Water Reactor, and it would be 21 years before construction on our next nuclear reactor began at Hinkley Point C. This was a different design, the EPR-1600. As we’ve noted, this basic reactor design had been used previously in France and Finland, however EDF has said the Office for Nuclear Regulation (ONR) required 7,000 design changes including 25% more concrete and 35% more steel for the reactor to be approved in the UK. The ONR disputes this, but irrespective of who was responsible, when the reactor becomes increasingly dissimilar, it is evermore difficult to transfer learning from the previous sites.

The combination of low levels of regulatory certainty, and no clear precedents for the one-of-a-kind reactor, mean that the supply chain might not have the confidence to justify preparing components until after the final construction decision is made. Likewise, with large gaps between projects, the ‘supply chain of skills’ is weaker: workers have lower incentives to develop the skills required for nuclear projects, and move away to other professions. For example, the delays to the approval of Sizewell C mean that nuclear welders will be unable to transition from Hinkley Point C directly into a new project.

The nuclear approval process is vetocratic—there is no positive force in the system to push back against time delays and cost increases.

The next largest contributor to slow and expensive nuclear projects is the diffusion of state responsibility for approvals. For a new nuclear power plant to be approved:

• The Office for Nuclear Regulation must grant a Nuclear Site Licence, which covers the location, the technology, and operation against accidents. The ONR is an independent statutory authority, and a public corporation in the Department for Work and Pensions.

• The Environment Agency must grant an Environmental Permit to cover the environmental effects of operation, if the reactor is in England; and in tandem with the respective devolved administration if it is elsewhere.33

• The Secretary of State in the Department for the Environment, Food, and Rural Affairs must confirm regulatory justification, which states that the benefits of using ionising radiation outweigh the costs.

• The Secretary of State for the Department for Energy Security and Net Zero must approve a Development Consent Order (DCO), which involves multiple rounds of consultation and an Environmental Impact Assessment (EIA).

The ONR is an independent statutory authority, responsible for nuclear fission safety. This means its mandate and responsibility is to prevent accidents that could be caused by nuclear power plants. The ONR has no authority or responsibility, to weigh the counterfactual risks from not building nuclear power: for example, the approximately 33 million life-years lost in the UK due to air pollution since nuclear power plant construction was slowed following Chernobyl, or the environmental damage from ongoing greenhouse gas emissions, or the impact of high energy prices on people or businesses, or any manner of other challenges.

The incentive and responsibility of the ONR is to minimise the risk of accidents from nuclear. The global standard for safety regulation, required in all industries in the UK, is that the risk of ionising radiation exposure is ‘As Low As Reasonably Practicable’ (nb. in some contexts this might be ‘As Low As Reasonably Achievable’). Because the ONR is not set up to balance aims, such reasonableness is defined as anything which can improve reactor safety, until a measure can be proven to be ‘grossly disproportionate’.

Similarly, the consequences of long and uncertain Environmental Impact Assessments are not considered. The EIA for Sizewell C was 44,260 pages; and for Hinkley Point C it was 31,401 pages. The issues of EIAs are not unique to nuclear power, and so we leave these to other sources, however, a report by Sam Dumitriu claims that EDF have spent, “hundreds of millions [of pounds]”, at Hinkley Point C to install underwater speakers, in order to deter roughly 112 fish from entering the water cooling system. 

Unlike the UK, in South Korea, the Nuclear Safety and Security Commission reports directly to the Prime Minister.34 This report suggests that political oversight changes the incentive equilibrium for regulators, to more appropriately balance the costs and benefits of incremental safety regulations. Though the ONR is independent, it is also a public corporation within the Department for Work and Pensions, and therefore there is less Ministerial interest or bandwidth for seeing that nuclear power gets built.

‘Regulatory justification’ has been misapplied

‘Regulatory justification’ is a requirement that stems from a 1996 EU directive that stipulates the benefits of ionising radiation for the production of energy must outweigh the costs. This does not seem, in principle, to be a bad idea—who would be for using ionising radiation where the costs exceeded the benefits? However, the requirement applies to each ‘practice’, which is an instance of the use of ionising radiation. It is currently interpreted that each reactor design is its own practice which must be separately assessed for regulatory justification. There are good legal arguments that nuclear power, or broad characteristics such as using low enriched uranium and light water as coolant and moderator, should be a single practice for which ‘regulatory justification’ is established once and for all.

Because of ‘functional separation’, authority for this decision sits with the Department for the Environment, Food, and Rural Affairs, and the decision takes two years. This is duplicative, because the purpose of the planning process is to weigh the relative merits of a new project. France, Finland, and Sweden incorporate the ‘regulatory justification’ into their planning decisions.

This superfluous step increases project uncertainty and duration and therefore raises the total costs of the project by causing longer and more risky borrowing.

To summarise, there are many actors in the system who can say ‘no’, or who might add incremental delays and cost increases to nuclear power plant construction, which amounts to a de facto ‘no’; but there is no positive force which pushes back against slowness, expensiveness, and the counterfactual damages of the two. With an approach that weighs cost and benefits, including the knock-on impacts to speed and certainty, the UK can build an internationally competitive nuclear planning and regulatory regime.

7. The UK should create ‘Special Compute Zones’

The purpose of this reform proposal is to solve an incongruence:

• The UK’s AI datacentre capacity is an imperative for economic security, growth in former places of industry, and long-term prosperity; as this previous section set out.

• But the planning and regulatory approval process for building new AI datacentres and associated power cannot permit the amount of construction, at the speed required.

Below is our proposal on how this might be resolved.

Within ‘Special Compute Zones’, there is an alternative planning and regulatory approval process for nuclear reactors, AI datacentres, and the transmission and networking infrastructure they require. The goal is to provide the certainty, speed, and hence, the opportunity to be cheap, that would make the UK the most competitive place in the world to build AI datacentres.

Within the Zones, there would be ‘deemed consent’—meaning the default answer to construction is ‘yes’—and permission to construct would have to be challenged within three months of an application. Ordinarily, planning decisions weigh the relative merits of each project; but within a ‘Special Compute Zone’, it would be decided that by creating a zone, this cost-benefit has already been considered, and therefore approval would depend on a ‘condition-based approach’. This means that if a developer can prove the project meets particular standards, it goes ahead. There is precedent for location-based policies from Spain and the EU.35 In Spain, the Government passed a decree which allowed renewable projects to forego the Environmental Impact Assessments, so long as the project met some conditions:

• Wind and solar projects are below 75 MW and 150 MW respectively.

• Projects are in areas of low or moderate environmental sensitivity.

• Grid connection lines are not longer than 15km or above 220kV.

• Authorities do not lodge an objection within two months.

This report notes the change doubled the speed of projects, and increased the forecast of solar construction by 13GW for 2030.36

The EU has mandated “Renewable Acceleration Areas”, as of September 2023, which require Member States to designate at least one area by February 2026. To implement them, Member States prepare, “a mitigation ‘rulebook’ consisting of a set of rules on mitigation measures to adopt in the specific area, aimed at avoiding or where not possible significantly reducing the environmental impacts resulting from the installation of projects in those areas.”

The Secretary of State in the Department for Science, Information, and Technology would be able to provide a single sign-off for the projects. This means nuclear power, AI datacentres, and networking and transmission infrastructure would be approved together, to improve project certainty. As with renewable zones, developers would apply to the Secretary of State, who would have three months to object to the application for construction, after which the project would automatically have permission. This aligns the incentives of governments to quickly respond to applications, as the average consideration period of a Development Consent Order is 22 months.37 Furthermore, a non-objection procedure, rather than a positive decision on each site, reduces the risk of judicial review. Judicial review proceedings could slow construction by roughly two years; and so developers are likely to want clarity that everything possible has been done to avoid judicial review. In order to further reduce the risk of judicial review, the primary legislation for Special Compute Zones could:

• Specify the grounds on which a challenge to the Act can be brought.

• Exclude “oral renewal” for judicial review.

• Ensure reviews are brought for procedural reasons and not ones based on the principle, policy or merits of the proposal.

One area for further investigation is to what extent all of the application process needs to be frontloaded before construction can begin, or whether this can be parallelised.  For example, some environmental permitting and mitigation (i.e. dealing with environmental effects of plant operation) might be performed concurrently with construction, subject to a condition that the plant cannot start operation until they have been addressed. There is international precedent for this: nuclear power developers in the US can opt to licence their reactors through ‘Part 50’ of the U.S. Code of Federal Regulations, which grants them separate construction and operating licences. (Ordinarily developers use Part 52, which grants construction and operation licences together.) The developer takes on some risk that they aren’t granted an operating licence, when they begin construction without. 

The “Special Compute Zones” will need to permit nuclear power operators to make ‘behind the meter’ power purchase agreements with datacentre operators. The grid network fees are 20-25% of the cost of grid power, and so it is necessary to make the UK a competitive place to build.

How should the Zones be designated?

The primary legislation for Special Compute Zones could designate all former nuclear, coal, and natural gas power plant sites, former steel plants, and ports as Special Compute Zones, and give the Secretary of State the power to de-designate or designate individual sites or classes of sites. The primary legislation would need to designate sufficiently broad classes of sites to avoid being a hybrid bill. This is because hybrid bills have a much longer parliamentary process, and therefore would take much longer to pass, and so it would make it much more difficult for the UK to participate in building AI datacentres. By making all of these classes Special Compute Zones through primary legislation, and providing the Secretary of State power to object and de-designate, the risk of judicial review is considerably lower than if the Secretary of State had to make a decision to designate each particular site. This in turn, means it is more likely that AI datacentres would be built.

The Secretary of State should also be given the power to determine which environmental conditions apply to each zone, based on a high level environmental assessment, so that environmental protection is provided. Compliance with those conditions would, under the Act, be a basis for removing the need for EIA.

Former sites of energy generation and steel production are especially suitable to be Special Compute Zones as they have grid connections and previously had environmental impacts that are likely equal to or, indeed, much greater than nuclear power plant operation. This improves the ease and strength of argument for streamlining the regulatory and planning process. Ports are suitable for conventional nuclear because bringing materials to the site by boat is substantially easier than to other sites. For example, at Hinkley Point C, wharfs needed to be constructed for delivery to reduce the number of lorries.

There may be additional options for planning and regulatory approval, worth including as alternative pathways, although they are unlikely to deliver results within the necessary timeframes as the proposal suggested above. For example, the local planning authority could grant consent for new nuclear power and datacentres, provided the application met the conditions required by the Special Compute Zones. To incentivise local governments to allow more power and datacentres, they might be permitted to retain 100% of the business rates in perpetuity for projects approved by this mechanism.

What should the conditions be?

For deemed consent to apply, the reactor must have been approved by a recognised international regulator (i.e. the US or the EU). Introducing international recognition would speed up both approval and construction.38 Practically, international recognition would allow international teams who have already built fleets of reactors to come to the UK, upskill local workers, and replicate their previous work, saving costs with the lessons they learned during that work.

The conditions for radiation dose exposure for workers and the public could be taken from the Ionising Radiations Regulations 2017.

Some conditions would relate to environmental impacts of construction and operation, to streamline and replace environmental permitting and impact assessments, and could include options such as requiring the developer to pay into a rewilding fund. This will ensure environmental effects are addressed without creating uncertainty and expensive delays.

Some conditions would address questions generally addressed in Nuclear Site Licences. For example, there should be requirements about emergency planning in scenarios with technical failure, and population density around the site.

Crucially, these conditions should differ between conventional nuclear and SMRs. For example, safety requirements should be proportionate to the size of the plant. 

How should we deal with ‘regulatory justification’?

Because regulatory justification is an EU directive, it would either need to be incorporated into a planning decision, per the French, Finnish, and Swedish approach, or it could be addressed by a Regulation 9 decision declaring all classes of nuclear power as a new justified practice or issuing a Regulation 12 determination that nuclear energy is an existing practice due to the operation of Magnox, Advanced Gas Reactors, and Sizewell B before the introduction of the directive.

Technical Appendix

The model solves for the optimal solar, wind and gas combination to minimise the levelised wholesale cost of electricity.39 The model generated 100 possible sets of 365 days outputs of solar and wind, and optimised the selection over 4000 possible combinations of solar, wind and batteries output. 

Solar

Solar energy has two sources of variance in the model - one caused by variation in average output across months and the other caused by daily variation caused by cloud cover and similar. Average output varies due to the northern hemisphere tilting away from the sun in winter - which both directly reduces incidence by trigonometry but also indirectly does so by forcing sunlight to pass through thicker air before reaching the Earth’s surface. Average solar output in December is thus 72% lower than its level in summer - with this effect being much greater than temperature variations imply as air can move around the earth. Cloud cover also reduces output, giving daily variation following a gamma distribution with shape 3.5. In the UK, solar energy has a capacity factor of 10%, a levelised cost of £49/MWh, and 1GW occupies 20km². In Texas, the levelised cost ranges from $24 to $96, giving a central estimate of $46/MWh. Solar output in Texas is much less volatile, declining by only 40% in winter.

Wind 

Wind energy varies in output due to variations in wind speed, which occur both across days and seasons. During winter, UK wind speeds are 25% higher than summer, giving it the beneficial property of being weakly negatively correlated to solar. However, the underlying energy stored within wind goes as the cube of wind speed, as each individual molecule’s kinetic energy goes as the square of wind speed and linearly more air molecules hit the turbine’s blades per second as wind speed increases. Although wind power in practice hits diminishing returns and eventually decreases in power output in wind speed, for most of the wind speed distribution in the UK we remain in the cubic - and so highly volatile - portion. This is compounded by the high volatility in the underlying distribution - Rayleigh, with a standard deviation of half its mean. In the UK, wind has a capacity factor of 27%, a levelised price of £53/MWh and 1GW of wind occupies 110km². The cost ranges from $24 to $75, implying a central estimate of £38/MWh.

Batteries

Batteries offer the ability to smooth variable-output processes and provide the constant stream of power needed for efficient datacentre operation. Additionally, as they are primarily used in summer the scale-up of solar over the day still only means 12GWh are required for continuous operation. However, they still remain impractically expensive to participate in balancing on timescales longer than a day, and also remain limited by generally only being able to charge at the same rate as they discharge - meaning that very large amounts of excess generation by renewables across seasons is difficult to exploit. Existing grid-scale storage has a price of $400/MWh and lasts for at least 2000 cycles. 

Gas

Gas has a levelised price of £136/MWh, just over half of which is carbon pricing. The capacity factor is close to 100%, and the area occupied is very small in comparison to renewables. In Texas, the cost is $39-$68, so a central estimate of £41. 

Results

In the optimal allocation, 8GW of solar and 0.4GW of wind was procured, at a cost of £106/MWh with 28% of electricity being gas generated. If it was possible to sell to the grid at £30/MWh during periods of renewables surplus (plausibly too high for the unsubsidised price as this is so correlated nationally), then the optimal solar quantity rises to 12GW, the optimal wind quantity stays unchanged on 0.4GW and the carbon intensity drops to 0.17GW, and the levelised cost drops to £98/MWh. The levelised decline is so small in response to the grid connection because the previous allocation only wasted 18% of its total output as curtailed renewables, so without a large adjustment on the renewables side (which are unprofitable without a guaranteed price on sale induced by subsidies) this leads to little change in prices. Given that the grid connection would of course have to be built and maintained, this renders its provision unlikely to reduce overall costs. Total expected deaths/GW-year are 7.06. The cheapest equivalent-emissions option for Texas costs £74/MWh, being composed of 0.7GW of wind and 4.7GW of solar. If Texas faced the same carbon pricing as British gas, the cost remains substantially lower at £89/MWh. 

Subscribe now