Skip to main content

[@DwarkeshPatel] I thought superintelligence was a decade away. Not anymore.

· 21 min read

@DwarkeshPatel - "I thought superintelligence was a decade away. Not anymore."

Link: https://youtu.be/1NHPDzpfR6Y

Duration: 44 min

Transcript: Download plain text

Short Summary​

The narrator, a researcher who spent the past year and a half arguing that AI's key missing capability was learning from on-the-job experience, now argues that automating AI research is only "a couple of years — possibly a couple of months" away, with a median estimate of 2028 for AIs matching the best human AI researchers. He expects a compressed jump in 2027–2030 and splits the odds 50/50 between a software-only singularity and a "spark" scenario. The evidence: Anthropic's Claude now leads 26% of Anthropic's AI R&D work (up from 1% in February), and METR found Anthropic's rate of AI progress doubled over a year.

Key Quotes​

  1. "If I had to pick my median year for when automated researchers will be as good as the very best human AI researchers at doing AI research, I'd guess 2028." (00:08:18)
  2. "the heaviest 10% of AI users amongst OpenAI's researchers went from using about $1,000 a day of internal coding agents at the beginning of the summer to over $7,000 a day by the end of August." (00:01:34)
  3. "While current AI systems are starting to automate mathematical proof, there is currently very little evidence that they can ask powerful questions or find profound new fields of mathematical inquiry." (00:03:47)
  4. "Current AI systems have solved a millennium prize problem all without the ability to ask powerful questions or find profound new fields of mathematical inquiry." (00:04:04)
  5. "As to which one will actually happen, I think it's basically a coin toss." (00:15:57)

Detailed Summary​

Detailed Summary

1. The revised timeline for automated AI research​

After three months of updating his views, the researcher has moved from thinking that AI research automation was at least “a better part of a decade” away to estimating it could be only “a couple of years — possibly a couple of months” away. His median estimate is 2028 for AIs to match the best human AI researchers, conditional on labs continuing at full speed without intentional or governmental slowdown.

  • The near-term scenario is a compressed period of progress in which AIs become at least as good as the best human experts across every field by 2030, possibly as early as the following year.
  • The researcher assigns at least a 25% chance that best-human-level AI research automation will not arrive until after 2032.
  • That downside case is driven primarily by the possibility that continual learning will remain unsolved for several years and models will not compensate through other capabilities.
  • Once AI research is fully automated, he expects the resulting systems to reach top-human performance across many domains within months, putting the broader transition around 2028 or 2029 even in his median case.

2. Evidence that AI R&D is already accelerating​

The central empirical argument is that AI-assisted research is no longer marginal: adoption and measured progress rates have risen sharply over the past year. The researcher combines Anthropic’s internal deployment data with external evaluations and forecasts from forecasting platforms.

  • METR spent three months embedded inside Anthropic and concluded that Anthropic’s rate of AI progress on well-scoped tasks was twice what it had been one year earlier.
  • METR attributed most of that acceleration to Anthropic’s extensive use of AI in its own R&D.
  • In a September OpenAI blog post, the heaviest 10% of AI users among OpenAI researchers increased their internal coding-agent usage from about $1,000 per day at the beginning of summer to more than $7,000 per day by the end of August.
  • Anthropic reported in September that Claude was leading 26% of its AI R&D work, up from only 1% as recently as February.
  • A new Metaculus report also found that Anthropic’s rate of AI progress had doubled from a year earlier under a takeoff model.
  • Ajeya Cotra noted that if AI is nearly doubling the rate of capability progress and the acceleration remains smooth, the naive implication is reaching extreme superintelligence within a year.
  • Some well-informed researchers with strong forecasting track records now expect AIs to beat the best humans at AI research as early as 2027.

3. AI’s rapid mathematical record and remaining limitations​

The researcher uses AI’s improving mathematical performance as evidence that systems are moving from simple calculation toward difficult research. He also distinguishes automating existing proofs from choosing new and important questions to investigate.

  • In 2023, models could not reliably add four-digit numbers; in 2024, they barely solved high-school math competition problems; and in 2025, they won gold at the International Math Olympiad.
  • Earlier in the year covered by the episode, AI systems began solving niche open problems, and a month before the episode OpenAI had solved a Millennium Prize problem in 88 hours.
  • Hundreds of mathematical results that leading mathematicians and computer scientists had spent decades trying to establish are now being produced as side quests while labs hype future model releases.
  • Toby Observed that current systems are beginning to automate mathematical proof, but there is little evidence that they can formulate powerful new questions or identify profound new fields of inquiry.
  • AI labs prioritize mathematical work mainly when it produces an algorithmic recipe that empirically improves training efficiency, not mathematics pursued for its own explanatory value through tools such as topology, group theory, or the Cartesian grid.

4. Would a modern model have invented the Transformer?​

The episode uses the Transformer as a test of whether AI systems could reproduce the kind of research synthesis that has historically produced major architectural breakthroughs. The researcher presents the historical record as challenging, but not decisive, evidence against that possibility.

  • Sholto Douglas maintains a perennially unfinished blog post arguing that the Transformer could not have been invented two years before it actually appeared.
  • Chris Olah said at the time that the architecture was a natural combination of existing ideas, including attention and residual networks, pushed to aggressive limits.
  • The Transformer was published in 2017, following AlexNet in 2012, but only became clearly superior after roughly half a dozen additional tweaks and innovations already present in the literature had been combined through many small experiments.
  • The researcher argues that Claude 7 could have invented the Transformer if Chris Olah had asked it to try hard.

5. Why algorithmic progress could accelerate​

The researcher’s acceleration mechanism has three components: automated systems would be faster, individually highly capable, and much more numerous than human research teams. He then adds an important constraint: multiplying researcher labor does not multiply the speed of experiments that remain constrained by compute.

  • Today’s AI systems already operate at roughly 10 times human speed, and automated researchers working continuously could complete around 50 times more work per calendar week than a human researcher.
  • That 50-fold work advantage would not necessarily produce 50-fold faster AI progress, because experiments that cannot run faster would remain compute bottlenecks and would exhibit steeply diminishing returns.
  • The researcher estimates that around 80% of algorithmic progress at a frontier lab is driven by its 50 leading researchers, connecting this concentration to Zuckerberg’s willingness to pay hundreds of millions of dollars for top AI talent.
  • He characterizes AI research as closer to formalizing a theorem than writing a novel because it decomposes into modular subtasks that can be checked independently.
  • His illustration is the image of clouds jointly formalizing Fermat’s Last Theorem and then a swarm trying a thousand different approaches across pipeline stages.
  • Real AI companies already divide those stages among separate teams working on data filtering, synthetic reinforcement-learning environments, attention kernels, and related systems.
  • He says AI companies must quadruple their R&D compute and double their workforce every year merely to maintain the current rate of progress, indicating that additional research ideas are already becoming harder to find.

6. Two scenarios: software-only singularity versus a “spark”​

The researcher treats the future as a coin toss between two feedback regimes. In one, AI improves faster than research ideas become scarce; in the other, research ideas become harder to find faster than AI improves.

  • If AI becomes smarter faster than ideas become harder to discover, a software-only singularity occurs in which a swarm builds ever-smarter versions of itself and reaches superintelligence within months, without adding compute beyond the physical limits of current chips.
  • If ideas become harder faster, the “spark” scenario delivers three to six years of AI progress at the current rate within one calendar year, after which progress simmers until additional compute can be built.
  • He assigns roughly 50% probability to the spark scenario and 50% to the software-only singularity.
  • Even the spark scenario is expected to compress roughly four years of progress into the first calendar year after research automation, a change compared to moving from GPT-3.5 to “GPT-6 Astra” in 12 months.
  • GPT-3.5 could write a passable limerick and autocomplete a trivial Python function, whereas the imagined GPT-6 Astra could build entire video games and produce Fields Medal-worthy mathematical contributions in routine volumes.
  • Repeating the last four years’ progress—described as moving from a preschooler to a savant intern—starting in 2027 or 2028 would yield an AI that is vastly superhuman in domains that can be somewhat verified.
  • One example is outperforming Nvidia at designing the next generation of AI accelerators.
  • With equal market access, such a system could make more money than the current human versions of Citadel and Jane Street, while remote access to a materials-science laboratory could make it better than humanity at finding a room-temperature superconductor.

7. Reinforcement learning, general skills, and real-world tasks​

Current reinforcement-learning environments favor tasks whose outcomes can be checked automatically inside a data center. The researcher argues that scaling this training process will cultivate general learning skills that let AIs ramp up in domains that cannot currently be verified so easily.

  • Existing RL uses automatically gradable tasks such as passing unit tests for a million lines of code or solving additional open mathematical problems.
  • That setup does not directly cover building a business from scratch, winning court cases, trading profitably, or helping a candidate win an election because these outcomes require real-world interaction.
  • The researcher expects scaled RL training to give AIs general skills rather than merely storing domain-specific answers in their weights.
  • He defines those general skills as gathering information, planning, pushing through mistakes, tracking learning, and collaborating.
  • During recursive self-improvement, automated engineers would create tens of thousands of times more RL environments than exist today.
  • Those environments would be larger, longer-running, higher-quality, more open-ended, and cover a much broader range of activities.
  • The researcher compares AIs navigating such virtual worlds to “a Bugatti relative to a horse and buggy.”

8. Evolution, deployment, and rapid accumulation of experience​

Human general-purpose intelligence emerged through evolution’s imperfect optimization process, while AI training is a more deliberate effort to optimize directly for present-day capabilities. The researcher uses this analogy to argue that deployed AI could acquire years of useful experience within days.

  • Evolution optimized a crude verifier—whether an organism’s genes passed on—against ancestral survival tasks such as hunting, fending off predators, and maintaining tribal standing.
  • Those tasks represented only a tiny subset of what humans now do, yet the resulting architecture became a general-purpose learner.
  • AI training is a deliberate analog of evolution, but it uses tasks closely matched to success in the modern world.
  • Labs are explicitly trying to improve transfer by ensuring that performance improves on difficult tasks held out of training.
  • Even a superintelligent learner cannot discover everything from inside a data center and would need real-world experience.
  • The researcher nevertheless expects deployed AI to acquire that experience faster than people anticipate and beat top human experts within months.
  • Napoleon reportedly stacked victories against larger European armies by age 27, while Rockefeller was operating one of the world’s largest refineries by age 25, only two years after entering the oil business.
  • These examples are intended to show that a couple of years of focused experience can suffice for exceptional performance, rather than decades being strictly necessary.
  • A hypothetical “shadowing” AI could read every company email, Slack message, and document on its first morning, then use copies of itself to interview every employee and major client in parallel by the end of day one.
  • Such a system still could not access feedback that arrives many months later, because that feedback will not yet exist.

9. Physical scaling under the spark scenario​

Even if software progress slows after the initial spark, the researcher expects jaggedly superhuman systems to continue expanding the physical substrate on which they run. The limiting factor becomes the time needed to design chips, construct fabs, and manufacture data-center infrastructure.

  • Even without AI assistance, global AI compute has been growing at roughly three times per year over the last half-decade.
  • In the spark scenario, superhuman AI would keep scaling compute by designing new chips, laying out fabs, and teleoperating robots to manufacture parts of fabs and data centers.
  • This physical expansion takes years, unlike the software-only singularity, which could reach its endpoint within months.
  • The resulting endpoint is described as a superintelligence that is to humans as humans are to chimpanzees.

10. The three buckets of AI R&D and the role of verifiability​

The researcher divides algorithmic progress into pre-training, post-training, and inference efficiency. Each category can be evaluated with measurable proxies, making AI research unusually suitable for reinforcement learning and parallel experimentation.

  • The three buckets together cover all algorithmic progress: pre-training, post-training, and inference efficiency.
  • Verifiable proxies let labs train models on AI research through thousands of rounds of practice rather than waiting for direct human judgment on every result.
  • They also let individual agents check a leaderboard score empirically before merging their contributions, supporting parallel work.
  • AI’s most serious deficiency relative to humans is sample efficiency: neural networks require far more data than people to learn the same capability.
  • Sample efficiency can be measured through the data required to reach a given pre-training loss, the number of tasks required to reach a target RL score, and the feedback and within-session time needed for in-context learning.
  • The researcher treats sample efficiency and online learning as two sides of the same problem and calls it one of machine learning’s deepest open problems—roughly its equivalent of a Millennium Prize problem.

11. Automating pre-training research​

Pre-training offers several concrete, automatically verifiable objectives for AI researchers. The main difficulty is ensuring that gains demonstrated at small scale will survive at frontier scale, a form of “research taste” that the researcher believes can itself be trained.

  • One proposed environment would RL a GPT-7 to minimize the compute needed to train a GPT-2-sized model to a fixed pre-training loss.
  • A swarm of GPT-7s could then train GPT-3-sized models to the lowest possible perplexity, with held-out test loss providing an objective check.
  • Scaling that idea from 100-million-parameter experiments to 10-billion-parameter models would require another RL environment in which GPT-7 predicts whether a small-scale algorithmic improvement will hold at large scale.
  • That scale-validation environment would cost more to train than a NanoGPT speed run, but the researcher considers it feasible at frontier-lab compute levels.
  • One mechanism for faster pre-training progress is that “compression between compression and intelligence” yields broader intelligence gains.
  • A second is that better algorithms let labs train smaller models that match previously larger ones, reducing the cost of experiments and RL rollouts while allowing a larger population of researchers to operate.
  • A third is that faster training of smaller models shortens the recursive self-improvement loop.
  • During RSI, labs would abandon three-month pre-training runs spread across several data-center campuses because automated researchers would find improvements so quickly that a giant run would be obsolete before it finished.
  • A smaller model launched only a week later could replace it, which the researcher illustrates with a ship to Alpha Centauri being overtaken in transit by one launched a century later.
  • AI Lab Magic recently pre-trained a model matching leading open-source base models such as DeepSeek V4 Pro using half the compute used for the original GPT-3.
  • At today’s prices, training the original GPT-3 would cost roughly $500,000, suggesting that orders-of-magnitude further pre-training improvements remain plausible.

12. Automating post-training research​

Post-training research has several direct proxies, ranging from making RL cheaper to measuring how quickly a model learns an unseen environment. Anthropic’s internal CoBench provides a concrete example of turning real research work into an AI training environment.

  • One proxy is improving post-training algorithms so that RL on new environments consumes less compute, data, or time.
  • Another is measuring how quickly a model learns on the fly in environments it was never trained on, such as a video game it has never played.
  • ByteDance’s EdgeBench is cited as a benchmark for this kind of rapid adaptation.
  • A third proxy directly measures AI R&D capability.
  • Anthropic’s internal CoBench contains around 500 real problems solved by its researchers and engineers during the year.
  • Because every benchmark can also become an RL environment, CoBench could let Anthropic’s models practice reproducing the last three to six months of human progress at the research frontier.

13. Inference efficiency and the width of the bottleneck​

Inference efficiency concerns how much useful computation can be extracted from existing hardware through model and systems design. Unlike some deeper research problems, its outputs can often be measured directly, which supports another form of automated optimization.

  • Inference-efficiency work includes designing the model and surrounding software to obtain more computation or “juice” from each unit of hardware.
  • Relevant metrics include GPU utilization, KV-cache size per token, total floating-point operations in a forward pass, and latency.
  • The researcher considers these bottlenecks serious but not especially crippling, describing the neck of the bottle as “pretty wide.”
  • He therefore expects something like a GPT-3.5-to-GPT-6-scale jump in 2028 rather than a crazy overnight singularity beginning immediately.

14. Continual learning as the key open problem​

For the previous year and a half, the researcher’s principal reason for doubting near-term AI research automation was the absence of robust learning from on-the-job experience. He has now changed his mind, largely because he believes much longer effective in-context sessions may substitute for a conventional continual learner.

  • Under his earlier view, a model session retained only a couple of hours of information, so learning did not accumulate across weeks or months.
  • In December, that lack of continual learning was his main reason for doubting that AI research could be automated soon.
  • He now thinks scaling in-context learning could produce sessions effectively many months long, and he points to many published ideas for dramatically longer effective context windows.
  • One possibility is that researchers will not need a dedicated continual learner to solve continual learning, because online learning is itself an ML research problem with clean, verifiable proxies.
  • Two years earlier, it was not obvious that the Navier–Stokes problem would be easier than continual learning, yet models already solve Millennium Prize Problems while remaining effectively amnesiac.
  • Over the same two-year period, continual learning became clearer as a harder problem than Navier–Stokes, raising the question of whether amnesiac systems can continue winning the race despite that disadvantage.
  • The researcher describes the situation as “like watching somebody do cartwheels down the track and still keep pace with Usain Bolt.”
  • His remaining 25% probability that Ilya-level AI researchers will not be automated until after 2032 is tied to the serious possibility that continual learning remains unsolved within the next few years.

15. The objection that researchers may not know which proxy to optimize​

A central objection to the RSI argument is that AI progress may be bottlenecked by selecting the right objective, because choosing what to optimize requires research taste. The researcher answers that existing proxy environments and broader outer-loop measures can survive paradigm shifts.

  • The objection is that choosing which “hill” GPT-7 should climb is itself an important part of AI research and requires a form of research taste that may be difficult to train.
  • Before 2018, almost nobody claimed that minimizing next-token prediction loss would be the most tractable route to advanced AI, so nobody had created NanoGPT-style speed-run environments in which an agent swarm minimizes time to a target pre-training loss.
  • His first answer is that the Transformer was invented to beat existing translation benchmarks, while later reasoning systems relied on established coding and mathematics benchmarks.
  • Those examples show that proxies created before a paradigm leap can still capture large step changes because general AI learning algorithms can exploit them.
  • His second answer is to use paradigm-agnostic outer-loop proxies.
  • Possible outer-loop measures include the number of Millennium Prize Problems a model solves, how quickly it discovers general relativity when trained only on data preceding 1905, and how rapidly it masters completely unseen video games.

16. Data and environments may matter more than model architecture​

Research cited by the episode suggests that much of the recent improvement attributed to “the model” actually comes from better data, tasks, and environments. This supports the view that ordinary R&D work can be automated through measurable environments.

  • Research conducted with Jerry Han, a Princeton student, found that over the last six years, more than three times as much pre-training algorithmic progress came from data improvements as from model improvements.
  • Data improvement is largely ordinary AI research, including filtering existing data or generating synthetic data that lowers loss on a held-out corpus.
  • A DeepMind paper states that changes to the basic RL algorithm did not matter nearly as much as systematic improvements in the scale, diversity, and verifiability of synthesized data and environments.
  • The DeepMind paper says those systematic improvements “account for essentially all of the observed gains.”
  • Its described pipelines train models iteratively to construct better tasks, using difficulty and correctness as reward signals.

17. The compute bottleneck and more efficient use of experiments​

AI research differs from pure mathematics because proposed improvements must be tested through real experiments that consume compute and time. The researcher argues that an RSI swarm could use its larger labor supply more intelligently rather than simply dividing a fixed experimental budget among more agents.

  • Mathematical discoveries can sometimes be reasoned out with pen and paper, while AI research requires experiments that consume actual compute and wall-clock time.
  • As the number of automated researchers grows, each agent receives a smaller slice of total experimental compute unless the swarm finds more efficient ways to use it.
  • Frontier labs have been more than quadrupling their compute every year.
  • Under that scaling, by the end of 2027 OpenAI could give roughly 10,000 agents enough compute for each to run a GPT-3-sized training run every day.
  • The 10,000-agent figure is compared with the number involved in solving the Millennium Prize problem.
  • For scale, the final model in the 2017 Transformer paper took three days on eight P100 GPUs and would cost about $50 to rent at today’s prices.
  • That Transformer training run used roughly one ten-thousandth of GPT-3’s training compute.
  • Top AI researchers, including names mentioned alongside Noam Shazeer and Ilya Sutskever in the episode, are individually paid hundreds of millions of dollars per year.
  • Current compute usage therefore reflects researcher-time scarcity more than compute efficiency and says little about how effectively compute would be used once compute itself became the principal bottleneck.
  • One proposed efficiency mechanism trades labor for compute by replacing wasteful small pre-training experiments with huge parallel multi-agent RL runs.
  • In that model, agents read every RL transcript, rebuild broken environments, run small-scale ablations, and spend effectively subjective decades designing and analyzing pooled experiments.
  • A second mechanism is developing better science for learning from small-scale experiments, allowing smaller trials to predict the value of larger runs more reliably.