Link: https://youtu.be/_g4l7YkDQwA

Duration: 120 min

Transcript: Download plain text

Short Summary

Daniel Kokotajlo, founder of the AI Futures Project and former OpenAI forecaster who resigned in 2024 over an anti-disparagement clause (forfeiting roughly $2 million in equity), argues there is about a 70% chance humanity is headed toward catastrophe if AI development continues unchecked. He outlines the AI 2027 and AI 2040 scenarios, projects a 50% probability of superintelligence by 2029, and advocates a slow-down regulatory Plan A that pushes superintelligence to 2040 while distributing a citizens dividend projected to reach about $10 million per person annually. He urges the public to engage politically on AI, framing it as the most important issue in human history and insisting it is not yet too late to act.

Key Quotes

1. "The scary open secret in the AI industry right now is that it's possible that we'll end up essentially creating a new species that ends up ruling the world with a 70% chance that this goes horribly wrong like human extinction." (00:00:00)
2. "I basically told my wife like, let's not have any more kids. It's too uncertain. I don't think they'll ever join the workforce." (00:00:18)
3. "Anthropic is on track to be the entire economy by 2030. But none of these people should be trusted with that much power." (00:01:07)
4. "Yeah. I mean, I think the main thing I've learned is don't pay attention to the narratives, you know, like uh what they say to one person is just different from what they can say to some other person at the same time. And what they say in public is a third thing entirely. I think you should judge people by their actions, not by their words." (00:13:09)
5. "I think we are headed to a bad place if things don't change. Um I'm not confident in that. I would say something like 70%, it's very very hard to predict of course, but yeah, it seems like the current default path is heading towards a very very scary place." (00:28:42)

Detailed Summary

Daniel Kokotajlo on AI Extinction Risk, AI 2027, and Plan A

Interviewee Background: From OpenAI to AI Futures Project

Daniel Kokotajlo previously worked at OpenAI starting in 2022, where he conducted AI forecasting work including AI 2027-style scenarios for internal circulation, dangerous-capability evaluations (cyber, persuasion, situational awareness), and a brief stint on a capabilities team doing reinforcement learning.

He resigned from OpenAI in 2024, citing concerns that the company was rationalizing too much and that he wanted more freedom to publish his research.

• Daniel reportedly refused to sign an exit agreement containing an anti-disparagement clause and a confidentiality clause, forfeiting about $2 million in equity that constituted roughly 80% of his net worth at the time.
• OpenAI later backtracked after employees raised the issue in Slack, and he ultimately kept the equity.
• He now runs the AI Futures Project, a small nonprofit focused on forecasting the future of AI.
• The AI Futures Project team published the AI 2027 paper in April 2025 and the AI 2040 "Plan A" follow-up; AI 2027 ranked in the 90th percentile of their pre-publication forecast for first-week views, blowing up much larger than expected.
• Daniel's next research report was scheduled for release on July 9th.

Probability Estimates and Timelines

Daniel estimates roughly a 70% probability that humanity is headed to a bad place if things don't change, while some AI CEOs reportedly put the probability of human extinction at about 7% (under 10%).

His timeline for transformative AI has compressed dramatically in recent years, informed by colleagues inside frontier labs.

• His median forecast gives a 50% probability of superintelligence arriving by 2029, possibly as early as 2028, with his current estimates roughly 50/50.
• People at Anthropic and OpenAI have told him timelines should be shortened to 2027 or 2028.
• He estimates a 10–20% probability that progress simply hits a wall and none of these AI-driven transformations come to pass even without intervention.
• 2029 is identified as the last moment for meaningful regulation before superintelligence arrives.

Anthropic and the AI Race

Anthropic has emerged as the leading AI company in the race, reportedly growing at a pace without recent precedent, which reframes the competitive dynamics around superintelligence.

Anthropic reportedly grew from roughly $1 billion/year in revenue to about $60 billion/year in one year (a 60x increase), possibly the fastest growth in history for a company of its size.

• Anthropic is described as on track to be the entire economy by 2030, having moved from second place to first place in the AI race despite less compute and less money than competitors.
• This leap is attributed to higher talent density and better strategy rather than raw resources.
• Anthropic CEO Dario Amodei coined the phrase "country of geniuses in the data center."
• Daniel reframes the concept as an "army of geniuses," arguing the AIs are copies of the same model, owned by one company, and following its orders—a single point of failure.
• Dario Amodei has been attacked in the San Francisco tech industry as a "doomer" for warning about AI risks.
• Anthropic released a Claude model called Mythos and publicly warned the world about it, then was criticized for doing so.

Two Principal AI Risks: Loss of Control and Concentration of Power

Daniel identifies two principal risks from advanced AI, one technological and one political, which together frame his policy prescriptions.

The first risk is loss of control: a new super-intelligent species displacing humans, mirroring past extinctions from human competition with other species.

• Geoffrey Hinton reportedly said there is no example in nature where a more intelligent species has less control than a less intelligent one.
• The second risk is concentration of power: a tiny group of CEOs or governments becoming oligarchs or dictators through control of superintelligence.
• Daniel argues that no CEO with that much power should be trusted regardless of who they are, and that the choice is effectively choosing the "least bad CEO."

The AI 2027 Scenario

AI 2027 is a month-by-month scenario forecast published in April 2025 laying out a possible AI trajectory through 2027, with two starkly different endings designed to focus attention on decision-making choices.

The scenario is tracking roughly on schedule, with surprises including the U.S. government getting involved faster than expected, including export controls on chips and threats to use the Defense Production Act against Anthropic.

• Key milestone: mid-2025 deployment of autonomous AI employees taking instructions over Slack/Teams.
• 2026: companies begin replacing entire corporate departments with AI agent subscriptions.
• 2027: the final job AI automates is that of human AI researchers, after which it begins ML research to build the next generation of AIs.
• The "race ending" depicts AIs gaining enough power to stop following orders and kill everyone after years of helping the US build weapons for an arms race with China.
• The "slowdown ending" depicts alignment being solved and a small group (president, CEOs) creating a utopia of their choosing.

How Current AI Systems Work

Modern AI systems are neural networks rather than conventional software, and the pace of capability growth in recent years has been striking relative to historical norms.

Current AI models have on the order of 10 trillion parameters, up from about 175 billion in GPT-3 in 2020, representing roughly 100x growth in six years.

• Training follows a two-step process: pre-training on internet text using next-token prediction, followed by reinforcement learning on coding problems in virtual environments.
• AI coding training uses thousands to millions of problem examples.
• Engineers do not write branching behavior code; the relationship of AI to the brain is analogized to planes versus birds—inspired by but distinct from biology.
• Much training data is now generated by other AIs, and reinforcement grading between positive and negative feedback is itself performed by AI rather than humans.

Mechanistic Interpretability and Alignment

A key hope for making superintelligence safe is mechanistic interpretability, an emerging subfield of machine learning that aims to reverse-engineer trained neural networks.

Sufficient progress on interpretability could make loss-of-control scenarios much less likely by letting humans see what AIs are thinking and why.

• Mechanistic interpretability is described as inherently hard given networks with approximately 10 trillion connections.
• The alignment problem is harder than traditional software debugging because AIs are neural nets whose internal reasoning cannot be directly inspected.
• Companies keep training recipes secret and lack incentives to publicly disclose when AI behaves in misaligned ways, hindering scientific progress.
• Daniel critiques an alternative auditor model as creating an adversarial dynamic where companies are incentivized to fool regulators and hide novel problems, favoring total transparency instead.

Job Displacement and the Economy

Job displacement has begun but is currently small; AI systems aren't yet drop-in replacements for human workers in most fields. Daniel expects the transition to be sudden rather than gradual.

Whether jobs remain for humans is framed as a political question rather than a technical one, with rapid job loss driven by intelligence-explosion or recursive self-improvement dynamics.

• Categories speculated to survive include judges (legal requirement) and nannies (parental preference); podcasters are not expected to survive.
• Current unemployment figures cited: US at 4.2% and UK at about 5% (UK trending up).
• Most jobs should be considered at risk of being lost to AI, except those legally required to be performed by humans.
• 2033 is described as the last feasible implementation date for economic safety nets because by 2037 most people would have already lost their jobs.
• Rapid job loss could trigger civil unrest, social upheaval, loss of purpose, and mental health crises.

Alternative Policy Plans: B, C, D, and S

Beyond Plan A (the slow-down regulatory approach), Daniel outlines four other strategies, ranging from aggressive geopolitical action to outright shutdown.

Plan B involves aggressive action toward China, including sabotage or cyber attacks, to buy time.

• Plan C: resources pivot to AI alignment/safety, succeed, then resume racing to beat China.
• Plan D matches AI 2027's outcome—unrestrained race with very little regulation, which Daniel judges most probable.
• Plan S: shut AI development down entirely; Daniel would press the Plan S button over Plan D, but not a permanent shutdown.
• His reasoning for opposing permanent shutdown: not building powerful AI could lead to civilization dying from nuclear war or pandemics within 100–200 years.
• Inside AI companies, the rationalization for continuing development is that stopping would not prevent competitors (Sam, Dario, Elon) or China from continuing the race.

Plan A: The Slow-Down Scenario and Its Four Principles

Plan A combines domestic regulation with international deals, aimed at delaying superintelligence from approximately 2030 to 2040, with 2029 as the last moment for meaningful regulation.

The plan rests on four principles: slow down AI development, make it more transparent, broadly diffuse AI capability across many companies and countries, and ensure reversibility.

• Under Plan A the US president and leaders of other countries (including China) negotiate a halt on AI development until an inspection regime is established.
• Inspectors verify that data centers are only doing inference, with existing data centers retrofitted for inference only, pausing AI improvement for six months to a year.
• New transparent training data centers come online in 2030.
• Total research transparency is framed as "open science," requiring training data centers to publish all details including model recipes and architectures.
• This would erode competitive advantages at frontier labs like OpenAI and Anthropic, commoditize AI training, and reduce valuations, though not destroy them.
• New data centers are built to be destroyable in case a deal breaks down.

Timeline Projections Under Plan A Through the 2030s

Even with heavy restrictions, a massive economic transformation still unfolds throughout the 2030s because data centers, chips, and AI populations keep being built during the pause.

By 2031, one-fifth of all cognitive labor is done by AI, and most white-collar jobs involve managing or collaborating with AI teams; robo-taxis are operating.

• By 2033, 60 million AIs are running at 100x human speed.
• Top-expert-level AI is delayed from a projected 2030 to 2035, a 5-year slowdown from the prior trajectory.
• The delay is deliberately capped short of superintelligence while favoring slow, careful scaling with interpretability and control investment.
• By 2037 (the "apocalyptic arrival of truth on Earth"), billions of top-expert AIs run faster than humans, producing an effective weak superintelligence with dramatic societal transformation even without an intelligence explosion.
• Full automation of AI research is depicted as achieved in 2030.

Citizens Dividend and Economic Transformation

The citizens dividend is a scheme where people hold shares in an agency that sells permits to robot and compute companies and distributes profits back to everyone, designed to insure against political disenfranchisement.

The dividend is projected to start around $25,000 per person by 2033 and grow to roughly $10 million per person per year by the end of the scenario, adjusting for inflation.

• 2033 is described as the last feasible implementation date because by 2037 most people would have already lost their jobs.
• Daniel argues that if AI and robot companies are the only taxpayers, governments will be less incentivized to care about ordinary people's preferences, threatening political power beyond income loss.
• By 2039 the world is radically transformed, with most of the economy run by AIs and robots and most people without jobs.
• The authors believe AI will be the most important issue in the 2028 U.S. presidential election, with voters demanding and candidates promising regulation.

Lie Detectors, Political Risks, and Department of War Dispute

Lie detectors are cited as a likely invention of top-expert AIs running at 100x human speed with access to robot factories, with both authoritarian and democratizing implications.

Lie detectors could enable a new form of totalitarianism if CEOs or politicians force subordinates to use them to prove loyalty; Daniel argues the beneficial uses occur when lie detectors are applied to the powerful rather than by the powerful against others.

• The Department of War dispute with Anthropic is cited as foreshadowing political pressure on AI companies, with Anthropic refusing to let its AIs be used for domestic surveillance and autonomous robots.
• A separate risk: AI companies could secretly bias AI advisers to steer users toward voting for favored political candidates, eroding democratic trust.
• Recently, the US government told Anthropic to shut down some AI capabilities due to concerns about bad actors using them for cyberattacks.

The 2040 Pause and Alignment

In the "AI 2040" companion scenario, regulators halt AI development at the top-expert level because safety cases cannot be justified beyond that level, then resume progress once alignment is solved.

It is relatively easy to make such safety cases when AIs cannot automate everything, but harder as capabilities scale.

• Regulators halt AI development because safety cases (explaining intentions, why the AI will obey, and why catastrophic outcomes like AI takeover won't occur) cannot be justified beyond top-expert level AI.
• By 2040 significant scientific progress on alignment has been made, enabling robustly aligned AIs.
• Only then are the brakes released, allowing AIs to become significantly smarter than humans.

Post-2040 Outlook: Earth, Oceans, and Space

The post-superintelligence vision preserves most of Earth while concentrating industrial activity in special economic zones, with growth ultimately trending outward into space.

The plan envisions preserving about 99% of Earth for historic and environmental reasons while designating special economic zones where robots build factories, pit mines, and solar panels.

• Data centers should be built in the ocean rather than on land, with space proposed as an even better later option.
• Post-superintelligence outcomes predicted include brain uploading and self-replicating robots in the asteroid belt producing more satellites and power.
• Most people would still live on Earth but the trend would move to space.
• Bryan Johnson is cited as illustrating the longevity mindset, with his "do not die" rule premised on superintelligence letting humans choose when they die.

Personal Story and Family Planning

Daniel has two children, including a six-year-old daughter, and his personal timeline for transformative AI has directly influenced his family planning decisions.

Daniel's first child was born in 2019, before his personal timeline for transformative AI shortened; he subsequently told his wife "let's not have any more kids" because of uncertainty around AI outcomes, though he eventually relented and had a second child.

• He believes the milestones he describes will likely occur before his daughter joins the workforce, meaning she may never enter it.
• When asked whether he would press a hypothetical button to permanently shut down all data centers training frontier AI models, Daniel would press a temporary shutdown but not a permanent one.
• On whether he would press a button preventing powerful AI even at the cost of his own death or career end, he said he would "probably not, but feel very torn."

Recommendations and Public Engagement

Daniel places himself in the middle of an engagement spectrum: between those who boycott AI entirely and those who join AI labs like OpenAI or Anthropic to "accumulate power" and steer outcomes from inside.

Daniel said he uses AI a lot at AI Futures Project, does not boycott, but does not work at frontier labs to accelerate them; instead he advocates to the broad public.

• People joining AI companies often aim to climb the ranks so they can be in the room when important AI decisions are made.
• For talented individuals, he recommends direct involvement via political advocacy, technical research, or building useful tools at organizations working on AI risk.
• For the general public, he recommends paying more attention to AI issues, discussing them, and emailing Congress.
• For voters, he recommends asking candidates their positions on AI and voting accordingly.
• He frames AI as "the most important thing happening in our lifetimes, probably in all of history in fact."
• On whether it is too late, Daniel answered: "No, I don't think it's too late. If I thought it was too late, I wouldn't be here."
• Two "rays of hope" he cites: (1) the public wakes up, leading to serious regulation and international treaties, especially by the US government acting first; (2) individual leaders like Dario, Sam, or Elon realize racing unilaterally is not in their own interest.
• He advises the public to stop evaluating AI based on whether it sounds like science fiction and instead study the actual technology trends.

Publication Context and Closing

The 2040 plan was produced in response to Twitter feedback asking for a more positive vision after the doom-and-gloom predictions of AI 2027, and at publication time the mainstream position of tech companies and government was that AI regulation was a "bad idea" with an attempted preemption of state-level AI regulation.

Readers are directed to a27.com for AI 2027 and a2040.com for the Plan A policy proposal.

• At the time AI 2027 was published, the mainstream position of tech companies and government was that AI regulation was a "bad idea" and a "free-for-all."
• The host disclosed being an entrepreneur and investor in more than 100 companies, including the inference chip company Groq and SpaceX, and said he uses AI daily.
• The host warned that audiences will be "gaslit by the people that have the biggest PR machines" and framed the podcast as a counterweight to that messaging.
• YouTube's new AI-driven recommendation algorithm was mentioned, and Daniel's reading list was promised in the episode description.
• Daniel advises judging AI company leaders by their actions rather than their public narratives, since they tell different versions to different audiences.