The scary open secret in the AI industry
right now is that it's possible that
we'll end up essentially creating a new
species that ends up ruling the world
with a 70% chance that this goes
horribly wrong like human extinction.
That's one possibility. There's many
more. It's quite chilling what you're
saying.
>> Yeah, it's uh it gets me down sometimes.
I basically told my wife like, let's not
have any more kids. It's too uncertain.
I don't think they'll ever join the
workforce.
Everybody should be afraid that their
jobs are going to be lost. And I know
this because I went to open air in 2022.
What I did there was forecasting as to
what the next couple years might look
like. And unfortunately, most of the
world is kind of asleep at the wheel and
doesn't really realize what's going on
with AI. So, I resigned.
>> I read it somewhere that you lost $2
million for not signing an
anti-disparagement clause, meaning you
couldn't criticize the company.
>> Yes. For reasons I'm happy to get into.
But the main thing I've learned is when
I go talk to people at Anthropic and
OpenAI about forecasting, they're like,
"It's not going to take that long. You
need to shorten them again. get them
back to 2027 or 228 because these
powerful CEOs, Ariel or Sam or Elon are
racing each other to be in control of
the most powerful AIs and are literally
afraid that if the other guy gets there
first, he might become dictator. I mean,
Anthropic is on track to be the entire
economy by 2030. But none of these
people should be trusted with that much
power. So, this is the most important
thing happening in our lifetimes,
probably in all of history, in fact. And
it's very important that it go well. So,
I think that there's a lot we can do to
like steer things in a better direction.
There's loads of benefits that we could
get from AI if we do it right. And if we
do solve the problems, then things could
be absolutely amazing for everyone.
>> Well, this report here in 2021, it was
remarkably [music] accurate. And then
you just published this one.
>> Yeah. So, this is our new scenario.
>> So, let's go through these slowly and
one at a time.
>> I would be incredibly happy if all my
predictions turn out to be wrong.
>> This is super interesting to me. My team
given me this report to show me how many
of you that watch this show subscribe.
And some of you have told us according
to this that you are unsubscribed from
the channel randomly. So, favor to ask
all of you. Please could you check right
now if you've hit the subscribe button
if you are a regular viewer of the show
and you like what we do here. We're
approaching quite a significant landmark
on this show in terms of a subscriber
number. So, if there was one simple free
thing that you could do to help us, my
team, everyone here to keep this show
free, to keep it improving year over
year and week over week, it is just to
hit that subscribe button and to double
check if you've hit it. Only thing I'll
ever ask of you, do we have a deal? If
you do it, I'll tell you what I'll do.
I'll make sure every single week, every
single month, we fight harder and harder
and harder and harder to bring you the
guests and conversations that you want
to hear. I've stayed true to that
promise since the very beginning of the
D of Coio, and I will not let you down.
Please help us. Really appreciate it.
Let's get on with the show. [music]
>> Daniel Cocatello,
at the very heart of what you do, um,
what is your mission and why? So what
would you do if you thought that super
intelligence was coming in a few years?
>> I guess it depends what the consequences
were. Well, let's talk about it. So
super intelligence, AIs that are better
than the best humans at everything while
also being faster and cheaper. Also able
to operate robots that can do everything
in the physical world that humans can
do, but better, faster, and cheaper. If
that really is coming in a few years,
then we need to prepare and we need to
think about how to make it go well
instead of poorly. So that's sort of my
answer is like I'm doing that to the
best of my ability.
>> So you believe it's coming in a few
years?
>> Yes.
>> How could you be so sure?
>> I spend a lot of time trying to forecast
this sort of thing. My sort of median
estimate 50% chance is currently in
2029. Maybe it'll slip to 28 28. It's
possible that it'll take significantly
longer, like maybe 10 years or something
like that, but uh you know, for reasons
I'm happy to get into, seems to me like
it's probably happening by the end of
the decade. What's less important is the
the sense of how close we are. What's
more important is the pace of the
trends. Anthropic
this time last year was making something
like a billion dollars a year and now
they're making something like $60
billion a year. So that's 60x growth in
one year, which is extremely impressive
even for very small startups. But for a
company of their size, it might be the
fastest growth in history. Um, we expect
that rate of growth to slow down, but
even if it slows down quite a lot,
they're still on track to be, you know,
the entire economy by 2030 or so.
>> Why should the average person care? The
high level thing is absolutely
everything is going to change for the
whole world and including therefore for
them and their families. Um could change
for the better, could change for the
worse depending on the details of how
it's done. So for example, everyone
could die you know um this is the
classic loss of control scenario or one
version of it. If we do build these
super intelligences and we use them to
automate all the jobs and we put them in
the military and we, you know, have them
giving advice to politicians and so
forth, they will eventually have
accumulated enough real world power that
they don't need humans anymore and
they're smarter than us, they're more
strategic, etc. At that point, we sort
of have to hope that they are virtuous,
that they have, you know, the goals that
we wanted them to have, the values that
we wanted them to have, etc. And the
sort of scary open secret in the AI
industry right now is that right now
that is kind of just a hope. It's not
something that we can be at all
confident in. And in fact there's lots
of evidence and arguments that it we're
not on track to achieve that. So there's
lots of reason like current AIS for
example will often lie uh to people or
they will like you tell them to do
something and they go do something else
and then pretend that they did it right.
So, it's an inherently difficult problem
to make something that's super
intelligent and also has the values and
virtues that you want it to have. And it
doesn't seem like we're on track to
solve that problem. Also, it seems like
the sort of problem that you could think
you've solved when you haven't actually
solved it, right? Uh that's a big reason
why this is scary. So, for all those
reasons, it's possible that we'll end up
essentially creating a new species that
ends up ruling the world instead of us.
And then maybe we go the way of other
extinct species in the past that were
out competed by humans. That's one
possibility. There's many more. Even if
you're not worried about that and you
think that the AI will be totally
controlled, there's the question of who
controls the AIS, right? When there's a
couple corporations that have made these
super intelligences and are using them
to automate all the jobs. Well, that's a
lot of power. You know, that's a lot of
money. It's a lot of political power.
They'll have the best strategists, the
best advisers, you know, they'll think
faster militarily. uh the countries that
has these AIs will be able to absolutely
wipe the floor with all the other
countries. The AIS themselves it's it's
kind of a single point of failure like
central uh control system where you know
the CEO of Enthropic Dario he coined
this phrase the country of geniuses in
the data data center that was his phrase
to describe what they're trying to
build. You know I think that's a little
bit misleading. I think it would be more
accurate to describe it as army of
geniuses in the data center because it's
not like it's a bunch of diverse
different AIs, you know, living in their
different parts of the data center.
They're all copies of the same big model
and they're owned by the company and so
they all follow the orders given by the
company, right? People should be asking
questions of like who controls this army
or these armies and what are they going
to be doing with them? I think that we
could very easily end up in a sort of
uh a situation where some tiny group of
people are essentially oligarchs or
dictators. And ironically,
both of these risks, the loss of control
and the constitution of power are things
that people in the industry have been
thinking about for decades. Um even
before the AI industry existed, you
know, people thinking about AI were
talking and writing about these things.
And then part of the founding narrative
or the founding myth of deep mind and
open AAI and anthropic is these problems
are real. So we need to get there first
so that we can handle it responsibly.
Those are I think the big two reasons.
But then I can go on there's lots more
reasons as well. So one thing is you
know World War II geopolitical conflict.
Um if AI does in fact get incredibly
powerful that's going to change the
balance of power between nations. That's
going to disrupt a lot of things. That
puts us at increased risk of crisis more
generally, right? Another one, what
about those jobs?
You you're going to lose your taxi job,
but not just the taxi driver, everybody
pretty much. Um there might be a few
exceptions like people whose jobs for
legal reasons are only allowed to be
done by humans, but for the most part,
everybody should be afraid that their
jobs are going to be lost even if we
manage to avoid all the other problems.
Right? this narrative has started to
emerge and I've had several interviews
on this show where I've interviewed
people who are very very scared and
anxious about AI and these are people
that have worked in the industry for
sometimes decades. Yeah.
>> Um the counternarrative coming over the
hill is that this is doomerism that
these people are for whatever reason
just trying to scare people and that
they don't really understand what
they're talking about. How do you
respond to that sort of
counternarrative? And you must have seen
this emerging yourself especially from
people who stand to benefit dare I say.
>> Yeah, exactly. this counter narrative is
fairly recent and it's been pushed by
the people who stand to benefit um from
it and it's not true. Like these these
concerns have been around for decades
since before the AI industry existed.
They're actually pretty reasonable
concerns. Like if you take the companies
at their word and imagine that they are
in fact going to build super
intelligence. Well, it raises a lot of
questions like who's going to control
it? Will anybody control it? What about
the jobs? You know, like these are just
kind of obvious implications to be
thinking about and worrying about. Who
are you and what's your story?
>> My name is Dan Coatello. Um, I currently
run the AI Futures Project, which is a
small nonprofit that mostly focuses on
forecasting the future of AI. Before
that, I worked at OpenAI.
>> AI forecasting.
>> Yeah. So think about how like you know
industry analysts who work for hedge
funds and stuff will make these
forecasts of like here's you know how
many cars Tesla will be selling 5 years
from now or like here's what the price
of electricity will be in 2 years right
that's forecasting I was doing that but
specifically focused on AI the reason I
was doing it is because it's incredibly
important to to see where this is all
headed
>> why did you go to open AI what did you
do there what did you observe while you
were there and how did Did it change
your perspective on the future of AI but
also I guess open AI as a company and
for anybody that doesn't know OpenAI are
the company that produced chatbt?
>> Yeah, so I went to OpenAI in 2022. Uh a
large part of what I did there was more
forecasting. AI 2027 is a scenario that
you may have heard of. I did like
smaller, you know, lower effort versions
of them internally for just internal
circulation of like here's some guesses
as to what the next couple years might
look like. I also worked on evaluations
for dangerous capabilities. So, you
know, trying to measure the AI's cyber
abilities or persuasion abilities or
situational awareness. And I also
briefly was on a uh a capabilities team
doing reinforcement learning to create
agents. AI is in fact getting uh a lot
better and I can say more about why you
know scaling laws um deep neural nets
bigger trained on more data become more
efficient, more competent at those
things. I also
became a bit more disillusioned with the
AI industry. So, OpenAI, Anthropic, and
Deep Mind all had these sort of founding
narratives of like, yes, these risks are
real, but we've thought about them and
we're going to try to handle them
responsibly. And that's why it's
important for us to keep doing what
we're doing. And I increasingly came to
think that these were rationalizations
to justify what they were doing rather
than sort of like deeply guiding their
actual behavior and that when push comes
to shove, they'll follow their
incentives rather than
do what's actually good.
>> So you're inside Open AI at the time and
you start to believe that they're
following commercial incentives versus
the I guess social in or societal
incentives that they founded themselves
on.
>> Sort of. I mean, I wouldn't actually
describe it as commercial incentives. I
think I would describe it as um
power seeking incentives. So,
[clears throat]
>> like it's true that the companies care a
lot about making a lot of money, but
especially at the very top of these
companies, like the leaders,
they understand that this is about more
than just money. You know, there are
these emails that came up in, you know,
the the lawsuit between Musk and and um
OpenAI. A bunch of emails were surfaced
in that lawsuit, which you can go read.
And in some of them, the founders of
OpenAI were talking back in like 2017
about how the reason why we made OpenAI
was because we were worried that Demos
at Google was going to become dictator
with AGI. Even back then, they were this
is obviously about more than just money.
Like these these powerful CEOs are
literally afraid that if the other guy
gets there first, he might become
dictator. And they don't trust each
other. And so that's why they are racing
as hard as they can so that they're the
ones who get there first, so to speak.
>> Have you met Sam Alman?
>> Yeah.
>> And did did that shape your opinion of
his incentives or what why he's doing
what he's doing? Cuz there's a lot, you
know, speculated about what his
incentives are. I mean, his most recent
narrative says uh for the good of
humanity. I think that's what
>> Yeah. I mean, I think the main thing
I've learned is don't pay attention to
the narratives, you know, like uh what
they say to one person is just different
from what they can say to some other
person at the same time. And what they
say in public is a third thing entirely.
I think you should judge people by their
actions, not by their words.
>> And why are you no longer at OpenAI?
>> Largely for the reason that I mentioned.
So I became gradually disillusioned with
how the company was going to behave. For
example, when I first joined in 2022, at
least the people I talked to, my
colleagues at the company, there was
this general sense of like, of course,
we wouldn't actually just build super
intelligence as soon as possible. Once
we started getting really close, like
once we started getting to AIS that
could maybe automate the AI research
process, we would pause and figure out
how to make it safe.
That's cuz we're the good guys. And
that's obviously the safe thing you
should do rather than just going full
speed ahead. But we're worried about
other people who might not pause. You
know, our competitors, Google for
example. And so that's why we need to be
in the lead so that we have that room to
do the safe stuff, right? That was sort
of like a thing that seemed like maybe
like the median position or something
among the colleagues I talked to when I
was there when I started, including
people like Sam, you know, including the
leadership. And then by the time I left,
I was like, "Oh man, they're really not
going to do that, are they?" Like
[laughter]
like they they've sort of, you know,
partly because this has become more
politicized and they've become bigger
and under more scrutiny, people have
started asking like, "Why are you doing
this in the first place if it's so
risky?" And so they've pivoted their
narrative to being more like actually
it's not that risky, you know? Um, and
so yeah, I mean it seems like they're
just going to keep going roughly as fast
as they can and hope that they can
figure it out on the way.
>> How did your time at OpenAI come to an
end?
>> Uh, I resigned in 2024. I had a nice
goodbye party.
>> What were the reasons you gave for
quitting OpenAI?
>> I thought that we were rationalizing too
much and that we needed to think more
about what would actually be good for
the world. Um, I wanted more freedom to
publish.
So at OpenAI as it became a bigger
company it became more of a normal tech
company with incentives and you know a
PR department and things like that and
so it started becoming more difficult to
um to publish the sort of research that
I was doing for example those scenarios
that I mentioned couldn't uh couldn't
publish those right they're just for
internal use I thought that that was a
shame because right now most of the
world is kind of asleep at the wheel and
doesn't really realize what's going on
with AI and doesn't really realize
what's coming in the pipeline a couple
years from now and the companies aren't
really incentivized to tell people that
much about it. I mean they say some
vague stuff in a sort of hypy way but um
you know well they didn't want me to
publish the scenario for example laying
out like here's how things might
actually look.
>> I'm just I'm super curious as to what
it's like being in a company like that
when they you know chat GBT3 is
released. You were there at that time
right?
>> Mhm. um which was a moment where I think
the whole world stood up and realized
that this technology was powerful.
>> Yeah.
>> Um and the conversation really began
from a society level. Um the company
starts growing super quickly.
>> Yeah.
>> Quicker than I think anybody could ever
have imagined. And what what was it like
inside there? What did you see change um
over over that period of time?
>> I remember one all hands meeting where
Ilia said something like
>> Ilia being
>> Ilia Sgver who was um head of research
at that time. He said something like,
"Okay, now the world is starting to pay
attention. Each of you is going to be
the most popular person at every party
uh for the next year. Don't let it get
to your head. Focus on the mission. Got
to build AGI." [laughter] The company
grew a lot. It already wasn't really
feeling like a nonprofit when I joined,
but it definitely didn't feel like a
nonprofit by the time I left. Um, lots
of new people came in. Ironically, the
like amount of conversation about
super intelligence and the implications
of super intelligence arguably sort of
went down over time due to this growth,
right? So, because the company would
like double and then double again and
then double again, all these new people
were coming in from other parts of the
tech industry who hadn't really been
thinking about these things and were
attracted by the high salaries. You lost
$2 million for not signing an
anti-disparagement clause, which would
mean you could speak, you couldn't
criticize the company.
>> Uh, yes. Well, so, um, I got to keep the
money.
>> Oh, you got to keep
>> So, what happened was after I had left,
said my goodbyes, etc., um, I got the
the exit paperwork, and it included this
clause that said you basically have to
agree not to criticize the company
again. Um, and also a clause saying, you
can't tell anyone about this. And so I
thought that was kind of rich coming
from a nonprofit that's supposed to be,
you know, for the benefit of all
humanity. So I didn't sign it. And if
you don't sign, you don't get to keep
your equity. So your compensation, you
know, what what they pay you is a bunch
of money and then also a bunch of stock
basically. But then they had this stuff
in the contract that they get to yank
back your your stock if you don't sign
this thing. Um, and my wife and I, you
know, were uh upset about this. We
talked about it for like a month or two,
consulted some lawyers, um, and then
ultimately decided to just refuse to
sign,
>> which would mean you lost, you would
have lost $2 million.
>> That's right. Which was like 80% of our
net worth at the time. Um, fortunately,
uh, it didn't go the way we expected. It
blew up basically on the internet. Like
when people heard that that we had done
this and that we had said no, it became
like this huge scandal. Employees at the
company started like asking questions in
Slack and like asking leadership like,
"Wait, what? Like why are you going to
take away our equity? What is this?" You
know, cuz a lot of people hadn't really
noticed this before. It had been
whispered about, but it hadn't been sort
of like a thing that most employees knew
about. Um, and so they backtracked and
they said, "Never mind. Never mind.
We'll change the paperwork. You can keep
the equity. It's fine." And Samman came
out and said he was embarrassed that he
didn't realize this was
>> Yeah, he had no idea apparently.
>> You don't believe him?
>> No, I think he probably knew. And if he
didn't know, then people close to him
probably did, such as his head lawyer.
>> Why did you decide not to take the $2
million?
>> I mean,
>> most people would have. I think
>> it's true. Most people would have and
most people did. And you know, money is
nice, but like it's not the only thing.
You know, sometimes it's good to take a
stand on principle. I I keep mentioning
super intelligence. Perhaps I should say
more about like the the sequence of
events that the companies are planning
to do. So, right now they're focusing on
automating coding. They're taking their
AIS, they're making them bigger, they're
training them for longer, and they're
especially focusing the training on
getting them to be good at autonomously
writing and editing code because uh that
will help the companies go faster,
right? Right? If they can automate the
code, then they can do their own work
better and faster and accelerate
progress. The next step, which they've
already begun, is to look at the rest of
the research process as well. Coming up
with ideas, um, analyzing experiments,
communicating those results, all the
other parts of of the research process,
they're trying to figure out how to
train AIs to be good at those as well,
so that they can have AIs do the entire
thing autonomously. When you say do the
entire thing, what do you
[clears throat] mean do the entire
thing?
>> So like Anthropic and OpenAI in
particular are trying to automate
themselves. Like they're trying to make
it the case that um they don't really
need human employees anymore. Uh they
just have a giant army of AIS that's
turnurning away doing all this
autonomous research to make better AIs
to train the new AIs, put them in charge
so they can make even better AIs and so
forth. And of course, not just not all
just happening internally, but also like
interfacing with the world, right? Like
going out and talking to people,
collecting the data, setting up the
training environments, doing the
business deals, and so forth. Like
they're they're trying to automate all
of that. The reason why they're doing
this is because they're trying to get to
a position where they have AIs that are
superhuman at everything, super
intelligence, and they're trying to get
there before their competitors do.
Needless to say, this is incredibly
dangerous, I would say. You know, and in
addition to being dangerous, it's a
power grab, right? Like, if they
actually succeed at this, then they'll
be sitting on top of this army of
superhuman AIs that will give them
immense leverage over all sorts of other
actors in the economy in so far as they
can work out something with the
president and, you know, integrate it
into the military or whatever, then that
would give the US immense hard power
over all other countries, right?
Obviously, nobody knows exactly when
this is happening, but a very disquing
thing has happened over the last year to
me, which is that when we published
2027,
people were generally of the opinion
that my timelines were too short and
that like probably it would take more
than 2027 until we got to the sort of
events that I was just mentioning. you
know, recursive self-improvement, AI is
automating the whole research process,
super intelligence. These these types of
milestones um they happen in 2027 in AI
2027,
>> which is this research paper you
published.
>> That's right. It's it's a scenario
forecast that sort of lays out like
month by month a possible future
trajectory. There was sort of like at
the time that we started writing it was
my best guess as to what would actually
happen. Obviously, there's lots of
uncertainty, but you know, I I I thought
it's valuable to make a concrete guess
just to sort of see what it might look
like. And at the time we were writing
this, a lot of my friends in the AI
industry and in nonprofits and so forth
that work on AI, a lot of people were
saying like, "Yeah, that stuff's going
to happen, but like it'll probably take
a couple years longer than you think."
And now
it's more 50/50. Especially when I go
talk to people at Anthropica and OpenAI,
they're often like, "Yeah, no, 2027,
that's basically what's going to happen,
just like you wrote. Why did you why did
you become why did you update your
timelines?" Oh, yeah. Context for this
is um after after writing 2027, I
shifted my timelines to be a little bit
more conservative. So, at the time that
we published, my 50% mark was in 2028,
not in 2027. And then after we published
progress just seemed like it was going a
bit slower and so I updated to 2030
which is you know it still could happen
sooner could happen later 2030. Um but
now when I talk to people in the
companies they're like it's not going to
take that long. They're like oh you need
to shorten them again like get them back
to 2027 or 2028 you know. Um so that's a
bit disquing. Um, again, don't know how
long it's going to take, but this is the
stated plans of the AI companies is to
do this incredibly dangerous thing, and
they think that they're just a few years
away. So, you wrote this um report here,
what 2026 looks like, and you wrote this
in 2021,
>> and it was remarkably accurate, helped
make a name for yourself amongst um
amongst everybody in AI. And I Which one
was it that JD Vance, the vice
president, read? I think it was this
one, wasn't it? Yeah, this one. Um and
then so then you published this one AI
2027 and this was published I believe in
2025.
>> Uh yes that's right April.
>> Yeah.
>> What were you forecasting in here? What
is what are the key things that you said
in here for people that haven't read it?
>> The high level version of it is they
automate the coding then they automate
the rest of the research process. Then
the pace of progress accelerates
dramatically. They get the super
intelligence. They're working with the
government who specifically the
president the executive branch naturally
wants to control this technology and
otherwise wants to use it to beat China
and integrate into military and so
forth. By [snorts] this point it's sort
of doing basically all the work itself.
I mean it's it's super intelligent. So
it's coming up with all these great
ideas for how to integrate itself into
everything and all these new
technologies it's invented and so forth
and uh because of the race dynamics and
because of the profit motive they end up
deploying it everywhere and it builds
robot factories. They build more robots.
to build more robot factories etc.
transforms the world entirely and then
at some point it has enough power it
meaning the AIs have enough power that
they don't have to pretend to to be
aligned anymore right um then they stop
listening to orders
that's the race ending of AI27
we also wrote a sort of different branch
which is the slowdown ending which was
intended to sort of illustrate the
concentration of power issues um that I
mentioned previously. So, what if
hypothetically the alignment issues get
sorted out sufficiently quickly? Like
what if it turns out that like it's not
too hard with two months of slowdown, we
can figure out how to make the AIS
robustly do what we want um and have the
values that we want them to have. So,
that's one possible branch. And in that
branch, uh it looks pretty similar. You
know, they take the jobs, beat China,
etc. Um, but instead of the AIS
ultimately killing everyone, they create
this sort of amazing utopia. But the
amazing utopia is whatever the people
who control the AIS wanted it to be,
right? And so that would be a very small
group of people like the president, some
CEOs, etc.
>> There should be a button just down below
here. And if it says subscribed, you're
already subscribed. If it says
subscriber, that means you're not yet.
And if you're not subscribed, please
could you do us a favor and hit that
button? It helps the show more than you
know. And according to the algorithm,
you're someone that watches our show,
but you haven't yet hit that button.
Thank you so much.
>> Is there any possibility, do you think,
that we never get to this thing called
AGI? And and how do we distinguish AGI
from this term super intelligence?
What's the difference?
>> Yeah, so the difference is that AGI is a
more vague uh and weak term. So super
intelligence is a bit more precisely
defined. It's better than the best
humans at everything, faster and
cheaper. Um, AGI is more like it stands
for artificial general intelligence,
which means AI is that can do things in
general rather than like some specific
task.
>> Yeah.
>> And so arguably we've already achieved
AGI, right? If you use cloud code or
something like that, it's like it can do
a lot of stuff. It's it's almost kind of
like a little employee that you can like
have go do stuff. So it's it is quite
general. It's not maximally general
though. It can't do everything. Whereas
super intelligence by definition can do
all the things that a human can do but
better.
>> And how does this sort of overlap with
robotics? Because obviously we're seeing
this huge robotics boom at the moment.
There are some real world things that
humans can still do because these AIs
are still stunk on my computer.
>> The way people talk about this is that
they basically just say we've achieved
super intelligence for cognitive tasks.
Then you can talk about like full super
intelligence that can do this physical
stuff.
>> And are we going to get there? Are we
going to get there with both?
>> I think so. I mean again this is not
something that we can be certain about.
Um you asked like is it possible we'll
never get there? Yes, it's possible
we'll never get there. I don't think
it's likely though. I think that there's
nothing sort of like magical about the
human brain. It's, you know, um it's
just a bunch of neurons. It is possible
for a digital system to do similar
functions in the same way that like, you
know, a plane can fly just like a bird.
Not in the same way as a bird
necessarily, like it doesn't have it's
not flying in the same way that a bird
flies, but it flies, you know.
>> Um so, so it does seem like yeah, like
>> seems possible.
>> You've written all these, you know,
these research reports. You're working
on another one that will be released um
likely on the 9th of July. You have
worked inside OpenAI. You then quit
OpenAI because you were concerned about
what was going on there and about the
future of the industry. You know more
than I do. Are you optimistic about the
future or pessimistic? Are we heading to
a bad place if things don't change? Um
based on everything that you know,
>> I think we are headed to a bad place if
things don't change. Um I'm not
confident in that. I would say something
like 70%, it's very very hard to predict
of course, but yeah, it seems like the
current default path is heading towards
a very very scary place.
>> How do you contend with that personally
and emotionally?
>> Um, it's rough. I mean, I think it it's
the sort of thing that like
gets me down
on a regular basis, but also I've been
dealing with this for so many years now
that I've sort of gotten used to it, if
that makes sense. Um,
yeah. Yeah. I I'll put it this way. I
would be incredibly happy if all my
predictions turn out to be wrong and uh
an AI hits the wall, for example.
>> It gets you down on a regular basis.
>> I used to be known as a pretty chipper
and optimistic person. But um in 2020 my
AI timelines predictions started
collapsing due to GPT3 and the scaling
laws papers and um the bioankers report
which I can talk about if you're
interested but basically some events
happened in 2020 that convinced me that
actually this stuff was like quite
plausibly coming by the end of the
decade and humanity is very obviously
not ready for this you know in a whole
bunch of different ways and so that's
obviously very scary
>> and that's an extremely scary world
because of all things you've said but
but again because of this recursive
self-improvement where AIs can train
themselves and at such point we're
starting to lose hold of what's going on
here. I mean the AIS are already
training themselves to be clear. It's
more like closing the entire research
loop, right? Doing everything.
>> Yeah. Like right right now a lot of the
training data is generated by AIS. A lot
of the reinforcement like the grading
that happens out of positive and
negative reinforcement is itself done by
AI.
>> Can you explain that in layman's terms
for
>> Yeah. So an important thing for
everybody to understand is that modern
AI systems are not software in the
normal sense. I mean they are
technically software but they're not
lines of code you know it's not like
some engineers at anthropic went and
wrote lines of code that basically says
like you know when the user asks for
this type of thing then go do this type
of thing for this many steps or
whatever. There's nothing like that.
Instead it's a neural net. You know
>> what's that?
>> Well think about how the brain is a
bunch of neurons connected to each
other.
>> Yeah.
>> That are firing um signals back and
forth. The brain learns over time the
types of patterns of firing that caused
success that caused a dopamine rush or
various other types of feedback get
reinforced and fire more often. And the
types of patterns that caused failure
like touching a hot stove get
anti-reinforced, get, you know, um,
destroyed so that they fire less often.
And as a result of all of that, you over
the course of years learn to act in the
world and you learn all sorts of skills
and you learn world models. You learn
like beliefs about the world and you can
sort of like mentally simulate how it's
going and stuff like that. So artificial
neurallets are like that except
artificial. So it's it starts off as a
giant
tangled spaghetti mess of randomly
generated uh artificial connections
called parameters. these days there
might be something like 10 trillion
parameters uh in the biggest AIS. So it
starts off randomly generated. So, it's
of course completely useless. Like if
you give it some input, it'll just
produce gibberish as an output. But then
they train it. And they start with
pre-training, which is where you give it
a bunch of internet text, and you show
it the first piece of text, and you put
that in as the input, and then it gives
a gibberish output, and then you
positively or negatively reinforce it
based on how accurate that output was at
predicting the next piece of text. Um,
so it's basically playing this game of
like predict the next word.
>> Isn't that how it happens with babies? I
had a I think I had a neuroscientist
tell me that babies have more neural
connections um than adults. And yeah, it
says yeah, toddlers have twice as many
neural connections as adults. And they I
guess they whittle down through
reinforcement.
>> Yep.
>> We have more pathways when we're younger
and just like the process of training an
AI, we're trained down to like remove
the ones that aren't useful and build up
on the ones that are.
>> Yeah. It's both pruning and
strengthening. Okay, it seems like in
humans it's actually more pruning than
strengthening, but it's both. Uh, and in
AI it's the same thing. It's both. So
the first portion of training is where
they train the AI to predict text, which
is kind of like training it to read. Um,
and it a similar thing does happen in
humans. So basically the the random
tangle gradually takes shape and
gradually sort of coaleses into more
useful circuitry that has stored lots of
facts about the world and has stored
lots of skills for how to you know
process information and transform it and
then produce predictions. That's just
the first step. After they do the
pre-training then they try to teach it
more useful skills besides just
predicting text. And so, you know, by
the end of the process, they've thrown
lots of coding problems at it and
they've said like, "Here's a coding
problem. Go. Here's a coding problem.
Here's an environment. You have access
to this virtual computer. Here's like
the codebase you're working with. You
can write code. You can edit the code.
You can run the code. You can read it.
You can use the internet. Go." And it
does that for a while. And then based on
how successful it is, reinforcement
happens. and they have thousands maybe
millions of examples of coding problems
like that that they train it on and
that's why they're so good at coding
now.
>> So what does super intelligence look
like in this regard? Is it just more of
these connections and how would they get
more connections? Can you explain that
to me like on
>> So there's different AI models, right?
So there's like you know GPT3 and GPT4
and GPD 4.5 and GPD5 and GPD 5.5 and
5.6, right? Sometimes they're just the
same previous model but with extra
training. Mhm.
>> Sometimes there are a whole new model
that's been trained from scratch,
including starting the whole
pre-training process again. Over the
last couple years, they've done several
new rounds of starting over from
scratch. And typically, when they start
over from scratch, they make the whole
thing bigger, the artificial brain much
bigger.
>> Okay.
>> Right now, they're at something like 10
trillion parameters. Back in 2020, um it
was more like 175 billion.
>> So, we've grown like two orders of
magnitude uh in six years.
>> Two orders of magnitude. Yeah, like two
10 x's, so 100x, right?
>> So that process is continuing. Um,
they're also improving the algorithms
themselves. So they're not literally
just the same type of AI, but bigger.
They've also come up with all sorts of
ideas for how to change the structure of
the of the connections and the neurons
and so forth and change the like
reinforcement
algorithms that they're using and to
change the training data that they're
training on. All sorts of tweaks that
have made this whole thing more
efficient. We're literally building a
brain
>> basically. Yeah. As they make more
brains, they're getting better at making
they're making them bigger and making
them more efficient and so forth.
>> And it's literally modeled on the brain,
like the way it works, right?
>> It's it's certainly heavily inspired by
the brain, but I I shouldn't overstate
the analogy. Like there's lots of
differences, too. So, for example, the
transformer architecture um
>> which is
>> which is the architecture that they use
for for these LLMs, uh is not really
recurrent. So the information sort of
flows one way rather than allowing all
these sort of little loops on the
inside. Also the the back propagation
algorithm is different from the sort of
um learning that naturally happens in
human brain. So there are some
differences but yes like broadly
speaking uh we are sort of making
artificial brains. It's kind of like for
brains what like a plane is for a bird.
>> Mhm. That's a [clears throat] really
good analogy. Yeah,
>> that that analogy helped me think
through a bunch of questions people
often ask about AI when they said can it
be creative but actually that analogy
kind of helps me understand that
actually that maybe that's not the
question it's can it produce something
that you would consider to be creative
because [clears throat] creativity is
people think of it as like a process but
actually it's it's judged based on the
output isn't it? I mean, you you can get
philosophical about like, is it truly
creativity that they have, but you can
also be like, well, I mean, just look at
all the stuff they're accomplishing,
[laughter]
you know, and uh it seems like they're
going to be accomplishing a lot more in
the near future.
>> Yeah, I do I I asked the question about
how this weighs on you personally
because I can I can sense that you're
actually personally bothered.
>> I mean, I think the situation is crazy.
Like,
first of all, it's very exciting. Like,
AI is really fascinating and interesting
stuff. I've been following the field for
more than a decade now. Um, I've been
part of it uh for some years and um it's
really cool, really interesting and it's
really fun to think about what's going
on inside these artificial brains and
why they are the way that they are. And
it's really cool to see all the
applications of this technology out in
the world. But it really seems like
we're on a pretty scary path. And the
more you think about it, the more
worried you get. And you know, in
stories it always ends well, but this is
real life. And I I think we have to sort
of stare reality in the face and tell it
and realize that like it might not
actually end well, you know.
>> Were there any recent dare I say I was
going to say Eureka moments, but
paradigm shifting moments where even
your own sort of mental model of what's
going on here and how this is going to
look were changed for better or for
worse.
>> For better or for worse and probably for
worse, things are kind of on track for
AI 2027. There are a few things that
have been different. not exactly like
paradigm shift differences but like
there have been some differences from
what we expected at the time we wrote
this. So the government has actually got
involved faster than we expected and has
been more aggressive than we expected.
So the export controls on mythos being
uh the biggest example and also
threatening anthropic with uh being
destroyed by the defense production act.
Um [clears throat]
>> another thing that's been surprising to
us is that anthropic in particular has
gone from second place to first place in
the sort of in the race basically. Why
do you think that happened? Because it
seemed like chat Gvt were out front and
clear as it relates to OpenAI were out
front and clear, but suddenly Anthropic
have uh lapped them.
>> Yeah, I mean I guess they have um
probably higher talent density um and
better strategy, but not by a lot, but
enough to make the difference.
>> Why do you think they have more talent?
>> Well, they don't have more compute. Like
what are the inputs, right? Like they're
in the lead now. They used to be behind.
What are the possible explanations for
this? Well, it could have been that they
had more resources, like more compute,
more money. But that's not true. They
have less resources and less money,
right? So then I guess talent is is the
next best alternative. You could maybe
say strategy,
>> some combination of those things. Yeah.
>> Something that wasn't just like the
amount of resources they had.
>> Just like Jon Jones, where marginal
improvements in your cognitive
performance can have a massive impact.
Sometimes I podcast for 10 hours a day.
Over the last couple of weeks, I've been
in filming for a TV show and then I have
like one or two days off to get all of
my work done, which means there's lots
of cognitive load. And so, I turned to
ketones because I find myself more
articulate, able to think more clearly,
able to work out better when I'm fueled
by ketones. And so, the reason I became
a co-owner of this company and the
reason why they now are a sponsor of
this podcast is because I remember one
of my team members called Cristiana, she
tried it once and came up to my desk and
she goes, "This is the best product ever
made." And I think in part that's
because she really cares about those
cognitive benefits as I do, as John
Jones does, and as I think most of my
listeners probably will. So, if you
haven't tried these yet, all you have to
do is go to ketone.com/stephven
and you'll also get 30% off your first
subscription order. You'll get exclusive
Ketone IQ merch and of course, cognitive
benefits that might just change your
life. Much of the reason most people
haven't posted content or built their
personal brand is because it's hard and
it's timeconuming and we're all very
very busy. And if you've never posted
something before, there's so many
factors in your psychology that stop you
wanting to post, what people will think
of you. Am I doing this right? Is the
thing I'm saying absolutely stupid? All
of these result in paralysis, which
means you don't post and your feed goes
bare. I'm an investor in a company
called Stanto, which you've probably
heard me talk about. And what they've
been building is this new tool called
Stanley that uses AI, looks at your
feed, looks at your tone of voice, looks
at your history, looks at your best
performing posts, and tells you what you
should post, makes those posts for you.
You can also just use it for
inspiration. And sometimes what we need
when we're thinking about doing a post
for our social media channels is
inspiration. Building an audience has
fundamentally changed my life, and I
think it could change yours, too. So,
I'm inviting you to give this new tool a
shot and let me know what you think. All
you have to do is search coach.stand
stand.store now to get started.
>> Uh, a friend of mine who knows some of
these people sat me down once upon a
time in London. He's actually said this
a few times to me, but I remember one
particular conversation where he says
that some of these AI CEOs predict the
probability of extinction at being I
think he said 7%. I don't know why I
have that number in my head, but I
remember it being less than 10%. And the
point he was making to me was that even
if it was 1%, like if there was a 100
buttons on this table now
>> Yeah. and one of them would end the
world. Would I dare press any of them?
You know, [laughter]
um,
>> no,
>> I wouldn't press any of them. But he
made the case to me that these AI CEOs
are very smart and they understand super
intelligence and that they think
actually if there was a 100 buttons on
this table right now, maybe 10 of them
could end the world. I've heard you say,
I think it was on the the the Daily
Show, the interview did, you said that
you think there's a 70% chance of human
extinction due to AI.
>> I wouldn't say human extinction exactly.
I would say something like 70% chance
that this goes horribly wrong like human
extinction. But that's just one of
several possibilities. But yeah,
basically
like for example, possibly the AI take
over and then don't actually kill
everyone. You know, maybe they do
something else. Like just just because
they've taken over doesn't mean they're
definitely going to kill us, right? They
might, but they could do something else.
So that's what that's that's why I don't
usually say like 70% chance of like
actual human extinction, but 70% chance
of like something like AI is taking over
some some sort of very big catastrophe
like that that could lead to human.
>> I guess I got two points there, which is
you've been around these CEOs. I mean,
you've worked for Sam Alman at OpenAI
before you quit. Do you think that they
think there's a chance of human
extinction?
>> Yes. But I think that an important thing
to understand is that like people sort
of believe what they need to believe in
order to think that they're good people
and that they need to keep doing what
they're doing. This is what
rationalization is. And so I think that
the tech CEOs have like genuinely
convinced themselves that like probably
things are going to be fine and that the
way to make things fine is for them to
keep doing what they're doing. And like
they need to like make sure that like
you know Sam needs to make Sam's
probably thinking like can't let Dario
or Elon get there first. You know I know
Dario is thinking Sam can't get there
first. Elon's thinking that like you
know they they they've all probably
convinced themselves that like oh yeah
like maybe it'll go horribly wrong but
like probably it's going to be fine and
probably, you know, I should be the one
in charge. It appears to me that
Anthropic are the only ones that are
talking about the potential chance of
extinction or a catastrophic event or um
the down the real downside still. They
seem to be the only ones that are still
publishing on it and now they're
actually becoming the enemy in many
respects of the
>> the tech industry in San Francisco. I'm
watching a lot of interviews and it's
everyone's attacking Dario because he's
saying listen things could go bad.
They're calling him a doomer and
questioning his incentives. Even with
Mythos, which is a a claude model that
they started to warn the world about,
again, he is attacked immediately for
saying that.
>> Yeah.
>> My question is, do you see him as being
slightly different from Sam in this
regard?
>> Yeah. I mean, it seems like Anthropic
and Stereo have been more willing to
say and do things that are costly to
their bottom line uh at least in the
last year or so. That's an example of
it. um like I don't think that really
wins them favors in the administration
or among their investors to say that
type of thing and you know a better
example is just the whole fight between
the department of war and anthropic was
an example of them doing something that
like cost them a lot of money and even
more importantly cost them a lot of
power for something like like they could
have just signed a contract you know
that said I really don't want to be in a
situation where we're like which CEO is
the least bad CEO let's support that one
you know like none of these people
should be trusted did uh with that much
power basically.
>> Nobody should.
>> Nobody should
>> regardless.
>> Regardless. Yeah.
>> Mhm. So on this point of the buttons,
you you you do believe that they think
there's a credible chance of extinction.
>> Yeah. But [clears throat] they've
convinced themselves that like it's
probably fine and also it'll be even
worse if I'm not doing it, you know?
Like that that's what they'll say inside
the companies too. Like two people will
be like, "Okay, well if we stop, what
about the other guys?" Like they're not
going to stop, you know?
>> Yeah. This is this has always been why
I've had this outstanding question which
is how does this not go bad when human
incentives seem to rule the day when you
look at history and all of the human
incentives are saying well if you you're
damned if you do i.e. you're damned if
you carry on developing these bigger and
bigger and bigger AI brains but you're
also then damned if you don't from a
geographical perspective because the
United States will lose to that country
or this company will lose to that
company. So when you just look at human
incentives and goes how does how does if
just purely incentives and disincentives
how does this end? Well it carries on
going. seems like it. I mean there there
is a caveat to that which is a hopeful
caveat which is that first of all if the
world wakes up to all of this then there
can be a more serious conversation about
regulation and international treaties
and things like that and that can change
the incentives right so the government
could come in and say like actually here
are some rules that you all have to
follow and because they're rules that
you all have to follow then you're not
incentivized to like break them anymore
because you get punished if you break
and everyone else is also following them
too and so you know it's fine. So so
there is that sort of like ray of hope
that like we can change the incentives
if the government and especially the US
government but then later other
countries act to to change the
incentives but that's not going to
happen until people sort of wake up to
all of this. The second thing is that
even individually at some point,
you know, Dario or Sam or Elon might
realize that like actually it's like not
even in their own interest to to keep
racing unilaterally.
And it the problem with that is it's
only if it gets extremely obvious and
extremely dire. So like in in AI 2027 in
that scenario, there's this choice point
that I mentioned and in one case the AI
are misaligned, in the other case the
AIS are aligned. At that choice point,
we have like one branch that depicts the
the misalignment ending and one branch
that depicts like they they slow down a
bit and solve the alignment issues.
>> The instigator for that choice point is
they see some evidence that their AI
might be misaligned and plotting against
them, right? So, if you actually see
that evidence,
>> then it's like, oh gosh, uh maybe we
shouldn't put it in charge of everything
and let it rip, you know, because that's
evidence is staring us right in the face
that it's it's untrustworthy, you know.
But if they don't see that sort of very
clear evidence, then I think they're
going to convince themselves that they
need to keep going, you know, but maybe
they will see very clear evidence like
that. In which case, even if we don't
have regulation, they might just sort of
voluntarily stop. Um, so that's the
second ray of hope. Like overall, I
don't think that we're like definitely
doomed. You know, like [snorts] I said
70%, but like I could see it working out
pretty well as well.
>> H what about uh jobs?
>> Yeah.
So I think I think I'm excited to at
some point get into the new thing which
is the more optimistic [clears throat]
positive vision.
>> Uh and that will have a lot to say about
this because in the in the in the
prediction you know in AI 2027 by the
time everyone loses their jobs there are
worse things happening or like it's it's
kind of like too late by that point. Um,
but yes, like once if I mean just just
think about it. If the companies do
manage to build super intelligence, then
by definition they're going to be able
to take almost all the jobs or all the
jobs, right? Because it's better,
faster, cheaper than the best humans at
everything.
>> And that again, the timeline is by the
end of sort of 2030, you reckon you
think super intelligence might arrive?
I'm trying to think about when we could
start to see job displacement in the
economy.
>> We're already starting to see a little
bit of it now, but not very much.
>> Why?
>> Um, cuz the AI aren't good enough yet.
like they they're they're they're
impressive, but they're not like they're
not just a drop in replacement for a
human worker in almost any field.
>> And do you think that'll be sudden?
>> I think it'll be sudden because of the
intelligence explosion dynamics or
recursive self-improvement dynamics. So
you can imagine a different world where
it's gradual.
>> Mhm. And and [clears throat] this is
this is maybe how it is in a lot of
science fiction is you know the AIS
gradually get better at a bunch of
things and you know they gradually
automate like this one industry like
pharma then they automate like you know
steering drones then they automate like
driving cars or something like that. Um
but what's different about the real
world is that the companies have
converged on this strategy of automating
themselves first, you know, automating
the AI research process. And so
if they're allowed to continue with this
strategy,
we're not going to see like, you know,
the robo taxis and like the plumber
robots and, you know, the lawyer AIs.
We're not going to see that sort of like
broad diffusion of AI into the economy
happening first because that's not what
they're focusing on first. They're
focusing on automating themselves,
automating their own research so that
they can do everything that they're
doing faster. And they want that to sort
of get going and get to, you know, very
high levels of intelligence, very high
levels of general intelligence. Um, and
then deploy more out into the economy,
right? So by the time it's actually
coming for like all these different jobs
they will have had fully autonomous AI
research happening for months maybe
years you know and that means that like
the AIs will be vastly superhuman at AI
research and probably also vastly
superhuman at lots of other things just
as a side effect you know if you're
wondering what this looks like well we
wrote about what it looks like it's sort
of like this this wave smashing through
the economy after they do the
intelligence explosion internally.
>> What I'm hearing there is that because
the AI will be able to improve itself
and train itself, it'll be getting
better at everything at once and then it
will be released at kind of once. Is
that accurate?
>> But it's it's not it's not even exactly
that because even if it's mostly just
getting better at the things that it's
doing like re research that'll have some
spillover effects to other skills as
well and then when it turns to focusing
on the those other skills it'll be able
to do them very fast.
>> What jobs remain in such a scenario do
you think?
I think that's actually a political
question, not a technical question
>> because
>> because on a technical level, all the
jobs can be done by the AIS if they've
reached that level. And so it's a
question of what jobs are allowed for
them to do
>> and what kind of jobs wouldn't be
allowed, do you think?
>> That depends on who's in charge. So
there'd be some sort of political
conversation about like what we're going
to allow and disallow.
>> I mean, in this scenario, the humans are
still controlling them. The AIs
>> depends on what you mean by control,
right? So there's like there's do the
AIs actually have the goals and values
that you want them to have and are they
going to robustly do that and behave as
intended into the future and then
there's like are they obeying your
orders for now?
>> Are they obeying the orders is really
what I'm saying.
>> Yeah. So like even in AI27 in the
scenario where the AI take over and kill
everyone, there's a period of like
several years where they're still
obeying orders and they're, you know,
taking some jobs but not other jobs and
they're helping to make better weapons
that the US government can use to like
do its arms race with China and so
forth. And that's why they're able to
get so much power so quickly is because
the governments and the corporations and
so forth trusts them and is deliberately
deploying them into all of these
positions because it thinks that things
are fine. But because these things are
neural nets, you can't just like look
inside and see what it's really
thinking. You can't really tell. I think
this is a really important point because
unlike software we can look at the code
and see what's going on theoretically
with AI you're saying that we don't know
what why it's making the decisions that
it's making because we can't get inside.
>> One note of optimism is that it doesn't
necessarily have to be that way. Like
there's a a sub field of machine
learning called mechanistic
interpretability and a broader subfield
called interpretability more generally
that's trying to solve that problem and
trying to take these these trained
artificial neural nets and piece
[snorts] them apart and understand like
how the information is flowing and how
the decisions are being made so to
speak. Um the problem is just it's a
very inherently hard problem. If you
have 10 trillion connections to look at,
you can look at any particular group of
them and be like, okay, so this is how
like this particular connection works,
but like how do you get a sense of the
whole, you know, how do you get a sense
of like
>> what's happening at a high level? And
the answer is, well, it might be
impossible, but people are working on it
and they are making progress and if they
can make enough progress, then we're in
a very different and much brighter
world. I think that it would be much
less likely for us to get into those
loss of control scenarios if we could
just actually see what our AIS were
thinking and why and how at any given
time, right?
>> Yeah.
>> So, we would still have the other
problems to worry about, but at least we
could mostly solve that one.
>> It is pretty crazy to think that we're
building a technology, a brain that we
don't understand.
>> Yeah, it's pretty crazy. I mean, it's
one of those things where like
>> in a movie, like a sci-fi movie, a bunch
of scientists stood around this big
brain and they're all just like they're
making it more they're feeding it.
>> Yeah. Yeah,
>> but they don't really know what the
>> Yeah, I mean it's it's kind of just like
obviously a dangerous thing to be doing.
>> Um, but we're doing it anyway because of
this history of how the field has
developed in the last 10 years where,
you know, people were like, "Oh, wow.
Yeah, that's obviously dangerous. Oh,
no. What if someone else did it and did
a bad job of it, therefore we should do
it and do a good job of it." And now
they're in this race where where they're
racing each other. And they're also
under all sorts of political pressure to
like pretend that it's not as bad as it
seems because they don't want to like
anger their investors. They don't want
to anger the White House.
>> One of the the key questions we had from
our audience was which and I kind of
asked you this in part, but which jobs
are genuinely likely to survive AI and
what skills should people/ students
focus on over the next 10 years? That's
kind of like like imagine if you were
someone living in Mexico in like 1500
and then you hear that like the
concistadors are coming.
You could be asking yourself like okay
well what sort of job should I be
switching to to like survive this
transition. But like you have a lot more
to worry about besides that. But yes, I
think I would say that like if we manage
to avoid the loss of control problem and
we end up with humans still in charge of
the AI and humans can like say what the
AI's goals and values are supposed to be
even as they become much smarter than
humans and even as they run the whole
economy then probably there will be
regulation that protects some areas and
you can try to guess at what those areas
might be maybe stuff that's more like
like like judges potentially. What about
podcasters?
>> Be honest.
>> Probably not podcasters, I think. Um,
stuff like, you know, being a nanny,
maybe, right? Like, I think it's even if
there's a robot nanny that's like really
really good, I think a bunch of people
might prefer to have an actual human
because they might be creeped out by the
idea of a really good robot nanny. So,
you can sort of you can sort of reason
like that. There's also like stuff that
might be legally protected, like maybe
judges, for example, like are going to
be legally required to be humans and not
robots. Some people say though there's
going to be so many jobs created that we
can't foresee right now like there was
in the industrial revolution or the
internet boom or whatever.
>> The problem with that is that um past
technological advancements have been
more narrow. They've like automated some
things but not everything. But we are
talking about a hypothetical future
situation in which everything gets
automated. So there isn't any new job
that you could do that the AI couldn't
also do except if it's like protected by
regulation or something. That's that's
that's also a thing. But so like for
example, right now there's this sort of
like cycle where
you know the AI has learned to do a
certain thing like write copy or like
draft code or like debug something and
then humans who used to do that thing
switch to managing AIS or switch to
doing the other stuff that the AIS can't
do. And that's why there's been this
dynamic historically of, you know, new
jobs opening up and people flooding to
them. But if it gets to the point where
the AI can do everything that humans can
do and better and faster and cheaper,
then whatever that new job is that you
might have switched to, like the AI can
switch to that, too. And they'll already
be better at it than you.
>> Because we haven't seen widespread
unemployment yet in the economy, do you
think people are getting a little bit
complacent? Because what I'm seeing on
my timeline is a lot of people saying,
"I told you so. I told you everything
would be fine." And when you look at the
the US unemployment rate, currently the
it's flat to slightly down. If you look
at the UK, it is up. The trend is up
compared to last year. We're at about 5%
unemployment. The US is at 4.2%
unemployment.
>> Yeah. Basically, nobody has said that
there would be mass unemployment by now.
Or at least we didn't say that, you
know, and we were historically one of
the more bullish people on AI progress
in AI 2027 because of the dynamics that
we just described. The mass unemployment
doesn't happen until 2028 or 2029 after
they already have super intelligence
because again the companies aren't
trying to cause mass unemployment as
step one. That's like step three after
you know it's like step one automate
themselves. Step two have this recursive
self-improvement to get to super
intelligence. Step three expand out into
the economy and automate everything. And
so this is really unfortunate from
humanity's perspective because one might
have hoped that if there was this broad
wave of automation going through the
economy, people would sit up and pay
attention and think about where all this
is headed and demand good regulations
from the government. But that's not
actually the strategy the companies are
taking. you know, they're going to be
getting the super intelligence first and
then doing the broad wave of automation,
which means that by the time they're
actually doing all of that, uh, well,
it's already going to be moving very
fast and the AI will already be very
powerful.
>> In your 2027 report, so you wrote that
in 2025, but it is called AI 2027. You
said that in mid 2025, we'd have the
autonomous employee, which is sort of
like AI agents taking instructions over
Slack or Teams.
>> That happened. I've actually got an AI
agent in my WhatsApp, which I talk to,
of course. I've got Clawbot exploded
obviously around the world and and now
um you know Claude have talked about uh
their new Slack integration but lots of
people are using agents now and that
happened I'd say for us at the we really
sort of caught on to it the the start of
2026 you also said by by 2026 companies
begin replacing entire corporate
departments with AI agent subscriptions
2027 the final job AI automates the job
of the human AI researchers themselves
and begins the machine learning research
to upgrade and build the next generation
of AIs.
>> Yeah. Yeah. So again timelines we are
uncertain about how long it will take to
achieve these milestones in this
scenario they happen at those times but
by the time we had actually published
the scenario our timelines had shifted
back a little bit specifically mine had
so like my 50% mark was 2028
>> for that for the full automation of AI
research milestone not 2027
uh and then other people on my team had
more like 2030 2031 things like that so
I I I kind of want to Excellent.
>> Maybe try to illustrate this with, you
know, we have like this probability
distribution. It's like a smeared out
probability mass and like the 50% mark
is this particular year, but there's
like a lot of possibility that it
happens
>> years earlier or years later, right?
>> What is this AI 2040?
>> So AI 257 was a best guess prediction as
to how things would actually go.
>> Yeah.
>> AI 2040 plan A is our recommendation for
how things should go. So we called it AI
2040 because in this scenario uh they
build super intelligence in 2040 instead
of much sooner because they delay
things.
>> Why do they delay things?
>> To manage the risks and make sure that
power is distributed equitably. They
basically like
regulate AI development so that it still
continues but at a slower more
reasonable pace in a more transparent
and safe way and spread out over more
countries and companies and as a result
they get to super intelligence in 2040
instead of in say 2030 and then we call
it plan A because well it's our
recommendation like we've we've come up
with a plan for what government should
do and uh the scenario is an
illustration of what it might look like
to imple ment that plan in a similar way
to how AI27 is kind of an an
illustration of what it might look like
to do what the companies are currently
planning to do. That makes sense.
>> And is this wishful thinking or is this
what you think is going to happen?
>> No, it's definitely not what we think is
going to happen.
>> It's not what you think is going to
happen.
>> No. No. What we think is going to happen
is still something more like this,
right? We we don't expect the world to
listen to us, right? this is our
recommendation, but we mean we hope that
that people do something like this and
we think it's possible, but it's not our
like prediction for what's going to
happen by default, you know. So, I do
want to run through the plans, the
potential plans and also plan A, but um
just to close off on how things might
look after the year cuz I think I wanted
to touch on robotics too and I've got
this graph here which talks about share
of labor output.
>> Yeah.
>> Um which I found to be quite striking. I
I've been sat here wondering as an
employer who employs hundreds and
hundreds of people
>> when when all this stuff is going to
happen and you know we're still hiring
more people as things stand there are
some roles where our consideration is
changing shifting considerably
>> and I'd have to say that you know we're
probably in a phase where our teams are
AI powered and they're using agents to
do some of their work now
>> but I'm wondering as an employer like
when is it when does this happen?
>> Yeah, great question. So if we could
maybe zoom in on this a little bit.
>> Yeah, we'll put it on the screen. Um, so
this is in the AI 2040 plan A scenario
and notably in that scenario there's
significant regulation introduced in
2029 that slows down the pace of AI
development. In the scenario, they do
that sort of at the last moment. So in
the scenario, if they hadn't done that,
then it was about to take off similar to
how it does in AI 247. Um, but as you
can see, like in the scenario, there's
still a bunch of jobs at the point that
they implement it. And this gets back to
what I was saying earlier is that if you
wait until most people have lost their
jobs to regulate the AI companies,
that's already too late because they
will probably already have super
intelligent AI by then because their
strategy is to first get super
intelligent AI and then do all that
stuff.
>> And I think you say that it would
collapse the economy and cause even more
harm to suddenly regulate something that
all of us and all of our lives were then
at that point relying on.
>> Oh, but it's a risk well worth taking. I
mean, we it's true that right now a lot
of people use AI for a lot of things,
but like if we could somehow slow or
halt AI development now to set up a
better way to do it, that would be well
worth it. Um, even though there would be
significant costs,
>> but you can't over here, right, can you
at this point where AI and robotics are
doing most of the labor output.
>> That's right. But in but in but in in
this scenario in the AI 2040 plan A
scenario they put in the regulations in
2029
>> and then they slowly and carefully
develop AI in a way that avoids all the
problems which we can get into in a
little bit. And so eventually, yes,
eventually the AIs take the jobs.
Eventually, basically the whole economy
is run by AIs and robots, but it it
happens gradually over the course of the
2030s
>> instead of happening in this sort of
crazy shock, you know, a year later,
>> right?
>> Because in this scenario, they don't let
the companies recursively self-improve
and get to super intelligence as fast as
possible. Instead, they regulate AI
development so that the core
capabilities of the AIS are improving at
a more reasonable pace and also in a
more transparent way so that the
scientific community can see what's
going on and help make it safe.
>> But it's uh I guess I noticed here that
in both your scenarios eventually AI and
robotics do pretty much all the jobs.
>> Yes.
>> So you kind of side there with Elon when
Elon says that working will be a choice.
Uh
>> because I mean what happens?
>> Yes. I mean if [laughter] it by
definition if it can do all the things
then it can do all the things. I think
that there's a question of like should
we allow there to be AIS that can do all
the things right? Some people think that
the answer is no and we should just shut
it all down and prevent these types of
AIs from being created in the first
place. And we're actually kind of
sympathetic to that. We we have our
Should we bring out the plans diagram?
>> Yeah.
>> Thanks. Yeah. So
our scenario is called AI 240 plan A.
It's a scenario in which they slow down
AI development to make super
intelligence happen in 2040 instead of
earlier. And plan A is our
recommendation. So this is sort of
illustrating our recommendation. But for
comparison, we made like mini scenarios
illustrating different alternative plans
which we call plan S, plan B, plan C,
and plan D. Plan D is basically the same
thing that happens in AI27. Like the
race continues. there's very little
regulation. Um, you can read about that
in AI27. Plan C also very similar to
what happens in the slowdown ending of
AI27 where they solve the alignment
problems. So, in that ending, they like
slow down a little bit, pivot more
resources to AI alignment and AI safety
research, get lucky and succeed, and now
they have aligned AIS and then they
speed up again and take all the jobs and
beat China and all those things. Plan B
is
it's kind of like plan C in that well
basically in plan B you're uh being more
aggressive towards China and you're like
taking actions to sabotage or cyber
attack them to like keep them behind so
that you have more breathing room to to
solve the alignment problems yourself.
Plan A is our recommendation. It's uh
domestic regulation and then an
international deal to continue building
AI, but in a much better way. Plan S is
shut it all down. If you want to have a
future where there aren't AIs running
around that can do everything better and
faster than humans, you kind of want
something like plan S.
>> What What do you want?
>> Plan A is our recommendation. I think
that I'm sympathetic to plan S, but for
reasons we explain, we recommend plan A
instead. And what do you think is most
probable if you're being honest?
>> Plan D,
>> which is that they just
>> type of thing where they keep racing,
they don't really slow down
significantly. Um, and uh things happen
extremely fast. The diagram sort of
explains like roughly the reasoning
behind this too. So like there's this
high level thing of like do you want to
keep racing as fast as possible to make
the AI smarter and smarter to put them
in charge of more things so that we can
be China? You know, if you're happy with
that, then you can get down into this
variation of optins here. If you are
worried about that, well, you get to
something like this. There's more
different options besides these, but
this is kind of like the ones that we
could compress onto a screen.
>> Do you have children?
>> Yeah, I have two children.
It's kind of sad. Like
I think that one way or another this
will probably all be over by the time
they're old enough to join the
workforce.
So I don't think they'll ever join the
workforce.
>> When you say this will be all over by
the time they join the what do you mean
by this will be all over?
>> So these milestones that I described
like AI is automating the AI research AI
is getting super intelligent. Um
AI is then exploding out into the
economy, taking the jobs, building robot
factories to build more robots, to build
more factories, etc. GDP starting to go
vertical. That sort of thing is what I
mean like all of those events
transpiring. Maybe there's like, you
know, 10 20% chance or something that
hits the wall and and none of this comes
to pass even if you don't do anything.
>> How old your oldest?
>> Six.
>> Six. Boy, girl,
>> girl,
>> girl. So your daughter comes to you and
says, "Dad, what shall I um what shall I
study in school?"
>> I mean, again, like if these radical
transformations happen, then the world
would just look completely different and
what sort of jobs you set yourself up
for basically won't matter that much
probably. I would say um that the thing
to do is well a try to make it actually
go well. Like if you can exert any
influence at all on history and how this
all develops, you should be trying very
hard to steer the future in better
directions. And then separately from
that on a personal level you should
focus on well
being a good person and doing things
that are sort of good in the for their
own sake rather than good because
they'll set you up for later employment
because that later employment is going
to be very uncertain. Um basically
>> Elon talks about this age of abundance
we're heading towards
age of abundance.
>> There'll definitely be abundance. The
question is who controls the abundance
and what do they do with it? Right? Are
the AIs controlled by anyone or are they
doing their own thing? And then if they
are controlled by people, who controls
them and what do they do? And what's the
sort of like political structure
governing how they make those decisions?
>> I think it was Jeffrey Hinton that said
to me, he said there's no example in
nature where a more intelligent species
is has less control than a less
intelligent species. Thus saying that
we're quite arrogant to think that in a
world where there's this artificial
brain that's a gazillion times the size
of mine that I'm going to give it
orders.
>> Yeah. I mean that that's the thing is I
I think it's like that should be our
default assumption is that like well
there's these brains. We can't see
exactly what they're thinking. We're
going to make them smarter than us and
put them in charge of everything
>> and then we're going to give them
bodies.
>> Yeah. And then they're going to be
autonomously building new factors and so
forth. And like how is this supposed to
end well again? Like isn't this just
exactly like us picking a new species
that's then going to out compete us when
it doesn't need us anymore? Like I think
that is just the default trajectory. Now
there's a whole argument we can get into
about like ways that we could get off of
that default trajectory. So for example,
there's research into interpretability
that I described previously. And if that
research bears fruit, then you will be
able to actually see what they're
thinking and then that would be an
excellent tool for shaping them and
controlling them and making sure that
they do what we want. Right? there's
other sorts of um AI alignment research
agendas that are making progress and if
enough of those agendas succeed
sufficiently we can avoid this problem
of course also there's the regulatory
side too where like part of what makes
this difficult is that we're building
these AIs in race conditions you know
like the the companies are secretive
about their recipes for making these AIs
because it's secrets that they want to
protect so that other people can't copy
them and so a lot of it is happening you
know behind closed doors only a few
people can really see the recipes that
they're using to train these AIs and and
so forth. And then often times when the
AI behave in unexpected ways or even
just like blatantly misaligned ways,
sometimes that information doesn't
really flow out to the public because
the companies are not really
incentivized to tell everyone about how
they messed up and how their AI is evil.
It's just not very conducive to
scientific progress on these issues. If
the regulatory system was different,
then perhaps we could be in a better
situation, make faster progress. Also,
of course, we wouldn't be planning to
put these AIs in charge of everything as
fast as possible, and we wouldn't be
planning to like let themsel improve,
you know, like the these are choices
that we could not make, you know?
I don't speak Vietnamese, but this show
can because of AI video technology from
our sponsor, Hey Jen. I get messages
every single week from those of you
listening to the D of Co all around the
world, and you express how much impact
it's had on you and your life. And if
that's true, then those conversations
shouldn't only reach people in English.
Ken can take one recording of me and
deliver it in any language while keeping
my voice, timing, and expressions
intact. But you don't need a studio like
this to make it work for you. Record 15
seconds of yourself and get an AI avatar
that delivers studio quality video in
over 175 languages. We're up to 20
languages now, and we're not the only
ones using it. Hey Jen is already used
by 30 million people, including 85% of
the Fortune 100. Whether you're building
an audience on social media, launching
an online course, or rolling out
training across your team, check out
Heyen now. Your first three videos are
totally free at heyjen.com/doac.
That's hygen.com/doac.
See you there.
Ilia was, as you said, he was one of the
leaders at OpenAI and he left and he
started his own company now, Safe Super
Intelligence. very curious name of a
company, Safe Super Intelligence, after
leaving OpenAI. Did you ever get to work
with him?
>> Uh, I wasn't directly working with him.
I had a couple chats with him.
>> Do you think he's he's genuinely
concerned as well?
>> I think he is, but I think he's I think
he's similar to these other CEOs where
I mean, just think about the sort of
incentives that they're under, right?
like
they can sort of see the problem and
then they can be like okay but like if I
don't if I stop if I quit my job and or
do something else that's not going to
solve the problem cuz the other CEOs are
going to keep going and even if all of
us didn't go then maybe China would keep
going. So like man seems like this is
just going to happen one way or another
whether I do anything about it or not. I
guess I should be involved, you know,
and like maybe I can make it go well.
And at any rate, like I don't want to be
out in the cold while these other people
I don't trust are in charge of
everything. So they all sort of like
reason through all of this and then
convince themselves that like the thing
to do is for them
>> to build it and to do it better. And I
think Ilia is just the latest example of
this. Elon's another example. Dario is
another example.
>> You know, arguably OpenAI at the
beginning Sam was an example. Although
like Elon and Dario were at OpenAI early
on. So
>> what do you think they should all do
then? So I think what should happen is
some sort of international regulation or
at least domestic regulation similar to
what we described in plan A.
>> Okay. So walk me through plan A.
>> Yeah. So in this scenario, AI takes
longer to the to get to recursive
self-improvement and full automation of
AI research than it does in AI 27. We
figured that we should try to illustrate
like a range of different possibilities
because we do have those sort of
uncertainty intervals. So we chose 2030
as the moment when full automation would
finally be achieved and things would
really kick off. And then working
backwards from that, when's the last
moment you could really have good
regulation? 2029. So in this scenario,
AI progress slows down a little bit
naturally and the AI companies keep keep
racing, but they don't quite succeed in
automating uh themselves in 2027 or in
2028 or in 2029, but they're getting
really close and they're going to do it
in 2030. And then in 2029, the
government steps in and regulates them.
What regulations do they do? Well, they
basically just shut it down temporarily.
>> Can I ask um
how does the elections overlay with your
time frames here? Because there's going
to be a big election, isn't there, in
2028?
>> And it seems now that sentiment has
really really turned against AI in in
sort of in the general public and that
it will be one of the big ticket items
on the on the ballot.
>> We think that it'll be maybe the most
important issue in the presidential
election in 2028. Um, I think a lot of
people mo most people will be quite
concerned about where things are headed.
And that's part of why we we chose to
depict things the way they were doing in
this scenario because that helps explain
why they might do this sort of
regulation in 2029 is that the voters
have been demanding it and the
presidential candidates have been
promising it.
>> And in this scenario and in 2027, would
the general public have felt the
consequences of AI much more severely
than they have now by then?
>> Yes. Although still even in 2029 in this
scenario they still mostly have the jobs
as as depicted here. Right. So in in
2029 in this scenario lots of jobs now
involve managing AI agents. You you
mentioned you have an AI agent right?
Well in 2029 in this scenario the AI
agents will be much better. Still though
not enough to just completely do
everything. You know that was the sort
of thing that would come in 2030
>> in this in this timeline. Again we're
uncertain about timelines. Things could
go faster than depicted in this
scenario. And in fact, I think things
probably will go a bit faster than
depicted in this scenario, but we're
uncertain. We already did the very fast
timeline scenario, so now we're doing a
slower timeline scenario. But maybe we
should talk about the high level goals.
So
they want to have AI continue, but in a
slower pace.
>> Who's that?
>> So they can make it safe. The
politicians, you know, the president and
the people who voted for the president
and, you know, the heads of other
governments and so forth. So goal one,
slow things down. Um, goal two, make it
more transparent so that the scientific
community can catch up to this stuff and
make more progress and also so that we
don't have to take the company's word
for it when they say that their systems
are safe and when they say that they
haven't, you know, put in any biases
into their systems, for example. That's
a concentration of power issue. We also
want to avoid a situation where there's
an intense concentration of power. So in
addition to these the transparency and
the slowdown, we actually think it's
actively good for there to be multiple
AI companies across multiple different
countries that have similar levels of
very advanced AI capability and for
there to be like broad diffusion of AI
into society rather than you know a
single mega project that has all the
best AIs for example. And the nice thing
about that is you kind of get that by
default if you do the first two things.
If you slow it down and if you make it
more transparent, then that means
there's breathing room for other
projects to sort of catch up, right? And
the transparency just like literally
helps them catch up because then they
can like copy copy some of the ideas.
And then I think the fourth thing would
be reversibility. So in what follows in
the scenario, we are going to be
building up a lot of data centers and a
lot of robots. We're going to be
transforming the world at a in a sort of
like slower pace, though still a very
fast pace, but slower. And if things go
wrong and the deal breaks down and
everyone starts racing each other again
to get to super intelligence as fast as
possible, that would be very scary. And
so the fourth principle is basically
build the new data centers in such a way
that if everything breaks down and
everyone starts racing again, the newly
built data centers get destroyed so that
we're sort of back to square one again
instead of in an even worse race where
there's even more AI and robust and
compute everywhere. Um, so I can sort of
walk you through the timeline if you're
interested. Sure.
>> Or the president talks to China, talks
to the leaders of a bunch of other
countries and says, "We're going to
basically halt AI development until we
can figure out a plan for how to do it
in the in ways that achieve these
goals." So they basically send
inspectors to each other's data centers.
Like Chinese inspectors come to US data
centers, US inspectors go to Chinese
data centers and verify that they are
doing inference and not training.
developing new AIS that's that involves
training them. But just taking existing
AIS and using them to serve customers
that's called inference. And so the sort
of like solution they come up with here
in this scenario is will allow them to
keep doing inference but not training
for now until we can get the new
training data center set up. So they
retrofit the existing data centers to
serve inference. People can still keep
talking to their AI agents, but they're
going to stop getting better and better
for like six months to a year while they
build the new data centers that are
going to be the transparent data
centers. And that's where the training
is going to happen. Once they get those
new data centers set up in 2030, then AI
research continues. This is a bit spicy.
We advocate for total research
transparency which means that on the
training data centers that are training
the new models they basically have to
publish everything which means you get
to see all the details of the recipes
for training these models you get to see
the architecture etc. We think that's
sort of open science is really important
for solving the alignment problem fast
enough because you don't want to have
these sort of biased companies making
the decisions about whether the AIS are
safe. Um and we also think it's
important for just good regulations more
generally because right now most of the
expertise in the world on AI is sort of
concentrated in Silicon Valley and the
the governments in particular kind of
are don't really understand AI that well
and imagine an alternative instead of
total research transparency you had like
an auditor system where the government
says here are some rules for how to make
the AI safe and then we're going to have
like an agency that like goes into the
companies and asks them questions and
tries to make sure that they're
following the rules that creates this
sort of adversarial dynamic where the
company is incentivized to like fool the
the regulator, [clears throat] you know,
and and also if they if they discover
some new problem that's not even on the
government's radar,
>> they're might be incentivized to like
not tell the government about it, right?
So, if you have the total transparency,
it helps the government make better
decisions faster, but
>> it kills that competitive advantage.
>> Yes, Dropbox's not going to like this.
You know, OpenAI is not going to like
this. This would be uh probably bad for
the valuations. I don't think it would
kill them completely, but it means that
it would commoditize more, right? So, it
means that there'd be like a bunch of AI
companies that would catch up to the
frontier. They would train AIs that are
like roughly similar, roughly
equivalent. They could still make money
by doing that and then selling their
AIs, but they wouldn't have a monopoly.
They wouldn't have anything close to
monopoly, which I think is good for
humanity, although it's bad for the
bottom line of those particular
companies. Notably, it's good for the
bottom line of lots of other companies.
Like if you're a company that's behind
and you don't you're not anthropic or
you're not open AI then you would love
this because this helps you catch up you
know or this this helps you to like um
capture more of the value from the chips
you're selling for example or from the
like downstream product that you're
making
>> and by 2031 then you have 1ifth of all
cognitive labor done by AI.
>> Yeah. So what's happening here is that
we're imagining that the government of
the United States and the government of
these other countries that are involved
in this agreement that are sort of
implementing similar regulations. Um
they don't have to be exactly the same.
Uh but that's another thing that's nice
about the transparency is that if you
have this sort of transparency then if
two governments are implementing
different regulations like if one of
them is like telling their companies to
go slower or like banning more stuff
than the other one is they can both see
>> Yeah. like, oh, you're letting them do
that sort of thing and you're not like
maybe we should let them do this too,
you know? So, it helps to sort of
naturally equalize the regulations to
some extent without having there to be a
central power that just gets to make
regulations for everybody.
>> Mhm.
>> So, anyhow, we're imagining that when
they when they get this transparency set
up, they basically agree to ban the
dangerous stuff to allow the not so
dangerous stuff. And there's a constant
ongoing conversation about like, well,
what's dangerous and what's not? What
should we ban? What should we allow?
What about this country? what about that
country? That conversation evolves over
time, but the gist of it is, at least if
they do it the way that we recommend it,
is that they don't do an intelligence
explosion. They don't let the AI, you
know, autonomously self-improve.
Instead, they slowly and carefully scale
up the AIS that they currently have and
invest lots into finding ways to make
them more interpretable, uh, to make
them more easy to control, to understand
better how they work, and so forth. The
result is that AI progress continues but
it's not quite as fast and it's much
much much safer and more transparent.
>> But still through these you we see job
disruption.
>> It is continuing because they are
building more data centers right like
this whole time they're building more
and more data centers more and more
chips and they're continuing to like
make there be a larger and larger
population of AIS so to speak and that
causes this huge transformation over the
course of the 2030s. sort of a big thing
that we sort of want people to take away
is that even if you heavily restrict AI
progress,
you still get this sort of crazy
transformation. You know, in this
scenario, they basically allow progress
to continue but at a slower, more safe
pace here in 2030 and then as a result,
it takes until 2035 to get to top expert
level AI. So remember, they were on
track to do that in 2030, but then sort
of at the last moment they stopped. But
because it was sort of so close to the
last moment, that means that like they
can sort of get there pretty soon if
they want to. And it's just a matter of
like how long they they allow it to go,
right? So they sort of they sort of slow
it down, spread it out, leisurely arrive
at this level after 5 years. By this
point, they've built up massive amounts
of data centers everywhere. So it's not
just that the AIs are smarter and able
to do all the things that humans can do,
but also there's a lot more of them and
there's a lot of robots and so forth. So
by this by this point you kind of have
the economy that a lot of people would
have imagined with AGI where there's AIS
there's lots of them they're able to do
all sorts of jobs there's robots there's
lots of them they're able to do all
sorts of physical work and basically the
economy is being run by these machines
>> so in 20 31 you you have the 1/5if of
all cognitive labor done by AI in 2023
you have 60 million AIs running at 100x
speed in 2033
there's cash dividend to all Americans.
>> Mhm.
>> Um I've got to
explain explain this to me.
>> Yeah. So, if the AIS are going to be
taking people's jobs, then it's very
important that people not starve to
death and still have money. And if
companies are going to be using AI and
robots to take all these jobs, then that
means that there needs to be some sort
of taxation scheme or something to like
make sure that people still have a slice
of that pie. M
>> the pie is going to grow huge, but you
still need to actually give people a
slice of the pie. And our proposal for
how to do that, we call it the citizens
dividend. Basically,
people have shares in a agency that
sells permits to the robot companies and
to the compute companies and makes
profit from selling those permits and
then those are people have shares in
that entity. It starts off small. It
starts off something like $25,000 per
person. Uh, and then by the end it's
something like $10 million per citizen
>> per person.
>> Per person per year.
>> Factoring in inflation. Like what you
mean?
>> Factoring in inflation.
>> So we're all going to be
multi-millionaires.
>> Yes. If this happens, which they
probably won't, but if it happens, this
is where it go. And again, this is the
thing I want to emphasize is that if you
get to the point where your AIs are
close to being able to do all the
research and then you sort of pause and
slow down, that means that like you
still have a lot of transformation ahead
of you because if you allow those AIs to
like still proceed slowly and like start
to automate various jobs and so forth
after some years they will in fact have
done that and they will have, you know,
built huge amounts of new data centers,
huge amounts of new chip fabs, huge
amounts of new robots, robot factories,
etc. You know, we're not sure obviously
how fast this will go exactly, but we've
thought about it a lot and we have our
our guesses and this is sort of like our
median guess.
>> What does this mean? 2037, the
apocalyptic arrival of truth on Earth.
>> Yeah. So, like this is the point where
we say they get to top expert level AI.
So, it's not super intelligence in the
sense that it's not like vastly smarter
than humans at things because they
deliberately pause it at the level of
top experts. So, so here they're going
slow. here they've just actually stopped
but they've stopped at a point where the
AIs are just actually really good at
everything. So kind of they've
definitely got AI maybe they got like
weak super intelligence
because they have so many of these AIs
and because they think faster than
humans you know they just run much
faster that's going to transform society
dramatically. So we talk about some of
the ways in which it transforms society
like this is sort of life after work we
talk about what it would be like to be
living on your cit citizens dividend and
not have a job anymore in this sort of
world. Um, here we talk about all the
scientific changes and all the social
changes that would come from all of the
intellectual progress and activity that
would be generated by all of these AIs.
So, for example, here is things like
cancer cures and like you know people
living in apartments that were built by
robots 2 years ago,
>> 2036
>> providing again we stop in 2029.
>> Yeah.
>> And providing I mean a conservative this
is a conservative time frame.
>> Yeah. Like unfortunately, I actually
think that things will happen faster
than this by default and that if we
don't slow down, things will happen much
faster than this. Once you get to the
point where you've got, you know, a
billion AIs running day and night and
they're each better than the best humans
at everything. And so they're doing a
lot of science. They're doing a lot of
talking to each other. They're doing a
lot of thinking. Everyone's constantly
talking to their AI assistants and so
forth. There's going to be a lot of
scientific progress. There's going to be
a lot of changes to politics, to
ideologies. It's going to be very
disruptive and crazy and we get into
some of the ways in which it is uh
later. Basically,
>> I still not super clear on what this
means. The apocalyptic arrival of truth
on Earth. It's just it's just because
there's so many eyes AIs that are so
smart that they're uncovering making new
discoveries in sciences.
>> Let me give you an example. Lie
detectors.
>> Yeah.
>> So, that's an example of a technology
that might be invented.
>> Yeah.
>> You know, right now we don't have good
lie detectors. We have very bad lie
detectors that like sort of work but
don't don't fully work. But once you've
had these top expert level AIs thinking
for many years at, you know, 100x human
speed and there's billions of them and
they have access to robot factories to
do research and stuff.
>> They'll probably invent a ton of
technologies. Maybe they'll invent on
lie detectors that actually work on real
humans. That'll have big social effects,
right? Imagine a presidential candidate
who's like, "Those allegations are
false, and to prove them, I will go
under a lie detector and say that
they're false." I was just thinking
about the whole justice system and how
that would be overturned. Um, in fact,
you could, you know, theoretically walk
down the street and be
>> Yeah,
>> it's both terrifying and exciting. One
thing that we talk about in this in this
section is like the invention of lie
detectors could be really bad. Like it
could be that it enables a new form of
totalitarianism where the powerful
people, you know, the CEOs and the
politicians
>> force the people under them to go under
lie detectors and say like, "Yes, I'm
loyal to the dear leader. I would never
do anything against the cheerleader,
right?
>> And if you're lying, then you're in.
>> And then if you're lying, you get fired,
right? So like there's there's a ton of
like very harmful uses of lie detector
technology. There's also the good uses.
And broadly speaking, I would say the
good uses are when lie detectors are
used on the powerful instead of by the
powerful.
>> What's this 2040 passing the torch to
AIS?
>> Yeah, great. So here they pause at the
top expert AAI level. And the reason why
they pause is because their safety cases
aren't good enough for going beyond that
level. Um, so in the sort of regulatory
systems that they set up over the course
of these years, roughly speaking, the
way they would work is when you're
making a new AI and then when you're
trying to deploy the AI into something,
you have to have some sort of safety
case explaining like what your
intentions are and like why you think
it's going to work the way that you want
it to work. And in particular, why the
AI is going to like
>> do as it's told, for example, and why
nothing super terrible is going to
happen like AI takeover. It's relatively
easy to make safety cases like this when
your AIs are still not capable of
automating everything,
>> but the more powerful they get, the more
difficult it is to actually argue that
things are going to be fine because the
AIS are just more capable and they can
they can get up to more stuff. And if
you if they're actually untrustworthy,
the the possible downsides are bigger.
So that's why they stop at this level is
that they they realize that if they keep
going, then they might actually lose
control of everything. But at the
current level, they're convinced by
safety cases that it's fine, but they
don't want to go further. So they stop
there. And then what happens in 2040 is
they've made significant progress
scientifically, including on alignment.
And they figured out how to make AIs
that are actually aligned in a robust
way
>> with humans
>> with humans. So they can actually trust
those AIs and they can allow them to
become much smarter again. So that's why
we call the whole thing AI 2040 because
in 2040 they sort of let off the brakes
and allow the AIs to become
significantly smarter than humans.
>> I guess you know this is a this is a
plan and this is a hope.
>> Yes.
>> But in reality this is not what you
think probabilistically if you had to
>> that's right it's important to
distinguish like this is what we
recommend. This is what we want to
happen from like this is what we
actually think will happen by default.
Now, we do think it's possible for this
to happen, but you know, that will
require a lot of people to sort of wake
up and pay more attention and advocate
for something like this to happen. So,
our main scenario is mostly talking
about the policy choices made and the
broad scale effects on society. We
figured it would also be nice to
accompany this with a little mini
scenario that describes what it would
actually feel like to live through this
from an ordinary person's perspective.
>> Okay.
>> Um 2029, everyone's yelling at each
other. the presidents are negotiating
something and they've paused AI but you
still have access to the existing AI so
it doesn't really feel that different
although it definitely is like something
exciting happening 2031 they've started
progress again the AIS are really smart
more people have lost their jobs it's
like really starting to actually affect
things but I think still most people
have their jobs but their jobs have sort
of transformed so like by 2031 it's like
most white collar jobs involve working
with AIs to a large extent or managing
teams of AIS or collaborating with them
somehow also there are some things like
robo taxis that are basically just
working citizens dividend, you know,
ideally this would happen sooner. Like
in our scenario, they kind of do things
at the last minute, you know, so like a
lot of these policy things are like
happening kind of like just in time.
Obviously, we would recommend that you
do them sooner and and do a better job
of them too, but so 2033 you start
getting your your checks from your
dividend. So you're forecasting that
there will be a citizens check that your
model says it could be around 25,000 at
the start per person
>> and then it would grow as the economy
grows.
>> But also as I guess job displacement
takes hold. They're going to need to to
grow that. Check make sure you
>> and that's why it's kind of the last
possible moment because if you waited to
implement this until like 2037 then like
everyone would have already lost their
jobs by the time that happens, right?
people losing their jobs, especially if
it happens quickly like like we see on
this sort of graph here, is going to
cause lots of problems in terms of civil
unrest, social unrest, purpose, mental
health, these kinds of things
theoretically.
>> Yes.
>> How do you think about that?
>> Uh it's it's going to be rough and
hopefully we can navigate that well. We
think that at a high level people need
to have money and also people need to
have power. And I think these are like
somewhat different things. It's like why
are jobs important? Well, there's a lot
of reasons why jobs are important, but I
think the main ones are um well, it's
how people get money so they can survive
and get the things that they want by
buying the things that they want. So, if
people are going to be losing their
jobs, you need some other way of people
getting money. And then there's also the
power thing, which is that right now
people have political power in part due
to their economic power. people can
threaten to go on strike for example or
you know countries that are ruled by
dictators can't
just completely you know genocide an
entire subop or they can but like it's
costly for them to do so because
then they'll have less money because
that subop is contributing to their
economy and contributing tax revenue and
so forth but if you end up in a world
where actually nobody's contributing tax
menu revenue except for the AI companies
and the robot companies then you're you
the government are less incentivized to
care about what you know the common
people think. So, so when people lose
their jobs, they're not just threatened
with lack of loss of income. They're
also threatened with loss of political
power. And so, we think that it's
important to like do things to push
against that.
>> What does that look like? How do you how
do people have power in such a world?
>> Well, in democracies at least, they
still have votes.
>> Okay. So I think that it's very
important for there to be uh regulations
on the use of AI that help make the
public discourse more sane and more um
actually giving the people what is in
their interest and what they want and
avoiding a sort of um opposite outcome
where you know the masses are easily
manipulated by AI powered media for
example or where everyone's talking all
to their AI advisers and the AI advisers
are like subtly steering them away from
voting for the candidate that would not
be what the AI companies want because
the AI companies have this other
candidate that they like better and
they're like secretly biasing their AIS
to like steer people towards voting for
that candidate, right? So, so we want to
be in a situation where um people have
AIs that are actually trustworthy and
that are truth seeeking AIs, honest AIs
and that don't have any sort of like
political agendas put into them by the
AI companies or by the government. You
know, you want to avoid a situation
where the AI company where where the
government has issued some sort of
secret order that like the AI have to be
such and such a way. Yeah. The
Department of War dispute versus
Enthropic is like a an interesting sort
of foreshadowing of this, right? where
um Enthropic was giving their AIS to the
Department of War. The Department of War
wanted to use them for certain things
and was upset that Enthropic's AIS were
like not supposed to be used for those
things. Uh that things in particular
were domestic surveillance and uh
autonomous robots.
>> Mhm. There's going to be a lot more
issues like that coming up and you want
it to be the case that like people know
what they're getting and that if people
are like spending hours a day talking to
their chatbot that chatbot doesn't have
political biases put into it or a secret
agenda or things like that and instead
has been trained to like give honest
true answers to things. And I think if
you can do that it can improve the
discourse and help people to use their
votes to put even better regulations and
even better politicians in place and so
forth. it can sort of potentially
bootstrap this to having something where
people's power is even more secure than
it is today.
>> A lot of the stuff we've we've covered
in part, so you know the wars and drones
and missiles, we're already seeing this
around the world at the moment, which is
really really interesting. Um, and we've
talked about robots outnumbering humans
as well, which is part of this
prediction. Some of the ones down here I
found to be really curious, which is
people will be protected by AIS wherever
they go.
>> Yeah. In this scenario, they delay the
creation of super intelligence until
2040 and in fact they pause from 2035
but then they let it go after then and
then they let the AIS become vastly
super intelligent. And we think that
once the AIs are vastly super
intelligent, the world will transform
even more radically than what happens in
the 2030s in this scenario. So in the
2030s in this scenario, it's more like
human level. You know, the AIS are not
they're they're doing the same sorts of
things that human experts would have
done. They're just doing it a bit
better, a bit faster, and a lot cheaper,
and there's a lot more of them. And the
robots are still, you know, doing the
same sorts of things that human workers
would have done. There's just more of
them, and they're cheaper. And because
of exponential growth, uh you start with
a world that looks not that different
from today in 2029, and then by 2039,
you end in a world that's radically
transformed, where everyone's living in
these like fancy new apartments that
were built by robots two years ago.
There's like giant special economic
zones that are full of robots and solar
panels and factories producing more
robots and solar panels and factories
and so forth. Most of the economy is AIS
and robots and people don't have jobs
anymore. That sort of transformation is
what you get if you pause at human
level. But if you go beyond the super
intelligence, there's a whole another
transformation coming that's going to
look more like magic. Think about how
the technology of today would look like
magic to someone from 500 years ago.
Mhm.
>> You know, and that's without even like a
qualitative improvement in intelligence,
right? Like the humans of today aren't
like qualitatively smarter than the
humans from 500 years ago. It's just
that we've had more time to do research
>> and we have more like money and
resources to build, you know, prototypes
and experiments and run experiments and
so forth. But if you had a point where
there were billions and billions of AIs
that were not only faster than humans,
but like qualitatively way way way
better at everything and in particular
at doing scientific research, we should
expect that some of the things that they
develop will seem like magic to us
>> and we'll just completely like we did
not think that was even possible. You
know, people don't want to die. People
don't want to be hit by cars. People
don't want to be like attacked by a
random mass murderer.
>> Cancer's gone. I mean not just cancer
like [snorts] you know all all a lot of
the stuff that happens in science
fiction will probably have happened by
then. So things like people scanning
their brains and uploading into into
computers right or self-replicating
robots in the asteroid belt uh creating
more and more satellites to
uh produce more and more power to
produce more and more self-replicating
robots and so forth.
>> Most people still live on Earth but the
trend is to move to space.
>> That's right. Yeah. So like if if you
end up in the situation where the entire
human economy is just like a tiny drop
in the bucket that is the entire economy
and it's just like this huge amounts of
robots and AIs that are moving
incredibly quickly, then what you want
is Earth to be mostly left as something
like a preserve. You know, I think a lot
of people are worried about the
environment being destroyed, which it
totally would be if it wasn't protected.
And uh you know there's a lot of people
who sort of like their lives as it is
and don't want to be uploaded or live in
some crazy new future thing. And it
seems to us like the reasonable solution
to these issues is uh create new living
spaces off the planet with some of that
vast economic wealth and activity that's
happening for the people who want that
sort of thing and then that way the
earth can be preserved.
>> Data picture here of data centers in the
ocean. Uh I mean there's three images
there of different environments where
humans might live.
>> Again like our proposal was you preserve
like 99% of the earth uh mostly as is as
historic or environmental from historic
or environmental reasons but then like
some parts of it you designate as
special economic zones where the robots
can go crazy and dig giant pit mines and
produce factories and so forth. Um, we
were thinking it would be good to build
the data centers on the ocean instead of
um on land for a variety of reasons,
although later space would be better and
we could see that being reasonable as
well.
>> What about immortality in a world of AI?
Um, 20 well 30 45 you say you've lived a
dozen lifetimes and are immortal,
passing from life to life as if by
reincarnation.
I mean, there's a lot of billionaires at
the moment that are focused on
longevity. I mean, Brian Johnson's said
he's got this central rule, which is do
not die right now.
>> Yeah.
>> Because we're in the age of AI and it's
conceivable that with super
intelligence, we'll be able to choose
when we die.
>> Yep. I think that's probably right. We
don't depict that happening in this part
because at this part they only have, you
know, human level AIs, but that's one of
those things that seems quite plausible
that super intelligence could achieve um
through a variety of means.
What is your hope with all of this stuff
now? Why did you do this? Why did you
make this 2040 plan ape?
>> In the like first week after we
published AI 2027, it it blew up a lot
bigger than we expected, by the way.
Like after we published AI 2027, it it
blew up a lot bigger than we expected,
by the way. Like we actually made
forecasts beforehand of like how many
views it would get and stuff like that,
and it was like 90th percentile outcome.
So like um very much not what we
expected. Um, but in like the Twitter
storm that happened, various people were
like, "Are why are you giving us all
this like doom and gloom uh predictions?
Like, how about a more positive vision
of like what you think we should do
instead?" And I think that that seed
sort of like implanted in us and then we
were like, "Yeah, that's reasonable."
Like, we've sort of depicted what we
think the default path looks like and
why we think it's pretty scary. Now
maybe we should switch facts and come up
with some actual recommendations and
then depict that as well
>> even though you don't believe they're
pro probable.
>> Yeah. I mean you can vote for a
political candidate even if you aren't
confident that they're going to win, you
know, and and you can say like here's
what I think we should do even if you
think that people are probably not going
to do it. You shouldn't say this if you
think it's completely unlikely. Like if
you think there's no chance then like
maybe you shouldn't bother. But we think
there's a chance. like in particular for
the reasons that we described in the
scenario we think that people are going
to wake up to the power of AI over the
next few years
>> because of something happens
>> the companies are saying that they're
going to do this [clears throat]
>> and
they are kind of on track and it just
sort of makes sense that like if they
get anywhere close to this level of AI
then there's like big issues and big
problems and like we need to like do
something about this and so I think that
Even if there's not any like very
dramatic warning shot or something, I
think that just naturally people are
going to start paying more attention to
this and reasoning through the
implications and trying to predict
what's going to happen. And so naturally
people are going to be more interested
in regulation of AI for example. And in
fact
there's actually like there's there's
actually more of this happening than we
predicted.
>> More of what happening?
>> Serious interest in reg AI regulation.
So, at the time that we published AI
2427, the sort of like mainstream
position of the tech companies and in
the government was kind of like AI
regulation, bad idea,
>> free-for-all.
>> Free-for-all. Yeah. In fact, there was
even an attempt to um preemptively ban
states from regulating AI.
>> Yeah.
>> You remember that? Now, it seems like
the conversation has changed a lot. Like
now the the US government just told
Enthropic they have to shut down their
AI because they were worried that bad
actors would use it for cyber attacks.
you know, the government is like waking
up and doing more stuff than we expected
uh already. And we're actually hopeful
that that trend will just continue and
that before it's actually too late,
there will be very serious conversations
happening inside the government and
outside the government and in the
broader society about all of these
issues and trying to chart a course that
is um avoids the loss of control and
concentration of power risk that we
mentioned. you um you've spent well must
be almost coming up to 15 years thinking
about this stuff. Um if this here was a
button and if you press that button your
plan S would occur and it would shut
down every data center that is currently
training a frontier AI model uh for
good. There would never be any other
>> AI labs um working on these problems.
Would you press that button? I was I was
about to slam it until you said for
good.
>> Okay.
>> Like I think I think if it was a sort of
temporary shutdown, I would totally slam
that button because we are not ready to
do this. You know, like civilization is
not ready to have these companies
automate themselves and then get smarter
and smarter and then have the super
intelligent like no. There's a bunch of
reasons why that's really uh dangerous.
But I would be at least hesitant to
press this button if it permanently
foreclosed the possibility of ever doing
it again. for sure.
>> But but if you think that plan D is
probable, which is this race we're on to
super intelligence,
>> if I had a choice between D and S, I
think I would press it.
>> Well, it's it comes down to what you
think, right? Cuz if you think that's
that is what's going to happen, plan D
and the only alternative.
>> I didn't say this is what's going to
happen
>> proistically.
>> Yeah. Yeah. Like like I'd be like this
is the most likely, maybe this is the
second most likely, maybe this is the
third most likely. They are all
possible. So with your current
perspective on whatever one you think is
going to happen, would you press the
button? I'm giving you an S, a definite
S or whatever you think is going to
happen.
>> That's tough.
[sighs]
>> What is the scope of the shutdown? So is
it
>> it's no one can train an AI model again,
ever again.
>> That's real rough because like I said,
there's loads of benefits that we could
get from AI if we do it right. Um,
>> I think I I've almost put you in the
position of Sam Holtman.
>> Yeah. [laughter]
>> To some degree.
>> Yeah.
>> Um, let me Do you mind if I just take a
moment to think about this?
>> I think I prefer you to think.
>> Yeah.
I think I would not press the button,
but I'm I feel very torn about it. Um,
the reason why I think I would not press
the button is that I still have
substantial hope that we can get
something much better than this,
something more like this. And I think
that
basically I think that if we don't build
powerful AI systems eventually,
then
we're probably going to die as a
civilization.
eventually, you know, like 100 years
from now, 200 years from now, something
like that, like nuclear war, pandemic or
something, you know,
I I don't think human civilization right
now is like super super stable. Um, and
so I think that basically what I was
about to say was the possible benefits
for posterity and for all the billions
and billions of people who could live in
the future outweigh the like
the current level of risk. But actually,
>> I've heard that narrative before.
>> Yeah. I don't know. Like,
>> yeah, like maybe maybe it's just like,
nope, the people right now are the
people we should prioritize. People
right now are in grave danger. They're
going to be fine for at least the next
couple decades.
So, never mind posterity.
Prioritize the people right now. Um, and
people right now definitely don't want
to do this lottery, I would say. Um,
[sighs and gasps]
yeah, you've really asked me a tough
question.
>> So, would you press the button if that
was the button?
>> Probably not, but I would feel very
torn.
>> Okay. So, what I I always think about
the personas of like the audience that
are watching and these are, you know,
they're they're very curious people,
especially on the subject of AI as we've
seen, but they they want to know like
what it means for them. I think a lot of
them also want to know what they can do.
>> Oh, yes. Yeah. What can people do? Well,
I think that if you either have talent
or passion, you can get directly
involved. There's lots of organizations
that are worried about these things and
that are trying to do something about
it, like political advocacy or technical
research or like building useful tools
that will hopefully help people be
better and stuff. But if you don't want
to like make any major career changes or
or things like that, then I would say
just pay more attention to these issues
and talk about them more with people. do
stuff like, you know, emailing your
congressman or whatever, it it doesn't
change things that much, but it does
help. I think that especially for this
particular issue, the core problem is
that people aren't taking it seriously
yet. Like if the sorts of things that I
was just saying to you for the last hour
or two were just like top of everybody's
mind,
we wouldn't even be here. Like there
there would already be much more
significant regulation in place, you
know, and not only would there be more
heavy regulation in place, but there
would have been better regulation in
place that's less, you know, less like a
cudgel and more like a scalpel and
that's like more sensitive to what's
actually bad and what's not so bad and
so forth. And there'd be more expert
people in the government and advising
the government and so forth. So, just in
general, like the more people wake up to
these concerns and to these projections,
uh, I think the more likely it is that
we can do good stuff before it's too
late.
>> What about how they should vote at the
polls? We've got an election coming up
in the United States in a couple of
years time, but there's elections
happening all over the world all the
time.
>> You should ask your candidates what they
think about all this AI stuff. You
should try to get them to like have
opinions, and then you should vote for
the candidates whose opinions are better
on this topic. This is the most
important thing happening. uh in our
lifetimes, probably in all of history in
fact and it's very important that it go
well and so it's what all the all the
leaders of all the countries should be
thinking about and making plans for.
>> Isn't it such a weird thing to be alive
at this moment in time? Like I was
thinking about all the times that I
could have been born and I guess my
ancestors probably thought the same, but
I was thinking as you were speaking I
was like I think it's when you referred
to it as like the final show.
>> Yeah.
>> What was the phraseology you used?
>> I said the the clim run up to the climax
or something.
>> Yeah. I mean, what a what a crazy thing
to be born in the run-up to the climax
where everything you're describing here
is within my lifetime, conceivably.
Hopefully.
>> Yeah.
>> Um or maybe not hopefully.
>> What a crazy time to be alive.
>> Certainly.
>> I noticed that when I met asked you if
you had kids, your demeanor changed
quite considerably.
>> Well, it's Yeah.
>> It's like you dropped into a different
state. Obviously, that's been central to
the
rumination that you've been
experiencing. Well, it's a sad topic,
right? Like when when I had kids, like
the reason to have kids is in large part
about the future, you know, like it's
not just like a cuddly thing to have
with you in the moment. It's because you
have all these hopes and dreams about
how they'll grow up and how they'll go
do their own thing and be their own
person and stuff. And because of what's
happening with AI, I think a lot of
those dreams are in jeopardy.
>> Presumably, you still would have had
kids.
>> I've actually flip-flopped on this
occasionally. Yeah. Basically, the
topline answer is I'm not sure. the my
first child was had we we had her when
we were um in 2019 she was born 2019.
>> Yeah.
>> So this is before my timeline shortened
a lot. So at this at this point I was
interested in AI I was tracking the
field. I was making forecasts but I
didn't like actually expect it to happen
soon
>> you know.
>> Mhm.
>> And then this caused like when I when I
did start thinking like oh my gosh it's
going to be happening like real soon um
like by 2030 you know. um that caused
some reconsidering and so I basically
told my wife like let's not have any
more kids. It's too uncertain, you know.
But that turned out to be really hard
because especially for my wife, like we
already had one kid and like no
siblings. Um so eventually I sort of
gave in and was like, "Okay, well, you
know what? We already have one. It's
gonna be all right. Like maybe maybe the
future will be good. And even if it's
not like, well, we're all in the same
boat together.
>> It's quite chilling what you're saying.
It's chilling because you know more than
me. And if you're at home saying to your
wife, listen, maybe we should pause on
having more children and building a
family because of what's going on with
AI.
>> To be clear, this is Yes. I mean, yes,
it's very concerning. I am I am chilled.
Uh this is bad. This is what I've been
saying. I hope things go well. I think
things might go well. Um I think that
there's a lot we can do to like steer
things in a better direction. I mean,
one of those things as well, I have to
say, is just speaking about it. It's I
think a lot of the progress we've seen
with governments waking up and, you
know, we've seen certain things with
people booing certain people at certain
events.
>> Yeah.
>> Um is is a downstream from people like
yourself actually coming on shows like
this and all the other podcasts and
>> Yeah.
>> telling us what's going on.
>> Yeah.
>> Because elsewhere, to be fair, we're
going to be gaslighted by the people
that have the biggest PR machines.
>> Yeah. So, um, I often I think it's
probably worth me saying I find myself
kind of in two minds because I'm an
entrepreneur and I'm I'm an investor.
I'm an investor in probably more than
100 companies now. And so many of those
companies are using AI. I invested in
Grock, the inference chip company. I've
invested in SpaceX which now own another
Grock and they're doing AI. I use AI
every day in my life. I've been using it
through this conversation to understand
different things that you've said.
>> So that's one side of me which is like
business builder entrepreneur who has
seen the benefits of AI in my own life.
And then there's the other side of me.
And it's funny because I think sometimes
people think you have to pick a camp.
But through all of my life, even when I
was a social media CEO and I was saying,
by the way, listen, I'm building a
social media business, but I think
there's some downsides to social media.
I find myself at the same moment where
I'm like, I build with AI. I have AI
investments. And at the same time, as a
civilian, I'm like,
>> yeah, I mean, I think that is attention.
I think that there's there's different
ways you can draw the line. So, and I
know lots of people who draw the line in
lots of different ways. So, like there's
some people who just like, I'm not going
to use AI. I think this stuff is bad um
and on a bad trajectory, so I'm going to
like boycott AI, right? I'm not one of
those people. I use AI a lot. We all do
at AI features project. Um it's helpful
for a lot of our work. The opposite end
of the spectrum is
people being like, well, it seems like
it's on a trajectory to happen, so the
thing to do to make it go well is to
like get involved and accumulate power
and try to like steer it from the
inside. Mhm.
>> And so I'm going to go work at OpenAI or
Anthropic and like try to like climb the
ranks and then like you know be someone
who matters when the important decisions
are being made. And I know loads of
people like that. That was like what I
was doing when I was that wasn't what I
was doing exactly but like
>> that was the path
>> that was like that was I mean this in
some sense this is what the whole
narrative of the companies are right
like this is why they tell themselves
it's okay to do what they're doing is
that they're worried about the other
guys you know and so like all these
people are are deciding like we're going
to like lean really hard into it. are
going to like be there in the room when
the important decisions are being made,
you know? So, there's a whole spectrum
and I'm sort of like somewhere in the
middle. Like, I'm not at the AI
companies. I'm not helping them go
faster. Instead, I'm talking to the
broad public and trying to advocate for
what I think is the my current best
guess as to the way out, you know, the
way forward. Um, but I'm not like
boycotting all the AIs. I'm I'm not
like, you know, uh trying to I'm not
refusing to like engage with it in that
way.
Do you think it's too late?
>> No, I don't think it's too late. If I
thought it was too late, I wouldn't be
here. H
>> Where would [clears throat] you be?
>> With my family.
>> What's your closing message to the
general public? If you had to have a
closing statement to them,
>> maybe I would say that like you're going
to hear a lot of things and you already
have been hearing a lot of things about
AI and it's going to sound like science
fiction, but sometimes things which
sound like science fiction happen in
reality. And in fact, many times
historically things used to be science
fiction have then become reality. And
people need to stop thinking about what
does or doesn't sound like science
fiction and just start thinking about
like the trends and you know the actual
trends that this technology is on and
reading and forecasting how it's going
to go and then taking seriously the
possibility that it could go something
like this and then thinking about what
should be done about that.
>> And where would you direct them to get
more information?
>> You can go to a27.com to read our
previous scenario. You can go to
a2040.com plan A to read our new
proposal for what is to be done. Um,
these things are not just a sci-fi
story. They also have lots of like
explainers and links to other things.
And so they're kind of like a nice
jumping off point to to learn about all
of this stuff. M
>> um if you want I could um after this is
over like give a reading list of like
other papers and articles and
>> blogs to follow and so forth
>> and I'll link them all below in the
comment section. So if you're listening
now go ahead and take a look at the
comment se the um description of this
episode and you'll see a bunch of links
which is Daniel's recommendations of
what you should read. You know I think
it's it's just a really really great
moment in time to get educated on this
stuff. Um humans have an inclination
because of cognitive dissonance where we
feel uncomfortable about something to
bury our heads in the sand and avoid it.
But actually I think this is one such
time to do the very opposite for many
reasons to to inform yourself so you
know what actions to take but also
because AI you know unavoidably is going
to be a huge part of all of our lives
and careers.
>> Yeah. Yeah. Thank you. And that that's a
good way to to say it. It's going to
matter a lot. It's going to it's going
to be everywhere soon and um we need to
do something about it before it's too
late. And
>> what about AI future project?
>> That's our organization. We spent a year
writing AI 2027 after I left OpenAI and
then we spent another year writing AI
2040 plan A.
>> Daniel, thank you.
>> Thank you.
>> Thank you for all the work that you do.
I can see how much you care about this
stuff and it's your care. It's funny.
Care itself makes others feel care. and
um seeing how personal this is for you
and seeing how much you've dedicated
your life to this, but also hearing that
you you basically walked away from $2
million to be able to speak to the
public about this information
[clears throat] is incredibly admirable
and uh I I think voices like yours are
more important now than they've ever
been on this subject. So, please do keep
fighting the fight that you're fighting.
And that's one of information. It is of
honesty and it is uh of saying what what
is often the quiet part out loud and
doing really really smart research. I'll
link everything we've discussed today
below. Um, and I hope we can chat again
sometime soon.
>> Thank you.
>> YouTube have this new crazy algorithm
where they know exactly what video you
would like to watch next based on AI and
all of your viewing behavior. And the
algorithm says that this video is the
perfect video for you. It's different
for everybody looking right now. Check
this video out and I bet you you might
love